When she beings with 'I don't understand', the Doctor plainly answers, "I don't blame you."

Pudding-brains rarely understood anything. But since he had already met his younger self, he could answer the question.

"There's not two of me. We're the same person. I'm just... a great deal older than him."

And with an entirely different face and very different personality, but they're still the same man. Which means, it's not a 'police box, too', it's the same box.