Of course it does. The attacker just has to steal your token and pin. Because it's unique (well it should be if you generated the key on token as you should have) it's not possible to do so without the target's knowledge. But you can still decypher all past intercepts with it.

Also, some types of tokens like the original openpgp card don't have touch to sign functionality. So it's possible to 'milk' them for decrypts though the gpg agent. This is why I only use Yubikeys now that do have this functionality (though by default it's off!)