I don't. Each agent gets one key, and that key alone. That's what helps.

https://github.com/ccontavalli/ssh-ident is what I use to manage those agents - one agent per key.