Alright, that vein of arguments are the biggest cop-out ever. It's like trying to say the 4th amendment doesn't apply to houses because no one forces me to have a house.
The constitution recognizes the inherent right to be secure in your personage and effects. This should include your financial information.
I think it odd that people accept the abrogation of their rights w.r.t. driving. Yes, the government can definitely tax you for road use, but imagine having to carry an ID around that proved you paid your income tax, and that you were required to be IDable at almost all times to be sure you were not committing tax evasion.
No one forces you to have a bank account, just like no one forces you to have a driver's license - which also involves giving up some rights, btw.