I'm not sure if it is counter-intuitive. With text you can describe exactly what it is you are selling and thus provide a reason for clicking the banner.
Stuff that is mainly graphics is easy to just mentally filter out with the rest of the website chrome.
To take it a step further, we are so used to obnoxious banner ads on the internet that I think most people automatically ignore them.
Just a guess, but the less "bannery" the ad is, probably the better chance it has of being noticed. Ironic, since banners got so obnoxious in the first place as an attention-grabbing mechanism.
Also, have you tried using CPM rather CPC?