If you charge 3% just to credit card users, they are now subsidizing the cash customers. I am guessing you are assuming the cost of cash handling to be zero.
Not to mention, as someone who spent a few years in that industry, in many cases those cash customers are not getting reported to the IRS, or local authorities for sales tax.
So they could already be "saving" up to 10% or more on cash customers...