I do the opposite. I almost never brush (maybe once a month), but get my teeth cleaned twice a year religiously.

Never had a cavity, (almost) never have bad breath. Normal diet.

My guess is that the specific bacteria in your mouth make all the difference.