Avoiding fruits only comes after cutting out all processed sugar, or added sugar on the label. Then you should cut out all processed grains. Flour is basically powdered carbohydrate, which spikes your insulin in a manner not much different from table sugar. Then you can look at sugar. I would still include berries, they're low cal and the antioxidant content is insane.

Basically, you need to earn your carbs. Sugar is fine if you burn it off, that's why athletes drink Gatorade mid workout. If you eat an apple before a walk/jog/swim/etc, there's no issue. If you eat a pile of bananas while doing nothing, that glycogen has to go somewhere.