Milk is a funny thing, being a rich medium for bacteria.

Take a pint of fresh-from-the-cow milk and a pint of your typical store-bought pasteurized milk and leave them open on the counter for 2 or 3 days at room temperature. The former will turn into a tart, pleasant smelling mix of curds and whey I'd be willing to taste and possibly consume while the latter will be a smelly, gag-inducing spoiled mess that will almost certainly make you ill.

I've witnessed this first-hand.

Raw milk is loaded with beneficial bacteria that can be naturally culture itself to various ends, more often then not out-competing environmental bad bacterial. Pasteurized milk, devoid of most bacteria to start, will (most often) be overrun by bacteria that will make people ill. I does not surprise me that our "normal" milk is a non-trivial vector of food-borne illness.

While I can understand the assumption Raw milk is far more likely to be associated with food born illness for exactly the reasons you mention. However while overall consumption well below 1% it's still responsible for a and much higher than expected fraction of illnesses and a few deaths.

Importantly, people are more likely to die from food that looks good but can kill them than they are to eat food that looks and tastes nasty.