How skewed is the perception? Riding transit isn't a death sentence, but it does suck: slow, unreliable, expensive, dirty/smelly/germy, and with the potential of being a victim of violent crime.
It really depends on the city. Some places it's just fine, and pretty popular, some places it's a rolling bathroom and pretty popular - the problems are the ones where it is unpopular and a rolling bathroom; that's the hardest to expand - which means it is perpetually stuck in a dead zone where it can't be used because it can't be expanded because it can't be used.
Meanwhile things like the San Diego Trolley has gone from 16 miles and 17 stops in 1981 to 47,000 people a weekday in 1996 and to almost 70 miles and 62 stations today, with ridership peaking at around 38 million a year in 2019 (and slowly recovering back up).