That's just one logistical problem (of many). The real reason we don't have flying cars is because we don't want them bad enough. If we spent enough capital, we could solve all the logistical and technical challenges and have flying cars. But the cost and effort it would take is so significant that nobody has yet had a good enough reason to do it.
Planes are good enough for going a long distance fast, and cars/trucks/trains are good enough for going a long distance slow. We could make a flying car, but for what? For individual people to go a moderate-distance fast? To go a short distance really fast? Cars and planes are good enough for our needs today without the huge investment in development of a new tech.
This is exactly the same reason internal combustion engines have ruled the roads for 100 years. Electric cars were preferred over ICE 100 years ago, but gasoline enabled us to go further for cheaper, so we accepted all the downsides, and industry made it convenient. It's only because we're suddenly afraid of our climate killing us that we're switching back to electric.
The actual "physical limitation" is that the typical average human simply cannot be trusted with a flying car. They can barely be trusted with regular ordinary run-of-the-mill cars.
