If you really wanted to quantify the impact in a completely market-oriented way, you'd have factor in not just the road construction and maintenance and environmental impact, but also the negative value of the road noise, high danger to others (much higher per mile traveled per person than other modes), worsening of each other mode (car-dominant street design makes other modes slower and less safe), and value of the land (car infrastructure takes up far more space per traveler than other modes).

By the time you're done factoring those in, driving would be expensive indeed, probably similar to the gas prices in Europe.