But adding a detour is easy in a car (even during rush hour - congestion is mainly on big axes). Restaurant doesn't need to be right where the traffic is.
Congestion is mainly anywhere near the big axes, which is exactly the problem. Even if the place is "only" three blocks away, that's three stinking red lights in each direction. Meanwhile you may not have any strong preference between two similar burger joints, so if one is directly on the main road and the other isn't, where are you stopping?