You're assuming "retirement" is still at 65, but increasingly it isn't. The median is actually highest at the oldest age group on that chart.
Everybody also forgets to include the net present value of social security (basically an annuity you were forced to buy), which is a disproportionately large amount of the net worth for lower income people both because they don't have as many other assets and because the income cap limits how much it adds to the net worth of the people at the top. (Though people would generally have a higher net worth without it; it pays back less than you'd have from investing the same money in an index fund.)
The median net worth for a 60-64 year old is 225k a far cry from 500k. And 66% of the population have less than 500k at that age.