Yes. There as difference between unfair and unreal; someone who is malnourished when growing up will forever likely be weaker than someone who received a proper sequence of meals.

We should perhaps recognize that and try to compensate for it, and it's not a value judgement on the person so afflicted, but pretending it doesn't exist just confuses matters.

If the difference is real I don’t think the test is the place to compensate as its function is to select people who will succeed in that area.

That's been the entire fight over the last 20+ years, does the test identify anything real and if so, what should be done with it (equality of outcomes vs equality of opportunity, e.g.).