A big thing is not that the example is missing, but that it counts as a negative example.
I.e. if during training a ML system notices the ambiguous combination (i.e. a woman pushing a baby stroller or a crowd) and marks it as a pedestrian, then it gets penalized in a manner that teaches it to ignore these ambigious combinations and treat it as nothing; while in practice it should probably treat such ambiguous combinations as even more "avoid-worthy" as an ordinary pedestrian.
The problem is that the default assumption is "clear road, you can drive there" - so what we need isn't "pedestrian detection" that finds pedestrians and only pedestrians, we need detection of random stuff that you shouldn't drive over. If a kid is wearing a weird Halloween costume, that doesn't look like a pedestrian, but it is one; If somebody has set up a tent in the middle of a supermarket parking lot, that's not a pedestrian but it should be avoided just like one.
That's a feature, not a bug. Swerving for potholes can be very dangerous, more dangerous than having undercarriage damage. If a pothole surprises you enough that you have to swerve you were either not paying attention to the road or you are following too close.
Regardless of what's the appropriate action to take given the context (ignoring, swerving, slowing down, a timely proper change of lane) it's probably not controversial that potholes should be identified by a car vision system and taken into account. And from a computer vision perspective there's no qualitative difference between "just" a deep pothole and a lane-wide ten foot deep sinkhole or a construction pit that's unmarked for some reason, it's just a matter of size.
tbf there's still the evaluate whether slowly changing your lane position to avoid the pothole will impede or confuse other road users, and do so only if that isn't the case option to avoid potholes, and I don't imagine [semi]autonomous driving systems do that either?
I.e. if during training a ML system notices the ambiguous combination (i.e. a woman pushing a baby stroller or a crowd) and marks it as a pedestrian, then it gets penalized in a manner that teaches it to ignore these ambigious combinations and treat it as nothing; while in practice it should probably treat such ambiguous combinations as even more "avoid-worthy" as an ordinary pedestrian.
The problem is that the default assumption is "clear road, you can drive there" - so what we need isn't "pedestrian detection" that finds pedestrians and only pedestrians, we need detection of random stuff that you shouldn't drive over. If a kid is wearing a weird Halloween costume, that doesn't look like a pedestrian, but it is one; If somebody has set up a tent in the middle of a supermarket parking lot, that's not a pedestrian but it should be avoided just like one.