Well, if an autonomous vehicle outfit were running unmonitored Level 4 vehicles on public roads using only an open source data set I'd be worried. Even if it was labelled thoroughly and correctly, there isn't nearly enough data in any open source dataset to train an autonomous vehicle perception system that can operate safely without human supervision. This is not a safety critical issue.
The Uber vehicle that ended up killing the pedestrian in Arizona was running under conditions similar to the ones that you outline here. The only notable difference was that a person was monitoring the vehicle.
And it will happen again. There were cases of people forgetting Nuclear Weapons on a runway for a whole day by accident. People, eventually, will fuck up. Mislabelled datasets will be used.
