Let’s say there is an event in NYC. Within milliseconds of the event, all the robot cars can be notified and their human drivers can take over, only the cars immediately in the vicinity of the event need robot operators. It’s not straightforward, but it can be done.
Even if you assume requiring human takeover to be a relatively uncommon event (whatever that means), as soon as you posit it as something that will be needed from time to time, you've significantly constrained the car's usage models. You now must have a licensed, unimpaired driver in the car at all times. Even if they don't have to be paying attention, this means no empty cars, no unaccompanied children, no "driving" home from the night out, etc.
Oh, I would too assuming it were relatively affordable. I'd be pretty happy with one that even just let me doze off when highway driving in a limited set of weather conditions.
I was just pointing out that, if you can't guarantee you won't need to handoff to a physically present driver, then there are a lot of things you can't do with the car even if needed interventions are just an occasional thing.