The issue is that software making these decisions can't possibly have the full context when making the decision because it was coded long before the decision point occurred.
What does that mean in practice? Have you ever saw a flash in the corner of your eye, looked for a second, then realised it was just a reflection? For that split second, your brain identified a potential threat and was trying to quickly decide whether to hide, fight or run. But underneath that was a nagging of "it's probably nothing," that caused you to delay pulling out your gun and opening fire on a passing car.
A computer programme may or may not have this coded in. A machine designed for an active war-zone, or a car assuming it'll only ever auto-pilot on a motorway, is used in a suburban area. That same programme that's fine in its original intended location suddenly opens fire on a grandma with a shiny stroller, or swerves into a pedestrian because "pedestrians don't walk on motorways."
> Have you ever saw a flash in the corner of your eye, looked for a second, then realised it was just a reflection? For that split second, your brain identified a potential threat and was trying to quickly decide whether to hide, fight or run. But underneath that was a nagging of "it's probably nothing," that caused you to delay pulling out your gun and opening fire on a passing car.
.. or vice-versa. The average war has plenty of stressed people firing semi blindly at half-identified shapes. And if we want to be literal, there's the case of Lee Clegg and the exact circumstances in which it is legal or not to open fire on a passing car and kill a teenage girl.
I’m not denying that awful shit like this happens with humans. I’m pointing out the moral hazard in turning it into a preprogrammed system. We’ve seen software bugs empty bank accounts simply because running that faulty system for 5 minutes iterates 100,000 times. Now give the power to decide who lives and dies and it’s only limited by the ammo it holds.
What does that mean in practice? Have you ever saw a flash in the corner of your eye, looked for a second, then realised it was just a reflection? For that split second, your brain identified a potential threat and was trying to quickly decide whether to hide, fight or run. But underneath that was a nagging of "it's probably nothing," that caused you to delay pulling out your gun and opening fire on a passing car.
A computer programme may or may not have this coded in. A machine designed for an active war-zone, or a car assuming it'll only ever auto-pilot on a motorway, is used in a suburban area. That same programme that's fine in its original intended location suddenly opens fire on a grandma with a shiny stroller, or swerves into a pedestrian because "pedestrians don't walk on motorways."
That's what we worry about.