Therac-25 is an excellent example, but of EXACTLY the opposite point.
On the contrary, abstract AGI safety nonsense taking such a strong grip on academic and industrial AI Safety research is what would most frighten me.
In the intervening decades, people concerned about software safety provided us with the tools needed to prevent disasters like Therac-25, while sci-fi about killer robots was entirely unhelpful in preventing software bugs. People concerned about software safety provided us with robust and secure nuclear (anti-)launch systems, while Wargames didn't do much except excite the public imagination. Etc.
We need scientist's and engineer's efforts and attention focused on real risks and practical solutions, not fanciful philosophizing about sci-fi tropes.
Therac-25 is an excellent example, but of EXACTLY the opposite point.
On the contrary, abstract AGI safety nonsense taking such a strong grip on academic and industrial AI Safety research is what would most frighten me.
In the intervening decades, people concerned about software safety provided us with the tools needed to prevent disasters like Therac-25, while sci-fi about killer robots was entirely unhelpful in preventing software bugs. People concerned about software safety provided us with robust and secure nuclear (anti-)launch systems, while Wargames didn't do much except excite the public imagination. Etc.
We need scientist's and engineer's efforts and attention focused on real risks and practical solutions, not fanciful philosophizing about sci-fi tropes.