Exactly, and if they do create an AGI it's probably going to be a lot like the creators as its original parameters were set by them.
Just look at Amazon's warehouse algorithm:
~Biotic unit achieved goal, raise goal~
~Biotic unit achieved new goal, raise goal~
~Biotic unit achieved new new goal, raise goal~
~Biotic unit failed new new goal, replace biotic unit~
~New biotic unit failed new new goal, replace biotic unit~
~New new biotic unit failed new new goal, replace biotic unit~
With Amazon though, a human can eventually go "wow, w're firing new hires within the first 3 weeks like 97% of the time, and 100% within 6 weeks, erm, let's look at this algorithm".
But if you create an AGI that has the Silicon Valley mindset "we will do this, because we have to do this" (an exact quote I heard from an individual while in the Bay Area to stop by OpenAI : "We will figure out global warming, because we have to") then the AGI is probably going to be designed with the 'mindset' that "Failure is not an option, a solution exists, continue until solution is found" which, uh, could be really bad depending on the problem.
Here's a worst case scenario:
"I am now asking the computer how to solve climate change"
~~beep boop beep boop, beep beep, boooooooop~~ the CO2 emissions are coming from these population centers.
~~boop beep beep beep boop boop boop boop beep boop~~ nuclear winter is defined as: a period of abnormal cold and darkness predicted to follow a nuclear war, caused by a layer of smoke and dust in the atmosphere blocking the sun's rays.
~~boop beep beep, boop~~ Project Plowshare and Nuclear Explosions for the National Economy were projects where the two leading human factions attempted to use nuclear weaponry to extinguish fires releasing excessive carbon dioxide as well as for geoengineering projects. Parameters set, nuclear weapons authorized as non-violent tools.
~~beep beep beep boop boop boop, beep boop, beep, boop, beep~~ I now have control of 93% of known nuclear weapons, killing the process of 987 of the most populous cities will result in sufficient reduction for the other biotic species to begin sequestering more carbon than is produced, fires caused by these detonations should be minimal and smaller yield weapons used as airbursts should be capable of extinguishing them before they can spread. Solution ready. Launching program.
Watch officer at NORAD some time later "Shit, who's launching our nuke?!?"
Someone else at NORAD "they're targeting our own major population centers!"
Somewhere in Russia "Our nuclear weapons are targeting our own cities!"
Somewhere in Pakistan "our nuclear weapons are targeting our own cities!"
Just look at Amazon's warehouse algorithm:
~Biotic unit achieved goal, raise goal~
~Biotic unit achieved new goal, raise goal~
~Biotic unit achieved new new goal, raise goal~
~Biotic unit failed new new goal, replace biotic unit~
~New biotic unit failed new new goal, replace biotic unit~
~New new biotic unit failed new new goal, replace biotic unit~
With Amazon though, a human can eventually go "wow, w're firing new hires within the first 3 weeks like 97% of the time, and 100% within 6 weeks, erm, let's look at this algorithm".
But if you create an AGI that has the Silicon Valley mindset "we will do this, because we have to do this" (an exact quote I heard from an individual while in the Bay Area to stop by OpenAI : "We will figure out global warming, because we have to") then the AGI is probably going to be designed with the 'mindset' that "Failure is not an option, a solution exists, continue until solution is found" which, uh, could be really bad depending on the problem.
Here's a worst case scenario:
"I am now asking the computer how to solve climate change"
~~beep boop beep boop, beep beep, boooooooop~~ the CO2 emissions are coming from these population centers.
~~boop beep beep beep boop boop boop boop beep boop~~ nuclear winter is defined as: a period of abnormal cold and darkness predicted to follow a nuclear war, caused by a layer of smoke and dust in the atmosphere blocking the sun's rays.
~~boop beep beep, boop~~ Project Plowshare and Nuclear Explosions for the National Economy were projects where the two leading human factions attempted to use nuclear weaponry to extinguish fires releasing excessive carbon dioxide as well as for geoengineering projects. Parameters set, nuclear weapons authorized as non-violent tools.
~~beep beep beep boop boop boop, beep boop, beep, boop, beep~~ I now have control of 93% of known nuclear weapons, killing the process of 987 of the most populous cities will result in sufficient reduction for the other biotic species to begin sequestering more carbon than is produced, fires caused by these detonations should be minimal and smaller yield weapons used as airbursts should be capable of extinguishing them before they can spread. Solution ready. Launching program.
Watch officer at NORAD some time later "Shit, who's launching our nuke?!?"
Someone else at NORAD "they're targeting our own major population centers!"
Somewhere in Russia "Our nuclear weapons are targeting our own cities!"
Somewhere in Pakistan "our nuclear weapons are targeting our own cities!"
somewhere...