I'm anxious, excited, and nervous about AI, but this feels more akin to luddite-thinking, the world's about to shift, but I'm hoping the shift is a positive one.
I agree to the point that it is totally unrealistic that it shuts down. There are no difficult barriers for entry into this space. No possible way of containment other than imposing hardware restrictions.
However, it is indeed a shift, and a massive power shift. With any substantial increase of power comes abuse of that power. The more power we have there is a tendency for humanity to manage it unwisely.
Hardware restrictions are unrealistic given all the GPUs that already exist. It's not like centrifuges which are big and few in number. RTX 4090s are everywhere.
Depends on what is required for AGI/ASI. Some say we still need much more power, others think we are almost there. So, yes, unrealistic if we are almost there already.
Given that GPT-4 is purported to be not bigger than GPT-3, but simply a better trained one or with some minor modifications, I'm inclined to believe that we are almost there and existing hardware does most of what is needed, even though better hardware would make things cheaper.
The author and most of his associates generally qualify as techno-utopians, and speak routinely of the "glorious transhumanist future".
Smarter-than-human AGI really is different from all previous technologies, in much the same way that homo sapiens is different from all preceding life on earth.
