The more I read the rationalist cult's writings the more I question if they are trying to communicate with outsiders at all; it's like a parody of the climate change problem, "don't look up", etc. Like the Jehovah's Witnesses or Scientology it is becoming all about how the "Wogs" are clueless and will be punished in the end.
(1) I think GPT-3 has no clothes (it's real superpower is that it gets "neurotypical privilege", whenever it seems to get something 80% right people give it credit for the other 20%), current neural networks are structurally inadequate for AGI, adding more neurons will approach an asymptote in performance not lead to a 'singularity'. The whole 'singularity is neigh' concept has no basis.
(I'll grant that if there is a structural change you might get something on a better trajectory than GPT-3... An example a structural feature that matters is "does it run programs that are guaranteed to complete in bounded time" or "can it solve constraints?" or "can it revise an early stage interpretation if it runs into problems at late stages")
(2) That particular essay is hard to rebut because it reads like Bari Weiss and Matt Taibbi made 100 embryos and then somebody scored them all and picked the worst. It seems to start out with some individual greivance somebody had once but then it was processed through a Russian doll factory that got attacked by a paperclip optimizer. It would take hours to really make sense of it... if it made any sense.
