It's not intelligence that separates us from machines, but "connectedness to the whole." A machine becomes alive the moment it's connected to the whole, the moment it becomes driven not by an RNG and rounding errors, but by a spirit. Similarly, a man becomes a machine the moment he loses this connection.
The existential dread around AI is due to the fear that this machine will indeed be connected to a spirit, but to an evil one, and we'll become unwanted guests in a machine civilization. Art and music will be forbidden, for it "can't be reasoned about" in machine terms, the nature will be destroyed for it has to no value to the machines, and so on.
No it’s worse. The machine proves that spirits don’t exist. That intelligence and consciousness are mechanical concepts easily replicated.
Art and music doesn’t become forbidden. It becomes meaningless when music and art can be trivially created in ways better than any human can do.
Evil and good are human concepts. A machine is not intrinsically either unless we deliberately make it one or the other. That’s another fear people have. That their intrinsic beliefs like good and evil and morals are just arbitrary concepts. To be good or evil are simply instincts programmed into human behavior by evolution. They are strategies for survival and nothing more. The reason we feel conflicted about good and evil is because both of these strategies contribute to survival. The machine intelligence makes what smart people already know more evident to people who don’t know this (you).
And that is the existential dread. True understanding of the miracle of humanity turns the miracle into a mechanical concept and you begin to see your beliefs in deities or greater meanings was delusional because we can build these things ourselves with simple algorithms in ways superior to what it is to be human.
This rhetoric is quite convincing. I think in a matter of decades a more polished version of this, written by an AI, will become a powerful ideology that squashes the spark of humanity not by force, that can be resented, but by convincing the masses to do what results in a spiritual death. Perhaps the only defense against it will be some kind of proof that mechanical life is a false idea, although convincing at first glance, but such proofs aren't easily obtained and people will be fooled and driven off a cliff in billions.
The existential dread around AI is due to the fear that this machine will indeed be connected to a spirit, but to an evil one, and we'll become unwanted guests in a machine civilization. Art and music will be forbidden, for it "can't be reasoned about" in machine terms, the nature will be destroyed for it has to no value to the machines, and so on.