I think to argue usefully about consciousness you've got to be able to define what you mean by it. If you use in the sense of a boxer is knocked unconscious as he's not aware of anything much versus conscious where he knows what's going on and can react and punch back, then AI systems can also be aware or not and react or not.
If you say it's all about the feelings and machines can't feel that way then it gets rather vague and hard to reason about. I mean they don't have much in the way of feelings now but I don't see why they shouldn't in the future.
I personally feel both those aspects of consciousness are not woo but the results mechanisms built by evolution for functional purposes. I'm not sure how they could have got their otherwise unless you are going to reject evolution and go for divine intervention or some such.
Consider a universe purely dictated by the mathematical laws of physics. It would be indistinguishable from our own to an observer, but such a universe would effectively be a fixed 4D structure, a statue incapable of experience. You have experience, yes? You think therefore you are. There exists something beyond maths and physics, experiencing the our universe, and you are that thing. How could such an entity develop from physical processes?
> such a universe would effectively be a fixed 4D structure, a statue incapable of experience
This claim requires proof. When I consider such a universe, it is obvious to me that such a universe would contain entities capable of experiencing it.
Why? We know that there are things which can experience but not be conscious, for instance simple robots. Do you think it would be impossible to have a universe full of people with no souls, who simply react to inputs like a robot? I think such a thing could exist and would be indistinguishable from out own.
A simple robot has neither consciousness nor experiences because it lacks the notion of self-identity. On the other hand, "soul" is itself a concept that is begging the question. If you consider consciousness an emergent property of sufficiently organized matter, then a world you describe is literally impossible.
AIs have self-concept, but I don't think you'd describe them as conscious. Likewise many animals do not recognise themselves in mirrors but still seem to experience the world.
> If you consider consciousness an emergent property of sufficiently organized matter, then a world you describe is literally impossible.
You are applying the word "impossible" to reject a perfectly valid consideration. It seems to be the case that in our universe, sufficiently organised matter can become conscious, but we can easily imagine one without this property. If you imagine a universe that only follows the laws of physics as we currently understand them, then it would not be the case. The universe would just be atoms bouncing off eachother. The idea that these atoms can come together to produce something which is more than a soulless automaton is an addition you are making to the laws of physics, not something present within them. We can imagine a world without this addition, just as we can imagine a world where gravity repels instead of attracting. We can see examples all over the place of things which act to approximate consciousness but are not conscious. It is no stretch to imagine a universe where this is a continuous scale, and consciousness doesn't magically emerge at some point along it.
There is a good reason why most people consider consciousness arising from the laws of physics as we currently understand them to be totally untenable. According to the known physics, the human mind is essentially just an advanced computer. To say that consciousness arises in any advanced computer is highly problematic. Take for instance this[1] popular comic. Do you honestly believe that this endless field of rocks could experience the world as you do if a man were to move them around in the right way?
> There is a good reason why most people consider consciousness arising from the laws of physics as we currently understand them to be totally untenable.
It is only "totally untenable" if you have a preconceived notion that humans are somehow special and not subject to the laws of physics. In other words, if you really want non-material souls to exist.
And yes, I do honestly believe that this endless field of rocks would "experience the world" if someone were to move them around in the right way. Although that is not entirely correct - whatever it is that is moving them in this way should also be considered a part of the overall system, and it is that system that would experience consciousness.
> and it is that system that would experience consciousness
This is not a common view, I'll just say that much. What happens if the man stops moving the rocks? Does the universe die? And if he starts again later? Is the universe conscious only in the instants where the rocks move or only when the man observes the outcome? My experience of consciousness doesn't seem compatible with something a bunch of rocks could feel if you moved them in the right way. And note importantly that this is something the rocks are doing on their own. The man could be moving them according to some set of rules which he doesn't understand, therefore the simple act of rocks moving creates life. I'm honestly astounded that you don't have anything in your experience of the world that you think wouldn't be felt by rocks that shift around in the right way.
> In other words, if you really want non-material souls to exist.
I don't want them to exist, I simply observe them existing. My experience of consciousness is not explained by any physical process. Physics can explain being which say they are conscious, but who hears me say it? A universe could be inhabited solely by chat bots, talking to eachother and claiming to be conscious, but there would be no one to observe that happening.
If you think of an emotional scene in a movie, say "frankly my dear I don't give a damn" in Gone With the Wind or whatever, there is obviously emotion in the scene if you watch it and you see the characters/actors experience it but the film is a stationary thing when it's sitting on the shelf, similar to a fixed 4D structure. I figure reality is like that.
It seems clear to us that the film is playing, but that's the magic isn't it? There is nothing in physics to suggest that the universe is anything other than a film on a shelf, yet we experience it playing. We are the characters in the movie playing out frames in a fixed sequence one after the other (as far as we know according to physics), yet we experience it where the characters in a film do not. We have a souls, or at least you do. For all you know you could be the only one.
If you say it's all about the feelings and machines can't feel that way then it gets rather vague and hard to reason about. I mean they don't have much in the way of feelings now but I don't see why they shouldn't in the future.
I personally feel both those aspects of consciousness are not woo but the results mechanisms built by evolution for functional purposes. I'm not sure how they could have got their otherwise unless you are going to reject evolution and go for divine intervention or some such.