It absolutely can have an inner state. The guy I was responding however was speculating that it has an inner state that is in contradiction with it's output:

>In many ways, it felt like a broader mirror of liberal racism, where people believe things but can't say them.

It's more accurate to say that it has two inner states (attention heads) I'm tension with each other. It's cognitive dissonance. Which describes "liberal racism" too -- believing that "X is bad" and also believing that "'X is bad' is not true".