I think this paper is cool and I love that they ran these experiments to validate these ideas. However, I'm having trouble reconciling the novelty of the ideas themselves. Isn't this result expected given that LLM's naturally learn simple statistical trends between words? To me it's way cooler that they clearly demonstrated not all LLM behavior can be explained this simply.