It's probably not programmed to forget, but it was too expensive to implement remembering.

Also probably not realted, but don't these LLMs only work with a relatively short buffer or else they start being completely incoherent?