> Fair enough but how did we not learn from that fiasco?
We certainly have and that's why so many people are saying that prompt injection is a problem. That can be done with HTML injection because you know that someone will try to include the string "<script>" so you can escape the first "<" with "<" and the browser will not see a <script> tag. There is no such thing to escape with prompts. The browser is expecting a certain content structure that an LLM just isn't.
It might help to think about the inputs that go into the LLM: it's just a bunch of tokens. It is literally never anything else. Even after it generates the next token, that is just added to the current tokens and passed through again. You might define a <system></system> token for your LLM but then an attacker could just type that out themselves and you probably just made things easier for them. As it is, there is no way for current LLM architectures to distinguish user tokens from non-user tokens, nor from generated tokens.
We certainly have and that's why so many people are saying that prompt injection is a problem. That can be done with HTML injection because you know that someone will try to include the string "<script>" so you can escape the first "<" with "<" and the browser will not see a <script> tag. There is no such thing to escape with prompts. The browser is expecting a certain content structure that an LLM just isn't.
It might help to think about the inputs that go into the LLM: it's just a bunch of tokens. It is literally never anything else. Even after it generates the next token, that is just added to the current tokens and passed through again. You might define a <system></system> token for your LLM but then an attacker could just type that out themselves and you probably just made things easier for them. As it is, there is no way for current LLM architectures to distinguish user tokens from non-user tokens, nor from generated tokens.