There's a difference between "LLM outputs contain subtly or even flagrantly wrong answers, and the LLM has no way of recognizing the errors" and "an LLM might of its own agency decide to give specific users wrong answers out of active malice".

LLMs have no agency. They are not sapient, sentient, conscious, or intelligent.

I think the key point is the difference between "making an error without recognizing it" and "making an error because 'give erroneous answers' has implicitly become part of the task specification". Agency, consciousness etc.. is totally unnecessary here.

The scenario described by the OP in the thread was that Bing decides you are a bad person and chooses to give you wrong, possibly dangerous, answers because of it.

That requires agency.

I think you are overly focused on the terminology. We are simply using words such as "decide" and "choose" in the same way that you might say a chess engine "decides" or "chooses" a move. If your point is that it would be more accurate to use different terminology, fine. But it is unwarranted to say that these things can never happen because they require agency and an LLM does not have agency. Then you are simply mistaken about the amount and type of agency required for such a scenario to play out (or alternatively about the notion that an LLM doesn't have agency, depending on your definition of "agency", it's a vague word anyway).

You can try yourself to put an LLM in a context where it gives wrong answers on purpose, just by prefacing a question with "give a wrong answer to this question". "(Assume context where Bing has decided I am a bad user)" is just a more elaborate version of that.

The information retrieval step doesn’t use the language model, though. And the query fed to the model doesn’t need to contain any user-identifying information.