So can a human, yet we don't ban those. I don't think AI is going to get better at manipulating people than a sufficiently skilled human.

What might be scary is using AI for a mass influence operation, propaganda to convince people that, for example, using a weapon is necessary.

We do prosecute humans who misuse weapons. The problem with AI is that the potential for damage is hard to even gauge; potentially an extinction event, so we have to take more precautions than just prosecuting after the fact. And if the AI has agency, one might argue that it is responsible... what then?