This to me is like going "Jesus H. Christ" at the prompt you get when you run the "sqlite3" command. It is also crazy to point that command at a production database and do random stuff with it. But not at all crazy to use it during development. I don't think this issue is as complicated, or as LLM-specific, as it seems; it's really just recapitulating security issues we understood pretty clearly back in 2010.
Actually, in my experience doing software security assessments on all kinds of random stuff, it's remarkable how often the "web security model" (by which I mean not so much "same origin" and all that stuff, but just the space of attacks and countermeasures) maps to other unrelated domains. We spent a lot of time working out that security model; it's probably our most advanced/sophisticated space of attack/defense research.
(That claim would make a lot of vuln researchers recoil, but reminds me of something Dan Bernstein once said on Usenet, about how mathematics is actually one of the easiest and most accessible sciences, but that ease allowed the state of the art to get pushed much further than other sciences. You might need to be in my head right now to see how this is all fitting together for me.)
> It is also crazy to point that command at a production database and do random stuff with it
In a REPL, the output is printed. In a LLM interface w/ MCP, the output is, for all intents and purposes, evaluated. These are pretty fundamentally different; you're not doing "random" stuff with a REPL, you're evaluating a command and _only_ printing the output. This would be like someone copying the output from their SQL query back into the prompt, which is of course a bad idea.
I won't claim to be as well-versed as you are in security compliance -- in fact I will say I definitively am not. Why would you think that it isn't a meaningful difference here? I would never simply pipe sqlite3 output to `eval`, but that's effectively what the MCP tool output is doing.
> This to me is like going "Jesus H. Christ" at the prompt you get when you run the "sqlite3" command.
Sqlite is a replacement for fopen(). Its security model is inherited from the filesystem itself; it doesn't have any authentication or authorization model to speak of. What we're talking about here though is Postgres, which does have those things.
Similarly, I wouldn't be going "Jesus H. Christ" if their MCP server ran `cat /path/to/foo.csv` (symlink attacks aside), but I would be if it run `cat /etc/shadow`.
Actually, in my experience doing software security assessments on all kinds of random stuff, it's remarkable how often the "web security model" (by which I mean not so much "same origin" and all that stuff, but just the space of attacks and countermeasures) maps to other unrelated domains. We spent a lot of time working out that security model; it's probably our most advanced/sophisticated space of attack/defense research.
(That claim would make a lot of vuln researchers recoil, but reminds me of something Dan Bernstein once said on Usenet, about how mathematics is actually one of the easiest and most accessible sciences, but that ease allowed the state of the art to get pushed much further than other sciences. You might need to be in my head right now to see how this is all fitting together for me.)