What I'm concerned about is a system that flags me for a crime based on a database I can't audit based on mechanisms with an entirely too high false positive rate.
Because the database can't be audited by anyone but a select group we have to trust that it only contains actual bad images. I do not trust that such databases don't also contain images that are embarrassing to powerful/connected people. I also do not trust such databases don't contain false positives.
The sort of people that are super zealous about a topic aren't simultaneously super rational and objective about that topic. There's a non-zero probability that those databases contain lewd yet entirely legal images that the submitters just didn't like.
Because of the false positive rate a photo of my dog might trigger an alarm and then my phone sends an automated message to the police. I'm then told by proponents there will be some manual review. I have to then hope that the local DA doesn't have an election coming up and wants to push a "tough on crime" message so charges me with a crime despite a review.
In short these scanning systems require far too much unearned trust. They also present a slippery slope thanks to the incendiary nature of the topic. Today it's CSAM but what undesirable content will the systems be used for tomorrow? Such systems require trust in the stewards of today and tomorrow. Do you want people of the opposite ideology to you in charge of such systems? Do you trust they'll never be abused? Do you trust well meaning people never make mistakes?
I do not trust in any of those things. I'm not worried about myself doing actual bad things, I'm worried that demonstrable false positive rates will ruin my life with the mere accusation of doing something bad.
Nothing in this post (either the system implementation details or the supposed consequences of detection) is how it actually works in reality, so I don't think you should spent much time being upset that something you just made up has a security hole in it.
e.g. there isn't a high false positive rate, the attack in this article doesn't work because the attacker doesn't have access to all the hashing algorithms used, and it doesn't text the police.
> the attack in this article doesn't work because the attacker doesn't have access to all the hashing algorithms used
As far as I know, there are only two hashing algorithms used: ContentID and "the Facebook one", whose name I don't remember offhand at the moment. ContentID has leaked, been reverse engineered, been published, and been broken. A method to generate collisions in it has been published. The Facebook one has never been secret, and essentially the same method can generate collisions in it. And most users just use ContentID. [On edit: Oh, yeah, Apple has their "Neuralhash" thing.]
Are there others? If there are, I predict they're going to leak just like ContentID did, if they haven't already. You can't keep something like that secret. To actually use such an algorithm, you end up having to distribute code for it to too many places. [On the same edit: that applies to Neuralhash].
I assume you're right that the false positive rate is very low at the moment. Given the way they're done, I don't see how those hashes would match closely by accident. But the whole point of this discussion is that people have now figured out how to raise the false positive rate at will. It's a matter of when, not if, somebody finds a reason to drive the false positive rate way up. Even if that reason is pure lulz.
None of this "texts the police", but it does alert service providers who may delete files, lock accounts, or flag people for further surveillance and heightened suspicion. Much of that is entirely automated. And a lot of the other things you'd use as input, if you were writing a program to decide how suspicious you were, are even more prone to manipulation and false positives.
I believe the service providers also send many of the hits to varous "national clearinghouses", which are supposed to validate them. Those clearinghouses usually aren't the police, but they're close.
But the clearinghouses and the police aren't the main problem the false positives will cause. The main problem is the number of people and topics that disappear from the Internet because of risk scores that end up "too high".
> As far as I know, there are only two hashing algorithms used: ContentID and "the Facebook one", whose name I don't remember offhand at the moment.
Yes, those aren't suited for client-side scanning. If the server side can do any content scanning then you're not secure against them, so the protection isn't what kind of hashing they use, it's just that someone actually looks at the results.
> You can't keep something like that secret.
I didn't say it was secret, I said you don't have access to it. Well… that's kind of the same thing I guess, but anyway the important point is they can change it/reseed it.
> None of this "texts the police", but it does alert service providers who may delete files, lock accounts, or flag people for further surveillance and heightened suspicion.
Google has done this one looking for "novel CSAM" aka anyone's nudes, which is bad, so I recommend not doing that.
> Those clearinghouses usually aren't the police, but they're close.
No, it's extremely important that they're not the police (or other government organization); in the US NCMEC exists because, as they're a private organization, you get Fourth Amendment protections if you do get reported to them. But these systems don't automatically report to them either. Someone at the service looks at it first.
Because the database can't be audited by anyone but a select group we have to trust that it only contains actual bad images. I do not trust that such databases don't also contain images that are embarrassing to powerful/connected people. I also do not trust such databases don't contain false positives.
The sort of people that are super zealous about a topic aren't simultaneously super rational and objective about that topic. There's a non-zero probability that those databases contain lewd yet entirely legal images that the submitters just didn't like.
Because of the false positive rate a photo of my dog might trigger an alarm and then my phone sends an automated message to the police. I'm then told by proponents there will be some manual review. I have to then hope that the local DA doesn't have an election coming up and wants to push a "tough on crime" message so charges me with a crime despite a review.
In short these scanning systems require far too much unearned trust. They also present a slippery slope thanks to the incendiary nature of the topic. Today it's CSAM but what undesirable content will the systems be used for tomorrow? Such systems require trust in the stewards of today and tomorrow. Do you want people of the opposite ideology to you in charge of such systems? Do you trust they'll never be abused? Do you trust well meaning people never make mistakes?
I do not trust in any of those things. I'm not worried about myself doing actual bad things, I'm worried that demonstrable false positive rates will ruin my life with the mere accusation of doing something bad.