You don't even need this! You could target a whole "social cluster" of people without having any special privileges within this system.

As an example, lets say you want to attack environmental protesters.

For image A, you create a meme about climate change.

For image B, you procure something that looks, to humans, like CSAM (as described in the article).

Craft B′ such that f(B′) = f(A) (also as described).

Now, all you have to do is anonymously publish B′ to a platform that is actively moderated/monitored. Unfortunate users will see it and report it as CSAM, and if the platform fulfills its obligations, that report will bubble up to the relevant authorities, who will review it and add its fingerprint to the database.

Now you can start sending out image A, the meme, to your target demographic. You won't be able to post it on "mainstream" platforms with server-side fingerprint scanning, but there are plenty of other avenues for it to spread. If it's a good meme, it will propagate organically through group-chats and DMs, and eventually find its way onto devices with client-side scanning.

Apple's proposed device scanning system had a threshold before your device would be flagged, so repeat all these steps a whole bunch of times, until the average meme-savvy environmental protester's device gets flagged for further scrutiny.

Unwitting victims who do try to post the meme to a mainstream platform with fingerprint matching may risk getting their accounts flagged and taken down, and they might have no way of knowing what triggered it. This would lead to, of course, automated censorship of environmental protest groups.

Which always struck me as odd as it would extremely easy to spin this as “Apple detected CSAM on this device but their policy is only to alert authorities once a set quantity of CSAM is found…”

I also know it will only take one case of someone being arrested for a related crime, and material being found on their phone that's under the threshold for reporting, and the associated headlines that could be written.

I don't think NCMEC adds random images they find to the A1 list without knowing their origin.

> until the average meme-savvy environmental protester's device gets flagged for further scrutiny.

Who is this an attack on? A moderation contractor maybe, but not the protester. "Further scrutiny" doesn't mean "you go to jail", it means someone looks at the pictures.

“Further scrutiny” here means your images are reported to a provider, which (at that point) means your provider has a list of reports that defeat the guarantees of E2E encryption. The threat model looks something like this: https://www.justice.gov/opa/pr/former-twitter-employee-found...

Remember that case not too long ago with a telehealth appointment, they sent a picture of something on their 2? year old's penis to the doc, asking if it was an issue. The police cleared him, but he's forever guilty in Google's eyes.

And as there are some people (and even companies) who rely on google services and their account there it can really shoot you in the knee

I consider someone looking at pictures from my local device without my consent to be an attack on my privacy, regardless of the content, or whether they send me to jail afterwards.

The reason apple's threshold exists in the first place is because individual false-positives happen. Some of the images leaked from the victim's device may be entirely unrelated. In the specific example of an environmental protester, there might even be images documenting "crimes" (entirely unrelated to CSAM), due to increased criminalization of protest techniques.

A system that may be manipulated (anonymously, from great distance) to trigger spot-checks on the devices of anyone I don't like is a broken system.

I'm not sure either or I would've linked it. I have implemented such a reporting system though (compliant to US law, which is privacy-preserving relatively speaking, not any upcoming EU laws, which like all other EU law seems like a huge pain to live under.)

> I consider someone looking at pictures from my local device without my consent to be an attack on my privacy, regardless of the content, or whether they send me to jail afterwards.

To be clear, this is only if you're using a cloud storage service like iCloud Photos, Google Drive etc. It's meant to be a strict improvement over the usual setup, which is that your data isn't hidden from the cloud provider at all and they can just look at whatever. It would certainly be had to have any scanning if you're not opting into a cloud service.

Your account could certainly be locked until someone looks at it though, yes.

The attacker then claims the video, and claims revenue on the video rather than taking it down. There's an appeal process which seems to work about 0% of the time, and after that the only recourse the uploader has is to submit a copyright counter notification, which becomes the start of a potential lawsuit.

The attackers in question will then typically relent if challenged with a copyright counter claim, but most videos don't get counter claimed once this happened.

This pattern of behavior seems so common that it has led to two interesting patterns of behavior that I've seen.

1) Any video that uses popular public domain music will be claimed not once, but dozens or hundreds of times, because there are so many channels operating this way now, and

2) If you post a video and immediately get hit with a huge wave of claims, rather than going through Youtube's process, most uploaders just delete the video, remove whatever audio caused the problem, and reupload.

Or so I thought... Every single video was copyright claimed. Youtube falsely thought it was a recording (which do have copyright). The second a claim was created I stopped earning money. I would then dispute the claim, explaining that it's not even a sound recording and they had a month to react to my answer.

In almost all instances the claims were retracted but it would only take a couple of days until another claim would be filed on that video. Answering all those claims took up more time than producing the videos so I gave up after a while.

But in my experience it wasn't really malicious actors filing these claims but youtube's filter simply not managing to distinguish between a recording of a classical piano piece and a well synthesized version of it. I actually remember most claims being filed by Sony.

It's interesting to hear that it's still this way because I don't really believe that nowadays, youtube still can't distinguish between different recordings of classical music. I guess they have no incentive to improve in that regard.

But I think the fact that nobody but the large labels are able to earn money with classical music on these large platforms is actually an excellent argument against upload filters and in my experience it's an argument that non-tech people can much better relate to than hash-collisions.

But there are penalties for false claims. There is a fine for claiming copyright you don't own, and if you go further and ask for takedowns, you are also liable for damage.

The problem is that these are rarely enforced. Even a $100 fine for a false claim on YouTube would be enough to weed out bots and click farms. And for the most serious cases, have the infringer pay damage and a bigger fine. No need to change the law for that, it just has to be enforced.

The latter are overwhelmingly more rare, but overwhelmingly larger. Society in general thinks false penalties for crimes are worse than the crime itself.

"It is better that 10 guilty men go free, than that one innocent man should suffer." I think that if the penalty for copyright infringement is $250,000, the penalty for a false claim should be $2,500,000.

You're right that a $100 low-effort, frequently-enforced counterclaim process would weed out ContentID bot farms (just like a $0.01/email transaction cost would weed out spam). But remember that the whole ecosystem is already ridiculous; if the pro-copyright MPAA/RIAA are pushing for $250k/infringement you have to be equally ridiculous to balance counterclaims.

Why is it ridiculous, in a world where the penalty for sharing a single music file is $250,000?

Then again, I personally do think that knowingly filing a false report of law-breaking should be treated as a very serious crime. I think the harm of filing a false report of copyright infringement is greater than the act of infringing copyright itself.

For stuff that is actually copyrighted that probably could only be triggered by the actual rights holder.

(And, yes, I would apply that to the criminal justice world.)

For this reason, many would suggest not using it. It's a vague way of combining the separate issues of copyright, patents, and trademarks. It also illegitimately tries to equate those things to property, which changes how many feel about it.

https://www.gnu.org/philosophy/words-to-avoid.html#Intellect...

Certainly there are issues with how we've implemented property ownership, but I don't think the concept itself is inherently flawed or ridiculous.

Uploading popular (public domain) music and claiming you own it is just fraud

The CSAM detection isn't proper hash collision either, in so far as I understand it. There's some fuzzy matching formula that generates the fingerprint, it's not simply a byte for byte hash taken of the image, and therein lies the comparison.

The fraud in question is reliant on content ID attempting to fuzzy match audio content, in this specific instance.

Additionally, the other party is actively trying to compromise the YTer's other accounts and identity to damage him, so any new data point given to this person represents risk.

I don't really care for this YTer (they made a video essay about someone who doesn't really want to be in the media circus anymore, could just MYOB) but the methods documented in this video can be used to doxx any user that uploads a video. All you gotta do is convince youtube that you own some piece of common non-royalty media that a youtuber uses to gain leverage on them. A lot of this kind of media is old with unfindable owners, so even if YT does want to spend the time to validate, there's nowhere to go with it.

https://www.youtube.com/watch?v=hixwIOd_C44

If this thing gets a serious discussion in a parliament, get rid of the parliament.

When the allegation is as serious as CSAM, I would rather be very very certain before accusing someone, rather than taking a shotgun approach and just randomly searching phones. But what do I know.

The "vulnerable" depends on your definition: SHA-2 and SHA-3 are both still quite safe against preimage attacks, and even second preimage attacks require significant work to pull off for SHA-2, and I am unaware of any meaningful second preimage attack on SHA-3.

Of course, SHA isn't built for finding similar images, but for finding exact matches this should be safe enough.

They essentially take an image and scale it to a small thumbnail. The values of all those reduced pixels are the hash of the original image. When a new image is scanned it's just doing a similarity check against the database of those hashes of known "bad" images. A hit triggers checking against an image hash performed with a separate algorithm. Hits against multiple hashes triggers a "bad image alarm" and ruins a person's life.

Changing a few input bits in an image doesn't usually change the perceptual hash because they're meant to be resistant to small amounts of localized noise. It is possible to add noise that will change a perceptual hash. It's also possible to manipulate an image so transforms (scaling etc) get wildly different results.

This leads to two exploits. The first is an attacker manipulates a "good" image such that when hashed it matches perceptual hashes of a "bad" image. The attacker then sends a bunch of these to a target triggering the "bad image alarm" and essentially SWATs the target. The second is to manipulate bad images in a recoverable way to get them pasted bad image scanners.

There are numerous papers on doing bit manipulation to cause miss classification.

> This shows that the database can be tainted with non-CSAM material by an entity that can submit entries to it.

Actually, it can easily be tainted by anybody. Take your massaged hash-colliding image, which remember is still visually child porn, and post it on some pedos-R-us forum. The people who maintain the database actively troll those forums. They'll see the image and add the hash to the database for you.

This is definitely true. Good old “intentionally accesses a computer without authorization or exceeds authorized access” in the CFAA. It’s so vague as to be able to make almost _anything_ the government doesn’t like involving computers illegal.

Just being accused of having CSAM is a life ruining event. Even if someone is eventually cleared of charges their life is forever altered. The "We Got Him!!" headlines are front page news, retractions are filed in a basement filing cabinet with a sign that says "Beware of Leopard".

In many jurisdictions worldwide, producing/distributing/possessing "very realistic AI generated" child pornography is a crime.

According to Wikipedia, [0] it is criminal in these jurisdictions: Australia, Canada, Ecuador, Estonia, France, Ireland, Mexico, New Zealand, Norway, Poland, Russia, South Africa, South Korea, Switzerland, United Kingdom.

Furthermore, Wikipedia says it is in somewhat of a legal grey area in Argentina, Austria, Italy, Spain, Sweden, and the United States.

Regarding the US in particular: the Supreme Court ruled in the 2002 case of Ashcroft v Free Speech Coalition [1] that the child pornography exception to the First Amendment does not include "virtual child pornography", so long as it does not involve images of real children (i.e. using AI to take a non-pornographic image of a real child and turning it into a pornographic image of that child). However, while this bars prosecuting AI-generated child pornography under child pornography laws, it does not bar prosecuting it under obscenity laws. In the US, obscenity laws are much narrower than child pornography laws, so it is more difficult to get convictions, but people have gone to prison for violating them (e.g. Ira Isaacs [2] in 2012/2013, Paul F Little aka Max Hardcore [3] in 2008/2009). It can be difficult to convince a jury to convict, but realistic AI-generated child pornography may be one of those cases in which many juries would. Furthermore, Ashcroft v Free Speech Coalition is not set in stone–given technological developments since then, and the changed composition of the Supreme Court, it is possible that some prosecutor might seek to overturn it, and you can't say for certain they would fail. Given all this, I think Wikipedia is right to say it is a "legal grey area" in the US.

[0] https://en.wikipedia.org/wiki/Legal_status_of_fictional_porn...

[1] https://en.wikipedia.org/wiki/Ashcroft_v._Free_Speech_Coalit...

[2] https://en.wikipedia.org/wiki/Ira_Isaacs#Further_charges,_re...

[3] https://en.wikipedia.org/wiki/Max_Hardcore#2005_arrest_and_p...

I however have precisely zero concerns about impersonating hashes:

1. It's trivial to deal with tainting the database: both secondary hashing and more invasive hashes deal with that problem.

2. It's trivial to deal with impersonated hashes, all positives can be scanned on device in a second round with a different hash method. Those which are still positive can have be checked off device in an automated, yet privacy preserving, way such as by sending a low resolution crop with faces blurred to a system that does a likeness check against the matched CSA image: only the real image will bear likeness.

3. These systems have undisclosed minimums. A person sharing CSAM will be generating positive results at a significant rate. Receiving a few images with faked hashes won't set off alarms, and if a victim is suddenly receiving hundreds of images, that would already be obvious.

4. These articles over sensationalise what occurs when a positive match is found. They vaguely gesture to serious consequences from a single match. In reality even without secondary automated checks, the false positive will be noticed by a human and then discarded. If a person is receiving a lot of false positives that may lead to the individual being informed that they are begin targeted.

Overall the idea of scanning chat content seems ineffective, for much of the same reasons why it's ineffective to ban end-to-end encrypted chat: those with criminal intent will just use something else or roll their own.

Do we have such a hash function?

The increase in difficulty is multiplicative not additive.

I didn't feel that the comment trivialising multiple hash collisions was worth a direct reply, because the author is clearly not writing with bona fide intentions, or any real knowledge about an image scanning system. It's the fingers in ears, head in the sand kind of denialism that adds nothing to the conversation.

My earlier comment already addressed what happens after the hashes match: a visual comparison. This isn't an original concept, it's a standard approach and part of earlier CSAM scanning proposals.

One needs to act rather barefaced to pretend that a matched hash alone will have consequences, when we already have established that false positives are the known downside to using hash-based image matching.

It may be possible to do the same thing to the client actually.

Sorry, what does "more invasive hashes" mean?

> It's trivial to deal with impersonated hashes, all positives can be scanned on device in a second round with a different hash method.

The double hashing would definitely help. I doubt anybody's gotten a close collision for more than one hash at a time.

It's not inconceivable that they could do it, though. As far as I know, there are only two hash methods that get used for this, and I think both of them are based on sliding a window over a reduced version of the image and doing a DCT, so they're pretty similar. I wouldn't be surprised if somebody could fool both of them enough to at least force you to tighten up your similarity threhsholds. It would be safer to come up with something that worked on completely different principles. No idea how hard that would be.

I don't think that the second-stage backup, where you send the image off somewhere, is "privacy preserving" by any standard I'd be comfortable with. Well, OK, actually even completely foolproof client side scanning isn't privacy preserving enough for me, but I mean that's substantially worse.

I also don't think it'd be easy to find anybody to run that server. And it'd be a significant structural change from what's deployed now. It'd be much more disruptive than adding a second hash (which is isomorphic to just switching to a single composite hash). It took a long time to get the present system deployed, so that big a change doesn't seem "trivial".

> These systems have undisclosed minimums. A person sharing CSAM will be generating positive results at a significant rate. Receiving a few images with faked hashes won't set off alarms, and if a victim is suddenly receiving hundreds of images, that would already be obvious.

Minimum what? If you just say "undisclosed minimums", my mind immediately assumes you mean threshold similarity scores. But you seem to mean hit counts to trigger various actions.

Anyway, both of those are set by the service that's using the database. Those services vary in their strictness and clue level, so I'm not sure you can say anything very general about them.

> These articles over sensationalise what occurs when a positive match is found.

In the threat model of the original article, the people doing the review for false positives are most likely the same people who poisoned the database to begin with. They've corrupted it to give them hits on images they want to suppress, even though the system operators don't want to support them in suppressing those images. So that whole "reviewed by a human and discarded" step doesn't happen.

In the broader world, I suspect that the thresholds for "delete the file" tend to be very low... too low for there to be human review. The thresholds for "disable the user's account" probably aren't exactly stratospheric, either. You can have damaging consequences well short of midnight police raids.

Also, "framing" people isn't necessarily the only thing you might use it for. For example, you could try to use it to drive the cost of running the system up to unsupportable levels.

First, at what point does the authority go "uh... Where did you get this from?" Practically speaking, the people doing this would have to be authorized law enforcement, no? Like if an ordinary citizen showed up with csam the excuse "oh I found this JPEG in a trunk in my late grandpa's attic" isn't gonna fly.

My assumption is that the central authority that compiles the database isn't just taking fingerprints, they want the actual source images. And I'm assuming they actually look at them to ensure they are, in fact, csam. Otherwise anyone could put anything into the database.

Now let me be very clear: I hate the idea of my device scanning for csam on principle. But what's the attack here? That someone looks at my device and determines that I do not, in fact, have csam? And that false positive might even be investigated and reveal that the bad actor sought out or generated CSAM for the purposes of perpetrating this! Which is even worse!

If the police want to look at your device, they can do that already through any number of other sketchy means that don't involve what's likely weeks of effort. The US searches phones at the border when they're feeling like it. I don't want to defend this stuff, but the potential for abuse in this specific case seems vanishingly small.

Imagine that an unflattering or satirical image of Viktor Orban is circulating in France, and let's say it becomes viral, inciting discussions that the Hungarian government finds detrimental to its international image. The authorities might want to suppress this image, not just within Hungary but also in the whole of European Union.

The Hungarian government - who presumably has access to the EU CSAM database (or can coerce those who do), might attempt to add a fingerprint of a manipulated CSAM image that collides with the fingerprint of the satirical image. The principle here is not for public individuals to submit CSAM directly but for government actors to manipulate systems clandestinely.

Then what? What does that achieve? There would be a huge spike in images identified as CSAM which would obviously throw up red flags. It seems like this would mostly just be headache for the law enforcement across the EU. It isn't like France is going to arrest a huge number of people without any investigation or thought of how this one CSAM image spread so far so quickly. And if we are talking about the result in Hungary, that government doesn't need this tool to abuse its power. Why go through all that effort? They could just do the equivalent of rubber-hose cryptanalysis.

And that’s just one idea, probably there’s others.

Sure a conspiracy of all the relevant authorities across the whole EU would work, ... but that seems a stretch to enable what, political elites to _cooperatively_ censor images the public hold. There are easier ways, surely.

Orban's people won't face any pushback because they submitted an actual CSAM image. It's on you to prove somehow they intentionally poisoned the CSAM image such that its hash matches the Orban image.

I understand people's concerns with this tech, but this seems like a rather silly hypothetical to me.

Yes, the database maintainers would notice, but it could take them a while to get around to it. And they're not going to be eager to remove a collision, because that would effectively "legalize" the child porn member of the image pair. There are lots of images that could be useful to suppress temporarily.

And if you actually succeed in suppressing the false-positive image, it may not get passed around very fast, and it won't get vastly more hits than real target images, so it will take longer for anybody to notice or care.

There's also a possible end game where the child porn traders start perturbing their images to collide with really common images like flags, corporate logos, iconic movie stills, and whatever else. So now you either have to ban the US flag, or let this or that actual child porn image go.

I don't actually think that the false hits would ruin very many lives in most places. But it's worth noticing that the original article was talking about authoritarian regimes repurposing the system without the consent of the database maintainers. In the Orbán example, it's possible that the system might flag you for child porn, but you might actually get arrested for sedition. And that continues to happen to people until the database maintainers pull the hash.

They may not know it immediately, but the actual CSAM image wouldn't actually be shared in any real numbers which removes much of the concern with "legalizing" the image. You can argue this makes this system ineffectual, but that also means this isn't repeatable since weakening the impact of this system would quickly result in Hungary losing the power to add hashes to the database.

>And if you actually succeed in suppressing the false-positive image, it may not get passed around very fast, and it won't get vastly more hits than real target images, so it will take longer for anybody to notice or care.

Who is sharing the real target image? Is Hungary now an active creator and distributor of actual CSAM in addition to manipulating the database?

>There's also a possible end game where the child porn traders start perturbing their images to collide with really common images like flags, corporate logos, iconic movie stills, and whatever else. So now you either have to ban the US flag, or let this or that actual child porn image go.

Once again, this behavior would get Hungary booted pretty quickly. Also it is important to remember these are hashes. Not all images of the US flag would trigger the system only that specific image that has a hash collision.

>I don't actually think that the false hits would ruin very many lives in most places. But it's worth noticing that the original article was talking about authoritarian regimes repurposing the system without the consent of the database maintainers. In the Orbán example, it's possible that the system might flag you for child porn, but you might actually get arrested for sedition. And that continues to happen to people until the database maintainers pull the hash.

But this isn't a closed system within that authoritarian regime. It leaves bread crumbs of this behavior out for everyone to see. If Hungary is going to arrest people on made up charges, they can do that anyway.

I just think this in a complicated and ineffectual bullet that can only be fired once because it has a fingerprint of the person who fired it. That adds up to make this type of abuse less of a worry.

> Once again, this behavior would get Hungary booted pretty quickly.

I'm sorry; I should have made myself clearer. In that paragraph, I've taken an aside and moved from Hungary (or any other government) as the adversary to child-porn-sharers-in-general as the adversary, and also changed the adversary goal.

No matter what you do, you can't "boot" the child porn sharers, because they're the ones who actually define what images you're legitimately trying to block.

> Also it is important to remember these are hashes. Not all images of the US flag would trigger the system only that specific image that has a hash collision.

They're approximate perceptual hashes, designed to come up with close values on close pictures. The US flag has an officially defined appearance. You'll get the same hash for any two close-cropped, straight-on images of the flag, the kind you might embed in your Web site. They'll be at least as close as two reprocessed versions of the same child porn image.

I was oversimplifying, though. You're not going to be able to tweak just any child porn image to make it hash like just any flag, unless you're willing to distort it into unrecognizability. And flags and logos might be bad candidates in general, because they're going to give DCT output that's wildly different than what you'll get from most photos. But if you had a relatively large library of child porn and a relatively large library of heavily-used effectively unbannable images of whatever kind, you should be able to find a lot of the child porn images that you can tweak to hash like one or another of the heavily used ones.

... and if you're generating fake child porn from scratch using ML, you can probably hack your ML model to bake the hash of one or another unbannable image into everything it creates. You could probably make those matches pretty damned close.

So, once they got the whole thing down, they should be able to force the system to greatly tighten its match thresholds and/or deal with a really high false positive rate. In fact, they could probably make things bad enough that they could end up with a pretty large collection of child porn that the operators would be forced to completely exclude from detection.

Now back on the original authoritarian threat model:

> I just think this in a complicated and ineffectual bullet that can only be fired once because it has a fingerprint of the person who fired it.

For the "authoritarian" purpose, you may be right... although I wouldn't be surprised if it stayed under the radar for longer than you think. If the image you want to suppress only circulates among your own people, and if you're the authority who receives and verifies the reports on your own people, then all you have to do is to keep the overall volume down enough that it doesn't make anybody suspicious enough to demand that you show them the reports you're getting.

If you're a relatively small country, the number of people who share some local meme you care about may be quite a bit smaller than the number of people who share some new real child porn image.

It’s not inconceivable that someone at the authority could put in an image known to be stored by someone else into the hash list just to cause an extremely scary and confusing month for that person.

To be fair, that’s also the same problem of say, someone at your ISP dislikes you. There’s always a chance your supplier has a “bad apple”. Maybe it’s not an issue but I get the idea.

Isn't that exactly how it is run today, however? There are actual people at the National Center for Missing & Exploited Children who maintain the database and do in fact get incidentally exposed to the imagery. It's the one organization in the entire nation allowed to legally possess the actual images.

If this is easy (for some definition of easy), is it not easy to then alter the CSAM images every they trade it to another pervert, such that hashes never match? How much image alteration would that take?, would it be human perceptible? If perceptible, would it be ignorable?

I think the exploit here suggests that the whole system may end up obsolete sooner rather than later.

Right. Your target uses e.g. an iPhone. You use Kali on clean hardware.

Just post it to 4chan, it’s actively monitored by intelligence services

I know that 99% of people cannot tell an AI image from a real photo since that "Last giant irish greyhound 1902" photo has been going around on social media for weeks, and it is, to me, unbelievably obvious AI.

But the proposed scanning system is the wrong solution, regardless of any "real or AI" ambiguity, because it's possible to generate false positives with nonsense images that aren't even close to the expected CSAM, real or otherwise.

I disagree. The point is to reduce actual child abuse. The images are in a way only tangential. If an image is made with an AI with no actual child being abused, then it shouldn't be a crime.

In a way, it's better, because it will distract the crowd of people into this sort of stuff from activities that harm real people.

This is actually the main point in dispute, and almost everyone arguing one side or the other on this topic seems to assume one side or the other on this point and argue from there, rather than seeking to support their position on the fundamental disputed fact question.

Which results in the most of the debate being people talking past each other based on conflicting assumptions of fact.

>I disagree. The point is to reduce actual child abuse.

There are limited resources, practically the only way to do this is to make it illegal to have anything that looks real (or looks derived from a real situation, in a 'I will know it when I see it way'). Otherwise, you're just making an almost impassable defence of 'it is fake' or 'I thought it was fake'. Then you can't practically reduce actual child abuse.

Thus I see no reason they would differentiate. With normal adult porn do we care that makeup and such might be involved?

I don't understand your comparison with adult porn; that might be nonconsensual but typically isn't, whereas CP is nonconsensual by definition because minors aren't legally capable of agreeing. Obviously there are grey areas like two 17-yos sexting each other, but most courts take that context into account.

Yucky as it is I believe the answer here is to have image-generating AIs sign their work. Something properly signed is known not to involve any actual children and would thus be legal.

(My philosophy in general is that for something to be illegal the state should be able to show a non-consenting victim or the undue risk of a victim (ie, DUI). I do not believe disgusting things in private warrant a law.)

I see that assertion a lot. If that's how that works, why does the very large amount of CSAM already in existence not have the same effect? Why would synthesized CSAM distract pedophiles from their activities when real CSAM from their fellow pedophiles doesn't?

I don't think they meant it would be 100% effective. And real child pornography may deter future abuse. Research is inconclusive.

This needs to be balanced against rights to privacy and expression, which I personally think take precedence, but pretending that it can serve as harm reduction is just not correct.

But I'm sorry, I just don't believe anyone holed up in their room with a bunch of fake CSAM is "just using it as an outlet" or "protecting real kids from harm." I mean, it almost sounds like a threat: "If you don't let me look at these pictures of fake kids, I'll hurt real kids." If that's the case then they should be seeing a psychiatrist, at minimum.

Violent movies that appeal to teens reduce vandalism and the like--they're in the theater rather than out causing trouble. (And it's not displaced, rates don't spike later, they just return to normal.)

My point isn't that AI CSAM should be legal or not, but whether these tools can differentiate what the lawmakers have decided is a crime or not.

Those who have no sexual interest in children are neither going to have CSAM nor engage in abuse. The fact that they had CSAM already shows it's a highly non-random sample. The control would be pedophiles with no access to CSAM--but how do you find that control group????

From a purely ethical standpoint: why? What is the purpose of punishing someone who has harmed no one?

No victim means no crime.

In this case, by giving every CSAM criminal a potential excuse that they "thought it was AI generated," the real victims are further victimized by being deprived of justice or forced to prove their realness.

(There are times when it is important to have commonality while the choice of the common practice isn't important, which justifies regulations of obviously ethically unimportant things like "which side of the road is it correct to drive on relative to the direction of travel"; but where the purpose of a crime is purely to lower the evidentiary bar to punish people presumed guilty of a narrower crime, that's just an attempt to hack around the presumption of innocence and the burden of proof of guilt.)

To choose a less emotional subject, mattress tags.

The ethical reason for mattress tags is because historically people would sell mattresses stuffed full of all sorts of unsavory garbage. What we actually criminalized, or at least were trying to prevent, was some sort of fraud or public endangerment.

But we also along the way made it illegal for sellers to remove the tags from mattresses.

Removing the tag isn't inherently harmful; if you don't deceive the purchaser on the contents of the mattress, it's not even fraud.

But we broadened the definition of the crime to make it easier to enforce.

If you believe, as I do, that such a person is guilty of a crime, then we're not risking the false guiltiness of an innocent person. At best, we're risking their level of sentencing. And I'm open to the idea of reduced sentences for AI CSAM, but it shouldn't be a factor in determination of guilt (i.e. it should be a matter between the judge and the defendant, rather than something the prosecution needs to prove).

With regards to CSAM criminalization in general, there is a real risk of punishing innocent people that may have been framed by planted evidence. But this is a risk regardless of whether the evidence is "real" CSAM or not, so legalizing possession of AI-generated CSAM doesn't reduce the risk of an innocent person being framed. It might make it "easier" for a bad actor to frame someone, since they can now do it with AI content instead of real content. But if they're already planting evidence, do they really care whether they're committing a crime while preparing the evidence? And besides, if we keep the AI content illegal, then it's equally legally risky to frame someone with it as it is to frame them with real content.

The problem of prosecuting "innocent" people, whether you believe they're innocent because they were framed or because they're only guilty of possessing AI-generated CSAM, should be addressed at the time of enforcement. Stop using entrapment and fishing expeditions as an enforcement mechanism. Only open investigations when they start with a real and identifiable victim, rather than a potential perpetrator.

> If you believe, as I do, that such a person is guilty of a crime,

You just explicitly said upthread that ethically they are not, but argued that it is useful for them to be treated as criminals because it denies an excuse to those who are ethically guilty because they are possessors of genuine CSAM.

You seem to be moving your fundamental ethical premises around in response to it being pointed out that the argument you previously made conflicts with a different widely proclaimed ethical premise.

There is literally no victim of any kind, even conceptually, in the case of computer generated imagery. It should be protected artistic expression.

Your logic would seem to imply that there's no crime with possession of real CSAM either, and that the only crime lies with the original abuser who took the pictures.

Unless there is a very specific "attempt to acquire CSAM" law then no they're not fucking guilty of any crime. If you live in a state where marijuana is illegal and you smoke some oregano because you thought it was marijuana you're not guilty of actually possessing marijuana.

A criminal law is composed of a number of individual statutes. When a state is trying to prosecute someone for a crime they need to prove three elements for each statute: the criminal act (actus reus), intent (mens rea), and the concurrence of both of those.

If a cop sells you oregano and you think it's marijuana you might have the intent to buy marijuana but there's no actual criminal act because oregano isn't illegal. If you make a law that only requires intent then congratulations, you've created thought crimes.

If you want to make entirely fake CSAM possession illegal, that's essentially the same as an intent-only law and creates thought crimes. It's a slippery slope.

It wasn't a cop, but I recall a case some years back when someone sold something as cocaine when it wasn't. Among other things, he went down for fraud.

You're a lawyer, I take it? I'm not a lawyer, and I admit your analysis of this scenario confuses me. Is there no legal difference between merely having intent to commit a crime at some point in the future, and actually attempting to commit a crime?

> Is there no legal difference between merely having intent to commit a crime at some point in the future, and actually attempting to commit a crime?

That was my point. To be charged with and prosecuted for a crime you need to both intend to commit it and then actually/attempt to commit it. Attempted murder is a crime, I both intend to kill someone and try to do so even if I fail. It's not punished as severely as actual murder but it's still a crime. But attempted murder is actual a specific crime in the criminal code. There's elements of it that need to be proven in court.

Unless a jurisdiction has a crime of "attempted possession of marijuana", intending to buy marijuana but ending up with oregano isn't a crime someone can be charged with. If we start writing laws outlawing attempted possession it's a slippery slope that gets into outlawing thoughts. It also opens the door to stupid pre-crime ideas like someone would only use cryptography to get ahold of illegal content therefore anyone using cryptography is instantly guilty of attempting to get illegal material.

You can be sure this is what will happen because it's the very arguments the anti-cryptography groups use.

Source: decade in the criminal justice system.

I had previously proposed that if the abuser has been caught that the victim should get the rights to the images and once they are an adult be allowed to legally sell them (thus a list of legally permitted images), but the AI image revolution has changed that. Have AIs sign their images, CSAM with a proper signature is legal.

In a lot of jurisdictions the decision has been made, whether it is right or wrong, to criminalize AI CSAM. The people have spoken and the lawmakers have made the laws. If you or I think that is wrong then the options are to lobby for a change.

That is not an argument against the concept that that should not be the case.

To put it another way, consider a thought experiment where a police officer generates CSAM with AI and then sells it to someone who thinks it's a real picture of a real victim. We should arrest the buyer, right? They thought they were committing a crime.

If you looked at it through a purely ethical framework then you could never convict the person because there was never any "real victim." But is that the right way to look at it? It's certainly not the way most people look at it.

The reason I think it's a bad idea to differentiate between real or AI is similar to the arguments against "means testing" for distributing benefits. You don't want to put real victims in a situation where they're deprived of justice because they can't prove that their victimization was "real." Imagine a real CSAM criminal claiming a defense that they "thought it was AI generated." Do you want to give them that out?

If protecting those victims comes at a cost of punishing criminals possessing AI-generated CSAM with sentences equally as harsh as those for "real" CSAM, then it's a worthwhile cost to pay. They are still criminals, and they are definitely not innocent (unless they're being framed, but that's a risk with both real and AI images).

"I thought she was 18" doesn't work for physical sex either.

Saying "I thought this heroin was fake" is not a defense when caught with a bag of heroin, I don't see how this would be any different. It's not a magic out for anyone.

It’s that you have a bag of fake heroin, you are then arrested for it because someone thinks that by you having fake heroin you are encouraging real heroin users to do more real heroin.

These are situational circumstances, and no prosecutor in the world would choose to prosecute those - but there is 0% chance you could get away with saying "oh I thought it was fake" if caught with CP on your phone.

The issue is that the people (and the legislators) in each jurisdiction have made a choice that AI CSAM is not illegal, and therefore this runs the risk of falsely accusing someone of a crime.

[if you disagree that AI CSAM should be legal in your jurisdiction the solution isn't to arrest everyone, but to petition your lawmakers to change the law]

Why do you believe that?

For the same reason, victims don't get to pardon crimes committed against them.

... which is the way it should be, because criminal punishment should not be seen as a form of revenge, but as a deterrent.

For instance, there is no "victim" if I got caught enjoying cannabis in my own home in a jurisdiction where such a thing is illegal, but "the people" have made a decision that they don't like it and I should be punished for committing an anti-social act.

That is one of the fundamental aspects of democracy at work.

Why would it not be the case, unless the abuse center wanted to waste time and resources and have unreliable records? This essay is technically sophisticated and socially naive. Tech people keep spinning up the most unlikely and hard-to-explain reasons for why leveraging technology to interfere with the spread of CSAM is bad, instead of working toward any kind of privacy-respecting solution to target CSAM itself.

Constantly emphasizing the scope for government intrusion and privacy violation (legit but at the same time overblown to the point of paranoia) and constantly minimizing or dismissing the real harms of CSAM is a great way to alienate the normal non-technical people you need to have on side.

Your claims of things that don't happen are, unfortunately, false. Law enforcement agents whose jobs involve ever looking at CSAM basically cannot stand the mental toll of the job for any significant amount of time. Unthinkably horrible things happen to children to create these images. The idea that we would legalize it is not on the table, and suggesting such a thing is a great way to immediately lose support from everyone in the world.

We should not reject efforts to prevent the production and spread of CSAM unless we can show that the former group can abuse it. Unfortunately, so far, basically all suggested prevention mechanisms are vulnerable to abuse and corruption. It's an extremely difficult problem.

> i don't need a "privacy respecting solution to CSAM". what the hell do you think that would be? do we also need a freedom respecting solution to the murder problem? would you have my legs cut off and say i should just order door dash?

I don't think that analogy is hitting the way you're intending. Yes, we'd all love a "freedom-respecting solution to the murder problem". In fact convicted murderers still have some rights and freedoms, and balancing those against preventing murder and rehabilitating murderers is another difficult problem. Cutting off legs hasn't been seen as a reasonable punishment for a crime since the Bronze Age, so I'm not sure what that's supposed to mean.

literally just propaganda and i have no doubt that this is a massive exaggeration.

>The idea that we would legalize it is not on the table, and suggesting such a thing is a great way to immediately lose support from everyone in the world.

cool, and i was going to open my first post with "CP being contraband is out of the question and never should have became a thing".

> I don't think that analogy is hitting the way you're intending. Yes, we'd all love a "freedom-respecting solution to the murder problem". In fact convicted murderers still have some rights and freedoms, and balancing those against preventing murder and rehabilitating murderers is another difficult problem. Cutting off legs hasn't been seen as a reasonable punishment for a crime since the Bronze Age, so I'm not sure what that's supposed to mean.

i don't know what i didn't make clear. i said that i don't want a law to cut off everyone's legs at birth with a casus beli like "terrible people are going around murdering people with their bare hands now that we all live in a cage and if you don't want to get rid of legs you are part of the problem". the analogy is in fact absolutely perfect even better than i intended, since you know whoever argued this is hugely exaggerating.

and then we have the other guy i'm replying to who can't even write a coherent response (for instance calculating 30K instead of 300, and then ignoring the fact that most of those people will not actually do anything) because he's too angry from his inner sheep being offended because someone dared question is terrible law which is basically just racket where they can use fabricated and overblown cases instead of having to perpetrate the crimes themselves (although the FBI is also known for serving CP themselves, which contradicts a huge amount of the "CP should be illegal" arguments, such as "seeing it makes more pedos")

what % of those "child porn consumers" actually just looked at a picture of a naked 16 year old girl? answer that and stay fashionable.

Hello, I'd be the anyone in this scenario. I don't believe that the pros outweigh the cons, violating the privacy of every single person to protect us from the absolute minority (abusers) of a minority (those attracted to children) is not worth the exchange.

CSAM detection would not have protected me from abuse. At best it would have prevented continued sexual abuse.

And yet had my abuser been caught early I would have been returned to a different, brutally abusive man. One whose abuse no one particularly seems to care about, because there's less to get righteous about when there's no sex involved in the abuse.

I'm sorry that happened to you, but I presume you would also prefer that imagery including you did not continue to circulate?

> I'm sorry that happened to you, but I presume you would also prefer that imagery including you did not continue to circulate?

I've reported CSAM posts on tumblr and twitter and seen them survive for weeks before they finally disappear. It seems like that's a much riper target than propping up this kind of worst case solution.

It's outraging.

Couldn't those who possess this material also run those exact same CLIENT scanners (if not directly, than via isolated canary systems and emissions monitoring) to detect when files have been added to the scanners?

What about tools that keep shuffling files in ways that are not too perceivable but would defeat those scanners? Just look at copyright infringing material on Youtube (it's still there, just better hidden).

Nothing short of a government whitelist database containing all files that are legal to possess, or 100% analog human surveillance will solve CASM. Not AI and certainly not this.

I'm forced to agree with others here: This isn't about CSAM at all. It's a thinly disguised frontal attack on democracy. Imagine if the government had this technology when people were organizing to repeal discriminatory laws in the US. Imagine if authoritative governments in the east (or anywhere else) had this technology right now.

Take a long hard look at every person or organization who supports this nonsense and pay close attention to them. They are the ones who want to harm us and our children.

And since these hashes are presumably resistant to cropping and scaling, they might be able to target individual symbols, people or image macros, like the rainbow pride flag (you know /pol/ wouldn't be able to resist), politicians etc.

This is a really stupid idea. Politicians who are demanding this are way out of their depths.

Is it because on the phone, it is "invisible" and cheap?

Does it mean that what stops the government from treating everyone as a potential criminal is cost and inconvenience?

People were raising alarms about this years ago and were branded as conspiracy theorists. I guess the EU works well in their propaganda department painting themselves as do gooders, where in fact they are just wannabe Stalinists.

Underrated comparison. That is apt and the whole “for the children” part is smokescreen.

The logical conclusion is that someone is going to get a knock on their door and a warrant for search because of an unknown signature match on a file some where at some point?

And people seriously think this will stop at being “for the children”?

Anyone with sufficient power to want to target you wouldn’t use a ridiculous exploit like this. They’ll just go to google and say “change the update server for this user to this new address” and then breach your device with the next software update.

You could select pictures from targets publically available photos or more insidiously look to compromised accounts cloud storage and generate fake offensive images that match peoples actual baby pictures whether for harm or blackmail.

2. You brought in matters that have nothing to do with hashing or child porn, but tend to generate flame wars.

3. You put sneer quotes around "racist", which makes it sound very much like you don't think any such thing exists.

1. Oh okay. I wasn't very clear there. By, "this is basically what I suggested we do a few years ago" I meant that we should look to exploit the limitations of these fingering printing algorithms to render them impractical. But yeah, my suggestion was that we should do the reverse. Rather than taint the database, taint the content.

2. That wasn't my intention. I was suggesting that there are likely other reasons governments are interested in this technology than, "think of the children". Hence, why I think we should seek to ways to render these algorithms ineffective were they deployed. But yes, please feel free to downvote if you feel my comment is unproductive. On reflection I agree it might be. I'm not really adding much here

3. You know what's annoying? I did that because I didn't to make an assertion either way to be as neutral as possible. I'm using quotes because others deem it racist, and that it's irrelevant what my personal position on that is. If you want my position on whether racism exists though, obviously I believe it does – I'd urge you to just assume good faith in the future.

You can't do this without committing an illegal act (downloading CSAM) and it also wouldn't work (because there's a second server side hash), so no I don't think you should do this.

I do think there may be a time and a place for them though... On school and work devices, for example.

That's just my weakly held opinion though. Isn't an opinion I would seek to "promote", just one I wouldn't object to it.

[1] https://takeitdown.ncmec.org/

People just start to use chat programs in the EU that do not comply. This would be Signal, Telegram, others. The EU can fine and fight with Meta/WhatsApp, Apple, others, but that’s about it. The EU bureaucrats do not have power to magically insert spyware in our devices as long as we can install our own software.

… which of course means we know what’s going to come next.

Telegram famously lacks end-to-end encryption (unless you intentionally use its private-chat feature, which few people do) and shouldn’t be mentioned in the same context as Signal.

The secret chats were the biggest feature when it started, but now it's an after-thought.

And Signal avoids FDroid because they don't want someone else signing packages but they can always provide an FDroid repository like many others do and sign everything themselves.

If push comes to shove they'll be fine and pressure to black box signal in the EU is unlikely to hold up if they can just move users to another app store.

Signal’s developers have spoken on a number of occasions about how their aim is to ensure encryption for the masses, not a techie elite.

Mass adoption can continue elsewhere if the EU chooses to go down a route that would make play store support in the EU non-viable.

NOTE: F-Droid does not require app developers to share their keys, instead they build the application themselves and sign it with their keys -- something Signal is not a fan of.

tl;dr their rationale no longer applies.

Either way, this isn't about only offering the application on Google Play or only on F-Droid; but providing the option for both. Non-tech savvy users will always pick the more familiar and easy option on average, Google Play.

> If push comes to shove they'll be fine and pressure to black box signal in the EU is unlikely to hold up if they can just move users to another app store.

Having relatively easy alternatives in place would reduce the leverage the EU has to actually practically enforce this and hopefully pressure them into at minimum non-enforcement and preferably walking back the obviously unenforceable legislation.

But if it looks like they could practically force out Signal and co or force them to adopt this MITM if they want to continue existing, then they might be more likely to pursue it.

The app can then retrieve the message and decrypt it. Most of them will then update the notification shown to the user with the full message content.

If A and B talk regularly, there might not be many other C who by coincidence exchange notifications at the same time

So you might be vulnerable to a sort of traffic analysis

Some applications choose to make these notifications more 'secure' by not sending any content through them, just the fact that you have a new notification, but that makes the UX a little less friendly.

https://en.wikipedia.org/wiki/Apple_Push_Notification_servic...

https://en.wikipedia.org/wiki/Firebase_Cloud_Messaging

Nothing gets taken down from telegram and they ignore court orders they just go to null

https://balkaninsight.com/2023/09/25/who-benefits-inside-the...

> The same month, UK officials privately admitted to tech companies that there is no existing technology able to scan end-to-end encrypted messages without undermining users’ privacy

It's as though this is a secret that UK officials knew. Everyone technically minded knows this; it's a battle between those who regulate (the difficulty of whose job generally stops at "write down the rule and fine people if they don't do it") and those who know about current fundamental limitations.

I don't think they consider themselves limited by things technology is actually capable of when talking about what they are going to get technology to do.