False positives of the kind you're thinking of aren't possible--it's checking for hashes that match known bad images, not running machine learning/image detection to detect if the photo you just took contains bad content. The issue is that there's nothing stopping Apple/the government from marking anything it finds objectionable--like anti-government free speech--as a Bad Image, beyond CSAM.
The thing is Apple uses some custom hash thing with parameters generated by AIs. As some other article shows you can get conflicting hashes if some color patterns and shadows match. Also the threshold they mentioned is secret so it could be 1 or 2 or it could change in future.
Once the policy decision is made that it can run some kind of scanning, it opens the doors for any kind of scanning. Today it's that "neural hash", tomorrow it's going to do something even more invasive.
Really, hashes are sufficiently unique? The objections I saw for this news were along the lines that random images could be manipulated to have hashes that match the flagged cases, in a way that was undetectable by the naked eye.
Doesn't the hash change by exporting a photo as a new file type or by changing a few pixels in photoshop?
If this was the FINAL solution to catch every last child pornographer in one glorious roundup MAYBE it would be worth the massive risk of authoritarian abuse but this algorithm sounds stupidly easy to get around for the deviants while still throwing our collective privacy under the bus.
This is a PhotoDNA hash, not a file-content hash. It is a bit more powerful than a normal hash:
> In the same way that PhotoDNA can match an image that has been altered to avoid detection, PhotoDNA for Video can find child sexual exploitation content that’s been edited or spliced into a video that might otherwise appear harmless
I wouldn't be surprised if a completely innocent false positive gets you put on a list indefinitely, with little recourse.