This is part of iCloud photo sync, and on hitting some threshold of matching pictures it would trigger human review.
There would also presumably be human review involved in the legal process, e.g. law enforcement getting a subpoena based on Apple notifying them, and then using gathered evidence for a warrant.
The system is based on known image hashes, not arbitrary ML detection.
As this system is used only for iCloud photo uploads, the evidence gathering should be similar to that done by LE with other cloud hosting providers for years
There would also presumably be human review involved in the legal process, e.g. law enforcement getting a subpoena based on Apple notifying them, and then using gathered evidence for a warrant.
The system is based on known image hashes, not arbitrary ML detection.
As this system is used only for iCloud photo uploads, the evidence gathering should be similar to that done by LE with other cloud hosting providers for years