Zack Whittaker, TechCrunch:
Later this year, Apple will roll out a technology that will allow the company to detect and report known child sexual abuse material to law enforcement in a way it says will preserve user privacy.
Apple told TechCrunch that the detection of child sexual abuse material (CSAM) is one of several new features aimed at better protecting the children who use its services from online harm, including filters to block potentially sexually explicit photos sent and received through a child’s iMessage account. Another feature will intervene when a user tries to search for CSAM-related terms through Siri and Search.
At Apple, our goal is to create technology that empowers people and enriches their lives — while helping them stay safe. We want to help protect children from predators who use communication tools to recruit and exploit them, and limit the spread of Child Sexual Abuse Material (CSAM).
[…]
These features are coming later this year in updates to iOS 15, iPadOS 15, watchOS 8, and macOS Monterey. [Features available in the U.S.]
This program is ambitious, and protecting children is an important responsibility. These efforts will evolve and expand over time.
To state the blindingly obvious, any amount of child abuse is too much, and we should celebrate any effort to reduce it when the privacy and security of the general public is not compromised. Apple’s announcement sounds like it is walking that fine line, but this is perhaps the biggest change to Apple’s privacy practices in years given that it intercepts messages and scans photos. There are experts in cryptography and privacy that are troubled by today’s announcement, and those concerns are worth taking seriously.
This announcement is fresh, and I am sure the fulness of time will bring more questions, confusion, and clarity. This post is not going to be comprehensive, but it will be an early look at what we do know.
There are three parts to Apple’s announcement:
For devices signed into an Apple ID belonging to a user under the age of 18 in an iCloud Family, Messages will flag images that on-device machine learning has determined may be sexually explicit. If the user chooses to view or send the image anyway, users with a Parent/Guardian role in the iCloud Family will be notified if the child is under the age of 13.
iPhones and iPads will locally scan images destined for iCloud Photos. Images will be compared on-device to hashes derived from known CSAM imagery, and will attach an encrypted “voucher” with the results when the photo is uploaded. When enough positive vouchers are associated with an account, the flagged photos and vouchers will be decrypted, and the account will be reviewed by a real person at Apple. If the flagged images in the account are actually CSAM and not a false-positive, Apple will suspend it and notify the National Center for Missing and Exploited Children.
Siri and Spotlight will display messages when users try to search for CSAM or want to report abuse.
The latter is a good step, and it is relatively straightforward. I am skipping it for the purposes of this post. That leaves the interception of messages, and on-device scanning. Let’s start with the first one, which actually sounds pretty appealing as an option for all users. I think many people would appreciate if their device flagged potentially sensitive images — solicited or unsolicited.
But the mechanism by which it determines if the image is explicit is, like most machine learning, an unaudited mystery. If it errs on the side of caution, it may flag perfectly innocent images which, like the car alarm I have been subjected to for the past minute and a half, makes it less likely that any warning is treated seriously. If it is not strict enough, it may not be effective. Either way, trust in this system must strike a careful balance.
That unaudited system also raises the possibility of abuse. This feature has only been announced for the U.S., but it will surely expand to devices in countries that have required Apple’s complicity in censorship. If it can be used to intercept potentially sexual images, there is no technical reason the same technology could not be trained to identify other kinds of images. The likelihood of that theoretical concern becoming material depends only on the pressure Apple will face, and whether it will risk being unable to sell in a country if it does not comply.
The second feature announced today that has generated the most discussion is the way images destined for iCloud Photos will be checked for CSAM. And it is the one that has me the most confused. Partly, that is because the cryptography proof (PDF) is complete gibberish to me; mostly, it is because I am unclear why this is more concerning now than when Apple acknowledged last January it was checking images in iCloud Photos. I have asked for clarity and will update this post if I hear back. (Update: Apple was not checking iCloud Photos. I regret the error.)
Apple is not alone in scanning user materials for potential CSAM, either. Facebook, Twitter, and Microsoft use a database matching technology developed by the latter called PhotoDNA to detect this imagery, while Google has used a different database since 2008. In all cases — including Apple’s — the implementation is broadly similar: hashes are created from different parts of known images of child abuse. When any image is uploaded to services provided by these companies, its hash is checked against that database and the image is flagged if there is a match.
What is perhaps different about Apple’s approach is that there is some machine learning magic that will generate the same hash for images that have been altered (PDF):
The main purpose of the hash is to ensure that identical and visually similar images result in the same hash, and images that are different from one another result in different hashes. For example, an image that has been slightly cropped or resized should be considered identical to its original and have the same hash.
Apple’s example shows that the same hash is generated by colour and greyscale versions of the same image. If you’re wondering about the fallibility of this magic system, you are not alone. Alongside this announcement, Apple also released a handful of independent assessments of its cryptographic bonafides. But they did not impress Matthew Green of Johns Hopkins University.
Green was the first person to break the news of this then-impending announcement, and is one of several security professionals to express concerns in a Financial Times report by Madhumita Murgia and Tim Bradshaw:
“It is an absolutely appalling idea, because it is going to lead to distributed bulk surveillance of … our phones and laptops,” said Ross Anderson, professor of security engineering at the University of Cambridge.
[…]
“This will break the dam — governments will demand it from everyone,” said Matthew Green, a security professor at Johns Hopkins University, who is believed to be the first researcher to post a tweet about the issue.
Alec Muffett, a security researcher and privacy campaigner who formerly worked at Facebook and Deliveroo, said Apple’s move was “tectonic” and a “huge and regressive step for individual privacy”.
These concerns could also apply to Apple’s existing scanning of iCloud Photos. Regardless, it is a privacy worry when any kind of personal document is being scanned and flagged.
Please do not misinterpret this as a defence for a perceived right to create or retain any of this vile material. But — hard pivot here — it is worth asking whether it makes sense to deputize any company to monitor user accounts for anything illegal.
Let’s apply this logic to something less heinous like copyrighted materials. If these files are merely stored in a cloud service and are not being shared to any third party, should the service provider be monitoring its systems and terminating user accounts with files matching known pirated copies? I am not sure that makes much sense.
Once these files are shared beyond the user’s personal drive, I am fully on board with scanning for legal purposes. Google Drive, for example, will allow users to store copyrighted materials but attempts to restrict sharing options if those files match valid takedown requests. Apple and Google check email attachments for child abuse materials, while Facebook scans users’ messages. That makes sense to me. As with social media companies moderating their platforms, I think there ought to be different expectations when things are broadcast to the world.
Perhaps CSAM should be regarded as a special case where cloud storage companies are deputized to proactively prevent any storage of this material whatsoever. I get that argument, and I think it is worth considering given the gravity of these specific abuses. But, while I am not a privacy absolutist, I think this is another step in a concerning direction where the default device setup gives Apple the keys to users’ data.
iCloud backups were the first step. iPhones and iPads will back up to iCloud by default, and Apple has the ability to turn over backup contents in response to legal demands. In some sense, this is a valid compromise: users benefit from encryption that protects device data from man-in-the-middle attacks and reduces the device’s usefulness in case of theft, while law enforcement still gets access to almost all device data when it is required for an investigation. If you do not like this situation, you must remember to disable iCloud backups and use your own computer instead. But the power of this default means that a user who has gone through the effort of switching off iCloud backups gives the impression of having something to hide, even if they do not.
iCloud Photos is not turned on by default but, as Michael Tsai writes, Apple implicitly encourages its use:
One takeaway is that, CSAM detection aside, Apple already has access to these photos. You shouldn’t upload anything to the cloud that you want to keep private. But Apple isn’t giving users much choice. It doesn’t let you choose a truly private cloud backup or photo syncing provider. If you don’t use iCloud Photo Library, you have to use Image Capture, which is buggy. And you can’t use iCloud to sync some photos but not others. Would you rather give Apple all your photos or risk losing them?
You can use third-party alternatives that keep an entirely separate library — Adobe Lightroom is one example — but it feels like a second-rate experience on iOS.
Tsai:
And, now that the capability is built into Apple’s products, it’s hard to believe that they won’t eventually choose to or be compelled to use it for other purposes. They no longer have the excuse that they would have to “make a new version of the iPhone operating system.”
This is true, but I think a fair counterargument is that Apple’s more proactive approach to child safety takes away one of law enforcement’s favourite complaints about commonplace encryption.
But it represents a similar trade-off to the aforementioned iCloud backups example. Outside of the privacy absolutist’s fictional world, all of privacy is a series of compromises. Today’s announcements raise questions about whether these are the right compromises to be making. What Apple has built here is a local surveillance system that all users are supposed to trust. We must believe that it will not interfere with our use of our devices, that it will flag the accounts of abusers and criminals, and that none of us innocent users will find ourselves falsely implicated. And we must trust it because it is something Apple will be shipping in a future iOS update, and it will not have an “off” switch.
Perhaps this is the only way to make a meaningful dent in this atrocious abuse, especially since the New York Times and the NCMEC shamed Apple for its underwhelming reporting of CSAM on its platforms. But are we prepared for the likely expansion of its capabilities as Apple and other tech companies are increasingly pressured to shoulder more responsibility for the use of their products? I do not think so. This is a laudable effort, but enough academics and experts in this field have raised red flags for me to have some early concerns and many questions.