Apple Photos’ ‘Enhanced Visual Search’ Matches Possible Landmarks Remotely lapcatsoftware.com

Matthew Green on Bluesky:

I love that Apple is trying to do privacy-related services, but this [“Enhanced Visual Search” setting] just appeared at the bottom of my Settings screen over the holiday break when I wasn’t paying attention. It sends data about my private photos to Apple.

The first mention of this preference I can find is a Reddit thread from August.

Apple says it is an entirely private process:

Enhanced Visual Search in Photos allows you to search for photos using landmarks or points of interest. Your device privately matches places in your photos to a global index Apple maintains on our servers. We apply homomorphic encryption and differential privacy, and use an OHTTP relay that hides IP address. This prevents Apple from learning about the information in your photos. […]

The company goes into more technical detail in a Machine Learning blog post. What I am confused about is what this feature actually does. It sounds like it compares landmarks identified locally against a database too vast to store locally, thus enabling more accurate lookups. It also sounds like matching is done with entirely visual data, and it does not rely on photo metadata. But because Apple did not announce this feature and poorly documents it, we simply do not know. One document says trust us to analyze your photos remotely; another says here are all the technical reasons you can trust us. Nowhere does Apple plainly say what is going on.

Jeff Johnson:

Of course, this user never requested that my on-device experiences be “enriched” by phoning home to Cupertino. This choice was made by Apple, silently, without my consent.

From my own perspective, computing privacy is simple: if something happens entirely on my computer, then it’s private, whereas if my computer sends data to the manufacturer of the computer, then it’s not private, or at least not entirely private. Thus, the only way to guarantee computing privacy is to not send data off the device.

I see this feature implemented with responsibility and privacy in nearly every way, but, because it is poorly explained and enabled by default, it is difficult to trust. Photo libraries are inherently sensitive. It is completely fair for users to be suspicious of this feature.