That Pew Report Is Sure Looking Like a Foundational Text, Unfortunately ⇥ bbc.com

Thomas Germain, BBC News, covered the Pew report about the relationship between Google’s A.I. Overviews and click-through traffic:

Pew says it’s confident in its research. “Our findings are broadly consistent with independent studies conducted by web analytics firms,” [Pew’s Aaron] Smith says. Dozens of reports show AI Overviews cut search traffic as much as 30% to 70% depending on what people are Googling. [Amisive’s Lily] Ray says she’s personally seen this in data from hundreds of websites.

But Google tells the BBC you should disregard this, because it’s bad research, biased data and meaningless anecdotes. The company says web traffic fluctuates for many reasons, and AI Overviews link to a wider variety of sources and create new ways to discover websites. Google’s spokesperson says the clicks from AI answers are also higher quality because people spend more time on the sites they visit.

I will continue to harp on this report. I think it is going to be foundational, yet I do not think it is robust enough to sustain the number of articles and conclusions that will be derived from it. The same is also true of a lot of third-party research into Google’s search engine. A decline of 30% in click-through rates is significant, but 70% is catastrophic. It is hard to see how these are both valid results.

This research is also more complex than these headline findings. Take the report showing a 70% reduction in click-through rates when an A.I. Overview is present. That is true — it is a reduction from 2.94% to 0.84% — but the next finding is a near-doubling of the click-through rate when a link is a source in the A.I. Overview compared to if it is not. Not only does this appear to contradict Pew’s findings, it is also described as an “incremental” change despite click-through figures being as similarly small as the previous finding.

The overall trend seems undeniable, however — A.I. Overviews are generally clobbering search referral traffic. Publishers are aware of ebbs and flows in search referral traffic. A.I. Overviews are not having that kind of middling effect. I appreciate Germain’s yadda-yadda framing of Google’s response here; if it wishes to dispute the overall trend, Google should provide evidence.

Germain:

Ironically, Google’s own AI disagrees with its PR department. If you ask Google Gemini, it says AI Overviews hurt websites. […]

I get the joke but I wish it was not included in this article because it plays into myths of how generative A.I. works. Germain is smart enough to know Gemini is just parroting news articles about the subject. And if it did not, it would be deeply suspicious! If questions to Gemini only showed Google P.R.-approved responses, that would be much worse.