Countering Consumer Reports’ MacBook Pro Battery Tests ⇥ imore.com

Following Consumer Reports’ wildly inconsistent battery life tests of the new MacBook Pro lineup, Phil Schiller tweeted:

Working with CR to understand their battery tests. Results do not match our extensive lab tests or field data.

It’s pretty strange, isn’t it? Their test seems basic enough:

For the battery test, we download a series of 10 web pages sequentially, starting with the battery fully charged, and ending when the laptop shuts down. The web pages are stored on a server in our lab, and transmitted over a WiFi network set up specifically for this purpose. We conduct our battery tests using the computer’s default browser — Safari, in the case of the MacBook Pro laptops.

If anything, I’d argue that their test is far too basic and doesn’t do a good job of simulating real-world activity; very few people are just browsing the web on any laptop. But to get such inconsistent results from something as straightforward as web browsing makes me think that there’s some edge case at play here.

Anyway, get ready to close all your apps and spin up your fans — more skepticism comes from Rene Ritchie of iMore:

Those results make very little sense and I’d take apart my chain, link by link, until I found out what was going on. I’d check and re-check my tests, I’d watch the systems like a hawk, and I’d do everything possible to find what was causing the variance. I’d even — gasp — try testing different machines and something other than web pages to see if that revealed more information.

Something truly doesn’t seem right with these tests. Matthew Panzarino’s sources say that they’re not seeing this kind of variance in real-world data, and having such a wide range of results is something you’d think would be caught during Apple’s testing, if it’s something that’s typical for these models.

That’s not to say that these benchmarks are wrong, necessarily. If Consumer Reports managed to get 4.5 hours in one version of their test and nearly 20 in another, something is clearly wrong. The question is whether it’s with the hardware, the software, or their testing methodology.

Update: Walt Mossberg also saw unpredictable battery life, though not to the extent that Consumer Reports did:

The biggest surprise in my tests was just how inconsistent the Touch Bar Pro’s battery life was. I have tested hundreds of laptops over the years and Macs have almost always excelled at meeting or beating their promised battery lives, both in my longtime battery test regime and in typical daily use. But the 13-inch MacBook Pro with Touch Bar wasn’t as reliably consistent as previous Macs.

Mossberg’s reporting was the result of his impressions more than a controlled test.