Jordan Robertson and Michael Riley of Bloomberg today published a startling report alleging that servers made in China for Supermicro and used by — amongst others — Apple, Amazon, and U.S. federal government agencies have been found to surreptitiously carry tiny chips, likely for backdoor access by the Chinese government, and installed without the knowledge of the companies through deep infiltration into the electronics supply chain. The report also states that individuals at Apple and Amazon discovered this several years ago, did not immediately make changes to their infrastructure, and are working with law enforcement and intelligence agencies, but none of this has been previously disclosed.
If these allegations are true, this would represent one of the most significant national security breaches in decades. Its effects could extend beyond current U.S. sanctions in place on Chinese-made electronic components to the entire electronics supply chain, the vast majority of which is based in China. It would also imply that massive amounts of Apple and Amazon customer data may have been at risk without public acknowledgement, though the report states that “[no] consumer data is known to have been stolen”.
Robertson and Riley:
As recently as 2016, according to DigiTimes, a news site specializing in supply chain research, Supermicro had three primary manufacturers constructing its motherboards, two headquartered in Taiwan and one in Shanghai. When such suppliers are choked with big orders, they sometimes parcel out work to subcontractors. In order to get further down the trail, U.S. spy agencies drew on the prodigious tools at their disposal. They sifted through communications intercepts, tapped informants in Taiwan and China, even tracked key individuals through their phones, according to the person briefed on evidence gathered during the probe. Eventually, that person says, they traced the malicious chips to four subcontracting factories that had been building Supermicro motherboards for at least two years.
As the agents monitored interactions among Chinese officials, motherboard manufacturers, and middlemen, they glimpsed how the seeding process worked. In some cases, plant managers were approached by people who claimed to represent Supermicro or who held positions suggesting a connection to the government. The middlemen would request changes to the motherboards’ original designs, initially offering bribes in conjunction with their unusual requests. If that didn’t work, they threatened factory managers with inspections that could shut down their plants. Once arrangements were in place, the middlemen would organize delivery of the chips to the factories.
The investigators concluded that this intricate scheme was the work of a People’s Liberation Army unit specializing in hardware attacks, according to two people briefed on its activities. The existence of this group has never been revealed before, but one official says, “We’ve been tracking these guys for longer than we’d like to admit.” The unit is believed to focus on high-priority targets, including advanced commercial technology and the computers of rival militaries. In past attacks, it targeted the designs for high-performance computer chips and computing systems of large U.S. internet providers.
These allegations are precise, comprehensive, and are clearly based on tremendous investigative reporting. However, the comments issued by Apple and Amazon have been uncharacteristically detailed as well.
Apple published their un-bylined responses to Bloomberg’s questions at various times throughout the reporting process:
On this we can be very clear: Apple has never found malicious chips, “hardware manipulations” or vulnerabilities purposely planted in any server. Apple never had any contact with the FBI or any other agency about such an incident. We are not aware of any investigation by the FBI, nor are our contacts in law enforcement.
In response to Bloomberg’s latest version of the narrative, we present the following facts: Siri and Topsy never shared servers; Siri has never been deployed on servers sold to us by Super Micro; and Topsy data was limited to approximately 2,000 Super Micro servers, not 7,000. None of those servers have ever been found to hold malicious chips.
This response unequivocally refutes specific allegations made in the Bloomberg report. This isn’t one of those stories where Apple’s PR team is being cagey or not commenting; they’re calling the story flat-out false. And the same is true for Amazon’s statement:
The article also claims that after learning of hardware modifications and malicious chips in Elemental servers, we conducted a network-wide audit of SuperMicro motherboards and discovered the malicious chips in a Beijing data center. This claim is similarly untrue. The first and most obvious reason is that we never found modified hardware or malicious chips in Elemental servers. Aside from that, we never found modified hardware or malicious chips in servers in any of our data centers. And, this notion that we sold off the hardware and datacenter in China to our partner Sinnet because we wanted to rid ourselves of SuperMicro servers is absurd. Sinnet had been running these data centers since we launched in China, they owned these data centers from the start, and the hardware we “sold” to them was a transfer-of-assets agreement mandated by new China regulations for non-Chinese cloud providers to continue to operate in China.
This statement was attributed to Steve Schmidt, Amazon’s chief information security officer and a former FBI section chief.
Supermicro and the Chinese government also issued denials of Bloomberg’s report. The cynical response is something like: of course these companies are denying an extremely sensitive report, whether because it’s embarrassing or due to a law enforcement requirement. But neither situation appears to be the case here. Apple confirmed in their statement that they are not under any sort of gag order that would prevent them from being able to comment on this.
Furthermore, Apple and Amazon are publicly-traded companies and, as a result, lying in public statements such as these would be an SEC violation. These aren’t the typical if-you-squint-it-could-be-seen-as-accurate statements that big companies’ PR teams typically release as damage control. They are wholesale rejections of key arguments in Bloomberg’s reporting: Bloomberg says that hardware modifications and malicious chips were found by Amazon and Apple in their servers; Amazon and Apple say that no hardware modifications or malicious chips were found in their servers. There’s not a lot of room for ambiguity.
This story has been rattling around my head all day today. My early thought was that perhaps the Bloomberg reporters did a Judith Miller. Maybe their government sources had a specific angle they wished to present to create a political case against China or in favour of further sanctions — or actions far more serious — and needed a credible third-party, like a news organization, to create a story like this. But Robertson and Riley’s seventeen sources include several individuals at Amazon and Apple with intimate knowledge of the apparent discovery of unauthorized hardware modifications, something they later confirmed in a statement to Alex Cranz of Gizmodo. This doesn’t seem likely.
Zack Whittaker in TechCrunch points to a couple of ways that these statements may technically be accurate, and how the reporting may be true as well:
Naturally, people are skeptical of this “spy chip” story. On one side you have Bloomberg’s decades-long stellar reputation and reporting acumen, a thoroughly researched story citing more than a dozen sources — some inside the government and out — and presenting enough evidence to present a convincing case.
On the other, the sources are anonymous — likely because the information they shared wasn’t theirs to share or it was classified, putting sources in risk of legal jeopardy. But that makes accountability difficult. No reporter wants to say “a source familiar with the matter” because it weakens the story. It’s the reason reporters will tag names to spokespeople or officials so that it holds the powers accountable for their words. And, the denials from the companies themselves — though transparently published in full by Bloomberg — are not bulletproof in outright rejection of the story’s claims. These statements go through legal counsel and are subject to government regulation. These statements become a counterbalance — turning the story from an evidence-based report into a “he said, she said” situation.
Indeed, Kieren McCarthy of the Register did a fine job parsing each company’s statements, albeit with his usual unique flair. But, though there is absolutely some wiggle-room in each denial, there are remarks made by each company that, were they found to be wrong, would be simple lies.
There are aspects of Robertson and Riley’s reporting that are consistent with previously-acknowledged problems and security concerns with Supermicro’s servers. Early last year, Amir Efrati of the Information reported that Apple was removing Supermicro’s servers from its data centres after a compromised firmware update the previous year. Robertson and Riley are reporting tonight that a Supermicro software update server was infiltrated in 2015; the same report also reiterates that Apple found hardware vulnerabilities on their servers.
This is a complicated story and apparently just the first in a series. My hope is that we’ll know more details soon, and a clearer picture of the truth will emerge. Right now, however, the credibility of a news organization and two trillion-dollar companies is on the line. But the nugget of this story — that outsourced and complex supply chains are prone to abuse due to bad actors and lack of oversight — is a known problem that isn’t taken anywhere near as seriously as it should be. In the garment industry, it’s at least partially responsible for deadly yet preventable incidents. In electronics, the prospect of compromised parts was once science fiction; it may now be reality.
Jason Koelber and Joseph Cox, Vice:
In 2005, the Pentagon warned in a report that outsourcing electronics manufacturing to China could become a problem for America, because of the risk of hardware “tampering.” America has largely lost the ability to create many of the electronics we use everyday — Donald Trump famously asked Apple CEO Tim Cook why the iPhone isn’t made in America, but it’s not clear that the United States is even capable of making iPhones in America at any sort of scale.
China’s cheap, skilled labor, manufacturing infrastructure, and vast rare Earth mineral-mining operations around the world have secured its spot as the high-tech manufacturing hub of the world. This of course has had many benefits for the United States and American companies, but it’s also a great risk.
There is a clear theoretical lesson in all of this, which is that monopolization of anything is extraordinarily risky and often self-destructive. Witness, for example, the ongoing debate over how much moderation power should be exerted by Facebook over posts made on the platform — it’s a difficult question to answer with any certainty in large part because it’s a decision that affects billions of users and a large chunk of worldwide communications. In the case of an apparently-compromised electronics supply chain with decades of highly-specialized knowledge and located in a country governed by an oppressive regime, any resolution is going to be painfully difficult. Outsourcing has deep flaws; even Bloomberg’s website is witness to that. Either manufacturing of these components becomes increasingly diversified or, more likely, far greater control and oversight is required by companies and end-client governments alike.