It has been two and a half years since Bloomberg Businessweek published the now-legendary story of how servers made by Supermicro were compromised by Chinese intelligence at the time of manufacture — servers that ended up in data centres for “a major bank, government contractors”, Apple, and a company acquired by Amazon that counted among its clients the U.S. Department of Defense. Contemporary statements from the named affected companies were unequivocal: either the reporters were completely wrong, or these statements were lies that would carry severe penalties should evidence be found.
In the ensuing years, Jordan Robertson and Michael Riley, the two reporters on the story, have mostly stayed quiet despite frantic calls from security professionals for clarity. Its truthfulness has become something of an obsession for many, including me. On the first anniversary of its publication, I lamented the lack of followup: “either [it is] the greatest information security scoop of the decade or the biggest reporting fuck-up of its type”.
Nearly a year and a half has passed since I wrote that, and it has seemed like it would remain a bizarre stain on Bloomberg Businessweek’s credibility. And then, today, came the followup.
Jordan Robertson and Michael Riley, Bloomberg:
In 2010, the U.S. Department of Defense found thousands of its computer servers sending military network data to China — the result of code hidden in chips that handled the machines’ startup process.
In 2014, Intel Corp. discovered that an elite Chinese hacking group breached its network through a single server that downloaded malware from a supplier’s update site.
And in 2015, the Federal Bureau of Investigation warned multiple companies that Chinese operatives had concealed an extra chip loaded with backdoor code in one manufacturer’s servers.
Each of these distinct attacks had two things in common: China and Super Micro Computer Inc., a computer hardware maker in San Jose, California. They shared one other trait; U.S. spymasters discovered the manipulations but kept them largely secret as they tried to counter each one and learn more about China’s capabilities.
When I woke up this morning and saw Techmeme’s rewritten headline, “Sources: US investigators say hardware and firmware of Supermicro servers were tampered, with an extra chip loaded with a backdoor to send data to China”, I thought there must be some strange bug that is loading old news. Alas, this is a new story, with new sources — over fifty people spoke with the reporters, apparently — new evidence, and new allegations. But rather than clarifying the 2018 article, I find that I have many of the same questions now about two blockbuster articles.
Before I get into my confusion, a necessary caveat: I only have information that has been shared publicly and I am a hobbyist commentator, while Robertson and Riley are journalists who have been collecting details for years. These stories matter a lot, and their allegations are profound, but extraordinary claims demand extraordinary evidence. And based on everything that has been reported so far, I just don’t see it yet. Chalk it up to my own confusion and naïveté, but it seems like I am not alone in finding these reports insufficiently compelling.
Here’s the one-paragraph summary: Supermicro is a big company with lots of clients, any of which would be concerned about a backdoor to a foreign intelligence agency in their hardware. According to these reports, the U.S. intelligence apparatus was mobilized to counter the alleged threat. This has been a high-profile case since the first story was published. And I am supposed to believe that, in two and a half years, the only additional reporting that has been done on this story is from the same journalists at the same publication as the original. Why do I not buy that?
Robertson and Riley’s new report concerns the three specific incidents in the quoted portion above. There is no new information about the apparent victims described in their 2018 story. They do not attempt to expand upon stories about what was found on servers belonging to Apple or the Amazon-acquired company Elemental, nor do they retract any of those claims. The new report makes the case that this is a decade-long problem and that, if you believe the 2010, 2014, and 2015 incidents, you can trust those which were described in 2018. But if you don’t trust the 2018 reporting, it is hard to be convinced by this story.
This time around, there are many more sources, some of which agreed to be named. There is still no clear evidence, however. There are no photographs of chips or compromised motherboards. There are no demonstrations of this attack. There is no indication that any of these things were even shown to the reporters. The new incidents are often described by unnamed “former officials”, though there are a handful of people who are willing to have quotes attributed.
So let’s start with the claims of one of those on-the-record sources:
“In early 2018, two security companies that I advise were briefed by the FBI’s counterintelligence division investigating this discovery of added malicious chips on Supermicro’s motherboards,” said Mike Janke, a former Navy SEAL who co-founded DataTribe, a venture capital firm. “These two companies were subsequently involved in the government investigation, where they used advanced hardware forensics on the actual tampered Supermicro boards to validate the existence of the added malicious chips.”
Janke, whose firm has incubated startups with former members of the U.S. intelligence community, said the two companies are not allowed to speak publicly about that work but they did share details from their analysis with him. He agreed to discuss their findings generally to raise awareness about the threat of Chinese espionage within technology supply chains.
Do not be distracted by the description of Janke as a former Navy SEAL. It is irrelevant to this matter.
One of the companies that has received funding from DataTribe is Dragos, which promises “industrial strength cybersecurity for industrial infrastructure”. It is not clear whether Dragos was one of the firms that received an FBI briefing. However, Dragos’ CEO Robert M. Lee has been consistently critical of Robertson and Riley’s reporting. Lee continues to be skeptical of their claims, saying that they have “routinely shown they struggle on technical details”. That becomes apparent in a detail in this adjacent story of apparently compromised Lenovo ThinkPads used by U.S. forces in Iraq in 2008:
“A large amount of Lenovo laptops were sold to the U.S. military that had a chip encrypted on the motherboard that would record all the data that was being inputted into that laptop and send it back to China,” Lee Chieffalo, who managed a Marine network operations center near Fallujah, Iraq, testified during that 2010 case. “That was a huge security breach. We don’t have any idea how much data they got, but we had to take all those systems off the network.”
Three former U.S officials confirmed Chieffalo’s description of an added chip on Lenovo motherboards. The episode was a warning to the U.S. government about altered hardware, they said.
That quote was pulled from a court transcript, and Chieffalo really did say “a chip encrypted on the motherboard”. That phrase is gibberish. It seems likely that Chieffalo meant to say “a chip embedded on the motherboard”, but the transcript includes no attempt at correction. More worrying for this story, Chieffalo was quoted wholesale without any note from the reporters. It seems reasonable that they could not speculate about the intended word choice, but surely they could have reached Chieffalo for clarification. If not, it seems like an odd choice to approvingly quote it; it undermines my trust in the writers’ understanding.
That trust is critical, particularly as this report implies a much more severe allegation. In 2018, Robertson and Riley wrote that Supermicro servers were compromised at the subcontractor level:
During the ensuing top-secret probe, which remains open more than three years later, investigators determined that the chips allowed the attackers to create a stealth doorway into any network that included the altered machines. Multiple people familiar with the matter say investigators found that the chips had been inserted at factories run by manufacturing subcontractors in China.
That suggests some distance between Supermicro itself and its allegedly compromised boards. If this is true, the company has some wiggle room there to disclaim awareness and terminate that supplier relationship. But in today’s report, Robertson and Riley step up the level of Supermicro’s involvement:
Manufacturers like Supermicro typically license most of their BIOS code from third parties. But government experts determined that part of the implant resided in code customized by workers associated with Supermicro, according to six former U.S. officials briefed on the findings.
Investigators examined the BIOS code in Defense Department servers made by other vendors and found no similar issues. And they discovered the same unusual code in Supermicro servers made by different factories at different times, suggesting the implant was introduced in the design phase.
Overall, the findings pointed to infiltration of Supermicro’s BIOS engineering by China’s intelligence agencies, the six officials said.
The report is careful to say that there is no evidence of executive involvement, and that these changes would have been made by people in a position to be working directly with Supermicro’s server technologies. But that still implies knowledge of this alleged compromise at much closer proximity than some factory in China.
The BIOS manipulation above is dated to 2013. The following year, the report says, the FBI detected nefarious chips on “small batches” of Supermicro boards:
Alarmed by the devices’ sophistication, officials opted to warn a small number of potential targets in briefings that identified Supermicro by name. Executives from 10 companies and one large municipal utility told Bloomberg News that they’d received such warnings. While most executives asked not to be named to discuss sensitive cybersecurity matters, some agreed to go on the record.
In 2018, Businessweek said there were up to thirty companies; it is not clear how much overlap there is with the eleven above. But, as Robertson and Riley write, not a single one has said they found evidence of infiltration. Some blamed a dearth of information from the FBI for their inability to find a problem with their servers, but what if the supposed rogue chips simply did not exist? That would make it especially hard to find evidence for them. Just because government agencies are providing briefings of a possible problem, it does not necessarily mean that problem exists as described.
Here’s one more named source with a funny story:
Darren Mott, who oversaw counterintelligence investigations in the bureau’s Huntsville, Alabama, satellite office, said a well-placed FBI colleague described key details about the added chips for him in October 2018.
“What I was told was there was an additional little component on the Supermicro motherboards that was not supposed to be there,” said Mott, who has since retired. He emphasized that the information was shared in an unclassified setting. “The FBI knew the activity was being conducted by China, knew it was concerning, and alerted certain entities about it.”
If there is a phrase that is jumping out to you in this quote, it is probably “October 2018” because that is when Robertson and Riley published their original “Big Hack” piece. It seems completely plausible to me that Mott’s colleague was describing that Businessweek article. There is nothing here that suggests the colleague was referring to independent knowledge. On the contrary, the fact that this was shared in an “unclassified setting” runs counter to the repeated assertions in both articles about the sensitivity and secrecy of these operations — so secret that, apparently, not even Supermicro was supposed to know.
There is one more incident described in detail. This time, Intel was the supposed target in 2014:
Intel’s investigators found that a Supermicro server began communicating with APT 17 shortly after receiving a firmware patch from an update website that Supermicro had set up for customers. The firmware itself hadn’t been tampered with; the malware arrived as part of a ZIP file downloaded directly from the site, according to accounts of Intel’s presentation.
This delivery mechanism is similar to the one used in the recent SolarWinds hack, in which Russians allegedly targeted government agencies and private companies through software updates. But there was a key difference: In Intel’s case, the malware initially turned up in just one of the firm’s thousands of servers — and then in just one other a few months later. Intel’s investigators concluded that the attackers could target specific machines, making detection much less likely. By contrast, malicious code went to as many as 18,000 SolarWinds users.
This posits an incredibly sophisticated attack — but, again, without supporting evidence. The report says that two steel companies based outside of the U.S. received compromised firmware in 2015 and 2018 from that update site. The Bloomberg story does not mention a 2016 case where Apple found an “infected driver” on one of its servers, which it determined to be accidental. All of these cases point back to an update server that Supermicro’s statement implies was not being served over HTTPS — pause for effect — until some time after that 2018 incident. That’s pretty bad security.
But is it possible that these were more isolated events, and not precise attacks? I am not doubting Intel’s investigative competence, but I am questioning whether the details of this internal presentation have been been accurately relayed to Robertson and Riley. There is no indication that the reporters saw the presentation themselves. If you shed the narrative and look at what is being described here, it sounds like APT17 — an infiltration team that FireEye attributes to the Chinese government — might have compromised Supermicro’s update server and planted malware for its clients to inadvertently install. Both Apple and Intel have denied that this was of notable concern. Malware is certainly a worry, though I am having trouble after all this time trusting the reporting I am basing my theory on. But there is a vast chasm between what has become a routine breach of a supplier with high-value clientele, and the supply chain hardware attack that Bloomberg has been reporting for two and a half years now without turning up a single piece of direct evidence.
There is more in Robertson and Riley’s new piece that one can nitpick; Matt Tait put together a comprehensive Twitter thread of concerns, with an acceptable summary:
FWIW, my money is on this whole saga being, if you dig deeply enough, just briefings related to the 2016 supermicro bad firmware update incident filtered through so many games of telephone that it’s eventually twisted itself into a story about tiny chips that never happened.
The problem remains that we just do not know what is going on here. This is not a trivial matter: there are many companies that rely on Supermicro hardware, and they need to know if there is any chance that any of it is compromised. We now have two lengthy and deeply reported stories with ostensibly alarming conclusions that have produced more confusion than clear answers.
A key indicator of the risk seen in these reports is how Supermicro’s clients behaved after these incidents were disclosed. It turns out that many of them — including Intel, the Pentagon, and NASA — have continued to use Supermicro as a supplier. One would think that, if there were concerns about the security of the company’s products, clients would be cancelling contracts left and right.
Everything about this story is wild and hard to believe. Apparently, there were three different vectors of vulnerabilities in Supermicro products: BIOS manipulation, malicious chips, and insecure firmware updates. In Robertson and Riley’s telling, all three have been exploited over the last eleven years. These attacks cover a few dozen high-profile companies and are being investigated by U.S. intelligence agencies; those agencies are briefing other orgnanizations about the danger. Yet there are only two journalists who have heard anything about this, despite this supposed supply chain attack being one of the most-watched information security stories in recent memory, and Supermicro still is not a prohibited vendor.
I would find this more compelling if this story were corroborated by more outlets with different sources, or if Robertson and Riley were able to produce more rigorous evidence. Then, at least, there would be some clarity. Right now, it feels like I’ve seen this movie before.