Would be remiss in our duties if we didn't also link
AWS' blog, in response to the Bloomberg article.
It would be better for them(AMZN, SMCI, AAPL) to prove that these events did not take place - in court.
In the opposite case, even if this article is full of inaccuracies, judging by the discussions of security specialists, the scenario indicated in the article is quite possible.
Unpopulated SOIC-8 near populated SOIC-16 flash for BMC firmware is sweet spot for custom MCU - snooping on flash SPI(most likely) bus and probably altering some data.
At the same time there will be a good precedent, if this article is fabricated - such journalists need to be taught a lesson.
And if they wont go to the court, there is something to think about.
It would be better for them(AMZN, SMCI, AAPL) to prove that these
events did not take place - in court.
"Can't prove a negative."
In the opposite case, even if this article is full of inaccuracies,
judging by the discussions of security specialists, the scenario
indicated in the article is quite possible.
The Bloomberg article described them as looking like 'signal
conditioning couplers" on the motherboard. There is no such part on
server boards but maybe they meant optoisolators or power conditioning
capacitors. The former is a hard place to tweak the BMC from without a
high probability of crashing it. The latter doesn't touch the data
lines at all.
They also quoted someone describing such a hack as being "like
witnessing a unicorn jumping over a rainbow." I agree.
Regards,
Bill Herrin
You overlook the obvious case - that it *looks* like Yet Another Filter Cap
but it's actually a microcontroller wired into a useful SPI bus....
It would be better for them(AMZN, SMCI, AAPL) to prove that these events did not take place - in court."Can't prove a negative."
You can in effect do so by suing for defamation. It’s then up to the person who has made allegedly defamatory claims to prove their claims. If they can’t prove their claims in court then the claims are, in effect, proven to be false.
However, I’m not sure that Amazon, Apple or Supermicro have actually been defamed by the article in question. In other words, there could be nothing to sue for. The PLA and Chinese government would have been defamed (if the claims are untrue) but that’s a different matter. Any lawyers wants to offer an opinion?
The Bloomberg article described them as looking like 'signal conditioning couplers" on the motherboard. There is no such part on server boards but maybe they meant optoisolators or power conditioning capacitors. The former is a hard place to tweak the BMC from without a high probability of crashing it. The latter doesn't touch the data lines at all.
The mystery object in the pictures in the article seemed to me to (sort of) resemble a surface mount power conditioning capacitor. Note that there was no suggestion that the mystery objects were connected in place of capacitors; the article merely claimed that they were visually disguised. They would obviously have to connect to data lines somewhere to do what is claimed.
They also quoted someone describing such a hack as being "like witnessing a unicorn jumping over a rainbow." I agree.
It doesn’t seem so unreasonable to me. If true, this is not a matter of fitting the mystery components to random hardware and hoping that they go somewhere useful. Instead, these were specific models of hardware being manufactured for specific customers for use in specific locations/roles. In other words, it was near-guaranteed that the hardware (or at least some of it) would end up being used in a location that carried ‘interesting’ target data. As such, this would be, if true, an example of very carefully targetted espionage, not some random lucky miracle.
One wonders if, with the quality of BMC’s in general being as low as it is, and their security as bad, if any sort of extraneous hardware is necessary to facilitate a compromise of a system where any of these BMCs is present. Keep in mind many of these devices for some time included a “feature” where telnet’ing to a specific port and typing in a short string would result in a response containing a cleartext list of usernames and cleartext passwords. 
Though Bloomberg didn't go out of their way to say it, the photos were
"representative" of the chip supposedly found. Were they in possession
of any hard evidence of the chips' existence, they'd have said so.
Regards,
Bill Herrin
I was wondering about where this chip tapped into all of the data and timing lines it would need to have access to. It would seem that being really small creates even more problems making those connections. I am a little doubtful about the article. It would seem to me better to create a corrupted copy of something like a front side bus chipset, memory controller or some other component that handles data lines than create a new component that would then require a motherboard redesign to integrate correctly. It would seem that as soon as the motherboard design was changed someone would wonder "hey, where are all those data lines going?" It would also require less people in on the plan to corrupt or replace a device already in the design. All you need is a way to intercept the original chip supply and insert your rogue devices.
On the opposite side of the argument, does anyone think it is strange that all of the companies mentioned in the article along with the PRC managed to get a simultaneous response back to Bloomberg. Seems pretty pre-calculated to me. Or did some agency somewhere tell everyone they better shut up about the whole thing?
Steven Naslund
Chicago IL
Just theory - tapping on same lines as SPI flash (let's assume it is not QSPI), so we are "in parallel", as "snooper" chip.
First - it can easily snoop by listening MISO/MOSI/CS/CLK.
When required data pattern and block detected during snooping, it can remember offset(s) of required data.
When, later, BMC send over MOSI request for this "offset", we override BMC and force CS high (inactive), so main flash chip will not answer, and answer instead of him our, different data from "snooper".
Voila... instead of root:password we get root:nihao
As they mentioned in their responses, Bloomberg has been calling each
of the companies for comment on the developing article for months to a
year. That's why they all knew it was coming.
Regards,
Bill Herrin
It is definitely more desirable to try and tap a serialized data line than the parallel lines. The thing that made me most suspicious of the article is why would anyone add a chip. It requires power and connections that a highly detectable. Motherboard designs are very complex in the characteristics of data buses so it is not so easy to just extend or tap into them without having negative effects (which brings the board under scrutiny that we don't want). Why not embed our rogue chip inside the case of a chip that is already controlling the bus or memory we want to play with? It would be really hard to detect without x-ray of all of the system chipsets.
The other thing I am highly skeptical of is the suggestion of attempting to tap sensitive intel agency systems this way. Talking to a C&C server is suicide from within their network. How long do you think it would take them to detect a reach out to the Internet from inside? How are you going to get the data from the outside back into their network? You still have to defeat their firewalls to do it. If this was targeted to specialized video processing server then would it not be unusual for them to be talking to some random IP address on the Internet?
Steven Naslund
Chicago IL
I can read but I am really finding it hard to believe that they all agreed to even comment on it at all. Especially the PRC. Next question would be that if Bloomberg was calling me for "months to a year" why not get out in front of it in the first place? The whole story and its responses are very flakey at least to my BS detector.
Steven Naslund
Chicago IL
Oh, at least 2 or 3 years. Or that's how long it took to be noticed the *last* time.
The US’ extensive reliance on third party commercial contractors to implement a lot of programs, means that despite laws and SOW/PWS for their contracts, many contractors do have sensitive data on their networks with a gateway out to the public Internet. I have seen it. I have cringed at it. SIGINT agencies in many cases rely on people being less than perfectly reliable in their data hygiene practices to extract useful information.
I’m sure that all of the super secret squirrel stuff is going on properly inside SCIFs, but mistakes will be made. Now draw an imaginary venn diagram overlap of human mistakes with places that handle classified data.
If I understand the article correctly, all the ‘infected’ systems were built for outsourced service providers so not intended directly for the most sensitive of systems. Still, I agree that network activity is inevitably going to be seen in any modern competent network. In fact, the article states that odd network traffic is how Apple found out about the implants.
I have observed that a common trait in technically complex stories like this is that we are not seeing the whole story. Key facts that cause everything to make sense to technical readers are often left out, either because those who have the information cannot release it (for safety or security reasons) or because it’s perceived as too complex for the readership to understand. Sometimes these issues even result in deliberate inaccuracies being introduced.
To put it another way: Considering that, if true, these were carefully targeted attacks it is possible that there were other ways to exfiltrate the target data that have been glossed over.
That said, even in highly complex or high cost plans, people sometimes make basic errors. Misplaced decimal places, wrong units, etc. Perhaps relaying on network access was another basic error.