Fairly abstract - Facebook Engineering - https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A10158791436142200%7D&path=%2Fnotes%2Fnote%2F&_rdr
Also, Cloudflare’s take on the outage - https://blog.cloudflare.com/october-2021-facebook-outage/
FYI,
/John
The FB one seems to be from a previous event. Downtime doesn't match,
visible flaw effects don't either.
Rubens
Fairly abstract - Facebook Engineering - https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A10158791436142200%7D&path=%2Fnotes%2Fnote%2F&_rdr
My bad - might be best to ignore the above post as it is a unconfirmed/undated post-mortem that may reference a different event.
Also, Cloudflare’s take on the outage - https://blog.cloudflare.com/october-2021-facebook-outage/
The Cloudflare writeup looks quite solid (loss of network → no DNS servers → major issue)
/John
They have a monkey patch subsystem. Lol.
Mike
Fairly abstract - Facebook Engineering - https://m.facebook.com/nt/screen/?params={"note_id"%3A10158791436142200}&path=%2Fnotes%2Fnote%2F&_rdr <https://m.facebook.com/nt/screen/?params={"note_id":10158791436142200}&path=/notes/note/&_rdr>
I believe that the above link refers to a previous outage. The duration of the outage doesn't match today's, the technical explanation doesn't align very well, and many of the comments reference earlier dates.
Also, Cloudflare’s take on the outage - https://blog.cloudflare.com/october-2021-facebook-outage/
This appears to indeed reference today's event.
One of the replies say it's from February, so year.
Mike
Fairly abstract - Facebook Engineering -
https://m.facebook.com/nt/screen/?params={"note_id"%3A10158791436142200}&path=%2Fnotes%2Fnote%2F&_rdr
<https://m.facebook.com/nt/screen/?params={"note_id":10158791436142200}&path=/notes/note/&_rdr>My bad - might be best to ignore the above post as it is a
unconfirmed/undated post-mortem that may reference a different event.
If I'm reading the source correctly, the timestamp inside is for 09 SEP
2021 14:22:49 GMT (Unix time 1631197369). Then again, I may not be
reading it correctly. 
The CF post mortem looks sensible, and a good summary of what we all saw from the outside with BGP routes being withdrawn.
Given the fragility of BGP, this could still end up being a malicious attack.
-mel via cell
Update about the October 4th outage
Thanks for the posting. How come they couldn't access their routers via their OOB access?
-Hank
Yes, actually, they do. They use Chef extensively to configure
operating systems. Chef is written in Ruby. Ruby has something called
Monkey Patches. This is where at an arbitrary location in the code you
re-open an object defined elsewhere and change its methods.
Chef doesn't always do the right thing. You tell Chef to remove an RPM
and it does. Even if it has to remove half the operating system to
satisfy the dependencies. If you want it to do something reasonable,
say throw an error because you didn't actually tell it to remove half
the operating system, you have a choice: spin up a fork of chef with a
couple patches to the chef-rpm interaction or just monkey-patch it in
one of your chef recipes.
Regards,
Bill Herrin
129.134.30.0/23, 129.134.30.0/24, 129.134.31.0/24. The specific routes covering all 4 nameservers (a-d) were withdrawn from all FB peering at approximately 15:40 UTC.
Cheers,
Jeff
While Ruby indeed has a chain-saw (read: powerful, dangerous, still the tool of choice in certain cases) in its toolkit that is generally called “monkey-patching”, I think Michael was actually thinking about the “chaos monkey”,
https://netflix.github.io/chaosmonkey/
That was a Netflix invention, but see also
Grüße, Carsten
Maybe withdrawing those routes to their NS could have been mitigated by having NS in separate entities.
Let's check how these big companies are spreading their NS's.
$ dig +short facebook.com NS
d.ns.facebook.com.
b.ns.facebook.com.
c.ns.facebook.com.
a.ns.facebook.com.
$ dig +short google.com NS
ns1.google.com.
ns4.google.com.
ns2.google.com.
ns3.google.com.
$ dig +short apple.com NS
a.ns.apple.com.
b.ns.apple.com.
c.ns.apple.com.
d.ns.apple.com.
$ dig +short amazon.com NS
ns4.p31.dynect.net.
ns3.p31.dynect.net.
ns1.p31.dynect.net.
ns2.p31.dynect.net.
pdns6.ultradns.co.uk.
pdns1.ultradns.net.
$ dig +short netflix.com NS
ns-1372.awsdns-43.org.
ns-1984.awsdns-56.co.uk.
ns-659.awsdns-18.net.
ns-81.awsdns-10.com.
Amnazon and Netflix seem to not keep their eggs in the same basket. From a first look, they seem more resilient than facebook.com, google.com and apple.com
Jean
Per o comments, the linked Facebook outage was from around 5/15/21
I think that was from an outage in 2010:
https://engineering.fb.com/2010/09/23/uncategorized/more-details-on-today-s-outage/
Rumour is that when the FB route prefixes had been withdrawn their door authentication system stopped working and they could not get back into the building or server room
My speculative guess would be that OOB access to a few outbound-facing
routers per DC does not help much if a configuration error withdraws the
infrastructure prefixes down to the rack level while dedicated OOB to
each RSW would be prohibitive.
Well, doesn't really matter if you can resolve the A/AAAA/MX records, but you can't connect to the network that is hosting the services.
At any rate, having 3rd party DNS hosting for your domain is always a good thing to have. But in reality, it only hits the spot if the service is also available on a 3rd party network, otherwise, you keep DNS up, but get no service.
Mark.
Maybe withdrawing those routes to their NS could have been mitigated by having NS in separate entities.
Assuming they had such a thing in place , it would not have helped.
Facebook stopped announcing the vast majority of their IP space to the DFZ during this. So even they did have an offnet DNS server that could have provided answers to clients, those same clients probably wouldn’t have been able to connect to the IPs returned anyways.
If you are running your own auths like they are, you likely view your public network reachability as almost bulletproof and that it will never disappear. Which is probably true most of the time. Until yesterday happens and the 9’s in your reliability percentage change to 7’s.