Submitted without comment:
http://inside.godaddy.com/inside-story-happened-godaddy-com-sept-10-2012/
-Tk
I know that I should know better then comment on networks others then
my own, ( and I know to never comment on my own publicly 
)
But here goes, 210x the size of normal really? 210% I'd have a hard
time believing. Did anyone else anywhere see a route leak equal to
larger then the entire Internet that day, anywhere else that could of
caused this?
I won't even get into max-prefix and how we've managed this long with
someone people still not setting them.
-jim
If the device was only expecting 2K or so internal routes, getting hit with
the 440K routes in the DFZ would be 210x....
Yes that math would work, but if your device can't handle 1x Internet
routing and your running without some serious max-prefix/filters it
says even more about your IP eng team then I'd be willing to comment
on.
-jim
Is it plausible that Godaddy's internal network only normally has a few thousand BGP routes? 210 x a few thousand would run most modern gear out of FIB space.
The "my DNS is broken, are we really being DDoS'd on udp/53 at the same time?" thing, I've seen, and I can imagine it being very confusing to someone seeing it for the first time.
I know that I should know better then comment on networks others then
my own, ( and I know to never comment on my own publiclyBut here goes, 210x the size of normal really? 210% I'd have a hard
time believing. Did anyone else anywhere see a route leak equal to
larger then the entire Internet that day, anywhere else that could of
caused this?
it's pretty easy to inadvertently leak a copy of the internet from one vrf to another and effectively install two copies of the internet routes in your fib...
There are plently of cases where you might to that or something similar on purpose, which is all good and
well if you have 2million route fib capacity but less awesome if you have 512K route capacity linecards at this point. if you get those routes from a private peer on some non-internet-vrf well that might imply that your filter policy needs some tuning.
On outages GoDaddy provided a tiny bit more information.
[quote]
Obviously the explanation of the incident had to be consumed by the
general public, however we encountered an unknown bug that was found
which started the domino effect. Aside from this group, that level of
detail wouldn't be understood by a majority of the recipients.
With that said, please feel free to take this off list with Jason or
Myself.
Mike Dob
Manager, Network Engineering
[/quote]
No information has been provided on what sort of "unknown bug" this
was. A bug in code that GoDaddy wrote? A bug in their route servers
or router OS, which others may also use and might want to be aware of?
- -DMM
In case you missed it
Just ask yourself how many times you have seen a Godaddy IP/NOC person post anything to NANOG or to any other technical forum?
-Hank