Once upon a time, Don <don@calis.blacksun.org> said:
Some facts:
3. The attack was large enough to affect many other customers in the same
data center- one with a lot of bandwidth off AT&T's backbone.
Is this what got Red Hat over the last couple of days as well? I think
they have a lot of their stuff on AT&T's network.
First, thanx for the nod, but there are some here who have dealt with more than I have. But I think I've seen enough to know something about it.
You can try things like "filter IP addresses which should not be going to node X", but what happens if the DDoS changes the network topology enough that you can't be certain users are going where you did not? If the DDoS is large, this is pretty much guaranteed.
Worse, suppose the topology changes for reasons unrelated to a DDoS. You could end up DoS'ing end users without an attack! (You could theoretically only put the filters in place when an attack is happening, but that has other problems - which may or may not be worse.)
Filtering on things like duplicated query IDs is not possible on router hardware doing 10s of Gbps or millions of PPS. And doing it on the server is not useful if there are more bits / pps than the router can process. Remember, servers can't answer packets that are dropped before they get to the servers.
Etc., etc., etc.
Overall, we are losing the war. What good providers, like the roots, Ultra, etc., do is to minimize the effect of any attack. If a "miscreant" fires the "DDoS of biblical proportions" and only 5% of users are affected, I consider that a success. Unfortunately, those 5% don't think so, but one can only do what one can do. Besides, if it truly is an attack of biblical proportion, those 5% are probably having much larger problems than name resolution.
Couple other comments:
From all indications I've seen (and most are not authoritative, but it's all the info I have), this was not a DDoS of "biblical proportions". There were no whole networks to go offline, there were no massive swaths of address space flapping, there were no entire peering points being congested, etc. A few Gbps does not count as "biblical" any more.
Whether this attack used spoof-source or not, BCP38 is _VITAL_, IMHO, to helping curb these things. It guarantees, at the very least, that you know where the attack is sourced. Filtering become much easier. Reaching the right operators to help with the problem becomes orders of magnitude easier. And if the miscreants just start using BotNets with real IP address, GOOD. It's not the End All Be All answer, but it is a _huge_ step in the right direction.
Unfortunately, as Jared has pointed out, the equipment vendors have to help the operators support this. So let's all call your favorite router vendor and ask them when they will have the "ip bcp38" config option. 
Unfortunately, as Jared has pointed out, the equipment vendors have
to help the operators support this. So let's all call your favorite
router vendor and ask them when they will have the "ip bcp38" config
option.
Even better would be the option: "no ip bcp38"
Make it so a conscious action is needed to disable it, but PLEASE put that in the release notes so when the config doesn't "change" we know that something really did change... 
R
Tellurian Networks - Global Hosting Solutions Since 1995
http://www.tellurian.com | 888-TELLURIAN | 973-300-9211
"Well done is better than well said." - Benjamin Franklin
5. AT&T (at least when I've dealt with them in their datacenters) does not support BGP community strings for null routing (or any strings for that matter
Lest anyone take me too seriously on that last point- AT&T hosting does have community strings for certain features- unfortunately not for null routing.
-Don
(My apologies for the earlier lack of a full email name)
The network hardware vendors do need to include the feature to support BCP-38. It'll help us out on a number of fronts especially with some of the recent cyber attacks.
We're in process of reaching out to many of the companies and many providers to encourage the implementation of BCP-38. We've gotten a lot of great feedback from many of you and its greatly appreciated. You know who you are
Especially some of the feedback related to the hardware OS issues.
-Jerry
Jerry@jdixon.com or jerry.dixon@us-cert.gov
5. AT&T (at least when I've dealt with them in their datacenters) does not
support BGP community strings for null routing (or any strings for that
matter
would need to call AT&T and request a filter/null route. Since AT&T
operations is based in Singapore (again this was last time I dealt with
them) I'm sure getting those filters/routes in probably doesn't happen
nearly fast enough. I have heard that AT&T is currently in the process of
setting up communities- maybe someone who knows more could comment.
Well, this is not exactly true. AT&T does support BGP communities,
although their communities aren't all that powerful, IMO. To my
knowledge, you are correct when you say that they do not support any
null-routing capabilities. I would love to find out the procedure
and string required to request/implement null routing via a community.
For those who would like to see AT&T's official guide, it can be found at:
http://www.onesc.net/communities/as7018
charles
...
As pointed out by Rob Seastrom in private email, RFC2182 addresses things
of biblical proportions - such as dispersion of nameservers geographically
and topologically. Having 3 secondaries, only one of them on separate /24,
and none of them on topologically different network does not qualify.
...
ns1.register.com. 600 IN A 216.21.234.96
ns2.register.com. 600 IN A 216.21.226.96
ns3.register.com. 600 IN A 216.21.234.97
ns4.register.com. 600 IN A 216.21.226.97
This is two pairs, each pair in a single /24 (or /26), and there are
ways in which each of these hosts could be in a widely different spot
from the other three, or in several different spots.
Why am I saying this? Most of the folks here know this and how to do
this even better than I do.
I am not saying that register.com IS doing this, just that you can't say
that they're NOT just from this evidence.
And by now it's moot anyway.
I think Alex could have included a few lines of traceroute to these hosts
showing that they all end behind:
7 tbr1-p014001.wswdc.ip.att.net (12.123.8.98) 9.754 ms 9.685 ms 9.608
ms
8 tbr1-cl4.sl9mo.ip.att.net (12.122.10.30) 29.708 ms 29.593 ms 33.498
ms
9 12.122.85.178 (12.122.85.178) 36.300 ms 28.558 ms 28.521 ms
So... it sorta looks like both /24's are behind something in StLouis,
Missouri ( to me atleast ).
Once upon a time, Chris L. Morrow <christopher.morrow@verizonbusiness.com> said:
I think Alex could have included a few lines of traceroute to these hosts
showing that they all end behind:
7 tbr1-p014001.wswdc.ip.att.net (12.123.8.98) 9.754 ms 9.685 ms 9.608
ms
8 tbr1-cl4.sl9mo.ip.att.net (12.122.10.30) 29.708 ms 29.593 ms 33.498
ms
9 12.122.85.178 (12.122.85.178) 36.300 ms 28.558 ms 28.521 ms
Also, it looks like anyone filtering on ARIN boundaries won't even see
that. Register.com has 216.21.224.0/20 assigned, but announces 7 /24s
and 2 /22s out of it.
My tests from 2 years ago showed the same thing, both /24s were behind
the same system in Exodus' NYC DC in Manhattan (IIRC). That is what
prompted me to move everything to the rcom partner side which uses eNom.
-Jim P.
My tests from 2 years ago showed the same thing, both /24s were behind
the same system in Exodus' NYC DC in Manhattan (IIRC). That is what
prompted me to move everything to the rcom partner side which uses eNom.
I don't know about a "partner" side but their premium service was always run by Register.com themselves. The servers were in a number of locations across the world. Whether any of this remains true today I have no idea.
Register.com may have also resold eNom services but I doubt that had anything to do with their premium service.
-Don