I'm seeing *.register.com down (including ns*) from everywhere. Just a
heads-up. Would be interesting to see the RFO for that one, including the
"why we didn't have any DNS servers offsite or used anycast to at least
limit amount of damage".
-alex
just guessing but:
1) it's 'hard'
2) there is very little 'sla' on anything register.com does ? (no idea,
not a customer)
3) it's 'hard'
4) why? this is the 100yr flood scenario for them? (perhaps)
cost/benefit ... analyze 
It's possible that that 'analyze' part may
change if there is significant fall-out from this event though.
<rant>
The reason the public facing DNS is poorly set up at the majority of
institutions is the IT guy says "lets bring it in house to give us more
control, how hard can it be?".
When if they had left it with their ISP it would be done right (along with the
thousands of others that the ISP does right).
I've seen it done dozens of times when consulting.
I have data from a personal survey that confirms this is the leading cause of
poor DNS configuration and lack of redundancy in my part of the UK.
I even have a few domains we slave to servers across several continents, and
otherwise clueful IT people pick SOA settings that still cause their domains
to expire too quickly when, had they left it to us, it would "just work".
(okay I could override those settings, but if I do that why bother letting
them master it in the first place?! "we delegated control to you, and then
overrode all your settings because they were stupid?!"). So don't let the IT
guy be a hidden master either, just leave it to the ISP.
How I reach the zillions of IT guys out there to say "don't do DNS inhouse,
you'll only mess up" is the remaining question; slashdot?
</rant>
wanna present all this rant and the proper solution to <rant> at the next
nanog?
Chris L. Morrow wrote:
just guessing but:
1) it's 'hard'<rant>
How I reach the zillions of IT guys out there to say "don't do DNS inhouse,
you'll only mess up" is the remaining question; slashdot?
</rant>wanna present all this rant and the proper solution to <rant> at the next
nanog?
Perhaps we should be celebrating the upcoming 10th anniversary of bcp 17.
Preaching to the choir, I suspect. Or are the people who are known
offenders likely to actually be reached by the presentation?
This is a common thread I keep encountering - the sites with enough clue to
send somebody to NANOG aren't usually the sites I need to send a cluegram to.
If anybody has a viable suggestion for that...
I'm seeing *.register.com down (including ns*) from everywhere. Just a
heads-up.
I'll take your word on exhaustively checking every possible address. BTW, do you mean nameservers down, webservers down, or something else? Did the Internet break?
Would be interesting to see the RFO for that one, including the
"why we didn't have any DNS servers offsite
They colo in more than a half-dozen facilities around the world.
or used anycast to at least limit amount of damage".
I also have information from a pretty good source that they actually do quite a bit of anycast.
matto
--matt@snark.net------------------------------------------<darwin><
Moral indignation is a technique to endow the idiot with dignity.
- Marshall McLuhan
> I'm seeing *.register.com down (including ns*) from everywhere. Just a
> heads-up.I'll take your word on exhaustively checking every possible address.
BTW, do you mean nameservers down, webservers down, or something else?
Did the Internet break?
*.register.com means nameservers, webservers, whois servers, etc. Of
course, Internet does not break, but we've received quite a number of
calls about "internet is down" - given that register.com serves a large
number of domains, yes, this is operationally affecting.
> Would be interesting to see the RFO for that one, including the "why
> we didn't have any DNS servers offsiteThey colo in more than a half-dozen facilities around the world.
> or used anycast to at least limit amount of damage".
I also have information from a pretty good source that they actually do
quite a bit of anycast.
Not that I can see - possibly that depends on a specific domain's
webservers.
The glue servers for register.com themselves:
Name: ns1.register.com
Address: 216.21.234.96
Name: ns2.register.com
Address: 216.21.226.96
Name: ns3.register.com
Address: 216.21.234.97
Name: ns4.register.com
Address: 216.21.226.97
(note just two different /24s)
Both of those /24s were down/down about 30 minutes ago, and are
flapping/flapping now.
route-views.oregon-ix.net>show ip bgp 216.21.234.73
...
BGP routing table entry for 216.21.234.0/24, version 5214460
701 7018 4264 13910, (suppressed due to dampening)
157.130.10.233 from 157.130.10.233 (137.39.3.60)
Origin IGP, localpref 100, valid, external
Dampinfo: penalty 898, flapped 5 times in 00:35:15, reuse in 00:03:50
route-views.oregon-ix.net>show ip bgp 216.21.226.97
BGP routing table entry for 216.21.226.0/24, version 5214460
...
701 7018 4264 13910, (suppressed due to dampening)
157.130.10.233 (inaccessible) from 157.130.10.233 (137.39.3.60)
Origin IGP, localpref 100, valid, external
Dampinfo: penalty 861, flapped 5 times in 00:36:13, reuse in 00:03:00
From various vantage points, both /24s are routed exactly the same (7018
in NYC).
-alex
They are apparently under a multi-gbps ddos of "biblical proportions".
--matt@snark.net------------------------------------------<darwin><
Moral indignation is a technique to endow the idiot with dignity.
- Marshall McLuhan
They are apparently under a multi-gbps ddos of "biblical proportions".
locusts?
racks which face backwards being turned into pillars of salt?
death of the primaries?
floods?
sorry. been a hard day.
randy
As pointed out by Rob Seastrom in private email, RFC2182 addresses things
of biblical proportions - such as dispersion of nameservers geographically
and topologically. Having 3 secondaries, only one of them on separate /24,
and none of them on topologically different network does not qualify.
Given that register.com is/was public (I think?) - I wonder what are their
sarbox auditors saying about it now 
Compliance of icann-accredited gtld-registrars with rfc2182 might be a
good subject for research (again, thanks to rs for idea)....
-alex
> > I'm seeing *.register.com down (including ns*) from everywhere.
> They are apparently under a multi-gbps ddos of "biblical proportions".
i wonder if that's due to the spam they've been sending out?
As pointed out by Rob Seastrom in private email, RFC2182 addresses things
of biblical proportions -
no. really, not.
such as dispersion of nameservers geographically
and topologically. Having 3 secondaries, only one of them on separate /24,
and none of them on topologically different network does not qualify.
there is no zone anywhere, including COM, the root zone, or any other, that
is immune from worst-case DDoS. anycast all you want. diversify. build a
name service infrastructure larger than the earth's moon. none of that will
matter as long as OPNs (the scourge of internet robustness) still exist.
Given that register.com is/was public (I think?) - I wonder what are their
sarbox auditors saying about it now
that's an easy but catty criticism, and baseless. i'm sure that some way
could be found to improve register.com's infrastructure, and i don't just
mean by stopping the spamming they've been doing. but it's not trivial and
in the face of well-tuned worst-case DDoS, nothing will help.
Compliance of icann-accredited gtld-registrars with rfc2182 might be a
good subject for research (again, thanks to rs for idea)....
i've been wondering if ICANN's accredidation could be revoked for spammers,
and register.com has indeed been spamming. and it may also be that they
are out of compliance with RFC 2182. but that would be like catching al
capone for income tax evasion just because you couldn't pin murder on him.
(OPNs = Other People's Networks)
> > > I'm seeing *.register.com down (including ns*) from everywhere.
> > They are apparently under a multi-gbps ddos of "biblical
> > proportions".i wonder if that's due to the spam they've been sending out?
Paul, this isn't nanae. Let's not sling accusations like that wildly.
> As pointed out by Rob Seastrom in private email, RFC2182 addresses things
> of biblical proportions -no. really, not.
> such as dispersion of nameservers
> geographically and topologically. Having 3 secondaries, only one of
> them on separate /24, and none of them on topologically different
> network does not qualify.there is no zone anywhere, including COM, the root zone, or any other,
that is immune from worst-case DDoS. anycast all you want. diversify.
build a name service infrastructure larger than the earth's moon. none
of that will matter as long as OPNs (the scourge of internet robustness)
still exist.
This isn't 2001, and, I will argue that it *is*, in fact, possible to be
protected from a "worst case" ddos, and not at obscene price. However,
even if you argue that point, there's no excuse for not being prepared at
all, and not following the BCP. While we all may be guilty of not having
topologically/geographically diverse DNS - for someone whose core business
is DNS, that's unexcusable.
> Given that register.com is/was public (I think?) - I wonder what are their
> sarbox auditors saying about it nowthat's an easy but catty criticism, and baseless. i'm sure that some
way could be found to improve register.com's infrastructure, and i don't
just mean by stopping the spamming they've been doing. but it's not
trivial and in the face of well-tuned worst-case DDoS, nothing will
help.
Well, let's talk about "worst-case ddos". Let's say, 50mpps (I have not
heard of ddos larger that that number). Let's say, you can sink/filter
100kpps on each box (not unreasonable on higher-end box with nsd). That
means, you should be able to filter this attack with ~500 servers,
appropriately place. Say, because you don't know where the attack will
come in, you need 4 times more the estimated number of servers, that's
2000 servers. That's not entirely unreasonable number for a large enough
company.
I know that the above was just rough back-of-the-envelope, and things are
far more complicated than that, but this discussion does not really belong
to nanog-l.
> Compliance of icann-accredited gtld-registrars with rfc2182 might be a
> good subject for research (again, thanks to rs for idea)....
i've been wondering if ICANN's accredidation could be revoked for
spammers, and register.com has indeed been spamming. and it may also be
that they are out of compliance with RFC 2182. but that would be like
catching al capone for income tax evasion just because you couldn't pin
murder on him.
Things like that, and accusations like that, I don't think really belong
to nanog-l.
(speaking for myself only)
There are two sides to rcom, the mom&pop side (aka register.com) and the
partner side (Rconnection, for folks with ~25+ domains registered). On
the mom&pop side they don't have (as far as I am concerned) a highly
redundant and distributed DNS system. That opinion is based on a few
hours of research abt 2 years ago. Over on the partner side they
outsource the DNS systems for their customers to eNom, which does use a
highly redundant and distributed anycast setup. I haven't seen any
problems wrt DNS for my systems today (eNom via rcom), so I can only
presume the OP was referring to the mom&pop side of rcom.
-Jim P.
Good god. It isn’t like they are some borderline case or anything.
Chris
i wonder if that's due to the spam they've been sending out?
Paul, this isn't nanae. Let's not sling accusations like that wildly.
Accusations and objective facts are two separate things.
there is no zone anywhere, including COM, the root zone, or any other,
that is immune from worst-case DDoS. anycast all you want. diversify.
build a name service infrastructure larger than the earth's moon. none
of that will matter as long as OPNs (the scourge of internet robustness)
still exist.This isn't 2001, and, I will argue that it *is*, in fact, possible to be
protected from a "worst case" ddos, and not at obscene price.
You are mistaken.
However,
even if you argue that point, there's no excuse for not being prepared at
all, and not following the BCP. While we all may be guilty of not having
topologically/geographically diverse DNS - for someone whose core business
is DNS, that's unexcusable.
We agree.
Given that register.com is/was public (I think?) - I wonder what are their
sarbox auditors saying about it nowthat's an easy but catty criticism, and baseless. i'm sure that some
way could be found to improve register.com's infrastructure, and i don't
just mean by stopping the spamming they've been doing. but it's not
trivial and in the face of well-tuned worst-case DDoS, nothing will
help.Well, let's talk about "worst-case ddos". Let's say, 50mpps (I have not
heard of ddos larger that that number). Let's say, you can sink/filter
100kpps on each box (not unreasonable on higher-end box with nsd). That
means, you should be able to filter this attack with ~500 servers,
appropriately place. Say, because you don't know where the attack will
come in, you need 4 times more the estimated number of servers, that's
2000 servers. That's not entirely unreasonable number for a large enough
company.
Even assuming your numbers, which I do not grant, you are still mistaken.
There is no single "appropriately[sic] place" which can absorb 50Mpps. If you meant "appropriately placed" (as in topologically dispersed locations), a well crafted attack could still guarantee _at least_ a partial DoS from an end user PoV.
It is essentially impossible to distinguish end-user requests from (im)properly created DoS packets (especially until BCP38 is widely adopted - i.e. probably never). Since there is no single place - no 13 places - which can withstand a well crafted DoS, you are guaranteed that some users will not be able to reach any of your listed authorities.
This is not speculation, this is fact. All a good provider can do, even with 1000s of server, is minimize the impact of any DoS.
Oh, and putting 2K servers into the "right" places is not a trivial expense, even for a large company. Last time I checked, 10GE pipes were not handed out for free. And you can't just rack these things in mom-and-pop colo saying "well, it has a GigE on the motherboard" when the colo has an OC3 to the 'Net. The Cap- and Op-Ex involved in doing what you suggest properly is large enough to probably be prohibitively expensive for a company like register.com.
I know that the above was just rough back-of-the-envelope, and things are
far more complicated than that, but this discussion does not really belong
to nanog-l.
We disagree. Keeping large name servers running is _absolutely_ a network operations topic. Not only is the defense mostly network based (since the network is the most likely thing to break), network operators are the people who get the phone calls when DNS does break.
Botnets were the topic at today's Info Security conference in New York City. <http://www.infosecurityevent.com> Coincidences? Or just as random as your iPod shuffle?
Jose Nazario estimated that there were 10,352 botnets active on the Internet earlier this year. You will probably always be outnumbered on
the public Internet.
There is no single "appropriately[sic] place" which can absorb 50Mpps.
If you meant "appropriately placed" (as in topologically dispersed
locations), a well crafted attack could still guarantee _at least_ a
partial DoS from an end user PoV.It is essentially impossible to distinguish end-user requests from
(im)properly created DoS packets (especially until BCP38 is widely
adopted - i.e. probably never). Since there is no single place - no 13
places - which can withstand a well crafted DoS, you are guaranteed that
some users will not be able to reach any of your listed authorities.
Yeah - I know it hard-to-impossible to do that, and it is a tug-of-war
between worm writers (to generate queries indistinguishable from real
client-resolver-generated queries) and trying-to-detect-malformed-queries
(such as duplicated qid, or from IP space that shouldn't be hitting this
specific node). You probably dealt with more ddos than rest of us
combined, so I bow to your superior knowledge.
I know that the above was just rough back-of-the-envelope, and things
are far more complicated than that, but this discussion does not really
belong to nanog-l.We disagree. Keeping large name servers running is _absolutely_ a
network operations topic. Not only is the defense mostly network based
(since the network is the most likely thing to break), network operators
are the people who get the phone calls when DNS does break.
Sorry - I meant that discussion whether or not register.com is spamming
isn't somewhat offtopic. Of course, DNS operations (and particularly
dealing with "biblical scale" ddos) is very much on-topic.
-alex
There's nothing "wild" about it -- Paul is one of the most sober,
reasoned observers of the spam problem, and if he told me that
my servers were sending spam, then I'd darn well go investigate.
Right now.
Besides -- it's not like this isn't common knowledge in the anti-spam
world. I'm sure I'm not the only one who's had unsatisfying correspondence
with register.com wherein they refuse to lift a finger to stop the abuse
from/facilitated by their operation.
---Rsk
As pointed out by Rob Seastrom in private email, RFC2182 addresses things
of biblical proportions - such as dispersion of nameservers geographically
and topologically. Having 3 secondaries, only one of them on separate /24,
and none of them on topologically different network does not qualify.
Register.com offered several models for DNS service including distributed anycast based services. Considering what I've heard about the scale of the attack I'm glad they chose not host their own domain name on the anycast networks- it simply would have taken more people down.
Some facts:
1. I've spoken with some AT&T engineers about what was going on. According to them this was (as mentioned earlier) a multi gigabit attack that came in through every peer on the AT&T network. Anycasting would not have fixed this problem- the attack was too large and too diverse. (I guess if they had 10 gige pipes and pops all over the planet- maybe. But that's not exactly a valid business model.)
2. These were not spoofed source addresses. This looks like a rather large botnet sending real traffic.
3. The attack was large enough to affect many other customers in the same data center- one with a lot of bandwidth off AT&T's backbone.
4. DNS is a tiny protocol. It's possible to send a LOT of small, but perfectly valid, DNS packets. The fact that the attack was multi gigabit per second is bad enough. Couple that with the packets all being really tiny and you have a recipe for routing disaster.
5. AT&T (at least when I've dealt with them in their datacenters) does not support BGP community strings for null routing (or any strings for that matter 
Think about that for a second. To stop an attack Register.com would need to call AT&T and request a filter/null route. Since AT&T operations is based in Singapore (again this was last time I dealt with them) I'm sure getting those filters/routes in probably doesn't happen nearly fast enough. I have heard that AT&T is currently in the process of setting up communities- maybe someone who knows more could comment.
The truth is that none of us has all the facts about what happened.
Given that register.com is/was public (I think?) - I wonder what are their
sarbox auditors saying about it now
Register.com is not public (If I recall correctly they were bought out a couple of years ago by a private firm). Furthermore if they were public I would think their stockholders might have something to say about spending large sums of money to prevent a DDoS which probably would not work anyway.
-Don
