WANTED: ISPs with DDoS defense solutions

Mailman List Mirror (Read Only)
1

We have some DDoS-sensitive customers asking us to refer them to the best ISPs for "in-the-core" DDoS defense. Other than UUnet (hi Chris!) and MFN, I'm not aware of any ISPs in North America developing a reputation for consistent DDoS defense. Could folks contact me either off-list or on-list?

It seems that large content providers and Tier2/3 bandwidth buyers would do well to collaborate on group RFP's for this type of thing to send the message to ISPs it is something to invest in (dare I say productize?). While UUnet's detection/blocking is great, it would be wonderful to see some more intelligent filtering of DDoS traffic ala RiverHead or similar approach that doesn't completely blackhole victim IPs.

Cheers,
-Lane
Equinix

2

[ obnoxious text wordwrapped :slight_smile: ]

We have some DDoS-sensitive customers asking us to refer them to the
best ISPs for "in-the-core" DDoS defense. Other than UUnet (hi Chris!)
and MFN, I'm not aware of any ISPs in North America developing a
reputation for consistent DDoS defense. Could folks contact me either
off-list or on-list?

It seems that large content providers and Tier2/3 bandwidth buyers
would do well to collaborate on group RFP's for this type of thing
to send the message to ISPs it is something to invest in (dare I
say productize?). While UUnet's detection/blocking is great, it
would be wonderful to see some more intelligent filtering of DDoS
traffic ala RiverHead or similar approach that doesn't completely
blackhole victim IPs.

  Well, there are a few things/issues here.

  One is the "security" of such filtering. As many times as
it's come up here saying "Filter your customers, it's important", how
many people out there have a strict policy for filtering them?
Would you want these same customers and providers that can not
get the filtering right in the first place to have the ability to
accidentally (or intentionally) leak a blackhole route to
your larger network? Yes, there is the ability to log bgp
updates to have accountability amongst other things, but the
more serious issue is that people are not doing effective filtering
[of announcements] in the first place.

  As far as I can tell these days, the US depends on
the Internet to be a utility. Always-on, and there is (for the most
part) sufficent interconnection that the choice between the top few
providers isn't as much a technical decision, but more of a financial
one. (There is no need to connect to MCI, Sprint and UUNet each to
avoid the peering congestion points as in the past).

  Equinix itself is demonstrating this with your "change providers
monthly" service that you offer.

  I think it will be some time before there will be
adoption of this across most of the networks. We want people to contact
our security team instead of "blackhole and forget" type solutions.

  If someone abuses the PSTN, or other networks they eventually
will get their service terminated. If people abuse their access by
launching DoS attacks, we need to catch them and get their access
terminated. It's a bit harder to trace than PSTN (or other netowrks)
but I feel of value to do so.

  - Jared

3

        If someone abuses the PSTN, or other networks they eventually
will get their service terminated. If people abuse their access by
launching DoS attacks, we need to catch them and get their access

Gee, wouldnt that be nice. Having personally dealt with one that had ~ 500 hosts involved on several dozen networks, I can confirm that of all the repeated pleas for help to said networks to track down the controlling party, I had a grand total of ONE (yes, 1 as in one above zero) who actually responded with a response beyond the auto-responders.... And that was to let me know that the user in question had already formatted their hard drive before the admin could see what was on the machine and who might have been controlling the machine.

It took several _weeks_ for all the attacking hosts to be killed off with several reminder messages to various networks. So I dont hold much optimism for actually tracking down the actual attacker.

         ---Mike

4

> If someone abuses the PSTN, or other networks they eventually
>will get their service terminated. If people abuse their access by
>launching DoS attacks, we need to catch them and get their access

Gee, wouldnt that be nice. Having personally dealt with one that had ~ 500
hosts involved on several dozen networks, I can confirm that of all the
repeated pleas for help to said networks to track down the controlling
party, I had a grand total of ONE (yes, 1 as in one above zero) who
actually responded with a response beyond the auto-responders.... And that
was to let me know that the user in question had already formatted their
hard drive before the admin could see what was on the machine and who might
have been controlling the machine.

It took several _weeks_ for all the attacking hosts to be killed off with
several reminder messages to various networks. So I dont hold much
optimism for actually tracking down the actual attacker.

  While I can have sympathy for this situation, you removed my
argument about the "DoS and forget".

  Lets say I am running www.example.com.

  I have it load-shared across a series of 5-10 machines, and
they all get DoS attacked via some worm, etc.. (ala the www1.whitehouse.gov)
with a large set of traffic.

  I can't just deem that IP unusable on my ARIN justification and
have my providers absorb the cost of the traffic at zero cost to me or
them. (well, unless they're getting the traffic on a customer link
and want to continue billing at that bandwidth overage rate :wink: )

  The router ports my upstream has invested (for peering) and
circuits for their network have a cost.

  If an attack lasts 10 minutes, yes, the blackhole is easy
to move, but what if it is coded to follow dns entries, honor ttl,
and continue to pound on devices.

  You can't just submit a route/form/whatnot to your provider
and have them leave in a null0/discard route indenfiately.

  I'm sorry you had poor luck tracking them down, but without
the providers putting the access controls necessary to prevent the
route-leak misconfiguration, I don't want to think about the instability
you (or others) are speaking of introducing if there is the ability
to distribute a null0 route to your upstream and accidentally leak
it.

  (sorry LINX members but ..)

  You should see the number of people who post to the LINX ops
list a month saying "whoops, we leaked routes, can you clear your
max prefix counters?"

  Imagine someone accidentally leaking your routes to their
upstream and tagging them with the community due to misconfiguration.

  - Jared

5

I understand the point you are making, but I am speaking just to the side comment you made, "we need to catch them and get their access." I totally agree with you. But based on my recent experiences with organizational responses, it seems NO ONE agrees with it in practice.

It seems all the discussion around DDoSes center on ways of coping with DDoSes, or mitigating the effects and not making 'the solutions worse than the problem.' However, there does not seem to be enough discussion and effort in to catching and prosecuting the people doing it. I would be at least happy with the "catching part." I recall one of our users was involved in a DoS once a few years back when the "giant pings" could crash MS boxes. The fact that his perceived anonymity was removed was enough to keep him from repeating his attacks....

         ---Mike

6

That's the heart of the problem. Anyone who's owned enough boxes can sit
there happily running a DDoS anonymously against a target because:

1) The OS/software/default settings for a lot of internet connected
machines are weak, making it easy to attack from multiple locations.

2) A lot of networks have no customer or egress filtering and make it a
lot more difficult to trace DDoS traffic because it generally uses faked
source addresses.

If these issues are addressed then it becomes a lot harder to remain
anonymous and starting DDoS attacks against targets that can trace you
becomes a lot less attractive.

Cheers,

Rich

7

> I recall one of our users was involved in a DoS once a few years back
> when the "giant pings" could crash MS boxes. The fact that his perceived
> anonymity was removed was enough to keep him from repeating his
> attacks....

If these issues are addressed then it becomes a lot harder to remain
anonymous and starting DDoS attacks against targets that can trace you
becomes a lot less attractive.

Sure, trace my attacks to the linux box at UW, I didn't spoof the flood
and you can prove I did the attacking how? You can't because I and 7 other
hackers all are fighting eachother over ownership of the poor UW student
schlep's computer...

The problem isn't the network, nor the filtering/lack-of-filtering, its a
basic end host security problem. Until that is resolved, the ability of
attackers to own boxes in remote locations and use them for malfeasance
will continue to haunt us. I would guess that the other owners of the
machines attacking Mike (assuming they got the emails he sent... big
assumption) probably said: "Great another person getting attacked from
that joker's win2k machine, hurray:(" and moved on about thier business.
They know that they can't get the end user to secure their machine and
they know that if the get him/her to reload the OS or 'clean' it of the
'virus' the problem will arise anew within 17 minutes :frowning:

I'm all for raising the bar on attackers and having end networks implement
proper source filtering, but even with that 1000 nt machines pinging 2
packet per second is still enough to destroy a T1 customer, and likely
with 1500 byte packets a T3 customer as well. You can't stop this without
addressing the host security problem...

8

But in the telco world, how often do you have people's home phones
trojanned and directed to 'DoS' another company? To pull that off
with great magnitude, you need a whole lot of coordinated access
to the physical plant, which is either impossible or extremely
noticeable. But in a scenario like that, if a telco user gets their
access canned, it's most likely because the telco user themself was
abusing their privileges, not getting abused by some random fool
attacking another user/company via their facilities just to swing
their nuts around anonymously.

But don't get it twisted, I agree with your idea of cooperation and
tracking but this is like chasing suicide bombers. You can kill a
drone or two or fifty, but new ones will pop up in their place. You
can kill the drone controller, but the drones will continue to
execute their mission as they were doing before, but now, without
any method or controller to tell them to stop attacking.

Not to mention, by cutting off the drone's Internet access, regular
users get caught in the crosshairs of the drone hunters. At the
same time, if you tell a user their computer is trojanned, but you
would like to bait it to catch the culprit, they'll get worried
about their personal data and either go on a formatting campaign,
or abandon the computer altogether (trashing it, selling it, giving
it away, etc).

I think one way to definitely help is by user education. ISPs should
kick out newsletters or advisories to their users, informing them of
the latest scam, spam, or exploit and how to protect themselves from
it or how to determine if the user is a victim of the exploit in
question. This is where telcos (with fraud departments) are usually
successful, every now and then you'll get some sort of info on the
latest trend to watch out for. You either get it directly from the
telco, or from some other 3rd party source that got it from the
telco or another person (examples: news, community bulletins, office
e-mails, etc). Too often do new users get brand spanking new Internet
access, and maybe a trial version of anti-virus software and the ISP
calls it a day, then the user is left to wander through the
wilderness.

Another big plus is network cooperation. Too often have attacks gone
unnoticed until someone becomes a target of the DoS and then throws
a fit over how no one is doing anything. (No, I'm not singling anyone
out). Granted, the general response to Slammer was better than usual,
but how often do companies with small T1 customers getting smacked
with 10-200Mbps get to prosecute or even at the least, identify the
attacker before, during, or after the filtering?

Let me stop now, this e-mail is way too long.

9

Yo Omachonu!

I guess you have not read Kevin Mitnick's new book yet. Better read
it before you make more statements like this.

RGDS
GARY

10

Sure, trace my attacks to the linux box at UW, I didn't spoof the flood
and you can prove I did the attacking how?

You can at least TRY and see where the controlling traffic stream is originating from. i.e. if crap is coming out of box X, all the effort is spent on dealing with the spew coming from X through clever filtering and null routing, rather than trying to figure out who is controlling X. Good grief, is it really that difficult to put on an acl to log inbound tcp setup connections to the attacking host ?
"Proof" in a legal sense is probably impossible if its some kid in Kiev and highly cost prohibitive if its some kid in Boston and you are in New York. But you know what, the odds are it is from a western country and odds are its not some politically motivated attack, its some emboldened kid due to the anonymity of the Internet, pissed off that someone questioned his manhood on IRC and decides to take it out via some ego enlarging attack. In the cases we have dealt with where it was one of our customers, contacting the parents and explaining that what was being done was against the law, was enough to stop the kid from continuing. Even when the attacker was an adult, talking to the person, explaining its against our AUP and against the law was, in our cases, enough to stop the person. Its amazing how compliant and timid darksith2999@hushmail.com becomes when you talk to Joe-Brown@we-know-where-you-live

Are all these incidents bored teenage kids ? No. But I would put money on it the majority are. Really, how many of the very clever hackers you know are involved in DDoS attacks ?

You can't because I and 7 other
hackers all are fighting eachother over ownership of the poor UW student
schlep's computer...

Great, so of the 7 inbound streams, what effort is it to identify the IP address ? In our case
ipfw add 20 count log tcp from any to x.x.x.x setup

will it always work ? no. But it will catch more attackers than clever routing and filtering, as that just copes with the issue and does nothing to deal with it.

The problem isn't the network, nor the filtering/lack-of-filtering, its a
basic end host security problem.

I would say all have some responsibility. Its not just an end user problem, its not just a network operator problem. I would say a DDoS would violate everyone's AUP on this list no ? If you choose to not enforce your AUP, how are you not responsible ? This is like the cops saying, "people are going to drive drunk and do stoooopid things. We cant stop them from doing this, so we give up"

Until that is resolved, the ability of
attackers to own boxes in remote locations and use them for malfeasance
will continue to haunt us. I would guess that the other owners of the
machines attacking Mike (assuming they got the emails he sent...

I sent email to the listed abuse contacts first. If that bounced (as it did with several korean networks) I contacted the AS, or RADB contacts. I even contacted the APNIC registrar to inform them that all contacts bounced for one of the Korean ISPs. I then asked a Korean friend to look around the website for a "real person" and emailed that address. But the majority of the infected hosts were (surprise, surprise) in the largest networks e.g. AT&T, TW, Comcast, colo providers, and other resi broadband providers in Japan, Korea and Canada. Not because they have the lion's hare of dumb users, but because they have the lion's share of users period. Almost all had auto-responders saying "if spam, email here, if network abuse, email here"... If it was a different address, I then re-sent the complaints to the address instructed.

big
assumption) probably said: "Great another person getting attacked from
that joker's win2k machine, hurray:(" and moved on about thier business.

We dont do this. If a customer host is infected with virus/worm or is used in an attack, we contact the customer. If they dont do anything or choose to ignore us, we cut them off.

I'm all for raising the bar on attackers and having end networks implement
proper source filtering, but even with that 1000 nt machines pinging 2
packet per second is still enough to destroy a T1 customer, and likely
with 1500 byte packets a T3 customer as well. You can't stop this without
addressing the host security problem...

And kids will continue to attack / cause problems with impunity when there are no consequences for their actions. If network operators would enforce their AUPs, I think we would go a long way to reduce these types of headaches. This starts with putting *some* effort into identifying the controlling source.

         ---Mike

11

Hi, NANOGers.

Ooooo, you just knew I'd have to chime in eventually. :slight_smile:

] 1) The OS/software/default settings for a lot of internet connected
] machines are weak, making it easy to attack from multiple locations.

Yep, quite true. Vulnerable hosts are a commodity, not a scarce
resource. There are 728958 entries in my hacked device database
since 01 JAN 2003 that attest to this fact.

] 2) A lot of networks have no customer or egress filtering and make it a
] lot more difficult to trace DDoS traffic because it generally uses faked
] source addresses.

I've tracked 1787 DDoS attacks since 01 JAN 2003. Of that number,
only 32 used spoofed sources. I rarely see spoofed attacks now.
When a miscreant has 140415 bots (the largest botnet I've seen
this year), spoofing the source really isn't a requirement. :expressionless:

Filtering the bogons does help, and everyone should perform
anti-spoofing in the appropriate places. It isn't, however, a
silver bullet.

Thanks,
Rob.

12

] Sure, trace my attacks to the linux box at UW, I didn't spoof the flood
] and you can prove I did the attacking how? You can't because I and 7 other
] hackers all are fighting eachother over ownership of the poor UW student
] schlep's computer...

Only seven? Must be a lame box. :slight_smile:

13

it was at UW and that damned computer security guy, old Mr.
What's-His-Name-Dietrich was watching :slight_smile:

14

Filtering the bogons does help, and everyone should perform anti-spoofing
in the appropriate places. It isn't, however, a silver bullet.

it's necessary but not sufficient. but if we knew the source addresses were
authentic, then some pressure on the RIRs to make address block holders
reachable would yield entirely new echelons of accountability.

with the current anonymity of ddos sources, it's not possible to file a class
action lawsuit against suppliers of the equipment, or software, or services
which make highly damaging ddos's a fact of life for millions of potential
class members.

so please focus on "anti-spoofing"'s *necessity* and not on the fact that by
itself it won't be sufficient. "anti-spoofing" will enable solutions which
are completely beyond consideration at this time.

(we'll know the tide has turned when BCP38 certifications for ISPs are
available from the equivilent of "big 8" ("big 2" now?) accounting firms,
and these certifications will be prerequisite to getting BGP set up.)

15

I agree with Pauls’ position on anti-spoofing, without that, you are fighting A
losing battle.

Henry R Linneweh

16

Filtering the bogons does help, and everyone should perform anti-spoofing
in the appropriate places. It isn't, however, a silver bullet.

it's necessary but not sufficient.

anti-spoofing is useful, but vastly insufficient, and hence not necessary

randy

17

1) The OS/software/default settings for a lot of internet connected
machines are weak, making it easy to attack from multiple locations.

I�ll start looking for this to happen when Microsoft manages to release
an OS version which does not contain remote exploitable flaw before
the boxes hit the store self.

Remember, security is not a process, it�s lifestyle.

Pete

18

> 1) The OS/software/default settings for a lot of internet connected
> machines are weak, making it easy to attack from multiple locations.
>
I�ll start looking for this to happen when Microsoft manages to release
an OS version which does not contain remote exploitable flaw before
the boxes hit the store self.

lots of late night pondering tonight.

the anti-nat anti-firewall pure-end-to-end crowd has always argued in
favour of "every host for itself" but in a world with a hundred million
unmanaged but reprogrammable devices is that really practical?

if *all* dsl and cablemodem plants firewalled inbound SYN packets and/or
only permitted inbound UDP in direct response to prior valid outbound UDP,
would rob really have seen a ~140Khost botnet this year?

19

Paul Vixie wrote:

lots of late night pondering tonight.

the anti-nat anti-firewall pure-end-to-end crowd has always argued in
favour of "every host for itself" but in a world with a hundred million
unmanaged but reprogrammable devices is that really practical?

The most popular applications today either prefer or require bidirectional
connectivity. Peer2peer traffic is about half of total and there can be only
so many "corporate sponsored" SuperNodes .

Also, games and some other applications, like SIP and other VoIP stuff
require to be able to connect to the remote host. Obviously you can engineer
around all this but then, fixing the host is also "just software".

if *all* dsl and cablemodem plants firewalled inbound SYN packets and/or
only permitted inbound UDP in direct response to prior valid outbound UDP,
would rob really have seen a ~140Khost botnet this year?

Sure. One late remote exploit requires just a embedded MIDI file on a web
page which MS's browser will be happy to download and "execute". Or did you
think that the NAT box would allow only text based browsing and provide
HTTP to Gopher translation?

While you are at it, make sure all email-clients are safe and immune to viruses.

Pete

20

Sure, trace my attacks to the linux box at UW, I didn't spoof the flood
and you can prove I did the attacking how? You can't because I and 7 other
hackers all are fighting eachother over ownership of the poor UW student
schlep's computer...

You're quite right. This only means we'll be able to:

1) Stop the attack more quickly.

2) Alert the admins of the box that it's owned so that they can fix it and
begin tracing how it happened.

I'm all for raising the bar on attackers and having end networks implement
proper source filtering, but even with that 1000 nt machines pinging 2
packet per second is still enough to destroy a T1 customer, and likely
with 1500 byte packets a T3 customer as well. You can't stop this without
addressing the host security problem...

Agreed, we all (network providers, router vendors, software vendors and
end users) need to be working together to solve this problem. There is no
magic bullet.

Rich