DoS Attacks

Mailman List Mirror (Read Only)
1

Oh boy, what a fun night this was. After a 4 or so hours downtime, my
servers are back up and running.

Heres the gorey details.

At about 7pm EST, we began having unusual issues with our network, the
router, and several machines on the network. For the first part of the
attack, we were held down for a good 30-60 minutes. Took us a while to
figure out which one of our machines was being targeted. Turns out to be
our NAT firewall box. We tried several things to drop the attack, but it
still kept coming in strong (mind you, we don't have very much bandwidth,
but we can usually ride out DoS attacks pretty well - this was an exception)
Then suddenly, out of the blue it dropped. Outside connectivity was
restored and things were back to normal.

20 minutes later, the relentless attack began again. This time, we were
ready and waiting with tcpdump and several other handcrafted tools we use
for this type of thing. The attack was coming from a single source machine,
unspoofed (ballsy if you ask me), 128.186.11.215. Packets were UDP, random
from 2100-2299 source and 2400-2699 dest.

So, now for the fun part. Being offsite, I wasn't the one to place the
calls, but my admin on site started with FSU's abuse desk. No help
whatsoever. Claimed that because the abuse desk was gone, they had no
authority to deal with the problem. Frustrated, annoyed, and pissed off, he
tried again, and got hung up on twice. Nice people eh?

Our next call was a bit later (at this point, we were very unhappy and ready
to start raising hell with anyone we could find) - this time, to their
upstream Qwest. After dealing with the operator, they finally sent him to
one of the NOCs. Unfortunately, they sent him to the wrong NOC and not the
Qwest MD NOC. Luckily, we got someone with a clue - a nice guy by the name
of Richard Stein who tried to help us, but found that the other NOC was
unresponsive and couldn't do anything himself to solve the problem.

After hanging up with Qwest, we got a call back from FSU. After a good 20
minutes or so of talking with the net admin from FSU, things were finally
set in motion. After another good 10 minutes or so, connectivity was
restored and everything was back to normal. According to my guy, they
yanked the whole subnet at FSU. Problem solved.

So here I am, asking if anyone here has any advice on dealing with these
issues in the future? Its painfully apparent noone takes these situations
seriously enough. What should we do when we are put in a position like
this? Just sit back and hope it goes away itself?

Also, any ideas on how to deal with these attacks on lower bandwidth
connections? Right now, 2mbit.com / sosdg.org is sitting on a 1.5/256
business DSL line. I really can't afford to be buying T1s or T3s just to
hold up to attacks like this.

As always, thanks.

2

So here I am, asking if anyone here has any advice on dealing with these
issues in the future? Its painfully apparent noone takes these situations
seriously enough. What should we do when we are put in a position like
this? Just sit back and hope it goes away itself?

Also, any ideas on how to deal with these attacks on lower bandwidth
connections? Right now, 2mbit.com / sosdg.org is sitting on a 1.5/256
business DSL line. I really can't afford to be buying T1s or T3s just to
hold up to attacks like this.

As always, thanks.
--------------------------
Brian Bruns
The Summit Open Source Development Group

I think I would follow two avenues next time - the direct approach with FSU
(or wherever the traffic is coming from) as well as with your DSL provider.
Your upstream should be able to assist in at least keeping the traffic off
of your dedicated line.

Whether your DSL provider has the resources to sink the traffic may be
another matter -- but they are at least in a position to help you and
(since you are paying them) have an interest in dealing with you.

Mark Radabaugh
Amplex
(419) 720-3635

3

You were lucky to know where it was coming from.
What you depends on what you know.
You knew the sources are small and you knew where they were. You did the
right thing by contacting FSU, and then their upstream.
If either was unresponsive, they are being extremely neglegent.
I'm one of the few who still believes that immediate 24/7 response to
security problems should be a requirement for permanent internet
connectivity.

If you don't know the source, talking to your own upstream to see if
there is anything they can do is a good first move. Most decent places
will at least TRY and help you out.

But realise that sometimes you just need to ride it out.

4

I think I would follow two avenues next time - the direct approach with

FSU

(or wherever the traffic is coming from) as well as with your DSL

provider.

Your upstream should be able to assist in at least keeping the traffic off
of your dedicated line.

Whether your DSL provider has the resources to sink the traffic may be
another matter -- but they are at least in a position to help you and
(since you are paying them) have an interest in dealing with you.

I hate to say this, but Ameritech/SBC is utterly useless in matters like
this. I mean, at one point their redback was being nailed, and they didn't
seem to care one bit. After 5pm, everyone with a clue seems to leave, and
we are left with useless low level help desk techs.

Our DSL service isn't bad - in fact it rarely goes down. The problem is
that when we need their help with something out of our league, they are
completely useless. Anyone know of a contact number for SBC/Ameritech that
would be useful in a case like this?

5

Its generally a better idea to contact your own upstream provider first.
Your own upstream knows you, and is supposedly paid to help you. Your
upstream should have contacts with its BGP peers and eventually the
source.

A problem with calling random NOCs is they don't know you from someone
trying to social engineer something. So you end up delaying effective
response while they try to figure out who you are, and how your report
is related to them.

If you have a problem with your credit card, your own bank is in a
better position to help you than calling random other banks in the world
even if you did have their phone number. Other banks may care about
security, but they still don't know who you are.

If your own upstream won't help you, you may have no choice but to
beg other NOCs to help you.

6

First of all, have your tools ready so that whenever DoS pounds on you, you can
immediately activate them and they will give you an overview of the DoS attack
such as size of the attack, source/dest (random or one/two or spoofed?), et al.

Then you need to contact your upstream first to hve them deal with it, and yes
I understand, most SDSL providers do not like to cooperate.

Considering it takes me 1 hour of buerocracy to get an ACL put up during a DoS
to my current providers, getting an ACL activated by SDSL team is.....psh....
utterly hopeless unless you have people connections :frowning:

If you can't afford T1/T3 type of circuits where you can at least call up your
upstream (doesnt matter how long it takes them to put up the ACL, the point is,
will they?), then I hate to say... I don't think there is much you can do :frowning:

-hc

7

Can SLA's be used to cover this sort of thing. (starts to dig out his own contracts).

Surely you should be able to bounce it to your upstream provider who should deal with it for you??

Just a thought.

8

So, now for the fun part. Being offsite, I wasn't the one to place the
calls, but my admin on site started with FSU's abuse desk. No help
whatsoever. Claimed that because the abuse desk was gone, they had no
authority to deal with the problem. Frustrated, annoyed, and pissed off, he
tried again, and got hung up on twice. Nice people eh?

You were speaking with one of our 2nd shift operators. Unfortunately, our
2nd and 3rd operators are not hired on the basis of cluefulness, but for
their willingness to work these shifts and do so at minimal pay. They tend
to take things very literally, so despite the fact that we have network
admins on-call 7/24, unless it's reported as a "network problem" the
operators won't call.

This morning instructions have been issued (again) to treat security
incidents as "network problem" and act accordingly. Additionally, your
E-Mail has been hardcopied and posted in the operator's area, along with
some not-for-public-consumption comments of my own.

After hanging up with Qwest, we got a call back from FSU. After a good 20
minutes or so of talking with the net admin from FSU, things were finally
set in motion. After another good 10 minutes or so, connectivity was
restored and everything was back to normal. According to my guy, they
yanked the whole subnet at FSU. Problem solved.

The computer's MAC was blocked at the switch.

Our apologies for this incident. We will continue trying to improve the
responsiveness of our 2nd and 3rd shift operations staff.

If all else fails, call me at home: 850-385-4725

- SLS

9

Due to the efficiency of our upstream provider's abuse department,
opening efficiently at 8 am and closing just as efficiently at 5 pm (because we all know network abuse only occurs between 8 and 5), the ISP wasn't going to be of much help with an attack that started at 6:30pm localtime.

Andrew D Kirch
Security Admin - Summit Open Source Development Group
http://www.sosdg.org

10

Due to the efficiency of our upstream provider's abuse department,
opening efficiently at 8 am and closing just as efficiently at 5 pm

(because we all know network abuse only occurs between 8 and 5), the ISP
wasn't going to be of much help with an attack that started at 6:30pm
localtime.

Andrew D Kirch
Security Admin - Summit Open Source Development Group
http://www.sosdg.org

* A lot of Abuse Departments are going to close at around 5pm, because the
advanced staff that would know how to deal with such things goes home around
that time. But most of them should have an escalation procedure, which means
that there is someone(s) over the staff in the NOC or Call (Support) Center
which can be called if necessary. You should insist or demand the issue be
escalated if it warrants this. If they won't escalate it ask for the phone
number of the supervisor... If they escalate it and you don't get a call
back in 30-60 minutes ... call again (repeat until you get a call back). Be
prepared to justify why you have had your issue escalated complete with
emailable detailed information. Even if the guy that calls you back
(semi-technical manager type) doesn't have the proper clue, chances are he
has someone else on call that does.

If they don't have this escalation capability ... (IMHO) get a different
ISP.

11

<rant style="moaning and useless" context="naive">

I really don't give a ^H^H^H^H!H * !X *!X about what timeframe abuse departments operate. I just want more upstreams (or specifically my upstreams) to have a community that I can announce a /32 to null.

</rant>

-hc

12

Haesu wrote:

<rant style="moaning and useless" context="naive">

I really don't give a ^H^H^H^H!H * !X *!X about what timeframe abuse departments operate. I just want more upstreams (or specifically my upstreams) to have a community that I can announce a /32 to null.

</rant>

Seems like they ought to be made to offer a 1/3-the-price offering if
the connectivity is only offered for 1/3 the time.

No, really, I'll be alright, just me sit down here a minute.