Why does abuse handling take so long ?

Mailman List Mirror (Read Only)
1

Dear nanog members,

As current maintainer of DroneBL I happen to receive a lot of unwanted
packets in the form of DDoS attacks, now the DDoS itself is not the real
problem, dealing with it the fast way is.

Now most of you would think: Just filter it, put a big firewall in front
of it, bla bla bla bla. But what I'm really talking about is the
ignorance most providers show when it comes to handling the abuse when
it gets reported.
The issue in there being, it's way too slow, and my hoster needs to
temporary nullroute my ip range in order to protect his network.
We both mail all the involved providers and sometimes need to wait days
before hostings act upon the mail.

In most cases the only thing the abuse@ contacts do as hoster, is relay
the mail to the client but do not dare to do anything themself, even if
you provide them with a shitload of logs, even if you call them and say
that the attack from their source is still continueing, they refuse to
look into it and shutdown the source. And that pisses me off badly.

Why o why are isp's and hosters so ignorant in dealing with such issues
and act like they do not care?

Kind regards,
Alexander Maassen
Maintainer DroneBL

2

they don't act like they do not care. they really *don't* care. no acting.

1) you're not a direct customer, why should they do anything? by doing nothing it cost them nothing.
2) why should they do anything to shut down paying customers? shutting down abusive customers is shutting off revenue sources.
3) lifting a finger is too much like work. it costs the money and gains them nothing.

the only way to correct this behavior is to make it more expensive for
providers to retain abusive customers than it is to keep them.

3

> Why o why are isp's and hosters so ignorant in dealing with such issues
> and act like they do not care?

they don't act like they do not care. they really *don't* care. no acting.

Well now, I'd say this varies considerably. There are definitely ISPs
that care and *do* work hard at reducing abuse. But even so - assuming
I'm an ISP that cares,

- You're presenting me with evidence of abuse. OK, I don't know you.
Why should I believe your evidence? At best I'm going to take it as a
*hint*.
- If I take your evidence as a hint, I'm going to want to correlate it
with my own logs. This takes time.
- I probably have customer contracts in place that specify under what
circumstances I can actually take the customer off net. My tolerance
of abuse may not be the same as your. Also, "due process" means that
these things take time.

Steinar Haug, Nethelp consulting, sthaug@nethelp.no

4

well, they should care. if a customer is compromised and ddosing, it
costs the provider money (additional traffic being pushed bringing your
95% closer to your commit levels or possibly causing an overage to be
incurred.)

by doing nothing it may wind up costing them something - even if they
can make the money back by passing the overage onto the customer, there
is a high likelyhood that the customer will just jump ship and not pay
the invoice and go elsewhere.

william

5

* Alexander Maassen:

In most cases the only thing the abuse@ contacts do as hoster, is relay
the mail to the client but do not dare to do anything themself, even if
you provide them with a shitload of logs, even if you call them and say
that the attack from their source is still continueing, they refuse to
look into it and shutdown the source. And that pisses me off badly.

There is a relatively nice way of putting this.

If you can't contact the customer and don't know what they are doing,
it is difficult to estimate the risk from terminating the customer's
connectivity. Therefore, giving them some time to react---4 business
hours or perhaps even a business day---seems reasonable, and this can
be a very long time span for many types of network abuse, especially
when time zones are taken into account.

Why o why are isp's and hosters so ignorant in dealing with such issues
and act like they do not care?

The less nice way is that many hosters attract customers who don't
care if they are compromised. These customers do not perceive abuse
notifications as valuable, so the hoster gains nothing from forwarding
them: the abuse won't stop, and the customer is likely less happy than
before.

6

Is it time for another "notion of self-defense" in responding
to/retaliating against a DDoS attack of sufficient strength to hold down
a large network, or resource?

Andrew

7

The RIPE IRR database contains a systemic means for operators,
responsible for IP address blocks, to exchange PGP-signed messages
amongst each-other in relation to security incidents. It
unfortunately does not see much use: under 1% of allocations in RIPE's
database include any reference to one of only 235 "incident response
teams," which are conceptually similar to a POC.

Other things have been tried but haven't reached "critical mass" also,
such as dial-by-ASN VOIP connectivity.

The real problem with handling serious network abuse is it's pretty
hard to get through the "bozo filter" and actually reach anyone who
might understand your request or complaint (DDoS), let alone have the
power to act. The anti-spam folks have honestly made this problem
far, far worse, by slamming every role mailbox they can find for every
network operator, regardless of whether or not a specific mailbox for
email-related abuse exists or how good (or bad) a network may be at
keeping spam off its network. I hope this remark doesn't steer the
thread far off-topic, but I wish the anti-spam folks would realize how
counter-productive it is to intentionally send the same complaints to
a multitude of different abuse mailboxes.

For this reason, it really is necessary to have an automatic filtering
mechanism in place just to make sure the network abuse people don't
have to sift through messages which are mostly related to email abuse.

If operators would decide to use a system like IRT, supported in RIPE
IRR, then we would not only be able to filter out a lot of the B.S.,
we would also know that signed messages complaining of DDoS coming in
were actually from the security folks at the complaining organization,
people who have authority to make requests on behalf of the org that
"owns" related netblocks.

This pretty much eliminates the "why should I believe your evidence?"
argument, because we shouldn't have to believe anyone's evidence to at
least block traffic towards the netblocks they operate.

For example: if I am an end-user with address 192.0.2.80 and my web
site is being subject to DDoS which I believe is originating from
203.0.113.66, I would contact my ISP, who registers themselves as the
IRT for 192.0.2.0/24. My ISP would probably do a sanity check on my
claim, examine their netflow, etc. and then agree that 203.0.113.66 is
a source of the DDoS. They'd see that an IRT is registered for
203.0.113.0/24 and send over a PGP-signed message to the counter-party
IRT. That IRT would verify the PGP signature and association with the
target of the DoS, 192.0.2.80, and at that point, they would have
absolutely zero excuse for not immediately dropping all traffic from
203.0.113.66 towards me at 192.0.2.80.

It doesn't matter if there are any logs or "evidence," it matters that
the proven security/abuse contact for 192.0.2.0/24 requested that the
counter-party stop sending traffic to 192.0.2.0/24. Whether or not
the ISP for 203.0.113.66 decides to investigate any further is up to
them; maybe they log some traffic, find a compromised host, and shut
it down. Maybe they really don't care.

Now that you know people are capable of doing all that based on data
in RIPE's trusted IRR database, you may also realize that this process
could be streamlined to any point between "human reads email, checks
relationships, and configures network" all the way to "script reads
email, checks relationships, and configures network." Implementing
this could save NOCs time (if they really cared about outgoing DDoS
from their networks) and improve response to network abuse.

So ultimately, there is already a good framework in place to
substantially "fix" this problem. No one uses it. That is unlikely
to change until there is an economic incentive, such as a lawsuit by
someone targeted by DoS which can be proven to be originated from a
negligent network, causing calculable damages. Until some network has
to pay out a million bucks because they sat on their hands, I don't
see anything changing.

8

Why o why are isp's and hosters so ignorant in dealing with such issues
and act like they do not care?

they don't act like they do not care. they really *don't* care. no
acting.

1) you're not a direct customer, why should they do anything? by doing
nothing it cost them nothing.
2) why should they do anything to shut down paying customers? shutting
down abusive customers is shutting off revenue sources.
3) lifting a finger is too much like work. it costs the money and
gains them nothing.

the only way to correct this behavior is to make it more expensive for
providers to retain abusive customers than it is to keep them.

Is it time for another "notion of self-defense" in responding
to/retaliating against a DDoS attack of sufficient strength to hold down
a large network, or resource?

Because there just aren't enough internet vigilantes already...

9

Because network operators rarely get together and turn off routing to
abusive hosting. On the few occasions that has happened, it took years
of consensus building.

So, part of the problem is *your* upstream. Why didn't your upstream
actively remove the entire abusive netblock? Why didn't your upstream
contact other providers with your evidence, and together remove the
abusive network from the global routing tables?

As we get more experience with global "cyberwar", we're going to need
faster response mechanisms.

What will we do as some major government coordinates an attack on another?

What will we do as some major North American government coordinates an
attack on another region or facility?

10

Well now, I'd say this varies considerably. There are definitely ISPs
that care and*do* work hard at reducing abuse. But even so - assuming
I'm an ISP that cares,

- You're presenting me with evidence of abuse. OK, I don't know you.
Why should I believe your evidence? At best I'm going to take it as a
*hint*.
- If I take your evidence as a hint, I'm going to want to correlate it
with my own logs. This takes time.

This also applies in reverse when your asking to get out of a DNSbl. FWIW, when you deal with me on getting out of the AHBL, how well you handle my abuse report affects how well I handle your request to be delisted. :slight_smile:

effort in == effort out

- I probably have customer contracts in place that specify under what
circumstances I can actually take the customer off net. My tolerance
of abuse may not be the same as your. Also, "due process" means that
these things take time.

You aren't by chance related to Andrew Stevens? He's been going on recently about "due process" (quotes and all) to the point where certain newsgroups are flooded with socks.

If not, then you have my apology :slight_smile:

11

The problem does seem to persist. 10 years later and DDoS, it's
mitigation, and asleep at the switch abuse departments are still a problem.

12

In the case of a DoS, a call to the legal dept of the ISP might do the trick. One successful lawsuit against a provider for knowingly allowing their customers to DoS/DDoS would certainly change alot of attitudes about the value of an abuse desk.

13

* Jeff Wheeler:

14

Exactly.

Make this a question of economics and the problem will solve itself.

It has to become more expensive to ignore abuse than it is to deal with it.

Until that changes, the abuse will continue.

15

This is why my post highlights the underlying mechanism/system. It
can and should be used to streamline DDoS mitigation. It is
unfortunately not in practical use, since the cost of ignoring DoS
originating from one's network is generally low or zero.

16

My hoster did mail, his upstream is EGI, however, EGI does not want to
block/filter since it would pollute their routers they say.
I asked through my hoster if they would be willing to place a simple UDP
filter, blocking all of it. They refuse.

17

again make it a question of economics.

vote with your wallet, vote with your feet.

if they won't block, leave.

18

leaving is not always as easy as you imply. There are some areas with
only one real provider.

19

In a message written on Sun, Mar 13, 2011 at 12:45:04PM +0100, Alexander Maassen wrote:

Why o why are isp's and hosters so ignorant in dealing with such issues
and act like they do not care?

One of the things you have to remember is that ISP's get a ton of
reports, and most of them are of very low quality. Abuse queues
are full of people who sign up for a properly run mailing list and
then a year or two later mail abuse to get taken off saying its now
spam. Or folks who misconfigure their firewall / IDS and send in
reports of being DDOSed, by a nameserver, to which they are sending
queries and then flagging the responses as an "attack". There are
a lot of reports that don't include either the source or destination
IP, or leave out any time information.

Worst of all, there are the automated reports where someone has a
different opinion than the law, or even reality. They create systems
to basically DDOS abuse@, by reporting every case they can find
individually when in fact the "spammer" is doing things legally and
properly.

Of course it varies greatly ISP to ISP, depends on customer mix,
time of the day, time of the year and all sorts of other factors.
Still, there are times when I would say less than 1 in 50 e-mails
received to abuse@ is something that is a complete report and
actionable Keep that in mind, along with what others have pointed
out, that there is generally no "profit" in handling abuse.

Quite frankly, most ISP's aren't going to take your DDOS report
seriously via e-mail. If it's not bad enough to you that it is
worth your time and money to make a phone call and help them track
it down it is not worth their time and money to track it down and
make it stop.

In short, try picking up the phone. You'll bypass the entire e-mail
reporting cesspool I just described, and show the ISP you actually
care. 9 out of 10 times they will respond by showing they care as
well.

20

Op 14-3-2011 0:21, Leo Bicknell schreef:

Quite frankly, most ISP's aren't going to take your DDOS report
seriously via e-mail. If it's not bad enough to you that it is
worth your time and money to make a phone call and help them track
it down it is not worth their time and money to track it down and
make it stop.

In short, try picking up the phone. You'll bypass the entire e-mail
reporting cesspool I just described, and show the ISP you actually
care. 9 out of 10 times they will respond by showing they care as
well.

Quite frankly, been there, done that, got the t-shirt. And the answer I
get most of the time there is:
[loop]
- Sorry, email abuse and wait for a reply
- Sorry, I can't help you, wait for a reply on your abuse email
- Sorry, there is nothing I can do, my hands are bound, wait for a reply
from the abuse department
[/loop]

So much regarding the 9 out of 10. It's the 1 remaining that actually
cares and tries something.