Large DDoS, small extortion

Mailman List Mirror (Read Only)
1

Apologies for the non-personal email address, but I don't want to give
our attacker any additional information than I need to.

I'd be happy to send personal contact/ASN information to any nanog
admins or regular members of nanog if it's useful.

Over the past year or so, we (a decent sized tier 2 with a nationwide
US backbone) have had several large DDoS attacks from what appear to
be the same person who is (we presume) going down something like the
alexa list of top sites, attacking them,
and asking for small amounts of money to stop.

This has been going on for a long time -- almost every detail is
exactly the same as what is described here:

http://it.slashdot.org/story/12/11/03/1846252/ask-slashdot-how-to-deal-with-a-ddos-attack

and more recently:

http://techcrunch.com/2014/03/03/meetup-suffering-significant-ddos-attack-taking-it-offline-for-days/

and:

https://gist.github.com/dhh/9741477

And I believe attacks including vimeo, github, and others.

The attacker is smarter than many random attackers, or at least has
better tools. He watches when you mitigate the attack, and shifts his
attack to something new. He (or his tools) also watch DNS for the
thing he's attacking and the attack moves as DNS changes.

We've seen UDP amplification (NTP and DNS mainly), syn flood, syn/ack
flood, layer 7 cache busting
(https://isc.sans.edu/forums/diary/Wordpress+Pingback+DDoS+Attacks/17801/),
and others we haven't been able to fully mitigate/identify.

The largest we've seen (which isn't the largest we've read about)
attacks are over 50Gbit and 10s of millions of pps.

He is in regular communication (via whois info and other collected
contact data) asking for <$1000 USD sums to stop the attacks.

While we are interested in technical means to mitigate the attacks
(the syn and syn/acks are brutal, all cores pegged on multicore 10G
nic servers just dealing with interrupts), what I'd really like to
find out is how to help fix the problem.

We've tried to engage upstream providers to help trace the attacks,
but have gotten nowhere (they didn't seem to understand that the syn
attacks were spoofed, and looking at source IPs didn't matter, we
wanted to know the ingress points on their network.)

What are the best practices for this? Are there secret code words
(http://xkcd.com/806/) we can use to get to someone at our upstreams
who might know what we're talking about? Is it worth the time?

Is it worth talking to law enforcement? Some of these have been >500k
costs to the customer, but we assume the person doing it isn't in any
western country, so maybe it doesn't even matter?

Thanks.

2

Apologies for the non-personal email address, but I don't want to give
our attacker any additional information than I need to.

I'd be happy to send personal contact/ASN information to any nanog
admins or regular members of nanog if it's useful.

We've tried to engage upstream providers to help trace the attacks,
but have gotten nowhere (they didn't seem to understand that the syn
attacks were spoofed, and looking at source IPs didn't matter, we
wanted to know the ingress points on their network.)

this sounds like a tooling issue on their part. they should be able to pick a specific set of items and trace them back and mitigate some set of spoofed packets. Some attackers are advanced and will detect when you block their spoofed packets immediately (they have telemetry/data like we all do) and move to another attack vector.

What are the best practices for this? Are there secret code words
(http://xkcd.com/806/) we can use to get to someone at our upstreams
who might know what we're talking about? Is it worth the time?

You need to talk to the security team in their NOC. These are usually small and sometimes difficult to reach. I know our NOC can find them quickly and works with them on customer issues often.

Is it worth talking to law enforcement?

Absolutely. Even if the "lost costs" have been just payroll which already exist, this may be related to other activity. I suggest calling your local FBI office (assuming you are in the US). They can be quite helpful. If you don't get somewhere quickly, let me know and I can try to hunt someone in a local field office for you.

Some of these have been >500k
costs to the customer, but we assume the person doing it isn't in any
western country, so maybe it doesn't even matter?

I'll say it does matter, because even if they are in some "unreachable" location, these folks sometimes travel to locations where they can be picked up. It may not be immediate, but can help build the case.

It is sad, but I can likely guess who your upstreams are, and some are more responsive than others. I'm aware of one that puts almost no effort into tracking spoofed packets to clamp down on them.

- Jared

3

Here's how to get started:

<http://mailman.nanog.org/pipermail/nanog/2010-January/016747.html>

Ensure you have flow telemetry enabled at all your edges; there are open-source tools like nfsen/nfdump that you can get started with quickly.

4

This has been going on for a long time -- almost every detail is
exactly the same as what is described here:
http://techcrunch.com/2014/03/03/meetup-suffering-significant-ddos-attack-
taking-it-offline-for-days/

He is in regular communication (via whois info and other collected
contact data) asking for <$1000 USD sums to stop the attacks.

That article said that the company didn¹t want to negotiate with
criminals. As an aside I spent some time with a retired hostage negotiator
on Tuesday (which was fascinating BTW). He actually said negotiation is
always useful and sometimes paying a ransom demand can serve as a method
to track where the money goes, to identify all the actors involved for
later action (which may apply in this case). And sometimes financial
demands are dropped as a result of negotiation.

Is it worth talking to law enforcement? Some of these have been >500k
costs to the customer, but we assume the person doing it isn't in any
western country, so maybe it doesn't even matter?

You may find the law enforcement more interested in engaging within you
than you might think.

Jason

5

Bad advice for online stuff, as a) it's very, very rare that the perpetrators are caught, and b) word will get around that you're an easy mark - so, more attacks, more (and more expensive) extortion.

*Never* pay extortion money to a DDoSer.

6

Never pay extortion anyways. After you pay once, you¹ll pay again.

If I were you, I¹d pay someone a few bucks to pull this kids dox and drop
them on pastebin. Stainless Steel Testicles turn to itty bitty testicles
when your name and phone number are sitting on the internet. Or just write
back to him and tell him that if you don¹t stop being packeted, you¹re
going to take his mother to a nice seafood dinner and NEVER call her
again. What¹s he going to do, packet you? :wink:

7

World could use more opts like this.

8

negotiation is fine… a weakness is presuming to know what the perp wants (and many times they don;t know themselves)
so engagement is good “The Cuckoo's Egg” is worth the read…

/bill

9

I could attribute a fair number of misdeeds to that book. :wink:

10

Contact law enforcement -- they can combine intel from multiple cases to
hopefully identify the attacker.

Automate your analysis and reporting. If you send an email to the sources
of abuse you can reduce the attacker's capabilities. (To set expectations:
only about 10% will take action.)

If you have specific customers that are being targeted, you may want to
suggest they get behind a DDoS mitigation provider that can absorb large
attacks (up to 500Gbps).

Damian

11

You know what would be nice? Some real life experience and results,
case studies.

I see the "common sense" and "logic" to a lot of these suggestions but
that and $1.75 plus tax will get you a venti coffee of the day at
Starbucks.

Victim: I'd be very wary of these suggestions unless there's some
good, solid reason to believe they're based on reality not just "I've
simulated all of human psychology in my head and here's what I think
you should do..."

I think it's interesting that the guy asks for such small amounts,
under US$1000.

Maybe that's a lot of money for him.

Maybe he thinks it won't be worth investigating such a small amount.

Maybe he thinks it's not a very big crime so if he gets caught he's
more likely to walk.

Maybe he thinks he's poor/broke and this money is deservedly his to
demand, it's such a modest demand.

  Note: He could be factually/legally wrong but that's why I prefaced
  with "maybe he thinks..."

Maybe he's a sadist and gets a kick out of making you squirm and the
money is just his way of keeping score, making you do something
tangible, kind of like "kiss my boots!"

Maybe he's insane which voids all of the above.

Maybe it's some sort of penetration exercise by terrorists, a govt,
etc.

Maybe all I've said and $1.75 plus tax...

12

I will use this opportunity to solicit real world experience and use cases that
could be discussed at the Security Track at NANOG 61. While I've been
soliciting talks in operational security specific groups, this thread also
peaked my interest.

Nothing beats sharing the good, the bad, the ugly and how collectively we
can improve on how we mitigate against varying attacks.

Please respond to me in unicast and let me know if you'd be willing to share
some experiences. The Security Track is not recorded nor streamed and
you do not need a formal presentation.

- merike

13

Some of us have quite a bit of real-life experience and results in these situations.

14

Most of us wish we didn't. There are so much more productive ways to
spend the day than fighting a determined and adaptive attacker.

-Blake

15

Concur 100%.

16

Thanks everyone. There's been a lot of great on and off list
responses, and we have a much better list of contacts for the next
time this happens.

We are in contact with the FBI now (very impressed, particularly
compared to what I expected), and have access to resources that we
didn't know existed.

Hopefully I'll meet some of you in bellevue next week.

17

Thanks everyone. There's been a lot of great on and off list
responses, and we have a much better list of contacts for the next
time this happens.

We are in contact with the FBI now (very impressed, particularly
compared to what I expected), and have access to resources that we
didn't know existed.

Hopefully I'll meet some of you in bellevue next week.

seeing as the web version of the security track is content free, it
would be cool if you held a little open chat.

randy

18

I replied back privately, as the specific details
I had to share weren't really best aired in an
archived, public forum. I suspect others might
be in a similar position, unfortunately.
I can see why the decision to not webcast
the security forum is a good one.

Matt

19

Sure, of course, many of us have. But how is $VICTIM supposed to
distinguish the wheat from the chaff without reference to specific
cases and results?

Some reasonable-sounding suggestions could be counter-productive or
even downright dangerous (depending on the nature of the attacker.) Or
a waste of time.

20

Sure. Every circumstance is different. But there is *one* universal rule

Never pay.

Never, under any circumstances, pay. Not even if you've persuaded the Men from U.N.C.L.E. to help you, and they suggest you pay because they think they can trace the money, do not pay.

Why not?

Because, irrespective of what happens with this one attacker, you will be swarmed by countless others. Attackers brag when they're paid; they'll exaggerate how much they received, and then you have a much bigger problem.

So, yes - one's own experiences and what one did and how one did it and why one did it and how it turned out are very valuable to share.

But never, under any circumstances, for any reason, no matter who advises you to do so, should you pay.