Krebs on Security booted off Akamai network after DDoS attack proves pricey

> > Didn't realize Akamai kicked out or disabled customers
> >

Krebs on Security booted off Akamai network after DDoS attack proves pricey | ZDNET

> >
> > "Security blog Krebs on Security has been taken offline by host

Akamai

> > Technologies following a DDoS attack which reached 665 Gbps in

size."

>
>
> So ultimately the DDoS was successful, just in a different way.
>
> ~Seth
>
>
More technical information about the characteristics of these attacks

would be

very interesting such as the ultimate sources of the attack traffic
(compromised home pc's?), the nature of the traffic (dns / ssdp
amplification?), whether it was spoofed source (BCP38-adverse), and

whether

the recent takedown the vDOS was really complete or if it's likely

someone

else gained control of the C&C servers that controlled it's assets?

At least for the OVH case there is a bit of info:

https://twitter.com/olesovhcom/status/779297257199964160

"This botnet with 145607 cameras/dvr (1-30Mbps per IP) is able to send

1.5Tbps DDoS. Type: tcp/ack, tcp/ack+psh, tcp/syn."

Krebs said it was mostly GRE. Pulling from the archive.org copy of his post[1]:

"Preliminary analysis of the attack traffic suggests that perhaps the biggest chunk of the attack came in the form of traffic designed to look like it was generic routing encapsulation (GRE) data packets..."

This bothered me, though:

"McKeay explained that the source of GRE traffic can’t be spoofed or faked the same way DDoS attackers can spoof DNS traffic."

Please tell me why I can't spoof source IPs on a stateless protocol like GRE. If he specifically meant you can't spoof a source, hit a reflector, and gain amplification, sure, but I see zero reason why GRE can't have spoofed source IPs. It bothered me sufficiently that I wrote up some spit-balling ideas about reflecting GRE using double encapsulation[2]. Very rough and untested, but apparently I got a bee in my bonnet...

my guess is the GRE traffic was harder to filter because many providers use GRE to deliver ‘clean’ traffic back to origin sites.

- Jared

Please tell me why I can't spoof source IPs on a stateless protocol like GRE. If he specifically meant you can't spoof a source, hit a reflector, and gain amplification, sure, but I see zero reason why GRE can't have spoofed source IPs. It bothered me sufficiently that I wrote up some spit-balling ideas about reflecting GRE using double encapsulation[2]. Very rough and untested, but apparently I got a bee in my bonnet...

my guess is the GRE traffic was harder to filter because many providers use GRE to deliver ‘clean’ traffic back to origin sites.

But those tunnels generally wouldn't terminate on addresses within the protected block. If I'm protecting 192.0.22.0/24 for you, things would get confused if my GRE tunnel dest is somewhere in 192.0.22.0/24 because I would have a route for 192.0.22.0/24 across the GRE tunnel (either static or more conventionally learned via BGP). I'd bank it on either a provider touchdown or another unprotected block, otherwise I'd have a recursive routing issue where the tunnel destination would get routed through the GRE tunnel. I've been there with bouncy-bouncy tunnels on service turn-up when that was flubbed.

Alternatively, I could pin /32s for the tunnel destinations and still learn the shorter protected block, but even so I should then know which source (my) and dest (customer's) IPs to exclude explicitly from the GRE filtering.

If the attackers were hitting the GRE tunnel destination and spoofing the tunnel source that would make things harder, but that's starting to get into rather intimate knowledge of the scrubber's and customer's setup. I could still probably filter on e.g. TTLs or drop GRE further up to the northern edge on input rather than output, but agreed that is starting to get trickier...

A similar GRE attack was used against the Olympics:

"Once the Olympics got under way, LizardStresser along with a few other botnets ramped up their attack against organizations affiliated with the Olympics. The DDoS campaign launched attack traffic using the lesser-known IP protocol Generic Routing Encapsulation (GRE).”

-mel

Not at all. I refered to AUP's as a way people remove you from a service
when you use more of it then you are paying for.

My experiences are that under duress most people make poor choices and don’t properly filter these types of traffic.

How many times have you turned off a filter to debug something? Making a tunnel work is trickier than it seems and not all devices can terminate them.

In Cisco IOS land, you also have to have an Ip address on the tunnel for it to handle IP traffic, even if it’s “ip unnumbered”.

My guess is someone terminates on their P2P link to carrier, and that is easy enough to find w/ traceroute/mtr.

- Jared

Anycast + load balancers + high powered varnish?

notionally (because I have been paying zero attention to this) jon's
suggesting:
  1) setup a crapload of nginx/squid/etc configured tightly for things to
be accessed behind them
  2) ecmp to them across several layers (assume 32 ecmp at each layer, call
it 4 layers get craploads of machines running)
  3) change over the dns
  4) profit--

eh? If you can eat the PPS, you can spray across enough tcp listeners, you
can weed out the chaff and start filtering in the 'application'... perhaps
also run a 'low bandwidth' version of the target site...

hey look, we invented prolexic.

Well...by anycast, I meant BGP anycast, spreading the "target" geographically to a dozen or more well connected/peered origins. At that point, your ~600G DDoS might only be around 50G per site, and at that level, filtering the obvious crap gets much more reasonable. Then, doing the layer 7 scrubbing of the less obvious crap is more easily dealt with than a single site receiving 600G of attack traffic.

I haven't actually done this (specifically for DDoS mitigation)...just speculating as to how it might easily be done given sufficient resources. The trouble is, the attackers have virtually unlimited bandwidth, and aren't constrained by having to pay for the bandwidth.

Is CloudFlare able to filter Layer 7 these days? I was under the

impression CloudFlare was not able to do that.

There have been a lot of rumors about this attack. Some say reflection,
others say Layer 7, others say .. other stuff. If it is Layer 7, how are
you going to ÿÿstep in front of the cannonÿÿ? Would you just pass
through
all the traffic?

Anycast + load balancers + high powered varnish?

notionally (because I have been paying zero attention to this) jon's

suggesting:
1) setup a crapload of nginx/squid/etc configured tightly for things to
be accessed behind them
2) ecmp to them across several layers (assume 32 ecmp at each layer, call
it 4 layers get craploads of machines running)
3) change over the dns
4) profit--

eh? If you can eat the PPS, you can spray across enough tcp listeners, you
can weed out the chaff and start filtering in the 'application'... perhaps
also run a 'low bandwidth' version of the target site...

hey look, we invented prolexic.

Well...by anycast, I meant BGP anycast, spreading the "target"
geographically to a dozen or more well connected/peered origins. At that
point, your ~600G DDoS might only be around

anycast and tcp? the heck you say! :slight_smile:

50G per site, and at that level, filtering the obvious crap gets much more
reasonable. Then, doing the layer 7 scrubbing of the less obvious crap is
more easily dealt with than a single site receiving 600G of attack traffic.

sure, yes.

I haven't actually done this (specifically for DDoS mitigation)...just
speculating as to how it might easily be done given sufficient resources.
The trouble is, the attackers have virtually unlimited bandwidth, and
aren't constrained by having to pay for the bandwidth.

sounds like you got it all sorted out...

Well...by anycast, I meant BGP anycast, spreading the "target"
geographically to a dozen or more well connected/peered origins. At that
point, your ~600G DDoS might only be around

anycast and tcp? the heck you say! :slight_smile:

People who've tried it say it works fine. Routes don't flap that often.

It’s worked fine for 28 years, for me.

                                -Bill

boy, it'd sure be nice if there were some 'science' and 'measurement'
behind such statements.
Didn't k-root do some anycast studies ~8-10 years back?

-chris
(note I'm totally a believer in anycast for tcp in the 'right'
circumstances, but often it feels like talking to climate-change-deniers
when proffering it as a solution)

* morrowc.lists@gmail.com (Christopher Morrow) [Sat 24 Sep 2016, 18:55 CEST]:

boy, it'd sure be nice if there were some 'science' and 'measurement' behind such statements.
Didn't k-root do some anycast studies ~8-10 years back?

Not k-root but CacheFly 2006: https://www.nanog.org/meetings/nanog37/presentations/matt.levine.pdf

  -- Niels.

that's not the one I was thinking of, this is:
  <https://www.nanog.org/meetings/nanog39/presentations/larson.pdf>

which references your presentation, nice! and is about J-root, not K-root,
but mentions Lorenzo's work on K-root studies... In anycase, both seem to
say that 'tcp anycast works fine' (inside some set of parameters).

thanks!
-chris

that's not the one I was thinking of, this is:
<https://www.nanog.org/meetings/nanog39/presentations/larson.pdf&gt;

which references your presentation, nice! and is about J-root, not K-root,
but mentions Lorenzo's work on K-root studies... In anycase, both seem to
say that 'tcp anycast works fine' (inside some set of parameters).

Right… and we’ve known this since about… ? 1996?

DNS Results for query A krebsonsecurity.comAnswer:krebsonsecurity.com 157 IN A 130.211.45.45

On Google now.

I recommend running this command (or similar):

rndc flushname krebsonsecurity.com

if you still see 127.0.0.1

- Jared

And of course on windows ipconfig /flushdns

Still I had to wait for my corporate caching servers to update; I think the
TTL on the old A record was an hour.

DNS Results for query A krebsonsecurity.comAnswer:krebsonsecurity.com 157
IN A 130.211.45.45

On Google now.

Next question.

Will google use the information from the telemetry, rumored to be webcams,
to defang the bot ? Like post an informative message that the source
network is hosting a bad actor (same nat ipv4, /25, ... ). Or , work with
the access networks (Comcast, rcs/rds, china telecom) to disconnect and
manage the abusers ?