syn flood attacks from NL-based netblocks

Mailman List Mirror (Read Only)
1

Greetings,

I'm seeing slow-motion (a few per second, per IP/port pair) syn flood attacks ostensibly originating from 3 NL-based IP blocks: 88.208.0.0/18 , 5.11.80.0/21, and 78.140.128.0/18 ("ostensibly" because ... syn flood, and BCP 38 not yet fully adopted).

Why is this syn flood different from all other syn floods? Well ...

1. Rate seems too slow to do any actual damage (is anybody really bothered by a few bad SYN packets per second per service, at this point?); but

2. IPs/port combinations with actual open services are being targeted (I'm seeing ports 22, 443, and 53, just at a glance, to specific IPs with those services running), implying somebody checked for open services first;

3. I'm seeing this in at least 2 locations, to addresses in different, completely unrelated ASes, implying it may be pretty widespread.

Is anybody else seeing the same thing? Any thoughts on what's going on? Or should I just be ignoring this and getting on with the weekend?

Jim

2

We are seeing that here also. Saw similar traffic ostensibly originating from NL at least as long ago as last Sunday August 17.

3
  1. Rate seems too slow to do any actual damage (is anybody really
    bothered by a few bad SYN packets per second per service, at this
    point?); but

Common technique used by port scanners to evade detection as a DoS attack by fw/ids/etc.

  1. IPs/port combinations with actual open services are being targeted
    (I’m seeing ports 22, 443, and 53, just at a glance, to specific IPs
    with those services running), implying somebody checked for open
    services first;

Or they’re just checking if certain common ports are open with the intention of later trying known exploits against those which are reachable in order to attempt to compromise the hosts. Build the DB of reachable hosts/ports now, come back with exploits later.

  1. I’m seeing this in at least 2 locations, to addresses in different,
    completely unrelated ASes, implying it may be pretty widespread.

Sounds like a relatively common pattern though.

Is anybody else seeing the same thing? Any thoughts on what’s going on?
Or should I just be ignoring this and getting on with the weekend?

I wouldn’t worry too much about it unless you have reason to believe some of the likely-forthcoming exploits may actually work. Of course, if that’s the case, you should fix them anyhow.

Have a good weekend!

4

I would think Shodan/Zmap/pick your multi-IP-block-scanning-tool would portray similar behavior.

Echoing Matt’s “probably shouldn’t worry” sentiment, this could just be someone running an incantation of such tools for research or recreational purposes.

Best,
Jared

5

The traffic “from” 88.208.0.0/18, 5.11.80.0/21, and 78.140.128.0/18 doesn’t match the packet signatures for Masscan, ZMap, or any other well-known scanner. The traffic is likely spoofed.

6

Have been seeing these at $DAYJOB off and on for the past week.
First logged events began for on 2019-08-04, at approx 1500hrs PST.

Impact for us has been negligible, but some older ASA's were having trouble with the scan volume and their configured log levels which has since been remedied.

7

Thanks for the various responses. The pattern I (and apparently quite a few others) are seeing differs from an ordinary probe in that it is repeated a few times per second (if somebody wants to know who has a visible ssh server on port 22, and what version of sshd is running, they don't have to hit it multiple times per second). It differs from a SYN flood DoS attack in that its rate is too low to be effective. And it differs from both a port probe and a SYN flood attack (or somebody "learning how to use nmap") in that it is targeting a broad set of destinations in parallel; if source addresses are forged, they are from a fairly narrow set of source IPs.

The atypical pattern seems noteworthy in itself. Not a crisis, but not quite routine, either.

Jim

8

Seen a similar pattern a few years ago. Discovered it’s a couple of students basically developing mass scanning software for a bachelor’s degree who forgot to turn the running code off production before the summer break.

That’s the white noise of the Internet. Unless it’s hitting you multiple thousand times/s as opposed to multiple times/s, it’s only a matter of unpaid curiosity to start figuring out the reason. I guess Amazon or microsoft dot com have quite a museum of that staff.

9

This appears to be a TCP amplification attack. Similar to UDP amplification (DNS, NTP, etc) you can get some amplification by sending a SYN packet with a spoofed source, and watching your victims receive multiple SYN-ACK retries. It’s a fairly weak form of attack (as the amplification factor is small), but if the victim’s gear is vulnerable to high packet rates it may be effective.

The victim (or law enforcement) could identify the true source of the attack by asking transit providers to check their netflow to see where it enters their networks.

Damian

10

Hmm, I doubt this is the output of TCP amplification since Jim reported it as SYN spoofing, i.e., SYN packets, not SYN-ACK packets (as for typical TCP amplification). Unless the given hosts respond with multiple SYN-ACKs in which case these may be experiments by an attacker to measure if these IP:ports could be abused as TCP amplifiers.

But, at least if these IP:ports do not amplify, I suspect it’s more likely that, as Töma wrote, this is simply result of some learning-experiment by students (or by wanna-be hackers).

Yes, such (unethical, unprofessional) experimentation happens, and can be easily confused with some clever new attack… which is a pity since we always like to learn of new attacks!

Of course one (i.e., Jim) could try to trace back and find the source (likely spoofer) but it seems quite likely to be some students or script kiddies, which may be underwhelming…

11

Hmm, I doubt this is the output of TCP amplification since Jim reported it as SYN spoofing, i.e., SYN packets, not SYN-ACK packets (as for typical TCP amplification). Unless the given hosts respond with multiple SYN-ACKs in which case these may be experiments by an attacker to measure if these IP:ports could be abused as TCP amplifiers.

Clarifying for those unfamiliar with this attack:

  • Attacker is sending SYN packets spoofed “from” NL to Jim (and others)
  • Jim (and others) have applications listening on those ports and respond with SYN-ACK packets to the victim in NL
  • When the victim (NL) fails to complete the handshake (which they didn’t initiate!) Jim (and others) sends another SYN-ACK

So they’re not probing to see if Jim (and others) are abusable as TCP amplifiers… they’ve already determined they can be abused and are using those machines to conduct an actual attack against victims in NL.

Damian

12

Damian, sure, that’s what I meant - it’s possible, but only if Jim’s machines actually respond with multiple SYN-ACK packets. Which I think Jim probably would have noticed. Or maybe not ?

btw, some TCP amplifications can be quite severe, if anyone wants I can send the citation to a nice paper exploring this issue.

BR…

13

    I'm seeing slow-motion (a few per second, per IP/port pair) syn flood
    attacks ostensibly originating from 3 NL-based IP blocks:
    88.208.0.0/18 <http://88.208.0.0/18&gt;
    , 5.11.80.0/21 <http://5.11.80.0/21&gt;, and 78.140.128.0/18
    <http://78.140.128.0/18&gt; ("ostensibly" because ... syn flood,
    and BCP 38 not yet fully adopted).

    Is anybody else seeing the same thing? Any thoughts on what's
    going on?
    Or should I just be ignoring this and getting on with the weekend?

This appears to be a TCP amplification attack. Similar to UDP amplification (DNS, NTP, etc) you can get some amplification by sending a SYN packet with a spoofed source, and watching your victims receive multiple SYN-ACK retries. It's a fairly weak form of attack (as the amplification factor is small), but if the victim's gear is vulnerable to high packet rates it may be effective.

That thought crossed my mind, but it seems to me that the weak amplification factor, plus the broadly distributed set of forged source addresses (within the blocks cited above), would make the attack ineffective -- the whole point of DDoS being to focus a broadly distributed set of (illegitimately obtained) source resources on a narrow set of destination targets. Attacking 2 /18 blocks plus a /21 block in parallel with a weak-amplification attack doesn't look like a successful DDoS strategy to me.

Jim

14

I am seeing this against my recursive revolvers - one syn in, and many
syn-ack's back with no connection obviously. Low rate to be sure, but
what was surprising to me was that my revolvers (PowerDNS) definitely
have an allowed-networks ACL which specifies my permitted client
addresses, and I thought this would effectively filter any inbound
queries. But it looks like this ACL is applied only AFTER connection
setup. Maybe I have been blind this entire time thinking I wouldn't
respond to any packets sent to my resolver from non-allowed-networks
addresses. But anyways just a good data-point for anyone else to check
your revolvers too and consider the TCP case may behave as mine do. I
fixed it by implementing a revised iptables firewall which definitely
corrects the issue and drops outright all packets to
non-allowed-networks addresses, thank you ipset...

Mike-

15

The number of TCP syn-ack amplifiers is large. It may suffice to allow clogging a provider or IX, using low load per amplifier, as described. Such low load is likely to be undetected by most operators, and even when detected (e.g. by Jim), only few (e.g. Mike) will have sufficient motivation to block it - esp. considering that there blocking it would often be non-trivial, in Mike’s case, the amplifiers were DNS servers and sounds like he simply blocked packets to unallowed networks (good practice for DNS anyway - although I wonder why not block the incoming requests instead). Notice that one annoying aspect of these attacks is that tcp congestion control isn’t relevant.

The current packets could be part of a research experiment about this threat, or the instrumentation part of preparing such attack. I would not rule out research, since it isn’t trivial to know if the attack can be really viable to clog a provider or IX; in fact finding this out in an ethical way appears a non-trivial challenge, I’ll give it some thought (ideas welcome). Also I wonder what would be good defenses against such attack. Of course one way would be to prevent spoofed-IP packets, but that goal has proved quite difficult…

16

I just shared this idea over on the powerdns list, but I do have an idea
that may be potentially a good mitigation strategy and for the exact
reason stated above; low load to individual end points may still, in
aggregate, overwhelm an IX or provider, so cutting off the SYN-ACK
traffic to those hosts which have not requested connections is good
internet hygiene...

My idea is to maintain a penaltybox for any client IP that initiated a
connection but did not complete, while also maintaining a whitelist of
'frequent fliers' who have previously completed their connections
successful. The penalty could simply be to drop traffic sourced from
those client ips that do not complete the handshake, for some
configurable timeout period. The whitelisting feature could give a pass
to good clients and allow these to bypass the penalty filtering, for a
longer timeout period (but of course, passing it along so other ACL's
can take effect). I'd say, perhaps, a 5 minute timeout would be
sufficient for a penalty, while 1 day or longer would be sufficient for
whitelisting. It would depend on your traffic of course, and definitely
you would want something efficient such as linux ipset as opposed to
individual iptables rules.

While looking around, I came across the SYNPROXY netfilter module.. it
appears to be very complete but missing the above functionality to avoid
responding to spoofed clients. I'm going to see about hacking up a proof
of concept. I'll post here if I come up with something to play with.

Mike-

17

The high packet rate (millions of packets/second) and broad targeting (several /24s, not just a handful of machines) make it clear this was an actual attack, not an experiment or reconnaissance. (There are also other clear signals, such as the timing of the attacks, source(s) originating the spoofed packets, etc.)

Most kernels will return 3-5 SYN-ACK packets for an incoming SYN, so it’s not particularly interesting for attackers or defenders. While a long-term mitigation could be to limit ourselves to a single SYN-ACK (via /proc/sys/net/ipv4/tcp_synack_retries) it’s somewhat pointless to worry about a small amplification factor – an attacker could attack directly… or use UDP to get a massive bandwidth (or even significant packet) amplification. A counter-argument presented in the paper “Hell of a Handshake: Abusing TCP for Reflective Amplification DDoS Attacks” is that several broken TCP stacks that will provide a much greater amplification factor.

I disagree that preventing spoofed packets is difficult – it’s trivial for transit providers to use their netflow to identify the true source(s) of spoofed attacks, and then apply filters on their problematic customers (who refuse to apply egress filters themselves). If transit providers can’t be bothered to be proactive about this, law enforcement can serve them with legal process to identify the problem customers, who might then be held responsible for the attacks they’re facilitating. I commented on this approach in my talk at NANOG 76.

Damian

18

Peace,

[..] I do have an idea
that may be potentially a good mitigation strategy and for the exact
reason stated above; low load to individual end points may still, in
aggregate, overwhelm an IX or provider, so cutting off the SYN-ACK
traffic to those hosts which have not requested connections is good
internet hygiene...

In theory, yes, but it's incredibly complicated to do that properly at scale.

My idea is to maintain a penaltybox for any client IP that initiated a
connection but did not complete, while also maintaining a whitelist of
'frequent fliers' who have previously completed their connections
successful.

Unless a connection is completed, you do not know if the source IP
address of your client is spoofed or not. (Under certain
circumstances you don't know it even then, but it is unlikely that you
would have to take such a possibiity into account).

Therefore, you should not populate anything in your RAM from such a source.

See also my short talk from RIPE 77 for more information:
- https://ripe77.ripe.net/presentations/154-ddoswww_ripe77_004.pdf
- Archives – RIPE 77

Also, odds are a whitelist won't help either.

While looking around, I came across the SYNPROXY netfilter module.

Not sure if it's still supported. I think I've read in LKML that it
was dropped since Linux 4.4. Anyhow, it's impossible to scale without
a complete rewrite.

19

Peace,

Most kernels will return 3-5 SYN-ACK packets for an incoming
SYN, so it's not particularly interesting for attackers or defenders.

Well, producing 1000 Gbps as opposed to 200 Gbps is still pretty
impressive, isn't it?

More on that later, b/c the point here aren't even jiggabits,

it's somewhat pointless to worry about a small amplification
factor -- an attacker could [..] use UDP to get a massive
bandwidth (or even significant packet) amplification.

Most of the resources hosted by a typical hosting company are
essentially Web sites.[citation needed]

Unless you are really really dependant on QUIC (and, unless we're all
really unlucky and recent initiatives to get rid of TCP/TLS fallback
in HTTP/3 would gain support), as a Web hosting company, you can use
whatever you want to get rid of UDP completely very quickly, and that
won't harm your business a lot.

Dealing with TCP flags is a different story:

- Your ability to handle them with the likes of RFC 5575 depend on
what particular sort of equipment is deployed in your network;

- To make matters worse, for a huge portion of customers the ability
to connect to an external service/API gateway/Web site via TCP is
crucial. A simple example is Google which cannot survive for long if
Googlebot keeps being unable to operate. Think also OAuth,
Skyscanner, credit scoring systems, insurance companies, etc.;

- To ensure proper handling of spoofed SYN/ACKs while still
maintaining a possibility to connect to an external service you, as a
hosting company under an attack, would have to track all of the
outgoing SYNs to match them against received SYN/ACKs later.

This is where the "1kGbps-vs-200Gbps" argument becomes important, b/c
every existing free connection state tracking solution doesn't scale
beyond 200 Gbps at best given the best hardware money can buy given a
single machine, and no existing solution is able to share its state
across multiple machines.

[there are proprietary products doing that though, we have one, but
proprietary solutions are always a different kind of story]

20

I agree these attacks can be large: the one under discussion probably exceeded 10Mpps (Gbps is the wrong metric for small-packet attacks)
I agree they can cause significant outages: this style of attack played a role in the Liberia outages in 2016
My main disagreement is whether small amplification factors are noteworthy. A factor of 2 is “rounding error” and we probably shouldn’t waste our time on it (eg, by designing solutions to reduce amplification factors) when we could instead be targeting the sources of spoofed traffic.

I was highlighting this as a DDoS (rather than a port scan) mainly to raise awareness. This is definitely an interesting form of attack, largely for the reasons you state (it’s subtle to detect and therefore harder to mitigate). But this particular “carpet-bombing” attack isn’t likely to be mitigated at the network layer anyway… the load is distributed across thousands of machines which can each trivially handle the state. It’s more a question of bandwidth to the provider… and if you’re targeting the provider’s bandwidth you’d do better to use traditional UDP amplification.

Damian