Before you skim past this email because you already read the Prolexic
report on it or some other article on the internet, there are 2
disturbing properties that I haven't found anywhere else online.
1) After sending abuse emails to many networks, we received many angry
replies that they monitored their traffic for days without seeing
anything (even as we were being attacked) and that their IPs were
spoofed and would block us for spamming them.
What we discovered was that their firewalls/routers/gateways coming
from vendors like Cisco and SonicWall apparently didn't record SNMP
traffic going in or out of themselves. We confirmed this multiple
times by running a query to an IP that was claimed to be clean and
watching the response come 10-60 seconds later because the device was
being so heavily abused.
2) SNMP reflection offers the largest amplification factor by far,
even surpassing DNS, Chargen, or NTP by a wide margin. I have tested a
68 byte query and received responses of up to 30,000 to 60,000 bytes.
The trick is to use GetBulkRequest to start enumerating from the first
OID and setting max repetitions to a large number. This is contrary to
the other articles online which suggest a much smaller amplification
factor with other queries.
This protocol is also prevalent in many devices ranging from routers
to printers.
To solve this problem you should block SNMP traffic coming from
outside your network and whitelist outside IPs that require it.
This looks like more a security issue with the devices, not border security
issues.
If you're seeing replies of that size, it means the devices themselves are
set up to allow public queries of their information (not secured by even
keys), which no one should be comfortable with. People should never be
leaving the public access snmp strings on devices even if they are
internal. Edge blocking just masks the real issue.
-Blake
The problem isn't the people on this list leaving the public snmp
community on their devices, it's the vendors of home routers leaving it
there in their devices. Normal end users don't know or even care what snmp
is. (nor can we expect them too)
A simple scan of a large cable/dsl ISP's address space will likely net you
tens of thousands of devices which respond to the "public" snmp community.
Thomas
Hi,
The problem isn't the people on this list leaving the public snmp
community on their devices, it's the vendors of home routers leaving it
there in their devices. Normal end users don't know or even care what snmp
is. (nor can we expect them too)
A simple scan of a large cable/dsl ISP's address space will likely net you
tens of thousands of devices which respond to the "public" snmp community.
I can confirm this.
we did some enumeration (and discussed the said amplification attack) here:
at the time once you scanned "typical broadband segments" of major European carriers, pretty much every address responding to a ping had SNMP "public" also.
we gave the talk several times and demoed the amplification attack (with a slightly modified version of this tool: https://www.ernw.de/download/snmpattack.pl) against some of our systems, abusing $SOME_RANDOM_SEGMENT as amplifiers (we asked to stop [camera] recording in those cases where the talks were recorded) and it worked pretty much all the time (~20:1 ratio, initiated from the respective conferences' hotel wifi).
thanks
Enno
Agreed, but progressively breaking every service on the internet at the
edge because you think there might possibly be an issue just leads to bad
places.
Get better defaults sure, but don't slowly turn the internet into a cable
distribution system because "they're just users". It's bad enough already,
don't make it worse trying to solve every issue with the nuclear option
before trying anything else.
-Blake
These attacks are known as SAD attacks ;( ... yes very SAD ;(
SNMP Amplification DDoS
Kindest Regards
James Braunegg
P: 1300 769 972 | M: 0488 997 207 | D: (03) 9751 7616
E: james.braunegg@micron21.com | ABN: 12 109 977 666
W: www.micron21.com/tv-hosting T: @micron21
This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.
A relevant paper was released by the BITAG, see http://www.bitag.org/report-snmp-ddos-attacks.php Section 7 includes recommendations.
See also this blog post I wrote one day short of a year ago that may be of interest: http://corporate.comcast.com/comcast-voices/taking-steps-to-prevent-unintentional-network-abuse
A remaining issue out there for the community is taking action to reduce spoofing. A related project is the Open Resolver Project at http://openresolverproject.org/.
- Jason
Public SNMP being exploited for 8000x amplification is a very serious
issue. It is
arguably worse than open email relays.
Not only does it expose critical information from your users
but it offers the largest possible amplified DDoS by far, likely
bigger than DNS when you take into account the amplification size and
ubiquity. It will also cause your user's device to lag.
The most disturbing part is the lack of logging. We have tried
reporting these exploited servers for many weeks and because of the
logging problem most of them never get shut down because they just
assume they were being spoofed. We even had replies threatening to
block us because they thought were because they couldn't see they
were sending anything. When we were reported chargen attacks we
had much more positive responses.
Maybe you could refine the block by denying SNMP requests with the
public string. As network operators some compromises must be
made for a problem of this magnitude instead of just saying that you
should only be the best dumb pipe you can be.
We have seen attacks large enough to disturb 10G uplinks so as network
administrators you should not ignore this issue because you think it
is a small problem affecting only end users. This will affect you once
more people figure out how to get 8000x amplification from it.
It is great news that Comcast is trying to proactively solve this
problem on their network and hope that more networks would follow
their example.
Write into your TOS a block for SNMP. Deal with the whiners on a case by case basis. Problem solved.
Would it be possible to add SNMP to your (collective cable labs buddies) shapers and it would be taken care of prior to it leaving your network but after the cmts?
Flow telemetry can be of use in this instance.
I bet blocking all SYN packets and non related flow UDP packets to
customers would be even more effective. Why don't we do that and be done
with it instead of playing whack a mole every 3 months when someone finds
some new service that was poorly designed so that it can be used to send a
flood?
Yes, I'm being sarcastic above.
This is hardly the first finger of death amplification attack, and I
strongly doubt it'll be the last. Years ago it was smurf, then Quake
servers, then DNS, then Battlefield boxes, etc etc. Now it seems to be snmp
and recently chargen, and tomorrow it'll be some peer 2 peer service, the
next day it'll be a voice app. It will never end, and breaking the internet
port by port doesn't do anything to make it better.
I've been the victim of week long DDoS attacks that took down our
upstreams, not to mention us, and I still maintain the above.
It works better to fix the design issues than to play whack a mole by
blocking every imaginable service to your customers that responds to the
public with data larger than a FIN. Like getting their providers to more
proactively police their spew, manufactures to stop making negligent
devices, or implementing more intelligent filter communication so the only
option doesn't begin with calling your provider and asking them over the
phone to block X ip for you since you're off the internet.
Maybe even look into liability laws for allowing said attacks to originate
from your customers and not doing anything about it, or being manufacturer
of said devices that harm others through their lack of due diligence
implementing proper security. It's still way more effective than trying to
fix the *last instance* of the problem, instead of it's reasons for
enduring as an issue at a global scale.
-Blake
This vulnerability has been present ever since SNMP v2 was announced
back in 1993.
There is a reason why the biggest attacks these days are from
protocols that are decades old like DNS and Chargen.
People making widely spread protocols these days are aware of the
problem and are usually able to get their software to auto-update.
Enforcing source IP filtering seems like a lost cause. Not much
progress has been made that I can tell and it is difficult to discover
if the network allows it without being inside it.
I don't see many uses for unsecured SNMP access so this is not like
blocking all syn and udp packets.
The first time I became a pariah on NANOG was for postulating precisely that view--that if the originators of problems would stop, we would not have to figure out how to clean up after them. But I am past that now and out of work.
But it does occur to me for the first time that I can recall, that maybe the tremendous efforts to Get Control Of The Intertubes could be suckered into doing some good, say be establishing a certification authority to test and certify the safety of designs (is Scott B??? still around) and devise a way to not permit traffic from uncertified devices or configurations.
But after years of research I will tell you that there is no way to stop an avalanche once it has been released at the source.
http://youtu.be/60loeoblu0M
Anyone can make a device and connect it to the internet.
Because it breaks applications that people are paying to be able to use.
The way I see it; more and more samples keep getting found about
protocols abused because networks have not implemented BCP38.
The latest SNMP trend is just another uptick to the sample size, and
proof that Closing off perfectly OK recursive DNS services is
totally inadequate and not a useful long-term fix to the problem of
DDoS or IP/UDP reflection attacks.
Asking folks to improve the security of access to their SNMP instances
is just chasing the latest exploit implementation, with no attention
to the vulnerability or the root cause....
I realize the root cause is security-oblivious designers and one level
below that, lack of BCP38.
But realistically those 2 problems are not going to be solved any time
in the next decade. I have tested 7 large hosting networks only one of
them had BCP38.
To my knowledge it is practically impossible for someone outside a
multi-homed network to know if they allow spoofing which means it will
be difficult to punish. It also cost time and money to maintain these
ACLs, much more than blocking the occasional wide-spread protocol with
8000x amplification every couple of years.
I am here to talk about solutions today. BCP38 has been repeated to
death and people aren't going to start doing it because I said so. The
fact that the amplification factor is so high means that you need to
ensure near 100% conformity even if everyone started to become BCP38
compliant today.
I realize the root cause is security-oblivious designers and one level
below that, lack of BCP38.
But realistically those 2 problems are not going to be solved any time
in the next decade. I have tested 7 large hosting networks only one of
them had BCP38.
Then name and shame.
To my knowledge it is practically impossible for someone outside a
multi-homed network to know if they allow spoofing which means it will
be difficult to punish. It also cost time and money to maintain these
ACLs, much more than blocking the occasional wide-spread protocol with
8000x amplification every couple of years.
Just define a EDNS option which contains the real ip address and
update who-am-i services to look for that when replying, use a
modified DNS cookies or similar to prevent this being used as
amplifier. You can do this with a experimental EDNS option code
points today.
Send a non-spoofed request then send a spoofed request using the
cookie learnt from the first request.
Standardise it and every nameserver on the planet and be used to
detect if spoofing is possible. Using DNS puts up to costs for
ISP's that want to block people checking.
The blackhats already check whether spoofing is possible so there
is little point in blocking whitehats checking.
As for time and money to maintain these acls it really shouldn't
now that SIDR is becoming a reality. Present a properly signed
CERT to the other providers and automatically generate the acls
based on those. There is a little setup cost to get the system
running and almost no additional recurrent costs to keep it up
to date. The CERTs need to be generated for sBGP anyway.
I wonder if it's truly that unrealistic. If we target access networks, it
seems impractical target.
We have about 40k origin only ASNs and about 7k ASNs which offer transit,
who could arguably trivially ACL those 40k peers.
If we truly tried, as a community to make deploying these ACLs easy and
actively reach out those 7k ASNs and offer help, would it be unrealistic to
have ACL deployed to sufficiently large portion of networks to make
spoofing impractical/expensive?
Do we have other approaches? Can we make this ACL dynamic to a degree? Can
we extract ACL information from BGP table?
If origin only ASN advertises prefix to global table anywhere, allow it at
matching 'remote-as' port. Does not look like difficult feature to build,
does not require magic HW support, essentially dynamically built ACL.
After this spoof would require injected trash BGP route, which would also
steal return traffic, making it useless for DoS.
The following is a sorted list from worst to best of networks that allow spoofing: (cutoff here is 25k)
(full list - http://openresolverproject.org/full-spoofer-asn-list-201307.txt )
Count ASN#