In general, i think UDP is doomed to be blocked and rate limited --
tragedy of the commons. But, it would be nice if folks would just fix
the root of the issue so the rest of us don't have go there...
While I'm behind some of the inventory projects (so you can go ahead and fix.. let me know
if you need/want the URLs to see data for your networks)...
I must provide credit to those behind the "Amplification Hell" talk at NDSS. If you
are at all interested in what is going on, you should attend or review the content.
BCP-38 on your customers is going to be critical to prevent the abuse reaching your
network. Please ask your vendors for it, and ask for your providers to filter your
network to prevent you originating this abuse.
If you operate hosted VMs, servers, etc.. please make sure those netblocks are
secured as well.
You can easily check your network (As can the bad guys!) here:
For those of you paying attention to the outage list, this is a pretty
big deal that has had daily ramification for some very big networks The Outages February 2014 Archive by date
In general, i think UDP is doomed to be blocked and rate limited --
tragedy of the commons. But, it would be nice if folks would just fix
the root of the issue so the rest of us don't have go there...
UDP won't be blocked. There are some vendors that have their own hidden protocol inside UDP packets to control and communicate with their devices.
Thinking on it again, maybe blocking UDP isn't all that bad. Would force the vendors to not 'hide' their protocol.
Be careful what you wish for. I know some people have just blocked all NTP to keep their servers from participating in attacks. This is common in places where they hand off a VM/host to a customer and no longer have access despite it being in their environment.
I would actually like to ask for those folks to un-block NTP so there is proper data on the number of hosts for those researching this. The right thing to do is reconfigure them. I've seen a good trend line in NTP servers being fixed, and hope we will see more of that in the next few weeks.
I've seen maybe 100-200 per-ASN reports handed out to network operators. If you want yours, please e-mail ntp-scan@puck.nether.net to obtain it. Put your ASN in the subject line and/or body.
- Jared (and others like Patrick that presented on the projects behalf).
Depending on your OS, the fixes can be quite simple or
interesting.
On my FreeBSD servers, simply updating with "freebsd-update"
was enough to fix the issue (in addition to limiting
who/what can access the service).
On Cisco devices, the ACL's you can attach to the NTP
process are quite effective.
On Juniper devices, it is less intuitive, and even though
NTP is enabled only as a client, it, sadly, runs the server
as well. A firewall filter helps here when applied
correctly.
The right thing to do is for people to not permit services to operate
on hosts they do not intend to operate on and not to be visible to
those they do not intend to use them. In other words, to properly
manage their networks. If that means blocking all access to
potentially faulty implementations, then that's the right thing to do.
In short, companies should do what is right for their companies and
nevermind anyone else.
Never forget that researches are just part of the "public" and should
never consider that their usage of the internet is any more or less
valid to the average third party than the next guy.
Taken to the logical extreme, the "right thing" to do is to deny any
spoofed traffic from abusing these services altogether. NTP is not the
only one; there is also SNMP, DNS, etc.
- - ferg
- --
Paul Ferguson
VP Threat Intelligence, IID
PGP Public Key ID: 0x54DC85B2
UDP won't be blocked. There are some vendors that have their own hidden protocol inside UDP packets to control and communicate with their devices.
Thinking on it again, maybe blocking UDP isn't all that bad. Would force the vendors to not 'hide' their protocol.
Be careful what you wish for. I know some people have just blocked all NTP to keep their servers from participating in attacks. This is common in places where they hand off a VM/host to a customer and no longer have access despite it being in their environment.
I was being a bit extreme, I don't expect UDP to be blocked and there are valid uses for NTP and it needs to pass. Can you imagine the trading servers not having access to NTP?
The knee jerk reaction to just block NTP is a temporary measure that can be used while other mitigation steps are implemented.
I kinda hijacked the NTP issue a bit and expanded it to cover the undocumented uses of device control in UDP. I'll leave that issue for another day, just wanted to raise awareness if it was not already known.
Taken to the logical extreme, the "right thing" to do is to deny any
spoofed traffic from abusing these services altogether. NTP is not the
only one; there is also SNMP, DNS, etc.
...and then we're back to "implement BCP38 already!" (like one of
the authors of the document didn't think of that, ferg?
NB: Some Entities believe all filtering is 'bcp 38' and thus have
given this stone-dead logical and sane practice a bad rap. If
someone is sloppy with their IRR-based filters or can't drive loose
RPF correctly, that isn't the fault of BCP38.
The document specifically speaks to aggregation points, most clearly
in the introduction:
"In other words, if an ISP is aggregating routing announcements
for multiple downstream networks, strict traffic filtering should
be used to prohibit traffic which claims to have originated from
outside of these aggregated announcements."
This goes for access, hosting, and most recently virtual hosting
in teh cloude. Stop forgery at your edges and your life will be
easier.
Indeed -- I'm not in the business of bit-shipping these days, so I
can't endorse or advocate any particular method of blocking spoofed IP
packets in your gear.
I can, however, say with confidence that it is still a good idea.
Great idea, even.
- - ferg
- --
Paul Ferguson
VP Threat Intelligence, IID
PGP Public Key ID: 0x54DC85B2
Indeed -- I'm not in the business of bit-shipping these days, so I
can't endorse or advocate any particular method of blocking spoofed IP
packets in your gear.
If you're dead-end, a basic ACL that permits ONLY your prefixes on
egress, and blocks your prefixes on ingress, is perhaps the safest bet.
Strict uRPF has it's complications, and loose uRPF is almost too
forgiving. If you're providing transit, it gets much more complicated
much more quickly, but the same principles apply (they just get to be a
less-than-100% solution)
I can, however, say with confidence that it is still a good idea.
Great idea, even.
