Several times this year our customers have suffered DDoS' ranging from 30
Mbps to over 1 Gbps, sometimes sustained, sometimes in a several minute
spurts. They are targeted at one IP address, and most times our netflow
tool identifies that a large percentage of the traffic is "port 0". The one
from today had about 89% port 0 and 11% port 53 (DNS). If it happens
repeatedly or continuously we just have our upstream provider blackhole the
target (victim) IP address.
Unfortunately I don't have packet captures of any of the attacks, so I can't
exam them for more detail, but wondering if there was some collective wisdom
about blocking port 0.
Yes - don't do it, or you will break the Internet. These are non-initial fragments.
You or your customers are on the receiving end of DNS reflection/amplification attacks, and the large unsolicited DNS responses being used to packet you/them are fragmented. Use S/RTBH, flowspec, IDMS, and/or coordination with your peers/upstreams to block these attacks when they occur.
Do *not* perform wholesale blocking of non-initial fragments (i.e., src/dst port 0), or you will have many unhappy customers and soon-to-be former customers.
Unfortunately I don't have packet captures of any of the attacks, so I
can't exam them for more detail, but wondering if there was some collective
wisdom about blocking port 0.
It should be relatively safe to drop (non-fragment) packets to/from port 0.
If I recall correctly, there are some routers that perform a "helpful"
numeric value validation when the human is entering port numbers for
access list rules, that _do_ forward port 0 traffic, and through
some sort of oversight by the router/firewall vendor actually
_prevent_ the administrator from selecting port 0 in a deny rule, eg.
"Port to deny must be a number from 1 to 65535".
TCP/UDP port 0 is technically a legal port, but it's also a reserved
port, and very unusual for it to be used on the network for any
legitimate purpose. Various firewalls will discard anything TCP/UDP
sent to/from port 0.
Many TCP/UDP sockets implementations won't even let an application
select port 0. bind() to port 0 is treated as a signal that the
application wants the sockets API to pick a high-numbered ephemeral
port.
Thanks for confirming what was discussed in the NANOG archive.
I now have warm fuzzies knowing that all my protections are reactive. =) I will be talking with our upstream provider to see if they can enable some better automation (because they run a larger shop). I know they were able to null route in seconds, we just need a faster way to identify targets.
Without a packet capture to look at, that's really just a blind assumption.
A port number of a non-initial fragment does not exist at all, because
the Layer 4 info is unavailable in that case, something _might_ lie
and say the port number is 0, but it should not -- there is no TCP
header with any port numbers, the only fields available to check
against on such packets are Layer 3 fields such as protocol, source,
destination address.
The port number of the Layer 4 connection cannot be determined without
executing IP fragment reassembly in that case. Routers normally
reassemble fragments they receive, if possible.
An access list statement attempting to match against non-present
Layer 4 information, should not work; on a stateful firewall, the
presence of the rule might trigger a fragment reassembly, on a
router, the non-applicable ACL entry referring to a non-existent
port number will generally be ignored.
A full capture should not be necessary.
You determine if a packet is a fragment by examining the MF flag, bit
50, and fragmentation offset of the IPv4 header; bits 51 through 63.
You only need to look at the first 8 bytes of the IP header. If
the MF bit is set to 0, and the fragmentation offset is also all bits
0, then the packet is not part of a fragment.
The packet is a non-initial fragment if and only if, the
fragmentation offset is not set to zero.
Port number's not a field you look at for that.
The port number of the Layer 4 connection cannot be determined without
executing IP fragment reassembly in that case. Routers normally
reassemble fragments they receive, if possible.
No, routers normally do *not* reassemble fragments. This is typically
done by hosts and firewalls.
NetFlow reports source/dest port 0 for non-initial fragments. That, coupled with the description of the attack, makes it a near-certainty that the observed attack was a DNS reflection/amplification attack.
Furthermore, most routers can't perform the type of filtering necessary to check deeply into the packet header in order to determine if a given packet is a well-formed non-initial fragment or not.
And finally, many router implementations interpret source/dest port 0 as - yes, you guessed it - non-initial fragments. Hence, it's not a good idea to filter on source/dest port 0.
Some UDP applications will use zero as a source port when they do not
expect a response, which is how many one-way UDP-based apps operate,
though not all. This behavior is spelled out in the IETF RFC 768:
"Source Port is an optional field, when meaningful, it indicates the
port of the sending process, and may be assumed to be the port to
which a reply should be addressed in the absence of any other
information. If not used, a value of zero is inserted."
That would only be applicable if the box was expecting to receive UDP
and not send a response. I'm not sure I can think of anything but
specialized, vertical applications that would have that behavior with
port zero (syslog and SNMP traps send without expecting a response,
but they don't use port zero in any implementation I've seen, and
neither is generally allowed to be received from the internet at
large).
In addition to the fragments, these packets might also be non-TCP/UDP
(ICMP, GRE, 6to4 and other IP-IP, etc). If the host doesn't expect to
receive large UDP packets, you can block UDP fragments. Note that
recursive DNS servers would need UDP fragments (well, if you want to
do large DNS packets - if you set the right options, you can turn that
off). But if you aren't generally providing UDP services, blocking
UDP packets, especially to stop an attack, wouldn't hurt (you can also
block anything with the MF bit set). If you block these fragments at
your provider's router, and it is a DNS amplification attack, you're
problems are probably solved until the hacker figures it out. Just
make sure you think of things like recursive DNS and other
applications that may be using UDP fragments.
Can netflow _properly_ "capture" whether a packet is a fragment or not?
No.
If not, does IPFIX address this?
Yes.
But this is all a distraction. We are now down in the weeds.
Your customers were victims of a DNS reflection/amplification attack. The issue of fragmentation is moot. The defense methodologies already discussed are how folks typically deal with these attacks. There isn't an ovearching network access policy list you can apply at your edges or ask your peers/upstreams to apply which will mask them - the optimal approach is to deal with them on a case-by-case basis.
Another nice "emerging" tool [I say emerging because it's been around forever but nobody implements it] to deal with this is Flowspec, using flowspec you can instruct your Upstream to block traffic with much more granular characteristics.
Instead of dropping all traffic to the IP address, you can drop (for example) udp dst 80 traffic to the IP address, or traffic from a particular source to a particular DST.
It can also be initiated by your side without interaction from the upstream ISP.
Actually *all* IPv6 node are supposed to support EDNS so *all* IPv6
hosts should be expecting to receive fragmented UDP for DNS. Add
to that all hosts that do DNSSEC validation in the stub resolver /
application. With DANE this will be any host with a web browser.
flowspec is essentially S/RTBH with layer-4 granularity (it can do some other things, as well). I certainly hope that vendors who've not yet implemented it will do so, it's a great tool, as you say.
Even customer-triggered S/RTBH is very useful, and some ISPs have implemented it for their customers.