Curious if anyone else thinks filtering out NTP packets above a certain
packet size is a good or terrible idea.
From my brief testing it seems 90 bytes for IPv4 and 110 bytes for IPv6 are
typical for a client to successfully synchronize to an NTP server.
If I query a server for it's list of peers (ntpq -np <ip>) I've seen
packets as large as 522 bytes in a single packet in response to a 54 byte
query. I'll admit I'm not 100% clear of the what is happening
protocol-wise when I perform this query. I see there are multiple packets
back forth between me and the server depending on the number of peers it
has?
Would I be breaking something important if I started to filter NTP packets
If your equipment supports this, and you're seeing reflected NTP attacks, then it is an effective stopgap to block nearly all of the inbound attack traffic to affected hosts. Some still comes through from NTP servers running on nonstandard ports, but not much.
Standard IPv4 NTP response packets are 76 bytes (plus any link-level headers), based on my testing. I have been internally filtering packets of other sizes against attack targets for some time now with no ill-effect.
You can filter packets that are 440 bytes in size and it will do a lot to help the problem, but make sure you conjoin these with protocol udp and port=123 rules to avoid collateral damage.
You may also want to look at filtering UDP/80 outright as well, as that is commonly used as an "I'm going to attack port 80" by attackers that don't quite understand the difference between UDP and TCP.
Next up, we will see the proto=0 and proto=255 attacks again..
Filtering will always break something. Filtering 'abusive' network traffic is intentionally difficult - you either just let it be, or you filter it along with the 'good' network traffic that it's pretending to be. How can you even tell it's NTP traffic - maybe by the port numbers? What if someone is running OpenVPN on those ports? What about IP options? Maybe some servers return extra data?
This is really not a network operator problem, it's an application problem if anything. While it makes sense to temporarily filter a large flood to keep the rest of your customers online, it's a very blunt instrument, as the affected customer is usually still taken offline - but I'm talking about specific targeted filters anyway. Doing blanket filtering based on packet sizes is sure to generate some really hard to debug failure cases that you didn't account for.
Unfortunately, as long as Facebook loads, most of the users are happy, and so these kinds of practices will likely be implemented in many places, with some people opting to completely filter NTP or UDP. Maybe it will buy you a little peace and quiet today, but tomorrow it's just going to be happening on a different port/protocol that you can't inspect deeply, and you don't dare block. I can imagine 10 years from now, where we're writing code that fragments replies into 100 byte packets to get past this, and everyone loses. Your filter is circumvented, the application performs slower, and the 'bad guys' found another way that you can't filter. When all that's left is TCP port 443, that's what all the 'abuse' traffic will be using too.
While filtering NTP packets may be a work-around, for any network with firewall isolation from the general Internet it would make more sense to:
1. Establish an internal peer group of NTP Server instances. As noted, a distributed group of four is the absolute minimum, six is more than sufficient.
2. Default restrict noquery on all internal NTP servers.
2. Use a common list of external NTP servers for all internal servers.
3. Provide that list of external NTP servers to the firewall engineer to add to a permit ACL (deny all others)
>> Curious if anyone else thinks filtering out NTP packets above a certain
>> packet size is a good or terrible idea.
>>
>> From my brief testing it seems 90 bytes for IPv4 and 110 bytes for IPv6
are
>> typical for a client to successfully synchronize to an NTP server.
>>
>> If I query a server for it's list of peers (ntpq -np <ip>) I've seen
>> packets as large as 522 bytes in a single packet in response to a 54
byte
>> query. I'll admit I'm not 100% clear of the what is happening
>> protocol-wise when I perform this query. I see there are multiple
packets
>> back forth between me and the server depending on the number of peers it
>> has?
>
> If your equipment supports this, and you're seeing reflected NTP
attacks, then it is an effective stopgap to block nearly all of the inbound
attack traffic to affected hosts. Some still comes through from NTP servers
running on nonstandard ports, but not much.
Also, don't forget to ask those sending the attack traffic to trace where
the spoofed packets ingressed their networks.
> Standard IPv4 NTP response packets are 76 bytes (plus any link-level
headers), based on my testing. I have been internally filtering packets of
other sizes against attack targets for some time now with no ill-effect.
You can filter packets that are 440 bytes in size and it will do a lot to
help the problem, but make sure you conjoin these with protocol udp and
port=123 rules to avoid collateral damage.
Preferably just source-port 123.
You may also want to look at filtering UDP/80 outright as well, as that is
commonly used as an "I'm going to attack port 80" by attackers that don't
quite understand the difference between UDP and TCP.
Please don't filter UDP/80. It's used by QUIC ( QUIC - Wikipedia).
We are filtering a range of packet sizes for UDP/123 at the edge and it
has definitely helped thwart some of the NTP attacks. I hate to do
blanket ACLs blocking traffic but multi-Gbps of attack traffic (not
counting the reflected traffic) is hard to ignore and it's worth the risk
of blocking a minute amount of legitimate traffic.
Filtering out packets this size from UDP/anything to UDP/123 allows time-sync requests and responses to work, but squelches both the level-6/-7 commands used to trigger amplification as well as amplified attack traffic.
Operators are using this size-based filtering to effect without breaking the world.
Be sure to pilot this first, and understand whether packet-size classification on your hardware of choice includes framing or not.
Also, note that this filtering should be utilized to mitigate attacks, not as a permanent policy.
In order to minimize overblocking of this nature, filtering of this nature should be used with the highest possible degree of granularity, and the minimal necessary scope. One way to accomplish this is to divert traffic towards destinations in question into a mitigation/center sinkhole, applying this filtering on the coreward interfaces of the mitigation center/sinkhole gateway (some re-injection mechanism such as GRE, VRF, selective filtering of the diversion route announcements coupled w/PBR, etc. must be used to re-inject non-matching traffic towards the destinations in question) or via other mitigation mechanisms.
In emergencies, the concept of partial service recovery may dictate temporary filtering of coarser granularity in order to preserve overall network availability; we've run into situations in the past week-and-a-half where networks were experiencing severe strain due to the sheer volume of ntp reflection/amplification attack traffic, and it was necessary to start out with more general filtering, then work towards more specific filtering once the network was stabilized.
But you raise a very important point which should be re-emphasized - general filtering of traffic is to be avoided whenever possible in order to avoid breaking applications/services.
However, the converse notion that emergency situations sometimes entail necessary restrictions should also be taken into account. Operators should use their best judgement as to the scope of any filtering, and should always pilot any proposed mitigation methodologies prior to wider deployment.
The folks at QUIC have been advised to not use UDP for a new protocol,
and they would be very well advised to not use UDP:80 since that is a
well known target port used in the DDoS reflection attacks.
As Jared noted, UDP:80 is a cesspool today. Attempting to use it for
legit traffic is not smart.
Please suggest which protocol has less blocking on the internet today
(keeping in mind the full end-to-end stack of CPE, various ISPs,
country-level proxies, backbone providers, etc).
> You may also want to look at filtering UDP/80 outright as well, as
that is
>> commonly used as an "I'm going to attack port 80" by attackers that
don't
>> quite understand the difference between UDP and TCP.
>
> Please don't filter UDP/80. It's used by QUIC (
> QUIC - Wikipedia).
The folks at QUIC have been advised to not use UDP for a new protocol,
and they would be very well advised to not use UDP:80 since that is a
well known target port used in the DDoS reflection attacks.
Please suggest which protocol has less blocking on the internet today
(keeping in mind the full end-to-end stack of CPE, various ISPs,
country-level proxies, backbone providers, etc).
Damian
Tcp.
But the actual answer is , if you want a new transport protocol, create a
new transport protocol with a new protocol number. Overloading the clearly
polluted UDP pool will have problems. Happy eyeballs negotiation may be
required for L4.
QUIC can do what it wants. Like anyone else, they pay their money and take
their chances. But, the data point that UDP is polluted is clearly
documented with several folks on this list suggesting tactical fixes that
involve limiting UDP, especially udp:80
If we want new L4 protocol which works today, we must first ride on top of
UDP, since that will work on lot more people day 1, this will avoid
chicken-egg problem (kit won't be fixed,as no one uses new L4, no one uses new
L4 as lot of kit drops it)
I'm surprised MinimaLT and QUIC have have not put transport area people in
high gear towards standardization of new PKI based L4 protocol, I think its
elegant solution to many practical reoccurring problem, solution which has
become practical only rather recently.
Oh, the transport area people *are* in their high gear.
Their frantic movements may just seem static to you as they operate on more drawn-out time scales.
(The last transport protocol I worked on became standards-track 16 years after I started working on it.)
At this IETF, there will be a “Transport Services” BOF to help find out what exactly the services are that a new transport protocol should provide to the applications. Research platforms such as QUIC, tcpcrypt, MINION etc. are very much in the focus of attention.
This time, it would be nice if the operations people got to have a say early on in what gets standardized.
(Just be careful not to try to “fight yesterday’s war”.)
yesterday's war = don't bring up that operators are having a real
problem with UDP, and that operators have and will continue to block
it? Because, i think that is what this thread is about.
i did bring yesterday's war to the IETF RTCWweb group and got the
expected answer