UDP/123 policers & status

Steven Sommars said:

The secure time transfer of NTS was designed to avoid amplification attacks.

I work on NTP software (ntpsec). I have a couple of low cost cloud servers in
the pool where I can test things and collect data.

I see bursts of 10K to several million packets "from" the same IP Address at
1K to 10K packets per second. Ballpark of 100 events per day, depending on
the size cutoff. I saw one that lasted for most of a day at 1K packeets/sec.

All the packets I've seen have been vanilla NTP requests - no attempt at
amplification. I'm only checking a very small fraction of the garbage.

I haven't seen any pattern in the target IP Address. Reverse DNS names that
look like servers are rare. I see legitimate NTP requests from some of the
targets.

Would data be useful? If so, who, what, ... (poke me off list)

I don't see any good solution that a NTP server can implement. If I block
them all, the victim can't get time. If I let some fraction through, that
just reduces the size of the DDoS. I don't see a fraction that lets enough
through so the victim is likely to get a response to a legitimate request
without also getting a big chunk of garbage. I'm currently using a fraction
of 0. If the victim is using several servers, one server getting knocked out
shouldn't be a big deal. (The pool mode of ntpd should drop that system and
use DNS to get another.)

If NTS is used, it would be possible to include the clients IP Address in the
cookie and only respond to requests with cookies that were issued to the
client. That has privacy/tracking complications.

Hello,

I am one of the authors of the NTS for NTP specification,
<https://datatracker.ietf.org/doc/draft-ietf-ntp-using-nts-for-ntp/>.

Steven described this well, and as he wrote, the first step in the NTS
procedure is to contact a Key Establishment (KE) server, the KE server
will point to the NTP server and port to use, also taking into
consideration what the client requested, if it did.

The NTP packets will be larger than what they are today, since they
contain one or sometimes more than one “cookies” or “cookie placeholders”
(a measure to make amplification impossible).

Today, some points in the internet still filter port 123 on size.

If this continues, NTS enabled NTP server owners will likely not run
the corresponding NTP server on port 123, since there is no need to,
they can run it on an arbitrary port.

There seems to be no willingness from the ISP community to try to
clean up the old NTP traffic amplifiers that are still out there.

Is this really what the ISP community wants - to kill off port 123,
and force NTP to move to random ports?

Ragnar

Make NST attenuation vector, so that reply is guaranteed to be
significantly smaller than request, and by standard drop small
requests.

The NTP replies on port 123 are of the same size as the request, or
smaller on error.

If filtering on request/reply (or “mode” in NTP lingo), you could filter
out the control packets which have the amplification problems in very old
configurations.
You could allow request and reply, mode 3 and 4, but disallow control
packets, mode 6.
This kind of filtering may not be possible in all equipment though.

Another option is to rate limit the traffic, even though that is not
entirely without problems either - public servers are supposed to get
a lot more traffic than a typical client generates.

I know that ISP:s have been hunting down machine with other services
that could be used for e.g. amplification or spam, like SMTP relays,
DNS resolvers, HTTP proxies, and similar.
This would be fully possible also with these bad NTP configurations
that have not been updated in many many years.
I think only the ISP:s are in a position to both find out who they
are, and to force them to be fixed.

Ragnar

Answers to all of these questions are readily available via search engines, searching the archive of this and related listservs, searching the presentation archives of NANOG/RIPE/APRICOT/APNIC/AusNOG/NZNOG/LACNIC, et. al. My archive of public presentations is available here:

<https://app.box.com/s/4h2l6f4m8is6jnwk28cg>

These topics are well-understood and -documented, and a bit of research can help bring one up to speed on them pretty quickly.

but why isn’t BCP 38 widely deployed?

Because it costs time and money. People have been asking for it to be implemented for decades. It is never going to be deployed on every network.

What fraction of the
world does implement BCP 38?

Not enough. Everyone has to use it for it to work. Otherwise the hackers will still find a network that doesn’t have it.

I’d also be interested in general background info on DDoS. Who is DDoS-ing
whom and/or why? Is this gamers trying to get an advantage on a competitor?
Bad guys making a test run to see if the server can be used for a real run?

Most motivations for attacks can’t be traced. But this is not just a gaming problem. It is used to extort businesses for money, destroy competitors, shutdown government critics, fame.

Is DDoS software widely available on the dark web?

You don’t need the dark web. It is widely available on Github like most other attack types.

https://github.com/search?q=ntp+ddos

Broken protocols need to be removed and blacklisted at every edge. Pushing the responsibility to BCP38 is unrealistic.

    but why isn't BCP 38 widely deployed?

Because it costs time and money. People have been asking for it to be
implemented for decades. It is never going to be deployed on every network.

So you are claiming BCP 38 has to be all or nothing? That there is *no*
benefit to incremental deployment?

    What fraction of the
    world does implement BCP 38?

Not enough. Everyone has to use it for it to work. Otherwise the
hackers will still find a network that doesn't have it.

I disagree. Enough people have to use it for it to work. And as more
folks use it, there is increasing motivation for more folks to use it.
As the number of deployments increases, one can assume (perhaps
correctly) that it will become less expensive to deploy, and that
additional measures will be found to help accomplish the same thing.

    I'd also be interested in general background info on DDoS. Who is
    DDoS-ing
    whom and/or why? Is this gamers trying to get an advantage on a
    competitor?
    Bad guys making a test run to see if the server can be used for a
    real run?

Most motivations for attacks can't be traced. But this is not just a
gaming problem. It is used to extort businesses for money, destroy
competitors, shutdown government critics, fame.

     Is DDoS software widely available on the dark web?

You don't need the dark web. It is widely available on Github like most
other attack types.

Repository search results · GitHub

Broken protocols need to be removed and blacklisted at every edge.
Pushing the responsibility to BCP38 is unrealistic.

The monlist attack was mitigated many years' ago. The problem is that
too many folks don't upgrade their software.

...

Broken protocols need to be removed and blacklisted at every edge.

A protocol isn’t broken just because it can be abused when spoofed,
it is abused. Even TCP can be abused in that way.
Should we blacklist and remove TCP?

Pushing the responsibility to BCP38 is unrealistic.

It would help quite a bit against a lot if abuse, and it would be
reasonable to include it on a lowest level of technical level to
actually get to be called an ISP.

So what do the ISP:s want - earn money while doing nothing until the
Internet is unusable? I don’t get it.
There are enough threats against the open Internet as it is, we
don’t need that too.

Ragnar

Steven Sommars said:

The secure time transfer of NTS was designed to avoid

   amplification attacks.

Uh, no.

Yes, it was.

As Steven said, “The secure time transfer of NTS was designed to
avoid amplification attacks”. I would even say - to make it
impossible to use for amplification attacks.

If you understand what's going on from the perspective of both the
client and the server and think about the various cases, I think you'll
see what I mean.

Hopefully, no-one exposes mode 6 or mode 7 on the internet anymore
at least not unauthenticated, and at least not the commands that are
not safe from amplification attacks. Those just can not be allowed
to be used anonymously.

NTS is a task-specific hammer.

Yes.

Ragnar

Ragnar,

Steven Sommars said:

The secure time transfer of NTS was designed to avoid

   amplification attacks.

Uh, no.

Yes, it was.

As Steven said, “The secure time transfer of NTS was designed to
avoid amplification attacks”. I would even say - to make it
impossible to use for amplification attacks.

Please tell me how. I've been part of this specific topic since the
original NTS spec. For what y'all are saying to be true, there are some
underlying assumptions that would need to be in place, and they are
clearly not in place now and won't be until people update their
software, and even better, tweak their configs.

If you understand what's going on from the perspective of both the
client and the server and think about the various cases, I think you'll
see what I mean.

Hopefully, no-one exposes mode 6 or mode 7 on the internet anymore
at least not unauthenticated, and at least not the commands that are
not safe from amplification attacks. Those just can not be allowed
to be used anonymously.

But mode 6/7 is completely independent of NTS.

It's disingenuous for people to imply otherwise.

Ragnar,

Steven Sommars said:

The secure time transfer of NTS was designed to avoid

  amplification attacks.

Uh, no.

Yes, it was.

As Steven said, “The secure time transfer of NTS was designed to
avoid amplification attacks”. I would even say - to make it
impossible to use for amplification attacks.

Please tell me how. I've been part of this specific topic since the
original NTS spec. For what y'all are saying to be true, there are some
underlying assumptions that would need to be in place, and they are
clearly not in place now and won't be until people update their
software, and even better, tweak their configs.

The NTS protected NTP request is always of the same size, or in some
cases larger, than the NTS protected NTP response. It is carefully
designed to work that way.

Hence, [what Steven said].

If you understand what's going on from the perspective of both the
client and the server and think about the various cases, I think you'll
see what I mean.

Hopefully, no-one exposes mode 6 or mode 7 on the internet anymore
at least not unauthenticated, and at least not the commands that are
not safe from amplification attacks. Those just can not be allowed
to be used anonymously.

But mode 6/7 is completely independent of NTS.

Exactly. No one needs to, or should, expose mode6/7 at all. They were
designed at a time when the internet was thought to be nice place were
people behaved, decades ago, today they are just huge pains in the
rear. Sadly allowing anonymous mode 6/7 was left in there far to long
(admittedly being wise in hindsight is so much easier than in advance).
And here we are, with UDP port 123 still being abused by the bad
guys, and still being filtered by the networks.

It's disingenuous for people to imply otherwise.

I couldn’t say, I don’t even know of an example of someone who does.

Ragnar

Ragnar,

Ragnar,

Steven Sommars said:

The secure time transfer of NTS was designed to avoid

  amplification attacks.

Uh, no.

Yes, it was.

As Steven said, “The secure time transfer of NTS was designed to
avoid amplification attacks”. I would even say - to make it
impossible to use for amplification attacks.

Please tell me how. I've been part of this specific topic since the
original NTS spec. For what y'all are saying to be true, there are some
underlying assumptions that would need to be in place, and they are
clearly not in place now and won't be until people update their
software, and even better, tweak their configs.

The NTS protected NTP request is always of the same size, or in some
cases larger, than the NTS protected NTP response. It is carefully
designed to work that way.

So what?

The use of NTS is completely independent of whether or not a server will
respond to a packet.

And an unauthenticated NTP request that generates an unauthenticated
response is *always* smaller than an authenticated request, regardless
of whether or not the server responds.

Claiming that amplification is a significant issue in the case where
there's no amplification but the base packet size is bigger is ignoring
a key piece of information, and is disingenuous in my book.

Hence, [what Steven said].

If you understand what's going on from the perspective of both the
client and the server and think about the various cases, I think you'll
see what I mean.

Hopefully, no-one exposes mode 6 or mode 7 on the internet anymore
at least not unauthenticated, and at least not the commands that are
not safe from amplification attacks. Those just can not be allowed
to be used anonymously.

But mode 6/7 is completely independent of NTS.

Exactly. No one needs to, or should, expose mode6/7 at all. They were
designed at a time when the internet was thought to be nice place were
people behaved, decades ago, today they are just huge pains in the
rear. Sadly allowing anonymous mode 6/7 was left in there far to long
(admittedly being wise in hindsight is so much easier than in advance).
And here we are, with UDP port 123 still being abused by the bad
guys, and still being filtered by the networks.

Your statement about "exposing" is imprecise and bordering on incorrect,
at least in some cases.

But again, the use of mode 6/7 is completely independent of NTS. You
are trying to tie them together.

It's disingenuous for people to imply otherwise.

I couldn’t say, I don’t even know of an example of someone who does.

You've done it in two cases here, from everything I have seen.

Ragnar,

Ragnar,

Steven Sommars said:

The secure time transfer of NTS was designed to avoid

amplification attacks.

Uh, no.

Yes, it was.

As Steven said, “The secure time transfer of NTS was designed to
avoid amplification attacks”. I would even say - to make it
impossible to use for amplification attacks.

Please tell me how. I've been part of this specific topic since the
original NTS spec. For what y'all are saying to be true, there are some
underlying assumptions that would need to be in place, and they are
clearly not in place now and won't be until people update their
software, and even better, tweak their configs.

The NTS protected NTP request is always of the same size, or in some
cases larger, than the NTS protected NTP response. It is carefully
designed to work that way.

So what?

The use of NTS is completely independent of whether or not a server will
respond to a packet.

And an unauthenticated NTP request that generates an unauthenticated
response is *always* smaller than an authenticated request, regardless
of whether or not the server responds.

Claiming that amplification is a significant issue in the case where
there's no amplification but the base packet size is bigger is ignoring
a key piece of information, and is disingenuous in my book.

You are now comparing unauthenticated mode 3 and mode 4 packets to
cryptographically secured ones, which is a completely different thing.

Disingenuous?

A protocol with varying packet size, as the NTS protected NTP is,
can easily have the bad property of having responses larger than the
requests if not taken care. Don’t you see that?

Hence, [what Steven said].

If you understand what's going on from the perspective of both the
client and the server and think about the various cases, I think you'll
see what I mean.

Hopefully, no-one exposes mode 6 or mode 7 on the internet anymore
at least not unauthenticated, and at least not the commands that are
not safe from amplification attacks. Those just can not be allowed
to be used anonymously.

But mode 6/7 is completely independent of NTS.

Exactly. No one needs to, or should, expose mode6/7 at all. They were
designed at a time when the internet was thought to be nice place were
people behaved, decades ago, today they are just huge pains in the
rear. Sadly allowing anonymous mode 6/7 was left in there far to long
(admittedly being wise in hindsight is so much easier than in advance).
And here we are, with UDP port 123 still being abused by the bad
guys, and still being filtered by the networks.

Your statement about "exposing" is imprecise and bordering on incorrect,
at least in some cases.

Exposing to the Internet, or anyone but the system owner.

I just can’t imagine that you didn’t fully understand that.

But again, the use of mode 6/7 is completely independent of NTS. You
are trying to tie them together.

I am certainly not, and I think that should be perfectly clear from
the above.

Mode 6/7 packets should generally never be exposed outside localhost,
and should probably be replaced by something entirely different.

They are just a extremely troublesome relics from a time long ago,
and they should have been removed from anonymous exposure on the
Internet twenty years ago at least.

Don’t you understand that?
If you don't, you are part of the problem of killing UDP port 123,
not part of the solution for saving it.

It's disingenuous for people to imply otherwise.

I couldn’t say, I don’t even know of an example of someone who does.

You've done it in two cases here, from everything I have seen.

I have not. End of story.

Ragnar

I think I see the disconnect.

One of the design goals of NTS was to prevent NTS-protected time
requests from being used in amplification attacks. Yes, that's true.

I've been interpreting this thread as people claiming that NTS will
solve a wider class of amplification vectors, and that's simply not true.

"Universal affirmatives can only be partially converted: all of Alma
Cogen is dead, but only some of the class of dead people are Alma Cogen."

It's also true, and generally not stated, that to the extent that
NTS-protected packets are exchanged, they will require increased network
capacity. A cynic could argue that requiring additional internet
bandwidth is a profitable goal, and the drama about requiring that extra
protection is the distraction that normalizes that cost.

H

Ragnar,

Ragnar,

Steven Sommars said:

The secure time transfer of NTS was designed to avoid

amplification attacks.

Uh, no.

Yes, it was.

As Steven said, “The secure time transfer of NTS was designed to
avoid amplification attacks”. I would even say - to make it
impossible to use for amplification attacks.

Please tell me how. I've been part of this specific topic since the
original NTS spec. For what y'all are saying to be true, there are some
underlying assumptions that would need to be in place, and they are
clearly not in place now and won't be until people update their
software, and even better, tweak their configs.

The NTS protected NTP request is always of the same size, or in some
cases larger, than the NTS protected NTP response. It is carefully
designed to work that way.

So what?

The use of NTS is completely independent of whether or not a server will
respond to a packet.

And an unauthenticated NTP request that generates an unauthenticated
response is *always* smaller than an authenticated request, regardless
of whether or not the server responds.

Claiming that amplification is a significant issue in the case where
there's no amplification but the base packet size is bigger is ignoring
a key piece of information, and is disingenuous in my book.

You are now comparing unauthenticated mode 3 and mode 4 packets to
cryptographically secured ones, which is a completely different thing.

Disingenuous?

I made no such claim.

I was talking about:

As Steven said, “The secure time transfer of NTS was designed to
avoid amplification attacks”. I would even say - to make it
impossible to use for amplification attacks.

and that statement is not as clear as it could be. Specifically:

NTS was designed to avoid amplification attacks

is vague.

You have just now written:

You are now comparing unauthenticated mode 3 and mode 4 packets to
cryptographically secured ones, which is a completely different thing.

Completely different? How?

Where is the amplification in an unauthenticated mode 3 request and an
unauthenticated response?

How does cryptographically securing these packets make any difference here?

A protocol with varying packet size, as the NTS protected NTP is,
can easily have the bad property of having responses larger than the
requests if not taken care. Don’t you see that?

I sure think I understand these points.

Who here has said that there was any problem with there being an
amplification issue with properly-authenticated NTS packets?

The current NTS spec is *only* written for mode 3/4 exchanges. While it
might be applicable for mode 6/7, I haven't seen any specs for this
usage. In the NTP Project's Reference implementation there are extra
implementation-specific protections built in to mode 6/7 exchanges.
While some of this might be addressed in the NTS spec, I don't recall
ever seeing this.

So why are you talking about mode 6/7 in this context?

Hence, [what Steven said].

If you understand what's going on from the perspective of both the
client and the server and think about the various cases, I think you'll
see what I mean.

Hopefully, no-one exposes mode 6 or mode 7 on the internet anymore
at least not unauthenticated, and at least not the commands that are
not safe from amplification attacks. Those just can not be allowed
to be used anonymously.

But mode 6/7 is completely independent of NTS.

Exactly. No one needs to, or should, expose mode6/7 at all. They were
designed at a time when the internet was thought to be nice place were
people behaved, decades ago, today they are just huge pains in the
rear. Sadly allowing anonymous mode 6/7 was left in there far to long
(admittedly being wise in hindsight is so much easier than in advance).
And here we are, with UDP port 123 still being abused by the bad
guys, and still being filtered by the networks.

Your statement about "exposing" is imprecise and bordering on incorrect,
at least in some cases.

Exposing to the Internet, or anyone but the system owner.

I think you're in the right ballpark, but the edges of your boundaries
are a bit off.

I just can’t imagine that you didn’t fully understand that.

I think I have a pretty wide and deep understanding of these issues.

If I'm correct, then perhaps we are simply communicating imprecisely.

If I'm missing something, I'd like to know what it is.

But again, the use of mode 6/7 is completely independent of NTS. You
are trying to tie them together.

I am certainly not, and I think that should be perfectly clear from
the above.

I took my lead from the exchanges above.

Mode 6/7 packets should generally never be exposed outside localhost,
and should probably be replaced by something entirely different.

My initial inclination is to disagree with you first clause. Then I
noticed you used the word "generally" and perhaps we agree and have
different ways of expressing ourselves.

As for your second clause, yes, that might be a good solution. But
there's still a staggering number of v3 systems out there.

Simply creating a new mechanism and believing it will fix the problem
seems naive to me. But I also think incremental and appropriate use of
BCP 38, especially at the connection with the customer, is a good and
workable idea. I say this arrogantly, as I'm not in the business of
providing network access to entities.

They are just a extremely troublesome relics from a time long ago,
and they should have been removed from anonymous exposure on the
Internet twenty years ago at least.

Don’t you understand that?
If you don't, you are part of the problem of killing UDP port 123,
not part of the solution for saving it.

I think we're talking past each other.

It's disingenuous for people to imply otherwise.

I couldn’t say, I don’t even know of an example of someone who does.

You've done it in two cases here, from everything I have seen.

I have not. End of story.

Ragnar

I think we're talking past each other.

Hi Harlan,

I am quite sure that we actually generally agree and are just talking
past each other, and so are you judging from your mail below.

Let’s move this discussion from the list.

Regards,

Ragnar

Why? Why not pad requests to guarantee attenuation vector until
authenticity of packets can be verified?

MinimaLT does this. I think all UDP based and initial TCP should do
it, doing it for existing protocols may not be possible, but why not
for new?

I proposed similar method for proxy-trace (bidir tracerouting) -
https://github.com/ytti/proxy-trace/blob/master/draft-ytti-intarea-proxy-trace.xml#L169

Please help me understand this.

Exactly how bad is it if the query and response packets are of a
different size? Does it matter at 4 bytes? 32?

what are the expected costs of different %s of response packets being
sent back? I think this needs to be understood for a variety of
different sized query/response packets.

Then factor in the costs of padding the shorter packets. Include the
cost of the actual bump in network bandwidth, which I suspect is
negligible, and at the same time we can do a reality check on the folks
who claim that NTP packets are a significant cause of the bufferbloat
problem.

Then we need to thoroughly study how the various size differences
between the client request and server response packets affect the
quality of time synchronization, in various network scenarios. Some
have claimed this is clearly noticeable and significant. I'd like to
see the experiments and the data.

NTF is very happy to do this work, incrementally if needed, if we can
get the necessary support and cooperation/collaboration.

Presumably, if it's attenuation vector (1byte or more), presumably
attacker will use any of the other many vectors which are
amplification vectors or will directly attack from the zombie machines
they pwn. Since NST would have negative ROI on attack if there is
_any_ attenuation.

OK, and exactly how bad is a single byte attenuation, when compared
against the cost of 100% of all of the 1-byte shorter NTP packets being
made bigger to make the attenuation vector 0?