NTP Sync Issue Across Tata (Europe)

Hi all.

I have NTP servers in Europe that are choosing Tata (6453) to get to 0.freebsd.pool.ntp.org which lives on 197.224.66.40:

traceroute -I 0.freebsd.pool.ntp.org
traceroute to 0.freebsd.pool.ntp.org (197.224.66.40), 64 hops max, 48 byte packets
1 ae-2-24.er-01-ams.nl.seacomnet.com (105.26.64.13) 0.300 ms 0.301 ms 0.215 ms
2 ce-0-0-11.cr-01-mrs.fr.seacomnet.com (105.16.8.201) 22.163 ms 22.370 ms 22.084 ms
3 ce-0-0-3.br-01-mrs.fr.seacomnet.com (105.16.32.254) 20.230 ms 20.243 ms 20.139 ms
4 ix-hge-0-0-0-28.ecore3.emrs2-marseille.as6453.net (80.231.165.52) 21.875 ms 21.679 ms 21.762 ms
5 * if-be-3-2.ecore2.emrs2-marseille.as6453.net (195.219.175.0) 42.751 ms *
6 if-ae-25-2.tcore1.ldn-london.as6453.net (195.219.175.5) 43.509 ms 43.280 ms 43.353 ms
7 195.219.83.158 (195.219.83.158) 203.310 ms 203.452 ms 203.209 ms
8 196.20.225.84 (196.20.225.84) 208.289 ms 208.637 ms 208.374 ms
9 197.226.230.13 (197.226.230.13) 209.657 ms 209.658 ms 209.830 ms
10 197.224.66.40 (197.224.66.40) 208.638 ms 208.632 ms 208.712 ms

NTP is not sync’ing to that address, and sessions stay in an Init state.

Other NTP servers I have in Africa are reaching the same address via local Anycast hosters, and sync’ing just fine.

Anyone else seeing this particular issue across Tata in Europe?

Mark.

Hi Mark.

Wouldn’t a local server be more optimal?

IE:

	   server 0.nl.pool.ntp.org
	   server 1.nl.pool.ntp.org
	   server 2.nl.pool.ntp.org
	   server 3.nl.pool.ntp.org

Or for Africa
   server 0.za.pool.ntp.org
	   server 1.za.pool.ntp.org
	   server 2.za.pool.ntp.org
	   server 3.za.pool.ntp.org

Matthew

I have a bunch of servers all over Europe I’d prefer not to touch if this is just a transient issue. I was speaking about Europe. There is no problem in Africa. Mark.

See for yourself how his pool server scores at https://www.ntppool.org/scores/197.224.66.40

I am not sure why it would be inserted into DNS answers for a worldwide pool like 0.freebsd as it clearly does have connectivity issues from some of the pool project’s own sensors.
-andreas

Many thanks, Andreas.

I'll take this up with the FreeBSD folk.

Mark.

Once upon a time, Mark Tinka <mark@tinka.africa> said:

Yes, Andreas clarified in unicast. Will do. Thanks.

Mark.

Mark,

You might consider setting up your own GPS-based NTP network. Commercial Ethernet GPS-sourced NTP servers, such as the Time Machines, TM1000A, are as little as $400. Or you can roll your own using a Raspberry Pi or similar nano computer with a GPS module and antenna. We use these exclusively in data centers now rather than depending on Internet NTP servers, primarily for security, because financial transactions in e-commerce can be sensitive to false time information. There are also a variety of NTP-based Internet attacks, so if you can block NTP at your border you’ve eliminated another attack surface.

-mel via cell

If the path has simmetric one way latencies you don’t have to pick a lower latency faulty one. Perhaps creating a selection at startup and then using that collection ?

Rubens

GPS time is monitored (and when necessary, adjusted) from the U.S.
Naval Observatory Master Clock, which is -the- authoritative time
source for the United States. The USNO also provides an NTP time
source from the same master clock:

https://www.cnmoc.usff.navy.mil/Our-Commands/United-States-Naval-Observatory/Precise-Time-Department/Network-Time-Protocol-NTP/

You -should not- just point your servers there, but it's useful to
point a few servers each at one of them in order to serve as your
network stratum 2 sources that keep the rest of your machines in sync
with each other.

That last point is key. You don't want your servers in sync with
random Internet time sources. You want them in sync with each other.

Regards,
Bill Herrin

Bill,

That still leaves you open to NTP attacks. The USNO accuracy and monitoring is worthless if you suffer, for example, an NTP DDoS attack.

[
ddos-lc.png

NTP amplification DDoS attack
cloudflare.com

](https://www.cloudflare.com/learning/ddos/ntp-amplification-ddos-attack/)

There are also replay and Man in the middle attacks (MITM) which can corrupt local NTP servers’ time basis. Worse, security flaws in NTP make others security protocols, such as SSL, vulnerable.

https://www.sidn.nl/en/news-and-blogs/security-flaws-in-network-time-protocol-make-other-security-protocols-vulnerable

if you can eliminate such security problems for $400, I say it’s cheap at twice the price.

-mel

* mel@beckman.org (Mel Beckman) [Sun 06 Aug 2023, 04:26 CEST]:

if you can eliminate such security problems for $400, I say it’s cheap at twice the price.

You must be unfamiliar with the prices neutral colocation facilities charge for roof access.

  -- Niels.

That still leaves you open to NTP attacks. The USNO accuracy and monitoring is worthless if you suffer, for example, an NTP DDoS attack.

Hi Mel,

From what I can tell, a fairly simple firewall policy of allow UDP 123
from known NTP clients and established connections (I sent them a UDP
packet recently) stops every one of those attacks (that's actually an
NTP attack and not something else like a DNS attack) except for
upstream address hijack that happens to coincide with your system
boot. And it still depends on the attacker executing an additional
sophisticated attack to do more than cause you a denial of service.

The links you sent are very interesting, at least in an academic
sense, but they don't cause me to be unduly concerned about employing
NTP.

if you can eliminate such security problems for $400, I say it’s cheap at twice the price.

Except you can't. Redundancy is required for any critical service. At
the $400 price point, your approach has multiple
single-points-of-failure. The device itself of course. Your ability to
receive continuous non-jammed GPS signals at the location where you're
able to place an antenna. And in your plan you'll need one of these in
every discontiguous network where you have equipment since you're not
doing NTP over the Internet.

Not to mention the operations cost. Keeping track of a six inch brick
with a wall wart and an antenna installed at a remote site is... not
entirely abnormal but it's a one-off that consumes manpower.

And then you're only vulnerable to the litany of Internet attacks
which don't involve NTP. Yay!

Don't get me wrong: the Time Machines TM1000A you recommended looks
like a cool little device well worth checking into. As a supplement to
Internet NTP, not a replacement.

Regards,
Bill Herrin

William,

Due to flaws in the NTP protocol, a simple UDP filter is not enough. These flaws make it trivial to spoof NTP packets, and many firewalls have no specific protection against this. in one attack the malefactor simply fires a continuous stream of NTP packets with invalid time at your firewall. When your NTP client queries the spoofed server, the malicious packet is the one you likely receive.

That’s just one attack vector. There are several others, and all have complex remediation. Why should people bother being exposed to the risk at all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve already described. Having suffered through such attacks more than once, I can say from personal experience that you don’t want to risk it.

-mel

Naively, instead of abstaining :wink: … isn’t robust diversity of NTP peering a reasonable mitigation for this, as designed?

A carefully selected set of stratum 0 sources for a set of stratum 1 servers is the heart of good NTP source design. With at least four “local” stratum 1 servers, Dr. Mills algorithm is excellent at distinguishing truechimers from falsetickers and providing a reliable source of monotonic time. DOS is a separate problem.

My NTP network deployment experience for a major auto manufacturer, among others, is in agreement with William Herin. A GPS NTP source is a valid Stratum 0 source, but relying on a single instance for local time is not exceedingly better than querying time.apple.com or a similar source.

In a nutshell, no. Refer to my prior cites for detailed explanations. For a list of real-world attack incidents, see

https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#

-mel

Respectfully, that Wikipedia article (which is mostly about legit but unauthorized clients overwhelming a given peer) and your cites don’t seem to cover what I was responding to - the “don’t peer with public NTP because someone can flood your firewall and spoof the responses” problem. I just want to make sure that I’m understanding the distinction betwen this class and other classes of attack.

Wouldn’t a robust implementation of peering - say, seven peers, with the NTP algorithm handily selecting a subset to peer with for each cycle - require an attacker to know and overwhelm not just one, but a majority of the peer IPs?

This is not to say that I’m advocating against maintaining local stratum 0s as part of a proper operator-grade NTP mesh. (My definition of “robust implementation” would include both local stratum 0 and a healthy serving of Internet stratum 1s). I’m just suggesting “don’t peer with public servers” seems draconian given the deliberate design choices of the protocol.

For this audience, this would recommend a tiered system where multiple mixed-stratum internal stratrum 0+1s would peer with each other, and maintain internal-facing consensus for all other downstream internal devices - such that the vast majority of internal peers would indeed not be talking to the public Internet (but the “parent” peers would, and have the mesh and mix that I describe).

And I’m well aware of who I’m saying this to … so I expect to find out where I’m wrong, as I keep digging. :smiley:

This entirely discounts the fact that bcp-38 and bcp-84 which, more or less, eliminate this “problem space” entirely.

I find it hard to believe ntp reflection is actually a problem in the year 2023, assuming you’re not running a ridiculously old ntp client and have taken really simple steps to protect your network.

Or one can select NTS-capable NTP servers, like those 5:
a.st1.ntp.br
b.st1.ntp.br
c.st1.ntp.br
d.st1.ntp.br
gps.ntp.br

Or any other NTP server that has NTS deployed. Game-over for NTP impersonation.

Rubens