NTP Sync Issue Across Tata (Europe)

Mailman List Mirror (Read Only)
21

Wouldn't a robust implementation of peering - say, seven peers, with the NTP algorithm handily selecting a subset to peer with for each cycle - require an attacker to know and overwhelm not just one, but a majority of the peer IPs?

Hi Royce,

More or less, yes. There are some corner cases where that may not be
true, but in terms of designing a useful attack, the attacker has to
be able to figure out which NTP servers you're talking to, flood you
with enough packets forged from all of them that the forged packet
arrives ahead of the real one, generally no more than 10s of
milliseconds behind the sent packet that only happens once every 15
minutes or so. And then it has to move you off real time gradually
enough for the NTP daemon not to decide the peer has become a false
ticker.

It's like DNS cache poisoning but a few orders of magnitude harder.

This is *not* to say that I'm advocating against maintaining local stratum 0s as part of a proper operator-grade NTP mesh. (My definition of "robust implementation" would include both local stratum 0 and a healthy serving of Internet stratum 1s). I'm just suggesting "don't peer with public servers" seems draconian given the deliberate design choices of the protocol.

For this audience, this would recommend a tiered system where multiple mixed-stratum internal stratrum 0+1s would peer with each other, and maintain internal-facing consensus for all other downstream internal devices - such that the vast majority of internal peers would indeed not be talking to the public Internet (but the "parent" peers would, and have the mesh and mix that I describe).

Well, it's not really one-size-fits-all.

Some systems consist of a well defined network containing routers and
servers with clear boundaries between self and Internet. Your approach
would work well there.

Some systems consist of equipment scattered at various data centers,
interconnected by the Internet itself. Implementing a GPS time
receiver at every one could get very expensive very fast. Standard
security rules apply: don't spend more protecting yourself than the
risk-cost. Risk is threat times vulnerability. Given the complexity of
the attack, you have to consider whether you're enough of a target
(threat) for anyone to bother.

Some systems are heavily virtualized, scattered over many vendors and
locations. You could -try- to track down a "secured" NTP source from
the vendor at each location, but that way lies configuration insanity.

Some systems are air-gapped from the Internet. You're going to get
time from GPS or the cellular phone network. GPS devices like the one
Mel pointed out are probably cheaper and more accurate.

Regards,
Bill Herrin

22

Bill,

You’re mistaking targeted NTP attacks with global ones. Yes, to attack your specific NTP client, the attacker has to know which NTP servers you’re using. But to simply succeed at random attacks, the attacker need only spoof popular servers. This is how time-shifting attacks work. Once an attack succeeds with a random victim, the hacker goes to work compromising time-sensitive security services.

And don’t forget that anyone can join pool.ntp.org. Including hackers. There is no credentialed vetting process.

-mel

23

The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need.

BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits.

OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it).

Rubens

24

Or one can read recent research papers that thoroughly document the incredible fragility of the existing NTP hierarchy and soberly consider their recommendations for remediation:

The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need.

Well DNS can be cryptographically secured. There really isn’t any good reasons to not sign your zones today. The majority of responses from authoritative servers are validated today so if you sign the responses will be checked. Unfortunately most to those validations still result in insecure instead of secure because people are not signing their zones.

25

GPS Selective Availability did not disrupt the timing chain of GPS, only the ephemeris (position information). But a government-disrupted timebase scenario has never occurred, while hackers are a documented threat.

DNS has DNSSec, which while not deployed as broadly as we might like, at least lets us know which servers we can trust.

Your own atomic clocks still have to be synced to a common standard to be useful. To what are they sync’d? GPS, I’ll wager.

I sense hand-waving :slight_smile:

-mel via cell

26

The problem with relying exclusively on GPS to do time distribution is the ease with which one can spoof the GPS signals.

With a budget of around $1K, not including a laptop, anyone with decent technical skills could convince a typical GPS receiver it was at any position and was at any time in the world. All it takes is a decent directional antenna, some SDR hardware, and depending on the location and directivity of your antenna maybe a smallish amplifier. There is much discussion right now in the PNT (Position, Navigation and Timing) community as to how best to secure the GNSS network, but right now one should consider the data from GPS to be no more trustworthy than some random NTP server on the internet.

In order to build a resilient NTP server infrastructure you need multiple sources of time distributed by multiple methods - typically both via satellite (GPS) and by terrestrial (NTP) methods. NTP does a pretty good job of sorting out multiple time servers and discarding sources that are lying. But to do this you need multiple time sources. A common recommendation is to run a couple/few NTP servers which only get time from a GPS receiver and only serve time to a second tier of servers that pull from both those in-house GPS-timed-NTP servers and other trusted NTP servers. I’d recommend selecting the time servers to gain geographic diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly both.

Note that NIST will exchange (via mail) a set of keys with you to talk encrypted NTP with you. See https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-authenticated-ntp-service .

27

To be honest, we have decent resiliency across the network to weather issues such as these. While going the GPS route is not a bad idea, I have more pressing problems to fix right now.

All that said, the issue seems to have corrected itself re: the Tata path (even if it's still the same path).

Appreciate all the feedback and support. Thanks.

Mark.

28

I forgot to finish my thought in the third paragraph before hitting send.

What I was going to express was that one should choose not only close, trusted, NTP servers, but also perhaps ones from different government agencies, or different sources. Sourcing time from multiple sources not likely to be affected by the same outage and/or attack is good robust design.

29

Hi Mark,

I have NTP servers in Europe that are choosing Tata (6453) to get to
0.freebsd.pool.ntp.org which lives on 197.224.66.40:

NTP is not sync'ing to that address, and sessions stay in an Init
state.

TL;DR: I'd guess your NTP Server IP address is geolocated to Mauritius. The Mauritius zone[0] on the pool has only one server, so you'll only see this one. To fix it, use europe.pool.ntp.org (_do not_ use pool.ntp.org).

Longer answer:

NTP pool folks use GeoDNS[1], which is their DNS server to map clients to servers.

The `0.freebsd.pool.ntp.org` name is just an alias for them -- what they really do is this:

  * Get geolocation_data(client_IP_address): <country, continent>
  * check country subzone in NTP pool (e.g, nl.pool.ntp.org [2]):
    * If there are >=1 servers in the zone, return (up to) 4 or them
    * If there is one, then return just one (this is a _known issue_)
    * if there is none, then fall back to the continent zone (Europe[3])

I've seen the same issue before with Guernsey clients (only one server). We have contact the pool operators and they are working now on a new GeoDNS version to prevent this from happening [4]

More details in [5].

In short, change your ntp configuration; the issue you have is that despite having 4k servers on the Pool, this strict GeoDNS mapping prevents you from accessing the other servers just bc of your IP address. The reasoning is to prevent asymmetric routing [4], but they are working on a fix to prevent these scenarios.

/giovane

[0] pool.ntp.org: NTP Servers in Mauritius, mu.pool.ntp.org
[1] GitHub - abh/geodns: DNS server with per-client targeted responses
[2] pool.ntp.org: NTP Servers in Netherlands, nl.pool.ntp.org
[3] pool.ntp.org: NTP Servers in Europe, europe.pool.ntp.org
[4] Minor new features on the website - #8 by ask - Server operators - NTP Pool Project
[5] https://www.sidnlabs.nl/downloads/5aPx86UtFmvKs6WE3LHwbU/c6acce6a012fe07256bab8caefff54af/Diving_into_the_NTP_Pool.pdf

30

So the Anycast address our devices use internally to find the closest NTP server is geo-mapped to MU. However, the physical server is geo-mapped to the specific countries in Europe, e.g., GB, NL, FR, DE, e.t.c.

Unless the geo data ntp.org are using is inconsistent, I'd imagine the servers should be mapped to a European pool, since the physical address from which the server queries the pool is geo-mapped locally, for this specific reason.

Mark.

31

So the Anycast address our devices use internally to find the closest
NTP server is geo-mapped to MU.

So indeed, the pool will only send you a single NTP server in this case.
GeoDNS essentially map you to mu.pool.ntp.org.

You can verify what NTP servers you can expect from the Pool by querying it directly (and thus bypassing GeoDNS mappings)

$ dig mu.pool.ntp.org

mu.pool.ntp.org. 62 IN A 197.224.66.40

However, the physical server is geo-mapped to the specific countries in Europe, e.g., GB, NL, FR, DE,

What really matters from GeoDNS is the IP address of your client -- the one that goes in the NTP query. So if you are using your anycast address to query, it does not matter what are the unicast addresses of your servers.

Unless the geo data ntp.org are using is inconsistent, I'd imagine
the servers should be mapped to a European pool, since the physical
address from which the server queries the pool is geo-mapped locally,
for this specific reason.

They also use the latest Maxmind mappings, and I confirmed it experimentally. ( I think it's fully automated their update method)

/giovane

32

GPS Selective Availability did not disrupt the timing chain of GPS, only the ephemeris (position information). But a government-disrupted timebase scenario has never occurred, while hackers are a documented threat.

DNS has DNSSec, which while not deployed as broadly as we might like, at least lets us know which servers we can trust.

NTP has NTS, which all 5 servers I mentioned have, and many more also have.

Your own atomic clocks still have to be synced to a common standard to be useful. To what are they sync’d? GPS, I’ll wager.

Nope, they are synced to the reference atomic clock of National
Observatory, pretty similar to USNO but local.

Rubens

33

> The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need.

Well DNS can be cryptographically secured. There really isn’t any good reasons to not sign your zones today. The majority of responses from authoritative servers are validated today so if you sign the responses will be checked. Unfortunately most to those validations still result in insecure instead of secure because people are not signing their zones.

So does NTP, with NTS.

Rubens

34

Diversity from GPS might also be obtained by setting one receiver for GPS and another for Galileo. I think I’d skip GLONASS for now :slight_smile:

35

The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need.

BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits.

OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it).

Rubens

Rubens

36

Forrest,

GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but commercial industrial NTP servers have specific anti-spoofing mitigations. There are also antenna diversity strategies that vendors support to ensure the signal being relied upon is coming from the right direction. It’s a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network.

-mel

37

Those particular boxes are not cheap. (Yes I know the units you talk about). Note that some of them rely on terrestrial communication of ephermis data to validate the GPS data to further make the time more robust.

I was hopefully trying to dispel the seemingly common thread in this discussion that somehow a $400 GPS derived ntp clock is more robust than using the public NTP infrastructure. Without understanding that GPS receivers are also subject to similar attacks as NTP, one comes to the wrong conclusion about how to build a robust timing infrastructure. GPS is subject to DoS and spoofing attacks just like NTP, and as such, it’s important to mitigate those as well if you care about time stability.

In the end, it’s all about risk vs cost to mitigate risk. If you’re running a tiny network, deriving your time from the public NTP servers is fine.

At the other end of the scale is using exotic hardened solutions which derive their own clock source and can hold over time for days or weeks in the event of a failure or attack.

In the middle tends to be a more moderate solution which involves a mix of time transmission methods from a variety of geographically and/or network diverse sources. Taking time from the public trusted ntp servers and adding lower cost GPS receivers at diverse points in your network seems like a good compromise in the middle. That way, only coordinated attacks will be successful.

38

Forrest Christian (List Account) wrote:

In the middle tends to be a more moderate solution which involves a mix of
time transmission methods from a variety of geographically and/or network
diverse sources. Taking time from the public trusted ntp servers and
adding lower cost GPS receivers at diverse points in your network seems
like a good compromise in the middle. That way, only coordinated attacks
will be successful.

Instead, just rely on atomic clocks operated by you. They are not
so expensive (several thousand dollars) and should be accurate
enough without adjustment for hundreds of years. There can be no
coordinated attacks. They may be remotely accessed through
secured NTP.

          Masataka Ohta

39

Masataka,

To be useful, any atomic clocks you operate must be synchronized to a Stratum Zero time source, such as GPS. Such clocks are useful when you need exceptional accuracy, such as for Building Integrated Timing Service (BITS), but unless they’re synchronized you can’t coordinate time-sensitive activities such as digital certificate validation with anyone else on the Internet.

From https://www.gps.gov/applications/timing/

Each GPS satellite contains multiple atomic clocks that contribute very precise time data to the GPS signals. GPS receivers decode these signals, effectively synchronizing each receiver to the atomic clocks. This enables users to determine the time to within 100 billionths of a second, without the cost of owning and operating atomic clocks.

Precise time is crucial to a variety of economic activities around the world. Communication systems, electrical power grids, and financial networks all rely on precision timing for synchronization and operational efficiency. The free availability of GPS time has enabled cost savings for companies that depend on precise time and has led to significant advances in capability.

-mel via cell

40

Depends on how synchronized you need to be.

In the context of running airgapped:

A rubidium oscillator or Chip Scale Atomic Clock is in the price range you quote. However, these can drift enough that you should occasionally synchronize with a reference time source. This is to ensure continued millisecond accuracy. Of course it all depends on how much drift you’ll tolerate, and if you’re OK with being within a second, then a rubidium might be ok.

Caesium oscillators which have much lower drift are in the $30K-50K range. These would require significantly less frequent synchronization, but are definitely not a few thousand dollars.

Note that these are both just oscillators and they need additional support hardware to be able to be queried by NTP. Or stated differently, they still need a NTP server. Yes, there are products out there which integrate everything in one box at an additional cost.