Leap second reminder

Mailman List Mirror (Read Only)
1

Just a reminder, at midnight UTC there's a leap second added to most time systems.

Some time systems will stop the clock at 23:59:59.999999 for 1 second, some will display 23:59:60 for a second.

Since the last leap second (1998), "leap second aware" time keeping systems(NTP, GPS, etc) have become much more prevalent, so it's much more likely this time that applications and NTP sync'ed devices will see a leap second happen(rather than have them manually corrected later, or not corrected at all). But, I'm not too sure how well everyone has tested applications and devices for how they handle the clock stopping for a second OR an "invalid" time of 23:59:60.

If anyone sees anything die at 00:00:00UTC I'd be interested to know.

-- Kevin

2

Just a reminder, at midnight UTC there’s a leap second added to most time systems.

I’ve had quite a few replies already off list, that I’ll sum up:

  1. ISO 8601 specifically defines “:60” as a valid representation for seconds, just for this event. If your app breaks, it’s probably not obeying the relevant RFC.

  2. Anything using the obvious implementation of regular expressions to check for a valid time is probably going to fail “23:59:60”, leading to events not being logged, or transient weirdness for that second.

One example I thought of after a few minutes is SpamAssassin, who will flag a mail header:

As “INVALID_DATE”, bumping up its spam score by 1.7 to 2.0(by default), because of the regexp it uses which contains “[0-5][0-9]”

  1. Two people have noted that their Juniper routers spat out some strange ntpd messages earlier today, such as:

Dec 31 06:26:46 hostname xntpd[66479]: time reset 0.999681 s
Dec 31 06:48:11 hostname xntpd[66479]: time reset -0.437785 s
(local time set to America/Chicago)

  1. You can’t set your clock to 23:59:60 manually, which makes testing what’s going to happen difficult.

  2. The “leap second pending” bit that’s supposed to have been set by now in NTP isn’t being set by some clock sources, leading some to wonder what they’re planning on doing at midnight.

3

The most comprehensive supplier information I have found so far.
http://www.ntp-systems.com/leapsecond/

Colin Johnston
HHT IT

4

My Juniper seems to be aware:

xxx@juniper> show ntp status
status=4694 leap_add_sec, sync_ntp, 9 events, event_peer/strat_chg,
version="ntpd 4.1.0-a Thu Jul 14 23:46:40 GMT 2005 (1)",
processor="i386", system="JUNOS7.3R1.5", leap=01, stratum=3,
precision=-30, rootdelay=40.669, rootdispersion=49.522, peer=35302,
refid=ntpx.xxx.xxx,
reftime=c7615c1c.65f78359 Sat, Dec 31 2005 13:35:56.398, poll=10,
clock=c7615d8e.d66d698f Sat, Dec 31 2005 13:42:06.837, state=4,
offset=-2.649, frequency=73.810, jitter=5.194, stability=0.024

note the leap=01 and leap_add_sec

5

Cisco seems happy as well. (adding leap second, leap second occurs at), etc.

>sh ntp status
Clock is synchronized (adding leap second), stratum 2, reference is xxx.xxx.xxx.xxx
nominal freq is 250.0000 Hz, actual freq is 249.9975 Hz, precision is 2**18
reference time is C7617E39.3A0B3D8C (17:01:29.226 EST Sat Dec 31 2005)
clock offset is 0.5332 msec, root delay is 5.11 msec
root dispersion is 7.72 msec, peer dispersion is 7.14 msec
Leap second occurs at C7619A00.00000000 (19:00:00.000 EST Sat Dec 31 2005)

Happy New Year!

Deepak

Gerry Boudreaux wrote:

6

Last NTP spam:

I'm by no means an NTP expert, if anyone else is, please pipe up.

About 30 minutes before the leap second should have occurred, several of our systems reported "xntpd[13742]: time reset 0.958385 s", which was really strange. They moved the wrong direction, and they did it early. Shortly after, those systems lost ntp association and began drifting. About 10 minutes after midnight all have regained sync. I wasn't checking things that early to see why, it's possible some of our NTP sources started disagreeing on what the correct time was, and would also match what other people have reported off-list, going back as far as 18 hours before midnight.

Several public NTP sources are now indicating a "leap second alarm" (setting the leap bits to 11), which will cause most NTP clients to rule them out as a source. ntp-2.gw.uiuc.edu is an example:

130.126.24.44: Server dropped: Leap not in sync
server 130.126.24.44, port 123
stratum 2, precision -19, leap 11, trust 000
refid [128.174.38.133], delay 0.03357, dispersion 0.00049

According to ntpdate, its clock seems to have stopped about 5 minutes before midnight, and hasn't yet recovered.

Other NTP servers haven't cleared their "today is a leap second day" bit, which they should have by now. Some NTP implementations rule out servers that don't agree with what their "master" server thinks the leap second bits should be. My reading of the NTP spec says that at 00:00:00 the leap bits should have been returned to zero. Attempting to sync from one of these servers will produce a "Next leap second occurs at 00:00:00.000 UTC Sun Jan 01 2006" message, but that should be harmless as long as they correct themselves a while before midnight.

Still others have their clocks off by a significant amount(10+ minutes) and think they're still in sync, but since I started typing this email, they all have corrected themselves.

While I can't say anything broke on our network as a result of the leap second, a good percentage of our gear lost NTP sync or had some kind of NTP problem around midnight UTC. You may want to check your NTP status at some point, in case something drifted quite a way off and won't step itself back now because the difference is too great.

-- Kevin

7

Kevin Day wrote:

Last NTP spam:

I'm by no means an NTP expert, if anyone else is, please pipe up.

About 30 minutes before the leap second should have occurred, several of our systems reported "xntpd[13742]: time reset 0.958385 s", which was really strange. They moved the wrong direction, and they did it early. Shortly after, those systems lost ntp association and began drifting. About 10 minutes after midnight all have regained sync. I wasn't checking things that early to see why, it's possible some of our NTP sources started disagreeing on what the correct time was, and would also match what other people have reported off-list, going back as far as 18 hours before midnight.

Several public NTP sources are now indicating a "leap second alarm" (setting the leap bits to 11), which will cause most NTP clients to rule them out as a source. ntp-2.gw.uiuc.edu is an example:

130.126.24.44: Server dropped: Leap not in sync
server 130.126.24.44, port 123
stratum 2, precision -19, leap 11, trust 000
refid [128.174.38.133], delay 0.03357, dispersion 0.00049

According to ntpdate, its clock seems to have stopped about 5 minutes before midnight, and hasn't yet recovered.

Other NTP servers haven't cleared their "today is a leap second day" bit, which they should have by now. Some NTP implementations rule out servers that don't agree with what their "master" server thinks the leap second bits should be. My reading of the NTP spec says that at 00:00:00 the leap bits should have been returned to zero. Attempting to sync from one of these servers will produce a "Next leap second occurs at 00:00:00.000 UTC Sun Jan 01 2006" message, but that should be harmless as long as they correct themselves a while before midnight.

Still others have their clocks off by a significant amount(10+ minutes) and think they're still in sync, but since I started typing this email, they all have corrected themselves.

While I can't say anything broke on our network as a result of the leap second, a good percentage of our gear lost NTP sync or had some kind of NTP problem around midnight UTC. You may want to check your NTP status at some point, in case something drifted quite a way off and won't step itself back now because the difference is too great.

-- Kevin

There is at least one stratum-1 server here on the West coast that my NTP says is now off by 1 second. Several stratum-2 are synced to it and are now off also. So checking servers might be a good idea

Roy Engehausen

8

Are they a GPS sync (or do you know?)

  The GPS system doesn't really handle the leap
second situation the same as others, there is a little blurb
here that talks about it:

http://gpsinformation.net/main/gpstime.htm

  I know that I saw my GPS output the following:

@051231235958
@051231235959
@060101000000
@060101000000
@060101000001
@060101000002
@060101000003

  Which is mostly correct, it should have really read 5960 instead.

  I saw my el-cheapo clocks drift by a second at midnight utc.

  I suggest changing your clock sources to something more reliable
if you're seeing folks that are not diligent stratum-1 sources.

  I suggest this as a source:

  http://www.stupi.se/Time/

  I'm kinda curious what CDMA network clocks said around this time
or if they just drifted, i'm sure someone here put their cmda phone in debug
and watched it.

  - jared

9

Once upon a time, Kevin Day <toasty@dragondata.com> said:

While I can't say anything broke on our network as a result of the
leap second, a good percentage of our gear lost NTP sync or had some
kind of NTP problem around midnight UTC. You may want to check your
NTP status at some point, in case something drifted quite a way off
and won't step itself back now because the difference is too great.

I watched my Tru64 5.1B and Linux 2.2-2.6 servers (NTP wasn't running on
my Solaris 9 server accidentally) and Juniper and Cisco gear, and they
all stayed in sync. The Linux systems logged:

Dec 31 17:59:59 kosh kernel: Clock: inserting leap second 23:59:60 UTC

17:59:59 lasted 2 seconds in local time (since the normal time zone data
doesn't pass through leap seconds).

I saw the following from JUNOS:

Dec 31 18:00:00 hsvrouter /kernel: microuptime() went backwards (8144133.847075 -> 8144132.881330)

Tru64 and Cisco didn't log anything.

10

We've Nagios monitoring a majority of our NTP devices. Around the appropriate time I got a pretty big flurry of ntp sync warnings, took about half an hour for everything to get in sync. Everything looks normal as of right now (has been for a while).

I hadn't thought to turn off the alarms even though I was aware of the leap. That resulted in a lot of notifications going out to our on-call people.

11

Linux seemed to survive happily too
[ytti@foo ~]% dmesg|tail -n1
Clock: inserting leap second 23:59:60 UTC

Curious though that not so many people who've I asked got these messages,
only explanation I can think of is that their NTP peers weren't telling
it.

Without much NTP clues, could someone explain what steps NTP takes to
protect itself from attackers spoofing packets and forcing you to leap?
Probably sane implementation would restrict leaping to happen only
at 23:59:59, so you'd drift maximum of 1s/d? But crappy implementation
might allow you to leap every second.

This http://www.ntp.org/ntpfaq/NTP-s-config-adv.htm#AEN2950
just tells
Most users of NTP do not need authentication as the protocol contains
several filters against bad time.

So I guess it's pretty implementation spesific how the input is
sanitized.

12

Interesting. ntp-2.gw.uiuc.edu is an old Cisco box running 12.0(27). I wasn't around for the leap second, so I don't have any information about what happened, and its log is completely silent. FWIW, its clock and NTP process appear fine now. It's synced off our GPS clock, which may be a part of the problem.

ntp-2>show ntp sta
Clock is synchronized, stratum 2, reference is 128.174.38.133
nominal freq is 250.0000 Hz, actual freq is 249.9982 Hz, precision is 2**19
reference time is C7642455.CAE491EF (16:14:45.792 CST Mon Jan 2 2006)
clock offset is 0.0594 msec, root delay is 3.57 msec
root dispersion is 6.30 msec, peer dispersion is 0.12 msec

ntp-2>show ntp asso

       address ref clock st when poll reach delay offset disp
+~130.126.24.24 128.174.38.133 2 192 1024 336 6.4 -0.15 4.0
+~192.5.41.40 .USNO. 1 477 1024 377 32.5 -2.24 1.5
*~128.174.38.133 .PPS. 1 442 1024 377 3.6 0.06 0.1
+~130.126.24.53 128.174.38.133 2 623 1024 377 8.5 1.20 3.1
* master (synced), # master (unsynced), + selected, - candidate, ~ configured

/cvk