Is anyone else seeing GPS timing source outages across the U. S. In the last two hours?
I’ve heard from people having the issue in South Africa as well.
Yes, lots of our key wireless network timing elements reported GPS timing sync failure during the past two hours.
Appears to be resolving now, within the past few minutes.
On what hardware/firmware are you seeing this?
Nothing unusual in the Bay Area so far, in other words "All Quiet on the
Western Front". I am seeing receivers synced to at least 6 or 7 sats. Two
different locations; one Garmin 18LVC, one uBlox 6M hardware. There is
also no chatter about it on the time-nuts list where I would suspect to
see it first.
-andreas
There’s a fair amount of hardware that uses GPS sync for wireless and I’m guessing this is likely a vendor bug w/ UBNT hardware, but it’s also good to know if it’s broader than that.
I know there’s some interesting java bugs related to the switchover for 2020 as well, eg:
https://twitter.com/NmVAson/status/1207820284268597249
I know of at least once place that was hit by this.
- Jared
No issues to report from Miami, FL. Clocks are marked as stable and keeping good time.
Ryan
Matt,
I saw a similar issue today with Cambium PMP450i’s losing sync from our PacketFlux rack injectors. May not be useful information to you know considering there’s chatter about this already on social media.
Thanks,
One of my hats is to design/manufacture/sell GPS third party timing receivers for Cambium Radios. It seems like something happened around 2PM MST today which caused (at a minimum) certain Globaltop/Sierra Wireless GPS modules to quit receiving signals from the GPS constellations. Because these modules are pretty well respected as far as the quality of the timing signal (normally) and relative low cost, they tend to be integrated in a lot of devices, including my products, the official Cambium Products, and I understand the gear from other manufacturers.
So far, a power cycle/reset of the GPS module seems to solve the problem.
I haven’t had enough time to process all of the data I have to know more details yet, but I can confirm the widespread nature of this event.
The following is from one ntpd instance in Chennai, India. We use Garmin
GPS 18x LVC receivers with our ntpd instances. ntpd interfaces directly
to the receiver using its NMEA driver and also receives a PPS signal
from the GPS receiver.
During last night, the ntpd process suddenly dropped its GPS reference
clock. This is the first time we've observed this happening. ntpq -np
showed an 'x' in the remote field where an 'o' should be. 'x' means "Out
of tolerance, do not use (discarded by intersection algorithm);"
(https://nlug.ml1.co.uk/2012/01/ntpq-p-output/831)
It was noticed in the morning. A drift is observed in
/var/log/peerstats.20191231. Corresponding to the log times of where the
drift starts, the $GPRMC sentences from /var/log/clockstats.20191231
show the following:
58848 73710.974 127.127.20.0 $GPRMC,202830,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73726.974 127.127.20.0 $GPRMC,202846,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73742.974 127.127.20.0 $GPRMC,202902,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73758.974 127.127.20.0 $GPRMC,202918,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73774.974 127.127.20.0 $GPRMC,202934,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73790.974 127.127.20.0 $GPRMC,202950,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73806.974 127.127.20.0 $GPRMC,203006,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73822.974 127.127.20.0 $GPRMC,203025,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73838.973 127.127.20.0 $GPRMC,203041,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73854.973 127.127.20.0 $GPRMC,203057,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73870.973 127.127.20.0 $GPRMC,203113,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73886.974 127.127.20.0 $GPRMC,203129,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73902.973 127.127.20.0 $GPRMC,203145,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__
58848 73918.973 127.127.20.0 $GPRMC,203201,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__
58848 73934.973 127.127.20.0 $GPRMC,203217,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__
58848 73950.973 127.127.20.0 $GPRMC,203233,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__
58848 73966.973 127.127.20.0 $GPRMC,203249,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__
58848 73982.973 127.127.20.0 $GPRMC,203305,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__
58848 73998.973 127.127.20.0 $GPRMC,203321,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__
58848 74014.973 127.127.20.0 $GPRMC,203337,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__
58848 74030.973 127.127.20.0 $GPRMC,203350,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__
58848 74046.973 127.127.20.0 $GPRMC,203406,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__
58848 74062.973 127.127.20.0 $GPRMC,203422,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__
58848 74078.973 127.127.20.0 $GPRMC,203438,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__
58848 74094.973 127.127.20.0 $GPRMC,203454,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__
58848 74110.973 127.127.20.0 $GPRMC,203510,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__
58848 74126.973 127.127.20.0 $GPRMC,203526,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__
58848 74142.974 127.127.20.0 $GPRMC,203542,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__
58848 74158.973 127.127.20.0 $GPRMC,203558,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__
These messages are logged once every 16 seconds. The $GPRMC time (UTC)
jumped from 203006 to 203025 (delta 19 seconds), and then corrected back
from 203337 to 203350 (delta 13 seconds). Note that these $GPRMC lines
are the (derived) output of the GPS receiver.
ntpd was restarted this morning, and appears to use the GPS timesource
properly.
Was there ever any additional information to come out about this? My Cambuim lost sync starting at 21:00:23 UTC 12/31/2019 and didn’t come back until I rebooted the unit…
Brandon Price
Senior Network Engineer
City of Sherwood, Sherwood Broadband
Desk: 503.625.4258
Cell: 971.979.2182
Over the weekend, tests confirmed that all of these issues are related to a end of the year GLONASS day rollover bug in the GNSS receiver. A power cycle seems to correct it until the next rollover in 2024.
If you’re still seeing these issues then power cycle the device using a method appropriate to the device. On the RackInjector in particular there is a power control button on the GNSS statusimulation screen.
I hate composing email on my phone. I always seems to mangle something.
Clarification: On the RackInjector, you can power cycle the GNSS receiver using a button on the GNSS status page.
Was there ever any additional information to come out about this? My Cambuim lost sync starting at 21:00:23 UTC 12/31/2019 and didn’t come back until I rebooted the unit…
Brandon Price
Senior Network Engineer
City of Sherwood, Sherwood Broadband
Desk: 503.625.4258
Cell: 971.979.2182
“It’s a glitch in the Matrix.”
Momentarily donning my tinfoil hat here, as I certainly don’t claim to know the nitty gritty of how GPS timing works, but…
What if, since the GPS system is operated by the US Air Force, there was something slipped into the GPS system relating to timing or accuracy or similar due to the events in the tinderbox of the Middle East?
</tinfoil_hat>
GPS != GLONASS
Only if it was the Russian Air Force...