Starlink terminal data acquisition for network engineers

I thought about posting this to only NANOG, but since a great concentration of beta testers of a technical/network engineering inclination are located in the Pacific NW, decided to also include the SIX chat list.

You may have seen the Starlink android or ios consumer-friendly app, which displays network traffic, uptime/downtime, and other link stats. I believe this to be polled directly from the antenna unit itself over grpc.

The beta antennas are always 192.168.100.1. If you are using your own router with the starlink beta system, in addition to its WAN interface being an ordinary DHCP client in cgnat IP space, you’ll need to manually give it an address in that /24 and set up routing to reach the .1 IP as needed.

reference: https://github.com/sparky8512/starlink-grpc-tools

reference: https://github.com/fullstorydev/grpcui

you’ll need a fairly normal Linux or BSD box with:

git

go

python3
pip

use pip to install grpcio and grpcio-tools

install grpcurl: https://github.com/fullstorydev/grpcurl

do a git clone of the starlink-grpc-tools url above, also take a look at its readme info

get the dish’s protoset file and write it to new file dish.protoset , this is an index of all data that can be polled
cd /home/eric/starlink-grpc-tools
/home/eric/go/bin/grpcurl -plaintext
-protoset-out dish.protoset
192.168.100.1:9200
describe SpaceX.API.Device.Device

/home/eric/go/bin/grpcurl
-plaintext
-d {“get_history”:{}}
192.168.100.1:9200
SpaceX.API.Device.Device/Handle | python parseJsonHistory.py

output of the above looks like this:
2021-02-06T08:15:56,3600,19.92034332,14,2,0.3125,0,0,0.0,0

full CSV header for the above:
datetimestamp_utc,samples,total_ping_drop,count_full_ping_drop,count_obstructed,total_obstructed_ping_drop,count_full_obstructed_ping_drop,count_unscheduled,total_unscheduled_ping_drop,count_full_unscheduled_ping_drop

since we are able to acquire the above in a comma-delimited csv format, it’s fairly easy to write a script storing the integers from any one of those particular columns into a mariadb db, sqlite, influxdb, or whatever.

the following will output about 3.8MB of text for the full history (I believe this to be the full copy of the ring buffer stored in RAM for the terminal’s statistics) , pipe it into a text file if you want to manually look at it.

/home/eric/go/bin/grpcurl -plaintext -d {“get_history”:{}} 192.168.100.1:9200 SpaceX.API.Device.Device/Handle

same as the above but human readable output

/home/eric/go/bin/grpcurl
-plaintext
-d {“get_history”:{}}
192.168.100.1:9200
SpaceX.API.Device.Device/Handle | python parseJsonHistory.py -v

current counter: 299673
All samples: 43200
Valid samples: 43200
Parsed samples: 3600
Total ping drop: 20.03700998
Count of drop == 1: 14
Obstructed: 2
Obstructed ping drop: 0.3125
Obstructed drop == 1: 0
Unscheduled: 0
Unscheduled ping drop: 0.0
Unscheduled drop == 1: 0

see the get_history_notes.txt file for more info

SOME EXAMPLE QUERIES
these should match with what the json query is in the grpc GUI

get status

/home/eric/go/bin/grpcurl
-plaintext
-d {“getStatus”:{}}
192.168.100.1:9200
SpaceX.API.Device.Device/Handle

get device info

/home/eric/go/bin/grpcurl
-plaintext
-d {“getDeviceInfo”:{}}
192.168.100.1:9200
SpaceX.API.Device.Device/Handle

get history, this outputs a huge amount of data

/home/eric/go/bin/grpcurl
-plaintext
-d {“getHistory”:{}}
192.168.100.1:9200
SpaceX.API.Device.Device/Handle

The following is copied/pasted from my notes file on things we can acquire, and then use a tiny shell script with awk, sed, regex, whatever as needed to trim out just the numbers, and put them somewhere else for polling by snmp extend

/home/eric/go/bin/grpcurl
-plaintext
-d {“getStatus”:{}}
192.168.100.1:9200
SpaceX.API.Device.Device/Handle

notes on what’s what:
figures we care about to parse out and turn into just the integers
snr can never be higher than 9
fractionobstructed appears to be the percentage of the time that the view is obstructed, as long as the view
remains unobstructed, this number appears to slowly decrement over time
validS is valid seconds? i think
the S is almost likely always Seconds of time

“uptimeS”: “304439”
“snr”: 9,
“fractionObstructed”: 0.0013524292,
“validS”: 61815.74,
“last24hObstructedS”: 53
“downlinkThroughputBps”: 47915.73,
“uplinkThroughputBps”: 34980.496,
“popPingLatencyMs”: 29.266666

example of running the command above twice, the second time a few minutes after the first,
to see that fractionObstructed does decrement itself over time
first run: 0.0013524292
second run: 0.0013467998

Thanks Eric, this is great information. I just installed my Starlink
"dishymcdishface" this weekend and it's humming along nicely, even in
the -25F (-51F with windchill) temps here in northern MN! I've got a
bunch of general monitoring of the link using a few Docker + grafana
widgets [1], but hadn't looked into direct metrics gathering from the
terminal yet. This will surely get me on that path :slight_smile:

--Tim

[1] Docker Hub

Tim Nelson
Network Engineer / Cloud Services
Phone: (218)727-4332 x4501 / Fax: (866)716-0229
5019 Airport Road, Hermantown, MN 55811
Check us out at www.sangoma.com

Anyone else noticing major MAJOR problems with various MS services?

Geoff

There is chatter on outages list outages@outages.org and some other forums so I think you are not alone.

Notice we received indicated that they were experiencing authentication issues across all MS services.

LQM

Can you be a bit more specific regarding what you’re seeing or not seeing?

Are you reaching MS through IP transit/peer connections, or are you having issues reaching MS cloud services over ExpressRoute circuits?

Thank you
jms

I didn’t troubleshoot at all (not my job), but yes, we are having all sorts of issues accessing O365/Teams/etc

Azure is having outages atm - https://twitter.com/MSFT365Status?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor

WVD seems to be affected as well… tak tsk tsk. I guess this is part of Monday blues? :stuck_out_tongue:

teams is failing, powerbi is failing, azure is failing. I don’t use hotmail, so I don’t know…

https://status.azure.com/en-us/status

SUMMARY OF IMPACT: Starting at approximately 19:15 UTC on 15 Mar 2021, a subset of customers may experience issues authenticating into Microsoft services, including Microsoft Teams, Office and/or Dynamics, Xbox Live, and the Azure Portal.

CURRENT STATUS: Engineering teams have been engaged and are currently investigating. The next update will be provided in 60 minutes or as events warrant.

This message was last updated at 19:58 UTC on 15 March 2021

Good afternoon,

I work retail at a tech store in Canada as my day job, and we’ve been experiencing issues today only with activating Microsoft product cards. Unsure if it’s related to the other Microsoft issues, but it’s a good possibility as no other product card activations for other vendors are affected. Another neighboring retailer is having the same issue as well, so it’s definitely a possibility.

Regards,

Peter P.

https://status.office365.com/