Hi All,
We have continued our investigation and data seem to show a more focused issue at the peering between Level 3 (CenturyLink) and MSN in LA. Can someone look at our data (new data below) and see if this seems like a reasonable conclusion?
From our network (San Diego) to an MSN Azure sever through the "Cogent | MSN Portal in LA" has no loss, but the same server has loss when going through the "Level 3 | MSN Portal in LA"
Also, we have data showing no packet loss through the same Level 3 LA (LosAngeles1) nodes to Cogent to Texas, so data seems to clear the Level 3 Los Angeles and Cogent peering.
Thanks,
Greg
Testing data:
IPERF UDP results looks good on path through Level 3 (CenturyLink) in LA (LosAngeles1) when going to a server in Texas (non-MSN Azure)
gfxc0.localdomain (0.0.0.0) Thu Nov 30 16:16:02 2017
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 216.75.40.1 0.0% 86 0.3 7.1 0.3 172.4 27.4
2. xe-8-3-3.bar1.SanDiego1.Level3.net 0.0% 86 3.3 3.6 3.3 25.4 2.4
3. ae-3-3.ebr1.LosAngeles1.Level3.net 90.6% 86 3.6 3.6 3.6 3.7 0.0
4. ae-1-51.ear2.LosAngeles1.Level3.net 95.3% 86 7225. 7157. 7134. 7225. 45.4
5. Cogent-level3-100G.LosAngeles1.Level3.net 0.0% 86 3.8 3.9 3.6 5.7 0.2
6. be3360.ccr42.lax01.atlas.cogentco.com 0.0% 86 3.6 3.7 3.5 4.0 0.0
7. be2932.ccr32.phx01.atlas.cogentco.com 0.0% 86 12.6 12.6 12.4 13.0 0.0
8. be2930.ccr21.elp01.atlas.cogentco.com 0.0% 85 20.7 20.9 20.6 22.5 0.2
9. be2928.ccr42.iah01.atlas.cogentco.com 0.0% 85 36.8 36.6 36.4 37.1 0.0
10. be2443.ccr32.dfw01.atlas.cogentco.com 0.0% 85 41.6 41.7 41.5 42.6 0.0
11. be2939.rcr21.dfw04.atlas.cogentco.com 0.0% 85 42.6 42.7 42.3 44.0 0.2
12. te0-0-1-1.nr12.b028597-0.dfw04.atlas.cogentco.com 0.0% 85 43.2 43.2 43.1 43.5 0.0
13. 38.122.200.202 0.0% 85 42.4 42.4 42.3 42.7 0.0
14. 138.128.243.167 0.0% 85 42.6 42.7 42.4 48.2 0.7
[root@gfxc0 ~]# iperf3 -uZVc 138.128.243.167 -b10m -t15 --get-server-output
iperf 3.1.3
Linux gfxc0.localdomain 4.12.5-300.fc26.x86_64 #1 SMP Mon Aug 7 15:27:25 UTC 2017 x86_64
Time: Fri, 01 Dec 2017 00:17:19 GMT
Connecting to host 138.128.243.167, port 5201
Cookie: gfxc0.localdomain.1512087439.341597.
[ 4] local 216.75.40.2 port 42277 connected to 138.128.243.167 port 5201
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 15 second test
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-1.00 sec 1.09 MBytes 9.11 Mbits/sec 139
[ 4] 1.00-2.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 2.00-3.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 3.00-4.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 4.00-5.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 5.00-6.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 6.00-7.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 7.00-8.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 8.00-9.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 9.00-10.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 10.00-11.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 11.00-12.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 12.00-13.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 13.00-14.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 14.00-15.00 sec 1.19 MBytes 9.96 Mbits/sec 152
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-15.00 sec 17.8 MBytes 9.94 Mbits/sec 0.057 ms 0/2274 (0%)
[ 4] Sent 2274 datagrams
CPU Utilization: local/sender 0.7% (0.1%u/0.6%s), remote/receiver 0.1% (0.1%u/0.0%s)
Server output:
Accepted connection from 216.75.40.2, port 43300
[ 5] local 138.128.243.167 port 5201 connected to 216.75.40.2 port 42277
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 1.08 MBytes 9.04 Mbits/sec 0.068 ms 0/138 (0%)
[ 5] 1.00-2.00 sec 1.20 MBytes 10.0 Mbits/sec 0.060 ms 0/153 (0%)
[ 5] 2.00-3.00 sec 1.19 MBytes 9.96 Mbits/sec 0.058 ms 0/152 (0%)
[ 5] 3.00-4.00 sec 1.20 MBytes 10.0 Mbits/sec 0.058 ms 0/153 (0%)
[ 5] 4.00-5.00 sec 1.19 MBytes 9.96 Mbits/sec 0.061 ms 0/152 (0%)
[ 5] 5.00-6.00 sec 1.20 MBytes 10.0 Mbits/sec 0.121 ms 0/153 (0%)
[ 5] 6.00-7.00 sec 1.19 MBytes 9.96 Mbits/sec 0.057 ms 0/152 (0%)
[ 5] 7.00-8.00 sec 1.20 MBytes 10.0 Mbits/sec 0.056 ms 0/153 (0%)
[ 5] 8.00-9.00 sec 1.20 MBytes 10.0 Mbits/sec 0.106 ms 0/153 (0%)
[ 5] 9.00-10.00 sec 1.19 MBytes 9.96 Mbits/sec 0.083 ms 0/152 (0%)
[ 5] 10.00-11.00 sec 1.20 MBytes 9.99 Mbits/sec 0.069 ms 0/153 (0%)
[ 5] 11.00-12.00 sec 1.19 MBytes 10.0 Mbits/sec 0.054 ms 0/152 (0%)
[ 5] 12.00-13.00 sec 1.20 MBytes 10.0 Mbits/sec 0.061 ms 0/153 (0%)
[ 5] 13.00-14.00 sec 1.20 MBytes 10.0 Mbits/sec 0.063 ms 0/153 (0%)
[ 5] 14.00-15.00 sec 1.19 MBytes 9.96 Mbits/sec 0.057 ms 0/152 (0%)
iperf Done.
[root@gfxc0 ~]#
Results still look NOT OK when going through same Level 3 (CenturyLink) LA (LosAngeles1) nodes ebr1 and ear2 to an MSN Azure server
My traceroute [v0.87]
gfxc0.localdomain (0.0.0.0) Thu Nov 30 16:24:54 2017
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 216.75.40.1 0.0% 95 0.3 10.9 0.2 199.5 38.9
2. xe-8-3-3.bar1.SanDiego1.Level3.net 0.0% 95 3.3 3.7 3.2 20.6 1.9
3. ae-3-3.ebr1.LosAngeles1.Level3.net 98.9% 95 4.0 4.0 4.0 4.0 0.0
4. ae-1-51.ear2.LosAngeles1.Level3.net 96.8% 95 7101. 7046. 7004. 7101. 50.0
5. Microsoft-level3-20G.LosAngeles1.Level3.net 0.0% 95 15.2 17.7 9.3 20.8 2.4
6. be-63-0.ibr01.lax03.ntwk.msn.net 1.1% 95 29.9 30.1 22.7 34.0 2.6
7. be-4-0.ibr01.by2.ntwk.msn.net 1.1% 95 31.6 30.2 20.4 33.9 2.6
8. 104.44.7.198 0.0% 94 29.8 29.9 20.3 33.7 2.7
9. ae102-0.icr02.by21.ntwk.msn.net 0.0% 94 35.7 29.2 20.5 41.4 2.7
10. ???
[root@gfxc0 ~]# iperf3 -uZVc 13.91.55.110 -b10m -t15 --get-server-output
iperf 3.1.3
Linux gfxc0.localdomain 4.12.5-300.fc26.x86_64 #1 SMP Mon Aug 7 15:27:25 UTC 2017 x86_64
Time: Fri, 01 Dec 2017 00:25:50 GMT
Connecting to host 13.91.55.110, port 5201
Cookie: gfxc0.localdomain.1512087950.883346.
[ 4] local 216.75.40.2 port 34611 connected to 13.91.55.110 port 5201
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 15 second test
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-1.00 sec 1.09 MBytes 9.11 Mbits/sec 139
[ 4] 1.00-2.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 2.00-3.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 3.00-4.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 4.00-5.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 5.00-6.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 6.00-7.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 7.00-8.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 8.00-9.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 9.00-10.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 10.00-11.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 11.00-12.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 12.00-13.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 13.00-14.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 14.00-15.00 sec 1.19 MBytes 9.96 Mbits/sec 152
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-15.00 sec 17.8 MBytes 9.94 Mbits/sec 0.141 ms 102/2274 (4.5%)
[ 4] Sent 2274 datagrams
CPU Utilization: local/sender 0.7% (0.1%u/0.6%s), remote/receiver 0.1% (0.0%u/0.1%s)
Server output: