If there's anybody from Level 3 Transport available, I'd like to discuss some bizarre results when traversing through your network, namely in Dallas, TX over the past few months? I'm working this through your NOC as well, but figured I would cover all avenues as this issue is pretty chronic.
Taken from your Looking Glass in Dallas to a destination of 195.110.36.136
1 vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 120 msec
vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 120 msec
vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 120 msec
2 ae-93-93.ebr3.Dallas1.Level3.net (4.69.151.170) 120 msec
ae-73-73.ebr3.Dallas1.Level3.net (4.69.151.146) 120 msec
ae-63-63.ebr3.Dallas1.Level3.net (4.69.151.134) 120 msec
3 ae-7-7.ebr4.Atlanta2.Level3.net (4.69.134.22) 36 msec 20 msec 20 msec
4 ae-2-2.ebr1.Washington1.Level3.net (4.69.132.86) 120 msec 116 msec 120 msec
5 ae-71-71.csw2.Washington1.Level3.net (4.69.134.134) 120 msec
ae-81-81.csw3.Washington1.Level3.net (4.69.134.138) 120 msec
ae-71-71.csw2.Washington1.Level3.net (4.69.134.134) 120 msec
6 ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) 120 msec 120 msec
ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) 124 msec
7 ae-44-44.ebr2.Paris1.Level3.net (4.69.137.61) 120 msec
ae-42-42.ebr2.Paris1.Level3.net (4.69.137.53) 116 msec 116 msec
8 ae-62-62.csw1.Paris1.Level3.net (4.69.161.94) 124 msec
ae-72-72.csw2.Paris1.Level3.net (4.69.161.98) 120 msec 120 msec
9 ae-81-81.ebr1.Paris1.Level3.net (4.69.161.85) 120 msec
ae-91-91.ebr1.Paris1.Level3.net (4.69.161.89) 124 msec
ae-71-71.ebr1.Paris1.Level3.net (4.69.161.81) 120 msec
10 ae-41-41.ebr2.London2.Level3.net (4.69.159.81) 120 msec 120 msec
ae-43-43.ebr2.London2.Level3.net (4.69.159.89) 120 msec
11 ae-12-3202.edge4.London2.Level3.net (4.69.202.182) 116 msec 120 msec 120 msec
12 ae-26-26.car2.London2.Level3.net (4.69.200.98) 120 msec 120 msec 120 msec
13 BACKBONE-CO.car2.London2.Level3.net (195.50.121.50) 120 msec 120 msec 120 msec
14 * * *
15 * * *
16 * * *
I'm confused how the first and last hops are showing equal latency, while being half way around the world. I'm also confused why it's 120ms to the first hop from a route server local to that market. While I wouldn't rely on a traceroute as a true measurement of end to end latency, I'm having problems explaining to customers experiencing tangible issues when their traceroute looks like this:
traceroute source 24.155.144.226 195.110.36.136
traceroute to 195.110.36.136 (195.110.36.136) from 24.155.144.226, 30 hops max, 40 byte packets
1 lag-8-868.ear1.Dallas1.Level3.net (4.30.74.53) 924.705 ms 545.117 ms 512.992 ms
2 4.69.146.5 (4.69.146.5) 124.812 ms 4.69.146.21 (4.69.146.21) 125.686 ms 4.69.146.9 (4.69.146.9) 124.018 ms
MPLS Label=1965 CoS=0 TTL=1 S=1
3 ae-73-73.ebr3.Dallas1.Level3.net (4.69.151.146) 125.012 ms ae-83-83.ebr3.Dallas1.Level3.net (4.69.151.158) 141.585 ms ae-63-63.ebr3.Dallas1.Level3.net (4.69.151.134) 125.005 ms
MPLS Label=1810 CoS=0 TTL=1 S=1
4 * * *
5 ae-2-2.ebr1.Washington1.Level3.net (4.69.132.86) 126.085 ms 125.994 ms 127.148 ms
MPLS Label=1692 CoS=0 TTL=1 S=1
6 ae-61-61.csw1.Washington1.Level3.net (4.69.134.130) 124.500 ms ae-81-81.csw3.Washington1.Level3.net (4.69.134.138) 125.369 ms ae-61-61.csw1.Washington1.Level3.net (4.69.134.130) 124.306 ms
MPLS Label=1555 CoS=0 TTL=1 S=1
7 ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) 131.126 ms 128.607 ms ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) 123.955 ms
MPLS Label=1636 CoS=0 TTL=1 S=1
8 ae-42-42.ebr2.Paris1.Level3.net (4.69.137.53) 127.156 ms ae-41-41.ebr2.Paris1.Level3.net (4.69.137.49) 125.736 ms ae-43-43.ebr2.Paris1.Level3.net (4.69.137.57) 143.070 ms
MPLS Label=1801 CoS=0 TTL=1 S=1
9 ae-82-82.csw3.Paris1.Level3.net (4.69.161.102) 122.998 ms 122.245 ms ae-72-72.csw2.Paris1.Level3.net (4.69.161.98) 124.499 ms
MPLS Label=1564 CoS=0 TTL=1 S=1
10 ae-61-61.ebr1.Paris1.Level3.net (4.69.161.77) 129.705 ms ae-81-81.ebr1.Paris1.Level3.net (4.69.161.85) 122.705 ms ae-91-91.ebr1.Paris1.Level3.net (4.69.161.89) 126.454 ms
MPLS Label=1322 CoS=0 TTL=1 S=1
11 ae-42-42.ebr2.London2.Level3.net (4.69.159.85) 126.019 ms 125.672 ms ae-44-44.ebr2.London2.Level3.net (4.69.159.93) 127.026 ms
MPLS Label=1911 CoS=0 TTL=1 S=1
12 ae-12-3202.edge4.London2.Level3.net (4.69.202.182) 122.058 ms 124.301 ms 123.397 ms
MPLS Label=300112 CoS=0 TTL=1 S=1
13 ae-26-26.car2.London2.Level3.net (4.69.200.98) 124.003 ms 124.567 ms 130.346 ms
14 BACKBONE-CO.car2.London2.Level3.net (195.50.121.50) 125.316 ms 125.991 ms 124.968 ms
15 * * *
This also doesn't seem to be circuit specific (Eliminating possible physical circuit errors, etc) as I have a secondary 10-Gig sourcing out of San Antonio, TX (Which ultimately terminates on your network in Dallas, same as the Austin Circuit) which demonstrates similar results (120+ ms at the second hop in Dallas when entering your MPLS cloud). Killing the BGP adjacency to both the Austin and San Antonio peers clears the issue up, but obviously that's not a long term viable option. Any help would be appreciated.
JJ Stonebraker
IP Network Engineering
Grande Communications
512.878.5627