Help me make sense of these traceroutes please

Mailman List Mirror (Read Only)
1

Hello Nanog community,
I would like to enlist your help with understanding this latency I'm seeing.

First some background,
I have Level3 circuits in the US and some services in Europe. From Comcast to the US level3 IPs the performance is excellent. The same traceroute to Europe is terrible. The strange part is the problem appears to begin stateside on the same infrastructure that carriers the us traffic.

Here is a trace to one of my IPs in the US from Comcast

Tracing route to 4.30.x.x over a maximum of 30 hops

   1 3 ms 1 ms 1 ms 10.1.1.1
   2 30 ms 29 ms 29 ms 71.62.150.1
   3 9 ms 9 ms 9 ms xe-0-1-0-32767-sur01.winchester.va.richmond.comc
ast.net [68.85.71.165]
   4 9 ms 14 ms 10 ms xe-9-0-3-0-ar02.staplesmllrd.va.richmond.comcast
.net [68.86.125.149]
   5 32 ms 30 ms 34 ms 68.86.91.153
   6 36 ms 38 ms 53 ms 23.30.207.98
   7 34 ms 28 ms 33 ms vlan51.ebr1.Atlanta2.Level3.net [4.69.150.62]
   8 29 ms 28 ms 20 ms ae-63-63.ebr3.Atlanta2.Level3.net [4.69.148.241]

   9 27 ms 29 ms 30 ms ae-2-2.ebr1.Washington1.Level3.net [4.69.132.86]

  10 24 ms 30 ms 24 ms ae-71-71.csw2.Washington1.Level3.net [4.69.134.1
34]
  11 29 ms 31 ms 39 ms ae-41-90.car1.Washington1.Level3.net [4.69.149.1
95]
  12 30 ms 30 ms 29 ms ae-2-23.edge7.Washington1.Level3.net [4.68.106.2
38]
  13 38 ms 44 ms 43 ms 4.79.x.x
  14 * * * Request timed out. (My firewall)
  15 39 ms 39 ms 39 ms 4.30.x.x

Trace complete.

Now here is the same computer tracing to a level3 circuit in Ireland.

Tracing route to xxx.yyy.ie [193.1.x.x]
over a maximum of 30 hops:

   1 1 ms 1 ms 1 ms 10.1.1.1
   2 38 ms 33 ms 25 ms 71.62.150.1
   3 10 ms 9 ms 9 ms xe-0-1-0-32767-sur01.winchester.va.richmond.comc
ast.net [68.85.71.165]
   4 14 ms 15 ms 15 ms xe-9-0-3-0-ar02.staplesmllrd.va.richmond.comcast
.net [68.86.125.149]
   5 28 ms 30 ms 30 ms 68.86.95.65
   6 37 ms 37 ms 37 ms 23.30.207.98
   7 118 ms * 218 ms vlan51.ebr1.Atlanta2.Level3.net [4.69.150.62]
   8 119 ms 218 ms 119 ms ae-63-63.ebr3.Atlanta2.Level3.net [4.69.148.241]

   9 221 ms 119 ms 119 ms ae-2-2.ebr1.Washington1.Level3.net [4.69.132.86]

  10 118 ms 119 ms 118 ms ae-91-91.csw4.Washington1.Level3.net [4.69.134.1
42]
  11 119 ms 119 ms 119 ms ae-92-92.ebr2.Washington1.Level3.net [4.69.134.1
57]
  12 117 ms 126 ms 120 ms ae-43-43.ebr2.Paris1.Level3.net [4.69.137.57]
  13 128 ms 118 ms 120 ms ae-6-6.car1.Dublin3.Level3.net [4.69.148.53]
  14 122 ms 225 ms 124 ms 4.69.148.58
  15 124 ms 118 ms 120 ms ae-11-11.car1.Dublin1.Level3.net [4.69.136.93]

Notice that the hop from 23.30.207.98 to 4.69.150.62 seems very respectable at around 30ms for US bound traffic. However when I'm tracing from the same Comcast network to an IP that is in Europe the very same hope produces 100ms of latency and about 12% packet loss. Why does this hop treat traffic differently based on it's destination? Is this some weird result of complex asymmetrical routing or something else?

I can route around this problem, but it does seem strange and I want to understand it

Thanks,
Sam Moats

2

Hello Nanog community,
I would like to enlist your help with understanding this latency I'm
seeing.

You are likely seeing the effects of asymmetric routing.

[..]

Tracing route to xxx.yyy.ie [193.1.x.x]

www.heanet.ie by chance? :slight_smile:

Though you could use for instance:
http://planchet.heanet.ie/toolkit/gui/reverse_traceroute.cgi

to do a reverse traceroute, do make sure you force your connectivity to
IPv4 as that host will do IPv6 too. (locally nullrouting the destination
/128 is the trick I use for 'disabling' IPv6 temporarily).

Otherwise the HEANET folks are extremely helpful and clued in, you can
always ask them for help with issues. It is the end-of-year though and
those Irish folks have lots of really good whiskey, Guinness thus you
might have to be patient till the new year.

Alternatively, you could use a tool like 'tracepath' or 'mtr' as those
reports multiple answers to a response and also check for the TTL on the
return packets.

Greets,
Jeroen

3

Hello Nanog community,
I would like to enlist your help with understanding this latency I'm
seeing.

You are likely seeing the effects of asymmetric routing.

That's what I was thinking to.

[..]

Tracing route to xxx.yyy.ie [193.1.x.x]

www.heanet.ie by chance? :slight_smile:

Yes they were the owners of the IP I used for the example case and the heanet folks are actually totally awesome :slight_smile:

Though you could use for instance:
traceroute from 193.1.201.141 (planchet.heanet.ie) to your web browser at 5.161.46.87 (server-prof771.discoursehosting.net) for 5.161.46.87

to do a reverse traceroute, do make sure you force your connectivity to
IPv4 as that host will do IPv6 too. (locally nullrouting the destination
/128 is the trick I use for 'disabling' IPv6 temporarily).

Otherwise the HEANET folks are extremely helpful and clued in, you can
always ask them for help with issues. It is the end-of-year though and
those Irish folks have lots of really good whiskey, Guinness thus you
might have to be patient till the new year.

Also you'd be amazed how many network issues can be solved with a bunch of IT folks and an ample supply of Guinness

Alternatively, you could use a tool like 'tracepath' or 'mtr' as those
reports multiple answers to a response and also check for the TTL on the
return packets.

Greets,
Jeroen

Thanks, this isn't affecting my service now I've worked around it so it's more a curiosity than anything. It seems really odd to me that the same L3 edge router would route the ICMP unreachable back to me via different paths based on the final destination IP of the of the ICMP echo packet.

Well its Christmas eve here and the customers are happy so Guinness seems like the best approach now :slight_smile:

Thanks and have a good Holiday,
Sam Moats

4

Hello Nanog community,
I would like to enlist your help with understanding this latency I'm
seeing.

You are likely seeing the effects of asymmetric routing.

That's what I was thinking to.

[..]

Tracing route to xxx.yyy.ie [193.1.x.x]

www.heanet.ie by chance? :slight_smile:

Yes they were the owners of the IP I used for the example case and the
heanet folks are actually totally awesome :slight_smile:

Though you could use for instance:
traceroute from 193.1.201.141 (planchet.heanet.ie) to your web browser at 5.161.46.87 (server-prof771.discoursehosting.net) for 5.161.46.87

to do a reverse traceroute, do make sure you force your connectivity to
IPv4 as that host will do IPv6 too. (locally nullrouting the destination
/128 is the trick I use for 'disabling' IPv6 temporarily).

Otherwise the HEANET folks are extremely helpful and clued in, you can
always ask them for help with issues. It is the end-of-year though and
those Irish folks have lots of really good whiskey, Guinness thus you
might have to be patient till the new year.

Also you'd be amazed how many network issues can be solved with a bunch of
IT folks and an ample supply of Guinness

Alternatively, you could use a tool like 'tracepath' or 'mtr' as those
reports multiple answers to a response and also check for the TTL on the
return packets.

Greets,
Jeroen

Thanks, this isn't affecting my service now I've worked around it so it's
more a curiosity than anything. It seems really odd to me that the same L3
edge router would route the ICMP unreachable back to me via different paths
based on the final destination IP of the of the ICMP echo packet.

Based on the data you provided, my guess is some kind of MPLS transport
(please refer to
https://www.nanog.org/meetings/nanog45/presentations/Sunday/RAS_traceroute_N45.pdf,
pages 46-48).

HTH.

5

I once heard the claim that if you couldn't explain your network design and
have the listener understand it after you had split a pitcher of Guiness,
it was probably too complicated.

6

Pitcher of Guinness!?! What blasphemy is this, the only way to drink it is
via individually poured pint glasses.

Back to the issues I'd say MPLS or GHCQ before NSA.

7

Thats why you're a bacon zombie. If you were a living person you'd know free beer tastes the same irrespective of the containment vessel. :wink:

I hope Santa brought all of you what you wanted. If not, blame UPS.