Help interpret a strange traceroute?

Mailman List Mirror (Read Only)
1

Happy monday all!

Any idea how a traceroute (into my network) could end up this fubar'd? Discovered this wierd routing while investigating horrendously slow speeds (albeit no packet loss) to a particular ISP abroad.

It's like - coming into us - the packets are taking every available path, simultaniously. Tracing back to him looks perfectly normal.

Thank you in advance for any comments on or off list.

Him to us:
traceroute to mirror.ash.fastserv.com (208.85.240.29), 64 hops max, 52 byte packets
  1 192.168.1.1 (192.168.1.1) 1.552 ms 1.522 ms 1.261 ms
  2 br1.l.hiweb.ir (46.224.0.12) 17.750 ms 18.932 ms 18.811 ms
  3 172.16.5.65 (172.16.5.65) 18.505 ms 20.044 ms 18.414 ms
  4 int.gr1.s.hiweb.ir (46.224.0.125) 19.096 ms 18.447 ms 18.752 ms
  5 int.gr1.l.hiweb.ir (46.224.0.117) 18.454 ms 18.768 ms 103.441 ms
  6 10.21.252.166 (10.21.252.166) 66.308 ms 18.728 ms 18.867 ms
  7 * * *
  8 185.100.209.139 (185.100.209.139) 162.126 ms
     ae5.istanbul1.ist.seabone.net (93.186.132.212) 210.124 ms 126.208 ms
  9 et7-1-0.franco71.fra.seabone.net (195.22.214.131) 102.694 ms
     if-ae-9-3.tcore1.pvu-paris.as6453.net (195.219.87.14) 189.955 ms
     185.100.209.197 (185.100.209.197) 218.539 ms
10 te0-3-0-4.rcr21.b023101-0.lon01.atlas.cogentco.com (149.14.144.89) 201.223 ms 215.996 ms
     185.100.209.1 (185.100.209.1) 164.381 ms
11 if-ae-2-2.tcore2.av2-amsterdam.as6453.net (195.219.194.6) 199.161 ms
     be2950.ccr22.lon01.atlas.cogentco.com (130.117.2.109) 216.648 ms
     if-ae-3-6.tcore1.l78-london.as6453.net (80.231.130.85) 271.750 ms
12 if-ae-4-2.thar1.njy-newark.as6453.net (80.231.130.34) 183.984 ms *
     be2814.ccr42.ams03.atlas.cogentco.com (130.117.0.141) 109.045 ms
13 if-ae-1-3.thar2.njy-newark.as6453.net (216.6.57.2) 207.371 ms
     be2870.ccr41.lon13.atlas.cogentco.com (154.54.58.173) 186.804 ms
     be12488.ccr42.lon13.atlas.cogentco.com (130.117.51.41) 190.236 ms
14 if-ae-14-14.tcore2.nto-new-york.as6453.net (66.198.111.126) 203.156 ms
     if-ae-9-2.tcore1.n75-new-york.as6453.net (63.243.128.122) 273.177 ms
     be2807.ccr42.dca01.atlas.cogentco.com (154.54.40.110) 235.632 ms
15 66.110.96.134 (66.110.96.134) 182.695 ms
     be2806.ccr41.dca01.atlas.cogentco.com (154.54.40.106) 278.593 ms 240.075 ms
16 66.110.96.142 (66.110.96.142) 245.454 ms
     be3084.ccr41.iad02.atlas.cogentco.com (154.54.30.66) 191.973 ms
     be3083.ccr41.iad02.atlas.cogentco.com (154.54.30.54) 240.740 ms
17 te0-0-0-0.nr13.b023801-0.iad01.atlas.cogentco.com (154.24.36.18) 257.027 ms
     be3083.ccr41.iad02.atlas.cogentco.com (154.54.30.54) 236.617 ms
     hu-1-3-0-2-cr02.newyork.ny.ibone.comcast.net (68.86.83.97) 221.186 ms
18 be-10102-cr02.ashburn.va.ibone.comcast.net (68.86.85.161) 194.766 ms 250.493 ms
     be-10203-cr01.newark.nj.ibone.comcast.net (68.86.85.185) 191.052 ms
19 be-10102-cr02.ashburn.va.ibone.comcast.net (68.86.85.161) 197.694 ms
     te0-0-2-0.nr11.b023801-0.iad01.atlas.cogentco.com (154.24.36.2) 266.914 ms 235.444 ms
20 te0-0-2-3.nr11.b023801-0.iad01.atlas.cogentco.com (154.24.36.6) 255.756 ms
     38.88.249.210 (38.88.249.210) 216.677 ms
     te0-0-2-0.nr11.b023801-0.iad01.atlas.cogentco.com (154.24.36.2) 211.364 ms
21 208.85.240.29 (208.85.240.29) 274.692 ms 212.896 ms
     ash01-mls-dc-core-a.latisys.net (67.217.175.37) 208.976 ms

And back to him:
traceroute to 46.224.145.65 (46.224.145.65), 30 hops max, 60 byte packets
  1 104.153.64.146 (104.153.64.146) 22.067 ms 22.049 ms *
  2 xe-3-1-1.er1.iad10.us.above.net (208.185.24.1) 0.398 ms 0.392 ms 0.420 ms
  3 zayo-tata.iad10.us.zip.zayo.com (64.125.13.170) 0.708 ms 0.580 ms 0.623 ms
  4 if-ae-11-2.thar2.NJY-Newark.as6453.net (216.6.87.138) 92.900 ms if-ae-11-3.thar2.NJY-Newark.as6453.net (216.6.87.242) 92.802 ms if-ae-11-4.thar2.NJY-Newark.as6453.net (216.6.87.169) 97.177 ms
  5 if-ae-1-3.thar1.NJY-Newark.as6453.net (216.6.57.1) 92.940 ms 93.008 ms 89.431 ms
  6 if-ae-8-2.tcore1.LDN-London.as6453.net (66.198.70.174) 97.315 ms 96.049 ms 95.855 ms
  7 if-ae-17-2.tcore1.L78-London.as6453.net (80.231.130.129) 98.074 ms if-ae-3-6.tcore1.PYE-Paris.as6453.net (80.231.130.86) 92.733 ms 92.898 ms
  8 if-ae-2-2.tcore1.PVU-Paris.as6453.net (80.231.154.17) 93.636 ms if-ae-3-6.tcore1.PYE-Paris.as6453.net (80.231.130.86) 97.917 ms 98.363 ms
  9 if-ae-2-2.tcore1.PVU-Paris.as6453.net (80.231.154.17) 100.909 ms if-ae-9-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.13) 92.055 ms 93.173 ms
10 195.219.87.46 (195.219.87.46) 208.894 ms 209.042 ms if-ae-9-2.tcore2.FNM-Frankfurt.as6453.net (195.219.87.9) 94.718 ms
11 * 195.219.87.46 (195.219.87.46) 209.640 ms int.gr1.s.hiweb.ir (46.224.0.118) 202.201 ms
12 int.cr1.s.hiweb.ir (46.224.0.126) 205.150 ms * *
13 * * *
14 int.gr1.s.hiweb.ir (46.224.0.118) 201.803 ms 201.799 ms int.cr1.s.hiweb.ir (46.224.0.126) 201.819 ms
15 int.cr1.s.hiweb.ir (46.224.0.126) 204.952 ms 198.969 ms *
16 46.224.145.65 (46.224.145.65) 210.507 ms 211.144 ms 206.998 ms

2

Hi Randy,

This is per-packet load balancing. In the forward path the alternates
are different lengths but the traceroute stops as soon as at least one
of the paths reaches the destination.

The return path is also engaged in per-packet load balancing but the
paths are all the same length.

Regards,
Bill Herrin

3

Hi Randy,

ECMP loadbalancing is most frequently done on layer3+layer4 headers, and unixlike traceroute use UDP with increasing destination port number for each packet (usually starting at 33434), which allows to see the different available paths, as wrote William.

Would you want/need to stick to only one traceroute path, you may use ICMP traceroute instead of UDP traceroute (no port in ICMP, so only layer 3 available to loadbalance, so all packets will go through the same interface).

Usually it is achieved by using traceroute -I yourdest
Windows tracert is ICMP only traceroute by the way. MTR tool is also ICMP based by default.

Keep in mind that it looses some useful information, though (since you see only one path and don't decide which).
So, you can also use UDP traceroute with fixed port (by example 33434 with no port increase), and try again the same traceroute with another destport (with fixed port too, by example 33435), which would display two different paths in a more readable way. RTFM is required since the options depend on your traceroute particular specie :slight_smile:

Olivier

4

This is also a handy tool that addresses the same issues:

https://paris-traceroute.net/

5

Hi,

I am trying to determine the physical diversity of the Zayo and Level3 networks vis-a-vis each other on the European racetrack - London/Amsterdam/Frankfurt/Paris/London. It is for a client of mine.

Regards,

Roderick.

6

Seems like a lot of bandwidth trying to save bandwidth. Or does that only happen to ICMP?

7

try Telegeography.com

best regards
Wolfgang

8

Does anyone have an IP that involves a load balancing router to test with?