www.att.net ipv6 traceroute

Anyone else noticing odd ipv6 traceroutes to www.att.net
(2001:1890:1c01:2::40)?

For example, the first traceroute originating in Chicago, and if the
hostnames are anything to go by:

Chicago -> Dallas -> Atlanta -> Miama -> NYC -> Amsterdam ->
Frankfurt -> France -> D.C. -> Atlanta...

The first 2 traceroutes are on SixXs tunnels and the third is native.

From 2604:8800::/32 (Chicago, Sixxs)

2 gw-122.chi-03.us.sixxs.net 25.925 ms 14.816 ms 13.645 ms
3 uschi03.sixxs.net 18.651 ms 15.544 ms 34.06 ms
4 2620:0:6b0:a::1 27.401 ms 15.677 ms 14.83 ms
5 tge6-3.fr3.ord4.ipv6.llnw.net 14.482 ms 18.875 ms 21.112 ms
6 ve8.fr3.ord.ipv6.llnw.net 28.965 ms 24.031 ms 15.947 ms
7 tge2-1.fr3.dal.ipv6.llnw.net 39.672 ms 41.431 ms 49.888 ms
8 tge4-1.fr3.atl1.ipv6.llnw.net 59.908 ms 62.198 ms 72.244 ms
9 tge2-1.fr3.mia1.ipv6.llnw.net 78.641 ms 75.351 ms 74.425 ms
10 2600:804:11f::11 91.243 ms 90.002 ms 87.802 ms
11 2600:802::23 119.804 ms 100.727 ms 100.737 ms
12 ge-0-3-0.IL1.NYC12.ALTER.NET 147.803 ms 145.406 ms 147.878 ms
13 ae1.XT2.AMS2.ALTER.NET 148.699 ms ae1.XT1.AMS2.ALTER.NET 146.322 ms 152.503 ms
14 GigabitEthernet6-0-0.GW5.AMS6.ALTER.NET 145.967 ms 148.903 ms 165.926 ms
15 FastEthernet0-0.6B1.AMS6.ALTER.NET 183.716 ms 156.131 ms 194.141 ms
16 tu3.6B1.FFT4.ALTER.NET 184.867 ms 184.005 ms 170.47 ms
17 ffm-s1-rou-1006.DE.eurorings.net 159.565 ms * 173.012 ms
18 ffm-s1-rou-1102.DE.eurorings.net 200.82 ms 225.979 ms 207.712 ms
19 kehl-s2-rou-1103.DE.eurorings.net 228.884 ms 208.576 ms 203.722 ms
20 nntr-s1-rou-1101.FR.eurorings.net 192.901 ms 210.818 ms 210.279 ms
21 * 2001:680:0:134:222:226:160:1 189.452 ms 171.714 ms
22 2001:1890:c00:c001::1171:ecca 192.306 ms 164.33 ms 192.835 ms
23 2001:1890:c00:c001::ee71:ecca 222.488 ms 218.21 ms 194.537 ms
24 2001:1890:ff:ffff:12:122:105:90 214.309 ms 216.21 ms 216.51 ms
25 wswdc22crs.ipv6.att.net 206.215 ms 220.884 ms 202.472 ms
26 attga21crs.ipv6.att.net 213.776 ms 220.918 ms 205.205 ms
27 attga409me3.ipv6.att.net 190.66 ms 228.532 ms 200.606 ms
28 2001:1890:c00:8802::11cc:6c6d 217.65 ms 222.145 ms 210.213 ms

From 2a02:2528::/32 (Switzerland, Sixxs)

2 gw-91.gva-01.ch.sixxs.net 28.932 ms 28.415 ms 29.091 ms
3 2a02:2528:ff:1::4 28.082 ms 28.873 ms 41.525 ms
4 ge0-1.gv1.ip-max.sixxs.net 29.546 ms 37.835 ms 29.366 ms
5 ge3-14.ar01.gva253.ip-max.net 30.502 ms 32.925 ms 29.370 ms
6 ae1.cr01.gva253.ip-max.net 45.019 ms 41.335 ms 47.675 ms
7 xe0-0-1.cr01.gva252.ip-max.net 28.903 ms 33.530 ms 41.954 ms
8 po1.br01.gva252.ip-max.net 39.794 ms 34.197 ms 28.843 ms
9 r1gva1.core.init7.net 28.849 ms 32.812 ms 31.988 ms
10 r1zug1.core.init7.net 47.583 ms 42.633 ms 37.785 ms
11 r1glb1.core.init7.net 36.323 ms 56.574 ms 34.909 ms
12 r1zrh1.core.init7.net 50.896 ms 56.617 ms 42.967 ms
13 ae0-12.zur11.ip6.tinet.net 42.462 ms 41.909 ms 41.557 ms
14 xe-0-1-1.nyc20.ip6.tinet.net 128.395 ms
    xe-0-1-2.nyc20.ip6.tinet.net 127.736 ms
    xe-0-1-1.nyc20.ip6.tinet.net 120.968 ms
15 nyiix.iij.net 121.285 ms 121.758 ms 120.904 ms
16 nyc002bb10.iij.net 121.436 ms 122.211 ms 120.475 ms
17 abn001bb10.iij.net 121.829 ms 123.181 ms 121.787 ms
18 2600:803:92f::1 143.860 ms 125.196 ms 179.323 ms
19 2600:802::23 139.937 ms 132.728 ms 140.416 ms
20 ge-1-2-0.il1.nyc12.alter.net 130.477 ms
    ge-0-3-0.il1.nyc12.alter.net 130.349 ms 140.231 ms
21 * * *
22 gigabitethernet6-0-0.gw5.ams6.alter.net 349.594 ms 129.648 ms
    gigabitethernet7-0-0.gw5.ams6.alter.net 130.583 ms
23 fastethernet0-0.6b1.ams6.alter.net 138.677 ms 169.896 ms 402.511 ms
24 tu3.6b1.fft4.alter.net 379.628 ms 156.939 ms 138.227 ms
25 * * *
26 ffm-s1-rou-1102.de.eurorings.net 219.035 ms 227.354 ms 236.133 ms
27 kehl-s2-rou-1103.de.eurorings.net 228.896 ms 221.190 ms 239.280 ms
28 nntr-s1-rou-1101.fr.eurorings.net 219.410 ms 226.779 ms 230.940 ms
29 2001:680::134:222:226:160:1 247.598 ms 243.938 ms 218.783 ms
30 2001:1890:c00:c001::1171:ecca 217.493 ms 242.119 ms 217.778 ms
31 2001:1890:c00:c001::ee71:ecca 238.297 ms 219.211 ms 218.524 ms
32 2001:1890:ff:ffff:12:122:105:90 257.048 ms 250.699 ms 252.160 ms
33 wswdc22crs.ipv6.att.net 248.764 ms 268.970 ms 416.443 ms
34 *
    attga21crs.ipv6.att.net 239.030 ms 241.492 ms
35 attga409me3.ipv6.att.net 249.902 ms 268.800 ms 253.631 ms
36 2001:1890:c00:8802::11cc:6c6d 260.203 ms 236.484 ms 245.413 ms

From 2001:1b60::/32 (Germany, Native)

2 2001:1b60:0:1:0:1:0:1 0.275 ms 0.247 ms 0.237 ms
3 Pironet.FRA-12-eth012-615.v6.lambdanet.net 3.199 ms 3.198 ms 4.578 ms
4 ae5.irt1.fra44.de.v6.as13237.net 6.518 ms 6.499 ms 6.507 ms
5 xe-0-3-0-0.fra21.ip6.tinet.net 6.570 ms 6.563 ms 6.543 ms
6 xe-0-1-2.nyc20.ip6.tinet.net 86.740 ms
    xe-0-0-2.nyc20.ip6.tinet.net 86.833 ms
    xe-0-2-1.nyc20.ip6.tinet.net 86.715 ms
7 NYIIX.IIJ.Net 88.196 ms 88.046 ms 88.016 ms
8 nyc002bb10.iij.net 85.712 ms 85.844 ms 85.978 ms
9 abn001bb10.iij.net 92.920 ms 93.113 ms 92.785 ms
10 2600:803:92f::1 93.331 ms 92.965 ms 92.713 ms
11 2600:802::23 94.009 ms 94.048 ms 100.621 ms
12 ge-0-3-0.IL1.NYC12.ALTER.NET 99.927 ms 99.896 ms
    ge-1-2-0.IL1.NYC12.ALTER.NET 99.866 ms
13 ae1.XT2.AMS2.ALTER.NET 97.016 ms
    ae1.XT1.AMS2.ALTER.NET 96.994 ms 96.919 ms
14 GigabitEthernet7-0-0.GW5.AMS6.ALTER.NET 97.120 ms 98.054 ms
    GigabitEthernet6-0-0.GW5.AMS6.ALTER.NET 97.111 ms
15 FastEthernet0-0.6B1.AMS6.ALTER.NET 104.272 ms 106.899 ms 106.249 ms
16 tu3.6B1.FFT4.ALTER.NET 107.443 ms 100.970 ms 98.980 ms
17 * *
    ffm-s1-rou-1006.DE.eurorings.net 99.641 ms
18 ffm-s1-rou-1102.DE.eurorings.net 190.998 ms 183.395 ms 190.432 ms
19 kehl-s2-rou-1103.DE.eurorings.net 190.216 ms 203.128 ms 192.594 ms
20 nntr-s1-rou-1101.FR.eurorings.net 183.713 ms 193.837 ms 202.330 ms
21 2001:680:0:134:222:226:164:1 209.937 ms 182.641 ms 182.782 ms
22 2001:1890:c00:c001::1171:ecca 188.257 ms 183.682 ms 216.084 ms
23 2001:1890:c00:c001::ee71:ecca 205.741 ms 183.258 ms 204.891 ms
24 2001:1890:ff:ffff:12:122:105:90 247.476 ms 242.229 ms 251.679 ms
25 wswdc22crs.ipv6.att.net 237.167 ms 216.849 ms 228.173 ms
26 attga21crs.ipv6.att.net 208.246 ms 230.028 ms 232.396 ms
27 attga409me3.ipv6.att.net 255.300 ms 251.860 ms 249.848 ms
28 2001:1890:c00:8802::11cc:6c6d 233.211 ms 215.398 ms 206.866 ms

Looks fine to me:

mpetach@tftp1:~> traceroute6 www.att.net
traceroute6 to www.att.net (2001:1890:1c01:2::40) from
2001:4998:f002:1fe::100, 64 hops max, 12 byte packets
1 ge-0-3-1-d2.pat1.pao.yahoo.com 0.369 ms 0.352 ms 0.367 ms
2 ae-8.pat1.dnx.yahoo.com 25.348 ms 25.363 ms 123.170 ms
3 2605:6c00:303:303::69 35.219 ms 25.622 ms 25.956 ms
4 10gigabitethernet4-3.core1.chi1.he.net 52.456 ms 59.001 ms 53.043 ms
5 att-internet4-as7018.10gigabitethernet5.switch2.chi1.he.net 75.188 ms
77.365 ms 75.445 ms
6 cgcil21crs.ipv6.att.net 102.561 ms 104.090 ms 103.922 ms
7 cl2oh21crs.ipv6.att.net 105.803 ms 106.685 ms 103.180 ms
8 * * *
9 nsvtn21crs.ipv6.att.net 97.371 ms 119.415 ms 92.557 ms
10 * * *
11 attga409me3.ipv6.att.net 113.108 ms 119.130 ms 118.172 ms
12 2001:1890:c00:8802::11cc:6c6d 126.793 ms 117.539 ms 128.418 ms

Palo Alto->Denver->Chicago->Cleveland->Nashville->Atlanta

mpetach@tftp1:~> traceroute6 www.att.net
traceroute6 to www.att.net (2001:1890:1c01:2::40) from
2a00:1288:f00e:1fd::100, 64 hops max, 12 byte packets
1 ge-0-1-0-d2.pat1.ams.yahoo.com 0.594 ms 0.745 ms 0.449 ms
2 30gigabitethernet1-3.core1.ams1.he.net 7.836 ms 11.186 ms 0.621 ms
3 10gigabitethernet1-4.core1.lon1.he.net 7.235 ms 7.693 ms 7.537 ms
4 10gigabitethernet10-4.core1.nyc4.he.net 85.336 ms 75.730 ms 74.506 ms
5 as7018-att.10gigabitethernet2-3.core1.nyc4.he.net 84.458 ms 84.046
ms 84.470 ms
6 n54ny22crs.ipv6.att.net 108.346 ms 109.709 ms 106.708 ms
7 wswdc22crs.ipv6.att.net 109.593 ms 109.116 ms 109.738 ms
8 attga21crs.ipv6.att.net 107.695 ms 104.384 ms 108.077 ms
9 attga409me3.ipv6.att.net 104.221 ms 109.647 ms 104.790 ms
10 2001:1890:c00:8802::11cc:6c6d 105.524 ms 105.763 ms 105.345 ms

Amsterdam->London->NY->DC->Atlanta

Possibly just a transient issue?

Matt

Looks OK from here:

(From 2607:4B00::/32 - SF Bay Area, Native):

mushroom#traceroute www.att.net
Translating "www.att.net"...domain server (216.139.32.37) [OK]

Type escape sequence to abort.
Tracing the route to www.att.net (2001:1890:1C01:2::40)

  1 2001:550:2:C::2:1 0 msec 4 msec 4 msec
  2 2001:550::142 40 msec
    2001:550::113 52 msec
    2001:550::142 48 msec
  3 2001:550::1040 40 msec
    2001:550::1041 48 msec
    2001:550::1040 48 msec
  4 2001:550:4::4C 44 msec 36 msec 44 msec
  5 2001:550:3::7A 40 msec 40 msec 44 msec
  6 sffca22crs.ipv6.att.net (2001:1890:FF:FFFF:12:122:86:122) 112 msec
108 msec 112 msec
  7 la2ca22crs.ipv6.att.net (2001:1890:FF:FFFF:12:122:31:133) 104 msec
112 msec 108 msec
  8 dlstx22crs.ipv6.att.net (2001:1890:FF:FFFF:12:122:28:177) 108 msec
104 msec 112 msec
  9 attga21crs.ipv6.att.net (2001:1890:FF:FFFF:12:122:28:173) 108 msec
100 msec 108 msec
10 attga409me3.ipv6.att.net (2001:1890:FF:FFFF:12:122:120:244) 112 msec
124 msec 100 msec
11 *
    2001:1890:C00:8802::11CC:6C6D 108 msec 104 msec
12 * * *
13 * * *
14 * * *

-Robert

David Hill writes:
> Anyone else noticing odd ipv6 traceroutes to www.att.net
> (2001:1890:1c01:2::40)?
>

David,

We see what's going on. It currently affects only a portion of the v6
Internet. Working on a fix...

            Jay B.

Jay -

Any update?

David