Anyone know whats up ? I have seen some strange routing depending on the payload's protocol to a site in one of their colos in Toronto.
e.g. all tcp packets are making to the host
marble# traceroute -s 199.212.134.2 -Ptcp ITMsite.sentex.ca
traceroute to ITMsite.sentex.ca (209.167.35.xx), 64 hops max, 56 byte packets
1 iolite4-fxp2 (199.212.134.10) 0.124 ms 0.101 ms 0.097 ms
2 allstream-vl108 (67.43.129.246) 5.663 ms 5.443 ms 5.580 ms
3 216.191.68.65 (216.191.68.65) 6.997 ms 6.050 ms 6.667 ms
4 65.195.241.149 (65.195.241.149) 17.438 ms 17.448 ms 18.513 ms
5 0.so-1-0-0.XL2.CHI1.ALTER.NET (152.63.70.70) 18.040 ms 17.950 ms 18.198 ms
6 0.so-0-0-0.TL2.CHI2.ALTER.NET (152.63.68.89) 18.476 ms 18.442 ms 20.224 ms
7 0.so-6-0-0.TL2.TOR2.ALTER.NET (152.63.0.201) 38.112 ms 34.369 ms 34.827 ms
8 0.so-1-0-0.XL2.TOR2.ALTER.NET (152.63.3.90) 37.353 ms 37.305 ms 36.856 ms
9 POS7-0.GW4.TOR2.ALTER.NET (152.63.131.141) 40.099 ms 36.807 ms 35.595 ms
10 enoo3-gw.customer.alter.net (205.150.100.214) 43.112 ms 41.565 ms 41.931 ms
^C
But other protocols (UDP, ICMP and most importantly for us, ESP) are hosed.
marble# traceroute -s 199.212.134.2 -Pesp ITMsite.sentex.ca
traceroute to ITMsite.sentex.ca (209.167.35.xx), 64 hops max, 40 byte packets
1 iolite4-fxp2 (199.212.134.10) 0.120 ms 0.097 ms 0.093 ms
2 allstream-vl108 (67.43.129.246) 5.460 ms 5.303 ms 5.524 ms
3 216.191.68.65 (216.191.68.65) 6.563 ms 6.289 ms 6.211 ms
4 65.195.241.149 (65.195.241.149) 18.219 ms 17.612 ms 17.991 ms
5 0.so-1-0-0.XL2.CHI1.ALTER.NET (152.63.70.70) 17.587 ms 17.629 ms 18.492 ms
6 0.so-0-0-0.TL2.CHI4.ALTER.NET (152.63.13.34) 26.075 ms 18.130 ms 18.879 ms
7 0.so-0-0-0.XL2.CHI4.ALTER.NET (152.63.13.33) 17.981 ms 18.160 ms 18.516 ms
8 0.so-0-0-0.TL2.CHI4.ALTER.NET (152.63.13.34) 19.138 ms 18.356 ms 18.593 ms
9 0.so-0-0-0.XL2.CHI4.ALTER.NET (152.63.13.33) 18.677 ms 18.477 ms 17.973 ms
10 0.so-0-0-0.TL2.CHI4.ALTER.NET (152.63.13.34) 19.088 ms 18.936 ms 18.692 ms
11 0.so-0-0-0.XL2.CHI4.ALTER.NET (152.63.13.33) 18.841 ms 18.308 ms 18.996 ms
12 0.so-0-0-0.TL2.CHI4.ALTER.NET (152.63.13.34) 18.362 ms 18.645 ms 18.584 ms
13 0.so-0-0-0.XL2.CHI4.ALTER.NET (152.63.13.33) 18.329 ms 18.125 ms 18.890 ms
14 0.so-0-0-0.TL2.CHI4.ALTER.NET (152.63.13.34) 18.583 ms 18.635 ms 18.933 ms
15 0.so-0-0-0.XL2.CHI4.ALTER.NET (152.63.13.33) 46.420 ms 19.088 ms 18.217 ms
16 0.so-0-0-0.TL2.CHI4.ALTER.NET (152.63.13.34) 62.137 ms 18.515 ms 18.349 ms
17 0.so-0-0-0.XL2.CHI4.ALTER.NET (152.63.13.33) 18.243 ms 18.558 ms 18.404 ms
18 0.so-0-0-0.TL2.CHI4.ALTER.NET (152.63.13.34) 18.506 ms 18.568 ms 19.341 ms
19 0.so-0-0-0.XL2.CHI4.ALTER.NET (152.63.13.33) 18.724 ms 19.178 ms 18.367 ms
20 0.so-0-0-0.TL2.CHI4.ALTER.NET (152.63.13.34) 18.644 ms 18.321 ms 18.545 ms
21 0.so-0-0-0.XL2.CHI4.ALTER.NET (152.63.13.33) 19.225 ms 18.410 ms 18.596 ms
22 0.so-0-0-0.TL2.CHI4.ALTER.NET (152.63.13.34) 18.758 ms 20.040 ms 18.488 ms
---Mike
Don't know if it is related, but we can't route email to bellsouth.net -- no
route to host. When I checked it rattles around inside their network
(assuming traceroute is trustworthy).
Their customer support (first number I found) report they have a ticket
already open. They know they have a problem, but didn't supply any details, I
didn't ask. I just wanted to know it wasn't specific to our own address
space, and they knew they had a problem, it isn't, they do.
Snip first few hops.
9 dhr4-pos-8-0.Weehawkennj2.savvis.net (204.70.1.6) 87.029 ms 78.880 ms
93.625 ms
10 0.ge-5-1-0.XL4.NYC4.ALTER.NET (152.63.3.121) 80.281 ms 76.763 ms 85.844
ms
11 0.so-6-0-0.XL2.ATL5.ALTER.NET (152.63.10.105) 111.373 ms 110.133 ms
106.262 ms
12 0.so-7-0-0.GW13.ATL5.ALTER.NET (152.63.84.109) 100.764 ms 114.852 ms
107.897 ms
13 bellsouth-atl5-gw.customer.alter.net (157.130.71.170) 103.411 ms 102.371
ms 95.479 ms
14 axr00asm-1-0-0.bellsouth.net (65.83.236.3) 95.399 ms 99.486 ms 99.574
ms
15 ixc00asm-4-0.bellsouth.net (65.83.237.1) 96.381 ms 96.492 ms 101.427 ms
16 acs01asm.asm.bellsouth.net (205.152.37.66) 109.901 ms 118.113 ms
111.783 ms
17 axr01asm-1-3-1.bellsouth.net (65.83.237.6) 96.847 ms 102.503 ms 96.839
ms
18 65.83.238.40 (65.83.238.40) 96.554 ms 96.587 ms 96.480 ms
19 65.83.238.37 (65.83.238.37) 110.986 ms 110.987 ms 118.938 ms
20 65.83.239.29 (65.83.239.29) 110.261 ms 115.079 ms 110.075 ms
21 65.83.239.102 (65.83.239.102) 110.098 ms 110.116 ms 110.158 ms
22 205.152.45.65 (205.152.45.65) 110.158 ms 110.263 ms 110.142 ms
23 205.152.161.63 (205.152.161.63) 109.775 ms 110.116 ms 115.274 ms
24 205.152.161.65 (205.152.161.65) 110.571 ms 110.576 ms 110.512 ms
25 205.152.161.48 (205.152.161.48) 118.745 ms 116.560 ms 112.485 ms
26 * * *
27 205.152.156.25 (205.152.156.25) 128.332 ms 128.274 ms 133.991 ms
28 205.152.161.49 (205.152.161.49) 132.822 ms 141.624 ms 143.536 ms
29 205.152.161.64 (205.152.161.64) 142.537 ms 131.862 ms 132.765 ms
30 205.152.161.62 (205.152.161.62) 132.566 ms 131.734 ms 134.176 ms
Hi,
Thanks for replying, it doesnt seem to be related based on the traceroute you provided. I am still seeing the issue this morning, but its not so wide spread across protocols. I worked around the issue by dumping out my traffic to them via Teleglobe instead of MTSAllstream. Teleglobe seems to talk to them on a different router in Chicago which doesnt seem to be showing the behaviour as much as the other router or at least its not on the source IP addresses I am using. However, its stil blocking IPSEC packets
I got an email from the datacenter early this AM saying the problem was fixed according to UUnet, but it still there when I just tried now. Is this the result of some protocol prioritizing config gone bad ? Or busted software ?
e.g as of this morning, icmp,tcp, and udp packets make it to the other side but not IPSEC
traceroute -s 67.43.129.246 -Pudp 209.167.35
traceroute to 209.167.35.xx (209.167.35.xx) from 67.43.129.246, 64 hops max, 40 byte packets
1 216.191.68.65 (216.191.68.65) 0.877 ms 0.708 ms 0.786 ms
2 65.195.241.149 (65.195.241.149) 12.225 ms 12.262 ms 12.239 ms
3 0.so-1-0-0.XL1.CHI1.ALTER.NET (152.63.70.78) 12.524 ms 12.424 ms 12.443 ms
4 0.so-0-0-0.TL1.CHI2.ALTER.NET (152.63.68.82) 12.870 ms 12.832 ms 12.940 ms
5 0.so-3-0-0.TL1.TOR2.ALTER.NET (152.63.2.85) 28.846 ms 28.633 ms 28.739 ms
6 0.so-1-0-0.XL1.TOR2.ALTER.NET (152.63.3.82) 28.886 ms 74.966 ms 28.789 ms
7 POS6-1.GW4.TOR2.ALTER.NET (152.63.131.129) 28.888 ms 28.691 ms 28.605 ms
8 enoo3-gw.customer.alter.net (205.150.100.214) 34.739 ms 34.998 ms 34.395 ms
traceroute -s 67.43.129.246 -Pesp 209.167.35.xx
traceroute to 209.167.35.xx (209.167.35.xx) from 67.43.129.246, 64 hops max, 36 byte packets
1 216.191.68.65 (216.191.68.65) 0.918 ms 0.724 ms 0.712 ms
2 65.195.241.149 (65.195.241.149) 12.244 ms 12.311 ms 12.297 ms
3 0.so-1-0-0.XL1.CHI1.ALTER.NET (152.63.70.78) 12.361 ms 12.456 ms 12.404 ms
4 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 12.792 ms 12.784 ms 12.751 ms
5 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 13.356 ms 13.405 ms 13.489 ms
6 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 13.208 ms 13.113 ms 13.131 ms
7 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 13.820 ms 13.833 ms 13.903 ms
8 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 13.666 ms 13.567 ms 13.733 ms
9 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 14.308 ms 14.316 ms 14.345 ms
10 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 14.018 ms 14.113 ms 14.073 ms
11 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 14.807 ms 14.824 ms 14.943 ms
12 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 14.558 ms 14.611 ms 14.595 ms
13 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 15.327 ms 15.335 ms 15.174 ms
14 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 15.031 ms 14.948 ms 15.148 ms
15 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 15.680 ms 15.678 ms 15.688 ms
16 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 15.510 ms 15.498 ms 15.534 ms
17 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 16.189 ms 16.239 ms 16.167 ms
18 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 15.964 ms 15.855 ms 15.924 ms
19 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 16.688 ms 16.629 ms 16.753 ms
20 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 16.385 ms 16.464 ms 16.473 ms
21 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 17.099 ms 17.034 ms 17.134 ms
22 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 77.096 ms 16.821 ms 16.875 ms
23 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 17.488 ms 17.474 ms 17.474 ms
24 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 17.310 ms 17.348 ms 17.323 ms
25 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 18.018 ms 18.137 ms 17.953 ms
26 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 17.838 ms 17.740 ms 17.820 ms
27 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 18.595 ms 18.466 ms 18.484 ms
28 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 18.220 ms 18.256 ms 18.349 ms
29 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 48.198 ms 18.987 ms 18.877 ms
30 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 18.669 ms 18.709 ms 18.769 ms
31 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 19.361 ms 19.565 ms 19.442 ms
32 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 19.121 ms 19.158 ms 19.114 ms
33 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 19.780 ms 20.010 ms 19.907 ms
34 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 19.610 ms 19.614 ms 19.587 ms
35 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 20.329 ms 20.239 ms 20.269 ms
36 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 20.119 ms 20.179 ms 20.076 ms
37 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 20.906 ms 20.700 ms 20.985 ms
38 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 20.487 ms 21.070 ms 20.650 ms
39 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 21.259 ms 21.198 ms 21.244 ms
40 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 21.035 ms 20.946 ms 21.008 ms
41 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 21.699 ms 21.767 ms 21.733 ms
42 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 21.418 ms 21.589 ms 21.425 ms
43 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 22.177 ms 22.190 ms 22.213 ms
44 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 22.157 ms 22.063 ms 21.864 ms
45 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 22.620 ms 22.618 ms 22.570 ms
46 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 22.454 ms 22.307 ms 22.383 ms
47 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 23.049 ms 23.064 ms 23.056 ms
48 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 22.802 ms 27.152 ms 22.712 ms
49 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 23.585 ms 23.603 ms 23.435 ms
50 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 23.478 ms 23.300 ms 23.317 ms
51 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 23.946 ms 23.967 ms 24.061 ms
52 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 23.802 ms 23.804 ms 23.820 ms
53 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 24.434 ms 24.392 ms 24.532 ms
54 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 48.177 ms 24.157 ms 24.267 ms
55 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 24.951 ms 25.016 ms 24.947 ms
56 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 24.734 ms 62.234 ms 24.696 ms
57 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 25.405 ms 25.378 ms 25.452 ms
58 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 24.992 ms 25.122 ms 25.256 ms
59 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 25.831 ms 25.818 ms 25.812 ms
60 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 25.638 ms 25.676 ms 25.636 ms
61 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 26.440 ms 26.350 ms 26.377 ms
62 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 26.006 ms 26.136 ms 26.180 ms
63 0.so-7-0-0.XL1.CHI6.ALTER.NET (152.63.65.161) 26.902 ms 26.770 ms 26.734 ms
64 0.so-0-0-0.TL1.CHI4.ALTER.NET (152.63.13.26) 26.506 ms 26.525 ms 26.494 ms
Sorry - my mistake
Saw the 205.150 prefix and confused in with 205.152, which are totally
different of course.
bellsouth.net have sorted their issue (from our perspective).