XO Outage

Mailman List Mirror (Read Only)
1

We are seeing some issues w/ XO/Savvis peering..

Trace from XO to Savvis IP space (64.75.10.151)

Keys: Help Display mode Restart statistics Order of fields quit
                                                                                                         Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. scrubbed
                                                                       0.0% 6 0.6 0.5 0.4 0.6 0.1
2. ip65-44-114-97.z114-44-65.customer.algx.net 0.0% 6 1.3 1.3 1.2 1.4 0.1
3. ???

Trace from Savvis to XO IP space (65.44.114.97)

1. scrubbed
                                                                                       0.0% 38 0.4 0.4 0.3 0.5 0.1
2. 64.41.199.129 0.0% 37 1.0 24.0 0.6 330.2 80.4
3. hr1-ge-7-47.santaclarasc5.savvis.net 0.0% 37 0.7 1.4 0.6 27.3 4.4
4. er1-te-1-0-0.sanjose3equinix.savvis.net 0.0% 37 0.7 5.2 0.6 140.3 23.2
5. cr1-tenge-0-7-5-0.sanfrancisco.savvis.net 2.7% 37 2.9 4.0 2.6 16.6 2.5
6. cr2-pos-0-0-3-3.dallas.savvis.net 0.0% 37 42.6 43.1 42.3 51.4 1.4
7. dpr1-ge-4-0-0.dallasequinix.savvis.net 0.0% 37 43.1 44.8 42.9 76.9 6.7
8. er1-te-2-1.dallasequinix.savvis.net 0.0% 37 43.3 49.2 42.8 233.6 31.6
9. 208.175.175.90 0.0% 37 43.0 42.8 42.6 43.6 0.2
10. 65.106.1.102 75.0% 37 43.5 46.5 43.4 62.9 6.3
11. 65.106.1.101 0.0% 37 43.4 47.8 43.2 112.3 12.5
12. 65.106.0.41 0.0% 37 57.5 65.1 57.1 177.3 21.0
13. 65.106.1.73 0.0% 37 57.4 66.5 57.1 162.1 24.2
14. ???

Trying to call into XO and they aren't even taking calls, they mention something about network issues in Spokane. Any ideas as to what is going on/ETA to fix?

--Justin

2

yeah, we noticed it as well. looks like they're back now. i noticed
their routes drop off at about 14:16 PT and come back just after 14:30 PT.

+m

Justin Sharp wrote:

3

Also seeing some issues with XO out here:

Tracing route to yahoo.com [68.180.206.184]
over a maximum of 30 hops:

  1 2 ms 1 ms 1 ms 10.100.20.1
  2 2 ms 1 ms 2 ms sjcisr01-int.wyse.com [10.100.1.15]
  3 3 ms 2 ms 2 ms 132.237.245.1
  4 3 ms 3 ms 2 ms 207.88.3.37.ptr.us.xo.net [207.88.3.37]
  5 7 ms 3 ms 3 ms p3-0-0.mar2.fremont-ca.us.xo.net [207.88.80.181]

  6 16 ms 5 ms 3 ms p4-0-0.rar2.sanjose-ca.us.xo.net [65.106.5.137]

  7 12 ms 14 ms 11 ms p6-0-0.rar1.la-ca.us.xo.net [65.106.0.17]
  8 17 ms 11 ms 11 ms p0-0-0d0.rar2.la-ca.us.xo.net [65.106.1.50]
  9 44 ms 50 ms 52 ms p6-0-0.rar1.dallas-tx.us.xo.net [65.106.0.13]
10 * * * Request timed out.
11 44 ms 44 ms 44 ms 207.88.12.150.ptr.us.xo.net [207.88.12.150]
12 53 ms 85 ms 52 ms 206.111.5.42.ptr.us.xo.net [206.111.5.42]
^C

4

Seems like Savvis/XO routes have been restored, albeit through an unusual Dallas route (usually takes an XO Los Angeles route)..

1. scrubbed
                                                                       0.0% 204 0.4 0.7 0.3 7.0 0.9
2. ip65-44-114-97.z114-44-65.customer.algx.net 0.0% 204 1.8 1.8 1.1 8.4 1.2
3. ip65-46-48-29.z48-46-65.customer.algx.net 0.0% 204 3.3 6.3 2.8 96.4 9.0
4. p6-0-0-0.mar1.saltlake-ut.us.xo.net 0.5% 204 8.1 14.4 3.1 161.0 26.3
5. p4-1-0-0.rar1.denver-co.us.xo.net 0.0% 204 15.5 25.6 15.3 191.4 26.7
6. p0-0-0d0.rar2.denver-co.us.xo.net 0.0% 204 16.2 22.4 16.1 129.2 14.4
7. p1-0-0.rar2.dallas-tx.us.xo.net 0.0% 204 45.3 38.3 29.9 201.3 21.3
8. te-3-4-0.rar3.dallas-tx.us.xo.net 66.0% 204 30.3 34.2 29.6 116.2 11.6
9. 207.88.12.146.ptr.us.xo.net 0.0% 204 33.1 32.7 29.4 106.7 8.5
10. 208.175.175.89 0.0% 204 29.8 35.7 29.5 195.4 20.0
11. dpr1-ge-2-0-0.dallasequinix.savvis.net 0.0% 204 34.3 33.8 29.6 115.2 10.5
12. cr2-tengige0-7-5-0.dallas.savvis.net 1.5% 204 30.2 33.3 29.7 68.6 5.4
13. cr2-loopback.sfo.savvis.net 0.0% 203 75.1 209.8 74.0 3270. 493.7
14. er1-te-2-0-1.sanjose3equinix.savvis.net 0.0% 203 71.8 75.6 71.7 219.5 11.7
15. hr1-te-2-0-0.santaclarasc5.savvis.net 0.0% 203 72.1 75.5 72.0 194.0 9.4
16. 216.34.2.226 0.0% 203 72.3 105.4 72.1 723.7 102.5
17. 64.41.199.140 28.6% 203 77.5 74.9 72.0 183.1 9.7
18. scrubbed
                                                                        0.0% 203 73.8 76.0 72.4 207.9 11.3

Hop 8 looks busy..

--Justin

Mike Lyon wrote:

5

I am seeing it on my end also:

traceroute: Warning: www.cnn.com has multiple addresses; using 157.166.224.25
traceroute to www.cnn.com (157.166.224.25), 64 hops max, 40 byte packets
  1 hq-rtr1.genius.local (64.244.66.1) 0.891 ms 0.429 ms 0.449 ms
  2 ip65-46-253-157.z253-46-65.customer.algx.net (65.46.253.157) 1.856 ms 2.860 ms 1.881 ms
  3 p3-0-0.mar2.fremont-ca.us.xo.net (207.88.80.181) 16.922 ms 2.041 ms 2.013 ms
  4 p4-3-0.rar2.sanjose-ca.us.xo.net (65.106.5.161) 2.637 ms 2.192 ms 2.823 ms
  5 p6-0-0.rar1.la-ca.us.xo.net (65.106.0.17) 10.308 ms 10.258 ms 10.386 ms
  6 207.88.13.22.ptr.us.xo.net (207.88.13.22) 10.931 ms 10.535 ms 10.037 ms
  7 *^C

Justin Sharp wrote: