cymru down?

Mailman List Mirror (Read Only)
1

Unable to geto to www.cymru.com and 68.22.187.24 has been down for 5+ hours. Known issue?

2

I think Ameritech/SBC/AT&T has problems here in Chicago. We have had
problems getting to several sites here. Traces across Ameritech's
network die in some spots. We are trying to get the issues resolved,
but Ameritech's people are...less than clueful.

Sargon

3

www.cymru.com resolves to 68.22.187.27 which is reachable from AS1103.

Roy

4

Getting hung up in savvis - can't ping through -

  1. v140.core1.irv.intelenet.net
  2. v8-ge4-1.border4.irv.intelenet.net
  3. so-6-0-0.ar3.LAX1.gblx.net
  4. so6-0-0-2488M.ar2.PAO2.gblx.net
  5. bpr1-so-2-0-0.PaloAltoPaix.savvis.net
  6. dcr2-so-3-3-0.SanFranciscosfo.savvis.net
  7. dcr1-so-5-0-0.SanFranciscosfo.savvis.net
  8. dcr2-so-0-0-0.Denver.savvis.net
  9. dcr1-so-7-0-0.Chicago.savvis.net
10. acr2-so-1-0-0.Chicago.savvis.net
11. acr2-so-0-0-0.Chicago.savvis.net
12. s228110-1.savvis-internet.uschcg1-bsn.savvis.net
13. ???

Sargon wrote:

5

Odd - I am seeing my route die here (from AS7949):

1 router-sf-t3sf.wmis.net (69.39.94.5) 4 msec 8 msec 4 msec
2 69.39.68.169 [AS 12129] 4 msec 12 msec 4 msec
3 edge-1.sfld-mi.123.net (216.234.96.2) [AS 12129] 4 msec 4 msec 4 msec
4 POS2-1.GW4.DET5.ALTER.NET (65.206.123.245) [AS 701] 8 msec 8 msec 4 msec
5 500.at-6-3-0.CL2.DET5.ALTER.NET (152.63.27.106) [AS 701] 8 msec 8 msec 4 msec
6 0.so-5-0-0.XL2.BOS4.ALTER.NET (152.63.16.126) [AS 701] 40 msec 40 msec 36 msec
7 0.so-7-0-0.XR2.BOS4.ALTER.NET (152.63.16.134) [AS 701] 48 msec 36 msec 40 msec
8 178.at-3-0-0.HR1.BOS4.ALTER.NET (152.63.21.245) [AS 701] 36 msec 40 msec 36 msec
9 0.so-0-2-0.AR1.BOS24.ALTER.NET (152.63.3.14) [AS 701] 36 msec 40 msec 36 msec
10 * * *

My BGP session to cymru has been down 5hs33min

6

just to through more fuel on the fire, from a 701 perspective it hops
though savvis and dies here:

11 acr2-so-1-0-0.Chicago.savvis.net (208.172.3.86) 25.014 ms 24.956 ms
25.833 ms
12 s228110-1.savvis-internet.uschcg1-bsn.savvis.net (64.240.84.82)
30.395 ms 29.377 ms 29.371 ms
13 *^C

I think this is the upstream device to the 'customer' network... perhaps
firewall problems? or a local host problem. Generally they are pretty
quick to fix these sorts of things.

7

Maybe its just _scary_ Halloween routing?

It works from my view of SBC/Ameritech... even though it looks like its going through a Cisco Systems customer connection.

  6 ex1-g9-0-0.pxatga.sbcglobal.net (151.164.249.9) 17.078 ms 16.964 ms 16.89
1 ms
  7 bb2-p3-0.atlnga.sbcglobal.net (151.164.42.14) 17.586 ms 75.782 ms 17.365
ms
  8 bb1-p14-0.atlnga.sbcglobal.net (151.164.189.21) 17.599 ms 17.469 ms 17.37
0 ms
  9 core1-p5-0.cratga.sbcglobal.net (151.164.40.230) 17.232 ms 17.243 ms 17.2
13 ms
10 core3-p3-0.crchil.sbcglobal.net (151.164.241.122) 31.975 ms 31.739 ms 31.
704 ms
11 core1-p9-0.crchil.sbcglobal.net (151.164.243.18) 32.315 ms 32.086 ms 31.9
77 ms
12 bb1-p6-0.chcgil.ameritech.net (151.164.42.184) 66.854 ms 33.085 ms 110.77
5 ms
13 ded2-g1-3-0.chcgil.ameritech.net (151.164.241.113) 33.132 ms 64.020 ms 33
.133 ms
14 Cisco-Systems-1054141.cust-rtr.ameritech.net (65.42.139.42) 41.344 ms 37.3
96 ms 37.158 ms

Happy Halloween,

Deepak

Christopher L. Morrow wrote:

8

So, how many people from nanog DOES it take to troubleshoot a simple
network outage anyways? :slight_smile:

With each passing day I find myself wishing more and more that there was a
Cisco Certified Traceroute Engineer program, and a convenient system to
ignore emails with traceroutes from anyone else.

9

Hi, NANOGers.

] Unable to geto to www.cymru.com and 68.22.187.24 has been down
] for 5+ hours. Known issue?

We brought up a new BGP peer today, oddly enough to add additional
redundancy and improved performance. :slight_smile: The turn-up appears to
have gone awry somewhere in the provider's network, and we're
debugging that now. The primary site has been online the entire
time, but the routing oddness has caused everyone a bit of pain.

We are also seeing some Chicago-based oddness that others are
enduring. We're pretty sure it has nothing to do with one of our
members, an ardent Cubs fan, cursing the Sox. :wink:

We continue to debug it with our peers. Stay tuned!

Apologies for the inconvenience.

Thanks,
Rob.

10

Hi, NANOGers.

] Maybe its just _scary_ Halloween routing?

It's definitely Halloween, but this isn't as scary as it looks. :slight_smile:

] 14 Cisco-Systems-1054141.cust-rtr.ameritech.net...

Back when we were part of Cisco, SBC named our link to them
accordingly. Now that we're on our own, I should probably ask
them to change that.

Thanks,
Rob.

11

ah-ha! me thinks someone is in the midst of a new customer turnup, though
it's back to replying to tcp/80 requests via savvis now.

12

Hi, NANOGers.

Just a quick interruption in your day to update you on the routing
fun we had. :slight_smile: The problem was two-fold:

   1. An errant provider router that randomly selected which
      prefixes to propagate. Mix that with uRPF, and you
      have FUN FUN FUN! :slight_smile:

   2. An interesting use of LOCAL_PREF by the provider, now
      mitigated.

Our tests from multiple external pods and route servers show that
we're globally visible again. If you're having any difficulty
reaching us, please let us know at <team-cymru@cymru.com>.

Again we apologize for the inconvenience!

Thanks,
Rob, for Team Cymru.