I’m in Louisiana and just lost my OC12 to Bwing/L3. Circuit didn’t die, actually received a BGP message to terminate the session.
Anyone else seeing anything or got an update? ALL the numbers I have to L3 are busy…
Seeing the same exact thing in Newark, DE.
Ross
on 2007-09-19 13:25 W. Kevin Hunt said the following:
I�m in Louisiana and just lost my OC12 to Bwing/L3. Circuit didn�t die, actually received a BGP message to terminate the session.
Anyone else seeing anything or got an update? ALL the numbers I have to L3 are busy...
Same same for our client who's co-loed in the Bwing facility in NYC. Traceroutes die:
18 p6-0.c0.ftwo.broadwing.net (216.140.17.53) 161.712 ms 158.098 ms 158.154 ms
19 p4-0.c0.atln.broadwing.net (216.140.17.114) 174.425 ms 178.960 ms 177.797 ms
20 p3-0.c0.wash.broadwing.net (216.140.8.109) 188.125 ms 187.199 ms 188.288 ms
21 p0-2-0.a1.wash.broadwing.net (216.140.8.90) 187.539 ms 190.421 ms 185.878 ms
different point A, same point Z:
23 p4-0.c0.atln.broadwing.net (216.140.17.114) 145.159 ms 143.587 ms 145.530 ms
24 p3-0.c0.wash.broadwing.net (216.140.8.109) 159.735 ms 161.113 ms 159.258 ms
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
and all the usual NOC numbers are either busy or fast-busy (!)
It's probably not a total melt-down, as our voice PRIs there are all still OK.
//jbaltz
It just came up back here in SE PA. Last time this happened, the
Broadwing 888 number was fast busy, as it was this time. The Level 3
Corporate HQ numbers were all normal busy, probably just swamped with
calls from people like us.
Traceroute from my Verizon FIOS account initially looked like this:
traceroute to pil.net (207.7.198.3), 30 hops max, 40 byte packets
1 wireless_broadband_router (192.168.1.1) 1.146 ms 0.482 ms 0.464 ms
2 l100.vfttp-40.phlapa.verizon-gni.net (72.94.48.1) 7.87 ms 6.712 ms
7.088 ms
3 mail.pil.net (207.7.198.3) 15.164 ms !H
Then went to this, shortly before coming back up:
traceroute to 207.7.198.3 (207.7.198.3), 30 hops max, 40 byte packets
1 wireless_broadband_router (192.168.1.1) 1.237 ms 0.489 ms 0.436 ms
2 l100.vfttp-40.phlapa.verizon-gni.net (72.94.48.1) 7.528 ms 9.455 ms
19.71 ms
3 p1-1.lcr-04.phlapa.verizon-gni.net (130.81.59.102) 6.606 ms 18.624
ms 6.345 ms
4 130.81.29.214 (130.81.29.214) 9.617 ms 7.354 ms 7.097 ms
5 0.so-6-0-0.xl2.phl6.alter.net (152.63.3.81) 5.876 ms 7.501 ms 6.563
ms
6 0.so-6-0-0.xl4.iad8.alter.net (152.63.0.130) 12.968 ms 11.937 ms
11.486 ms
7 0.ge-7-0-0.br2.iad8.alter.net (152.63.41.157) 27.124 ms 17.941 ms
10.892 ms
8 4.68.127.25 (4.68.127.25) 23.224 ms 11.511 ms 13.252 ms
9 vlan99.csw4.washington1.level3.net (4.68.17.254) 24.319 ms
vlan79.csw2.washington1.level3.net (4.68.17.126) 15.146 ms
vlan99.csw4.washington1.level3.net (4.68.17.254) 21.856 ms
10 ae-64-64.ebr4.washington1.level3.net (4.69.134.177) 16.428 ms
ae-84-84.ebr4.washington1.level3.net (4.69.134.185) 22.225 ms
ae-74-74.ebr4.washington1.level3.net (4.69.134.181) 15.351 ms
11 ae-4.ebr3.losangeles1.level3.net (4.69.132.81) 84.88 ms 84.218 ms
89.179 ms
12 ae-2.ebr3.sanjose1.level3.net (4.69.132.9) 93.175 ms 100.726 ms
91.776 ms
13 ae-83-83.csw3.sanjose1.level3.net (4.69.134.234) 93.029 ms 84.361 ms
98.956 ms
14 ae-33-89.car3.sanjose1.level3.net (4.68.18.133) 90.771 ms 217.911 ms
118.959 ms
15 te-4-1-73.rp0-1.snjsca13.level3.net (4.68.63.6) 85.783 ms * *
16 216.140.3.65 (216.140.3.65) 98.25 ms 96.062 ms 96.013 ms
17 * * *
18 * * *
19 p4-0.c0.atln.broadwing.net (216.140.17.114) 152.745 ms 150.813 ms
151.869 ms
20 * * *
21 p0-2-0.a1.wash.broadwing.net (216.140.8.90) 171.344 ms 170.289 ms
163.337 ms
22 216.140.8.238 (216.140.8.238) 171.446 ms 157.893 ms 162.619 ms
23 66-243-174-57.focaldata.net (66.243.174.57) 175.231 ms 177.866 ms
171.83 ms
24 * * *
25 * * *
James Smallacombe PlantageNet, Inc. CEO and Janitor
up@3.am http://3.am
Same thing in Chicago.
Brian Knoll
Yep - Chicago also.
Looks like a change management Oops. I'm sure they are putting things back as fast as they can.