RE: Majore North Jersey outage...

Still feeling the effects -- this outage was big. 2000 (That's right, TWO
THOUSAND) DS3s were down for 12 hours or more. Alcatel has been on-site for
9 of them.


Was big? IS big... I watched all the circuits into
the ATM cloud come up while ours still languishes...

Alex has the scoop, but I understand there's only 50% or so restored
now. No real good info from anyone on what the cause is. They mumbled
something about fiber and electricity on the support line, Northpoint said
it was a botched upgrade...


It's a DACs upgrade that went awry.

It's been an interesting day.

Service is 90% up, 10% more to go. dacs is up in simplex mode,
no redundancy per-shelf but as a whole traffic should flow with
no effect noticeable by customers.

Engineers will be working throughout the night on the b-side
(redundant) end of each shelf.


Last I heard, ~ 95% of their DACS shelves were restored (albeit in
"simplex" mode), and ~ 80% of their circuits were back up. While most
mission-critical services are now restored, some are not, most notably,
a large hospital's DS3 ATM ports...

More news to follow...