We just had a qwest outage of about 2 mins at 1:41am pst. When I called to report it I was told it was a 200+ emergency software upgrade due to a security concern, and that we will get a notice later after the fact. Normally we get notices in advance, even for software upgrades due to security or other important issues, so I am curious if other qwest customers had the same experience and wether this is how it's going to be from here on in? The affected platform was juniper and I'd love to know the specfic case being addressed here.
I'm located in the NW (Talent, Oregon, just over CA border..) and we have a few customers on Qwest T1's and the like. We also have a customer who gets MPLS directly from Q.
We've yet to hear of any outages for our customers - but I suppose the night is still young...
We just had a qwest outage of about 2 mins at 1:41am pst. When I called
to report it I was told it was a 200+ emergency software upgrade due to
a security concern, and that we will get a notice later after the fact.
Normally we get notices in advance, even for software upgrades due to
security or other important issues, so I am curious if other qwest
customers had the same experience and wether this is how it's going to
be from here on in? The affected platform was juniper and I'd love to
know the specfic case being addressed here.
We received 7 Juniper Security Advisories today. My guess is that this
is the reason for the Qwest outage you've seen.
Give the nature of the issue from juniper I am guessing that a large number of companies were doing upgrades over the last few days as fast as they could. http://j.mp/8XaReK has the details I know of now.
We just had a qwest outage of about 2 mins at 1:41am pst. When I called
to report it I was told it was a 200+ emergency software upgrade due to
a security concern, and that we will get a notice later after the fact.
Normally we get notices in advance, even for software upgrades due to
security or other important issues, so I am curious if other qwest
customers had the same experience and wether this is how it's going to
be from here on in? The affected platform was juniper and I'd love to
know the specfic case being addressed here.
I got 3 notices about the outage related to our 1 Qwest OC-3.
As for the Juniper security issues, see juniper-nsp archives.
We experienced the outage, but so far have not received any notifications.
Steve
- --
- ---------------------------------------------------------------
Steven Saner <ssaner@hubris.net>
Director of Network Operations
Hubris Communications
Yeah, they refused to notify due to security concerns from what they told me last night. Notification was performed after maintenance was complete.
Ok, so the next question is, what harm would a simple advance notice of 'emergency maintenance' caused, vs the very real hassle and inconvenience that DS3-down in the middle of the night caused for operations staff? I personally was yanked out of bed by my network monitoring systems and had to spend at least 5 minutes looking at logs and perf monitors and so forth so I had enough information before making the call to qwest support, thinking we took a hit on our fiber. Some others reported getting 7 or more hours of advance notice - this would have been enough for me, even tho of course I don't like it - and I suspect it could have saved many many others from a similar fate of responding to the event so late.
Just saying "we need to do this", does not give the bad guys any ammo with which to attack, and would go a long ways twords preventing needless heroics such as middle of the night investigations by senior staff. The follow up email we subsequently received was fine and we appreciated learning it really was a necessary upgrade and seems justified now with that knowledge, I would simply have appreciated not having to engage my emergency processes for something that was planned.
We just had a qwest outage of about 2 mins at 1:41am pst. When I called
to report it I was told it was a 200+ emergency software upgrade due to
a security concern, and that we will get a notice later after the fact.
Hmm - I got notice in advance. I'll have to go search for the email, but it
was sometime before Christmas, since I had it on my calendar.
This is one reason why companies should use twitter...great for those
impromptu emergency messages. Our electrical utility company uses
twitter and I have to admit that it's nice to know what is going on
ahead of time even if it is a short message!
Same thing for us in Minnesota. Brief outage and emergency outage notification came after the outage. The outage window was for 6:00-12:00GMT, and the outage came at 6:15GMT. We didn't get the notice until 10:30GMT.
Qwest had a major outage over the Xmas weekend in MN. I wonder if this is related. They told me it was a bad switch, but that sounded funny.
We just had a qwest outage of about 2 mins at 1:41am pst. When I called to
report it I was told it was a 200+ emergency software upgrade due to a
security concern, and that we will get a notice later after the fact.
That's not a maintenance, that's an outage.
I hope everybody impacted on this list is claiming SLA.
Qwest NEVER EVER provides SLA adjustments, no longer how long it's down or what their own role in it being down is. They toss it from department to department and then hand down judgments that 'we're not providing credits' and that's that. So in the three years we've had qwest we've had probably 10 major service impacting failures and at least 3 over 6 hours each, and a 9 month period where the (ssshhh!! secret!!) mpls tunnel they put our 45mbs 'clear channel ds3' on was oversubscribed with another user, resulting on 600+ms latencies at random times and they refused to accept our mrtg and smokeping traces showing the problem. No adjustments at all, just pay up, and oh we'll get around to that someday when it suits us but you have nothing to say about it that we care about.
If they honored every SLA adjustment they would not be able to pay the current stockholders a 6.8% yield!
At 20.63% over their 200day and 12.6% over their 50day moving average their stock is only 4.52% below
their 52 week high, and about 50% down from its 2007 high, makes their stock a better investment than
their service. Well that is assuming you do not look back to 2000 when it was a $50 stock before the roller
coaster ride to a low of a little over $2 in oct 2009 and is going back up again due to that extra little bit of
cash they can squeeze for those damb customers. If it was not for the customers, this would be a good
business investment.