RE: Qwest outage In NY

Qwest has confirmed a DOS attach against two of their Juniper routers in the
NY POP. I believe they had a UDP attack last week also (maybe on Saturday).
This time the DOS was a TCP attack on the 100Mb management interface on the
Juniper, leaving the box unable to pass packets, hence BGP stays up and a
full routing table but you cannot get anywhere.

Patrick

We saw the same issue on Saturday morning too (eastern time). BGP sessions
to Qwest stayed up, but no data was moving across Qwest links. Kind of
ugly.

Emails to Qwest seeking information on Saturday went unanswered.

Drew Linsalata
The Gotham Bus Company
Internet Server and Carrier Neutral Co-Location
http://www.gothambus.com

Ok I'll bite... What crackpipe are you smoking from?

If the link from the RE to the PFE (the fxp1) became saturated, or enough
packets hit the RE to blow away the processor, BGP (and the CLI, and
everything else) would certainly fall over.

Much like with any other router using distributed forwarding, if the
management processor dies, the traffic will continue to forward until the
routing protocols timed out and the rest of the network stopped sending it
traffic. The attack would then stop hitting the box in question, it would
come back up, and the cycle would repeat. This assumes that there are
actual routing protocols, in the case where it's statically routed the
box just stays down. :slight_smile:

But Juniper is more resilient to this form of attack than most, and you
have the ability to filter packets going to the RE on any IP rev.

Let me clarify, our directly connected Qwest router was not under DOS attack
so BGP stayed up and we had a full routing table. The router that got hosed
was 3 router hops into their backbone and it was definitely hosed good. :slight_smile:

Sounds like another point for the whole argument for separate management
networks and separation of the device control plane from the route
processor/packet forwarding plane. Nice to know some of the big guys
don't quite do it either :wink:

jms

Qwest has confirmed a DOS attach against two of their Juniper routers in the
NY POP. I believe they had a UDP attack last week also (maybe on Saturday).
This time the DOS was a TCP attack on the 100Mb management interface on the
Juniper, leaving the box unable to pass packets, hence BGP stays up and a
full routing table but you cannot get anywhere.

The story I just got from Qwest (from a NOCie who was reading from
their ticket, so take this with a grain of salt) made it sound like
that were flooded with bogus routes from some BGP peer. I tend to
believe what you wrote above though. I mean, getting a bunch of bogus
routes via a BGP peer doesn't seem like the kind of thing where you'd
call the vendor onsite (several Qwest NOC'ies stated that Juniper was onsite)
whereas a large-scale DOS might... Anyways, that's the scoop that I've got

/me returns to lurking

Eric :slight_smile:

We have been working an issue with Qwest for the past two months where they
simple black hole all our traffic for no known reason. We had an escalation
procedure to get directly to the Ops Eng group when this event started this
morning as we are still trying to find out what causes it in the past.
Today's event had the very same symptoms as before but one router hop
further into the network from the past 2 times it happened. Below is what
the Ops Eng guy told us happened (very reluctantly).