One router and it takes there entire network off-line... Maybe someone needs
a Intro to Networks 101 class.
-jim
Well, if the memory errors corrupts the forwarding table placed on the
line cards or something similar, and still keeps its adjacancies up, then
you can get these problems. I've seen it happen on route-cache boxes where
certain entries in the ip-forwarding table was corrupted and thus
incorrectly routed.
It could be that they ran out of memory on linecards as well, perhaps
injected too many routes etc, and lost dCEF (dunno if the problems was on
gsr or juniper), been there, done that.
I've seen a case where a single error in the
configuration file of a $VENDOR_1 router was accepted
(due to an 'undocumented feature'), and this caused
the wholesale importation of BGP routes into the IGP,
which caused most of their $VENDOR_2 hardware to spaz
out. Locating the single error was a matter of hours,
not minutes, so effectively a typo took out that ISP -
and it's considered by most to be a relatively
well-designed network.
-David Barak
No matter what kind of technology or design you have there are always
kinds of faults which may bring the entire system down. The problem is
generally in recognizing when a fault has occured, so the the operation
may be switched over to a backup.
Particularly, the present Internet routing architecture is (mis)designed
in such a way that it is incredibly easy for a local fault or human error
to bring a significant portion of the network down. Even single-box
_hardware_ faults may lead to global crashes.
Long long time ago I had to track down a problem which made US and EU
pretty much disconnected for several hours. This turned out to be a
hardware problem in 7000's SSE card, which happily worked with packets
originating and terminating in the router itself, but silently dropped all
transit packets. Voila! Neighbour boxes were convinced that this one's
working - because all routing protocols were happy, and were trying to
send lots of traffic through it, which was simply going to a blackhole to
the mighty annoyance of everyone. I've got a speeding ticket showing over
100mph on Dulles hwy at 3am, too, as a memento of rushing to DC with a
spare card...
So, in the absense of details, I would reserve judgement on soundness of
design practices.
--vadim
I've seen a case where a single error in the
configuration file of a $VENDOR_1 router was accepted
(due to an 'undocumented feature'), and this caused
the wholesale importation of BGP routes into the IGP,
which caused most of their $VENDOR_2 hardware to spaz
out. Locating the single error was a matter of hours,
not minutes, so effectively a typo took out that ISP -
and it's considered by most to be a relatively
well-designed network.
I have also seen a variation of this where the boxes which got flooded
by large IGP tables run out of memory and not recovering (because there
was no memory left) after the broken router was withdrawn. Eventually
the network got fixed by restarting every box in succession.
Not sure if there are safeguards for this now or if everybody buys all their
IGP routers with 512M or more.
Pete