Whoops! 2 hours to find routers w/o an IGP tsk tsk.
Dear AT&T IP Services Customer,
Please be advised of the following:
I think it's worth full credit that they actually say what was wrong and
why it took so long to fix it. A lot of other companies would mumbo-jumbo
a human error like this in order to not lose face.
Massive stupidity shouldn't win you points just because you admit to it
(note that I consider the *massive* stupidity part to start at the
inability to troubleshoot the problem, not the failure in change
management).
But to be fair, when a NOC or Customer Service person writes an RFO, it
usually comes out sounding very very bad, regardless of the actual
complexity of the situation. I'm certain all the AT&T customers hope that
was the case here. 
Massive stupidity shouldn't win you points just because you admit to it
(note that I consider the *massive* stupidity part to start at the
inability to troubleshoot the problem, not the failure in change
management).
I don't think anyone can point to massive stupidity w/o knowing exactly
what happened and the exact steps taken to fix it. Even though the RFO
note indicated the "problem" and the "solution", my guess is that things
were a bit more complicated than finding a couple config statements,
putting them back in and watching the network magically start working
again.
Of course that is just my opinion, I could be wrong 
But to be fair, when a NOC or Customer Service person writes an RFO, it
usually comes out sounding very very bad, regardless of the actual
complexity of the situation. I'm certain all the AT&T customers hope that
was the case here.
Agreed, and sometimes the RFO tends to be a bit simplistic compared to the
actual trouble.
-sean
Spoon!
It would also be interesting to know which backbone/core product requires a reboot to
activate OSPF configuration changes. Sounds like something one should stay away from.
Pete
Frank Scalzo wrote:
This is impressive. It's very nice to see a carrier providing this level of
technical analysis to customers after an outage. Many carriers would be
embaressed or try to gloss over what has happened. Sprintlink, in the old
days, was also very good about this. Customers really appreciate honesty and
being kept in the loop on these sorts of things.
Sure, this was a dumb mistake, but everyone has made mistakes - no carrier
or engineer is perfect. I think AT&T gets big points on this one. It's a sad
statement about our industry that being honest with your customers is the
mark of outstanding customer service, rather than average customer service.
- Daniel Golding
Has anybody mentioned the benefits of ISIS as an IGP to them.
Kesva
:
:Has anybody mentioned the benefits of ISIS as an IGP to them.
:
Of course, ISIS is no more resilient against the deletion of igp
configuration than OSPF.
cheers,
brian
Has anybody mentioned the benefits of ISIS as an IGP to them.
Link-state protocols are evil, and when they break, they *really* break.
I still do not see a compeling argument for not using BGP as your IGP.
Alex
Slow convergence.
Greetz, Peter
As well there is the issues of running a full iBGP mesh. I've actually
been doing it, and now that I'm about to add my 5th router, OSPF is
looking a lot better than configuring 4 more BGP sessions. I've heard
some people recommend a route-reflector, but that would mean if the
route-reflector goes down you're screwed.
-Ralph
That's why you configure two.
-C
Planning on doing away with that pesky IBGP mesh and just redistributing
BGP into OSPF are we Ralph?
There is so much wrong with the above post that I can't do anything
except hang my head in shame for people running networks everywhere
around the world.
I wish this was a joke, but I know it's not.
Ralph, they are talking about running BGP as an IGP, not if they are going
to run BGP at all. Most large carriers run BGP everywhere. They also run an
IGP for next-hop reachability within their networks (loopbacks, interface
/30s, etc). The issue was whether you can get away with not running the IGP,
and just running BGP. The problem is, of course, BGP handles many routes
well, and converges relatively slowly. IGPs converge quickly, but only
handle a relatively small number of routes.
If you are offering transit to folks, you need to run BGP pretty much
everywhere. If you are peering, or have multiple transits in multiple
locations, with a network between them, BGP makes a lot of sense. If not,
just run BGP on your edge routers.
As far as an IGP - you need one.
You don't need route reflectors for 5 routers. You probably do need
something at double that number. Use two route reflectors, so if one goes
down, you're ok. Redundancy is wonderful. Or use confederation BGP, which
doesn't usually have single points of failure, but many be a bit too complex
for some networks.
Finally, if cutting and pasting a configuration is making you a "sad clown",
learn Expect or Perl, and write a script. That way, more routers can be
broken in a shorter period of time, leading to greater efficiency.
We now return to our regularly scheduled, low level of signal to noise.
- Daniel Golding
>
> > > > Has anybody mentioned the benefits of ISIS as an IGP to them.
> > > Link-state protocols are evil, and when they break, they *really*
break.
> > > I still do not see a compeling argument for not using BGP as your
IGP.
> >
> > Slow convergence.
>
> As well there is the issues of running a full iBGP mesh. I've actually
> been doing it, and now that I'm about to add my 5th router, OSPF is
> looking a lot better than configuring 4 more BGP sessions. I've heard
> some people recommend a route-reflector, but that would mean if the
> route-reflector goes down you're screwed.
What about a well choosen (wrt topo) pair of them...
Planning on doing away with that pesky IBGP mesh and just redistributing
BGP into OSPF are we Ralph?There is so much wrong with the above post that I can't do anything
except hang my head in shame for people running networks everywhere
around the world.
mh
Um. Set up more than one reflector....
So how many is enough? I would think 3 is a minimum to come close to the
reliability/redundancy of OSPF.
-Ralph
Um. Set up more than one reflector....
yes... and align your setup with your physical topology(so making it
useful);
use other proto for mapping your infra, etc, etc,..
mh
> > > Has anybody mentioned the benefits of ISIS as an IGP to them.
> > Link-state protocols are evil, and when they break, they *really*
break.
It depends a lot on your topgraphy. For example, if you have a specific
city/region aggregation center, your redundant boarder routers for that
city are probably also RRs. Those boarders peer with your boarders in
other cities as well as your intra-region routers.
Of course, this is just one way. Trying not to start a religious war.
Yup. I like using OSPF to set up the mesh to the loopbacks and then ibgp
as the IGP.