today's charlie foxtrot

Regarding today's Charlie Foxtrot:

  The problem that occured today appears to be caused by someone
  without clues redistributing their BGP (which contains a set of
  routes of people clueless enough to trust them to run an exchange
  point of sorts) into their IGP, back into the BGP tables. It is
  also possible that the lack of clues was involved in choosing a
  vendor that exhibited a bug similar to this.

  This caused the AS paths to be lost, allowing loopish things to
  occur. This caused router tables to flare, and improper paths to
  be propogated and used. Lots of churn for a stressed Internet.

  It seems to have propogated rather evilly by hiding in edge ASes
  and seeping back in via edge ASes to other providers (ie it comes in
  provider 1, and then goes out provider 2). This also seems to
  have been propogated by subasish behaviour of certain providers
  running confederations and multiple ASes.

  RSes mail re: email not being an effective mode for conversing is
  somewhat correct, nonetheless, I continued to receive email from
  NANOG through the issue.

  As well, several larger ISP folk were on the phone diagnosing the
  problem and taking action to minimize the effect.

  Enke's mail stating that filters could have prevented this is
  somewhat accurate, but it is not apparent that filtering folk were
  not at risk to this event. I'd be curious to hear what ANS saw.

  The duration appears to have been ~11:30 AM EDT to ~2:00 PM EDT.
  At no time was this a catestrophic Internet breaking event, so
  relax Bob.

  This could have been prevented by two things:

    1/ Strict adherence to peer-peer filters.

    2/ Someone not doing something stupid.

  I think we are getting too big to rely on 2 not happening. Yet the
  sense of liberty yearns for 1 not to happen.

  But it will. This will help accelerate it.