too many routes

Mailman List Mirror (Read Only)
1

Sprint recently announced that they are deploying 622Mbps
POS cross-country links, and the GSRs are the only things
you can put on the ends of such beasts that are available
today.

  This is untrue.

  L2 switches are available today that reliably receive OC-12 SONET
  circuits.

  These can be disaggregated into OC3 ATM pipes that can be fed into
  many routers with proven technology and reliability. Granted, the
  disproportionality of the edge ckts to the backbone ckts provides
  for interesting flow aggregation dynamics, but it does work.

  Your disdain for ATM does not stop its existence and use by the
  larger NSPs.

(Someone may point out that it may not be the only choice
for very long, however, it's the only one whose design I
know and have had any influence upon, and is therefore
frankly the only one I trust,

  Aggressive statement, though the timbre is occasionally marked by
  usefulness. I wonder how the world would develop without Sean's
  input..... {music, clouds, harps... visions of ATM faeries
  dancing with sugarplums...}

  There is another router vendor who has OC-12 router interfaces
  about ready for production. Another (#3) is expected to release a
  bang-up product real soon now.

  There is a choice in the market today for OC12 router interfaces.

  -alan

2

  There is a choice in the market today for OC12 router interfaces.

s/there is/we hope there soon will be/

3

Alan Hannan <hannan@bythetrees.com> writes:

> Sprint recently announced that they are deploying 622Mbps
> POS cross-country links, and the GSRs are the only things
> you can put on the ends of such beasts that are available
> today.

  This is untrue.

Other people have pointed out that the statement is
perfectly true, but we can ignore the semantic misunderstanding.

AFAICT there is only one orderable, deployable and tested
IP router which observably terminates multiple 622-Mbit/s bit
pipes, notably that made by cisco.

One thing I find amusing is your own company's recent
heavy investment in cisco's only likely competitor.

Maybe this will mean cisco will stop paying so much
attention to the ridiculous requirements UUWHO coughs up
because it insists on a heavily meshed architechture for
what has since become essentially purely political
reasons, and start paying attention to people who are
building sensible hierarchical networks that display much
better scaling properties on nearly all fronts. (I hope
so).

Maybe this will also make people shy about helping Juniper
because that translates directly into helping UUWHO as well.
Who knows, maybe such things aren't so important,
especially since the cisco product a/ works and b/ is *cheap*.

  These can be disaggregated into OC3 ATM pipes that can be fed into
  many routers with proven technology and reliability. Granted, the
  disproportionality of the edge ckts to the backbone ckts provides
  for interesting flow aggregation dynamics, but it does work.

Oh sure it works if you turn LS IGPs on their head and
try to avoid thinking too hard about how to build spanning
trees without building a physical VC topology for
multicast and can put up with interesting buffering and
switching effects and, as you say, the dynamics of WHY it
works vs. why it doesn't work when things break.

I hate to say it but I think I have become old and this
might explain why I like really simple and straightforward
failure modes.

  Your disdain for ATM does not stop its existence and use by the
  larger NSPs.

Technical merit has only been offered up as a reason for
using cells by your organization and ATMNET, AFAIK.

Everyone else who has been using ATM has done so because
it was one of only three obvious scaling paths, with the
other two being also gross (large numbers of parallel DS3s
using load-balancing and flattening the network with lots
of DS3s resembling lots of VCs, which is essentially what
UUNET did with its physical topology anyway).

  Aggressive statement, though the timbre is occasionally marked by
  usefulness. I wonder how the world would develop without Sean's
  input..... {music, clouds, harps... visions of ATM faeries
  dancing with sugarplums...}

Hahah. Well, don't forget the Russian and the Swede who
also sang from the same songbook, and the other
congregation members who were at NANOG meetings when
everyone was angry at the BELLCORE twits prior to the
NSFNET wind-down (hi Curtis).

Many of the people who did actual engineering work at
cisco were also not very agnostic when it came to cells,
and had an effect on the company, which is one of the
reasons I have always liked cisco compared to its
competitors, who were not quite agnostic either but in the
opposite direction.

  There is another router vendor who has OC-12 router interfaces
  about ready for production. Another (#3) is expected to release a
  bang-up product real soon now.

As Peter L�thberg pointed out, there are BFRs in
production right now. Sprint is also not the only vendor
who is using them to move traffic, although the majority
of other users are currently using the BFRs as really fast
star-shaped LANs, essentially. (I expect this to change
very soon)

  There is a choice in the market today for OC12 router interfaces.

Not today there isn't. Maybe next month.

There may be choice in what you can put at the end of an
OC12 though, but it's either something smart or something
you'd find at Hobson's Tavern.

  Sean.

4

AFAICT there is only one orderable, deployable and tested
IP router which observably terminates multiple 622-Mbit/s bit
pipes, notably that made by cisco.

  Hmm, well, the lack of knowledge I suppose shouldn't be held
  against you.

  www.ascend.com : nom de domaine enregistré chez Safebrands - Registrar Icann, Afnic, Eurid

  lists their support for such interfaces.

  ( nonsequiter comments deleted )

> These can be disaggregated into OC3 ATM pipes that can be fed into
> many routers with proven technology and reliability. Granted, the
> disproportionality of the edge ckts to the backbone ckts provides
> for interesting flow aggregation dynamics, but it does work.

Oh sure it works if you turn LS IGPs on their head and

  On their head? I can assure you the IGP is still a manageable
  problem in larger ISPs with meshed backbones. In fact, there is
  one solution available that is not yet even exercised.

try to avoid thinking too hard about how to build spanning
trees without building a physical VC topology for
multicast and can put up with interesting buffering and
switching effects

  Okay, I'll bite: Which interesting buffering and switching effects?

and, as you say, the dynamics of WHY it
works vs. why it doesn't work when things break.

  Yes, indeed this is a tradeoff.

  But, the more I think about the KISS principle, I become convinced
  that only our own limitations, ignorance, and hesitancy prevent us
  from adding complexity to achieve increased control.

  Certainly, when solution A and solution B have net effect Z, we
  should choose the simpler.

  But to say that the benefit of B, even with increased complexity,
  isn't needed because we don't understand it, is just silly (unless
  you live in the real world, and have to work with people, as we
  do, but still, we should consider the question "Is the benefit
  + the increased knowledge worth the education and increased risk"
  not just "which is simpler".)

I hate to say it but I think I have become old and this
might explain why I like really simple and straightforward
failure modes.

  See above comments.

> Your disdain for ATM does not stop its existence and use by the
> larger NSPs.

Technical merit has only been offered up as a reason for
using cells by your organization and ATMNET, AFAIK.

  Yes, we at 'bythetrees.com' are avid supporters of ATM in today's
  technology.

  Why do you always insist on linking one's place of business with
  their technological idealogy? Certainly I work at UUNET and my
  opinion is occasionally involved in certain decisions. However, I
  am tempered by people more wise, experienced, and quite frankly,
  smarter, than me.

  I don't speak for UUNET, I speak for me. Golly. Novel concept.

Many of the people who did actual engineering work at
cisco were also not very agnostic when it came to cells,
and had an effect on the company, which is one of the
reasons I have always liked cisco compared to its
competitors, who were not quite agnostic either but in the
opposite direction.

  Hmm, I tend to be atheistic about technology -- what can it do for
  me today, and what will it to do tomorrow with a reasonable chance
  of success.

There may be choice in what you can put at the end of an
OC12 though, but it's either something smart or something
you'd find at Hobson's Tavern.

  How do you define smart?

  -alan

5

Alan Hannan <hannan@bythetrees.com> writes:

> AFAICT there is only one orderable, deployable and tested
> IP router which observably terminates multiple 622-Mbit/s bit
> pipes, notably that made by cisco.

  Hmm, well, the lack of knowledge I suppose shouldn't be held
  against you.

  www.ascend.com : nom de domaine enregistré chez Safebrands - Registrar Icann, Afnic, Eurid

  lists their support for such interfaces.

  ( nonsequiter comments deleted )

I hope the comments privately prior to your writing any of
this that pointed out that nowhere in this URL or in
Ascend's other literature is the concept of non-ATM 622
mentioned. Oh wait, that's a non sequitur, because it's
the same thing as POS, right, only with extra features.

  On their head? I can assure you the IGP is still a manageable
  problem in larger ISPs with meshed backbones. In fact, there is
  one solution available that is not yet even exercised.

It's only manageable because Henk Smit (and before him
Dave Katz) has been going to lots of trouble to clean up
all the edge cases strangely flat backbones reveal, and
because Cisco has no trouble with the idea of adding in
richer metrics to iIS-IS. (Well, neither do I, actually :slight_smile: )

Really rich connectivity means that it is more work to
calculate new forwarding tables as point to point
connections come and go, and the results are much harder
to predict.

Of course, if you take your approach and DON'T route
dynamically between routers, but rather reroute VCs around
link failures, then you might not have these problems
except when fun things happen like ATM switches reloading
or router-to-switch failures happen. At that point you
have the fun of working out appropriate backup paths for
the traffic that used to fan out over a large number of
VCs. This is non trivial and scales geometrically with
the size of the mesh.

> try to avoid thinking too hard about how to build spanning
> trees without building a physical VC topology for
> multicast and can put up with interesting buffering and
> switching effects

  Okay, I'll bite: Which interesting buffering and switching effects?

You have to copy datagrams out to each branch of your
spanning tree. If you have a largd number VCs out one
interface overlaid with a dense spanning tree, you end up
with large buffering requirements on that card, or
alternatively do funny queue management so that you eat
lots of queueing slots if you use an indirection list
(i.e., you use pointers pack to the multicast PDU rather
than copy the PDU multiple times, however "boxing"
datagrams, while it saves memory-to-memory copies can
make it difficult to do clever cache prefetching from
packet memory, and may not reduce the number of memory
reads you have to do, and certainly will increase CPU
use). In other words, when using smart line cards like Wellpap
or Cisco do, if you are doing lots of multicast with a
fairly large distribution, you are better off keeping the
spanning tree small. If you have a large number of
downstream interfaces on a box, that box works harder. If
you have a large number of downstream interfaces on a
single card, that card works harder. Harder can be
described as larger CPU time and more memory references in
some combination, depending on the queueing mechanism and
what does the packet copying.

Also when you have a large concentration of downward links
on a box, that box tends to attract more join and prune
messages which also keeps it busier. You could add RSVP
issues to that too, if you thought RSVP were somehow
relevant to the Universe.

> and, as you say, the dynamics of WHY it
> works vs. why it doesn't work when things break.

  Yes, indeed this is a tradeoff.

Obviously you and I have different thoughts about
Byzantine failures. I've experienced enough of them
driven by multilayer operability issues with a small set
of vendors that widening the vendor base substantially
(i.e., adding lots of switch vendors and the like, not
just adding a second router vendor) or increasing the
number of layers which have to interoperate does not
appeal much to me.

  But, the more I think about the KISS principle, I become convinced
  that only our own limitations, ignorance, and hesitancy prevent us
  from adding complexity to achieve increased control.

Sure. How much do you want to spend on NOC staff is a
factor in how much complexity NOC staff can handle, and
that is a factor in how quickly Byzantine failures can be
dealt with.

Personally, Byzantine failures that require waking up
senior smart people at multiple vendors' companies are
scary and as Byzantine failures happen sometimes in big
networks, keeping the number of people potentially needed
to effect a fix for something really weird strikes me as a
win.

In short, defensive engineering.

Sure complexity seems attractive, but only until your
first major meltdown driven directly by or even only
exacerbated by that complexity.

> I hate to say it but I think I have become old and this
> might explain why I like really simple and straightforward
> failure modes.

  See above comments.

See bags under eyes and many scars.

  Yes, we at 'bythetrees.com' are avid supporters of ATM in today's
  technology.

Oh sure, you're bythetrees.com NOW. --:slight_smile:

  Why do you always insist on linking one's place of business with
  their technological idealogy?

Because:

  a) it makes people angry, and that's really funny
  b) it reminds people whom they are working for and
     what trade-offs they make between what they
           want and what their employers say they want
  c) a + b sometimes causes people to fix their
           employers into doing things "correctly", or
     leads people into realizing their employers are
           hopeless and that there are other opportunities
           elsewhere (which sometimes also fixes the employer)

the latter part of (c) is particularly edifying if it
involves physical violence to doors and so forth.
  
So, needle needle, if you are at variance with UUWHO's
technical policies, why are you still there?

Certainly I work at UUNET and my opinion is occasionally
  involved in certain decisions.

Oh, cool, so I can blame you for brain damage. Several
managers will be really happy now that you've volunteered
yourself to be picked on in technical lists instead of
them. :slight_smile:

  Hmm, I tend to be atheistic about technology -- what can it do for
  me today, and what will it to do tomorrow with a reasonable chance
  of success.

I think you mean agnostic, not atheistic. I distinctly
remember you comparing me to Jesus Christ.

> There may be choice in what you can put at the end of an
> OC12 though, but it's either something smart or something
> you'd find at Hobson's Tavern.

  How do you define smart?

When you put your brain into a blender and unpack it into
individual cells in hopes you can put it all back together
the way it was without having noticed, that's not very
smart.

  Sean. (don't send the next cheesecake as
    breadcrumbs please :slight_smile: )

6

> > AFAICT there is only one orderable, deployable and tested
> > IP router which observably terminates multiple 622-Mbit/s bit
> > pipes, notably that made by cisco.
>
> www.ascend.com : nom de domaine enregistré chez Safebrands - Registrar Icann, Afnic, Eurid
> lists their support for such interfaces.

I hope the comments privately prior to your writing any of
this that pointed out that nowhere in this URL or in
Ascend's other literature is the concept of non-ATM 622
mentioned. Oh wait, that's a non sequitur, because it's
the same thing as POS, right, only with extra features.

  I see now where we missed each other -- no, I know of no POSIP
  interface for OC12 with the exception of Cisco's.
  
  However, as it is not a tool that I can put into my great big bag
  of router tricks (with the exception of peerings or customer
  connections, which I'm pondering) it's not too important (to me)
  (right now).

  We differ in this manner -- you feel the negatives of atm forbid
  its use *period*, while I think its benefits outweight any negatives
  at this time, and couldn't consider growing a large network
  without its flexibility.

> On their head? I can assure you the IGP is still a manageable
> problem in larger ISPs with meshed backbones. In fact, there is
> one solution available that is not yet even exercised.

It's only manageable because

  But it's manageable.

because Cisco has no trouble with the idea of adding in
richer metrics to iIS-IS. (Well, neither do I, actually :slight_smile: )

  Funny you and cisco agreeing. Nonetheless, the metric set is
  its own problem, but moderately orthagonal to any of these
  discussions. On that tangent, however, flooding is what we're
  mostly concerned about, and it's just not that bad.

Of course, if you take your approach and DON'T route
dynamically between routers, but rather reroute VCs around
link failures, then you might not have these problems
except when fun things happen like ATM switches reloading
or router-to-switch failures happen. At that point you
have the fun of working out appropriate backup paths for
the traffic that used to fan out over a large number of
VCs. This is non trivial and scales geometrically with
the size of the mesh.

  The scaling issue is synonymous w/ an entirely L3 network. L2
  networks just put it off and make it somewhat less likely.

  See, I get the impression that people on this list can be
  categorized as two kinds -- router people and network people.
  That may be too strict a delineation.

> > multicast and can put up with interesting buffering and
> > switching effects
>
> Okay, I'll bite: Which interesting buffering and switching effects?

  (paragraph about buffer:flow ratios deleted)

  Hmmm. So how does this change by putting lots of virtual
  interfaces on a router and putting the buffering into L2?

> > and, as you say, the dynamics of WHY it
> > works vs. why it doesn't work when things break.
>
> Yes, indeed this is a tradeoff.

Obviously you and I have different thoughts about
Byzantine failures. I've experienced enough of them
driven by multilayer operability issues with a small set

  Perhaps it's a difference in the quality or quantity of
  those managing the network. :wink: The interoperability adds a
  variable, but the scalars in the equation minimize probability or
  problems.

of vendors that widening the vendor base substantially
(i.e., adding lots of switch vendors and the like, not
just adding a second router vendor) or increasing the
number of layers which have to interoperate does not
appeal much to me.

  That's why you do it your way and.....

> But, the more I think about the KISS principle, I become convinced
> that only our own limitations, ignorance, and hesitancy prevent us
> from adding complexity to achieve increased control.

Sure. How much do you want to spend on NOC staff is a
factor in how much complexity NOC staff can handle, and
that is a factor in how quickly Byzantine failures can be
dealt with.

   Yep.

Personally, Byzantine failures that require waking up
senior smart people at multiple vendors' companies are
scary and as Byzantine failures happen sometimes in big
networks, keeping the number of people potentially needed
to effect a fix for something really weird strikes me as a
win.

In short, defensive engineering.

  At the cost of scalability. Not a win. Anytime you limit growth
  to a variable you are doomed (ahh physical space you evil beast).
  Instead, you educate and.... this thread goes nowhere. I see your
  point, we disagree.

Sure complexity seems attractive, but only until your
first major meltdown driven directly by or even only
exacerbated by that complexity.

  The opportunity to have a complex network is arift with its own
  problems.

  Once your problem set becomes large enough your
  solution set becomes larger. Increase your richness in variables
  to solve more quickly and exactly.

> Why do you always insist on linking one's place of business with
> their technological idealogy?

Because:

  a) it makes people angry, and that's really funny
  b) it reminds people whom they are working for and
     what trade-offs they make between what they
           want and what their employers say they want
  c) a + b sometimes causes people to fix their
           employers into doing things "correctly", or
     leads people into realizing their employers are
           hopeless and that there are other opportunities
           elsewhere (which sometimes also fixes the employer)

the latter part of (c) is particularly edifying if it
involves physical violence to doors and so forth.
  
So, needle needle, if you are at variance with UUWHO's
technical policies, why are you still there?

  I don't accept your assertion, but in answer to the latter
  query: They/We rock. :slight_smile:

> Hmm, I tend to be atheistic about technology -- what can it do for
> me today, and what will it to do tomorrow with a reasonable chance
> of success.

I think you mean agnostic, not atheistic.

  No, I mean atheistic. I am without religion with regards to L2
  fabric selection for networking. You want to believe (smith's
  reference), but you're just not sure. I don't believe, I
  rely on facts. You do too, but you ignore some facts because of
  your religious tendencies.

I distinctly
remember you comparing me to Jesus Christ.
  
  Yes, remember the crucifix phrase?

> > There may be choice in what you can put at the end of an
> > OC12 though, but it's either something smart or something
> > you'd find at Hobson's Tavern.
>
> How do you define smart?

When you put your brain into a blender and unpack it into
individual cells in hopes you can put it all back together
the way it was without having noticed, that's not very
smart.

  Hmm, I thought that's what telnet did to keystrokes gifs. Or what
  sonet did to PPP or whatever those l2 things are that the cisco OC12
  interface sends out. Or what web servers did to IP when they send
  noodie gifs to pedophiles using PGP.

  Sean. (don't send the next cheesecake as
    breadcrumbs please :slight_smile: )

  You owe me. I'll take a pound of cheese to go with your next
  mail. :slight_smile:

  -alan

  ps. for those of you at home or work forced to endure this anecdotal
      parenthetical, I used to work at a particular NSP who used
      Sean's previous employer for internet connectivity. His
      always happy and never wavering commitment to customer
      service and reliability soon found us cheerful chums. In
      thanks to this level of service, I bestowed a cheesecake to
      him from a customer of ours at the time. I was not reimbursed
      for this from my company, but I guess it's too late....

7

hannan@bythetrees.com (Alan Hannan) writes:

  On that tangent, however, flooding is what we're
  mostly concerned about, and it's just not that bad.

Alan,

You are correctly concerned about flooding, however, I suspect that you're
underestimating the scalability problems of flooding. Please recall that
it's O(n^2) in the number of nodes in your mesh. This is a serious threat
to building very large domains. In addition, the full mesh prevents
hierarchical abstraction within the domain, defeating the other obvious
mechanism for IGP scalability.

Tony