ATM (was Re: too many routes)

Sean M. Doran wrote:

Cool, I love being talked down to by old guys. It's
refreshing and doesn't happen nearly frequently enough.

  I am not that old. And I did not intentionally talk down, I was tired
and grumpy, I am sorry if it came out that way. But, I would love to
respond to the technical points in your
extremely "learn-ed" email. ( That is a genuine compliment, not sarcasm.

Now remember, I am argueing for ATM, not about the natural color of my

Or, the speed of light when chasing a south bound sparrow. :wink:

PDH is dead.


End-to-end POTS is already dying. Worldcom is making a
big deal over relatively simple technology which shuffles
fax traffic over the Internet.

Yeah, and they are pushing it around over ATM.

However, there are neat
plans for SS7 and neater plans for doing clever things
with interpreting DTMF.

  Hey, did you ever notice that ATM is called broadband ISDN,
and regular ISDN is called narrow band.. And that SS7 is designed
to eventually intermesh these two technologies? This is still not
an argument: CON ATM.

  This is not intended to be flippant.... ?sp?

In this model POTS and historical
telco voice and data schemes become services rather than

   Yes, but the end user, in most cases, will still get POTS, as he knew
it. And don't argue for ISDN... It is far easier to push voice across a
channel T1 (Over ATM) using ISDN, than it is to map the RBS onto the
stuff. But I agree completely.

The reason that you see "strange" or at least "unsmooth"
load balancing along parallel paths is that except with
fib and slow switching cisco had always forwarded packets
towards the same destination out the same interface, and
load balancing was performed by assigning (upon a cache
fault) particular destinations to particular interfaces.

(Leon was invented to blow away cached entries so that
over time prefixes would slosh about from one interface to
another as they were re-demand-filled into the cache.

  So ,you are saying, it is only Cisco who demonstrates this variability
Can anyone back that ?

Points if you know who Leon and Viktor are. Hint: they're
both as dead as PDH.)

  This reminds me of Englands Queen. While "PDH may be dead",
it is, and will be, emulated for a long time to come.

  So, "The Queen is dead, long live the Queen"


With CEF these days you can load-balance on a per-packet
basis. This has the side effect that you cannot guarantee
that packets will remain in sequence if the one way delay
across the load-balanced paths is off by more than about
half a packet transmission time. However, you also get
much more even link utilization and no ugly
cache/uncache/recache at frequent intervals (which really
sucks because unfortunately you have to push a packet
through the slow path at every recache).

  I can't ever remember a sequencing issue over parallel ATM paths...
Perhaps I am just not experienced enough ;>

So anyway, as I was saying, I'm ignorant about such


>Audio sounds great with lots of variability

So if you aren't holding a full-duplex human-to-human
conversation you introduce delay on the receiver side
proportional to something like the 95th percentile and
throw away outliers. If you're holding a full-duplex
long-distance human-to-human conversation you can use POTS
(which is dying but which will live on in emulation) and
pay lots of money or you can use one of a number of rather
clever VON member packages and pay alot less money but put
up with little nagging problems. For local or toll-free
stuff, to expect better price-performance from an end-user
perspective now would require taking enormous doses of
reality-altering drugs.

   All is quiet in the BoardRoom, the hostile merger is about to occur.
All the Executives stand pensive, awaiting the last second instructions
from the CEO, in germany.

  " And now, whatever you do, don't forget the *snap*, *crackle*, *pop*,
    or it will cost us millions, Oops! gotta run, ta. (click)"

I am certainly not saying voice over IP is not viable... for the right

Matter of Fact, voice over IP, over ATM, is pretty solid :wink:

Soft real time things can be implemented across a wide
variety of unpredictable media depending on the window
available to service the real time events and the slope of
the utility decay function.

For instance, interactive voice and video have a number of
milliseconds leeway before a human audience will notice
lag. Inducing a delay to avoid missing the end of the
optimal window for receiving in-sequence frames or blobs
of compressed voice data is wise engineering, particularly
if the induced delay is adjusted to avoid it itself
leading to loss of data utility.

  So , what do you think CDVT is all about :wink: (I know , you already
knew this,
  I just couldn't resist )I am still looking for the CON ATM in this

> However, I have NEVER failed to get the bandwith
> "promised" in our nets.

Sure, the problem is with mixing TCP and other window-based
congestion control schemes which rely on implicit feedback
with a rate-based congestion control scheme, particularly
when it relies on explicit feedback. The problem is
exacerbated when the former overlaps the latter, such that
only a part of the path between transmitter and receiver
is congestion controlled by the same rate-based explicit
feedback mechanism.

What happens is that in the presence of transient
congestion unless timing is very tightly synchronized
(Van Jacobson has some really entertaining rants about
this) the "outer loop" will react by either hovering
around the equivalent of the CIR or by filling the pipe
until the rate based mechanism induces queue drops.
In easily observable pathological cases there is a stair
step or vacillation effect resembling an old TCP sawtooth
pattern rather than the much nicer patterns you get from a
modern TCP with FT/FR/1321 stamps/SACK.

In other words your goodput suffers dramatically.

   Good point. However, FGCRA helps, graceful drops and all that...
And yes, it will be a while before the entire mechanism is completely
tied together.
Oh, never mind, you make my point in the following. And remember, I
specifically noted that legacy equipment does not count.

> But, doesn't that same thing happen when you over-run the receiving
> router ???

Yes, and with OFRV's older equipment the lack of decent
buffering (where decent output buffering is, per port,
roughly the bandwidth x delay product across the network)
was obvious as bandwidth * delay products increased.

With this now fixed in modern equipment and WRED
available, the implicit feedback is not so much dropped
packets as delayed ACKs, which leads to a much nicer
subtractive slow-down by the transmitter, rather than a
multiplicative backing off.

    BYTW: I was arguing for ATM, not just ABR. But, the key here is to
provide a reliable method whereby ATM can convey information in the same
pro-active strategy.
My original point is that in the ATM exchange environment, it is far
less likely that an irresponsible neighbor will interrupt your service.
has scaling capacity .... And also a side effect I really like, only the
ones who
are abusing their lines, drop packets. (Usually...)

Finally another ABR demon is in the decay of the rate at
which a VS is allowed to send traffic, which in the face
of bursty traffic (as one tends to see with most TCP-based
protocols) throttles goodput rather dramatically. Having
to wait an RTT before an RM cell returns tends to produce
unfortunate effects, and the patch around this is to try
to adjust the scr contract to some decent but low value
and assure that there is enough buffering to allow a VS's
burst to wait to be serviced and hope that this doesn't
worsen the bursty pattern by bunching up alot of data
until an RM returns allowing the queue to drain suddenly.

  Or, just don't overaggregate to the Nth degree. ;\

Patient: Doctor, Doctor, it hurts when I do this.....
Doctor: So, don't do that. That will be 50 dollars. :wink:

  However, you have a very good point regarding the effect of delay
awaiting the RM to indicate congestion. The solution does not exist,
ATM is not without design considerations.
*Nothing* is a substitute for good planning. It is easier to avoid
bottlenecks when
the top circuit is 622mbs, and you can run parallel paths.

> Ahhh… We await the completion, and proper interaction of RM, ILMI,
> and OAM.
> These will, (and in some cases already DO), provide that information
> back to the router/tag switch.
> Now do they use it well ???
> That is a different story…

The problem is that you need the source to slow
transmission down, and the only mechanism to do that is to
delay ACKs or induce packet drops. Even translating
FECN/BECN into source quench or a drop close to the source
is unhelpful since the data in flight will already lead to
feedback which will slow down the source.

Agreed, it is not currently as pro-active as an ideal situation would
be. Weaknesses still exist providing layer 2 information to the layer 3
devices. The temporary fix is to provide large ingress and egress
buffers, throughout the path, and provide as much end-to-end feedback as
possible, preferably with the ability for all devices in a path to react
to that feedback.
OAM, RM, etc.
Then to do as you say, set a reasonable SCR, and hope you bought enough
buffers, and the CDVT is adequate to prevent loss.

The congestion control schemes are essentially
fundamentally incompatible.

   Lets just say "not optimal".

> > Therefore, unless ABR
> > is deliberately inducing queueing delays, there is no way
> > your delay can be decreased when you send lots of traffic
> > unless the ATM people have found a way to accelerate
> > photons given enough pressure in the queues.
> >
> More available bandwidth = quicker transmission.

   I still say lets switch to tachyons :wink:

(Just for those who don't know already, I am not serious)

> 6 (-.174.56.2) 567 ms
> 154 ms
> 292 ms
> >>>>>>>>>>>>>> Tell me this is a speed of light issue.
> >>>>>>>>>>>>>> From the FDDI to the HSSI on the same router.

This has nothing to do with the router's switching or
route lookup mechanism. Router requirements allow routers
to be selective in generating ICMP messages, and cisco's
implementation on non-CEF routers will hand the task of
generating ICMP time exceededs, port unreachables and echo
replies to the main processor, which gets to the task as a
low priority when it's good and ready. If the processor
is doing anything else at the time you get rather long
delays in replies, and if it's busy enough to start doing
SPD you get nothing.

  I have noted the conversations regarding this. However, I should note
that according to the data we collected, there is a direct correlation
the timings increasing, and the variability/latency/loss of packets
travelling the path.

My argument is based on our charting of timings of this nature, and the
results from BoardWatch reviews. Our timing charts matched almost
perfectly to boardwatches timing of
major carriers. The lower the timings, and closer to minimal variability
from these timings, the better the
final score from boardwatch.

Perhaps this relationship is merely demonstrating that, indeed, ICMP
timing is "retarded" during periods of high router utilization, due to
prioritizing of processes. However, the fact that this prioritization
is being "kicked in" proves to be a good indicator that a router is
experiencing a serious load. And routers experiencing heavy loads, are
*usually* the cuplrit in packet delay/variance/loss. Not conclusive
enough for
the courts, but quite informational , none-the-less.

> Flow switching does a route determination once per flow, after that
> the packets are switched down a predetermined path "The Flow". Hence the
> term "flow switching". This reduces the variability of
> the entire flow.

Um, no it doesn't. As with all demand-cached forwarding
schemes you have to process a packet heavily when you have
a cache miss. Darren Kerr did some really neat things to
make it less disgusting than previous demand-cached
switching schemes emanating out of OFRV, particularly
with respect to gleaning lots of useful information out of
the side-effects of a hand-tuned fast path that was
designed to account for all the header processing one
could expect.

  I have NO excuse for simplifying, other than a lack of need to get
more detailed. I am now a little more versed in my intended audience....
However, the concept still holds true.
The flow switching is "in essence" demonstrating the advantage of
minimizing interim processing. We have collected similar data, as well.
(Hand tuned)
BTW: Everyone seem to have missed that flow switching ,
in its current incarnation, does not scale well.
Sort of like ethernet, there is a critical mass point.
I was surprised no one attacked with that..
(not that it was important to the issue, or anything)

Flow switching does magic matching of related packets to
cache entries which describe the disposition of the packet
that in previous caching schemes could only be determined
by processing individual packets to see if they matched
various access lists and the like. It's principal neat
feature is that less per-packet processing means more pps

MPLS is conceptually related, btw.

Flow switching does not improve queueing delays or speed
up photons and electrons, however, nor does it worsen
them, therefore the effect of flow switching on
variability of normal traffic is nil.

  Hold it.... doesn't a better PPS throughput imply reduced que latency
Not the physical process of entering the que, or the physical process
of exiting the que, but the median time the data is "que homed".
(again, full latency of ingress to egress)
Kind of "by definition" ?

  The more packets that you can clear, the less congested the pipe.
The less congested a pipe, lower the probability that a packet
will need to experience delay. Or, have I over simplified ?

Or, is your operational word "normal traffic"? That would make sense.

Flow switching has mechanisms cleverer than Leon the
Cleaner to delete entries from the cache and consequently
there are much reduced odds of a cache fault during a
long-lived flow that is constantly sending at least
occasional traffic. You may see this as reducing
variability. I see it as fixing a openly-admitted design flaw.

Ok. But, it still works. Reduce the time spent processing the packet/or
cell, and things get smoother... Why should it be otherwise?

> PS MAC Layer Switching, and ATM switching are apples and oranges.
> Although, one could be used to do the other.
> (Told you Dorian)


  Sorry, A response to another users mail. They referenced a holy war in
the archives. I derived from that thread that it was an argument about
traditional LAN
switches, and their MAC based flow descisions -versus- Layer 3 Routing
descisions. I guess
technically speaking, it was related. But,it was not the immediate

  I guess in the picture of things I created flame bait.
  I claim naivette' (sp ?) to the NANOG archives .
  However, I found the discussion refreshing.

  It was a pleasure :wink: