FYI
--Jonny Ogawa
----- Forwarded message from Stephen H. Inden -----
It's all true.
Alse I've been weighing my router and it's 7 lbs heavier with the addition of all these new ip addresses in it's routing table.
-wil
From: Joao C. Mendes Ogawa
Sent: Thursday, March 31, 2011 6:14 PM
Subject: Fwd: IPv4 Address Exhaustion Effects on the EarthFYI
--Jonny Ogawa
----- Forwarded message from Stephen H. Inden -----
Dang, I was hoping to see an RFC on Bufferbloat in Avian Carriers and
how tail-drop is a messy solution that is to be avoided.
Oh well.
wil,
maybe after all this time you got the router, it gained 7lbs of all the
dust in it ?
Op 1-4-2011 3:26, Wil Schultz schreef:
Date: Sat, 02 Apr 2011 04:18:00 +0200
From: Alexander Maassen <outsider@scarynet.org>
Subject: Re: IPv4 Address Exhaustion Effects on the Earthwil,
maybe after all this time you got the router, it gained 7lbs of all the
dust in it ?
Consider what happens if the carrier encounters a route reflector --
flipping the bird??
Sigh... A major opportunity missed.
Unfortunately the bufferbloat problem isn't a laughing matter, though I do wish I had thought of this idea in time for my talk. I will include this joke as some levity about the mess we're in as I repeat the talk going forward, and would tie in very nicely with one of the amusing reasons that "RED in a different light" has never been published. I really hate giving such bad news without some levity as it can be a real downer both for me and the audience.
For those of you who missed my IETF talk, you can find the latest version (tweaked since IETF) at: http://mirrors.bufferbloat.net/Talks/PragueIETF/
I suspect audio is some place on the net as well; I presented at the transport area meeting. The questions after my talk are also very worth listening to. Time was precious in that venue, so I did feel rushed and hope to get a better opportunity in a month or two for that. It's a shorter version of my first talk given at Murray Hill http://mirrors.bufferbloat.net/Talks/BellLabs01192011/ which does have additional information impossible to fit in that short a time slot.
- Jim
Sigh... A major opportunity missed.
Unfortunately the bufferbloat problem isn't a laughing matter, though
I
do wish I had thought of this idea in time for my talk. I will
include
this joke as some levity about the mess we're in as I repeat the talk
going forward, and would tie in very nicely with one of the amusing
reasons that "RED in a different light" has never been published. I
really hate giving such bad news without some levity as it can be a
real
downer both for me and the audience.
Speaking of Van's paper, has that ever been located/revived? Is it
available beyond that one earlier draft that is available on the
Internet?
George
Also how port mirrors will cause a collision and the bird will die.
Van and Kathie managed to get a later version off a disk image and get it out of Framemaker. Unfortunately, the paper was only half edited when the second attempt at publication failed due to the circumstances I blogged about at:
http://gettys.wordpress.com/2011/01/06/a-committee-is-a-life-form-with-six-or-more-legs-and-no-brain-lazarus-long/
Right now, they are concentrating on trying to get a consistent description of the proposed RED Light algorithm captured correctly, so we can code it up and try it. Then they will work on finishing up the rest of the text for publication sometime this summer. What I have in hand from Kathie at the moment is not internally consistent, though much longer.
In the mean while, we've started work on various AQM and buffer management systems, at www.bufferbloat.net. SFB (Stochastic Fair Blue) went upstream into Linux to aid testing last month, and we have an implementation of eBDP as well with which we are experimenting. Wireless is much more of a challenge than the classic internet router case. Please come help.
Note that the paper "Characterizing Residential Broadband Networks" by Dischinger, et. al. indicates that a large fraction (in their 2 year old sample, 30% or so) of broadband head ends are running without RED, and should be doing so if at all possible; alternatives are years out by the time they are tested and deployed, and operators running without it in congested systems are inflicting pain on their customers.
- Jim
In the mean while, we've started work on various AQM and buffer
management systems, at www.bufferbloat.net. SFB (Stochastic Fair
Blue)
went upstream into Linux to aid testing last month, and we have an
implementation of eBDP as well with which we are experimenting.
Wireless is much more of a challenge than the classic internet router
case. Please come help.
In my context "Wireless" means "mobile" and the challenge there is that
"I lost a packet" doesn't mean "there is congestion". It most likely
means the user walked in front of a pole, the signal faded briefly, they
dropped a packet and are perfectly fine now. So in that context, tcp/ip
behaves as if it is seeing congestion when it is really seeing a
momentary loss of connectivity that comes right back as soon as the end
node moves 5 feet to the left. The right answer there might be
ubiquitous support of ECN and treating packet loss in the absence of ECN
as a connectivity issue and not a congestion issue but we are a long way
from proper ECN support.
I will have another look at the site, it has been a while since I was
there last.
George
Note that the paper "Characterizing Residential Broadband Networks" by
Dischinger, et. al. indicates that a large fraction (in their 2 year
old
sample, 30% or so) of broadband head ends are running without RED, and
should be doing so if at all possible; alternatives are years out by
the
time they are tested and deployed, and operators running without it in
congested systems are inflicting pain on their customers.
So, who's your contact at Google for KCK?
Cheers,
-- jra
Vint ;-).
- Jim
Something I've observed is if you are sending data 'upstream' on the cable modem setup i have (16 down/ 2 up) and you saturate the upstream, the buffering destroys any downstream capability at the same time. I'm not even sure where to start diagnosing to explaining this to the carrier involved, as this isn't the desired behavior of a "business class" service.
- Jared
Nope. Your acks get delayed to what you are sending upstream, behind the downstream traffic.
Bufferbloat hurts both directions, once saturation occurs and your latencies start to go up.
Note that on many of these links, the RTT becomes (literally) as though you are half way (or further than) the moon.
- Jim
I sent a private reply, but I guess i'll post some of it here:
1) there are no ways to identify the devices doing the buffering and/or drop counts
2) I can obviously feed the cable modem much faster on the lan vs what it can send upstream
Doing things like rate-limiting/QoS are merely just papering over the problem. I would take a T1 and rate-limit it to 1.2Mb/s for TCP to allow VoIP to work. Junipers can buffer up to 1 second on these low-speed interfaces, which obviously creates the problems you describe.
There are a lot more problems with the gateway devices, such as the forcible dns proxy that exists.
- Jared
Note that the paper "Characterizing Residential Broadband Networks" by Dischinger, et. al. indicates that a large fraction (in their 2 year old sample, 30% or so) of broadband head ends are running without RED, and should be doing so if at all possible; alternatives are years out by the time they are tested and deployed, and operators running without it in congested systems are inflicting pain on their customers.
Something I've observed is if you are sending data 'upstream' on the cable modem setup i have (16 down/ 2 up) and you saturate the upstream, the buffering destroys any downstream capability at the same time. I'm not even sure where to start diagnosing to explaining this to the carrier involved, as this isn't the desired behavior of a "business class" service.
- Jared
Isn't this just a case or prioritizing outbound ACKs?
Nope. Your acks get delayed to what you are sending upstream, behind the downstream traffic.
Bufferbloat hurts both directions, once saturation occurs and your latencies start to go up.
Note that on many of these links, the RTT becomes (literally) as though you are half way (or further than) the moon.
I sent a private reply, but I guess i'll post some of it here:
1) there are no ways to identify the devices doing the buffering and/or drop counts
This isn't really true: you basically do "ping" combined with "traceroute" while saturating a path. The hop where there is unexpected additional latency not present when the you don't saturate the path is the culprit.
You can't easily figure out where inside a bottleneck the buffering is, unless you have some way to monitor or control the buffering inside it (e.g. twist the transmit queue or transmit rings in Linux, as an example).
2) I can obviously feed the cable modem much faster on the lan vs what it can send upstream
Doing things like rate-limiting/QoS are merely just papering over the problem.
Papering over the problem isn't all bad, if it allows you to hide egregious size buffers in a bottleneck link over which you'd otherwise have no control.
It allows me to take my latency/jitter on my home cable service from about 400ms down to 10ms (at some cost in bandwidth). This means I actually can have voip or other latency sensitive applications work (so long as I ensure that the broadband link is the bottleneck; if you home wireless network's effective bandwidth drops below that of your broadband, then the bottleneck becomes the 802.11 hop, and you'll see queues in your host and home router, rather than in the broadband hop. Notice that this means with symmetric 25/25 FIOS service, you get bufferbloat in your host and wireless router, as I did (actual data transfer rates of 802.11g is only about 20Mbps, not the 54 signalling rate marketed).
I would take a T1 and rate-limit it to 1.2Mb/s for TCP to allow VoIP to work. Junipers can buffer up to 1 second on these low-speed interfaces, which obviously creates the problems you describe.
Bufferbloat is (nearly) everywhere.
You'd better see if you can run RED or some AQM. The issues with RED revolve around the fact that it is a flawed algorithm that requires proper tuning to be effective, and it can't handle situations such as wireless or broadband with time variable behaviour such as Comcast's Powerboost.
It is exactly the fact that RED requires tuning that means that it is often not enabled when it should be: but it is often the only tool we've got right now.
There are a lot more problems with the gateway devices, such as the forcible dns proxy that exists.
Right now, the home router market is a cesspool of scum and villainy. Our best hope is the rebel alliance; I think we'll get OpenWRT straightened out long before the commercial vendors do.
- Jim