Bufferbloat related censorship at Virgin Media

Mailman List Mirror (Read Only)
1

I have put this on a blog post, and my g+ also, here, and submitted
the story to slashdot and reddit. How I spend my sunday afternoons
these days!

The linky version:
http://the-edge.blogspot.com/2015/03/virgin-media-fixing-epidemic-of.html

or g+: https://plus.google.com/u/0/107942175615993706558/posts/E1yMgbWW81C

--snip snip--

To whom it may concern at Virgin Media:

My IP address is apparently now banned from accessing your site at
all, for "advertising", on this thread:

http://community.virginmedia.com/t5/Up-to-152Mb/Bufferbloat-High-Latency-amp-packet-loss-when-connection/td-p/2773495

Believe me, I understand the degree to which advertising pollutes the
internet. And certainly, given the brevity of my post, you could
assume that I was just some random guy, selling snake oil. Nothing
could be further from the truth. Admittedly, it was a short message,
it was kind of late, and I was in a hurry, being that I have so many
other networks to help fix. To clarify matters:

I am the co-founder of the bufferbloat project, and I like to think, a
world-wide acknowledged expert on the topic on this thread.

In particular, I worked pretty hard on part of the DOCSIS 3.1
standard, which was ratified years ago, and has a specific section on
it regarding technologies that can help fix *half* your bufferbloat
problem.

http://www.cablelabs.com/how-docsis-3-1-reduces-latency-with-active-queue-management/

I admit to some frustration as to how long it is taking DOCSIS 3.1 to roll out.

The cablelabs study that led up to the AQM component in the 3.1
standard - in which I participated and am cited in, is here:

http://www.cablelabs.com/wp-content/uploads/2013/11/Active_Queue_Management_Algorithms_DOCSIS_3_0.pdf

And while I continue to favor fq_codel as the best solution for low
and medium bandwidths - I have no problem with you somehow, soon,
getting DOCSIS-pie out the door.

If you continue to exist in denial of what your own R&D department for
your own industry is saying, ghu help you! After giving this talk at
uknof, the premier conference for network operators in the UK:

https://plus.google.com/u/0/107942175615993706558/posts/immF8Pkj19C

*over two years ago*, I met with 6+ technical members of Virgin
Media's staff, who all agreed they had a problem, understood what it
was, and grokked the various means to fix it. Judging from the
enthusiasm in the room, I figured you'd be rolling out fixes by now,
but was wrong.

A rather human readable explanation of what has gone into the pending
3.1 standard is in the IETF internet draft here:

https://datatracker.ietf.org/doc/draft-white-aqm-docsis-pie/

Sadly, just DOCSIS-pie rolling out on the modems is not enough - you
have to somehow, yourselves, fix the dramatic overbuffering on the
CMTS side, as shown here:

http://burntchrome.blogspot.com/2014/05/disabling-shaping-in-one-direction-with.html

These downlink problems have been discussed thoroughly on the
bufferbloat.net bloat and the ietf aqm mailing lists, and rather than
point at direct links I would encourage more people to join the
discussions there, and browse the archives.

As I have seen no visible progress on the CMTS front yet...

The best way to fix bufferbloat for your suffering customers *now*, is
to help them - and your customer service departments - recognise the
problem when it occurs and propose sane ways to fix it with stuff
available off the shelf which includes the free firmware upgrades
distributed by openwrt, or nearly any linux derived product and by the
products available downstream from those.

I have no financial interest in *free firmware*. I'm just trying to
fix bufferbloat on a billion+ devices and nearly every network in the
world as fast as humanly possible. Furthermore, me and a whole bunch
of Internet luminaries gave the theory and code away for *free* also,
in the hope that by doing so that might more quickly get the megacorps
of the world to adopt them and make the quality of experience of
internet access for billions of users of the world vastly better.

Fixing bufferbloat was a 50 year old network research problem, now
solved, with great joy, thoroughly, by some of the best minds in the
business, and the answers are now so simple as to fit into a few
hundred lines of code, easy to configure for end-users and easily
embeddible in your own devices and networks if only you would get on
the stick about it.

We have provided the code, are in the standardization process, and
provided free tools to diagnose and fix your epidemic bufferbloat
accurately on every kind of device you have.

Two examples of fixing bufferbloat on cablemodems:

http://burntchrome.blogspot.com/2014/05/fixing-bufferbloat-on-comcasts-blast.html

http://snapon.lab.bufferbloat.net/~cero2/jimreisert/results.html

And the *free* tool designed not only to accurately measure
bufferbloat, but one that you can setup internally to test your
networks and devices for it privately and quietly and then fix them
with, is here:

https://github.com/tohojo/netperf-wrapper

So, now, a rant:

Now, if me pointing a customer of yours that has correctly identified
the root cause of his own problems, at the solutions both available
now, and pending, is considered "advertising", then there really is an
orwellian mixup between the definition of that word, and the truth, on
your side of the water.

Please, unblock my dtaht account and unblock my IP, and allow in
better information about how customers of yours can solve the
incredibly serious, and incredibly epidemic problem of bufferbloat.

... A problem that is now easy to solve with cheap gear now all over
the market so that all your customers suffering can fix it for
themselves if they so choose.

And: I would like a public apology for blocking me, and a clear
statement from Virgin, as to how, when, and where, they will begin to
roll out their own fixes to bufferbloat across their subscriber base.
And perhaps, you could publish some guidelines - like accurate
up/download settings to use - to help your customers fix your problems
for themselves.

Sincerely,

Dave Taht
Co-founder, bufferbloat.net

2

Dave,

I appreciate all your work on buffer bloat. It looks like you have done quite a lot of selfless contribution. However, I don't think you're effectively communicating with the people who can change things.

After I read what you said, here is what I would have heard as a service provider:

"I am the smartest person in the room. You better listen to me, because if you don't there will be trouble. But you probably won't because you're too stupid. Your customers suffer because you are idiots. Listen to me! This issue is too important for me to be polite, or even coherent. If you can't figure out what I'm saying, do some research and figure it out! Plus, apologize to me! I demand it!"

Bees, honey, vinegar, etc.

-mel beckman

3

Dave,

I appreciate all your work on buffer bloat. It looks like you have done quite a lot of selfless contribution. However, I don't think you're effectively communicating with the people who can change things.

After I read what you said, here is what I would have heard as a service provider:

"I am the smartest person in the room.

I am pretty close to being the smartest person in the room. That said,
people like van jacobson, eric dumazet, tom herbert, jim gettys, eric
raymond, vint cerf, dave reed, fred baker, and many, many others,
smarter than me, have also been banging this drum, politely, and
rationally, to not much effect, for 4+ years now.

google for any of those names and the word "bufferbloat".

You better listen to me, because if you don't there will be trouble. But you probably won't because you're too stupid. Your customers suffer because you are idiots. Listen to me! This issue is too important for me to be polite, or even coherent. If you can't figure out what I'm saying, do some research and figure it out! Plus, apologize to me! I demand it!"

Oh, banning my ip for *3 links to sane benchmarks and fixes*,
realllllly pushed me over the edge.

Bees, honey, vinegar, etc.

I have been polite, constructive, and helpful, for four+ years. I have
worked both in the background and foreground with many companies, to
start hopefully, getting bufferbloat fixed across the entire edge of
the internet. It hasn't worked fast enough for my liking, and the last
batch of new products that claimed to fix it, didn't, and the market
is now rife with genuine lies as to whether they did or not.

So, this morning, I tried this. Sorry for the noise on these lists.
Honestly! I totally agree with your assessment of my tone, btw! but I
would rather like the cable industry in particular, to come clean,
with schedules for deployable fixes.

I am off to go fix wifi next, and I do hope that 2+ billion people in
the world - if not the isps, maybe - would like wifi to get better
also, and indeed, I spent the weekend constructively starting to
implement some of the fixes I outlined at the last 802.11 meeting I
attended. That part, on fixing wifi bufferbloat - a much harder
problem than edge bufferbloat - , is a lot of fun! For some info on
what we plan to do there, see:

http://snapon.lab.bufferbloat.net/~d/ieee802.11-sept-17-2014/11-14-1265-00-0wng-More-on-Bufferbloat.pdf

So I took a break from that, reared back, and got some stuff off my chest.

4

I don't see how codel is related to the customer complaint from their perspective. The problem appears to be high latency on downstream with little to no upstream. I'd probably call it off-topic advertising. The only thing that seems to relate to codel is the user's use of bufferbloat in the topic. Nothing the user can do will fix the downstream to my knowledge. Not that I'm extremely knowledgeable on the subject.

Jack

5

Well, with luck probably it will just bounce off their corporate hull and drift into the Kuiper belt.

Say hi to Sugar :wink:

-mel

6

My IP address is apparently now banned from accessing your site at
all, for "advertising", on this thread:

Archived - Virgin Media Community

I don't see how codel is related to the customer complaint from their
perspective. The problem appears to be high latency on downstream with
little to no upstream. I'd probably call it off-topic advertising. The only
thing that seems to relate to codel is the user's use of bufferbloat in the
topic. Nothing the user can do will fix the downstream to my knowledge. Not
that I'm extremely knowledgeable on the subject.

It is 100% possible to fix excessive downstream buffering from some
misconfigured device with a shaper on the download *on the CPE or
home router*.

I have been doing that for 15 years. So has everyone that uses nearly
any of the shapers that are available for Linux, at least.

http://burntchrome.blogspot.com/2014_05_01_archive.html

doing it yourself, right, requires a good measurement, and you lose
just a little bit of single-flow bandwidth - typically 5% - but you
get it all back with faster tcp ramp up times, huge improvements in
dns lookups, voip, gaming, and other traffic.

it generally works way better than policers do.

7

From OP: "However I've recently noticed periods of 500-800ms latency to the CMTS gateway when only using 15-20 of the 60Mbps total (and little to none upstream utilisation)."

I agree with you that it is better to run a shaper that insures your shaper hits saturation and handles queue policies before the upstream does. That is great if it is your pipe (and only its queue) that is saturating. I don't think this problem qualifies.

I find it difficult to believe that he's hitting a buffer bloat issue on a single (not shared with others) queue using 1/3rd of the total bandwidth available to him at those speeds and with that latency value. His problem is more likely lower down (unable to obtain max speed resulting in saturation) or a shared queue where others are saturating it and him applying a shaper will not keep others from doing so.

Jack

8

It is 100% possible to fix excessive downstream buffering from some
misconfigured device with a shaper on the download *on the CPE or
home router*.

From OP: "However I've recently noticed periods of 500-800ms latency to the
CMTS gateway when only using 15-20 of the 60Mbps total (and little to none
upstream utilisation)."

Might be. Again, all I did on that thread was provide a few pointers
to bufferbloat related resources, and pointed at the downlink being a
real problem quite often with links to stuff like this

and they yanked me. Certainly I could have tried again, from another
IP, but ya know, some sundays are more fun than others.

I agree with you that it is better to run a shaper that insures your shaper
hits saturation and handles queue policies before the upstream does. That is
great if it is your pipe (and only its queue) that is saturating. I don't
think this problem qualifies.

Might not. That said, it was hardly an accurate measurement. It is
also perfectly feasible for the upstream device or the downstream
device to be measuring these problems and deal with them
appropriately. It gets progressively easier cpu-wise, as the effective
bandwidth goes down.

It is unfortunately nearly impossible for the next device in line to
do (although we have some tools measuring interpacket "smoothness"
that can provide a hint now, they are not baked yet)

It was my hope, in working on the DOCSIS 3.1 standard that all the
possible downstream problems would be addressed. They weren't.