Ungodly packet loss rates

> Do you say "Yes, that's a good idea"? No.

Because I/we don't think it is. The applications and users feed
the network. Demand increases capacity. Ideally/expectedly
the capacity planning is done maturely enough that the network is
built to meet the demand. But this demand is rather large.
People don't foresee properly. Even when they do they ignore
hoping to increase their stretched profit margins.

... and, eventually, the users get fed up with low service levels, and
stop buying, and the network stops growing. That's what's known as
market feedback. If you anticipate the feedback, you can roll with
it and reduce its impact. If you ignore it, it hits you hard.

> Do you say "No, that won't
> work because <x>"?

Erm, maybe I missed something, but I don't see your Add Water
solution to the problem. Are you suggesting we limit customers
that have access to the Internet?

Actually, I did suggest that, or at least I suggested that providers
not take on new customers until they're sure they can reliably service
the ones they have... and, yes, I know that not everybody would do it,
and that the people who didn't do it would gain market share over the
ones who did. That could potentially be fixed by contractual
arrangements among the providers, or by refusing traffic from
"rogues", but that could get you into antitrust trouble. It's obviously
not a trivial problem, and it may not be possible at all, but the
alternative may be collapse.

> No. Do you say "We think we have a handle on the
> problem, and you can expect it to go away soon"?. No.

I hope not. The problem you are seeing is an example of one or
two (or maybe more) poorly connected AS entities. With providers
like that, the rest of the net looks really good. I don't accept
your premise that the majority of the net is broked.

The immediate outage (which, by the way, appears to have been caused
by an overload *within* a very well-connected backbone network) has
been or will be corrected. The problem I seem to see (admittedly based
on spotty information) is that this happens all the time, with many
providers. It's not a matter of one or two providers.

> Do you say "We
> don't think we can make the problem go away no matter what we do, so
> we'll try to do a better job of explaining the expected level of
> service to new users (and to old users who are losing the level of
> service they've been used to)?". No.

Imply a sociological analogy. Do we say that the United States is
broked because we've 4% unemployment? Because we've X murders /
year?

Some people do. The United States, however, is not, except in certain
ritual speeches that nobody believes anyway, marketed as being
perfect. The Internet *is* marketed, by essentially all providers, as
providing useful global connectivity. The implication of everbody's
marketing material is that, by connecting to their networks, you'll
get access to the whole Net, *including* people connected to other
people's networks.

The average person has only the most tenuous grasp of the relationship
between her own provider and the Net as a whole, and providers as a group
haven't done all that much to change that.

> Do you refer me to some existing
> document, prepared either by my own ISP or by NANOG or some other
> group, describing the quality of service I'm to expect, and point out
> to me that what I'm asking for is more than it guarantees? No.

Actually, I will. Your contract.

What, it doesn't say anything on there about quality of service?
Well, why not?

Sigh. In essentially all legal systems, all contracts, by default,
require that the goods or services delivered be largely as advertised,
and that they be useful for the advertised purposes.

I can't find my TLG contract, but I suspect that the contract I have
with TLGnet actually *does* mention quality of service... for the
explicit purpose of stating what is *not* guaranteed.

However, my TLGnet contract also says that either I or TLGnet can terminate
the arrangement at any time. Yes, that means that, as you and others
have suggested, I can go to another provider... if I can find one that
provides better service.

It also means that I can leave the Net completely. I, personally, am
unlikely to do that; I've been doing this for 12 or 13 years now.
I am firmly addicted. Other people may not have my loyalty... especially
if they have no way of finding a provider they have any reason to believe
is better than the one they're leaving... and even less so if they
don't even understand or believe that there's a difference between
providers.

> As far as I can tell, nobody's acknowledged that there's a problem.
> You really seem to believe that the quality of service provided over
> the Internet as a whole, as opposed to within any particular
> provider's network, is acceptable.

Indeed. I look forward to your definition of the Internet.
Contributions to benchmarking the performance of the "internet"
can be directed to the IPPM mailing list...

Well, I don't know if I can define the Internet, but I can give you a
draft of a pretty good working definition of the "Commercial Internet":

   The interconnected set of IP networks operated by organizations
   that use the words "Internet" or "Information Superhypeway" in their
   advertising, and who claim to offer "unrestricted access" thereto.

> I think there's a big difference between complaining about a
> connection "not [being] 100% perfect" and complaining about a huge
> packet loss rate making a path (and indeed all paths between me and at
> least one very major network) nearly unusable. There's even more of a
> difference between complaining about a single incident of such a loss
> rate and complaining about a pervasive pattern of such incidents.

But, you see, this is not our problem. It is the contributor
of the loss's problem. There exist paths that do not have this
problem.

It's your problem because your customers are affected by it. From the
point of view of 99 percent of your customers, any problem on the
network is your problem. If this particular incident doesn't affect
any of your customers, wait a while, and there'll be one that does.
In a very real sense, what you're selling to your customers is the
performance of the entire network, not just your part of it.

Look, I don't know what the internal topology of the Internet is. I
don't know who's connected to whom. I am capable of finding out, but I
have limited time to spend on such matters. Your average customer is
less capable, has even less time to spend, and wouldn't be able to
draw any conclusions that were of any real use to her if she *did*
find out.

It's fine to say that people should choose better providers, but that
only holds water if there's some useful way for them to do that,
and, incidentally, if there really is a difference between the
services offered... a difference that's meaningful to the *user*.

When I chose the provider I use, I took a quick look at their network
map, didn't see anything too obviously out of line in the internal
structure, didn't see anything too bad in terms of who they were
connected to where (within the confines of my very limited knowledge
of the structure of the Net), saw that they had connections to
multiple backbone providers, and decided that was good enough.

If you really want users to move to better providers, get busy on
a meaningful, easy-to-use rating system. It's not going to happen
without one.

Or you could get another provider, and encourage the sites
connected to poor providers to change as well.

Are you seriously suggesting that I go out and research every one of
hundreds of ISPs, so that I know which ones are "poor"? To put it in
your terms, the economic cost to me of such an activity would far
exceed the marginal utility of even the very best Internet
connectivity. One of the problems with the more naive forms of market
theory is that they ignore the cost of getting perfect information.

It's a free market. It's not designed to provide for the common
welfare. It's designed to reward the quick thinking and
resourceful. It's designed to endorse Darwinism.

Er, gee, Wally, I didn't know it was designed at all. I thought that
it grew out of various Enlightenment ideas about people's rights
to dispose of their property as they pleased, and that it gained
acceptance partly because of Adam Smith's work showing that it
would work to the benefit of most people most of the time.

In order to have networks succeed, you must have networks fail.

Huh? Why?

Networks that fail will lose customers and decrease their
potential to attract new ones.

OK, but why is that essential to having other networks succeed.

It's all well documented in many economics textbooks.

Economics is a fascinating study. Applied correctly, it can provide us
with many useful insights into human behavior and institutions.
Unfortunately, many people apply it incorrectly...

          -- J. Bashinski

JBash,

In my experience, the primary technical factors that are hurting Internet
reliability at this time are roughly (in no particular order):

1) Poorly performing routers. I would estimate that 50% of the packet loss
in the Internet today is wholly unnecessary and results soley from
defective gear. The route cache paradigm (and subsequent cache thrashing)
in current cisco product, the lack of memd, missing high performance
interfaces, and the lack of backplane bandwidth are all grave problems. I
will certainly accept some personal culpability here, but cisco has had far
too long to get its act together and has not responded. I refer you to
Mr. Bilger for further progress.

2) Lack of bandwidth. There are frequently cases where the bandwidth
simply does not exist. This may be due to lack of planning, lack of will,
greed, or whatever. Only economics will drive this forward.

3) Poor engineering. The number of ISP's who have no idea what they're
doing is still quite high. I refer you to Greg, Ravi, et. al. for war
stories on what really happens.

4) Lack of routing scalability. You expressed amazement that the BGP mesh
still holds together. My amazement exceeds yours. A new generation of
protocol is needed which, if nothing else, is much easier to manage, and
can scale to larger domains. You know some of the kludgery that was done
to aid this scaling. It's time for a new generation which incorporates
this as the base. Unfortunately, this isn't useful without #1.

Given this _technical_ environment where literally no one _CAN_ succeed in
delivering volume quality Internet connectivity, I'm not surprised that
you've gotten evasive, defensive, and non-sensical replies.

I wholeheartedly agree that the service level of the Internet as a whole is
pathetic, but given the above situation, your efforts are better spent
internally at cisco rather than trying to perform external social
engineering. Once the technical situation improves, the market will
eliminate those who (for whatever reason) are unable to deliver. If the
technical situation does not improve shortly, the market will simply
eliminate the Internet. ;-(

Regards,
Tony

[excellent technical summary deleted for bandwidth conservation]

Given this _technical_ environment where literally no one _CAN_ succeed in
delivering volume quality Internet connectivity, I'm not surprised that
you've gotten evasive, defensive, and non-sensical replies.

In addition to this, even for very savy customer, it's very difficult to
measure a good network. Tools that are available are at best incomplete, and
comprehensive metrics to judge a good network are very hard to come by. So
while in normal situations, economic factors will weed out worthless
providers, in the current climate, this isn't happening because you can't
measure/prove why provider X provides better service then provider Y, even if
they could engineer it.

All an end user is left is a murky feeling that their provider might be
broken. Hopefully, along with technical advances, we'll get advances in
network measurement area (hopefully IPPM effort of IETF will bear bountiful
fruit) so will force providers to the "right thing" or face loss of business.

-dorian

All an end user is left is a murky feeling that their provider might be
   broken. Hopefully, along with technical advances, we'll get advances in
   network measurement area (hopefully IPPM effort of IETF will bear
   bountiful fruit) so will force providers to the "right thing" or face
   loss of business.

Yes, but the end user need not be the only metric that a provider can gauge
itself by. For example, the provider has the very nice metric of "trouble
tickets per day per billi-packet". This gives him a real gauge of how many
problems his network is seeing. If the customer is happy, then the
provider must be doing something right...

Tony

The only problem with this metric is that under heavy load, it tends to have
high packet loss. Which is, of course, where over-engineering comes in... :wink:

-dorian