BBN/GTEI

Perhaps the best solution here is for BBN to purchase Exodus.

You mean GTE...

> However, there is a more general issue here. If a company primarily hosts
> websites, then their traffic will be asymmetric with many more bytes going
> out than coming in. They must engineer their network to handle this
> asymmetry and any transit providers they have must also do so.
> Traditionally, peering only occurred between providers with a national
> network with at least 4 points of contact, spread out geographically. This
> requirement is so that each provider carries a roughly equal load of the
> traffic across expensive national longhaul circuits. It is this balanced
> transit load that makes them peers.

Lets not forget here that shortly after GTE picked up BBN, they also
picked up Genuity. Genuity still pretty much exists as an independant
entity. So when that is taken into account, GTE ALSO does web
hosting.. They just seperate the traffic off from the "BBN" traffic.

> In the case of a web hosting company, the bulk of the traffic is outgoing
> and if they do the standard shortest-exit routing with their peers, then
> the peer will be carrying a much larger transit load than the webhosting
> company. Of course, this could be solved by both parties doing
> longest-exit which places the largest transit load on the webhosting
> provider. But there is more.
>
> When the webhosting network touches down at only a few exchange points to
> peer with a provider who has an extensive network you will have a
> situation in which the peer is providing regional transit for free and
> must engineer their regional networks to deal with an asymmetric traffic
> pattern. For instance, if a webhosting provider touches down at San Jose,
> Chicago, DC and Dallas then they appear to meet the eligibility
> requirements for peering. But these peers end up providing full transit
> from Boston to DC, LA to San Jose, Denver to Dallas, and St. Louis to
> Chicago. This arises because one provider hosts lots of equipment close to
> an exchange point and the other provider interconnects many POPs in many
> cities.

When you consider a web hosting company, often there are not the
resources to transit traffic to the best exit. Many companies buy
transit from one or more companies anyhow to be sure that they have
maximum reachability. If, for example, a company buys transit from MCI
and uses that to reach BBN, than it will have an impact on the traffic
ratios for the BBN/MCI interconnects. If enough similar companies take
MCI to get to BBN then it is entirely possible that the BBN/MCI
interconnect will become highly asymetric. So takeing into account
what just happened with Exodus, does this mean that BBN would try to
force MCI to buy transit from BBN?

This only goes so far before the whole thing breaks down. Granted that
the above is somewhat extreme example but it is most certainly
possible. The whole point behind private interconnects is to get
traffic to the destination with as few intermediary hops as possible
and to be able to better control congestion and loss between the
networks. If BBN's customers have trouble getting to certain popular
sites, they're going to complain to BBN. If the hosting company's
customers can't be accessed from certain sites, they're going to
complain to the hosting company. This means that both companies tech
support and operations centers get additional calls, additional work,
and additional headaches in trying to deal with traffic levels of some
third party. In many ways, its the third party that ultimately suffers
as they try to deal with both companies to resolve the issues.

IMO, asymetry is not a valid requirement for a private
interconnect. NOT having the interconnect hurts both companies.

web@typo.org (Wayne Bouchard) writes:

IMO, asymetry is not a valid requirement for a private
interconnect. NOT having the interconnect hurts both companies.

it hurts them asymmetrically though. or, that's the bet at any rate.

regardless of whatever we all charge or pay that's measured in bits
per second per month or whatever, the costs are all in terms of bit-
miles. ultimately this means you can only peer with someone whose
costs are similar to yours, almost regardless of the benefit they
receive from peering with you.

So takeing into account
what just happened with Exodus, does this mean that BBN would try to
force MCI to buy transit from BBN?

No. It means that BBN would bill MCI for the transit component of the
traffic exchange that would be measured and priced by an industry agreed
measurement methodology and pricing scheme. All peering agreements would
be the same in this scenario. Once you get accepted as a peer then you
would pay only for the transit that you use that is outside the accepted
rule of thumb for balanced peering which could be somthing like a 1.5:1 or
2:1 traffic ratio at the exchange point. Presumably, a pair of peers like
MCI and BBN would agree to settle these bills on an aggregate basis so
that MCI could pay a $2000 peering charges bill from BBN by offseting it
against a $1500 bill sent to BBN along with $500 cash.

If BBN's customers have trouble getting to certain popular
sites, they're going to complain to BBN. If the hosting company's
customers can't be accessed from certain sites, they're going to
complain to the hosting company. This means that both companies tech
support and operations centers get additional calls, additional work,
and additional headaches in trying to deal with traffic levels of some
third party. In many ways, its the third party that ultimately suffers
as they try to deal with both companies to resolve the issues.

However, if both companies have a scalable peering agreement with each
other, this problem becomes merely a problem of engineering for the
traffic levels. The payment aspects are settled automatically by applying
the settlement rules.

If such scalable peering already existed, I'm convinced that the current
situation between Exodus and BBN would not have developped. So while those
two companies figure out how to handle their relationship, maybe we could
all learn from this and figure out a way to make peering work in a more
scalable manner.

> So takeing into account
> what just happened with Exodus, does this mean that BBN would try to
> force MCI to buy transit from BBN?

No. It means that BBN would bill MCI for the transit component of the
traffic exchange that would be measured and priced by an industry agreed
measurement methodology and pricing scheme.

Hey Michael,

Wasn't it you on the inet-access list suggesting:

>> If you don't peer with or buy
>> transit from GTEI, sit back, shut up, and watch and learn (seriously
>> WATCH and LEARN).

So, were the above comments meant to reference everyone *except* you?

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
Patrick Greenwell (800) 299-1288 v
         Systems Administrator (925) 377-1212 v
                           NameSecure (925) 377-1414 f
\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/

Wasn't it you on the inet-access list suggesting:

> >> If you don't peer with or buy
> >> transit from GTEI, sit back, shut up, and watch and learn (seriously
> >> WATCH and LEARN).

Nope. Wasn't me. The above was quoted by me in a message.

So, were the above comments meant to reference everyone *except* you?

Nope. They were for everyone except *YOU*.

> So takeing into account
> what just happened with Exodus, does this mean that BBN would try to
> force MCI to buy transit from BBN?

No. It means that BBN would bill MCI for the transit component of the
traffic exchange that would be measured and priced by an industry agreed
measurement methodology and pricing scheme. All peering agreements would
be the same in this scenario. Once you get accepted as a peer then you
would pay only for the transit that you use that is outside the accepted
rule of thumb for balanced peering which could be somthing like a 1.5:1 or
2:1 traffic ratio at the exchange point. Presumably, a pair of peers like
MCI and BBN would agree to settle these bills on an aggregate basis so
that MCI could pay a $2000 peering charges bill from BBN by offseting it
against a $1500 bill sent to BBN along with $500 cash.

Why should they?

In fact, what you're advocating is billing the sender for *solicited data*
from the recipient's point of view! The end result of this will be
discontinuities in the network - since nobody in their right mind is
going to allow SOMEONE ELSE to write blank checks on THEIR account, when
they have no means to get that back from the party (since they have
no contract with the other side).

The result? You connect to a web site and get "XXX is asking us to pay for
what you already bought - please enter you VISA number at the prompt" in the
requestor..... or just a "XXX is being a jerk; go away!"

However, if both companies have a scalable peering agreement with each
other, this problem becomes merely a problem of engineering for the
traffic levels. The payment aspects are settled automatically by applying
the settlement rules.

You've got the problem backwards Michael.

Not at all. I am advocating paying for transit. When A and B use roughly
the same amount of each other's transit, there is no point in counting the
difference. But when you have an asymmetric situation, rather than cutting
off peering altogether because the other guy is too different, why not
have a scalable peering situation that directly addresses the asymmetry in
traffic flows. The only other solution that I can see is for the network
receiving the huge incoming flow to direct all that web traffic through
transparent web caches at each exchange point. However that just raises
increased barriers to peering and does not deal with non-cacheable
dataflows which are increasing over time. Scalable peering would reduce
the barriers to peering and make it easier for new players to buy in. They
would still have to build a truly national network, but at that point they
could not have the door slammed in their faces.

Regardless of whether my proposed solution is the correct one or just a
bad idea produced by indigestion, you cannot deny that the asymmetry
between networks is increasing as network providers specialize the
services they offer. The old-fashioned rough-cut peering is becoming more
and more unsuitable as the only peering option. We need new ways to do
this. Somebody has to take the first step. Somebody has to be a pioneer.

The details can always be hashed out later.

Has anyone yet come up with any good metrics for route-miles a packet
must pass through in one direction or another?

Right now, the closest thing I have seen to that is MEDs in a BGP
announcement, set by a provider who knows a little something about how
his network is built.

The problem I see is that if we base settlement on how the MEDs are set,
an unscrupulous peer could change his MEDs to offset a portion of his
transit or something along those lines.

Or every content provider could buy a same-sized access provider or vice
versa, and solve the problem that way.

-Deepak.

> In fact, what you're advocating is billing the sender for *solicited data*
> from the recipient's point of view!

Not at all. I am advocating paying for transit.

On the contrary.

If I buy a DS1 for transit from your network, I'm expecting the person I pay
to provide transit - ALL OF THE TRANSIT.

That's what I'm buying!

Now what you're saying is that "oh, no, that's not really what you bought"!

I'm sure the thousands of DS-1 connected CUSTOMERS (that is, transit
purchasers) will find this a very, very interesting interpretation.

Regardless of whether my proposed solution is the correct one or just a
bad idea produced by indigestion, you cannot deny that the asymmetry
between networks is increasing as network providers specialize the
services they offer. The old-fashioned rough-cut peering is becoming more
and more unsuitable as the only peering option. We need new ways to do
this. Somebody has to take the first step. Somebody has to be a pioneer.

No, its actually becoming MORE suitable. Instead of burning the entire
circuit in both directions, you're only burning half of it now (one
direction).

The details can always be hashed out later.

You're simply wrong, once again.

> > In fact, what you're advocating is billing the sender for *solicited data*
> > from the recipient's point of view!
>
> Not at all. I am advocating paying for transit.

On the contrary.

If I buy a DS1 for transit from your network, I'm expecting the person I pay
to provide transit - ALL OF THE TRANSIT.

Of course, and I agree with you 100%. But I was not talking about a
transit customer. I was talking about a peer whose traffic interchange is
asymmetric and who therefore uses some regional transit in the other guy's
network. I'm saying that instead of slamming the door in his face and
telling him to buy transit, we need to have a scalable peering option that
is a blend.

Maybe I am headed in the wrong direction with this but I do believe we
need a better solution for peering with asymmetric peers that reduces the
barriers to entry to $$$. Right now there are barriers to entry that
probably will not pass the scrutiny of the DOJ.

No, its actually becoming MORE suitable. Instead of burning the entire
circuit in both directions, you're only burning half of it now (one
direction).

You still have to pay for the whole circuit.

> > > In fact, what you're advocating is billing the sender for *solicited data*
> > > from the recipient's point of view!
> >
> > Not at all. I am advocating paying for transit.
>
> On the contrary.
>
> If I buy a DS1 for transit from your network, I'm expecting the person I pay
> to provide transit - ALL OF THE TRANSIT.

Of course, and I agree with you 100%. But I was not talking about a
transit customer.

YES YOU ARE!

Let's look at the instant case - BBN <> Exodus in a PEER relationship

BBN claims that Exodus sends more traffic (by a LOT) than they send.
To which I say "so what" - BBN'S TRANSIT CUSTOMERS ASKED FOR IT!

Now, if BBN wishes to claim that they have the right to bill Exodus for that
which THEIR TRANSIT CUSTOMERS ASKED FOR, I say "bullshit". Further, I
suspect that BBN's transit customers, upon learning that Exodus refuses to
pay, will ALSO say *bullshit* and demand that BBN either carry the traffic,
as they PAID for it be carried, or they will find a provider that will!

Now if BBN has MISPRICED their transit, that's THEIR problem. To try to
offload that problem on their PEERS, when those peers are sourcing that
traffic at the EXPLICIT request of their customers, is IMHO nuts.

I was talking about a peer whose traffic interchange is
asymmetric and who therefore uses some regional transit in the other guy's
network. I'm saying that instead of slamming the door in his face and
telling him to buy transit, we need to have a scalable peering option that
is a blend.

You're simply wrong here. No traffic passing over a peer connection is
"using some transit that is not paid for". That's an argument that in its
logical conclusion (two or three levels down the tree of logic) means that
someone has been ripped off and/or deceived - that they paid for something
that the provider in question doesn't intend to provide!

ALL traffic passing between peered providers is BY DEFINITION sourced by a
TRANSIT customer of one provider and sunk by a TRANSIT CUSTOMER of the other
network. The correlary to this is that all traffic passing between peered
providers was REQUESTED by one of the transit customers of one of those
networks, who PAID for that transit to be provided to them!

THIS is why claiming that "well, you're asymmetric so you pay!" is baloney.

Were I to find myself in that position my response would be "Bite me!",
followed same-day by a letter to EVERY ONE of the denying provider's
customers - on paper - announcing that their provider had just tried to bill
me for what the target of those letters might think they had already bought
and paid for!

The fallout from THAT would be immediate and severe.

Maybe I am headed in the wrong direction with this but I do believe we
need a better solution for peering with asymmetric peers that reduces the
barriers to entry to $$$. Right now there are barriers to entry that
probably will not pass the scrutiny of the DOJ.

NO barrier to peering will pass the scrutiny of the DOJ. Sorry, that's
the facts. A peer relationship is by *definition* not assignable as
something which has disparate value to one end or the other, since by
definition each provider involved is being paid to provide transport by
their customers, and those customers won't be if they can't get where they
want to go.

> No, its actually becoming MORE suitable. Instead of burning the entire
> circuit in both directions, you're only burning half of it now (one
> direction).

You still have to pay for the whole circuit.

And you have the unused half over which you can run other traffic.

>
> > So takeing into account
> > what just happened with Exodus, does this mean that BBN would try to
> > force MCI to buy transit from BBN?
>
> No. It means that BBN would bill MCI for the transit component of the
> traffic exchange that would be measured and priced by an industry agreed
> measurement methodology and pricing scheme. All peering agreements would
> be the same in this scenario. Once you get accepted as a peer then you
> would pay only for the transit that you use that is outside the accepted
> rule of thumb for balanced peering which could be somthing like a 1.5:1 or
> 2:1 traffic ratio at the exchange point. Presumably, a pair of peers like
> MCI and BBN would agree to settle these bills on an aggregate basis so
> that MCI could pay a $2000 peering charges bill from BBN by offseting it
> against a $1500 bill sent to BBN along with $500 cash.

Since I don't have Michael's original post, I'll just respond to this
quoted portion:

And any suitably informed party to a peering arrangement like this can
simply offer free hosting to search engine crawlers until they are
extracting the desired amount cash from their peer via settlements.

HECK with peering agreements like this, you could PAY search engine
crawlers to host with you. :slight_smile:

Cool!

Mike.

ps. If it isn't complete obvious at this point, the moral of the story is
that if you decide to pay people for any measure of traffic flow you have
given them incentive to provide you with exactly what you want and they
will make sure you end up paying them.

pps. In otherwords, if it involves money you can't help incentivizing some
form of behavior.

+------------------- H U R R I C A N E - E L E C T R I C -------------------+

Folks,

Think very carefully about what happens as traffic increases...

Given that the context is the introduction of "settlement-based" peering
for traffic imbalance in a shortest-exit world, then all of the various
schemes for generation of traffic (web crawlers, backup services, etc)
are no problem whatsoever, and may actually been seen as desirable...

Background:

  Imagine a nationwide Internet provider with customer A on the west
  coast and customer B on the east coast. Customer B sends queries
  to customer A's server, and receives quite a bit of data in return.
  (all of this traffic stays on the provider's backbone in this case)
  Presume for the moment that the provider charges the customers (A&B)
  on a usage-basis which takes both sending and received traffic levels
  into consideration. Now, one could charge each of A and B the full
  cost of the traffic exchanged (which would effectively be "double
  dipping" and quite profitable :slight_smile: but a competitive market makes it
  far more likely that the rate per traffic measure will be such that
  A & B together pay the costs associated with their traffic exchange

  Now, given that A & B are both on the provider's network, everything
  is fine. When we hypothesize that A is instead connected to a peer
  network, we don't know whether costs will be recovered. In a shortest-
  exit world, traffic is quickly exchanged to the original provider's
  network and hence the costs of transmission to B are almost identical
  to the case where the A was a customer. Given that there's only one
  customer (B) now paying for these costs, this presents a potential
  problem. Customer B expects to pay their fair share of the costs,
  but doesn't expect to pay a larger portion of the total costs simply
  because A is connected to another provider.

  The truth is, there is actually nothing wrong with this situation if
  the traffic to/from the peer is of the same order of magnitude. To
  elaborate: a peer provides traffic which must be carried without
  corresponding sender-side revenue. On the other hand, the peer also
  allows the Internet provider to provide a similar level of sender-paid
  traffic which (by nature of being handed off) has very little actual
  transmission costs. This is why typical peers are overall neutral
  to cost recovery, and why peers which send far more traffic than
  received are problematic.

  There are a few options that have been mentioned to this situation,
  including the "receiver pays full freight / sending is free" model,
  the use of something other than shortest-exit routing, and introduction
  of settlements for such traffic. For purposes of this thread, we're
  trying to determine if the settlement approach is doomed to failure
  due to ability to generate additional traffic flows through innovation.

Assessment:

  Presuming that a peer which is a net exporter of traffic begins to
  address this by encouraging additional traffic from customers to its
  network, this would result in more sender-paid originating traffic
  which quickly moves to the peer. Such traffic is (as noted above)
  quite profitable, as it includes full cost recover but little actual
  costs due to shortest to the peer. This is actually good for
  both networks, and us back to an Internet where more traffic is
  considered a good thing by ISP's at both ends of the connection.

/John

p.s. The fact that the sender of traffic should be paying some portion
      of the resulting costs is not a surprise to anyone; many of the
      content companies that I've spoken to believe they already are
      paying more as traffic increases, and were quite surprised to
      find that it doesn't actually make it to the networks which
      bear the brunt of the traffic carriage.

p.p.s. As noted, departure from shortest-exit is also another approach
        which may provide some answers to this situation, but that's a
        different topic which deserves its own thread. This message
        is simply noting that settling for peering traffic is quite
        viable, despite assertions to the contrary regarding traffic
        generation. As long as you're billing the senders on your
        network for increased usage (and handing it off shortest-exit),
        increased traffic is good thing.

All:

The issue of asymmetric data flows is one of the reasons why we are building
an optical Internet here in Canada. With an optical Internet you can
traffic engineer individual wavelengths to support Tx/Rx asymmetric data
flows in any way you want. This makes much more efficient use of the
underlying fiber indrastructure.

For more information

www.canarie.ca
www.canet2.net

Bill

There can be no 'request' without associated 'acceptance' of that requet.
Therefore we have two 'half-links' where the content provider thinks it
beneficial to pay for the other half-link in order to get its traffic
delivered to the content consumer paying the other half. A corporate website
covers its cost of the half-link in product sales while a more serious
content provider sells subscriptions.

A colo charges the website publisher not only for rack space etc. but also
to provide the half-link to content consumer. So the colo has the fees and
will pay if peering is asymmetric.