I'm sorry to interrupt the discussion of "how long is your rack?" and
"what do you do when your home ISP is down?" with something impacting
some folks' cost and manner of selling services, however Level 3 just
published its new peering guidelines, buried in comments on the L3/GX
merger:
http://fjallfoss.fcc.gov/ecfs/document/view?id=7021703819
http://fjallfoss.fcc.gov/ecfs/document/view?id=7021703818
FYI,
-a
In a message written on Fri, Aug 19, 2011 at 04:29:05PM -0400, Adam Rothschild wrote:
I like to see Level 3 arguing this with the regulators. AboveNet
persued this line of thinking with a number of ISP's in the late
1990's with some success, and I believe others did as well. AboveNet
implmented it by honoring MEDs from peers, and thus doing a cold
potato routing and carrying a higher bit-mile cost.
Ratio is the most broken part about modern peering agreements.
Ratio really has no bearing on the costs to either ISP, it is an
artifact of their position in the world. That is to say the type
of ISP (end user, content) location (urban, rural) or technologies
(dsl, cable, leased line) along with user behavior determine the
ratio. Early ISP's that had similar customer mixes, locations, and
technologies could use ratio as an easy proxy, but those days are
long gone.
The primary challenge to change is the technical community coming
up with some metric that is easy to measure and senior management
can understand. You can go to a VP and say "the ratio to them is
1.5:1" and they get it (or so they think). Trying to make the same
argument that on some vague level you are deriving "equal benefit"
is much larger. I like Level 3's effort in using the bit-mile cost
but I don't know any way to measure that metric easily on a large
network.
In a message written on Fri, Aug 19, 2011 at 04:29:05PM -0400, Adam Rothschild wrote:
I like to see Level 3 arguing this with the regulators. AboveNet
persued this line of thinking with a number of ISP's in the late
1990's with some success, and I believe others did as well. AboveNet
implmented it by honoring MEDs from peers, and thus doing a cold
potato routing and carrying a higher bit-mile cost.
Yes, Above.Net broke the original peering-ratio fight that way. Thank you for that. Too bad it didn't last.
Ratio is the most broken part about modern peering agreements.
Ratio really has no bearing on the costs to either ISP, it is an
artifact of their position in the world. That is to say the type
of ISP (end user, content) location (urban, rural) or technologies
(dsl, cable, leased line) along with user behavior determine the
ratio. Early ISP's that had similar customer mixes, locations, and
technologies could use ratio as an easy proxy, but those days are
long gone.
There are many broken parts of 'modern' peering agreements.
"You must spend as much on your backbone as I spend on mine." "You must be on at least 3 continents (no matter where the traffic goes to or from)." "'Peer' means 'equal'." "You have to be wearing a blue hat when you request peering." Whatever.
I've said it before, more times than I can count. Peering is a business tool, a means to an end. The goal, the 'end', of for-profit companies is to make a profit. Sounds obvious, but surprising how many people forget this. If peering with another network will increase your profit and you turn down the peering request, you should be fired. Your ego should not be substituted for business decisions.
Some people rationalize these decisions (read: "con themselves into believing") by saying if they do not peer with a network, they will gain revenue from that network. It even works sometimes. Most of the time it does not work, and in more & more cases ensures no revenue will -ever- flow between the companies.
Just so no one jumps all over me, I am not even hinting that peering requests must always be accepted. There are rational, business reasons to turn down peering. But a one-size-fits-all policy is idiotic, as are most of the reasons peering requests are denied these days.
More importantly, the real issue is power (leverage, whatever you want to call it). In the 90s, all users had to do to switch ISPs is change a few digits in their modem dialer. Content owners had to do quite a bit more to move a web site between hosting providers.
These days, it is very difficult for end users to change broadband ISPs, frequently requiring equipment changes (e.g. DSL -> Cable), moving phone & TV, etc. Worse, for many people in the US, they have only one provider who can give them more than 1.5 Mbps.[*] Content owners can move hosting companies easily.
Not hard to figure out who had more power in the 90s, and who has it now.
The primary challenge to change is the technical community coming
up with some metric that is easy to measure and senior management
can understand. You can go to a VP and say "the ratio to them is
1.5:1" and they get it (or so they think). Trying to make the same
argument that on some vague level you are deriving "equal benefit"
is much larger. I like Level 3's effort in using the bit-mile cost
but I don't know any way to measure that metric easily on a large
network.
Having a metric sr. mgmt can understand will not change which company has power. Companies that have power -will- exercise it to make money.
L3 is on the wrong side of the power equation today, so they are trying to move the discussion to something that has real business logic. Their only hope of the leverage some other companies have is to either get the gov't to step in (EWW!), or pray the threat of gov't regulation will be enough. Dangerous game, if you ask me.
I tried getting some data on that a few weeks back when reading the FCC Broadband Report. The best I could find was below which still leaves the users vs. markets unclear (which most likely skews the end result):
Today, 290 million Americans—95% of the U.S. population—live in housing units30 with access to terrestrial, fixed broadband infrastructure capable of supporting actual download speeds of at least 4 Mbps.31 Of those, more than 80% live in markets with more than one provider capable of offering actual download speeds of at least 4 Mbps.32 Meanwhile, 14 million people in the United States living in 7 million housing units do not have access to terrestrial broadband infrastructure capable of this speed.33 Although housing units without access to terrestrial broadband capable of 4 Mbps download speeds exist throughout the country, they are more common in rural areas (see Exhibit 3-D).34
Above is from http://www.broadband.gov/plan/3-current-state-of-the-ecosystem/ which I got referenced from the FCC Broadband Report. That further references the OBI report on broadband availability gap which I never finished reading so more details might be in there.
/nco
In a message written on Fri, Aug 19, 2011 at 05:40:43PM -0400, Patrick W. Gilmore wrote:
Yes, Above.Net broke the original peering-ratio fight that way. Thank you for that. Too bad it didn't last.
I forgot to write back when you first posted this, but the recent
follow ons reminded me...
While it is not a wide spread practice, I am 99.99% sure several
networks are still using this method in private. I also know of
at least two networks that have in the past deaggregated their large
supernets to peers to affect a similar result.
The interesting thing is that this is not done to elminate the
peering ratio argument in the aggregate, but to make up for the
geography of the two networks. Maybe one network insisted on peering
in a city that the other only has a few customers in, so the link
is mostly idle while others run too hot. Rather than provision
more bandwidth they agree to meds or deaggregates to get the traffic
to the lightly loaded link. The ratio in the aggregate is basically
unchanged.
I've said it before, more times than I can count. Peering is a
business tool, a means to an end. The goal, the 'end', of for-profit
companies is to make a profit. Sounds obvious, but surprising how
many people forget this. If peering with another network will
increase your profit and you turn down the peering request, you
should be fired. Your ego should not be substituted for business
decisions.
I agree with your premise, and think everyone involved _thinks_
they are making the best business decision for their company.
However I believe many folks are in fact making poor decisions for
their company because they lack data. It's interesting to look at
the first order costs of peering; the port, cross connect, and time
to set it up. In my experience though the second and third order
effects are far more important; how it affects your ratio with other
peers, your customer's perception of your network performance, etc.
These are much harder to measure, and near as I can tell virtually
no network does.
I've been asked many times how the government could step in and
help, aka regulate peering to make it better. In thinking about
all the things that could be mandated, the interactions that can
be regulated, I see them generally as a univeral bad. With one,
major, glaring exception....
Right now peering is all done in secret. You don't really know if
your competitor is getting free peering, paying to peer, or getting
paid to peer. You don't know if they are really insisting on the
same locations with other parties they insist with for you. It's
a perfect game of poker. This is in stark contrast to say, transit,
where you can get prices from 20 people and compare.
AboveNet also used to put graphs of almost all of its peering links
online, in real time. A few peers flat out refused, but most were
there. You could see where they peered, at what speed, and how
much traffic was flowing. You could also calculate the ratio.
I think the Government could bring a lot of sunshine to the party
if it simply required all ISP's to do just that. Post a graph of
every peering link or port to a public exchange, updated in real
time. They all have the data internally, they are all collecting
it for their own needs. You still wouldn't know if any money is
changing hands, but imagine how it would change many of the
interactions. Oh, you require me to hit 3Tb aggregate, when you
have 20 peers that are less than half that? Really, you're going
to require me to peer with you in Nome Alaska when you don't have
any other peers at that location?
It would also let ISP's make an informed decision about depeering
events. The next time cogent splits from someone you can go look
at the graphs on both sides, and make your own decision who was
being more reasonable. It would also let you, the transit customer,
evaluate the extent and free capacity to peers your ISP has before
buying.
I have yet to find a down side to this sort of sunshine. I'd wecome
anyone who thinks its a bad idea to educate me.
I have yet to find a down side to this sort of sunshine. I'd wecome
anyone who thinks its a bad idea to educate me.
Why is that any different than forcing businesses to explain which links are paid? Or any other internal data? Private businesses are private. Their relationships with other private businesses are private. Saying some data should be public while others are not is arbitrary.
Next time Cogent de-peers someone, customers do not care who was being more reasonable. They care that their links are broken. Customers do not care if links are out of ratio, they care if packets are dropped. Further there are companies who can push 75 Gbps on an 8x10 LAG for 12 hours without dropping a packet, while other companies cannot push 60 Gbps on that same link without some intermittent packet loss. Explaining to customers which are 'good' and which are 'bad' is an impossible task.
Finally, if this were to happen, a lot of links would be quickly disconnected. Perhaps that could be considered a feature, but I am not at all certain it would be.
In summary: I do not like gov't involvement, even if it is just to require some random set of data to be made public.
Wouldn't that mostly affect people who are silly enough to single-home to a Tier-1 that
gets involved in a peering dispute?
Which $PROBLEM does not affect only those who are "silly enough" to not put mitigation tools for $PROBLEM?
And the customers still don't care. They just care _that_ it affected them - at least during the problem. Although one can hope they care enough to change their behavior afterward.
What are thoughts on public disclosure limited to capacity constraints?
There is ample business reason for making the terms of specific
interconnects private. On the other hand, knowing definitively that
{mon,du}opoly broadband provider A is running its connections to
transit provider B hot could be in the public interest, and allow
operators to make informed routing decisions. Bringing these metrics
into the public light might also encourage operators to upgrade more
responsibly, though this could be wishful thinking on my part. 
(This is entirely food for thought, I've not yet formed any opinions.)
-a
And yet, people still single-home to Tier-1s. Go figure.
Because they care and don't care at the same time, although most don't
want to be bothered to spend the money on a second provider.
~Seth
People still single home to "tier-2s" that have a single transit provider - which is objectively worse.
People still smoke too. Go figure.
In a message written on Fri, Aug 26, 2011 at 09:32:00PM -0400, Patrick W. Gilmore wrote:
Why is that any different than forcing businesses to explain which links are paid? Or any other internal data? Private businesses are private. Their relationships with other private businesses are private. Saying some data should be public while others are not is arbitrary.
It is, but it's also in the effort of allowing consumers to be
informed. Why do we require food items to list the ingredients?
Why do public companies have to disclose particular events to the
SEC? Why do car companies have to list the MPG on their cars?
The FCC is very interested right now in finding out if when ISP X
says "your service is 16M down" you can get 16M. Peering has an
impact on that. If I have a 16M service, and every single pering
link they have is flatlined 10 hours a day I won't be able to get
16M down for those 10 hours.
In a message written on Fri, Aug 26, 2011 at 09:32:00PM -0400, Patrick W. Gilmore wrote:
Why is that any different than forcing businesses to explain which links are paid? Or any other internal data? Private businesses are private. Their relationships with other private businesses are private. Saying some data should be public while others are not is arbitrary.
It is, but it's also in the effort of allowing consumers to be
informed. Why do we require food items to list the ingredients?
Why do public companies have to disclose particular events to the
SEC? Why do car companies have to list the MPG on their cars?
While I see some similarities, I do believe there are differences between food ingredients & financial disclosures and peering ports. I guess one could argue MPG is a comparable metric, but it's not really a useful number. (Perhaps why it is comparable? 
The FCC is very interested right now in finding out if when ISP X
says "your service is 16M down" you can get 16M. Peering has an
impact on that. If I have a 16M service, and every single pering
link they have is flatlined 10 hours a day I won't be able to get
16M down for those 10 hours.
Without disagreeing with your last point, we all know it will not be that simple. Suppose I have 100 peering partners and 3 are congested. Further assume those 3 are the three smallest peers I have, and refuse to upgrade because they do not have the cash (or whatever)? How many customers are going to be mad at me, even though it likely has zero effect on them? And what should I do about it? (If your answer is "explain to them why this is not a problem", then I submit you have never spoken to a pissed-off, irrational customer.)
Some more questions:
Are you going to limit this to just networks that sell consumer access?
What will you do if the peering links are outside the US?
Will some countries become havens for peering (like Lichtenstein is for banking)?
How many links do you think will be dropped if this rule is put into place?
Are transit ports required to be shown?
Etc., etc.
That said, I'm not going to Washington to lobby against it if this law is written.