I was poking around to see what was happening with Cogent and AOL
and ran into some interesting info.
The test that Cogent failed was a 2:1 ratio; Cogent was at 3:1 and
AOL insisted they be at no more than 2:1 for free peering.
AOL wants Cogent to pay for peering - the pricing I've heard is
$50-/meg for paid peering - which I think is more than street price
for transit...
Hmm; I wonder if this change in policy has anything to do with John
Schanz's recent move from Sprint to AOL?
--asp
Further, if L3/Cogent are settlement-free and both parties are interested in
growing the size of their peering connections, wouldn't it make better sense
for Cogent all-around? If AOL is not interested in settlement-free peering
with them, then AOL can pay to get to them.
I seem to remember some old rule of thumb that basically said anyone who
peers with your upstream/transit provider is probably makes sense for you to
peer with (because you are otherwise paying to reach them).
I thought *THAT* was the point of peering vs transit for networks that are
not transit-free.
Deepak Jain
AiNET
Old rules, modern peering decisions arent made with such common sense ideas in
mind but based on power play and a desire for everyone to be your customer!
Connectivity, resilience, even commercial saving all seem to be increasingly
moved to be on a back burner for many peering managers!
I have this at the moment with an operator, we host content their access
customers use and have requested improved connectivity to, I see this provider
via mutual transit.. they wont peer.
Your analysis of the AOL/Cogent situation suggests we're not fully aware of the
facts, either that or they really are stupid!
Steve
Well, it only took the press 9 days to get a story out, I guess
that isn't all bad. The Washington Post now has a story on this
issue:
http://www.washingtonpost.com/wp-dyn/articles/A45819-2002Dec27.html
It claims AOL wants $75000/month. If we use the $50/meg Andrew
Partan posted that would be an even 1.5 Gig, which is an entirely
plausible number for the traffic level (given previous rumor of
2xOC-12, eg 1.2 Gig, recently upgraded to 2xOC-48).
I'll offer two comments from my own opinion:
- Peering should cost significantly less than transit. At least
half, probably less. If you have 1.5 Gig, getting $50 a meg
transit is trivial today. I can't imagine any company paying
$50 a meg for peering, no matter what the circumstances. Perhaps
that was the point though.
- In my opinion, if you want to enforce a ratio and charge people
who do not meet it, the charge should only be on the difference.
That is, say it was 1500 Mbps Cogent->AOL, and 500Mbps AOL->Cogent.
The first 1000 Mbps (2x500, 2:1 ratio), Cogent->AOL should be
free, as they would be if there was less traffic. Charging for
the extra 500, while not something I advocate, would be fair.
To make it such that 1000Mbps would be "free", but 1001 Mbps
means to pay for the first 1000 is just stupid. People don't
generally accept pricing models that have large jumps in them,
they want something progressive.
I wonder what Cogent's response would have been if the charge was
only for the amount over 2:1, and was a reasonable price for peering,
perhaps $15/Meg and AOL gets to pick the locations....
- Peering should cost significantly less than transit. At least
half, probably less. If you have 1.5 Gig, getting $50 a meg
transit is trivial today. I can't imagine any company paying
$50 a meg for peering, no matter what the circumstances.
I'll make one issue about that blanket statement of the price of peering.
Consider this example: If I buy 100Mbit of transit from AboveNet in IAD,
odds are you're gonna peer off 75% of my traffic locally, without it ever
having touched expensive longhaul circuits. If I buy 100Mbit of paid
peering, odds are you're going to be burning longhaul circuits carrying
most of it all over the world, plus the same longhaul carrying it all
back to me.
Depending on your situation, your transport costs per meg could easily end
up exceeding the price you charge for transit, and even more so for what
you would want to charge to peering. Now based on the price you're
charging the customer on the other end, it might still be worth it. But
forgetting about some of your huge costs just because you've already paid
them and you don't have to worry about it until you need to upgrade is
dangerous, and leads to situations like the ones many service providers
are facing today.
There is also a big distinction between what I would call "paid peering",
and "on-net transit". Many of the people I see inquiring about paid
peering are one-location wonders looking to lower their transit cost by
"buying" peering with everyone. This is significantly different from
someone who is in diverse locations, but just needs a little extra to make
the deal worth it. This might mean one side paying for the loops, or as
you suggested paying for usage over a ratio, or otherwise some "price"
which is less than transit.
Perhaps that was the point though.
At this point, I'm inclined to believe AOL is simply flexing their
newfound peering pecker on someone they perceive to be in a weak position.
But who knows, maybe AOL thinks they can make more money by helping drive
Cogent out of business, and inflate the price of transit again. 
Everyone has their own theory about how much to charge and who to charge
it too. Only time will tell who has it right.
In a message written on Sat, Dec 28, 2002 at 05:52:30PM -0500, Richard A Steenbergen wrote:
> - Peering should cost significantly less than transit. At least
> half, probably less. If you have 1.5 Gig, getting $50 a meg
> transit is trivial today. I can't imagine any company paying
> $50 a meg for peering, no matter what the circumstances.I'll make one issue about that blanket statement of the price of peering.
[issue deleted]
Your analysis is not completely wrong when considering only settlement
free peering, or only transit. I was intending to address the
middle case, a settlement based peer. I'm also assuming these
people are close to a settlement free peer.
That is, if you allow a 2:1 peer, and someone comes in at 1001:500
Mbps, charging them the same as the transit price for either the
full 1001 (which I contend is amazingly foolish), or just for the
1 (something I could live with, in some situations) doesn't make
sense. You're trying to even out a perceived inequity, and I will
argue strongly that the cost of moving an extra meg across an
existing peering circuit is _far_ less than moving a transit meg.
All in all, I find ratios an extremely poor way of validating a
peer. I can think of many cases where it is in both parties interest
to peer, but where the traffic might be extremely unbalanced. Yes,
the fact that it is unbalanced can shift costs from one provider
to another, and that's a very real problem to solve. The correct
way to solve it is not to force a transit model though, but to use
careful circuit provisioning and various technical tools to move
the cost back to something more equal. Heck, even a settlement,
much as I hate them, would be better than just turning the thing
off.
What's even funnier is that most people apply them equally in both
directions. They want to make the claim that being out of ratio
(such as in this case) shifts costs onto their network. Well, many
people do the same thing in reverse. If they saw a 1:3 they would
not peer with someone /even though they are shifting cost to the
other party/. I've never understood how someone can argue that a
ratio is about protecting their company from bearing a disproportionate
amount of the cost, and then also prevent their company from shifting
that cost to someone else (assuming the other party would agree).
All in all, I find ratios an extremely poor way of validating a
peer. I can think of many cases where it is in both parties interest
to peer, but where the traffic might be extremely unbalanced. Yes,
the fact that it is unbalanced can shift costs from one provider
to another, and that's a very real problem to solve. The correct
way to solve it is not to force a transit model though, but to use
careful circuit provisioning and various technical tools to move
the cost back to something more equal. Heck, even a settlement,
much as I hate them, would be better than just turning the thing
off.
When Cogent cut off their peering with AOL, AOL customers now find that they
experience lags reaching content providers that use Cogent. Also, of course,
Cogent customers find that AOL's customers have trouble reaching their
content.
I submit, however, that the pain is suffered more nearly equally. Assume for
the moment that AOL is very large and all eyeballs and Cogent is very small
and all content. The argument would likely be made that poor connectivity
between Cogent and AOL hurts Cogent more as a significant number of people
can't reach their customer's sites well.
But I submit that while the harm is lesser to each AOL customer (since only
a small fraction of the sites they might wish to reach are congested), there
are more AOL customers. The aggregate harm or benefit would be expected to be
nearly the same. After all, each delayed or dropped packet harms one customer
of each company.
In general, however, eyeballs are more sensitive to delays. If I get a slow
page load of a Microsoft site, Microsoft isn't harmed as much by that one
delay as I am. So the aggregate benefit to AOL's customers would be expected
to be greater than that to Cogent's.
What's even funnier is that most people apply them equally in both
directions. They want to make the claim that being out of ratio
(such as in this case) shifts costs onto their network. Well, many
people do the same thing in reverse. If they saw a 1:3 they would
not peer with someone /even though they are shifting cost to the
other party/. I've never understood how someone can argue that a
ratio is about protecting their company from bearing a disproportionate
amount of the cost, and then also prevent their company from shifting
that cost to someone else (assuming the other party would agree).
Well the pipes and routers go both ways at the same speed. So the aggregate
cost to set up, say, an OC12 peering connection is best utilized if the OC12
is nearly maxed both ways. It's not wholly unreasonable to say that if you're
going to go through the cost of setting up a peering connection at a
particular speed, you want to move as much total traffic over it as possible.
But, as I'm sure we all know, this whole charade is just about dreaming up a
way to charge someone money.
Currently, traffic between Cogent and AOL seems to be experiencing delays of
under one second each way. There is no packet loss. The situation is quite
tolerable though, of course, it could be better.
Also, one philosophical point that I think is especially important for
network operators to keep in mind. The usual definition of the 'Internet' has
IP in it somewhere. While this may be what currently defines the Internet,
the protocol could change entirely and it would still be the Internet.
The Internet is a philosophy and what that philosophy has brought into
being. The philosophy is about making a genuine best effort to
intercommunicate with anyone else who makes a similar effort. An Internet
product is one that reflects this philosophy, not one that happens to work
over today's Internet. An Internet company is one that operates under this
philosophy, not one that happens to own routers, pipes, or pass IP traffic.
DS
Speaking of this whole Cogent/AOL/Level3 mess.. sigh.
I got tired of trying getting anything out of Cogent. So, here's list of
questions perhaps someone might be able to answer.
1. I'm sure some of you are customers of Level3, and I'm sure
you do see 1-2 sec latency w/ Cogent, what's the official Level3 'position'
if/when you contact them? Do they have any plans upgrading capacity with
Cogent, what's their side of this story in general?
2. I think I asked this before, why wouldn't Cogent prepend
customer prefixes to Level3 or set BGP4 community for multihomed sites,
homed w/ Cogent + someone else.
(This is to control inbound, and please don't go into "this is not-standard
and Cogent won't do it".)
3. Did anyone suggested to Cogent to carry traffic (or some portion of it)
to AOL via MFN to offload its Level3 peering? I couldn't get any straight
answer from Cogent why this can't be done.
4. And another interesting perspective... I'm sure NDAs on peering are
involved, but anyhow -some of us don't really care about AOL that
much, assuming it is only outbound from Cogent into AOL (via Level3) that is
saturated, Cogent may try to push traffic as:
16631_174_3356_ excluding AOL' ASN(s) at one peering location
and keep saturating its Level3 peering connectivity at other locations. Any
thoughts?
-Basil
You got your answer to this before, what part wasn't clear? Cogent isn't
congested in the inbound direction, only outbound to Level 3. The best
they could do is lower their localpref for 3356, which I would surely hope
they have done already.
So are any ISPs pricing transit and/or paid-peering
bandwidth (significantly) lower if purchased at an exchange
point?
Pete.
... trying to even out a perceived inequity ...
peering is a business decision. if it's possible to force another network
into the role of "customer", then that's seen by many as good business since
revenue increases. "paid peering" or even "settlements" are not about
inequity, perceived or otherwise; rather, it's about not leaving money on
the table.
2. I think I asked this before, why wouldn't Cogent prepend
customer prefixes to Level3 or set BGP4 community for multihomed sites,
homed w/ Cogent + someone else.(This is to control inbound, and please don't go into "this is not-standard
and Cogent won't do it".)
This is non-standard and Cogent wont do it. It is their policy not to do
non-standard stuff. To change the policy, you simply need to take control of
Cogent.
3. Did anyone suggested to Cogent to carry traffic (or some portion of it)
to AOL via MFN to offload its Level3 peering? I couldn't get any straight
answer from Cogent why this can't be done.
Same reason as above.
Alex
> ... trying to even out a perceived inequity ...
peering is a business decision. if it's possible to force another network
into the role of "customer", then that's seen by many as good business since
revenue increases. "paid peering" or even "settlements" are not about
inequity, perceived or otherwise; rather, it's about not leaving money on
the table.
The perceived "money on the table" frequently doesn't exist and attempts
to get it may produce the opposite result.
Consider:
A) The former peer may shift the traffic to your transit provider.
B) The former peer may shift the traffic to their transit provider.
In which case the following apply:
* Who they shift the traffic to is now in control of the peering
relationship instead of you (they get to negotiate with the former peer
regarding where and how big of pipes to use for peering).
* Who they shift the traffic to is now a more important part of your
strategic business plan (they have more clout with you when negotiating
and you depend on them more in a risk management sense).
* Who they shift the traffic to may be your competitor.
If you assume the above three cases are costs and you add that to the cost
of the decreased efficiency of traffic to the target network you can
compare it to the probability that you can sell service to the former
peer. Depending on the relationship, you can guess the likelyhood.
Mike.
+----------------- H U R R I C A N E - E L E C T R I C -----------------+
Basil,
If you recall, we talked about purchasing cogent access from you quite
some time ago, as Five Elements is also in the Switch & Data facility.
If you need somewhere to off-load your AOL-bound traffic, we have some
excess Aleron transit, and they have AOL peering of some sort right in
Chicago. As we have excess capacity right now we could do it for a cost
that would be similar to your cogent cost on a month-to-month basis,
provided it does not exceed 40Mb/sec or so, and we'll let you know when
our excess starts to run out and we start to incur cost on it. Most
likely, by that time the Cogent/AOL issue will be resolved anyway, but
it protects us from getting into a long-term deal selling below cost,
while allowing you to improve network performance while maintaining your
current cost structure as long as we have excess that we have to pay
for, anyway. I'm not trying to "pitch" you, just help out 
Here is a traceroute from the router we could put you on. I have a free
100baseT port, or we could put you on a switch if you don't mind.
root@mr0.chcgil1:~$ traceroute -q1 www.atdn.net # us in switch & data
traceroute to atdn.net (64.12.181.62) from 199.166.200.16, 30 hops max,
38 byte packets
1 gige2.mr0.chcgil2.ings.com (199.166.200.2) 0.341 ms # us in equinix
2 ge3-7.as.eqxchiil.aleron.net (205.198.16.141) 0.403 ms
3 ge6-2.ar.eqxchiil.aleron.net (205.198.16.41) 0.365 ms
4 66.185.141.105 (66.185.141.105) 23.778 ms # first AOL TDN hop
5 66.185.148.66 (66.185.148.66) 24.009 ms
6 bb2-vie-P10-0.atdn.net (66.185.152.215) 24.041 ms
7 bb2-rtc-P0-2.atdn.net (66.185.152.116) 24.110 ms
8 bb2-mtc-P8-0.atdn.net (66.185.152.100) 24.661 ms
9 pop1-mtc-P12-0.atdn.net (66.185.143.195) 24.810 ms
10 ow1-mc1-so-0-0-0.atdn.net (66.185.143.202) 24.839 ms
11 172.20.148.22 (172.20.148.22) 25.150 ms
12 172.20.148.22 (172.20.148.22) 25.048 ms !A
I hope you don't mind my inquiry. Drop us a line if you think we can
help provide a stop-gap measure for the Cogent/AOL thing, or whatnot.
Oops. Obviously, I posted this to the list by mistake.
But in any case, for those of you who are "relying upon" cogent pricing
to make your business model work, it should be easy to figure out that
at some point, you might start getting what you are paying for.
If you only have one vendor that can sell you the product you need at
the price you need to make your business work, you are putting all your
eggs in one basket. Your investors and customers should be concerned.
It's time for companies in this situation to stop complaining at cogent
or AOL or the double-secret peering cabal, and start realizing that they
need to make arrangements with other vendors in order to give themselves
the flexibility to avoid problems such as this.
Sorry for the accidental post
Well, if L3 is a transit provider, would it not make sense for them to increase the peering pipes in order to bill their customers more? I am sure there are some smiles there right now.
I understand very well that there is no problem on the inbound of Cogent
from Level3.
For my not-so-bright customers I simply want traceroutes to look good when
they run one from Level3-homed site. Obviously a ~5-7 hops to us looks
really disturbing, try to explain to one of them that there is no problem.
Enough said.
-Basil
Then make a press release for your not so bright customers, and explain
every single detail rather than asking a company to change their entire
routing policy because you're too lazy to educate your customers on how
traceroute shouldn't be used as a tool to gauge performance. And if you
can't explain it to them, then you should step back and look at who you
really have for customers. "Enough said."
Great. Thanks for the PR tip. I never intended them to change "their entire
routing policy" just for my laziness, if you insist 
It's been going on
for 2 and 1/2+ weeks with no definitive date/solution/workaround on when
it's going to be fixed. Now I've made an attempt trying to show what could
be done; perhaps move some butts to actually get something done. Speaking of
my customers or perspective customers, I don't get to choose often who they
are. Maybe you do, I don't.
Happy New Year,
-Basil
After some off-list discussion I think I understand the issue. You do
not want customers who are doing a traceroute from Level3 or one of
their downstreams to see high latency on some of their traceroute hops
going toward you, because you cannot control the egress path of those
ttl_exceeded packets from cogent's network, even though you can control
your own egress.
So the obvious solution is to prepend your advertisements toward cogent,
which will cause them to carry less of your inbound traffic. This has a
negative impact for cogent, because they need that inbound traffic to
justify some of their peering agreements (think, ratios). Supposedly
this is the reason they couldn't keep the ATDN peering, eh? If all
their web host-type customers suddenly start prepending advertisements,
it will cause them to bleed inbound traffic.
If you want to encourage cogent to build a rich community set so you can
prepend only toward Level3, perhaps you should start prepending toward
cogent and make the point with your cogent rep that this is going to
cause them to lose your inbound traffic, and if they gave you more
control over route advertisements, it would not have such an impact.
On the other hand, maybe cogent doesn't want web hosts as customers.