Hi NANOG
We are a group of researchers and our focus is on the economy of
interconnection in the Internet. My question is mainly about the various
costs of an AS establishing a connection with another AS, including the
costs charged by the colocation providers. I am familiar with most of the
connection options such as public peering on IXP, and private peering
through xconnects. My understanding is that in addition to the cost of
transit that the smaller AS pays to the larger AS, in the former you pay a
monthly fee to establish a link to the switching fabric and then you can
connect to as many ASes that are member in the IXP, and in the later you
need to pay for as many xconnects that you need to connect to as many ASes
that you plan to peer with. Obviously in both cases there is the cost of
being in the colocation and renting a rack or whatever. What are the other
costs involved? How should the AS reach the colocation center in the first
place? I don't think every network can dig a hole an lay cables. Who should
they pay to get from one PoP to another? Do ASes have to pay for xconnect
to connect their PoP in a data center to the rest of their network?
I think there is no single answer as different businesses may have
different pricing models. I hope the discussion can help me understand the
whole ecosystem a little bit better.
Best Regards
Reza Motamedi (R.M)
Graduate Research Fellow
Oregon Network Research Group
Computer and Information Science
University of Oregon
Hi Reza,
I have a list of example items that need to be costed in below, it is
by no means a definitive list though:
https://docs.google.com/document/d/1i2bPZDt75hAwcR4iKMqaNSGIeM-nJSWLZ6SLTTnuXNs/edit?pref=2&pli=1#
Cheers,
James.
there is also the increasingly common pattern of "remote peering"
where you lease a circuit to an exchange point but do not establish a
presence in the facility. this can either be done with the last leg on
a dedicated cross-connect (so it looks to the exchange operator just
like any other connection except that it is to an intermediary and not
to you) or multiplexed on a single connection to the exchange operated
by a carrier that specialises in facilitating remote peering.
to the extent that this practice dramatically decouples the peering
graph from the underlying infrastructure graph it is debatable if this
is a wise or efficient strategy. on the other hand it significantly
widens the operational scope of bgp configuration knobs.
but the point is, you can do peering without a physical presence in a
location, and it is a common thing to do.
cheers,
-w
Thanks guys for the replies.
I wanted to clarify two things in my questions. First by peering I did not
necessarily mean "settlement free" interconnection. I meant any inter-AS
connection. My understanding is that in addition to the cost of transit
that should be paid to the transit provider, there also exists the cost of
the xconnect that is charged by the colocation provider. Secondly, my
question was more about the expenses, as opposed to the technical
costs/benefits. I have browsed through the "Peering Playbook", but I think
its more about providing a case "settlement free" peering.
Best Regards
Reza Motamedi (R.M)
Graduate Research Fellow
Oregon Network Research Group
Computer and Information Science
University of Oregon
Salam Reza,
You are asking an interesting set of questions, it might be easier to answer the over a phone call conversation (at least fro our perspective, being a small ISP/NSP ). I can write to you a lengthy reply, but am not sure if I will be able to convey to you the information properly.
The way you have posed the question, indicates to me that you are taking a particular view of a 'network', which is not the same view I hold. I am not sure if your view is more prevalent or our view as a service provider.
Regardless I think it would make for an interesting conversation. Feel free to call me at your convenience.
Regards.
Faisal Imtiaz
Snappy Internet & Telecom
7266 SW 48 Street
Miami, FL 33155
Tel: 305 663 5518 x 232
Help-desk: (305)663-5518 Option 2 or Email: Support@Snappytelecom.net
Not offlist.
Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373
Ouch... so much for off list .. 
Faisal Imtiaz
Snappy Internet & Telecom
7266 SW 48 Street
Miami, FL 33155
Tel: 305 663 5518 x 232
Help-desk: (305)663-5518 Option 2 or Email: Support@Snappytelecom.net
Faisal is new to the Internet. 
or More like getting too old to remember completing the edit of mail to: field !
LOL!
Faisal Imtiaz
Snappy Internet & Telecom
7266 SW 48 Street
Miami, FL 33155
Tel: 305 663 5518 x 232
Help-desk: (305)663-5518 Option 2 or Email: Support@Snappytelecom.net
There is only a xconnect expense if the peering happens at a third party
datacenter.
If it is a first party datacenter any xconnect fees can be considered part
of the peering price.
There is also the case where one party leases or builds a fiber to the
other party. The xconnect always goes to the DC owner but there is usually
a choice of companies that can lease fiber. The transit provider might own
the fiber and provide the fiber lease and transit charge as one combined
offer.
We buy transit from the incumbent teleco. There is no xconnect because we
are in the building already with our gpon equipment and all cost is
regulated by the government. The regulator put the price of xconnect at $0
because frankly xconnect is a made up cost.
Regards
Baldur
Hi Reza -
When researching the costs of peering you should perhaps categorize into the most popular forms of peering.
Public (many-to-many) peering solutions vs. private (one-to-one)
If you haven't already, you should read this
http://drpeering.net/core/bookOutline.html
Thanks guys for the replies.
I wanted to clarify two things in my questions. First by peering I did not
necessarily mean "settlement free" interconnection. I meant any inter-AS
connection. My understanding is that in addition to the cost of transit that
should be paid to the transit provider, there also exists the cost of the
xconnect that is charged by the colocation provider. Secondly, my question
was more about the expenses, as opposed to the technical costs/benefits. I
have browsed through the "Peering Playbook", but I think its more about
providing a case "settlement free" peering.
Dude, how are you going to weigh up the costs and benefits of peering
if you don't include the "costs". I refer you back to the same
documented I referred you to yesterday...
Peering Proposal v1.4 - Google Docs
This time look at section 4 of this huge and hard to navigate
document, "4. Peering Costs":
Loosely extrapolating:
Network transport: You need to be physically connected unless you
blagged space in the same rack and can patch in for free.
Hardware: You need tin to route packets.
Software: You need software to monitor the packet routing.
Colocation: You need space/power/cooler/security in which the tin can operate.
Staffing costs: Someone has to configure that tin.
Admin/engineering overhead: Someone has to manage the peering process
Peering port: You probably have to pay to peer.
Reseller ports: You might need remote connectivity to the LAN and
"network transport" above would refer to a cross connect to the remote
peering provider instead of directly into the IXP LAN.
James.
Thanks James,
I totally missed that section. Sorry about that. I think the picture is
becoming more clear in my head now.
Let me first make sure my terminology is right.
- With respect to peering, there are "transit" in which you pay the other
AS in 95-5 fashion or whatever, and "settlement-free" in which two ASes
agree not to charge each other for traffic exchange. In the latter, the
exchanged traffic is limited to the customer cone of the two ASes.
- Peering can happen either "publicly" though an IXP or "privately" through
directly linking the ports of two routers and exchanging BGP traffic.
Now, if I understood correctly, the difference between the cost of public
and private peering is that in public, one AS pays to be connected to the
IXP fabric, perhaps to the IXP provider and the colo provider. I'm assuming
IXP and colo provider are not always the same organization as it is the
case in SIX (Seattle IXP) and colo providers in Westin Building including
Equinix, ZAYO, etc. So the AS is being charged for becoming an IXP member,
and also to be allowed to take a line from its rack to the IXP's "meet me
room". Additionally if one decides to buy transit over the IXP it has to
pay the transit providers.
In Private peering however the AS pays the colo provider for the xconnect
per ASes that it wants to peer with. The cost of transit would be
additional if the peering is in fact a transit and not settlement free.
All the costs of HW, SW, personnel, administration, and perhaps
transmission between colos (including remote peering, being waved to
another location, tethering) would be the same, right?
Best Regards
Reza Motamedi (R.M)
Graduate Research Fellow
Oregon Network Research Group
Computer and Information Science
University of Oregon
You are still assuming there is a colo. But perhaps the most common case is
a multihomed company buying transit from two independent service providers.
The customer is at his office and the two service providers will have their
end somewhere in the city, perhaps even terminating their end of the
circuit in a street cabinet. The customer is multihomed and therefore has
his own AS. This is a peering situation with three AS numbers that fits
your description, it is private peering and there is no xconnect. Instead
there is usually a leased line cost, but this cost is often hidden from the
customer. Also the ISP might own the line (physical fiber) and the cost is
not a simple $/month.
But also two ISPs might peer in this way. Residual internet providers have
a ton of points of presences, so why choose a place where there is a
xconnect fee? We can peer anywhere in the city, including at a random
street cabinet. Often the cost of renting a dark fiber somewhere is lower
than a xconnect fee (a sign that datacenter owners are too greedy).
If one party is a content provider I give you that the peering point is
usually at a datacenter somewhere. But still, if the content provider is
big enough to run their own datacenter, we are back at the leased line case
again. Some content providers, even if small, prefer to just run their own
datacenter in the basement of their offices.
Regards,
Baldur
Aren't availability, guaranteed service and remote hands an incentive to do
peering inside a third party colocation?
I see very large numbers for xconnects for instance in Equnix [
https://blog.equinix.com/2013/08/equinix-cross-connects-hit-110000/] and it
made me believe buying xconnect is still a normal practice.
Best Regards
Reza Motamedi (R.M)
Graduate Research Fellow
Oregon Network Research Group
Computer and Information Science
University of Oregon
Sure. But there are places in the US where you have to decide whether the
cost of lighting 300 miles of fiber to the colo is worth the benefits, when
the other option is lighting fiber to a street cabinet across town.
Hi,
The counter example is the Netnod Stockholm IX that allows you to connect
via dark fiber from anywhere within Stockholm. The other large european
exchanges also offer multiple connection options.
"Aren't availability, guaranteed service and remote hands an incentive to
do peering inside a third party colocation? "
That really depends on your situation and who you are. As a residential ISP
our business is not in a DC. If we have any equipment at all there, it
would only be a router. But usually we do not want a router there, we just
want a connection. So in fact we do not want access to the facility at all.
If you are a content provider, you might have servers and what not. Then
sure. Or if you are large IP transit provider, you might want to host a
router, so you can sell to other customers at the DC (if it is a transit
neutral DC).
This is why we now see the rise of remote peering.
Regards,
Baldur
Also remember that 300 miles of fiber is going to go through a dozen of
street cabinets to get there.
Regards,
Baldur
