Dear Nanogers,
I have a question about common/best network interconnection practices.
Assume that two networks (let's refer to them as AS-a and AS-b) are present
in a colocation facility say Equinix LA. As many of you know, Equininx runs
an IXP in LA as well. So AS-as and AS-b can interconnct
1) using private cross-connect
2) through the public IXP's switching fabric.
Is it a common/good practice for the two networks to establish connections
both through the IXP and also using a private cross-connect?
I was thinking considering the cost of cross-connects (my understanding is
that the colocation provider charges the customers for each cross-connect
in addition to the rent of the rack or cage or whatever), it would not be
economically reasonable to have both. Although, if the cross-connect is the
primary method of interconnection, and the IXP provides a router-server the
public-peering over IXP would essentially be free. So it might makes sense
to assume that for the private cross-connect, there exists a back-up
connection though the IXP. Anyway, I guess some discussion may give more
insight about which one is more reasonable to assume and do.
Now my last question is that if the two connections exist (one private
cross-connect and another back-up through the IXP), what are the chances
that periodically launched traceroutes that pass the inter-AS connection in
that colo see both types of connection in a week. I guess what I'm asking
is how often back-up routes are taken? Can the networks do load balancing
on the two connection and essentially use them as primary routes?
Best Regards
Reza Motamedi (R.M)
Graduate Research Fellow
Oregon Network Research Group
Computer and Information Science
University of Oregon
Reza,
You maybe overthinking this one a bit. The economics are something to consider, however all public exchanges have different economics. With Equinix you pay pretty much a flat rate for a single 1Gbps/10Gbps link that includes the cost of facility cross-connect and public exchange access. It is a nice one to many connection for all those various network and content networks your end users would appreciate direct connectivity. Depending on the public exchange you either have a single BGP session or a BGP session per network you are peering. Really after that, it's just BGP routing and route management. You do need to be careful about not being too overly dependent on a single public switch link, in some cases like at Equinix you may want multiple connections to redundant public exchange switches at that site. There is a balance you want to seek of number of paid upstream network transit providers you are connected to versus how many direct peering arrangements you have setup. It's not usually practical for a smaller network to have loads of BGP peers. There are lots of good articles online about this fine balance and some good advice from experienced network operators.
To your later questions. For your simple example, if AS-a and AS-b were both already on the public IX, and the link wasn't too overly critical then using the public IX switch maybe a good first step. However as that relationship matures, they most likely in a real world example may look to split the cost of the private cross-connect. If it was mutually beneficial. There is much more to public peering and transit than the technical conversation. Most of the larger networks on the public switches won't peer privately with anyone or only with extremely larger networks. To get a provider such as this to peer both privately and on the public exchange is not a technical issue, it's more of a business overhead and management issue.
If you have a couple of quality upstream transit providers, they will be excellent failovers to a public switch outage. Plan for the public switch to have as many problems as any upstream provider.
Sincerely,
Nick Ellermann – CTO & VP Cloud Services
BroadAspect
E: nellermann@broadaspect.com
P: 703-297-4639
F: 703-996-4443
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
Hi Nick,
Thanks for the reply.
Let me clarify another issue first, since I thought the colo's business
model is different at least in the US. So if AS-a puts its router in
Equinix, it should pay the same amount in the following two scenario (only
considering the interconnection cost and not the rent for racks and remote
hands and ....)?
1) AS-a only connects to the IX and establishes all inter-AS connections
through the IX.
2) AS-a connects to the IX, in addition to privately connecting to bunch of
other colo customers (these private connections can be either transit or
settlement-free peerings).
My understanding was that colos in the US charge per cross connect, so the
more you connect privately, the more you pay. This article may be old, but
I don't think much has changed:
https://www.telegeography.com/press/press-releases/2015/02/26/colocation-cross-connect-price-disparities-remain-between-u-s-europe/index.html
With respect to my second question, I was asking if is practical/reasonable
to keep both the connection types to same network (say AS-b) at the same
time, i.e., connect privately over a cross-connect and keep the public
connection over the IX.
Best Regards
Reza Motamedi (R.M)
Graduate Research Fellow
Oregon Network Research Group
Computer and Information Science
University of Oregon
Router ports are expensive, so even if cross connects were free, you would
still use the public switch fabric until you reach a traffic level that
justifies a direct connection. The point of having a IX switch is that you
can connect to many others with just one single router port.
When you have the direct cross connect, you would not usually use the IX
switch in parallel for that AS. With the cross connect you have dedicated
bandwidth to the AS and you would want to reserve the IX switch port for
traffic to the remaining networks that you do not yet have a cross connect
to.
The cross connect is not a very good redundancy setup with regard to the IX
switch. Both usually go to the same router and share the same single point
of failure (your router is a single point of failure and the peer router is
a single point of failure). A cross connect is usual very reliable. You
would plan for your router to be down or the peer router to be down, and
have a backup path through some entirely geographic separate location.
In many cases your generic IP transit service is good enough backup. Your
direct peering is an optimization and if that is down, you go back to the
transit service.
Of course everyone are playing their own game and you might see anything
happening in the real world despite the above.
Regards,
Baldur
Ports on the colo's IX, Equinix for example, will likely cost more than just a cross connect. If you have peers with which you exchange enough traffic, it can make sense to remove that traffic from the IX and put it on PNI (cross connect) peering, leaving the IX port(s) for use primarily for peering with lots of "smaller peers" (in the amount of traffic exchanged).
Typically, if a peer is big enough to justify PNI, you won't want to fail-over to the IX as a backup, because doing so is likely to congest your or their IX links. Of course, there are exceptions. A PNI peer might not have enough ports to dedicate to PNI peering and might want to spread peering traffic over both PNI and IX evenly.
Hi All,
I wonder why a "VLAN exchange" does not exists. Or I do not know any?
In my understanding it should be a switch, and people connected can
easily order a private VLAN between each other (or to private group)
through some kind of web interface.
That should be a more easy and much less expensive way for private
interconnects than direct wires.
This does exist, often called an elastic fabric, e.g. Megaport
Regards,
Marty Strong
Hi Max,
These do exist, at least in the NREN part of the internet.
Have a look at netherlight (www.netherlight.net) and the bigger picture GLIF (www.glif.is) and where you read 'lightpath' replace that with ethernet p2p.
Regards,
Jac
As Marty said, it does exist. AN example from LONAP in the UK:
https://www.lonap.net/fees.php
Private VLANs between members = FREE
Another option is using a provider like IXReach (now "Console"), take
a peering to them, and then down multiple VLANs they can through you
peerings to different IXs from around the world and to other networks:
http://www.ixreach.com/
Cheers,
James.
In a message written on Sun, May 22, 2016 at 09:33:38AM +0300, Max Tulyev wrote:
That should be a more easy and much less expensive way for private
interconnects than direct wires.
The problem is peering is not an even distribution by traffic level.
When BigCDNCo connects to BigCableCo, they need 50x100GE. It's
actually cheaper to run the fiber between them at 10 locations for
5x100GE each than it is to run fiber from both of them to a switch,
and have the switch providing vendor engineer the switch to that
capacity. (Hint, running to the switch is 2x the fiber, plus
switch ports.)
On the other end of the spectrum, the guy who has 5Gbps of traffic
can buy a 10GE into the switched exchange, have lots of headroom
and connect to everyone with the same port.
The truth of the matter is there are 40 players in the big pile,
15,000 providers in the small pile, and perhaps only 100 oddballs
between the two.
As mentioned by others, they do exist, but usually not for exactly the reason you state.
In most cases, peers go to PNI instead of peering via the exchange when it does not make
sense to grow laterally at the exchange for significant bilateral traffic. It’s much
less expensive to get a cross-connect from my router to your router than for both of
us to add a cross-connect to the exchange and each pay for an additional exchange port.
Example: If I have 12.5 gigs of traffic to the exchange and 8 gigs of that is to
autonomous system X while the remaining 4.5 G goes to random other peers, then it
makes much more sense for both X and I to connect directly (PNI) than for each of
us to order an additional exchange port to support that traffic.
Owen
I'm glad we are having this discussion.
I want to clarify something, since I'm not sure I'm following the
terminology. What Max referred to as "VLAN exchange" is what Equinix
markets as "*private VLAN"*, right?
I just copy-pasted a portion of Equinix's IX brochure that covers the
services that they offer [
http://www.equinix.com/resources/data-sheets/equinix-internet-exchange/]
Standard Equinix Internet Exchange Features
• Public VLAN — offers access to all peering participants
• Supports industry standard IEEE 802.1Q trunking encapsulation
• Redundant MLPE route servers at each IX Point enabling efficient open
peering
• *Private VLAN* (Required: Unicast Peering VLAN enabled) — create a
private broadcast domain over the public switched infrastructure that can
be used for direct bi-lateral peering or to create a community of interest
My question is what is the point of having such an option for peering? I
understand the argument that Owen and Leo have, which is to move the bigger
portion of traffic away from the IX fabric and keep the IX for smaller
flows. but why would a pair of networks want a private point-to-point
connection on a shared switching fabric. Is this just because that shared
fabric has geographical reach, as in the case of IXReach?
I also see that links provided in this discussion show Europe based
networks that are using this peering type more often. Is this widely
accepted that US market is totally different from Europe?
Best Regards
Reza Motamedi (R.M)
Graduate Research Fellow
Oregon Network Research Group
Computer and Information Science
University of Oregon
The usefulness of an elastic fabric as far as I can see it are:
- Can give you a private VLAN to some *cloud* providers that provide direct access to them in some other fashion than peering (assumedly for enterprises)
- Is spread across multiple buildings across a metro area
- Is elastic so can be divided between different services for different time periods
In a traditional peering sense it doesn’t really offer much value.
Just my two pence.
Regards,
Marty Strong
And what benefit is there to this 'public' vlan service? A shared vlan between
all participants (with some well organized numbering/indexing scheme)?
TorIX (Toronto) is about to have an AGM here and this VLAN thing which has
been in the air for 3 years will certainly be brought up again.
/kc
Typically you would use a private VLAN between you and another participant in order to connect to them separately from the public peering VLAN. You would do this instead of a PNI in a situation where you’re in a different building from the other participant making a direct fibre more expensive than the value it would bring.
A public VLAN is essentially the peering VLAN anyway, so an all participants VLAN would be a little pointless. Perhaps a VLAN shared between a couple of members *may* be useful depending on those members’ use cases, although I can’t think of one off the top of my head.
Regards,
Marty Strong
The cost of an IX can be quite expensive actually. If you look at the RIPE
presentations from this week, there are stealth routing hijacks that come from
promiscuous peering as well as just the flat economics of connecting with a 10GE
or 100GE interface and the cost per gigabit you assign to the IX port. These
are flat rate ports, unlike transit that may offer you a price and commit rates
that allow you to reach everyone vs those just at the IX.
I’m hoping I don’t get in trouble for sharing this, but this collaboration exists
for europe on peering costs which are normalized in euro cents per megabit.
https://docs.google.com/spreadsheets/d/18ztPX_ysWYqEhJlf2SKQQsTNRbkwoxPSfaC6ScEZAG8/edit#gid=0
- Jared
If you dig into hijacking topic more, you will see that hijacks through
Tier1 is same or even more popular than through IXes.
And if someone want to make me a transit offer for the price of DE-CIX
(I do not even ask the price of DTEL-IX peering 
) - please, contact
me off-list, I will be really happy.
If you dig into hijacking topic more, you will see that hijacks through
Tier1 is same or even more popular than through IXes.
You may not have a view into that you’re being hijacked and used to send
SPAM for example:
Their space was hijacked and announced facing Yahoo. I’m hoping that
Yahoo is now feeding public route views services as a method to help
with detection. Same goes for Microsoft and Google and other e-mail
providers. Some sunlight here would help avoid similar localized hijacks.
And if someone want to make me a transit offer for the price of DE-CIX
(I do not even ask the price of DTEL-IX peering 
) - please, contact
me off-list, I will be really happy.
Pricing obviously varies based on location and a few other criteria, but
you should be shopping if this is a major part of your business.
- Jared
I'm right here at RIPE 72 now, so I saw it of course
The problem is not peering itself, but more general problem of filtering
nets, and it was told in the presentation.
I disagree somewhat, without a view of how you are being hijacked there often can be no remediation. Yahoo for example provides no cloud services so you can't purchase a view of their routing by getting a VM.
Jared Mauch