Vancouver IXP - VanTX - BCNet

Correct. The ones in black are exchanges, the ones in gray are things
that someone asserted to have been exchanges, or asserted will be
exchanges.

glad it's all so black and white, well grey. :slight_smile:

When different people are asserting different things (i.e. that something is, and is not, an IXP) the situation is, by definition, contentious. We move things into the "definitely an exchange" and show it in black text when we're able to observe a number of things:

- Three or more participants
- Shared layer-2 switch fabric across which participants peer with each other, exchanging customer routes
- New participation is not too rigorously constrained (at least a domestic ISP new market entrant should be able to participate)
- Participants do not receive a metered-rate bill based on utilization

In addition, we look for a number of signs of openness and transparency that would indicate that it's intended to be a good-faith effort to provide a point of interconnection between interested parties, rather than a regulatory compliance function, a set of private crossconnects that don't facilitation connection of new participants, a transit buyers' club, or a commercial layer-1/2 WAN carrier trying to re-brand their services. Which are, I would say, the four most common things that attempt to brand themselves as IXPs in disagreement with the general consensus that we observe.

New IXP founders typically contact our staff early in the formation process, and we collect the information above through conversations with participants, direct participation in the exchange, and on-site visits. The weak point in this process is that when IXPs go defunct, we're often lacking a clear date of dissolution in our records, because they tend to fade away gradually, with very little public notice.

Whenever anyone notices such a discrepancy, we very much appreciate their bringing it to our attention, so we can make the directory more accurate.

but how do you represent seattle colonolizing bc?

If, by that, you mean Canadian ISPs peering in Seattle, you'd see that in the participant list...

    Internet Exchange Directory | PCH
    Participants | www.seattleix.net

…and in theory the map here...

    https://prefix.pch.net/images/applications/ixpdir/origin_country_worldmap/345.png

…should be showing the Canadian participation visually, but the fact that it's not, at the moment, is indicating a data coding error on our ARIN whois import, which I'll have our guys take a look at.

                                -Bill

Just wondering aloud if an ISP that did have commercial interest could run a non-member driven exchange point successfully as long as they had pricing and policies that were similar to member driven exchange points.

I have a facility in Windsor, Ontario that is well connected, has all the physical infrastructure necessary, the ability to provide relatively low cost local fibre loops, has an open policy towards other carriers providing transport loops, but alas, it wouldn't be perceived as "neutral".

I would suggest that member driven exchanges typically produce the end product that people are interested in. Honestly, if TorIX wasn't member driven, but had the same policies as it does now, I'd still want to connect.

Community of interest of course is the other magical ingredient that is necessary. Not sure how many ISPs would want to peer in Windsor...

New IXP founders typically contact our staff

wow! i did not know we had the ixp god here! lemme go back to my
camera-ready dreadline. :slight_smile:

- Three or more participants
- Shared layer-2 switch fabric across which participants peer with
  each other, exchanging customer routes
- New participation is not too rigorously constrained (at least a
  domestic ISP new market entrant should be able to participate)

imiho, it is also nice if non-isp folk can participate, content, etc.

- Participants do not receive a metered-rate bill based on utilization

that's a new one. i am not sure i understand why. just seems a finer
grained case of 100mb for $1, 1g for $5, and 10g for $20 or whatever.

and i would add carrier neutrality, i can haul fiber from anyone into
the exchange. this is pretty critical in the exchanges where i have
played.

randy

Just wondering aloud if an ISP that did have commercial interest could run a non-member driven exchange point successfully as long as they had pricing and policies that were similar to member driven exchange points.

Verrrry interesting that you raise that.

IIRC, Albuquerque has NMIX which I think was setup as for-profit. (John Brown are you still here?) Well over a decade ago now, my recollection is fuzzy. I don't recall the reasoning in choosing for-profit over nont-for-profit.

As for ISPs doing it, there are clear examples in the wild today, but. Many buts. That ISP would have to be quite benevolent. In the long run.
New MGMT/owners and then.....?

I have a facility in Windsor, Ontario that is well connected, has all the physical infrastructure necessary, the ability to provide relatively low cost local fibre loops, has an open policy towards other carriers providing transport loops, but alas, it wouldn't be perceived as "neutral".

The only reason why we (OttIX) followed the path of not-for-porfit (and all that it comes with, from beloved loons to passionate supporters to the somewhat silent majority) was to give the community of interest (gawd what a PC-style phrase) assurance that the IXP would not be held hostage to a bottom-line or to the dictates of the single owner. In other words, neutral.

(Now going for-profit could have been tempered with issuing one share per peer and having share-holders, etc, but we're starting to delve into philosophical viewpoints which in turn have consequences, advantages and disadvantages too numerous to get into here.)

Community of interest of course is the other magical ingredient that is necessary. Not sure how many ISPs would want to peer in Windsor...

If I were looking strictly at bottomline and had the same cost option between connecting to an IX in Ottawa/Windsor as going to Toronto, I'd go to Toronto. $dayjob was public sector: We believed the more we peer with, the greater the benefit to public citizen (along being able to divide and conquer potential DDOS). Of course there are those who don't subscribe to that notion... so what do I know?

But, do what we did, throw it out there and try it just to see if there's any interest Windsor. Get the packets flowing, forget the paperwork and managerial super-structure for now. Talk to CIRA, get them to listen to you, you listen to them. OttIX started with a Paradyne DSLAM as switch core and many peers coming in on $40/month xDSL lines, just to see if there was a point.

That's one decade gone, already into another....

wfms

> New IXP founders typically contact our staff

wow! i did not know we had the ixp god here! lemme go back to my
camera-ready dreadline. :slight_smile:

> - Three or more participants
> - Shared layer-2 switch fabric across which participants peer with
> each other, exchanging customer routes
> - New participation is not too rigorously constrained (at least a
> domestic ISP new market entrant should be able to participate)

imiho, it is also nice if non-isp folk can participate, content, etc.

It provides for much more financial benefit for the participants if they
can. Pulling that traffic off of the wire at a N:N ratio usually results in
enough of a cost savings to be a win-win for both.

> - Participants do not receive a metered-rate bill based on utilization

that's a new one. i am not sure i understand why. just seems a finer
grained case of 100mb for $1, 1g for $5, and 10g for $20 or whatever.

I completely agree.

and i would add carrier neutrality, i can haul fiber from anyone into
the exchange. this is pretty critical in the exchanges where i have
played.

randy

Exchanges boxed in by incumbents and monopolies should absolutely be
contacting content sources directly (peering@) to determine if there is a
way that they can participate in the community directly. In most cases I
can definitively tell you that there is likely a way to resolve the
business issues that are roadblocks for both parties and to return
substantial benefit to the exchange and it's community. That train left the
station a few years ago. See Curacao.

Best Regards,

-M<

>Just wondering aloud if an ISP that did have commercial interest
>could run a non-member driven exchange point successfully as long
>as they had pricing and policies that were similar to member
>driven exchange points.

Verrrry interesting that you raise that.

IIRC, Albuquerque has NMIX which I think was setup as for-profit.
(John Brown are you still here?) Well over a decade ago now, my
recollection is fuzzy. I don't recall the reasoning in choosing
for-profit over nont-for-profit.

NMIX was a group of NM ISPs on a shared router at (last of?) the local
feeders into what was once WestNet in the NSF days. It had a local NNTP
server and (I believe) a couple of other services. It was useful back
in the days when you could plumb some T1s to an AGS+ and make people
happy.

Mr. Brown's attempt at an exchange (IXNM) lasted about 8 years, and can
probably be counted as an example of failure for such a model. The
political side overwhelmed any technical advantage in Albuquerque.

While it never became an importan IX, from the outside it looked like it
was a successful bandwidth co-op with several local ISPs buying from it
and benefiting from the local connectivity.

Perhaps others can make a go at it?

----------- IXNM Opening e-mail ---------------------

>Just wondering aloud if an ISP that did have commercial interest could run
>a non-member driven exchange point successfully as long as they had
>pricing and policies that were similar to member driven exchange points.

Verrrry interesting that you raise that.

IIRC, Albuquerque has NMIX which I think was setup as for-profit. (John
Brown are you still here?) Well over a decade ago now, my recollection is
fuzzy. I don't recall the reasoning in choosing for-profit over
nont-for-profit.

[NMIX couldn't pay its bills so it lost a lot of support/clients]

/bill

Facility neutrality especially. If the IXP is inside a non-neutral DC, it and its peers are always under constant threat of being squeezed out or shutdown by any number of circumstances. If the co-lo business were separate from the facility business, it may be a better environment since the IXP could convince the facility to host it, which the co-lo business could then be attracted to. All depends on the circumstances and environment.

wfms

Randy wasn't shooting at you about your definitions.

He was shooting at you for assuming people people were reading a technical
mailing list in a non-text format. It's all black to me, too. :slight_smile:

Cheers,
-- jra

Ah thanks for that update.

You've reminded me of another point: While it is admirable that CIRA (and probably other similar counterparts are watching) looking to establish IXPs, my anxiety lies with the future: Given everything that's already been written, are any of these IXPs capable of becoming self-sustaining in the future? It's a rhetorical question applicable to any starting IXP and requires an understanding of the local environment.

wfms

We run our colo facility as a separate business entity than our facilities/ISP business. Our customers actually get two invoices if they buy services and colocate. When we opened the colo, we invited any facilities based carrier in the region to place fibre. My rule was they could have rack space for a patch panel in the MMR for free for cables coming in from outside. If they needed space and power, then they would have to pay for that. They could use the entrance conduits from the first manhole outside the building for free, but they'd have to get there themselves.
Colo customers pay a standard fee for fibre pairs to the MMR patch panel, agnostic of which carrier they are connecting to - including our own services. We have some customers that don't buy services from us - just space and power. Just depends on their needs.

There are 3 fibre providers in the building now. It seems to work out. Now if there was a legitimate community of interest for establishing an IXP here, we could do it, but alas, as has been pointed out, the case for TorIX is so compelling, and so much needs to flow through Toronto regardless, it seems the natural place to interconnect.

I would hope that at least the 3 largest cities in Canada (Toronto, Montreal, Vancouver) should be able to sustain IXPs.

Hopefully Calgary, Edmonton and Winnipeg too considering the size of their populations and their distance from the largest 3 cities.

Omnibus reply warning. Skip this one unless you're really into IXP trivia.

- New participation is not too rigorously constrained (at least a
domestic ISP new market entrant should be able to participate)

imiho, it is also nice if non-isp folk can participate, content, etc.

Of course, and that is best-practice. I was listing the things that were, to my mind, bare minimums. I think we can agree that if two ISPs interconnect with each other, but prohibit a third or subsequent ISPs from interconnecting, that does not constitute an IXP. There are a small but significant subset of IXPs are the only option within their regions and are universally recognized to be IXPs, yet still place some restrictions upon who can participate, often requiring that participants hold a national ISP license. We generally work to influence them to ease or remove those restrictions over time. For example, we've succeeded in getting that restriction lifted from the Beirut exchange, while we're still working to move the management of the Buenos Aires exchange toward best-practices in this regard.

- Participants do not receive a metered-rate bill based on utilization

i am not sure i understand why. just seems a finer
grained case of 100mb for $1, 1g for $5, and 10g for $20 or whatever.

Certainly that's one way of looking at it. Can you think of any examples of something you'd consider to unequivocally be an IXP, yet charges a participant 1% more if they use 50.5mbps than if they use 50mbps? Generally this is one of the hallmarks of a layer-1/2 carrier that's trying to pass itself off as an IXP for marketing purposes. The principle drawback of such a charging scheme is that it removes the efficiency-of-scale incentive for ISPs to use the IXP more, and therefore removes their ability to charge their customers less as they use the IXP more, and thus the "exchange" fails to grow. Hypothetically, if the price were low enough, none of this would be a significant factor or disincentive, but in twenty years of IXPs and things-that-tried-to-be-IXPs, I haven't seen a successful example of such, while seeing quite a few that failed, where this was the primary distinguishing feature. It's not a common thing, but it's been a big red flag when it has shown up.

i would add carrier neutrality, i can haul fiber from anyone into
the exchange. this is pretty critical in the exchanges where i have
played.

I'd say that carrier neutrality is a hard requirement in markets where that's a decision being made by the IXP (or more often the colo that hosts the IXP). There are a lot of markets that don't have competitive carriers, so this issue isn't yet tested at the time we're trying to figure out what's what. For instance, It was twelve years between the point at which the first IXP was formed in Singapore, and the first time anyone was able to run a competitive piece of fiber into it.

Peer1 exchanges are only open to Peer1 customers, I believe. At least,
that's how it worked in Toronto the last time I looked.

that is not an exchange. most isps have switches in their transit
infrastructure.

I agree that ISPs have switches, and will usually happily sell transit (or colo) to customers connected to those switches, and that that doesn't constitute an exchange. Peer1 presents themselves as a colocation provider (that also sells hosting). I think most people other than Peer1 would agree that Peer1's facilities don't constitute exchanges, and they're not marked as exchanges by our staff, in the directory. However, the line between Peer1 and, for instance, Equinix or Global Switch, is fairly fine… I think you'd find general consensus that Equinix switches are IXPs, and Equinix is a colocation provider (that also occasionally sells transit, to some of their customers, on occasion). So while the position you're expressing is certainly the majority position, it's something of a matter of degree and focus, and to some degree a judgment in the eye of the beholder, rather than something easily expressible in a simple objective rule.

Just wondering aloud if an ISP that did have commercial interest could run a non-member driven exchange point successfully as long as they had pricing and policies that were similar to member driven exchange points.

ISPs tend to be very pragmatic on the issue of facility neutrality, whereas folks like regulators and those who are selling a product in the area get more religious about it. So ISPs don't tend to care much whether a facility is neutral, or run by one of their competitors, if they can see a clear value proposition in using it.

That said, the price-point of the non-neutral for-profit IXP you postulate is not necessarily equal to or greater than zero… First of all, operating a switch is well within the core competency of ISPs, so they all know that the cost of doing it themselves is de minimis. Second, they know that the capex of building out to someone else's facility can be substantial, so while they're willing to do that when the future value of that investment, and their ability to recoup it, is protected, they generally won't do so if they can't see very clear protections (cash and contracts, not assertions and good-will). Third, they know that, while they have to make that infrastructural investment to reach the exchange, the owner of the facility does not. When the owner is not an ISP, that's not an issue, but when the owner is an ISP, and one of their competitors, it constitutes a significant relative competitive disadvantage. Although both parties lose in an absolute sense if they fail to exchange traffic, the other guy didn't have to drop a lot of money, and doesn't have as much to lose, so can afford to play hardball in negotiating over who gets to keep excess rent.

In practice, you don't see this arrangement much (non-neutral IXPs) for two reasons: when IXPs are established through an open stakeholder process, all stakeholders make clear that they'd be happy to host the exchange within their facility, and once they've all gotten that off their chests, they move on to deciding upon a jointly-acceptable neutral location; and when a for-profit entity unilaterally establishes a non-neutral exchange, ISPs generally don't choose to use it, and it fails through the action of market forces.

The halfway-case, neutral for-profit exchanges, generally a feature of neutral for-profit colocation facilities, are an interesting compromise and are often quite successful in the marketplace. So it's useful to look at them as a point of comparison. ISPs still know that they can do the switch thing themselves just as easily, so the cost still needs to be relatively low, and the points of value need to be things that are more difficult for an ISP to do internally. First and foremost, the neutrality and the large datacenter facility attract other people to talk to. Second, also very compelling, getting to take advantage of the benefits of scale; generators, security, etc., in a location where you only need one or two cabinets. Third, many of these businesses make their money elsewhere, through real-estate speculation or by issuing shares, so the services ISPs are buying are subsidized by those other lines of business, and are, literally, less expensive than if the ISP were to build it themselves.

Community of interest of course is the other magical ingredient that is necessary. Not sure how many ISPs would want to peer in Windsor...

If there was a legitimate community of interest for establishing an IXP here, we could do it, but alas, as has been pointed out, the case for TorIX is so compelling, and so much needs to flow through Toronto regardless, it seems the natural place to interconnect.

Folks in London (England, not Ontario) said the same thing about Washington D.C. before they got an IXP of their own up, as well. It's very easy to look at the status-quo and observe that it's functioning, while it's not always so easy to imagine how things will be better, if things pan out, in five or ten years. My experience leads me to believe that Windsor can support an IXP, and that if you don't over-spend and gold-plate it, it'll make money for you, and will grow over time. As will TorIX. It's quite likely that an exchange in Windsor will never be as large as one in Toronto, but that's no reason not to do it. A restaurant or gas station in Toronto might be more successful than one in Windsor, but it doesn't mean that nobody bothers to start a restaurant or gas station in Windsor. Same thing. Speed times distance equals cost, and always will. Exchange traffic more locally to keep speeds high and costs low, and stay competitive. If you just use an IXP in Toronto, ISPs in Toronto will always be more competitive than you will, because their cost-of-goods will always be lower.

While it is admirable that CIRA (and probably other similar counterparts are watching) looking to establish IXPs…

I think CIRA has been very clear that they're not trying to establish IXPs, they're trying to provide any desired support to locally-based IXP efforts. There's a big difference. In order for an IXP to succeed, it needs a constituency of ISPs who understand that it's critical to their financial success. You can't just drop in and build an IXP for someone, and expect it to still be there a year later.

My anxiety lies with the future: Given everything that's already been written, are any of these IXPs capable of becoming self-sustaining in the future? It's a rhetorical question applicable to any starting IXP.

Indeed. I think that ISPs who understand their business model well enough to understand the effect the IXP will have on their average-per-bit-delivery-cost is essential. I think it's also essential that they have some basic familiarity with the different ways IXPs can fail, or fail to thrive, so that they can avoid mistakes others have made in the past. Over-spending, particularly on switches, is a huge killer of IXPs. Under-provisioning of circuits to the IXP is another big mistake. Failure to encourage local content and hosting is another.

                                -Bill

>
> but how do you represent seattle colonolizing bc?
"keep your potatoes out of my pig."

Ugh. Suddenly your talk of potatoes in pigs
makes "colon-olizing" seem almost meaningful

(methinks Randy meant "colonizing", but his
colon-os got a bit carried away...:smiley: )

Matt

* woody@pch.net (Bill Woodcock) [Wed 21 Aug 2013, 21:04 CEST]:
[..]

My anxiety lies with the future: Given everything that's already been written, are any of these IXPs capable of becoming self-sustaining in the future? It's a rhetorical question applicable to any starting IXP.

Indeed. I think that ISPs who understand their business model well enough to understand the effect the IXP will have on their average-per-bit-delivery-cost is essential. I think it's also essential that they have some basic familiarity with the different ways IXPs can fail, or fail to thrive, so that they can avoid mistakes others have made in the past. Over-spending, particularly on switches, is a huge killer of IXPs. Under-provisioning of circuits to the IXP is another big mistake. Failure to encourage local content and hosting is another.

Can you cite a few examples of an IXP going under because of overspending on switch hardware? You call this a "huge killer" so there must be dozens you can choose from.

  -- Niels.

Having recently been through the startup process at an independent
non-profit IXP, I can see spending too much on hardware being a problem
particularly on supporting the ongoing hardware/software maintenance on the
switch. Something else to think about, if an exchange is given an endowment
from from outside entity it might be harder to build the commitment levels
of participants/volunteers because it's too easy to solve the problems with
money as opposed to member contributions.

We went through 3 switch upgrades in 2 years and IMHO it built a lot of
community.

Each IXP will have a set of faithful founders and the key is growing that
group beyond the initial group before the founders lose interest and move
on to bigger things. Particularly before connecting to the IXP makes
business sense to a company that won't connect just because it's cool.

When the exchange gets over the hump were companies are saving real money
it is much easier to keep the ball rolling and the exchange growing. Many
exchange points never make it to that level.

Jay

What has CIRA done in Calgary?

CIRA has done a great job in Winnipeg and Montreal. Nothing in Calgary.

Bill, not true.

Following on our vision for Canada to have an IXP in every major city, specifically for Calgary, CIRA worked with CYBERA to organize a town hall meeting in Calgary, on September 14, 2013. At the meeting, we had interested members of the community (Content delivery, ISP, government, R&E, CIRA, others) together to start the development of a new IXP in Calgary.

Cybera being local, they took the lead in working with the community members in setting up governance, technical architecture, location, etc... CIRA actively participated in the various committees.

CIRA is planning the deployment of equipment for the IXP, a .ca DNS Anycast stack, NTP servers and space for M-Lab nodes. We also work with PCH to have a PCH anycast stack installed in Calgary. We talked to Akamai and Google to be part of the Calgary IXP. HE actually did build to Data Hive (Preferred data center in Calgary). Doing our part in trying to get as much content, ISP, eyeballs, core internet services to be present at IXP. The AlbertaIX is cash poor at that moment in time so CIRA was looking at options to fund equipment or provide start-up grants, nothing was done up to now with respect to providing equipment or funds.

Note: CIRA's job is not to go in and put a switch and leave. CIRA works with the community to get the IXP up and running. We have criteria to participate, the IXP must be a member based, vendor neutral non-profit organization with a viable trusted community to operate and sustain an IXP. CIRA is acting as a catalyst, not a doer. The community is the doer. Ask the people in Winnipeg www.mbix.ca and Montreal qix.ca . We are planning the launch of mbix this September as well. This is an example where the IXP build was successful.

Back to Calgary, something special occurred, while we were all working on setting up AlbertaIX (we started fast but the pace slowed down significantly), a new IXP came to 'life' YYCIX in Data Hive, yycix.ca. Long story short, CIRA is waiting for things to settle before we continue providing more support and invest in deployment our .CA infrastructure. When the "dust" settles, we will deploy our .ca infrastructure and be a member of the communities (peer).

There's a chance we're going to have two IXPs in Calgary, hopefully they would be in the same data center to leverage our investment, if we have two, then it's going to make it confusing for the new potential peers to pick the right one or both.

All in all, CIRA's objective was to have 1 IXP in Calgary, we have one now, perhaps 2 in the future, so mission accomplished. It's up to the community to get their act together (no pun intended), and if they need our help, then we're there to help with governance, funding, technical expertise, etc...

Also, we're doing our best with our limited resources,

Jack

(NOTE: English not my first language, if you're not sure what I mean, ask me, don't assume)

Is Bill Sandiford a member of the board of both CIRA[1] and AlbertaIX[5]?
Since AlbertaIX has no peers [2,3] is CIRA now going to investigate options
to help fund YYCIX which does have peers[3,4]?

References:
[1] http://www.cira.ca/about-cira/about-the-board/meet-the-board/
[2] http://www.albertaix.ca/peers/
[3] https://prefix.pch.net/applications/ixpdir/
[4] http://yycix.ca/peers.html
[5] http://www.tcpiputils.com/browse/as/54982