Do ATM-based Exchange Points make sense anymore?

Mailman List Mirror (Read Only)
1

Hi all -

I've been working with a number of ISPs on a research paper that builds on the previous peering research papers (Internet Service Providers and Peering, A Business Case for Peering, The Art of Peering, Interconnection Strategies for ISPs, etc.) that applies the Peering Modelling tools in a comparison of ATM and Ethernet-based Internet Exchanges. Both of these IXes are compared against each other and against the cost of buying transit. The paper applies recent price quotes for transport and transit, costs for ATM and Ethernet-based IX participation, to answer the question:

            Do ATM-based Exchange Points make sense anymore?

I'd like to speak with additional ISP Peering Coordinators and Network Architects (preferable ones that have experience with peering across both ATM and Ethenet-based IXes) to walk through this paper and help me check that I have the technical and business details right. I would need about 20 minutes or so on the phone to walk you through the paper, the financial models, the cost points, and get feedback on the conclusions...preferably sometime in the next couple weeks.

If you are a Peering Coordinator I think you will find at least a couple of findings in this research *very* interesting. In any case, if you can help, please send me an e-mail at wbn@equinix.com and let me know when we could chat.

Thanks -

Bill

PS - As with any these Peering White Papers, this white paper will be freely available once enough folks have walked through it and verify that we have things right.

------------------------------------------ Abstract

2

Hi again -

A couple points (based on some interactions with folks privately).

This is not an ATM is bad, or general ATM-bashing paper. It simply applies the same Peering Analysis that ISPs are applying to determine if and when IXes make sense. With the transit prices and transport prices dropping, this is a reasonable question, worthy of greater analysis than "well, ATM is expensive so ATM is bad."

To give you a flavor, given a set of assumptions, OC-3 (155Mbps) transport into an ATM-based IX has an "Effective Peering Range" (where peering across them is cheaper than transit) of 75-90Mbps, while given the same assumptions, Fast Ethernet-based IXes also at OC-3 have an Effective Peering Range of 40-70Mbps. The "Minimum Cost of Traffic Exchange" for this ATM solution is $122/Mbps while FastE is $80/Mbps.

At higher capacity the interconnect analysis is more dramatic: Given the relatively high price point of transport and port cost, the Effective Peering Range for ATM/OC-12 Peering is a narrow 236Mbps to 375Mbps with a Minimum Cost of Traffic Exchange of $69/Mbps. The GigE/OC-12 equivalent range is 109Mbps-466Mbps with a Minimum Cost of Traffic Exchange of $25/Mbps.

What was unexpected in this analysis was the Effective Peering Range Gap. When an ISP upgrades the ATM OC-3 to OC-12, the gap between the Effective Peering Bandwidth of the OC-3 (90Mbps) and the Peering Breakeven Point (the point at which the Peering Costs are totally offset by the cost savings of peering vs. transit) at 208Mbps is huge. This 118Mbps gap is where an ISP should rationally prefer to purchase transit until 208Mbps can be sent in peering relationships over the ATM fabric, and only then upgrade the peering connection to OC-12!

There is also an Ethernet EPR Gap but it is only about 40 Mbps, and once at the GigE/OC-12 capacity, it gets you an Effective Peering Range up to 475Mbps.

In any case, this is the analysis that the paper walks through, and since the spreadsheets are in the paper, one can muck around with the assumptions and cost points, key of which are:
      1) ATM OC-3 Port Cost $8000/mo, ATM OC-3 Circuit Cost $3000/mo,
          ATM OC-12 Port Cost $17000/mo, ATM OC-12 Circuit Cost $8000/mo
      2) FastE Port & Rack Space $2500/mo, OC-3 Circuit $3500/mo,
         GigE Port & Rack Price$5000/mo, OC-12 Circuit $7000/mo
      3) Transit Price: if you peer at OC-3, you probably pay $125/Mbps, peer at OC-12,$110/Mbps
      4) ATM Overhead (aka cell tax): 20%
      5) Assumption that ISP upgrade capacity when avg utilization >75% Effective Peering BW

Let me know if you violently object to any of these data points. These are culled from a lot of conversations in the field. The rest of the paper is simply plugging these data points into the equations and analyzing the results.

Bill

3

Everything I say below is IMHO.

This is not an ATM is bad, or general ATM-bashing paper. It simply applies
the same Peering Analysis that ISPs are applying to determine if and when
IXes make sense. With the transit prices and transport prices dropping,
this is a reasonable question, worthy of greater analysis than "well, ATM
is expensive so ATM is bad."

Yes. ATM is priced expensive from the ATM NAP providers, and no one seems
to know why. Personally, I don't believe that ATM is 'bad' for
shared-fabric exchange point. I mean, it works, and solves several
problems quite easy: a) it's easily distributed via SONET services to
folks who are not next to the ATM switch, b) it makes interconnection
between networks safer (ie, not dealing with broadcast issues on a
ethernet nap), c) virtual PI connections are easily accomplished, d) there
are varying degrees of interconnection speed (agreeably, less important),
e) it allows for things other than IP, or packet-based traffic to be
exchanged (a la Verizons' video-portal service) (agreeably, again, less
important).

To give you a flavor, given a set of assumptions, OC-3 (155Mbps) transport
into an ATM-based IX has an "Effective Peering Range" (where peering across
them is cheaper than transit) of 75-90Mbps, while given the same
assumptions, Fast Ethernet-based IXes also at OC-3 have an Effective
Peering Range of 40-70Mbps. The "Minimum Cost of Traffic Exchange" for this
ATM solution is $122/Mbps while FastE is $80/Mbps.

The pricing model is wholy irrelevant of the platform used. I think that
it just happens to be that AADS and WCOM rape the folks who peer.

At higher capacity the interconnect analysis is more dramatic: Given the
relatively high price point of transport and port cost, the Effective
Peering Range for ATM/OC-12 Peering is a narrow 236Mbps to 375Mbps with
a Minimum Cost of Traffic Exchange of $69/Mbps. The GigE/OC-12
equivalent range is 109Mbps-466Mbps with a Minimum Cost of Traffic
Exchange of $25/Mbps.

Agreed, however, even at $25/meg, when you factor in the cost of
equipment, rack space, and the build of network to get you to the place to
peer, you can easily be over the price of purchasing bandwidth again.

What was unexpected in this analysis was the Effective Peering Range
Gap. When an ISP upgrades the ATM OC-3 to OC-12, the gap between the
Effective Peering Bandwidth of the OC-3 (90Mbps) and the Peering
Breakeven Point (the point at which the Peering Costs are totally offset
by the cost savings of peering vs. transit) at 208Mbps is huge. This
118Mbps gap is where an ISP should rationally prefer to purchase transit
until 208Mbps can be sent in peering relationships over the ATM fabric,
and only then upgrade the peering connection to OC-12!

Certainly a problem, but this should more be addressed as a problem with
price points that they charged not being relative to actual costs.

In any case, this is the analysis that the paper walks through, and since
the spreadsheets are in the paper, one can muck around with the assumptions
and cost points, key of which are:
      1) ATM OC-3 Port Cost $8000/mo, ATM OC-3 Circuit Cost $3000/mo,
          ATM OC-12 Port Cost $17000/mo, ATM OC-12 Circuit Cost $8000/mo
      2) FastE Port & Rack Space $2500/mo, OC-3 Circuit $3500/mo,
         GigE Port & Rack Price$5000/mo, OC-12 Circuit $7000/mo
      3) Transit Price: if you peer at OC-3, you probably pay $125/Mbps,
peer at OC-12,$110/Mbps
      4) ATM Overhead (aka cell tax): 20%
      5) Assumption that ISP upgrade capacity when avg utilization >75%
Effective Peering BW

I am not looking to start a "my transit provider is cheaper than yours"
arguement, but at 100 mbit/second committments, they are a wide variety of
providers in the $50 to $100/meg range (with a nice cluster around $75).

-- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben --
-- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --

4

Personally, I don't believe that ATM is 'bad' for

    > shared-fabric exchange point. I mean, it works, and solves several
    > problems quite easy: a) it's easily distributed via SONET services to
    > folks who are not next to the ATM switch, b) it makes interconnection
    > between networks safer (ie, not dealing with broadcast issues on a
    > ethernet nap), c) virtual PI connections are easily accomplished, d) there
    > are varying degrees of interconnection speed (agreeably, less important),

All of the above are true of frame relay as well, which has the additional
benefit of not being funamentally incompatible with data networking. :slight_smile:

    > e) it allows for things other than IP, or packet-based traffic to be
    > exchanged (a la Verizons' video-portal service) (agreeably, again, less
    > important).

Ah, yes, I agree with this entirely. If you're building a voice exchange,
rather than a data exchange, ATM is excellent technology. I believe
Vinnie can give us some personal experience on this front.

                                -Bill

5

I doubt that any of the ATM-based echanges were built because
of a deep affection for ATM. More likely, it was the only
virtual circuit techonlogy around at the the time that a certain
router vendor supported at speeds greater than DS3.

ATM worked reasonably well for that application, once there
were switches with adequate buffering.

Anyone building a similar exchange today would have new choices
not available three or more years ago.

  Steve

6

You guys might find this interesting.... I'd like to share the more common "Religious debate points" regarding ATM-based vs. Ethernet-based IXes that I heard during the walk throughs (about 50 so far) of this paper (v1.6):

7

Interesting points, and although orthogonal to the analysis in "Do
ATM-based Internet Exchange Points Make Sense Anymore?", I am including
these in the appendix to show these alternate views of the world. Am I
missing any of the major (fact-based) views?

There is this "small" thing that higher speed ATM interfaces are not
actively
being developed. It�s very hard to come by STM16 ATM interface on a router
with STM64 being non-existent on both switches and routers. This is not due
technological barriers but commercial reality. ATM would be going away
faster if DSLAM�s wouldn�t be there increasing the traffic, but because of
the extensive investments to ADSL gear, it�ll be there in 5, probably 10
years
time.

Because of the above reason, there is no point in setting up an ATM based
exchange today. The first one here which was built 1996 is being retired by
the end of year, and the other probably sometime next year.

Pete