Market for diversity (was: Re: Cogent latency / congestion)

Speaking on Deep Background, the Press Secretary whispered:

> Or there might suddenly be a reason/market for properly physically diverse
> paths which provide partial 1:1 (ie, some services are guaranteed full backup
> bandwidth, other services get degraded access) IP paths..

I don't think the target customer in this case is really in the market
for properly physically diverse paths which provide partial 1:1. The
target customer seems to be looking for no-frills, cheap Internet.

Customers in a market for properly physically diverse paths with partial
1:1 probably are already buying Internet from other ISPs.

And still not getting it. A friend oversees various expensive USG
networks. They pay for physically diverse routing from multiple
sources. Yet every year, when they do an laborious audit down to
the "what fiber, in what bundle, in what trench" level; they find..

Guess What! Yep, someone has moved this circuit or that one to
where both pipes are intimate neighbors.

It's inevitable given buying throughput is rather like moving
something by ship. You never go to the ship [fiber] owner; you
go to a freight broker who deals with a consolidator who calls an
agent who knows who has chartered ships from A to B on DATE and....
[Look up the "GTS Katie" incident for a side effect of this.]

Speaking on Deep Background, the Press Secretary whispered:

Or there might suddenly be a reason/market for properly physically diverse
paths which provide partial 1:1 (ie, some services are guaranteed full backup
bandwidth, other services get degraded access) IP paths..

I don't think the target customer in this case is really in the market
for properly physically diverse paths which provide partial 1:1. The
target customer seems to be looking for no-frills, cheap Internet.

Customers in a market for properly physically diverse paths with partial
1:1 probably are already buying Internet from other ISPs.

And still not getting it. A friend oversees various expensive USG
networks. They pay for physically diverse routing from multiple
sources. Yet every year, when they do an laborious audit down to
the "what fiber, in what bundle, in what trench" level; they find..

Guess What! Yep, someone has moved this circuit or that one to
where both pipes are intimate neighbors.

My rule is that you never really know where bits are actually going unless
you put in the fiber yourself. (Of course, you can know where the bits _went_,
once the backhoe or the train crash takes out your circuit, but by an extension of the quantum measurement theory, that only applies to the past, not the future.)

It's inevitable given buying throughput is rather like moving
something by ship. You never go to the ship [fiber] owner; you
go to a freight broker who deals with a consolidator who calls an
agent who knows who has chartered ships from A to B on DATE and....
[Look up the "GTS Katie" incident for a side effect of this.]

Yes, and likewise you never know how your (literal) shipment is going
until you get it. (Vienna, Austria, to Norfolk, Virginia, with a stage by truck from St Louis, Missouri ? Happened to me. They must have been using some sort of hot potato routing.)

Totally OT, but it made me really happy to learn that Dulles Airport is officially a port, and I can literally send things by ship to IAD for customs clearance and pickup for the same price as to Norfolk, Virginia (an actual port, with docks and water and ships and all that). Of course, I don't have to care about their routing protocols or their last mile problems.

Regards
Marshall

It's inevitable given buying throughput is rather like moving
something by ship. You never go to the ship [fiber] owner; you
go to a freight broker who deals with a consolidator who calls an
agent who knows who has chartered ships from A to B on DATE and....

This may also have something to do with the fact that carriers/fiber owners tend to play a slight game of schizophrenia with themselves. On the same cables they try to make wildly different levels of compensation (say: SONET voice traffic vs SONET data vs IP -- in decreasing order of value per bit)

Then try to increase the marginal value of existing assets by increasing the total bit capacity (when we all know the highest value traffic grows at the slowest rate).

So they sell of large chunks of capacity to brokers so they don't have to play channel wars (openly) with themselves.

Then do anticompetitive things when those brokers themselves try to move up the value chain by selling data PL services at IP prices, etc.

Pardon my forwardness, but don't people just multi-home these days? If your network connection is that mission critical, then having at least two providers would be prudent. Keep Cogent for el cheapo/variable latency connection, but have a reliable second and/or third source (i.e. Sprint, UUNET/Verizon).

All of these issues have convinced me to multihome with Sprint as soon as my fiber-to-the-home-business is finished. Granted I still have a single point of failure with the single fiber into my cable co (which will allow me to multi-home with Sprint and cable co TWC, ASN 20001) but the crime rate in my area is MUCH lower in the last mile. Much less chance of gunfire taking it down. I am more worried about that critical path in Cleveland; hopefully Sprint and Level3/ATDN (TWC multihomed providers) keep their fiber buried more deeply in high crime/railroad areas...

Multihoming is great for when there is a total outage. In the case of Cogent on Monday, it wasnt "down"... In this case, there is only so much you can do to influence how packets come back at you as BGP doesnt know anything about a "lossy" or slow connections.

         ---Mike

Hi,
In my case, I have 6453 and 174 for transit. I want to get to 577 which is directly connected to 6453 and 174. 577 has a higher local pref on paths via 174. Short of shutting my 174 session (or some deaggregation), I dont have a way to influence how 577 gets back to me. I can easily exit out 6453, but it does nothing for the return packets. I have enough capacity on 6453 to handle all my traffic, but its a Draconian step to take and some traffic via 174 is fine and would be worse if I fully shut the session. (ie. peers of 174 in Toronto)

         ---Mike

We're connected to Teleglobe(6453), Telus(852), TeliaSonera(1299), MCI(701), and L3(3356)

We don't play any economic games with our traffic - our routing policy is (theoretically) designed to give the best possible product to our customers, and although we weren't dead in the water during the cable cut, we had major problems - especially to Bell(577) for the same reason as Mike.

So what is the solution? So do we connect to Bell as well? Even though they are the ones with the moronic routing policy? It would solve the problem, but it's certainly not the way to support "quality" carriers by purchasing only quality bandwidth...

Mike Tancsa wrote:

I'm posting too much this week and should stop, but...

Again, this is a matter of thinking about design goals. What were you trying to accomplish when you bought redundant connections? It probably wasn't "redundancy," but rather something that redundancy would give you. What redundancy gives you is a better statistical probability that not all of the redundant components will be broken at once.

It should be noted that multi-homing is just one of many areas of possible redundancy. Anything else that can break -- routers, switches, cables, etc. can all be set up redundantly. No amount of redundancy in any of those components guarantees reliability. What they do mean is that your network can keep functioning if some components break, as long as you still have enough of whatever component it is to keep running.

So, in a redundant setup, what happens when a component breaks? In an ideal situation, it breaks cleanly, fail-over happens automatically, and nobody notices. Then you just have to hope your monitoring system is good enough that you know there's something to fix. But, in an ideal situation, things wouldn't break at all, so designing your procedures around "ideal" failure scenarios doesn't make much sense. What redundancy really gives you is the ability to have outages not turn into major disruptions; the ability, when you see that a component is malfunctioning, to turn it off and go back to sleep. You can then do the real fix later, when it's more convenient or less disruptive.

Thought about that way, there's nothing "Draconian" about turning off a connection (or a switch, or a router, or any other redundant component) that's not doing what you want it to. Instead, you're taking advantage of a main feature of your design. If your other providers are doing 95th percentile billing, you even have a day and a half per month that you can leave a connection down at no financial cost. The alternative, as you seem to have noticed, is to spend your day stressing out about your network not working properly, and complaining about being helpless. You don't need redundancy for that.

-Steve

Thought about that way, there's nothing "Draconian" about turning off a connection (or a switch, or a router, or any other redundant component) that's not doing what you want it to.

While I agree in general with what you are getting at, one point to add is cost. All these goals are constrained within a business case to make. In my case, I could turn off my Cogent connection, but I would have ended up punishing connectivity to other networks that are off Cogent in Toronto only. This would have forced them to get to me via Cogent's pop in Chicago, which was overloaded. So to fix my connectivity into AS577, I would have to hose another group of users in Toronto. Now I could of course add more diversity by adding another connection in Toronto. But, I have to justify the business case to do that. Is it worth the extra money for the few times this particular type of outage happens ? In my case probably not. The cost to privately peer with 577 is quite high and there are no good transit providers at 151 Front that have good connectivity to Bell other than via Chicago.

  Instead, you're taking advantage of a main feature of your design. If your other providers are doing 95th percentile billing, you even have a day and a half per month that you can leave a connection down at no financial cost. The alternative, as you seem to have noticed, is to spend your day stressing out about your network not working properly, and complaining about being helpless. You don't need redundancy for that.

I didnt mean to sound complaintive. My original post to NANOG was more of trying to get details as to what was going on beyond the rather basic info 1st level support and the cogent status page was saying. After the original post, various questions / comments came up as to what could and could not be done in this situation.

         ---Mike

Hi, David, everyone --

And still not getting it. A friend oversees various expensive USG
networks. They pay for physically diverse routing from multiple
sources. Yet every year, when they do an laborious audit down to
the "what fiber, in what bundle, in what trench" level; they find..
Guess What! Yep, someone has moved this circuit or that one to
where both pipes are intimate neighbors.

Is it not possible to require that each of your suppliers provide over a specified path ? I'm planning a build-out that will require a diverse path between two points, and one supplier has named two routes, and promised that they wont change for the duration of the contract. Perhaps I am naive, but a promise should be a promise.

Also instead of buying path A from supplier X and path B from supplier Y, it might be worth buying paths A and B from supplier X and a spare path B from supplier Y too. Supplier X must know they only get dollar N because they can provide both paths.. and in addition diversity has to matter more than money because you are in effect paying for one path twice.

Andy

-a

Note: IANAL, nor do I play one on TV.

Spell out exactly what you want (read: make no assumptions about even the most mundane details) when talking to your account rep and go over the contract and accompanying schedules with a fine-toothed comb before you sign. You want to make sure all those details are spelled out the same way that you provided them to your salescritter and also check for legalese that gives the provider room to do things like re-groom your circuit/lamdba/whatever you're buying onto another (possibly convergent) path without your notification and consent.

Even then, be prepared to take your provider(s) to task and perform due diligence on those physical routes on a regular basis. Note that getting this information in the first place may require you to execute a non-
disclosure agreement. Check the wording of that agreement as well to make
sure that it won't prevent you from a) performing future due diligence and
b) seeking legal relief if a future round of due diligence shows that the
terms of your contract have been breached.

jms

Just naive. Most people make assumptions about what was promised. If it
sounds too good to be true, it probably is. What the sales person promises, the fine print takes away.

http://www.atis.org/ndai/ATIS_NDAI_Final_Report_2006.pdf

You will find out no one will sell to you if the contract requires some things, and the alternatives are rather limited.

I would be more concerned about suppliers that promise things that aren't possible than suppliers that decline to sell things that aren't possible.
Unrealastic buyers are just as much of a problem as non-performance by sellers.

If anyone promises their network will never do down, they will never have
single paths, they are perfect; you should grab your wallet and run away.

I agree with this, and many people take the Ts & Cs, MSA, etc the vendor anyway. We have a standing habit of reading over our new contracts with our attorney on a con call, we always edit them, send them back to the vendor and negotiate on any changes. Its amazing how much you can get things changed in your favor if you're persistent.

More on point for this thread, I always have new vendors bring in fiber maps and show me their paths. Images of the intended path specified on the map are part of the contract, including verbage regarding failover paths. Once I know where their fiber is, I can look for another vendor that takes a different path. Some locations are easier than others of course. A lot depends on what the motto is as to where they like to run fiber, or who they lease/bought their fiber from.

What I find hard to combat are M&A changing operations over time, overlooking contractual obligations on the vendor's part usually. This is a reason we always use 12 mo terms, we can change things fast enough to beat their changing things for us. Sometimes we even go back to the same vendor, just to make sure the new company and contract detail what we have and where it goes. Sounds a little tedious, but at least you know where your circuits go.

Sean Donelan wrote:

Jason LeBlanc wrote:

I agree with this, and many people take the Ts & Cs, MSA, etc the vendor anyway. We have a standing habit of reading over our new contracts with our attorney on a con call, we always edit them, send them back to the vendor and negotiate on any changes. Its amazing how much you can get things changed in your favor if you're persistent.

More on point for this thread, I always have new vendors bring in fiber maps and show me their paths. Images of the intended path specified on the map are part of the contract, including verbage regarding failover paths. Once I know where their fiber is, I can look for another vendor that takes a different path. Some locations are easier than others of course. A lot depends on what the motto is as to where they like to run fiber, or who they lease/bought their fiber from.
What I find hard to combat are M&A changing operations over time, overlooking contractual obligations on the vendor's part usually. This is a reason we always use 12 mo terms, we can change things fast enough to beat their changing things for us. Sometimes we even go back to the same vendor, just to make sure the new company and contract detail what we have and where it goes. Sounds a little tedious, but at least you know where your circuits go.

These are all business points. If you have enough bargaining power, you can get someone to agree to almost anything. If you are paranoid, that may even work for a while. If you seriously intend to renegotiate thousands of circuits involved in building global backbones every 12 months... well, your budgets are going to start skewing in a very funny way.

Its very simple, if I have a simple network between my neighbor and me, I can ensure 100% diversity. 1 Wireless and 1 wired link at all times... as the distances increase, and the number of total nodes and paths increase (paths increase at something like (n-1)^2 or thereabouts), um, it gets exponentially more complex and more expensive to monitor.

On top of that, good/diverse/cheap right-of-ways DECREASE at probably close to a square root rate (especially over long distances). That is a fundamentally difficult problem as n gets very large.

Deepak

More on point for this thread, I always have new vendors bring in fiber
maps and show me their paths. Images of the intended path specified on
the map are part of the contract, including verbage regarding failover
paths. Once I know where their fiber is, I can look for another vendor
that takes a different path.

This often won't get you the most cost-effective connections,
and sometimes it'll be bad for performance as well,
and doesn't always take advantage of available technology.

For instance, if Carrier 1 and Carrier 2 both use the same route for
their primary connection,
and you buy from Carrier 1 because they're 5% cheaper,
you may find that Carrier 2's second-best route is a lot more
expensive that Carrier 1's.
If you're buying from two carriers to get equipment diversity as well
as route diversity, you've lost.

Another kind of problem I've run into in the past - here in
California, to get from SF to LA,
you can either go down the coast or down the Central Valley, depending
on which railroads or highways you like. But there's another route
that takes a railroad connection from
SLO (middle of the coast) to Bakersfield (south/middle of the valley),
and if your primary connection uses that route, the options for
diverse routes go through Salt Lake City or Denver. Given the
history of what fiber got built when, you'll find that
for some speeds many of the carriers use that crossover route, while
for lower speeds
there's a lot more choice.

From a technology standpoint, a lot of carriers are starting to use

intelligent optical switches
that give them automated provisioning, automatic reroutes, etc., so
while they can show you
where their cable routes are, and where the most likely provisioning
and reroutes go,
in general you can't get a precise guaranteed route, because that's
not what the switches do.

What I find hard to combat are M&A changing operations over time,

In general, it's hard for one carrier to keep track of diversity
(though some can),
and much much harder for two carriers to keep diverse from each other.
And the tracking problems scale differently for large connections,
where you may build custom access rings, than for small connections where
most providers are reselling telco last-mile copper.

There's also the problem of diversity philosophy - it's not uncommon
for large East-Coast
companies to view equipment diversity as the critical problem, and
concentrate their
switches into a smaller number of larger sites where they can do
cost-effective sparing,
and have their fiber spread out across many different physical routes,
not remembering that customers in the West Coast expect that
buildings just fall down sometimes, so they care about building diversity,
and geographical and demographic considerations mean that there are only
a few good routes across the Rockies and along the coasts.
(Of course, sometimes this means that the West Coast customers buy from multiple
carriers to get building diversity and _still_ get caught when a telco
DACS fails :slight_smile:

....