Why don't ISPs peer with everyone?

Mailman List Mirror (Read Only)
21

I'd like to foster a discussion here to better understand this, not rile anyone up. That said, what I see so far is a representation of those who do not recall the halcyon days before a rabid profit motive was the driving force behind ISPs.

Peering in it's original sense is/was free. It was a swap of traffic. That profit motive has created the phrase "settlement free peering" to refer to the original definition so it seems like the free swap of traffic is the aberration. The big ISPs used to seek to balance content hosting and the customer load to avoid having to pay for any sort of transit. AOL was known to acquire companies which had huge downstream traffic for this purpose.

Now we see ISPs waging an economic war with content providers wanting to find a way to charge, say, Google for allowing them to to pass their YouTube content along to the ISP's subscribers. This is the result of letting non-technical, profit-driven managers run the show and not the usually eager to cooperate network engineers who actually understand how this stuff works.

The problem here is that the closer you are to the end user, the harder you're getting screwed, and not in a good way. The very large ISPs are doing real peering, and charging smaller, end-user focused ISPs high transit rates so that they can't possibly compete on price with the inferior, customer-service-impaired ISP end-user offerings. The US government has declined to enforce any sort of rule which might require the huge ISPs to grant wholesale-type access to their physical networks (for better or worse depending on your POV) or examine any of this cartel-type behavior under the light of monopoly rules.

So please, short of socialism, and in light of the rampant legislation-for-sale culture in our government (how many FCC commissioners get jobs with huge ISPs?) how do we fix this?

Please note: I'm not advocating socialism. I might advocate regulation a la public utilities. There is universal agreement that the internet is "critical infrastructure." deregulating other utilities hasn't been uniformly successful, especially when measured from the consumers' point of view. Thoughts?

Rob

22

True, but the point being made is: how *much* better? Is it enough better
to justify the cost of installing and maintaining another peering link?

- Matt

23

I concur, and I specifically would like to see a lot more *geographically*
local peering, so packets from Roar Runner[1] Tampa Bay to FiOS Tampa Bay don't
have to clog up an exchang point in Reston or Dallas; this stuff *will*
eventually bite us in another Katrina-scale event.

Cheers,
-- jra
[1]Roar Runner was a typo, but given most of what the Internet is used for
these days[2], it's so funny I'm going to leave it in.
[2]"Recreational Indignation".

24

What I've found interesting is the cost of circuits seem to not be distance-sensitive. I think this will contribute to mega-regional peering for the foreseeable future.

(ie: dc, sj, dfw, chi, nyc, etc…)

Unless these costs come closer to reflecting a balance then I suspect we will continue to see this regional networking. I had a hard time getting people to interconnect even in the CLEC COLO spaces. very few people had bgp capable devices in those locations, while they were big and had traffic, the gear for running bgp just wasn't there.

- Jared

25

Content providers (e.g. Netflix, Hulu, YouTube) will always try to get their content serviced for little to no cost. The low cost, web-only plan isn't sustainable, and the amount of Netflix traffic around the globe is a good example; There's a lot of traffic that they aren't paying for. The free market only works if entities self-police. But as has been expertly stated, there's no money in that.

I had an idea, I'm sure it's been said before:

If we actually had solid "Tier 1 vs Tier 2 vs Tier 3" thresholds, and we could come up with an agreeable metric, we might be able to minimize the impact of bandwidth hogs (sorry Netflix, pointing at you).

So, if you are a Tier 1, you are required to have at least 10 piers in 10 locations, 5 of which must be Tier 1 providers. If you are Tier 2, that number is halved. It could be a combination of having the "status" of being a Tier 1 provider, but the major benefit is a reduction of the diameter of the Internet. Even done by continent, this could offer enough parallel paths to help address (potentially) the cost of doing business.

I think we would need to have something similar for content providers. To reach Tier 1 status, you are required to have 10 piers in 10 locations, which should cover a set multiple of your total bandwidth (1 TB if it is 500 GB, etc....) For reaching different tiers, they could receive a price break on the cost of Internet circuits.

There would also need to be a middle ground somewhere. Circuits would either need to stop being unlimited or have service thresholds. For exceeding, the content provider would be liable to pay X amount per Gigabit of bandwidth. This would then force Content providers to scale their business rather than relying on the upstream providers' upstream provider to do so.

Not perfect by a great margin, but I think something like that could help.

Sincerely,

Brian A . Rettke

26

Additionally, we share at least one common transit provider, so we'd be trading <1ms for 1-2ms. Obviously, if we were talking about a leased line with any MRC, the answer would be hell no. Since we're able to utilize fiber inside the building with no MRC, the answer is more of a "why bother?" It's not going to save either of us any meaningful amount of transit bandwidth $/capacity.

27

First - I don't work for Netflix! I'm a consumer of their product and a
network engineer who mostly gets stuff.

So I'd like to offer a point of distinction that's kinda bugged me whenever
these conversations pop up here: Netflix the company
doesn't consume bandwidth nor are *they* a bandwidth hog. The consumer is
the bandwidth hog. And the consumer pays their ISP for that bandwidth.
ISP's over provision in the hopes that most folks won't use what they are
paying for and to help keep costs down (very valid). Companies like Netflix
and even Google (I don't know this for a fact - just making logical
assumptions) are not going to rely on peering arrangements of ISPs to
deliver 100% of their traffic. If they did they'd place their business
model in the hands of network operators who don't have Netflix's best
interests in mind. They are going to use caching like products or services
to bring their content closer to the consumer, develop them to be bandwidth
and latency aware, or even make peering arrangements on their own (to your
point). These peering arrangements and products they purchase / pay for are
most likely located within Tier 1 networks in the USA. So technically, if
my assumptions are correct, Netflix probably is paying for their bandwidth
that exits their network. And the consumer is paying for their bandwidth.
Now - Netflix like content providers may cause some of the ISP's to rethink
their over provisioning strategies, but that's not my problem. I'm paying
for my bandwidth, therefore, I want to use it for what I want when I want.
It's my ISP's job to deliver what I'm paying for. This is just my .02 and
that tangent is over for now!

To the original poster - I think it'd be technically impossible to have
every ISP plugged into every ISP, physically ($$ issues aside). How many
ISPs are there and how many routers / ports would you need? And I'm pretty
sure that most Tier 1 ISP's peer with each other - but that's an assumption
not made of fact.

Maybe someday when there really are no bandwidth or latency limitations an
overlay routing model could abstract the physical issues we all deal with
and everyone can logically peer with everyone (although I'm not sure even
that would make sense) but until then a hierarchical model (Tier 1 vs Tier2
etc..) seems to me to make the most sense. Anyway, the implementation of
that hierarchical Internet is driven by $$ of course.

Kenny

28

That's what it really boils down to. How much money can be saved versus performance. If I'm doing a lot of throughput to a specific network, it makes sense that I might want to connect to them, especially if that connection either 1) saves me money or 2) gives me superior QOS/load balancing without a cost increase.

Anything less than 200mbit of traffic isn't even worth me considering these days, and as I grow, I'm sure that number will increase. Content providers generally won't peer unless you meet certain traffic requirements for the same reason.

Jack

29

That's certainly a valid approach for direct (private) peering, it's
not applicable to IXPs offering route servers.

30

In my case, I have to justify the long haul to an IXP as appropriate cost savings, and given that haul often costs more than I pay for transit, it still hasn't justified. Perhaps when I get to multiple 10GE traffic loads and justify leasing a 600 mile dark fiber ring to DFW.

Jack

31

1. For those who can pull it off, getting paid twice for each packet
is better than getting paid once.

2. Your service has a value per byte and a cost per byte. If your
value is less than your cost, you go out of business. Open peering
facilitates greater consumption on the part of your customers. Unless
you're structured to charge them more for that increased consumption,
it reduces the value of each byte you pass.

Unless you're peering with someone in the same or higher tier (who
you'd otherwise have to pay for transit) the odds are you're reducing
the value of your bytes faster than you're reducing your cost.

Personally, I'd love to see 95th percentile billing applied
universally with everybody getting a large pipe the same way everybody
gets a 200 amp electrical service. The problem with that notion is
that A) consumers are hooked on "unlimited," and B) your toaster
doesn't get hacked and start consuming 200 amps all day without your
knowledge.

Regards,
Bill Herrin

32

[snip]

gets a 200 amp electrical service. The problem with that notion is
that A) consumers are hooked on "unlimited," and B) your toaster

Consumers aren't getting "unlimited right now".
They're getting (unknown number of databytes)/month, before the ISP
speed caps, throttles, rate limits them or turns them off for "excessive usage".

doesn't get hacked and start consuming 200 amps all day without your
knowledge.

Your toaster is plugged into an outlet that probably has a 20 amp
circuit breaker on it.
If someone hacks it without your knowledge to eat 200 amps, it will
get turned off.

A similar mechanism could be built into network CPEs.

33

[snip]

gets a 200 amp electrical service. The problem with that notion is
that A) consumers are hooked on "unlimited," and B) your toaster

Consumers aren't getting "unlimited right now".
They're getting (unknown number of databytes)/month, before the ISP
speed caps, throttles, rate limits them or turns them off for "excessive usage".

They're being told they're getting unlimited and for 99% of them it's
true in the sense that their usage does not induce their ISP to impose
its cap. Point is: they expect unlimited and a service which doesn't
claim to be unlimited is, therefore, a non-starter.

Back in the day I faced this problem at my dialup ISP. We had a 240
hour per month cap on dialup usage so that the 24/7 users would buy a
24/7 account or go elsewhere. We started losing business from folks
using 30 and 40 hours a month because the other guy was "unlimited."
So we did some fancy wordsmithing and came up with "unlimited
_attended_ hours" meaning you had to be in front of your computer. How
did we know? Because you sleep too so if you're online for 23+ hours
per day every day, your usage isn't "attended."

Our salesfolk tested the waters, but we couldn't sell a $5/month plus
$0.10/hour product even though that would have resulted in most
customers paying less.

When I say consumers are hooked on unlimited, that's what I'm talking about.

Your toaster is plugged into an outlet that probably has a 20 amp
circuit breaker on it.
If someone hacks it without your knowledge to eat 200 amps, it will
get turned off.

A similar mechanism could be built into network CPEs.

A similar mechanism is built in to network CPEs. It's called the port
speed and the choices are 10, 100 and 1000.

The electrical company metaphor breaks down here. Wiring an appliance
so it can consume your entire electrical service has no desirable
traits. Wiring your computing equipment so they can communicate at
higher speeds within the building than leaving the building is the
opposite.

Regards,
Bill Herrin