Cogent <-> Verizon peering congestion

Hello,

For the last several months, we have been tracking a congestion issue between Cogent <-> Verizon

  Host Loss% Snt Last Avg Best Wrst StDev
  1. router.garlic.com 0.0% 29 0.3 6.1 0.2 160.6 29.7
  2. vl203.mag03.sfo01.atlas.cogentco.com 0.0% 29 2.2 8.1 2.1 161.1 29.5
  3. te0-0-0-14.ccr22.sfo01.atlas.cogentco.com 0.0% 29 2.9 2.7 2.4 3.6 0.2
  4. be2165.ccr22.sjc01.atlas.cogentco.com 0.0% 29 4.1 4.0 3.7 4.8 0.2
  5. be2047.ccr21.sjc03.atlas.cogentco.com 0.0% 29 4.5 4.7 4.3 5.5 0.3
  6. verizon.sjc03.atlas.cogentco.com 22.2% 28 169.3 171.5 168.1 193.5 6.9
  7. so-1-0-0-0.SJC01-CORE-RTR2.verizon-gni.net 37.0% 28 205.8 180.6 171.6 271.6 24.8
  8. A12-0-135.SNFCCA-DSL-01.verizon-gni.net 33.3% 28 172.3 177.5 171.7 250.8 18.3
  9. pool-71-116-122-235.snfcca.btas.verizon.net 25.0% 28 197.9 197.6 195.5 199.2 0.8

We have smokeping's from our side showing 30%+ packet loss from us (AS4307) to Verizon.

All I have gotten from Cogent is a canned response:

I also see major congestion from Cogent to VZ. Amongst other major
networks.

http://i.imgur.com/1z2ZGOr.png

I've seen some Cogent-Sprint congestion today also. About 10% PL at the link.

Yep. Major oversub in our area (LA/SD) - worse for us is same with VZ <-> L3!

James Laszko
Mythos Technology Inc
jamesl@mythostech.com

Cogent support uses the same response when inquiring about Comcast,
CenturyLink, Tata, AT&T etc.

If the "Tier 1s" are really keeping each other congested, are they not
creating an environment where you have to buy from each of them to have a
chance at congestion free paths? Or peer around them.

FYI, here's the latest response from Cogent when I prodded them about
the issue (just received this about 30 minutes ago:

Just to make something clear. I do not own any stock, interest or
have any official relationship with Verizon or Cogent. The
opinions expressed are mine and mine alone as I have come to
understand some of the relationship without the aid of any privileged
information.

Cogent support uses the same response when inquiring about Comcast,
CenturyLink, Tata, AT&T etc.

Yup, certainly a common thread there. It wouldn't take a lot of
digging to determine that those "peers" are most likely compensated in
some way shape or form. Hence the lack of response by the remote
"peer" to upgrade what would most likely be a paid peer.

If the "Tier 1s" are really keeping each other congested, are they not
creating an environment where you have to buy from each of them to have a
chance at congestion free paths? Or peer around them.

The term tier 1 is a marketing term. The last time I looked Cogent
was default free, but certainly not settlement free. Good luck
peering around a lot of the networks in which they have congestion
from. The only real way would be to order additional capacity, buy
transit or live with the situation long enough for customers complain
to the remote network. If that doesn't work then force a partition
with the end objective of Cogent to change from a compensated peer to
a settlement free peer. It's a fun game, it's unfortunate that
customer flows have to be used to force it.

Let me help translate the canned response.

All I have gotten from Cogent is a canned response:

---
The latency and/or packet loss that you are experiencing to this
destination is due to occasional high traffic with Verizon.

    We are sorry that we transmit more traffic to Verizon than we are
willing to pay Verizon for.

We have
repeatedly requested augments to these congestion points and hope Verizon
will comply soon.

     We have demanded that the compensated peers be converted to
settlement free peers with greater capacity, with little to no
response.

While this has been escalated internally to the CEO
level, we encourage you to also contact Verizon customer support with your
concerns and complaints.

     Our CEO is aware that this will cost money but is unwilling to
pay for additional bandwidth, so we are in a stalemate and want
customers to complain to Verizon in the hopes that the squeaky wheel
will get the grease or in other terms, free capacity.

Their delay is a major impediment to internet
traffic overall and contrary to net neutrality requirements.

    Since the connectivity is compensated, Verizon refuses to provide
additional bandwidth at an acceptable (free) cost. So, no money
equals no additional customer capacity.

Our peering
engineers will continue to address this on a daily basis until resolved.

    Our peering folks will continue to pester Verizon's non-commercial
folks by requesting settlement free peers but until enough people
complain to Verizon the requests will fall on deaf ears.

thanks,
charles

He received a very similar response from Level3 when we complained to them about the issue. "We have contacted VZ with no response"

We see pings go from 8ms to 110ms+ from our office FIOS to one of our data centers in San Diego....

LAX handoffs of FIOS traffic to Cogent & Level3 appears the choke points to us.

James Laszko
Mythos Technlogy Inc
jamesl@mythostech.com

We've been having trouble with congestion between Verizon and Cogent in Chicago since May 2013. We had to move some traffic off of our Verizon connection to get around it. Verizon has apparently had an internal ticket open for the problem since February 2013. Their response in August 2013 was "Cogent is designing an upgrade to turn up the bandwidth". Their response a few weeks ago was "Verizon is working with Cogent on increasing more capacity there is no deadline of completion, issue ongoing since 2013 possible completion in 2014". In other words, they're blaming Cogent.

Ben Bridges
SpringNet