Hello all,
We have heard from many of our customers in the Northeast region,
specifically PA, MD, and VA who have difficulty connecting to our website
(very slow loading times). We have noticed that if our data center
provider specifically routes our outbound traffic over Hurricane Electric
rather than XO Communications, that connectivity is restored to normal for
our customers. The HE path goes from Salt Lake City to Denver where it is
handed off to TeliaSonera, and they send it to Chicago where it gets handed
off to Verizon. This works great and generally has a ping response of
20-25ms less than the bad route as posted below from one of our servers:
Tracing route to pool-108-52-xxx-xxx.phlapa.fios.verizon.net
[108.52.xxx.xxx]
over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 172.xxx.xxx.xxx (Internal IP)
2 <1 ms 10 ms <1 ms 209-41-xxx-xxx.c7dc.com [209.41.xxx.xxx]
3 <1 ms <1 ms <1 ms e1-5.rt08.gp1.c7dc.com [192.41.0.97]
4 2 ms 1 ms 1 ms ip65-46-51-113.z51-46-65.customer.algx.net
[65.46.51.113]
5 19 ms 18 ms 19 ms vb1611.rar3.sanjose-ca.us.xo.net
[216.156.0.5]
6 21 ms 18 ms 18 ms 207.88.14.226.ptr.us.xo.net [207.88.14.226]
7 18 ms 18 ms 18 ms 206.111.6.122.ptr.us.xo.net [206.111.6.122]
>>>> Verizon owned device with an XO IP address
8 87 ms 86 ms 87 ms b100.phlapa-lcr-22.verizon-gni.net
[130.81.209.187]
>>>> Next hop goes from CA to PA, probably just hidden routing info
9 * * * Request timed out.
10 88 ms 89 ms 90 ms pool-108-52-xxx-xxx.phlapa.fios.verizon.net
[108.52.xxx.xxx]
As you can see, XO sends this traffic from Salt Lake City to San Jose where
it is handed off to Verizon. We've worked with XO and determined that
there are no connectivity issues along their path, which seems to point to
an issue within Verizon's network. I have a couple of tickets open with
Verizon at the moment on behalf of some of our customers, but we have had
trouble breaking through Tier 1.
Our first thought was saturation of the peering point, but XO states that
the link between hops 6 and 7 is using 5Gbps out of 20Gbps capacity, and
the latency on hop 7 (the Verizon owned device with an XO IP) seems normal
whenever we test.
Any Verizon engineers out there willing to take a quick peek at this route
for any obvious issues? It would be a lot easier for us if Verizon
provided Looking Glass servers, but alas...
Thank you,
Ryan
Hello all,
We have heard from many of our customers in the Northeast region,
specifically PA, MD, and VA who have difficulty connecting to our website
there's probably vz customer folk on-list, maybe 'what should they
test for you' would be nice? 
(very slow loading times). We have noticed that if our data center
provider specifically routes our outbound traffic over Hurricane Electric
rather than XO Communications, that connectivity is restored to normal for
our customers. The HE path goes from Salt Lake City to Denver where it is
handed off to TeliaSonera, and they send it to Chicago where it gets handed
off to Verizon. This works great and generally has a ping response of
20-25ms less than the bad route as posted below from one of our servers:
Tracing route to pool-108-52-xxx-xxx.phlapa.fios.verizon.net
[108.52.xxx.xxx]
over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 172.xxx.xxx.xxx (Internal IP)
2 <1 ms 10 ms <1 ms 209-41-xxx-xxx.c7dc.com [209.41.xxx.xxx]
3 <1 ms <1 ms <1 ms e1-5.rt08.gp1.c7dc.com [192.41.0.97]
4 2 ms 1 ms 1 ms ip65-46-51-113.z51-46-65.customer.algx.net
[65.46.51.113]
5 19 ms 18 ms 19 ms vb1611.rar3.sanjose-ca.us.xo.net
[216.156.0.5]
6 21 ms 18 ms 18 ms 207.88.14.226.ptr.us.xo.net [207.88.14.226]
7 18 ms 18 ms 18 ms 206.111.6.122.ptr.us.xo.net [206.111.6.122]
>>>> Verizon owned device with an XO IP address
8 87 ms 86 ms 87 ms b100.phlapa-lcr-22.verizon-gni.net
[130.81.209.187]
>>>> Next hop goes from CA to PA, probably just hidden routing info
9 * * * Request timed out.
10 88 ms 89 ms 90 ms pool-108-52-xxx-xxx.phlapa.fios.verizon.net
[108.52.xxx.xxx]
As you can see, XO sends this traffic from Salt Lake City to San Jose where
it is handed off to Verizon. We've worked with XO and determined that
there are no connectivity issues along their path, which seems to point to
an issue within Verizon's network. I have a couple of tickets open with
Verizon at the moment on behalf of some of our customers, but we have had
trouble breaking through Tier 1.
Our first thought was saturation of the peering point, but XO states that
the link between hops 6 and 7 is using 5Gbps out of 20Gbps capacity, and
the latency on hop 7 (the Verizon owned device with an XO IP) seems normal
whenever we test.
Any Verizon engineers out there willing to take a quick peek at this route
for any obvious issues? It would be a lot easier for us if Verizon
provided Looking Glass servers, but alas...
they have route data sent to routeviews at least...
Hi Chris!
@Ryan, XO was accurate, the peering connection between 6 and 7 is not congested, nor is it taking any sort of errors.
I took a look. The significant increase in rtt seems to happen between 7 and 8 (an Verizon LSP consisting of multiple physical routers). That hop traverses the continental US from San Jose to Philadelphia, so you would expect that, and I don't see anything going on in that path. The end to end rtt is pretty normal, and in any case, it shouldn't prevent your customers from connecting. I am not seeing any packet loss on that path. It's possible that whatever they saw was a transient issue and may be over now.
So, please double check with your customers to verify that they still have problems. If so, continue to escalate the ticket.
-- rj
Hi Chris!
howdy!
@Ryan, XO was accurate, the peering connection between 6 and 7 is not
I had thought Ryan wasn't at XO but an XO customer... I could be
mistaken though.
congested, nor is it taking any sort of errors.
I took a look. The significant increase in rtt seems to happen between 7 and 8 (an Verizon LSP consisting of multiple physical routers). That hop traverses the
yikes lsp's from XT -> LCR now, neat. it's also possible (and
likely) that the return path from VZ customer -> Ryan is across either
the NYC or DCA links to XO (nyc4 or dca6?)... I agree that the ~65ms
cross-country seems on-par with SJC -> PHL on a good day though.
continental US from San Jose to Philadelphia, so you would expect
that, and I don't see anything going on in that path. The end to end
rtt is pretty normal, and in any case, it shouldn't prevent your
customers from connecting. I am not seeing any packet loss on that
path. It's possible that whatever they saw was a transient issue and
may be over now.
So, please double check with your customers to verify that they still have problems. If so, continue to escalate the ticket.
hopefully ryan's having a better day today