Seeing some packet loss via Cogent.
www.internetpulse.net seems to be lighting up.
I'm noticing it too. POP in Grand Rapids, circuit to Detroit. Packet loss,
but no total loss of connectivity.
Looking at from Level3 via San Jose, NLayer via Chicago, Cogent via NY.
Seems like the trouble starts after: 0.ge-5-0-0.CHI01-BB-RTR1.ALTER.NET
Substitute CHI for NY, SJC etc.
I was just looking at an issue between 701 in Toronto. Seems to be resolved now-- at least the issue I was seeing.
the bad traceroute, looked like
....
3 if-3-0-2.core3.TNK-Toronto.as6453.net (64.86.81.3) 0.988 ms 0.978 ms 1.578 ms
4 209.58.94.10 (209.58.94.10) 1.902 ms 71.416 ms 3.472 ms
5 if-3-1-0-0.tcore1.NJY-Newark.as6453.net (216.6.98.34) 22.286 ms 21.957 ms 29.472 ms
6 if-12-0.mcore3.NJY-Newark.as6453.net (66.198.70.14) 67.961 ms
if-3-0-0.mcore3.NJY-Newark.as6453.net (216.6.57.121) 21.449 ms 20.956 ms
7 if-10-0.core3.NTO-NewYork.as6453.net (216.6.57.66) 21.975 ms 22.467 ms 21.977 ms
8 Vlan1297.icore1.NTO-NewYork.as6453.net (209.58.26.49) 20.977 ms * 30.520 ms
9 0.ae20.BR2.NYC4.ALTER.NET (204.255.168.173) 21.478 ms 21.458 ms 21.477 ms
10 *
11 *
Now its working
....
3 if-3-0-2.core3.TNK-Toronto.as6453.net (64.86.81.3) 0.975 ms
4 209.58.94.10 (209.58.94.10) 1.975 ms
5 if-3-1-0-0.tcore1.NJY-Newark.as6453.net (216.6.98.34) 21.445 ms
6 if-12-0.mcore3.NJY-Newark.as6453.net (66.198.70.14) 54.472 ms
7 if-10-0.core3.NTO-NewYork.as6453.net (216.6.57.66) 112.426 ms
8 Vlan1297.icore1.NTO-NewYork.as6453.net (209.58.26.49) 22.964 ms
9 0.ae20.BR2.NYC4.ALTER.NET (204.255.168.173) 21.455 ms
10 0.ae2.XL4.NYC4.ALTER.NET (152.63.3.117) 21.466 ms
11 0.so-5-1-0.XT2.TOR2.ALTER.NET (152.63.128.121) 34.958 ms
12 0.POS7-1.GW2.TOR2.ALTER.NET (152.63.131.205) 33.958 ms
When it was not working, packets would not get from my AS (11647) to the target in IP in AS701. But packets from 701 would get back to my AS. The AS path in both directions are 701-6453-11647 and 11647 6453 701... I saw a similar outage to VPNs I have in AS15290 which I see as 11647 6453 701 15290. However, I did not have time to check if it was the same behaviour with loss being in one direction. In both cases, IPs that follow 11647 174 701 and 701 174 11647 and 11647 174 7018 15290 and 15290 7018 174 11647 were not impacted.
---Mike
Seeing some packet loss via Cogent.
www.internetpulse.net seems to be lighting up.I'm noticing it too. POP in Grand Rapids, circuit to Detroit. Packet loss,
but no total loss of connectivity.
Interesting. I began to notice trouble reaching google about a week ago from 2 different east-cost Verizon FiOS lines -- one business and one residential -- but my problem appears to be congestion at the AS701 <> AS15169 border. I'm not noticing any loss across AS701.
These problems began around 9am Eastern on Saturday Feb 19th -- see attached smokeping graph. You can see the diurnal pattern in the loss -- packet loss peaks between 9a-4p ET. Of course, I can get toipv6.google.com without any loss at all via my HE tunnel!
In both cases, mtr shows ~50% loss beginning at google-gw.customer.alter.net (152.179.50.62), the first hop in AS15169. It's clear that I must be losing more ICMP than TCP packets given that google webpages come up fairly quickly, but youtube videos hang ever since this started.
Anybody else seeing this?
Thanks,
-Chris
From our experience and smoke pings on Verizon's alternet, they ALWAYS have issues. Which is why we moved away from them.
-Bret
I've been seeing ~50% packet loss to google from FiOS (WDC area) for a while
now. Youtube completely unusable during the day for the most part, but that
has been going on for months to tell you the truth.
~Randy
I'm having issues through Verizon too...I have a server colocated in Vancouver...could it be a Canadian thing with Verizon?
2 l100.phlapa-vfttp-60.verizon-gni.net (98.114.95.1) 8.910 ms 8.760 ms 7.026 ms
3 g3-0-2-860.phlapa-lcr-08.verizon-gni.net (130.81.139.120) 10.711 ms 8.466 ms 10.698 ms
4 * * *
5 so-13-2-0-0.res-bb-rtr2.verizon-gni.net (130.81.19.118) 14.937 ms 15.975 ms 15.148 ms
6 0.ae2.br2.iad8.alter.net (152.63.34.73) 14.346 ms 13.943 ms 14.833 ms
7 * * *
8 * * *
I can ssh to the box from other networks, and here's a traceroute back to my Verzon FIOS IP...other Verizon customers (DSL, etc) report same problem:
2 static-209-17-142-114.gtcust.grouptelecom.net (209.17.142.114) 0.744 ms 0.642 ms 0.620 ms
3 static-66-38-255-9.gtcust.grouptelecom.net (66.38.255.9) 0.750 ms 0.657 ms 0.649 ms
4 GE3-0.PEERA-VANCBC.IP.GROUPTELECOM.NET (66.59.190.6) 0.730 ms 0.694 ms 0.693 ms
5 bx4-vancouver_G1-1-6.net.bell.ca (67.69.199.105) 0.847 ms 0.836 ms 0.828 ms
6 core4-vancouver_ge8-0-0.net.bell.ca (64.230.183.109) 4.637 ms
core3-vancouver_ge8-0-0.net.bell.ca (64.230.183.105) 113.794 ms 17.328 ms
7 core2-seattle_pos4-0-0_core.net.bell.ca (64.230.144.97) 23.686 ms
core1-seattle_pos6-0-0_core.net.bell.ca (64.230.144.89) 4.672 ms
core2-seattle_pos4-0-0_core.net.bell.ca (64.230.144.97) 5.720 ms
8 bx2-seattle_POS10-0-0.net.bell.ca (64.230.186.22) 4.358 ms
bx2-seattle_POS11-0-0.net.bell.ca (64.230.186.26) 4.396 ms
bx2-seattle_POS10-0-0.net.bell.ca (64.230.186.22) 4.353 ms
9 Comcast-peering.net.bell.ca (67.69.246.198) 4.991 ms 4.795 ms 4.795 ms
10 pos-0-4-0-0-cr01.seattle.wa.ibone.comcast.net (68.86.86.137) 8.285 ms 5.258 ms 4.919 ms
11 pos-0-6-0-0-cr01.denver.co.ibone.comcast.net (68.86.87.49) 46.892 ms 46.901 ms 46.952 ms
12 pos-0-13-0-0-cr01.chicago.il.ibone.comcast.net (68.86.85.245) 57.243 ms 57.359 ms 57.333 ms
13 pos-2-13-0-0-cr01.newyork.ny.ibone.comcast.net (68.86.87.25) 85.080 ms 85.020 ms 85.069 ms
14 te-2-1-pe01.philadelphia.pa.ibone.comcast.net (68.86.84.194) 87.264 ms 86.372 ms 86.453 ms
15 75.149.230.250 (75.149.230.250) 86.548 ms 86.489 ms 86.439 ms
Please disregard my reply...I used pine for the first time in months and although this was tagged as a New message, I didn't see the date was from months ago.
However, I AM seeing problems right now as described below...anybody aware of any Verizon issues?
Hi,
Yeah, I was just seeing some issues through TATA (AS6453) with routes
being blackholed in Newark, or at least that was where the packets
stopped. I had to shut my peer with them and I just finished opening a
trouble ticket.
A traceroute to 209.167.35.0/24 from a source addr in 205.211.164.0/24
byte packets
1 teleglobe-vl38-tor (205.211.165.121) 0.259 ms 0.465 ms 0.488 ms
2 if-2-3-513.core4.TNK-Toronto.as6453.net (209.58.16.21) 0.987 ms
0.981 ms 0.565 ms
3 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) 53.390
ms 2.973 ms 2.991 ms
4 if-3-1-0-0.tcore1.NJY-Newark.as6453.net (216.6.98.34) 20.481 ms
62.938 ms 20.967 ms
5 if-2-2.tcore2.NJY-Newark.as6453.net (66.198.70.2) 20.986 ms 21.059
ms 20.871 ms
6 * * *
Not sure if it was after them or coming back to them. Same source addr
through AS174 is fine
---Mike
I was told by TATA one of their core routers in NY is not reachable. So
perhaps some inadvertent black hole routing between them / by them.
---Mike
Do you have a ticket number, Mike? Seems like they are still blackholing traffic.
I will send the ticket offlist. The last update I got from them an hr ago