High latency/packetloss in nyc/nj for cogent/level3/zayo?

Hi,

Anyone also experiencing really high lancy and packetloss 80%+ in nyc/nj area for cogent/level3/zayo?

Hi,

wer're working with Telia and Hurricane in NYC and we only see some latency flaps in the HE network .... flapping from 0.3 to ~15ms. Nothing really bad. No visible packet loss.

Best regards

Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: jj@anexia.at
Web: http://www.anexia.at

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

From London, yes to a general problem, but not sure if it’s specific to NYC, just East Coast in general.

Start: Fri Sep 4 22:31:02 2015 BST
HOST: severn Loss% Snt Last Avg Best Wrst StDev
  1.|-- gw1.thn.exonetric.net 0.0% 10 7.7 0.9 0.2 7.7 2.4
  2.|-- e1-3-2.a00.londen03.uk.ra.gin.ntt.net 0.0% 10 0.6 0.8 0.5 1.4 0.0
  3.|-- be3028.ccr21.lon02.atlas.cogentco.com 0.0% 10 0.8 0.9 0.8 1.2 0.0
  4.|-- be2328.ccr21.lon01.atlas.cogentco.com 0.0% 10 1.0 0.9 0.8 1.2 0.0
  5.|-- be2178.ccr42.lon13.atlas.cogentco.com 0.0% 10 1.0 1.0 0.9 1.1 0.0
  6.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
  7.|-- be2268.ccr41.iad02.atlas.cogentco.com 80.0% 10 147.2 145.7 144.2 147.2 2.0
  8.|-- be2657.ccr42.dca01.atlas.cogentco.com 90.0% 10 145.4 145.4 145.4 145.4 0.0
  9.|-- be2149.ccr42.jfk02.atlas.cogentco.com 90.0% 10 145.0 145.0 145.0 145.0 0.0
10.|-- be2149.ccr42.dca01.atlas.cogentco.com 80.0% 10 154.7 152.4 150.1 154.7 3.2
11.|-- be2113.ccr42.atl01.atlas.cogentco.com 80.0% 10 163.3 162.6 161.8 163.3 1.0
12.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
13.|-- ??? 100.0 8 0.0 0.0 0.0 0.0 0.0
14.|-- te0-0-1-0.agr12.dfw01.atlas.cogentco.com 71.4% 7 182.0 181.9 181.7 182.0 0.0
15.|-- te0-0-2-3.nr11.b000821-1.dfw01.atlas.cogentco.com 71.4% 7 183.3 182.8 182.2 183.3 0.0
16.|-- ??? 100.0 7 0.0 0.0 0.0 0.0 0.0

1.|-- hosted-by-i3d.net 0.0% 10 8.1 17.3 0.3 144.6 45.0
2.|-- 80ge.cr0-br2-br3.smartdc.rtd.i3d.net 0.0% 10 0.3 2.1 0.2 9.4 3.0
3.|-- 40ge.cr1-cr0.smartdc.rtd.i3d.net 0.0% 10 0.3 7.3 0.3 13.3 5.7
4.|-- ae51.edge4.London1.Level3.net 0.0% 10 10.6 14.2 7.6 30.4 7.5
5.|-- 4.69.156.9 90.0% 10 224.7 224.7 224.7 224.7 0.0
6.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
7.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
8.|-- cs20.cs90.v.ewr.nyinternet.net 80.0% 10 169.4 168.8 168.1 169.4 0.9
9.|-- 96.47.77.134.static.nyinternet.net 90.0% 10 167.7 167.7 167.7 167.7 0.0
10.|-- ftw.nj.nyi.net 90.0% 10 169.4 169.4 169.4 169.4 0.0

Having this to almost every network located in NYC/NJ that is going through the said three carrier from many locations.

We're seeing issues between the US and northwest Europe (UK / Ireland),
that started around 40 minutes ago. They are fairly unrelated services
(AWS, Linode, L3VPN, commercial IP transit)...so I'm assuming there's some
kind of larger outage going on?

Oliver

Hi,

I do see one of our offices down ... but ftw.nj.nyi.net is reachable without any issue for us (but the path differ from yours):

                                                   Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com 0.0% 47 0.5 5.4 0.4 80.3 17.4
2. cs70.nyi.net 0.0% 47 0.9 6.7 0.8 118.0 21.4
3. cs70.cs80.v.ewr.nyinternet.net 0.0% 47 2.6 8.1 2.0 144.8 27.0
4. 96.47.77.142.static.nyinternet.net 0.0% 47 2.1 2.1 2.0 3.1 0.1
5. ftw.nj.nyi.net 0.0% 47 2.0 2.0 1.9 2.2 0.1

One other thing we do see: looks like Level3 is broken globally ... we can't send traffic from Europe via Level3 to the Asian region:

                                                      Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. er-03.0v-00-03.anx04.vie.at.anexia-it.com 0.0% 101 0.3 0.4 0.3 2.2 0.2
2. cr-04.0v-08-71.anx03.vie.at.anexia-it.com 0.0% 101 0.8 0.8 0.4 17.8 2.0
3. win-b4-link.telia.net 0.0% 101 0.5 1.6 0.5 29.1 4.0
4. level-ic-1573273-wien-b4.c.telia.net 0.0% 101 0.5 3.6 0.4 85.3 12.1
5. ???
6. 4.69.152.144 99.0% 100 307.4 307.4 307.4 307.4 0.0
7. 4.53.208.102 89.9% 100 253.8 255.7 251.3 265.8 3.9
8. TenGE4-2.br01.tok02.pccwbtn.net 87.9% 100 524.0 406.4 366.9 536.7 66.9
9. cr-01.0v-00-08.anx11.tyo.jp.anexia-it.com 91.9% 100 368.1 371.0 365.4 382.9 5.5
10. anx-lg-jp-tyo01.anexia-it.com 96.0% 100 368.9 368.6 366.9 369.3 1.1

We'll start a ticket with Level3 support ... hopefully they will share some information (in the next days ... ).

Best regards

Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: jj@anexia.at
Web: http://www.anexia.at

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

Hi,

We see the same hier from server pingdom servers towards Amsterdam, The Netherlands.
Last hops without loss in the traceroutes were fist:
• 4 xe-8-0-0.bar1.Tampa1.Level3.net (4.53.172.1) 57.509 ms 57.970 ms 58.201 ms
• 4 ae52.edge1.Washington4.Level3.net (4.53.112.25) 2.254 ms 2.453 ms 2.633 ms
and last few I saw 10min ago was:
• 4 xe-11-0-3.bar2.LasVegas1.Level3.net (205.129.18.249) 1.928 ms 2.211 ms 2.484 ms

Surprise, surprise ... the cleaning staff stopped his worked and connected back in the correct cord ...

Looks like everything went back to normal ....

Best regards

Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: jj@anexia.at
Web: http://www.anexia.at

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601