Packet Loss to Google, others

Peter_Beckman · May 14, 2009, 3:55pm

Anyone else getting reports of Google (and other destinations) slowness?

From Verizon FIOS Northern VA:

HOST: xxx.angryox.com Loss% Snt Last Avg Best Wrst StDev
   1. xxx2 0.0% 20 0.6 0.6 0.4 0.7 0.1
   2. 10.1.41.89 0.0% 20 1.6 1.8 1.6 3.0 0.3
   3. G2-0-3-891.WASHDC-LCR-08.ver 0.0% 20 1.6 1.7 1.6 2.0 0.1
   4. so-1-1-0-0.RES-BB-RTR2.veriz 0.0% 20 2.2 2.7 2.1 7.5 1.4
   5. 0.ge-5-2-0.BR2.IAD8.ALTER.NE 0.0% 20 2.9 2.9 2.7 3.1 0.1
   6. 204.255.168.30 0.0% 20 6.2 9.4 6.1 70.6 14.4
   7. cr2.wswdc.ip.att.net 0.0% 20 6.5 6.8 6.3 7.3 0.3
   8. 12.122.134.157 0.0% 20 6.2 27.8 6.1 196.8 51.4
   9. ??? 100.0 20 0.0 0.0 0.0 0.0 0.0
  10. 216.239.48.108 55.0% 20 328.2 335.7 326.8 389.6 20.3
  11. 64.233.175.109 50.0% 20 334.7 333.0 325.1 337.4 4.0
  12. 72.14.236.74 30.0% 20 328.3 328.3 324.3 335.5 3.5
  13. he-in-f147.google.com 40.0% 20 335.2 328.6 323.6 337.0 4.5

From Cox Fiber in Northern VA / DC:

HOST: xxx.angryox.com Loss% Snt Last Avg Best Wrst StDev
   1. 70.164.19.3 0.0% 20 0.2 0.2 0.2 0.4 0.1
   2. wsip-70-168-111-17.dc.dc.cox 0.0% 20 1.7 9.6 1.5 77.5 20.9
   3. mrfddsrj01-ge706.rd.dc.cox.n 0.0% 20 13.2 3.9 1.4 18.7 5.2
   4. ashbbbrj01-ae0.0.r2.as.cox.n 0.0% 20 2.5 2.7 2.3 6.4 0.9
   5. 209.85.240.136 0.0% 20 39.5 10.5 3.7 63.0 15.3
   6. 64.233.175.111 0.0% 20 4.9 4.7 4.1 5.9 0.4
   7. 72.14.236.74 0.0% 20 6.0 6.9 6.0 8.5 0.7
   8. he-in-f147.google.com 0.0% 20 4.4 4.6 4.1 7.1 0.8

From Level3 in New York:

HOST: xxx.angryox.com Loss% Snt Last Avg Best Wrst StDev
   1. 208.72.185.177 0.0% 20 0.4 89.0 0.3 303.9 86.1
   2. 208.72.184.245 0.0% 20 0.4 0.8 0.3 7.7 1.6
   3. ge-6-7.car3.NewYork1.Level3. 0.0% 20 0.5 2.3 0.5 29.9 6.6
   4. vlan69.csw1.NewYork1.Level3. 0.0% 20 4.5 6.1 0.6 14.0 4.7
   5. ae-61-61.ebr1.NewYork1.Level 0.0% 20 1.6 1.0 0.7 1.6 0.2
   6. ae-3-3.ebr4.Washington1.Leve 0.0% 20 11.3 10.9 5.4 19.1 4.3
   7. ae-64-64.csw1.Washington1.Le 0.0% 20 14.0 10.9 5.8 18.5 4.8
   8. ae-1-69.edge1.Washington1.Le 0.0% 20 6.2 9.2 5.8 60.1 12.0
   9. ??? 100.0 20 0.0 0.0 0.0 0.0 0.0
  10. 209.85.241.50 50.0% 20 330.9 330.3 329.1 331.6 0.9
  11. 64.233.175.219 55.0% 20 316.6 314.4 311.7 317.3 2.3
  12. 72.14.236.66 60.0% 20 317.4 319.6 314.1 328.0 4.5
  13. he-in-f147.google.com 45.0% 20 323.9 321.5 312.1 332.9 6.9

Andy_Ringsmuth2 · May 14, 2009, 4:01pm

Yeah, definitely seeing it here. Gmail via POP has been dead for a while too. This is via Unite Private Networks (via Qwest ultimately) in Lincoln, Neb.

Last login: Thu May 14 10:35:12 on ttys000
204-Andy:~ andyring$ ping google.com
PING google.com (74.125.45.100): 56 data bytes
64 bytes from 74.125.45.100: icmp_seq=2 ttl=233 time=323.535 ms
64 bytes from 74.125.45.100: icmp_seq=3 ttl=233 time=339.249 ms
64 bytes from 74.125.45.100: icmp_seq=5 ttl=233 time=334.392 ms
64 bytes from 74.125.45.100: icmp_seq=6 ttl=233 time=320.491 ms
64 bytes from 74.125.45.100: icmp_seq=7 ttl=233 time=321.207 ms
64 bytes from 74.125.45.100: icmp_seq=10 ttl=233 time=320.925 ms
64 bytes from 74.125.45.100: icmp_seq=12 ttl=233 time=330.371 ms
64 bytes from 74.125.45.100: icmp_seq=16 ttl=233 time=321.174 ms
64 bytes from 74.125.45.100: icmp_seq=21 ttl=233 time=336.478 ms
64 bytes from 74.125.45.100: icmp_seq=24 ttl=233 time=326.038 ms
64 bytes from 74.125.45.100: icmp_seq=26 ttl=233 time=323.370 ms
64 bytes from 74.125.45.100: icmp_seq=27 ttl=233 time=330.263 ms
64 bytes from 74.125.45.100: icmp_seq=29 ttl=233 time=327.915 ms
64 bytes from 74.125.45.100: icmp_seq=30 ttl=233 time=333.887 ms
64 bytes from 74.125.45.100: icmp_seq=32 ttl=233 time=334.694 ms
64 bytes from 74.125.45.100: icmp_seq=33 ttl=233 time=330.924 ms
64 bytes from 74.125.45.100: icmp_seq=34 ttl=233 time=330.133 ms
64 bytes from 74.125.45.100: icmp_seq=35 ttl=233 time=333.507 ms
^C
--- google.com ping statistics ---
37 packets transmitted, 18 packets received, 51% packet loss
round-trip min/avg/max/stddev = 320.491/328.808/339.249/5.800 ms
204-Andy:~ andyring$

Frederic · May 14, 2009, 4:12pm

From France. nothing

traceroute www.google.com
traceroute to www.l.google.com (74.125.39.104), 64 hops max, 40 byte
packets

3 gi9-13.ccr01.par04.atlas.cogentco.com (149.6.164.177) 1 ms 1 ms 1
ms
4 te1-3.mpd02.par01.atlas.cogentco.com (130.117.2.94) 2 ms
te3-3.mpd02.par01.atlas.cogentco.com (130.117.1.61) 2 ms
te1-3.mpd02.par01.atlas.cogentco.com (130.117.2.94) 2 ms
5 te8-7.mpd02.lon01.atlas.cogentco.com (130.117.3.6) 10 ms
te9-7.mpd02.lon01.atlas.cogentco.com (130.117.3.134) 10 ms
te8-7.mpd02.lon01.atlas.cogentco.com (130.117.3.6) 10 ms
6 google.lon01.atlas.cogentco.com (130.117.14.90) 10 ms 10 ms 10 ms
7 64.233.175.27 (64.233.175.27) 10 ms (TOS=128!) 10 ms 10 ms
8 72.14.233.63 (72.14.233.63) [MPLS: Label 386194 Exp 4] 14 ms
209.85.248.80 (209.85.248.80) [MPLS: Label 605502 Exp 4] 15 ms
209.85.241.84 (209.85.241.84) [MPLS: Label 529410 Exp 4] 11 ms
9 209.85.250.141 (209.85.250.141) 17 ms 209.85.248.80 (209.85.248.80)
[MPLS: Label 452895 Exp 4] 16 ms 209.85.250.141 (209.85.250.141) 19 ms
10 209.85.254.112 (209.85.254.112) 18 ms 209.85.248.44 (209.85.248.44)
16 ms 209.85.254.116 (209.85.254.116) 17 ms
11 209.85.254.126 (209.85.254.126) 18 ms 22 ms 18 ms
12 fx-in-f104.google.com (74.125.39.104) 19 ms (TOS=0!) 209.85.254.116
(209.85.254.116) 18 ms (TOS=128!) fx-in-f104.google.com (74.125.39.104)
18 ms (TOS=0!)

4 mai 2009 à 11:55 -0400, Peter Beckman a écrit :

William_McCall · May 14, 2009, 4:13pm

Issue has cleared on VzB/UUnet

rar · May 14, 2009, 4:13pm

We are seeing issues through our Verizon, and Level3 connections. Fine
through Comcast.
Our Comcast routing goes through Level 3.

Bob Roswell
System Source
broswell@syssrc.com
(410) 771-5544 ext 4336

Tomas_L_Byrnes · May 14, 2009, 4:24pm

All well from Cox in San Diego:

PING googlemail.l.google.com (74.125.19.18) 56(84) bytes of data.
64 bytes from cf-in-f18.google.com (74.125.19.18): icmp_seq=0 ttl=246
time=32.9 ms
64 bytes from cf-in-f18.google.com (74.125.19.18): icmp_seq=1 ttl=246
time=33.7 ms
64 bytes from cf-in-f18.google.com (74.125.19.18): icmp_seq=2 ttl=246
time=35.7 ms
64 bytes from cf-in-f18.google.com (74.125.19.18): icmp_seq=3 ttl=246
time=36.0 ms
64 bytes from cf-in-f18.google.com (74.125.19.18): icmp_seq=4 ttl=246
time=34.8 ms
64 bytes from cf-in-f18.google.com (74.125.19.18): icmp_seq=5 ttl=246
time=31.7 ms

Joel_Esler · May 14, 2009, 4:49pm

We have been receiving reports for about the past hour and a half at
http://isc.sans.org and via Twitter. The situation appears to be clearing
up now.
J

Adam_Hebert · May 14, 2009, 5:24pm

ah sent to wrong email:

Resolution:
Google ack'da maintenance on their core network did not go as planned-Forced traffic to one peer link that was unable to handle all the traffic. Maintenance has been rolled back. Issue has been restored

AdamH