www.google.com latency/packet loss/very slow thru savvis

hey is any one seeing a slow google to day... packet loss 22 hops...

visual traceroute show that it goes from chicago to los angeles to singapore
to italy to amsterdam and then finally to google in sunnyvale ca.

router@m40> ping google.com
PING google.com (72.14.207.99): 56 data bytes
64 bytes from 72.14.207.99: icmp_seq=1 ttl=243 time=447.727 ms
64 bytes from 72.14.207.99: icmp_seq=2 ttl=244 time=445.543 ms
64 bytes from 72.14.207.99: icmp_seq=3 ttl=243 time=442.937 ms
64 bytes from 72.14.207.99: icmp_seq=4 ttl=244 time=443.776 ms
64 bytes from 72.14.207.99: icmp_seq=5 ttl=243 time=446.607 ms
64 bytes from 72.14.207.99: icmp_seq=6 ttl=243 time=451.655 ms
64 bytes from 72.14.207.99: icmp_seq=7 ttl=243 time=451.842 ms
64 bytes from 72.14.207.99: icmp_seq=8 ttl=244 time=447.052 ms
64 bytes from 72.14.207.99: icmp_seq=9 ttl=243 time=448.432 ms
64 bytes from 72.14.207.99: icmp_seq=10 ttl=244 time=433.543 ms
64 bytes from 72.14.207.99: icmp_seq=12 ttl=244 time=455.547 ms
64 bytes from 72.14.207.99: icmp_seq=13 ttl=243 time=451.316 ms
64 bytes from 72.14.207.99: icmp_seq=14 ttl=243 time=444.915 ms
64 bytes from 72.14.207.99: icmp_seq=16 ttl=243 time=444.318 ms
^C
--- google.com ping statistics ---
17 packets transmitted, 14 packets received, 17% packet loss
round-trip min/avg/max/stddev = 433.543/446.801/455.547/5.075 ms

router@m40> traceroute google.com
traceroute to google.com (72.14.207.99), 30 hops max, 40 byte packets
1 s216240-1.rtr1s.216240.uschcg2-j20c.savvis.net (64.240.49.173) 5.275 ms
4.889 ms 4.378 ms
2 dcr2-so-2-2-0.Chicago.savvis.net (208.172.3.81) 5.191 ms 60.075 ms
4.844 ms
3 bcs2-so-0-0-0.Dallas.savvis.net (204.70.192.98) 30.408 ms 30.099 ms
30.683 ms
     MPLS Label=595600 CoS=0 TTL=1 S=1
4 bcs1-so-6-0-0.Dallas.savvis.net (204.70.192.49) 30.093 ms 30.035 ms
30.314 ms
     MPLS Label=294001 CoS=0 TTL=1 S=1
5 dcr2-so-2-0-0.LosAngeles.savvis.net (204.70.192.86) 62.299 ms 62.363
ms 62.326 ms
     MPLS Label=442832 CoS=0 TTL=1 S=1
6 dcr1-as0-0.LosAngeles.savvis.net (204.70.192.117) 158.574 ms 62.171 ms
62.628 ms
7 bpr3-so-0-0-0.LosAngelesEquinix.savvis.net (208.172.44.106) 62.246 ms
62.182 ms 64.274 ms
8 singapore-telecommunications-ltd.LosAngelesEquinix.savvis.net
(208.174.196.6) 64.209 ms 62.281 ms 62.374 ms
9 laxeq-cr1.ge-3-0-0.ix.singtel.com (203.208.182.45) 62.767 ms
ge-0-3-0.laxeq-cr1.ix.singtel.com (203.208.168.221) 62.958 ms 62.604 ms
     MPLS Label=126512 CoS=0 TTL=1 S=1
10 so-3-3-2.sngc3-cr1.ix.singtel.com (203.208.154.25) 227.841 ms 225.874
ms 226.254 ms
     MPLS Label=100288 CoS=0 TTL=1 S=1
11 ge-1-0-0.sngc3-ar2.ix.singtel.com (203.208.172.158) 232.373 ms
ge-0-0-0.sngc3-ar2.ix.singtel.com (203.208.172.166) 232.503 ms 239.240 ms
12 203.208.147.238 (203.208.147.238) 424.056 ms * 429.784 ms
13 pal6-pakistan-telecom-1-pk.pal.seabone.net (195.22.197.193) 477.452 ms
478.886 ms *
14 amsix-ams2-racc1.ams.seabone.net (195.22.213.221) 483.282 ms 491.266
ms 480.559 ms
15 core1.ams.net.google.com (195.69.144.247) 448.393 ms 442.054 ms
435.464 ms
16 216.239.43.90 (216.239.43.90) 452.569 ms 460.223 ms 451.533 ms
     MPLS Label=588419 CoS=0 TTL=1 S=1
17 72.14.236.216 (72.14.236.216) 429.854 ms 433.688 ms 448.938 ms
     MPLS Label=775643 CoS=0 TTL=1 S=1
18 66.249.94.98 (66.249.94.98) 448.831 ms 439.760 ms 453.181 ms
19 66.249.94.69 (66.249.94.69) 444.019 ms * 439.384 ms
20 72.14.236.183 (72.14.236.183) 446.575 ms 438.125 ms *
21 * 72.14.236.134 (72.14.236.134) 451.592 ms 457.254 ms
22 72.14.207.99 (72.14.207.99) 452.669 ms * 441.830 ms

Thanks

Erik

Looks ok here from Denver, CO:

PING google.com (64.233.187.99) 56(84) bytes of data.
64 bytes from 64.233.187.99: icmp_seq=1 ttl=235 time=56.4 ms
64 bytes from 64.233.187.99: icmp_seq=2 ttl=236 time=56.2 ms
64 bytes from 64.233.187.99: icmp_seq=3 ttl=235 time=55.9 ms
64 bytes from 64.233.187.99: icmp_seq=4 ttl=236 time=56.3 ms
64 bytes from 64.233.187.99: icmp_seq=5 ttl=235 time=56.2 ms
64 bytes from 64.233.187.99: icmp_seq=6 ttl=236 time=56.4 ms
64 bytes from 64.233.187.99: icmp_seq=7 ttl=235 time=56.0 ms
64 bytes from 64.233.187.99: icmp_seq=8 ttl=236 time=56.4 ms
64 bytes from 64.233.187.99: icmp_seq=9 ttl=235 time=56.1 ms
64 bytes from 64.233.187.99: icmp_seq=10 ttl=236 time=56.3 ms
64 bytes from 64.233.187.99: icmp_seq=11 ttl=235 time=56.0 ms
64 bytes from 64.233.187.99: icmp_seq=12 ttl=236 time=56.3 ms

traceroute to google.com (72.14.207.99), 30 hops max, 38 byte packets
1 10.10.1.254 (10.10.1.254) 1.008 ms 0.346 ms 0.314 ms
2 ai-core.axint.net (38.116.133.1) 0.771 ms 0.597 ms 0.542 ms
3 f0-2.na01.b009854-0.den01.atlas.cogentco.com (38.112.6.145) 1.059 ms
8.405 ms 0.854 ms
4 g0-3.core01.den01.atlas.cogentco.com (66.250.10.217) 4.955 ms 2.292 ms
1.090 ms
5 p5-0.core01.mci01.atlas.cogentco.com (66.28.4.30) 14.825 ms 12.485 ms
11.917 ms
6 p5-0.core02.ord01.atlas.cogentco.com (66.28.4.34) 23.278 ms 23.379 ms
23.330 ms
7 p15-0.core01.ord03.atlas.cogentco.com (154.54.2.242) 105.414 ms 198.623
ms 224.269 ms
8 google.ord03.atlas.cogentco.com (154.54.11.186) 25.238 ms 23.616 ms
23.705 ms
9 66.249.95.253 (66.249.95.253) 23.871 ms 23.776 ms 23.610 ms
10 66.249.95.247 (66.249.95.247) 25.818 ms 23.816 ms 66.249.95.120
(66.249.95.120) 23.601 ms
11 72.14.236.20 (72.14.236.20) 37.726 ms 38.636 ms 37.839 ms
12 66.249.94.68 (66.249.94.68) 37.847 ms 37.883 ms 37.842 ms
13 72.14.236.181 (72.14.236.181) 40.689 ms 38.346 ms 47.902 ms
14 66.249.94.73 (66.249.94.73) 38.500 ms 38.306 ms 72.14.236.130
(72.14.236.130) 43.042 ms
15 72.14.207.99 (72.14.207.99) 38.486 ms 38.637 ms 38.375 ms

You do realise you're talking to a 'different' google.com, right?

That said, I tested both of the above IP's without any problems, via reach.com, if it helps.

Mark.

Erik Sundberg wrote:

hey is any one seeing a slow google to day... packet loss 22 hops...

visual traceroute show that it goes from chicago to los angeles to singapore
to italy to amsterdam and then finally to google in sunnyvale ca.

router@m40> ping google.com
PING google.com (72.14.207.99): 56 data bytes
64 bytes from 72.14.207.99: icmp_seq=1 ttl=243 time=447.727 ms
64 bytes from 72.14.207.99: icmp_seq=2 ttl=244 time=445.543 ms
64 bytes from 72.14.207.99: icmp_seq=3 ttl=243 time=442.937 ms
64 bytes from 72.14.207.99: icmp_seq=4 ttl=244 time=443.776 ms
64 bytes from 72.14.207.99: icmp_seq=5 ttl=243 time=446.607 ms
64 bytes from 72.14.207.99: icmp_seq=6 ttl=243 time=451.655 ms
64 bytes from 72.14.207.99: icmp_seq=7 ttl=243 time=451.842 ms
64 bytes from 72.14.207.99: icmp_seq=8 ttl=244 time=447.052 ms
64 bytes from 72.14.207.99: icmp_seq=9 ttl=243 time=448.432 ms
64 bytes from 72.14.207.99: icmp_seq=10 ttl=244 time=433.543 ms
64 bytes from 72.14.207.99: icmp_seq=12 ttl=244 time=455.547 ms
64 bytes from 72.14.207.99: icmp_seq=13 ttl=243 time=451.316 ms
64 bytes from 72.14.207.99: icmp_seq=14 ttl=243 time=444.915 ms
64 bytes from 72.14.207.99: icmp_seq=16 ttl=243 time=444.318 ms
^C
--- google.com ping statistics ---
17 packets transmitted, 14 packets received, 17% packet loss
round-trip min/avg/max/stddev = 433.543/446.801/455.547/5.075 ms

router@m40> traceroute google.com
traceroute to google.com (72.14.207.99), 30 hops max, 40 byte packets
1 s216240-1.rtr1s.216240.uschcg2-j20c.savvis.net (64.240.49.173) 5.275 ms
4.889 ms 4.378 ms
2 dcr2-so-2-2-0.Chicago.savvis.net (208.172.3.81) 5.191 ms 60.075 ms
4.844 ms
3 bcs2-so-0-0-0.Dallas.savvis.net (204.70.192.98) 30.408 ms 30.099 ms
30.683 ms
     MPLS Label=595600 CoS=0 TTL=1 S=1
4 bcs1-so-6-0-0.Dallas.savvis.net (204.70.192.49) 30.093 ms 30.035 ms
30.314 ms
     MPLS Label=294001 CoS=0 TTL=1 S=1
5 dcr2-so-2-0-0.LosAngeles.savvis.net (204.70.192.86) 62.299 ms 62.363
ms 62.326 ms
     MPLS Label=442832 CoS=0 TTL=1 S=1
6 dcr1-as0-0.LosAngeles.savvis.net (204.70.192.117) 158.574 ms 62.171 ms
62.628 ms
7 bpr3-so-0-0-0.LosAngelesEquinix.savvis.net (208.172.44.106) 62.246 ms
62.182 ms 64.274 ms
8 singapore-telecommunications-ltd.LosAngelesEquinix.savvis.net
(208.174.196.6) 64.209 ms 62.281 ms 62.374 ms
9 laxeq-cr1.ge-3-0-0.ix.singtel.com (203.208.182.45) 62.767 ms
ge-0-3-0.laxeq-cr1.ix.singtel.com (203.208.168.221) 62.958 ms 62.604 ms
     MPLS Label=126512 CoS=0 TTL=1 S=1
10 so-3-3-2.sngc3-cr1.ix.singtel.com (203.208.154.25) 227.841 ms 225.874
ms 226.254 ms
     MPLS Label=100288 CoS=0 TTL=1 S=1
11 ge-1-0-0.sngc3-ar2.ix.singtel.com (203.208.172.158) 232.373 ms
ge-0-0-0.sngc3-ar2.ix.singtel.com (203.208.172.166) 232.503 ms 239.240 ms
12 203.208.147.238 (203.208.147.238) 424.056 ms * 429.784 ms
13 pal6-pakistan-telecom-1-pk.pal.seabone.net (195.22.197.193) 477.452 ms
478.886 ms *
14 amsix-ams2-racc1.ams.seabone.net (195.22.213.221) 483.282 ms 491.266
ms 480.559 ms
15 core1.ams.net.google.com (195.69.144.247) 448.393 ms 442.054 ms
435.464 ms
16 216.239.43.90 (216.239.43.90) 452.569 ms 460.223 ms 451.533 ms
     MPLS Label=588419 CoS=0 TTL=1 S=1
17 72.14.236.216 (72.14.236.216) 429.854 ms 433.688 ms 448.938 ms
     MPLS Label=775643 CoS=0 TTL=1 S=1
18 66.249.94.98 (66.249.94.98) 448.831 ms 439.760 ms 453.181 ms
19 66.249.94.69 (66.249.94.69) 444.019 ms * 439.384 ms
20 72.14.236.183 (72.14.236.183) 446.575 ms 438.125 ms *
21 * 72.14.236.134 (72.14.236.134) 451.592 ms 457.254 ms
22 72.14.207.99 (72.14.207.99) 452.669 ms * 441.830 ms

Thanks

Erik

From seattle.. it looks fine..
ritalin[~]$ traceroute 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 38 byte packets
1 vl101-d1.acc.sea2.hopone.net (209.160.32.2) 1.221 ms 1.307 ms 0.638 ms
2 ge6-0.core1.sea2.hopone.net (209.160.60.242) 1.462 ms 0.269 ms 0.152 ms
3 ge6-0.core2.sea2.hopone.net (209.160.60.243) 0.222 ms 0.301 ms 0.285 ms
4 ge-3-0-0--0.tkwl.eli.net (66.36.224.193) 0.729 ms 1.000 ms 1.104 ms
5 ge-4-1-0--0.cr02.tkwl.eli.net (207.173.115.209) 1.775 ms 0.849 ms 1.167 ms
6 so-0-0-0--0.cr01.mpls.eli.net (207.173.115.166) 37.471 ms 37.276 ms 37.863 ms
7 so-3-0-0--0.cr02.mpls.eli.net (207.173.114.186) 46.812 ms 37.432 ms 37.278 ms
8 so-1-0-0--0.cr01.chcg.eli.net (207.173.115.233) 47.082 ms 47.461 ms 46.652 ms
9 ge-1-0-0--0.er01.chcg.eli.net (207.173.115.138) 46.698 ms 46.376 ms 46.406 ms
10 core1-2-2-0.ord.net.google.com (206.223.119.21) 98.684 ms 47.532 ms 46.726 ms
11 216.239.46.1 (216.239.46.1) 46.967 ms 47.061 ms 46.759 ms
12 72.14.236.20 (72.14.236.20) 61.060 ms 61.015 ms 60.896 ms
13 72.14.236.183 (72.14.236.183) 61.848 ms 61.650 ms 61.801 ms
14 72.14.236.134 (72.14.236.134) 70.552 ms 64.497 ms 66.249.94.72 (66.249.94.72) 61.790 ms
15 72.14.236.130 (72.14.236.130) 65.446 ms 61.785 ms 70.031 ms
16 72.14.236.134 (72.14.236.134) 65.919 ms 64.584 ms 72.297 ms
17 72.14.207.99 (72.14.207.99) 62.743 ms 62.596 ms 62.203 ms

and from Tysons Corner, Mclean VA

$ traceroute 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 64 hops max, 40 byte packets
1 c-vl101-d2.acc.dca2.hopone.net (66.36.230.3) 0.386 ms 0.392 ms 0.507 ms
2 ge4-0.core2.dca2.hopone.net (66.36.224.243) 0.493 ms 0.257 ms 0.460 ms
3 ge2-0-241.core1.iad1.hopone.net (66.36.224.34) 1.221 ms 1.418 ms 1.483 ms
4 core1-2-2-0.ord.net.google.com (206.223.119.21) 28.714 ms 28.847 ms 29.247 ms
5 216.239.46.1 (216.239.46.1) 29.560 ms 66.249.95.249 (66.249.95.249) 29.081 ms 28.739 ms
6 66.249.95.245 (66.249.95.245) 28.622 ms 28.971 ms 28.824 ms
7 72.14.236.20 (72.14.236.20) 43.094 ms 42.856 ms 42.717 ms
8 72.14.236.183 (72.14.236.183) 43.652 ms 43.427 ms 43.673 ms
9 72.14.236.134 (72.14.236.134) 44.317 ms 72.14.236.181 (72.14.236.181) 43.343 ms 43.510 ms
10 72.14.236.130 (72.14.236.130) 44.430 ms 43.862 ms 72.14.207.99 (72.14.207.99) 43.532 ms

hey is any one seeing a slow google to day... packet loss 22 hops...

visual traceroute show that it goes from chicago to los angeles to singapore
to italy to amsterdam and then finally to google in sunnyvale ca.

...yadayadayada...

8 singapore-telecommunications-ltd.LosAngelesEquinix.savvis.net
(208.174.196.6) 64.209 ms 62.281 ms 62.374 ms
9 laxeq-cr1.ge-3-0-0.ix.singtel.com (203.208.182.45) 62.767 ms
ge-0-3-0.laxeq-cr1.ix.singtel.com (203.208.168.221) 62.958 ms 62.604 ms
     MPLS Label=126512 CoS=0 TTL=1 S=1
10 so-3-3-2.sngc3-cr1.ix.singtel.com (203.208.154.25) 227.841 ms 225.874
ms 226.254 ms
     MPLS Label=100288 CoS=0 TTL=1 S=1
11 ge-1-0-0.sngc3-ar2.ix.singtel.com (203.208.172.158) 232.373 ms
ge-0-0-0.sngc3-ar2.ix.singtel.com (203.208.172.166) 232.503 ms 239.240 ms
12 203.208.147.238 (203.208.147.238) 424.056 ms * 429.784 ms
13 pal6-pakistan-telecom-1-pk.pal.seabone.net (195.22.197.193) 477.452 ms
478.886 ms *
14 amsix-ams2-racc1.ams.seabone.net (195.22.213.221) 483.282 ms 491.266
ms 480.559 ms
15 core1.ams.net.google.com (195.69.144.247) 448.393 ms 442.054 ms
435.464 ms

You'd be better off with an AS-PATH, but just follow the chain... Savvis
-> Singtel -> Pakistan Telecom -> Seabone -> Google. Where I'm from, we
call that a routing leak. Doesn't take too much work to guess where
either. :slight_smile:

Oh and FYI it is still going on, though the route just changed 4 mins ago:

                    [BGP/170] 00:04:21, localpref 200
                      AS path: 7473 17557 17557 17557 17557 5400 15169 I

Singtel -> Pakistan Telecom -> British Telecom -> Google. AS17557 is
leaking its BGP table, and AS7473 is not filtering its customer. Bad
network, bad. No cookie.

and... sbgp would have fixed this for us! :slight_smile: Seriously though, in the last
12 months there have been atleast 4 major route leaks like this, eh? Is
the lesson not being learned to filter custoemrs?

SBGP would have. But SIDR produced security extension might not
(at least not initially) because cross-peer authentication is not
part of the initial work requirement at wg formation.