L3: Google from DC via the Netherlands?

Mailman List Mirror (Read Only)
1

Seems strange. Had a partial outage on Verizon network this morning around
9:50am EST, then when it came back around 10:05am, google routed via the
Netherlands. My guess is that there's some sort of routing problem making
my fastest or least cost route go to the Netherlands, but I wanted to find
out if there was an ongoing issue, or if the Netherlands simply now serves
the US East Coast for google.

Isn't there anything closer? Or is this just indicative of an ongoing L3
outage? I'm guessing it's cheaper/faster to get to Google somewhere in the
US, but maybe someone on NANOG knows better than I do.

HOST: octal.angryox.com Loss% Snt Last Avg Best Wrst StDev
   1. tomato.angryox.com 0.0% 20 0.6 0.6 0.5 0.7 0.0
   2. 10.1.41.15 0.0% 20 2.1 4.2 1.9 42.8 9.1
   3. p4-2.lcr-02.washdc.verizon-g 0.0% 20 1.5 1.5 1.3 2.3 0.2
   4. 130.81.29.218 0.0% 20 2.0 7.3 2.0 45.0 11.9
   5. 0.so-1-2-0.xl4.iad8.alter.ne 0.0% 20 2.4 2.4 2.3 3.3 0.2
   6. 0.ge-3-3-0.br2.iad8.alter.ne 0.0% 20 2.4 2.4 2.3 2.5 0.1
   7. te-11-3-0.edge1.washington4. 0.0% 20 2.4 12.2 2.3 151.2 33.6
   8. vlan99.csw4.washington1.leve 0.0% 20 3.1 7.3 3.0 15.0 4.3
   9. ae-92-92.ebr2.washington1.le 0.0% 20 3.2 3.3 3.1 3.6 0.1
  10. ae-42-42.ebr2.frankfurt1.lev 0.0% 20 94.1 94.4 93.7 104.0 2.3
  11. ae-2-2.ebr1.dusseldorf1.leve 0.0% 20 106.1 100.6 95.8 107.0 4.1
  12. ae-46-106.ebr2.dusseldorf1.l 0.0% 20 100.4 103.1 98.0 111.0 4.3
  13. ae-41.ebr1.amsterdam1.level3 0.0% 20 107.2 99.9 94.4 107.7 5.7
  14. ae-11-53.car1.amsterdam1.lev 0.0% 20 94.5 95.0 94.1 102.6 2.0
  15. 212.72.42.14 0.0% 20 94.9 98.8 94.5 133.9 12.0
  16. 209.85.254.250 0.0% 20 95.4 95.4 94.8 97.0 0.5
  17. 72.14.233.114 0.0% 20 100.9 107.2 100.7 140.6 9.7
  18. 209.85.255.166 0.0% 20 108.9 109.1 107.5 117.3 2.1
  19. 209.85.255.110 5.0% 20 109.5 112.4 106.3 119.9 4.7
  20. ew-in-f147.google.com 5.0% 20 107.9 108.0 107.5 108.9 0.3

Fri Feb 6 10:56:09 2009
                                             Packets Pings
  Host Loss% Snt Last Avg Best Wrst StDev
  1. tomato.angryox.com 0.0% 8 0.6 0.6 0.5 0.6 0.0
  2. 10.1.41.15 0.0% 8 2.1 2.4 2.1 3.3 0.4
  3. P4-2.LCR-02.WASHDC.verizon-gni.net 0.0% 8 1.4 1.4 1.3 1.5 0.1
  4. 130.81.29.218 0.0% 8 2.1 9.1 1.9 59.0 20.2
  5. 0.so-1-2-0.XL4.IAD8.ALTER.NET 0.0% 8 2.4 2.4 2.3 2.6 0.1
     0.so-6-1-0.XL4.IAD8.ALTER.NET
  6. 0.ge-7-1-0.BR2.IAD8.ALTER.NET 0.0% 8 2.5 2.5 2.4 2.6 0.1
     0.ge-3-3-0.BR2.IAD8.ALTER.NET
     0.ge-5-1-0.BR2.IAD8.ALTER.NET
     0.ge-7-0-0.BR2.IAD8.ALTER.NET
  7. te-11-3-0.edge1.Washington4.level3.net 0.0% 7 2.4 6.0 2.3 27.2 9.4
  8. vlan79.csw2.Washington1.Level3.net 0.0% 7 3.2 8.3 3.2 13.5 4.2
  9. ae-72-72.ebr2.Washington1.Level3.net 0.0% 7 3.7 3.5 3.1 4.0 0.3
10. ae-44-44.ebr2.Frankfurt1.Level3.net 0.0% 7 94.5 94.9 94.4 97.2 1.0
11. ae-2-2.ebr1.Dusseldorf1.Level3.net 0.0% 7 106.2 100.7 96.4 106.7 4.6
12. ae-46-106.ebr2.Dusseldorf1.Level3.net 0.0% 7 107.6 106.1 100.1 108.9 3.0
13. ae-41.ebr1.Amsterdam1.Level3.net 0.0% 7 95.5 100.6 95.4 108.3 5.9
14. ae-11-55.car1.Amsterdam1.Level3.net 0.0% 7 95.3 95.8 94.4 98.7 1.5
15. GOOGLE-INC.car1.Amsterdam1.Level3.net 0.0% 7 97.0 96.7 96.5 97.0 0.2
16. 209.85.254.250 0.0% 7 102.0 97.0 95.7 102.0 2.2
17. 72.14.233.114 0.0% 7 107.5 104.8 101.5 107.5 2.7
     209.85.248.79
18. 209.85.255.70 0.0% 7 109.8 109.1 107.9 110.3 0.8
     72.14.239.123
     209.85.255.166
     209.85.255.20
19. 209.85.255.98 0.0% 7 118.3 114.9 109.7 120.2 4.6
     209.85.255.102
     209.85.255.110
20. ew-in-f103.google.com 0.0% 7 110.0 109.4 108.0 110.6 1.1

2

Looks ok from Boston-

3 core2.po1-bbnet1.bsn.pnap.net (63.251.128.18) 2.590 ms 3.988 ms
3.181 ms
4 207.88.182.33.ptr.us.xo.net (207.88.182.33) 26.636 ms 7.651 ms
11.977 ms
5 207.88.182.18.ptr.us.xo.net (207.88.182.18) 7.603 ms 8.174 ms
7.405 ms
6 216.239.49.217 (216.239.49.217) 8.219 ms 8.388 ms 7.510 ms
7 72.14.236.213 (72.14.236.213) 8.468 ms 66.249.94.235
(66.249.94.235) 16.799 ms 72.14.236.213 (72.14.236.213) 9.196 ms
8 72.14.238.138 (72.14.238.138) 27.405 ms 209.85.248.216
(209.85.248.216) 16.352 ms 15.785 ms
9 72.14.238.235 (72.14.238.235) 15.691 ms 14.641 ms 15.628 ms
10 209.85.255.194 (209.85.255.194) 39.077 ms 72.14.239.136
(72.14.239.136) 31.206 ms 64.233.174.46 (64.233.174.46) 32.528 ms
11 209.85.254.247 (209.85.254.247) 28.897 ms 209.85.254.249
(209.85.254.249) 29.192 ms 29.649 ms
12 209.85.255.194 (209.85.255.194) 29.326 ms gw-in-f100.google.com
(74.125.67.100) 28.393 ms 209.85.255.194 (209.85.255.194) 35.464 ms

3

I'm OK to that IP as well, but when I query www.google.com, I get multiple
IPs, but here are the ones that in in 147:

DNS Server IP Route (for me)
205.234.170.217 (tiggee) 74.125.79.147 Amsterdam
208.67.222.222 (opendns) 64.233.183.147 Amsterdam
4.2.2.1 (verizon) 74.125.19.147 San Jose
198.6.1.3 (uu.net/verizon) 74.125.47.147 Washington DC (yay)

That's a lot of different answers for the same question!

So maybe the problem is that some DNS servers are smarter than others. I
am on Verizon FIOS in Northern VA (DC), and use tiggee's free resolving
service, http://www.resolvingnameserver.com/freerns.html (they are great
for DNS hosting).

I used to use 198.6.1.1, but it seems so did everyone else who used to work
at UUnet, and they shut it off. Rather than using 198.6.1.3 everywhere
(which I'm sure the network folk would appreciate), I found Tiggee's
resolving name service, which is nicely anycasted.

So what's the deal here? Caching?

Beckman

4

So someone from Google has been helpful in pointing out that the resolver
  IP, not YOUR IP, is the one that determines where you get routed to when
  you make a request for www.google.com. Which is simply due to the way
  things are implemented, which makes sense.

  The problem is, here I am, just some guy, and 99%* of the Internet resolves
  to the same IP(s) regardless of who I ask. But then the other 1%*, and
  this would likely be larger players who are diversified and have systems
  in multiple locations and networks, do something funky and give a
  different address depending on where your DNS server is in the network.

  Then throw in the possibility that YOUR DNS servers are anycasted for
  great justice, or at least for good reliability. Now when you base YOUR
  answer on the caching server's IP address, well, it may not make sense.
  It seems that Tiggee and OpenDNS are anycasted, as is DNS Advantage, as
  well as some root nameservers.

  Thus my problem -- because I ask two free resolving name services, which
  I believe to be anycasted, where to go, I get routed to Amsterdam instead
  of a few miles down the road in Ashburn, VA, and spend 100ms instead of
  10ms travelling the globe, costing someone more money for Atlantic Ocean
  transit when it was unnecessary.

  SO. Who's problem is this to fix? Is it:

     1. Me? Am I a dope for using a very reliable but anycasted resolving
        name service? Clearly, I could just use the handy dandy easy to
        remember because I worked there 198.6.1.x, or is that an Internet
        faux pas because technically I wasn't given permission to use it?

     2. Google? They probably have an interest in making sure my experience
        to their services are fast and as close to me as possible, but I'm
        probably a minority and not worth the effort of refactoring a giant
        DNS implementation just to fix my one problem, nay, inconvenience.

     3. Anycasted DNS Providers? Not sure how they could fix it, other than
        flag certain domains as special, and do something special for them,
        but man that smells like a hack.

     4. My ISP? Does the ISP have to gripe at Google for providing bad
        results that causes traffic to go over expensive lines when it could
        have easily gone locally and much more cheaply? I'm assuming that
        sending my traffic over the Atlantic to the Netherlands costs
        SOMEONE more money than if I had gone to a datacenter nearby, both
        physically and network-wise.

     5. Nobody? Is it just the price the customer (me, who helps generate
        income for Google by using Google and clicking AdWords ads all day)
        pays for the reliability, redundancy and fault tolerance that Google
        has implemented?

  I think things are working as implemented -- it's not "broken," but it
  seems it could be better. Then again, sometimes better is more expensive
  than the status quo, either in time or money or both.

  NOTE: I do not admit to knowing that 100% of what I've written is fact,
  and if you know better than I, please correct me and show me the light.

  * Numbers have no basis, just a guess.

Beckman

5

Peter Beckman wrote:

SO. Who's problem is this to fix? Is it:

    1. Me? Am I a dope for using a very reliable but anycasted resolving
       name service? Clearly, I could just use the handy dandy easy to
       remember because I worked there 198.6.1.x, or is that an Internet
       faux pas because technically I wasn't given permission to use it?

We are network operators. We all use robust distributed name servers,
along with our own (usually hidden) primary.

But as a network operator, why aren't you running your own caching resolver?

Not since the '80s have I ever needed to point at somebody else's resolver,
the DNS is much better distributed now.

6

you don't show the route to the DNS servers though...

the resolver's queries to the auth servers obviously aren't going
to be _sourced_ from the anycast address, or the auth servers' answers
would often likely end up going to the wrong instance of the resolver.
so, at least in google's nameservers opinion, the outgoing address
used when the name was looked-up is closer to their european site...

so perhaps you have ended up querying anycast instances of resolvers
close on the network to the servers with the addresses which get
returned, i.e. using resolvers nearer to SJC/AMS.

if that's the case, i'd be much more concerned about using a nearby
resolver for my dns queries, which i use all the time, than being
worried about an extra 100ms the times i use google. (ok, it's common,
but nowhere near as common as querying dns).

there's another possibility though; perhaps these public resolvers
share caches between their various anycast instances, and it so happens
that the lookup that got cached was done from a european site.

7

And you can add this one to your list:

DNS Server IP
212.27.40.240 (free/proxad) 2001:4860:A003:0:0:0:0:68

$ host -t AAAA www.google.com 212.27.40.240
www.google.com CNAME www.l.google.com
www.l.google.com AAAA 2001:4860:A003:0:0:0:0:68

As you can see, no need for http://ipv6.google.com/ to reach Google
over IPv6 using default DNS resolver provided by some French ISPs.
www.google.com starts to be resolved as an IPv4/IPv6 website.

8

IMHO, off the top of my head, on a weekend where I haven't had enough coffee
yet:

     3. Anycasted DNS Providers? Not sure how they could fix it, other than
        flag certain domains as special, and do something special for them,
        but man that smells like a hack.

Anycast is a good thing, but when geo-location style concerns are factored
in maybe they should have region-based anycast addresses.

Interestingly, with Google there could be another similar concern WRT the
IPv6 "trusted tester program" (or whatever the correct name of that is)
where the DNS resolver / organization could have sufficient IPv6
connectivity to qualify, but that capability might not expand to the clients
of / hosts within the service.

/TJ

From: Peter Beckman [mailto:beckman@angryox.com]
Sent: Friday, February 06, 2009 2:51 PM
To: nanog@nanog.org
Subject: RE: L3: Google from DC via the Netherlands?

I'm OK to that IP as well, but when I query www.google.com, I get
multiple IPs, but here are the ones that in in 147:

DNS Server IP Route (for me)
205.234.170.217 (tiggee) 74.125.79.147 Amsterdam
208.67.222.222 (opendns) 64.233.183.147 Amsterdam
4.2.2.1 (verizon) 74.125.19.147 San Jose
198.6.1.3 (uu.net/verizon) 74.125.47.147 Washington DC (yay)

So someone from Google has been helpful in pointing out that the resolver
IP, not YOUR IP, is the one that determines where you get routed to when
you make a request for www.google.com. Which is simply due to the way
things are implemented, which makes sense.

The problem is, here I am, just some guy, and 99%* of the Internet
resolves
to the same IP(s) regardless of who I ask. But then the other 1%*, and
this would likely be larger players who are diversified and have systems
in multiple locations and networks, do something funky and give a
different address depending on where your DNS server is in the network.

Then throw in the possibility that YOUR DNS servers are anycasted for
great justice, or at least for good reliability. Now when you base YOUR
answer on the caching server's IP address, well, it may not make sense.
It seems that Tiggee and OpenDNS are anycasted, as is DNS Advantage, as
well as some root nameservers.

Thus my problem -- because I ask two free resolving name services, which
I believe to be anycasted, where to go, I get routed to Amsterdam instead
of a few miles down the road in Ashburn, VA, and spend 100ms instead of
10ms travelling the globe, costing someone more money for Atlantic Ocean
transit when it was unnecessary.

SO. Who's problem is this to fix? Is it:

    1. Me? Am I a dope for using a very reliable but anycasted resolving
       name service? Clearly, I could just use the handy dandy easy to
       remember because I worked there 198.6.1.x, or is that an Internet
       faux pas because technically I wasn't given permission to use it?

    2. Google? They probably have an interest in making sure my

experience

       to their services are fast and as close to me as possible, but I'm
       probably a minority and not worth the effort of refactoring a giant
       DNS implementation just to fix my one problem, nay, inconvenience.

    3. Anycasted DNS Providers? Not sure how they could fix it, other than
       flag certain domains as special, and do something special for them,
       but man that smells like a hack.

    4. My ISP? Does the ISP have to gripe at Google for providing bad
       results that causes traffic to go over expensive lines when it

could

       have easily gone locally and much more cheaply? I'm assuming that
       sending my traffic over the Atlantic to the Netherlands costs
       SOMEONE more money than if I had gone to a datacenter nearby, both
       physically and network-wise.

    5. Nobody? Is it just the price the customer (me, who helps generate
       income for Google by using Google and clicking AdWords ads all day)
       pays for the reliability, redundancy and fault tolerance that

Google

9

After a few emails traded with David Ulevitch from OpenDNS, it is clear to
  me that they do NOT suffer from this issue, and have a work-around. My
  apologies to David and to OpenDNS for lumping them in and not doing better
  due dilligence when researching this issue.

IMHO, off the top of my head, on a weekend where I haven't had enough coffee
yet:

    3. Anycasted DNS Providers? Not sure how they could fix it, other than
       flag certain domains as special, and do something special for them,
       but man that smells like a hack.

Anycast is a good thing, but when geo-location style concerns are factored
in maybe they should have region-based anycast addresses.

  Anycast is extremely useful for fault tolerance, agreed. But what I
  personally didn't consider, and I don't think other people consider, when
  they chose to use an alternative DNS caching resolution providers is what
  might break or not operate as expected.

  Having traded a few private emails from people smarter than I at Google
  and OpenDNS, I understand the issue much better than when I first posted.
  Thank you to you both.

  Here's a theoretical solution to this problem that I'd like to open for
  discussion.

     In each location where a provider hosts their anycasted service, there
     is likely a local, non-anycasted IP address for each server. When
     receiving a DNS request that is not in the local cache, or has expired,
     make the new request on that local IP address interface, rather than on
     the anycasted IP address interface. In those cases, GSLB records would
     likely return a more accurate set of results for clients making DNS
     requests of it, and when those records were requested from the
     anycasted DNS resolving service, the cached records would more likely
     be closer from a network standpoint to the actual service.

  Obviously there are some issues:
     * need to patch BIND or PowerDNS to use a different interface for
       making new requests
     * possibility of the responding anycasted DNS server being close to
       server farm A, while being far away from DNS record requestor B

  I'm curious to find out if others on the list know what other companies
  are using GSLB, and what the actual impact of anycasted DNS caching
  nameservers has on GSLB records. If enough people are using anycasted DNS
  resolution services, implementing a fix like this would reduce network
  traffic. By how much, I don't know.

Beckman