Is it a fairly normal practice for large companies such as Yahoo! And Mozilla to send icmp/ping packets to DNS servers? If so, why? And a related question would be from a service provider standpoint is there any reason to deny ICMP/PING packets to name servers within your organization?
Thanks,
-Drew
Sounds like one of the global-scale load balancers - when you do a (presumably)
recursive DNS lookup of one of their hosts, they'll ping the nameserver from
several locations and see which one gets an answer the fastest.
Yes, it's a semi-borkken strategy, because it assumes that:
1) ICMP is handled at the same rate as TCP/UDP packets in all the routers
involved (so there's no danger of declaring a path "slow" when it really isn't,
just becase a router slow-pathed ICMP).
2) That the actual requester of service is reasonably near net-wise to the
server handling the end-user's recursive DNS lookup.
Is it a fairly normal practice for large companies such as Yahoo!
And Mozilla to send icmp/ping packets to DNS servers? If so, why?
Some of the DNS load balancing schemes do this, I assume to work out how far
away your server is so they can give geographically relevant answers. If you
are geographically close to your recursive name server, it might even work.
And a
related question would be from a service provider standpoint is there any
reason to deny ICMP/PING packets to name servers within your organization?
I tend to favour filtering some types of ICMP packets and not others, the
packets required for ping hold little fear for me (and are kind of useful),
but YMMV.
My ICMP filtering experience is not DNS specific, you might be able to do
better with DNS server specific rule, but that is too much like
micromanagement for me, others may know a lot more on this.
Drew Weaver wrote:
Is it a fairly normal practice for large companies such as Yahoo! And
Mozilla to send icmp/ping packets to DNS servers? If so, why? And a
related question would be from a service provider standpoint is there
any reason to deny ICMP/PING packets to name servers within your
organization?
Wearing my Mozilla hat here...
I blogged about this (blog.mozilla.com/mrz, somewhere there) and Asa blog'd about it over at http://weblogs.mozillazine.org/asa/archives/2007/03/trying_to_load.html .
Mozilla uses Citrix Netscalers and we're currently using dynamic proximity for load balancing between data centers.
After Asa's post, we found poorly documentation that led to misconfiguration of the probe settings. I've cut down the number of probes (default was icmp, udp and tcp:80 to a nameserver) and instead of the ~10 complaints a day I was getting, I get many one a month.
If you're still annoyed by the probes, ping me off-list.
- mz
> Is it a fairly normal practice for large companies such as Yahoo!
> And Mozilla to send icmp/ping packets to DNS servers? If so, why?Sounds like one of the global-scale load balancers - when you do a
(presumably) recursive DNS lookup of one of their hosts, they'll ping
the nameserver from several locations and see which one gets an
answer the fastest.Yes, it's a semi-borkken strategy, because it assumes that:
1) ICMP is handled at the same rate as TCP/UDP packets in all the
routers involved (so there's no danger of declaring a path "slow"
when it really isn't, just becase a router slow-pathed ICMP).
This is aimed at hosts, not routers, right? As far as I know, routers
don't slow-path forwarded ICMP. Hosts will probably reply to ICMP from
their kernel, so it's a faster response than a user-level DNS reply.
2) That the actual requester of service is reasonably near net-wise
to the server handling the end-user's recursive DNS lookup.
Right. But there's no particular reason to block it, unless the rate
is high enough that it's causing you CPU or network load problems. (If
it's your IDS that's getting overloaded, perhaps tell it not to worry
unless you see other load issues...)
--Steve Bellovin, Steven M. Bellovin
But why would they care where the nameserver is? Point 2 would seem to
be a little stupid a thing to assume. Also, what happens if, at that
moment, the ICMP packet is stuck in a queue for a few ms making the
shortest route longer.
While point 2 is a bad assertion if you depend completely upon it, it's
not necessarily a bad starting point if you have no other data to go on.
1. 90+% of resolvers are topologically proximate to either the
requestor, or, the requestors NAT box that you will have to
talk to anyway.
2. At the GLB level, you really don't have any data other than the
IP address of the resolver upon which to base your GLB decision.
Since you'll be right 90+% of the time, and, only sub-optimal,
not broken the other <10% of the time, it generally works OK.
3. When I worked for Netli, before they were acquired in what I would
call a much less than ethical transaction, we maintained an
exception table for cases where we learned that the DNS
resolver was not topologically proximate to the requestors
that flowed through it. We also spent a fair amount of time
explaining the benefits of having the resolver be topologically
proximate to our customers and their customers.
The Netli system was designed to be quite gentle in the amount of
probing it did, but, we did occasionally get messages from people
with paranoid IDS boxes. Usually, once we explained that our
efforts were directed at improving the quality of service to their
users, and how the system worked and how little traffic we sent
their way to accomplish this, they were happy to reconfigure their
alarm preferences.
I don't have first hand knowledge of anyone elses use of these
kinds of ICMP probes, but, I would say that generally, they are
somewhat useful and mostly harmless.
Owen
Sounds like one of the global-scale load balancers - when you do a
(presumably) recursive DNS lookup of one of their hosts, they'll ping
the nameserver from several locations and see which one gets an
answer the fastest.
Why would they ping rather than just sending the query to all of the
NS and see which one answers first? It's an IP round trip either way.
I agree that pinging is harmless, but for this application it seems
pointless, too.
R's,
John
If you have sites in San Fran, London, and Tokyo, and you launch a ping from
all 3 and see which one gets there first, you'll *know* the RTT from each site.
If you just send DNS replies from all 3, you don't have a good way of telling
which one got to the destination first.
Your method works if *I* want to know which one of the 3 sites is closest
(assuming I can identify an DNS server at the 3 sites). The problem of the
owner of the 3 sites trying to identify which one I'm closest to isn't
symmetric to it.
ping targets are of the caching nameserver type, not the authoritative
nameserver type.
Duane W.
Well, they don't *directly* slow-path it. But we've seen *plenty* of cases
of "multi-hop performance as indicated by ICMP Echo Request/Reply doesn't at
all match throughput/latency as indicated by TCP-level stats" mentioned on
this list...
Why would they ping rather than just sending the query to all of the
NS and see which one answers first? It's an IP round trip either way.If you have sites in San Fran, London, and Tokyo, and you launch a ping from
all 3 and see which one gets there first, you'll *know* the RTT from each site.If you just send DNS replies from all 3, you don't have a good way of telling
which one got to the destination first.
Um, unless I seriously misunderstand the client DNS cache wants to know which server is closest. So it sends DNS queries to all three NS at the same time. Then it waits for the answers. Whichever one answers first is the closest. What am I missing?
Regards,
John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies",
Information Superhighwayman wanna-be, http://www.johnlevine.com, ex-Mayor
"More Wiener schnitzel, please", said Tom, revealingly.
Well... we're talking about recursive resolvers. There's not
really a simple way for a third party to measure the round trip time to
the recursive resolver at the dns level.
It may not respond to external queries at all, and even if it does,
what query would you send that would cause an immediate reply
without any additional processing or network latency at the resolver?
There's lots of tricks you can play to do this, but most of them are
no better than a simple ICMP ping.
Cheers,
Steve
The client DNS doesn't know there is more than one server. It queries for www.$FOO.com, and the authority for $FOO.com replies with the IP address of the 'closest' web server.
This result could be pre-calculated by all the web servers pinging the client DNS. It could be done lots of ways, but that is what we are discussing today.
Owen said it worked well for his customers (in a past life), and he has operational experience with this. Can anyone give a serious counter example _from experience_? Or are we just discussing possibilities?
first I agree that in most cases the 'RTT to client cacheresolver'
probably works well enough. That said though...
Owen said it worked well for his customers (in a past life), and he
has operational experience with this. Can anyone give a serious
counter example _from experience_? Or are we just discussing
possibilities?
Sure, 75% of the people that use cache00.ns.uu.net and aren't in the
'mid-atlantic' region... (and someone's blocking echo-request to
cache00 but...) Unless there exist exceptions and other metrics for the
clients it's probably not very accurate in this instance (or the other
cacheXX.ns.uu.net cases really, since they tend to move about the network
as required...)
John L wrote:
Um, unless I seriously misunderstand the client DNS cache wants to know which server is closest. So it sends DNS queries to all three NS at the same time. Then it waits for the answers. Whichever one answers first is the closest. What am I missing?
The bind_garbadge_collection_delay.
From time to time bind goes to sleep, depending on the size
and number of zones the nameserver does host.
Maybe it is internal garbadge collection or memory reclaiming.
Maybe it is the operating system that causes the delay.
That delay is very random.
icmp is answered by the network layer.
dns is answered after the network layer passes the packet to the operating system,
after the operating system passes the packet to the tcp/ip subsystem,
after the tcp/ip subsystem passes the packet to the socket subsystem,
after the memorymanager swappes bind back into memory - and all the way back.
Each of the steps could envolve the memory manager swapping memory to disk or
from disk. Each of the delays could be longer that all network delays.
I have seen nameservers answering after seconds.
I have seen pings to those servers returning after only 30 millisecs.
Kind regards
Peter and Karin
It's harder than it looks, given the number of people who pop up on this list
and ask how to get BGP to Do The Right Thing when 2 paths are the same length
but vastly differing in bandwidth, number-of-actual-hops, and/or latency....
They use ICMP/Echo Request to calculate a rough surrogate latency estimate for potential users of that caching DNS server so they can return different DNS answers depending on your network topology. Yes its an approximation, and can be wrong. Some networks even re-route ICMP/Echo to a completely different box which just responsed to pings; so it may not
even go to the same place. Given all those caveats, many times its still the best guess you can make.
ICMP/ECHO is a separate protocol which is easy to filter if you want to, without affecting "normal" TCP/UDP/etc packets. But then expect to get "worse" default DNS answers from those same sites attempting to optimize their DNS answers.
It would be cool if people ran NTP port 123 on their DNS servers, and then we could get extreme measurements. But then I'm sure someone would point out 62 flaws with that. In the end, its up to each network operator to make its own decision. If your DNS servers aren't
being negatively impacted, and it helps your users get better answers,
you might keep it. If the answers are reversed, you might drop them.
My IDS is badly tuned.... Well maybe there is a fix for that.
I think you're talking about the performance of BIND 9 as a cache. This is pretty tangential, but there was a good presentation on work being done to fix that performance problem by Michael Graff at the recent DNS ops workshop in Chicago:
http://public.oarci.net/files/dnsops-2007/Graff-BIND9-cache.pdf
Joe
Plus you are assuming BGP knows the right answer.
People here are arguing ICMP Echo is a bad metric. BGP makes ICMP look like the gold standard.