ISP network design of non-authoritative caches

Mailman List Mirror (Read Only)
1

I appreciate the redirect, if there is a better list, but
my question is directed at network operators. namedroppers
is for dns protocols, dnsops is for operators of authoritative
name servers.

The majority of users on today's Internet will never directly
query any root name server, or any other authoritative name
server. Instead of a set of authoritative servers, the servers
which actually deliver direct DNS service to users/hosts are
non-authoritative, caching servers. There are more caching,
name servers at the edge of the net than there are Akamai boxes
in the world.

In the late 1980's and early 1990's, when the net was much more
interesting (i.e. flaky, low-bandwidth, expensive circuits) network
operators carefully planning where to place caching-only name servers,
and configuring end-systems to use the appropriate set of servers.
A well-configured set of caching-only name servers can maintain
the illusion of DNS for several hours, even during a network partition
or loss of many authoritative name servers, at least for the "popular"
names. They work so well, people forget they can still have problems.

During the boom times, ISPs couldn't individually configure millions
of DNS clients. They generally told subscribers to use two statically
configured name servers, or more recently used DHCP to set them. Several
national ISPs, including the one I use, with millions of subscribers,
appear to still do this.

We know this isn't good engineering practice, because another national
ISP with millions of subscribers configured their network the same way,
and experienced a multi-hour service disruption affecting most of their
users a couple of years ago when an error blocked access to their two
caching-only, name servers.

There is lots of "best practice" information for configuring authoritative
name servers (including the root and TLDs). The BOG, O'Reilly, DNSOPS,
RFCs, etc. There are several "managed service" companies which will
maintain authoritative name servers for you.

Although most of this stuff seems obvious, network providers seem
to get bitten by the same obvious things over and over again.
Is there a white paper, best common practice, or book which shows
the naive ISP (whether they have 10 or 10 million subscribers) how
to architect their DNS system? Medium and large organizations having
firewalls with internal/external DNS, which already includes local
caching. This seems to be mostly a large, national ISP issue. By
their nature, small ISP networks tend to have "shared fate" among
all of their systems anyway.

2

During the boom times, ISPs couldn't individually configure millions

    > of DNS clients. They generally told subscribers to use two statically
    > configured name servers.

Many of them, like us, tell subscribers to use two statically configured
_service addresses_ which describe the internal-DNS _service_, and are
resident on all customer-facing DNS servers throughout our infrastructure.

The only reason to give them two addresses rather than one is to keep the
clueless ones from complaining about lack of redundancy. Both addresses
are typically hosted on all machines, though.

                                -Bill

3

Some ISPs do this, its fairly easy to check. It is one of several
methods an ISP could use. Is there a paper, book, etc which we could
give to ISPs documenting such practices? Or do you have to hire the
right consultant, who knows the proper incantation?

If you look at some of largest consumer ISPs which outsource much of
their infrastructure, they don't have customer-facing servers distributed
throughout their infrastructure. Or they distribute customers among the
servers using a very unusual algorithm.

I'm using Mindspring/Earthlink tonight, and my DNS resolver is using
a server in Dallas (if you believe the in-addr.arpa traceroute).
Earthlink could intercept the DNS/UDP packets to port 53 and route
them differently, but I don't think that's true.

Tracing route to ns1.mindspring.com [207.69.188.185]
over a maximum of 30 hops:

  1 98 ms 88 ms 94 ms arc-6a.oak.mindspring.net [209.86.81.157]
  2 95 ms 99 ms 99 ms cisco-g4-0-0.oak.mindspring.net [209.86.81.129]
  3 110 ms 103 ms 104 ms cisco-s6-1-0.lax.mindspring.net [207.69.144.34]
  4 105 ms 103 ms 104 ms cisco-1-s5-0-1.pas.mindspring.net [209.86.67.10]
  5 130 ms 114 ms 119 ms cisco-s1-1-0.phx2.mindspring.net [209.86.67.13]
  6 150 ms 159 ms 148 ms cisco-s3-1-1.dal.mindspring.net [209.86.66.173]
  7 150 ms 149 ms 149 ms cisco-6-s2-0-0.dal2.mindspring.net [209.86.67.26]
  8 150 ms 154 ms 155 ms foundry-5-ve4.dal2.mindspring.net [207.69.217.227]
  9 160 ms 154 ms 153 ms ns1.mindspring.com [207.69.188.185]

ATT Worldnet appears to have more DNS caching name servers spread
around the country, but I get assigned servers in Missouri and DC
when I dial into a California POP. The RTT matches a coast to coast
trip.

4

I've always advocated that at least one caching DNS be located in every
POP. In the old days, I designed it right into the NAS router(s).

In the original IPv6 Neighbor Discovery, the routers announced the DNS,
so that it could be in the router or on a separate device.

Then, as commercial NAS's became available without DNS built in, and
SIP(P) IPv6 didn't take off, I've tried a "service" style using net 10,
but it was hard to manage and explain (at least by our entry level
support), so I gave up.

Ideally, clients would use DHCP INFORM to find the local POP servers.
But none of the clients seem to bother.

Nowadays, we just give the customers that statically configure a pair
of central site servers (still on separate subnets), and configure the
NAS's with the local POP server using the bogus M$ PPP options.

That seems to cover our users pretty well.

If we could agree on a standard service (sometimes called anycast) set
of IP addresses, that might work....

All these techniques should be written up in a BCP.

5

One of the reasons that centralized nameservers get used for big
dialup ISPs (such as Mindspring, Sean's earlier example) is that the
folks who provide outsourced dial ports often don't put a nameserver
in each POP, or are at least bad about giving their customers a map of
them so the RADIUS servers can do the right thing. An unfortunate
side-effect of this practice is that it can pessimize Akamai service
(can't speak for DI and other CDNs).

                                        ---Rob

6

dnsops is for operators of authoritative name servers.

dnsop (note singular) is for non-protocol, but still technical, aspects
of the dns. i am not aware of an ietf wg which limits parcipitation by
occupation. if you want cliques, go to icann :-).

Instead of a set of authoritative servers, the servers which actually
deliver direct DNS service to users/hosts are non-authoritative,
caching servers.

some measurements show a large number of combo servers, i.e. they are
authoritative for their local domain(s), say foo.com, but also act as
recursive caching servers for the users of a site.

During the boom times, ISPs couldn't individually configure millions
of DNS clients. They generally told subscribers to use two statically
configured name servers, or more recently used DHCP to set them. Several
national ISPs, including the one I use, with millions of subscribers,
appear to still do this.

We know this isn't good engineering practice

well, actually, a number of the large providers use many servers at the
same v4 anycast address. so they get fairly rich geographic/topologic
dispersion, but don't confuse users with a dozen addresses. i consider
this reasonably good engineering practice. ymmv.

setting up the routing for this is a bit of a hack, but not all that
hard. and the magma wg's work may give us some simpler tools.

Is there a white paper, best common practice, or book which shows
the naive ISP (whether they have 10 or 10 million subscribers) how
to architect their DNS system?

not of which i am aware. wanna help write a dnsop i-d?

randy

7

> dnsops is for operators of authoritative name servers.

dnsop (note singular) is for non-protocol, but still technical, aspects
of the dns. i am not aware of an ietf wg which limits parcipitation by
occupation. if you want cliques, go to icann :-).

Thanks for the correction. I read the DNSOP charter and parts of the
mailing list archive, and had the misconception the work was focused
on authoritative name servers.

> Is there a white paper, best common practice, or book which shows
> the naive ISP (whether they have 10 or 10 million subscribers) how
> to architect their DNS system?

not of which i am aware. wanna help write a dnsop i-d?

Yes. I didn't want to duplicate work. If it doesn't exist, I
can put together at least a few obvious good practices any operator
should consider.

8

You mean there are national ISPs out there, who have exactly 2
caching nameservers, as opposed to configure their clients with
2 IP addresses (perhaps always the same 2) which perform name
resolution? Wow. Is this some sort of retro fashion?

9

Is there a white paper, best common practice, or book which shows
the naive ISP (whether they have 10 or 10 million subscribers) how
to architect their DNS system?

not of which i am aware. wanna help write a dnsop i-d?

Yes. I didn't want to duplicate work. If it doesn't exist, I
can put together at least a few obvious good practices any operator
should consider.

the only work in this area of which i am aware is ripe-192, and that is
for a simple leaf site. note that it tool over a year to get consensus
on this document, but the authors were kinda busy folk.

randy

10

>>> Is there a white paper, best common practice, or book which shows
>>> the naive ISP (whether they have 10 or 10 million subscribers) how
>>> to architect their DNS system?
>> not of which i am aware. wanna help write a dnsop i-d?

The first thing is to change your root cache file to contain

A.ROOT-SERVERS.ORSC thru
M.ROOT-SERVERS.ORSC

or the PacRoot servers like CONDOR.TALLSHIP.NET

So that your customers can see all of the internet and not just what ICANN
wants you to
see.

There are instructions for ISPs on the website listed below. Its already
done.

John

11

Yes, there are.

No, it's not fashion. Though it's about as foolish.

Names avoided to prevent NDA lawsuits. You can fill in the blanks.

12

nanog@adns.net ("John Palmer (NANOG Acct)") writes:

The first thing is to change your root cache file to contain
...
or the PacRoot servers like ...

So that your customers can see all of the internet and not just what ICANN
wants you to see.

bzeep. nonsequitur. there can only be one root zone. if you change your
dns configuration to subscribe to something else, then you're off "the internet"
in a technical sense. (that some root-looking zones incorporate all icann data
past present and future is merely a testament to the fact that "the internet"
means "what the one true official root zone includes", and should not be
indicative of some kind off odd "value subtraction service" by a root-like zone
publisher.)

13

Sorry to burst your reality distortion bubble Paul, but there are a very
large number of us who don't consider ICANN to be either representative of
our best interests, nor the ultimate authority on "the Internet".

14

And there are a large number of trolls and horse beaters on this list
also. Let us not fall into the dnsroot and email filtering hellholes yet
again. Please...

andy

15

bzeep.

Ak! Troll food. Rinse & spit.

Alex Bligh
Personal Capacity

16

nanog@adns.net ("John Palmer (NANOG Acct)") writes:

> So that your customers can see all of the internet and not just what ICANN
> wants you to see.

bzeep. nonsequitur. there can only be one root zone.

Nonsense. BCP32 (RFC 2606) specifically alters a root zone by adding reserved TLDs. How many combinations of the root does that provide for? And RFC2826 has been clarified here on the Nanog list as "one root at a time" and not mixing multiple roots. OK by me, but at some point sanity should prevail and that running code and rough consensus demands the peering of non-conflicting TLDs for everyone's benefit. It's a common practise in other spaces, so why not in the DNS space? I don't understand the "my way or the highway" mentality.

http://www.ietf.org/internet-drafts/draft-higgs-virtual-root-00.txt

if you change your
dns configuration to subscribe to something else, then you're off "the internet"
in a technical sense.

Nonsense. I've not used the legacy root since I proposed the shared-TLD model (and explained it to Postel) back in 1995/6 (FYI, the last version of the Internet Draft is at the IAHC website). Everytime Verisign/NSI screws up the .COM glue I don't notice a thing. Something to be said for that. At least one ICANN board member recognizes the limitations under ROOT-SERVERS.NET and chooses another root for service. I don't blame 'em.

(that some root-looking zones incorporate all icann data
past present and future is merely a testament to the fact that "the internet"
means "what the one true official root zone includes", and should not be
indicative of some kind off odd "value subtraction service" by a root-like zone
publisher.)

F.ROOT-SERVERS.NET is a "value subtraction service". It misses published portions of the name space - so if you use it, it is of limited value. I can reach twice as many TLDs elsewhere. Oh yeah. It's "stable". I forgot.

What ticks me off is the zero-sum game being played. Anyone saying "there can only be one root zone" and supporting a closed and non-inclusive root is playing a zero sum game. "We win, you lose" is not the spirit of the internet, running code and rough-consensus.

Best Regards,

Simon

17

s/a/the/

18

F.ROOT-SERVERS.NET is a "value subtraction service". It misses published
portions of the name space - so if you use it, it is of limited value. I
can reach twice as many TLDs elsewhere. Oh yeah. It's "stable". I forgot.

F publishes whatever the zone's owner has edited into it. Never more or less.

What ticks me off is the zero-sum game being played. Anyone saying "there
can only be one root zone" and supporting a closed and non-inclusive root
is playing a zero sum game. "We win, you lose" is not the spirit of the
internet, running code and rough-consensus.

I'm making no arguments (at this time and in this place, at least) for one
single-owner versus another. However, there's only one real root zone at any
given moment, according to DNS's specification and at least one implementation.

One expects that an argument as to who that single owner ought to be is
welcome somewhere (other than nanog, that is), but any argument about how
many different actual root zones there can be would not be welcome at all
(except, I guess, if you're IPv8-connected.)

"What ticks me off" is the folks who want the owner/contents of the root zone
to be different than they are, and who therefore offer up completely bizarre
misinterpretations of hard technical fact in order to support a de-facto
ownership change (by changing one to many rather than arguing about the one.)

If you don't like ICANN, fine, there's a mailing list somewhere and you ought
to go make yourself heard on it. But don't pervert DNS, fragile and perverse
as it already is, in order to make or avoid various political realities.

19

(subject rename with apologies to Peter, Paul, and Mary)

Date: 18 Nov 2001 19:23:37 -0800
From: Paul Vixie <vixie@vix.com>

However, there's only one real root zone at any given moment,
according to DNS's specification and at least one
implementation.

Tired of domain inavailability? Burned by a scalper? Use us
as your root, and buy domains from me! Get any domain that you
really want!

Once we start down the slippery slope of "I'm a root too", how
many different ad hoc DNS "universes" (for lack of better
term) must we have before we decide that things are "broken"?

I'd hazard a guess that 99.9% of Internet users want things to
"just work" cohesively and consistently. Crimony, users of a
certain "get 1000 hours free your first month"[*] service have
troubles realizing that ".com" is not optional and automatically
assumed. I've seen countless people enter domain names in search
boxes, not realizing the difference between a search engine input
and an URL!

[*] Lest I get reminded how many hours are in a month, I know.
Note quotes. Interpret referenced text seriously at your own
risk. May cause drowsiness.

Maintaining a single, authoritative root seems, IMHO, to be a
Good Thing. Given multiple registries, namespace collisions
would get ugly -- and, even in the absence of collisions, let us
consider "reachability" issues.

This isn't a question of "I like ICANN" or not... it's a question
of the system. Do people decide that they like or dislike our
system of government based solely on specific elected officials?
(I suppose that some do, and this will open a tangent thread, but
I presently can't think of a better analogy.)

Cheap domains from the EverQuick root! Guaranteed to work at a
handful of local dialups and dedicated-line customers! :slight_smile:

I think that I'll start handing out ASNs and IP space, too...

Eddy

20

<snip>

Once we start down the slippery slope of "I'm a root too", how
many different ad hoc DNS "universes" (for lack of better
term) must we have before we decide that things are "broken"?

I'm pretty sure that if you were to ask most of us who use alternative
roots, the reason would be that we are, in our own little way, trying to
"fix" the DNS by doing so.

In my case, I believe that ICANN is totally corrupt, *completely*
oblivious [by choice] of how their corruption impacts the real world, and
finally, that because of this deeply entrenched corruption, the DNS is
de-facto "broken".

I'd hazard a guess that 99.9% of Internet users want things to
"just work" cohesively and consistently.

I agree completely, and it is this belief that seems to be missing at
ICANN. Unless of course a system can be set up whereby ICANN is the sole
financial recipient of the fruits of this consistency: then ICANN can be
most cooperative in "working for the public good".

<snip>

Maintaining a single, authoritative root seems, IMHO, to be a
Good Thing.

I agree totally. Pity we can't get ICANN to go along with this without
forking over our life's savings and bribes^H^H^H^H^H^H donations of our
first borns, huh?

Given multiple registries, namespace collisions
would get ugly -- and, even in the absence of collisions, let us
consider "reachability" issues.

I agree with all of this, but the issue is moot in my book: Since ICANN
felt the need to either own the world or break it, I went elsewhere. So
did a LOT of others. Look at the mass exodus from NS/Verisign - same
issues. IMNSHO, ICANN cares not a rats a@@ about the internet, they are
only interested in the money and the power. That makes their
"positions" totally meaningless to me, and a lot of others who feel the
same way.

This isn't a question of "I like ICANN" or not... it's a question
of the system. Do people decide that they like or dislike our
system of government based solely on specific elected officials?

No, but at least our [deeply flawed and bleeding] system keeps the
*pretense* of following the laws of the land. As for your inference that
the ICANN folken are all "elected" officials...

Cheap domains from the EverQuick root! Guaranteed to work at a
handful of local dialups and dedicated-line customers! :slight_smile:

Oh, didn't you hear? EverQuick *also* includes the records of it's
largest competitor, so you can go to ALL of the places you wanted, not
just the ones that live on EverQuik.

I think that I'll start handing out ASNs and IP space, too...

It's called subdelegation. Go for it. Totally different system, which
for the most part works. (Yes, there are flaws there too, but...)