F.ROOT-SERVERS.NET moved to Beijing?

I happened to notice the following at three separate sites around
the US and one site in Europe:

$ dig +short +norec @F.ROOT-SERVERS.NET HOSTNAME.BIND CHAOS TXT
"pek2a.f.root-servers.org"

and:

$ dig +short +norec @F.ROOT-SERVERS.NET HOSTNAME.BIND CHAOS TXT
"pek2b.f.root-servers.org"

After running a couple of traceroutes it appears that he.net has a
route for F's anycast IPv6 address (2001:500:2f::f) towards Beijing.
According to https://www.isc.org/community/f-root/sites the Beijing
node should be a "Local Node" (without IPv6 but I suppose the list
is not up to date).

I believe this means that a lot of DNS queries from IPv6 enabled
sites in US and other countries are going to Beijing. I wonder if
this is intentional? Chinese government (CNNIC) seems to be in the
path.

All my sites seem to have he.net somewhere in the IPv6 connectivity
path. I wonder if this is specific to he.net or more wide-spread
routing anomaly?

I have notified he.net NOC and F-root @ ISC.

Best Regards,

I see similar, intermittedly

# dig +short +norec @F.ROOT-SERVERS.NET HOSTNAME.BIND CHAOS TXT
"pek2a.f.root-servers.org"

# dig +short +norec @F.ROOT-SERVERS.NET HOSTNAME.BIND CHAOS TXT
"ord1b.f.root-servers.org"

Getting palo alto from east coast.

3 10gigabitethernet1-2.core1.atl1.he.net (2001:470:0:1b5::2) 8.166 ms
8.135 ms 8.103 ms
4 2001:470:0:ce::2 (2001:470:0:ce::2) 77.881 ms 77.866 ms 77.909 ms
5 iana.r1.atl1.isc.org (2001:500:61:6::1) 77.885 ms 77.924 ms 77.896 ms
6 int-0-5-0-1.r1.pao1.isc.org (2001:4f8:0:1::49:1) 76.846 ms 75.854 ms
75.819 ms
7 f.root-servers.net (2001:500:2f::f) 75.788 ms 75.756 ms 75.726 ms

In a message written on Sun, Oct 02, 2011 at 05:40:23PM +0000, Janne Snabb wrote:

I happened to notice the following at three separate sites around
the US and one site in Europe:

ISC has verified our PEK2 route was being leaked further than
intended, and for the moment we have pulled the route until we can
get confirmation from our partners that the problem has been resolved.
Service should be back to normal, but if anyone is still having
problems noc@isc.org will open a ticket.

leo, all,

in the past, name servers that operated inside of china were subject
to arbitrary rewriting or blocking of their results by the Great
Firewall.

this is obviously bad for Chinese citizens but it's *dramatically*
worse for people outside of china who end up reaching a root server in
china by mistake, no? people who ostensibly live free of this kind of
interference and censorship are now subject to it by mistake.

a previous time this happened renesys did a good write up on it.

http://www.renesys.com/blog/2010/06/two-strikes-i-root.shtml

i guess my questions now are:

1) how long was this happening?
2) can any root server operator who serves data inside of china verify
that the data that they serve have not been rewritten by the great
firewall?
3) does ISC (or <Insert Root Operator Here>) have a plan for
monitoring route distribution to ensure that this doesn't happen again
(without prompt detection and mitigation)?

i'm not really singling out ISC here--this is a serious problem for
anyone who chooses to operate a root server node on untrustworthy or
malicious network infrastructure (which is one appropriate way of
thinking of a rewriting firewall from the perspective of a root server
operator).

cheers,

t

2) can any root server operator who serves data inside of china verify
that the data that they serve have not been rewritten by the great
firewall?

DNSSEC should help this issue dramatically. This however could be problematic
if the Chinese govt (or any repressive regime) decides to ban the use of
technology that allows a user to identify when they're being repressed.

3) does ISC (or <Insert Root Operator Here>) have a plan for
monitoring route distribution to ensure that this doesn't happen again
(without prompt detection and mitigation)?

Leaked routes happen External monitors and looking glasses and filters and
communities are all things we should probably be doing more of, in order to
minimize routing bogosity. But when all is said and done, there's no real way
to have a dynamic routing protocol like BGP and at the same time *guarantee*
that some chucklehead NOC monkey won't bollix things up. At best, we'll be
able to get to "less than N brown-paper-bag moments per Tier-[12] per annum" for
some value of N.

So Leo - you don't have to give us a full reveal of the root cause, but did
the phrase "chuckleheaded NOC monkey" enter at all into the saga? :wink:

valdis, all,

2) can any root server operator who serves data inside of china verify
that the data that they serve have not been rewritten by the great
firewall?

DNSSEC should help this issue dramatically. This however could be problematic
if the Chinese govt (or any repressive regime) decides to ban the use of
technology that allows a user to identify when they're being repressed.

sure, but DNSSEC is still basically unused.

3) does ISC (or <Insert Root Operator Here>) have a plan for
monitoring route distribution to ensure that this doesn't happen again
(without prompt detection and mitigation)?

Leaked routes happen External monitors and looking glasses and filters and
communities are all things we should probably be doing more of, in order to
minimize routing bogosity. But when all is said and done, there's no real way
to have a dynamic routing protocol like BGP and at the same time *guarantee*
that some chucklehead NOC monkey won't bollix things up. At best, we'll be
able to get to "less than N brown-paper-bag moments per Tier-[12] per annum" for
some value of N.

yep. this is a *great* argument *against* running critical
information services on known-malicious network infrastructure, right?

i.e.: if you are sure you're going to be interfered with regularly
and you're positive you can't restrict the damage of that interference
narrowly to the people who were already suffering such interference,
perhaps you should choose to not locate your critical network
information resource on that network.

yes, i'm (again) suggesting that people take seriously not doing root
name service inside of china as long as the great firewall exists.

t

In a message written on Sun, Oct 02, 2011 at 05:30:37PM -0400, Todd Underwood wrote:

i guess my questions now are:

1) how long was this happening?
2) can any root server operator who serves data inside of china verify
that the data that they serve have not been rewritten by the great
firewall?
3) does ISC (or <Insert Root Operator Here>) have a plan for
monitoring route distribution to ensure that this doesn't happen again
(without prompt detection and mitigation)?

I can't answer #1 with precision yet, but will attempt to get a
precise answer soon.

I'd like to partially address #2 and #3. ISC can verify that the
responses sent from F-Root boxes are always the same, regardless
of which server returns the answer. That is, there is no filtering
or rewriting on any ISC root servers.

We do know there are a number of locations around the world that
have various rewriting and blocking systems employed. We have found
networks where a query sent to F-Root never reaches an ISC run
server. As a root operator we hate this, and believe the best way
to solve the problem is DNSSEC. Short of providing a method like
DNSSEC to verify the answer is legitimate, we know of no other
countermeasure. There are in fact networks in the world that
impersonate all 13 root servers, and we don't know of a lever to
make them stop (short of local empowerment).

In the case of transparent re-writers or blockers between us and
the end users there is no practical way for us to detect that the
modifications are happening, and thus I don't think anyone could
answer your second question with precision. DNSSEC will at least
let every user do the verification from their own vantage point,
which is part of why it is so important.

Regarding #3, ISC does monitor for leaked routes. Unfortunately
these monitors are only as good as the vantage points they occupy,
and so with upwards of 40,000 ASN's I don't know of any way to cover
them all with any certianty. In this case it was even harder, as
the leak (appears to have been) IPv6 only. There are a lot fewer
IPv6 monitors, and folks are generally sloppy with their IPv6 configs
so there is more leaking. The situation is improving rapidly.

i'm not really singling out ISC here--this is a serious problem for
anyone who chooses to operate a root server node on untrustworthy or
malicious network infrastructure (which is one appropriate way of
thinking of a rewriting firewall from the perspective of a root server
operator).

I think the problem goes a lot further than root operators. The
fact of the matter is that there are networks that tamper with your
packets. From the benign NAT, to the full on transparent content
filter/blocker. Most places that tamper with root queries also
tamper with lots of other things. Without sort of reliable end to
end crypto you really have no way of knowing.

The root zone is signed. You can enable DNSSEC validation in your
caching resolvers. There are plugins for popular browsers that
attempt to do DNSSEC validation and show the results to the end
user in some pleasing way. Much more work needs to be done in this
area, but the technology is usable today. If you care about authentic
responses, use it.

Lastly, for some reason a ton of people always jump to the conclusion
that these sort of events are the plot of $insert_bad_guy. I've
chased down many leaks of F-Root during my time, and 100% of them
to date have been an accident. The clueless NOC monkey. The poorly
written route map. Someone not reading the documentation. Even
if $insert_bad_guy wanted to hijack F-Root (or any other root),
doing it in this way is very visable and easy to work around. It
just doesn't make sense to even try.

We won't be permitted to see the repression inherent in the system?

Cheers,
-- jr 'Run Away!' a

help, help! I'm being repressed!

phil

You actually think China will be the first to ban DNSSEC? Maybe,
but It will probably be banned first indirectly, by governments
legislating requirements of SPs
that are incompatible with DNSSEC.

The repression is at home in the form of the US PROTECT IP bill that
will provide
a framework for DNS authorities, domain registries, and ISPs/operators of
non-authoritative nameservers to be sent letters requiring them to modify DNS
responses for other organization's domains based on allegations/suspicions.

china nukes 120,000 domains for going against the policy of the state.

oops! that wasn't china, was it?

perhaps, we should postpone telling others what to do until our side of
the street is clean?

randy

120K domains - basically cnnic seems to have finally got tired of russian
botmaster types registering thousands of domains at a time, and put in a
rule that says you need business registration in China / ID in china to
register a .cn

Beyond that, that's one ccTLD - however large. There are multiple gTLDs
that have already done a great job of cleanup (biz, info for example) and so
far I haven't heard of .us having an infestation of botmasters / spammers.

And of course there are all the registrars out there that need to be reached
out to / handled etc etc - but that's another kettle of fish.

We're discussing two different things here - apples and oranges, though it
does look like they're all part of the same fruit salad.

1. Action by different registrars / registries [in .cn's case, a government
controlled registry, to be sure]

2. State policy to route internet access and DNS through an inspecting +
rewriting firewall that blocks or replaces politically unacceptable content

--srs

No, I think Randy was referring to this sort of thing:

http://www.theregister.co.uk/2011/02/18/fed_domain_seizure_slammed/

Sure - but what was being discussed in this thread was transparent / on the
fly rewrites of root server responses getting exposed to people beyond
china.

Whether these responses should be altered / censored within china or not is
a different can of worms, and that too has nothing at all to do with either
registry policy, or law enforcement mandated domain seizure.

a message of 32 lines which said:

I happened to notice the following at three separate sites around
the US and one site in Europe:

Good analysis at <http://bgpmon.net/blog/?p=540>

a message of 107 lines which said:

We have found networks where a query sent to F-Root never reaches an
ISC run server.

For details on such behavior, i highly recommend the excellent paper
"Identifying and Characterizing Anycast in the Domain Name System"
<http://www.isi.edu/~xunfan/research/anycast_Tech_Report_ISI_TR_671.pdf>,
which shows, among other things, that such masquerading (by a false
root name server) happens.

http://www.theregister.co.uk/2011/02/18/fed_domain_seizure_slammed/
"Our government has gone rogue on us," Eric Goldman, a professor at Santa Clara University School of Law, said. "Our government is going into court with half-baked facts and half-baked legal theories and shutting down operations. This is exactly what we thought the government couldn't do. I'm scratching my head why we aren't' grabbing the pitchforks." �

I.C.E., our very own Gestapo-Without-Borders. Makes me proud.<sigh>

a message of 32 lines which said:

$ dig +short +norec @F.ROOT-SERVERS.NET HOSTNAME.BIND CHAOS TXT
"pek2a.f.root-servers.org"

The next time, I suggest to also run "data" queries such as "A
www.facebook.com" or "A www.twitter.com" to see if there is hard
evidence of an actual security problem. (Most articles on this case
mentioned that "we have no proof there was a rewriting of answers from
the F-root instance".)