HE.net problem

[Bandy_Rush1]

what foss dns monitoring tools do folk use to alert of
  - iminent delegation expiry
  - inconsistent service (lame, soa mismatches, ...)
  - dnssec signing and timer issues
  - etc.

GitHub - berthubert/simplomon: Very simple monitoring system with a single configuration file

thanks. may play

hak whacked me to add
http://dns.measurement-factory.com/tools/nagios-plugins/check_zone_rrsig_expiration.html
to my nagios deployment.

anyone have some known sick in various ways dns zones against which to
test?

randy

[Jeroen_Massar]

> On the other side of this, we all may be learning the value of not
> having all of you NS records in a single zone with a domain under a
> single registrar.

From some trainings I did on how to be sure your DNS was robust:

- don't have all your business critical domains under the same
   registrar (unless it's of the CSC/markmonitor class)

Please note that:

- Markmonitor is owned by Newfold Digital / Endurance International [1]
- Network Solutions is owned by Web.com <http://web.com/&gt; [2]
- Web.com <http://web.com/&gt; is... owned by Newfold Digital [3]

But yes, if you pay MM, you likely get to keep your domain.

Note that NetSol is also not the NetSol anymore from the 1990's that one got tshirts from when you got a domain.

[1] Markmonitor - Wikipedia
[2] Network Solutions - Wikipedia
[3] Endurance International Group - Wikipedia

And... we all still have ICANN as an ultimate power, and the TLD itself, next to the above registrar.

There is always going to be single point of failures in a hierarchical tree like that.

- don't have all your auth NS for your domain in bailiwick (within the
   domain being served)

If, as it is the example in the thread, he.net <http://he.net/&gt; is your primary domain, which is their case, then if somebody in the tree disables the delegation of he.net <http://he.net/&gt;, nothing is going to fix resolution to you. Having your DNS servers in another TLD will not make he.net <http://he.net/&gt; appear in the global DNS again...

And if you look at multiple large sites, the google.com <http://google.com/&gt; akamai.com of the world, their registrar is MM, DNS are in-bailiwick and in-AS.

Indeed, for domains that are not 'infrastructure', where one uses out of bailiwick DNS servers, that can help, but also cause issues.

But, if we have for instance:

example.com <http://example.com/&gt; NS ns1.example.net <http://ns1.example.net/&gt;
             NS ns2.example.org <http://ns2.example.org/&gt;
             NS ns3.example.com <http://ns3.example.com/&gt;

Then:
- if .com decided to kick you out, you are out
- if .net or .org decide to kick you out you lose 1/3rd of your queries and induce latency...
   (at which point one can still remove the glue, but that will be another hour before gone)

Thus one only increases the risk by having multiple TLDs. Choosing a trusted registrar (one you have good contact with; indeed MM qualifies) and a TLD that will not cause you issues, is thus more important.

- don't have all your auth NS in the same routing domain (anycast can
   be an exception to this if robust enough)

Yep, that is a decent rule. But if you control your routing domain (AS, different prefixes), less external issues that can happen

Many of these rules btw are tested with https://www.zonemaster.net/

Though, one will see that many of the big sites do not have that; but they also use MM, and have 24/7 teams to fix things.

- don't have the account registrar credential emails all within the
   domain, nor with personal emails like gmail. do have them all under
   control of your IT

Accounts in different domains can be indeed a rather good one to think about, definitely for communication when the main domain is broken, at least the other domains are then on file for contacting.

Hopefully those email addresses cannot be used for account recovery, as then one increases risk again.

- protect all account credentials with strong passwords, MFA
- have MX for your domain either with a very large provider or across
   multiple domain names

It's painfully easy to fall off the internet and be unreachable if
you're not thinking about all this for business critical domains. You
don't ever want to be hoping that some customer kept your NOC phone
number in their phone.

Indeed; The weakest point is where the chain breaks... one can have a simple chain or a very distributed chain.

Greets,
Jeroen

[_Bill_Woodcock]

Taking off on what Jeroen is saying here… A huge amount of PCH’s work is with TLD registries. Much of that is ccTLDs, national domains, but a fair bit is also with brand TLDs. I think a lot of people are dismissive of brand TLDs, thinking “oh, that’s just trademark protection.” And MarkMonitor and CSC were, admittedly, a part of the reason why people treat them dismissively. The majority of brand TLDs lie fallow, with little to no use.

That’s unfortunate, because a TLD of its own is one of the VERY BEST things an organization can do to reduce security externalities. It’s a really foundational building-block in modern security. You can do DNSSEC and DANE and use all of the security tools and processes that build upon those, without having to depend upon the (largely non-existent) security of the registrar-registry chain. There are more protocols and tools coming down the pike that build further on that foundation. There are browsers coming which will trust the existence or non-existence of a DANE cert, without allowing a downgrade attack to a bogus CA cert. There are Digital Emblems coming (participate in the BoF at the IETF if you care!). That leaves you with just the one (?) externality of the IANA (and the RZM agreement) which, yeah, you’re not going to get past. But that’s done very, very securely, so if you have to trust one external party, at least they’re _competent_ and well-funded and not going to get acquired by a Florida Man private-equity outfit.

ICANN’s going to open another round of TLD applications, and I expect a lot of companies to go into that with their eyes more open than last time, knowing why they’re doing it. It’s not about brand protection, it’s about disintermediating the root of trust and giving yourself a solid foundation for your security architecture.

                                -Bill

[Paul_Ebersman]

- don't have all your business critical domains under the same
  registrar (unless it's of the CSC/markmonitor class)

There is always going to be single point of failures in a
hierarchical tree like that.

Everything in internet/infrastructure is risk tradeoffs and cost/benefit
analysis. If we could be perfect as engineers, we would be.

Personally, the fact that the internet mostly functions most mornings
when I get up is still something that amazes me after years of using
it...

- don't have all your auth NS for your domain in bailiwick
  (within the domain being served)

If, as it is the example in the thread, he.net <http://he.net/&gt;
is your primary domain, which is their case, then if somebody in
the tree disables the delegation of he.net <http://he.net/&gt;,
nothing is going to fix resolution to you. Having your DNS
servers in another TLD will not make he.net <http://he.net/&gt;
appear in the global DNS again...

The above two points of mine tie together. If you can afford a registrar
who will be far more likely to care about you than random/bad
enforcement of external complaints and you're big/rich enough to be able
to use highly robust anycasted auth NS, in bailiwick is a much lower
risk.

If my "joe's fish shop and internet cafe" DNS is provided by "my mom let
me be a registrar if I ate my vegetables" diversity of TLD, registrar,
and auth NS (including out of bailiwick NS) becomes a much more
attractive and cheaper way to be more robust.

Thus one only increases the risk by having multiple
TLDs. Choosing a trusted registrar (one you have good contact
with; indeed MM qualifies) and a TLD that will not cause you
issues, is thus more important.

Again, this depends on scale. For SMB, multiple TLDs is more likely to
be a feature, for a really large business not so much. As Bill points
out, this is actually one of the few cases where brand TLD is a major
potential security upgrade.

[Robert_L_Mathews]

That’s not the case if you provide DNS servers for others, though.

It is true that if he.net has nameservers of “ns1.he.net” and “ns2.he.net”, making the second of those be “ns2.he.org” will not make “www.he.net” reachable if he.net is placed on clientHold.

However, if “example.com” uses “ns1.he.net” and “ns2.he.net” as its nameservers, having the second of those instead be “ns2.he.org” will keep “www.example.com” reachable if he.net is placed on clientHold.

That was presumably the emergency concern in this case – not so much that www.he.net itself was offline, but that all the other domains using their nameservers were offline.

I run a registrar so there’s no risk of our domain names getting put on clientHold, but I still don’t trust the registry not to put one of our domain names on their equivalent “serverHold”. We provide nameservers for our customers in .net, .biz and .org (run by separate companies) to mitigate that risk. And every time I see a story like what happened to he.net yesterday, I re-convince myself that the slight performance hit is worth it, and presumably, so do companies like Amazon:

$ dig +short amazon.com NS
ns1.amzndns.co.uk.
ns1.amzndns.com.
ns1.amzndns.net.
ns1.amzndns.org.
ns2.amzndns.co.uk.
ns2.amzndns.com.
ns2.amzndns.net.
ns2.amzndns.org.

[John_L]

It appears that Bill Woodcock <woody@pch.net> said:

ICANN’s going to open another round of TLD applications, and I expect a lot of companies to go into that with their eyes more
open than last time, knowing why they’re doing it. It’s not about brand protection, it’s about disintermediating the root
of trust and giving yourself a solid foundation for your security architecture.

I take your point, but I also observe ICANN's list of abandoned vanity
TLDs which now has 134 entries, companies that paid the fee, went all
the way through the application process, paid to get their TLD set up
and signed and put in the root, probably costing them on the order of
$500K in all. Then later they decided the TLDs are worthless and mailed
the keys back to ICANN. Here's the 134 who have abandoned so far, with
more every few months:

https://www.icann.org/resources/pages/gtld-registry-agreement-termination-2015-10-09-en

Also, getting your own TLD doesn't necessarily make your risks less, it just
makes them different. You now have a direct relationship with the registry
back end provider that you have to not screw up, and due to ICANN's rules,
there is a registrar between you and your registry for each of your names,
e.g. payme.hsbc is registered through Comlaude. I do not see why this would
be any better than a .COM registered through CSC.*

I will be interested to see how the next round goes. As seen from
anyone outside the ICANN bubble who can do simple arithmetic, the
current round has been an utter failure for everyone other than ICANN
and the people who sucked money out of the process (which includes me,
but not so much I particularly want to do it again.)

R's,
John

* - I have argued with CSC about updates to the IANA domains and found
it is nearly impossible to get them to change anything even when you
follow the process they set up. I would expect it to be even harder to
get them to make changes outside of the process. Their whole business
model is not to annoy their large company clients most of which also use
them for Delaware corporate paperwork.

[Rubens_Kuhl]

For up to 100 domains it's possible to use the back-end registry
directly instead of using a registrar.
So for *some* TLDs, those using less than a 100 registrations, they
don't need a registrar.

For those with more than a 100 domains, some back-end registries
implemented a two-person rule requiring both registrar and brand
client to agree on changes to the SRS and to the zone. So those have
better control compared to using the same registrar with other TLDs.

Rubens

[Jay_Ashworth]

From: "Robert L Mathews" <lists@tigertech.com>

However, if "example.com" uses "ns1.he.net" and "ns2.he.net" as its nameservers,
having the second of those instead be "ns2.he.org" will keep "www.example.com"
reachable if he.net is placed on clientHold.

That was presumably the emergency concern in this case -- not so much that
www.he.net itself was offline, but that all the other domains using their
nameservers were offline.

Correct. I was not the person who made the original query/report, but that was
the concern which made me run the event up the flagpole here and on Outages.

I run a registrar so there's no risk of our domain names getting put on
clientHold, but I still don't trust the *registry* not to put one of our domain
names on their equivalent "serverHold".

And it is there that perhaps I overreacted one step; I had thought from the
data I heard that that *was* a registry-side hold (and hence it didn't matter
that it was NetSol). Or perhaps that NetSol was still the registry for .net --
that's out of date now, isn't it?

Cheers,
-- jra

[John_L]

According to Jay R. Ashworth <jra@baylink.com>:

data I heard that that *was* a registry-side hold (and hence it didn't matter
that it was NetSol). Or perhaps that NetSol was still the registry for .net --
that's out of date now, isn't it?

Uh, yeah, Verisign spun off the NetSol registrar over 20 years ago in late 2003.

In early 2003 Verisign turned .ORG over to PIR, but they kept .NET and
.COM which they stil have. They are also the registry for a bunch of
small ccTLDs and new gTLDs. They paid $135 million in the auction for
.WEB which they may eventually run once the legal challenges are
settled.

NetSol was bought and sold and merged several times and since 2011 has been
part of web.com.

[Jay_Ashworth]

See how little it has been necessary for me to pay attention to them since my net handle was assigned back in the early 90s or maybe late 80s?

Cheers,
– jra3

[John_Von_Essen]

I've found this conversation hugely of interest… The below isn't really a question, more of a high level clarification/further thinking.

First, what actually happened and the impact (correct me if any of this is wrong):

A stupid phishing complaint to NetSol by a 3rd party got he.net put into client hold. As a result, assuming there is no cache protection, name servers around the world trying to lookup anything.he.net were failing because ROOT servers said go to NetSol for .net, and netsol had no answer for he.net due to client hold. This means:

1. he.net <http://he.net/&gt; website would have been down regardless if their Auth NS was split Auth and split across TLDs
2. customer.com <http://customer.com/&gt; website would be down is customer.com used NS1.HE.NET <http://ns1.he.net/&gt; and NS2.HE.NET <http://ns2.he.net/&gt; as Auth DNS because that resolution would fail due to he.net <http://he.net/&gt; being clienthold at NetSol
2.a. customer.com <http://customer.com/&gt; website would be UP if customer.com <http://customer.com/&gt; used NS1.HE.NET <http://ns1.he.net/&gt; and NS2.HE.ORG <http://ns1.he.org/&gt; as Auth DNS… assuming HE implemented secondary NS servers on another TLD, or secondary was Cloudflare or something.

Obviously the root cause was a glitch, but if it wasn’t a phishing report it could have been any other number of human errors, billing issue, internal NetSol glitch/fat fingering, etc.,. something requiring human intervention - which is hard to do these days because nobody has a 24x7 NOC with real people who can make real changes.

#2 could have been protected by #2A, but as others have said #1 isn’t really possible to 100% protect against. Yes, HE could get and use their own vanity TLD at huge expense (say .he TLD) and since they control it a glitch like this cant burn them, but you just trade risk because now you have to maintain the infra of this TLD. So #1’s the easiest fix - just use MarkMonitor.

Ok…. now a rabbit hole. I looked at some vanity TLDs, and it appears the ALOT of big companies have their names as TLDs, but almost none of them are using it for anything. Why is that? Is it just a copyright play to protect the name from some else taking it?

Then it got me interested, assuming a company already has the infra, what is a realistic cost to get your own TLD and actually use it for yourself (and maybe others)? I saw something online that said $250,000 but that didn’t make sense if its all paperwork. Again, this assumes you already have infra to use.

-John

[_Bill_Woodcock]

Heh. I see you are unfamiliar with ICANN. They’ve said that same paperwork is likely to cost $375k in ICANN staff time for the next round. Because, you know, inflation or something.

                                -Bill

[Paul_Ebersman]

I saw something online that said $250,000 but that didn't make
sense if its all paperwork.

Heh. I see you are unfamiliar with ICANN. They've said that
same paperwork is likely to cost $375k in ICANN staff time for
the next round. Because, you know, inflation or something.

And $250k was not really the cost last time either. By the time you did
legal fees, deposits, fees, NOC, etc. it was closer to $500k per TLD.

If there is similar inflation on all the extra costs, that probably
means $600-700k next time.

I've been surprised that none of the folks that got TLDs seem to be
leveraging the technical/security brand protection like they could.

If I have an LG TV and it wants to update to <mumble>.LG and LG is
DNSSEC signing the whole chain, that sure seems more likely to be legit
than <mumble>.lg.tv or some such.

Add in that if you do brand, short of the root zone dumping you for some
reason, you own your experience and your full resolution chain. You can
pick who/how/where for all the labels, servers, anycast infra, etc.

But I haven't seen a lot of interest in taking advantage of that added
potential layer of security.

[_Bill_Woodcock]

I've been surprised that none of the folks that got TLDs seem to be
leveraging the technical/security brand protection like they could.

A few are. A very few. SNCF. A few banks.

If I have an LG TV and it wants to update to <mumble>.LG and LG is
DNSSEC signing the whole chain, that sure seems more likely to be legit
than <mumble>.lg.tv or some such.

Ayup. Particularly if they don’t allow downgrade attacks to CA certs.

I think there are a few more brands looking to make this move to higher security in the new ngTLD round. At least everybody’s a lot more educated this time around.

                                -Bill

[Crist_Clark]

If you’re LG, you own the software, you do cert pinning.

Also, realize many (most? almost all?) are going to outsource the management of their vanity TLD to one of the existing companies in that market.

Think of a brand that sells, I don’t know, shoes. Running a TLD is not part of their core business. It makes no sense to do this in-house. So now it’s a contractual agreement with some third party again anyway. You depend on their help desk, their security, and all of the other vendors that they outsource other bits to.

Heck, even hugest of huge IT companies out source this stuff. .apple backend is outsourced to Afalias.

[John_L]

It appears that Bill Woodcock <woody@pch.net> said:

-=-=-=-=-=-

I saw something online that said $250,000 but that didn’t make sense if its all paperwork.

Heh. I see you are unfamiliar with ICANN. They’ve said that same paperwork is likely to cost $375k in ICANN staff time for the next
round. Because, you know, inflation or something.

Well, yeah, there's only $49 million of the current round's $308M left
unspent. Better safe than sorry.

They've already spent $18M of the current round's money to pay for
costs for the next round, because what could be fairer than to use
your application fee to pay for random strangers to compete with you?

R's,
John

[John_L]

According to Bill Woodcock <woody@pch.net>:

-=-=-=-=-=-

I've been surprised that none of the folks that got TLDs seem to be
leveraging the technical/security brand protection like they could.

A few are. A very few. SNCF. A few banks.

I can't help but note that if I connect to https://oui.sncf, it
now immediately redirects to https://www.sncf-connect.com.
http://restaurationabord.sncf/ is 404.
https://www.abonnement-regional.sncf leads to a fairly
lame login page that quickly switches to sncf.com.

All the other ones I checked are dead.

Wonder if they're getting ready to be #138.

If I have an LG TV and it wants to update to <mumble>.LG and LG is
DNSSEC signing the whole chain, that sure seems more likely to be legit
than <mumble>.lg.tv or some such.

Ayup. Particularly if they don’t allow downgrade attacks to CA certs.

I think there are a few more brands looking to make this move to higher security in the new ngTLD round. At least everybody’s a lot
more educated this time around.

I dunno, if they were better educated they'd realize it's a total
waste of money.

R's,
John

[Robert_McKay]

People aren't used to URLs not ending in .com or possibly their local ccTLD. Anything else looks suspicious or isn't even recognised as a URL and less people will visit it. It doesn't even make your URL shorter because you'd need to include "www." in front to have any chance of it being recognised as a URL. Perhaps even "https://www." to drive home the point.

Maybe this will change over time, but maybe not.. there's probably nothing to be gained by fighting it. You want people to visit your URL? Don't use a weird URL.

Rob

[Rubens_Kuhl]

If I have an LG TV and it wants to update to <mumble>.LG and LG is
DNSSEC signing the whole chain, that sure seems more likely to be legit
than <mumble>.lg.tv or some such.

.lg and .he were mentioned as possible brand TLDs, but those are not
allowed, because they are reserved for possible ccTLDs. gTLDs are
required to have at least 3 ASCII characters or 2 Unicode characters,
with the 2026 round bringing the possibility of 1 Unicode character
gTLDs in Han Script (mostly known as CJK - Chinese, Japanese, Korean
for the most representative languages using that script).

Rubens

[Brandon_Butterworth]

People aren't used to URLs not ending in .com or possibly their local ccTLD. Anything else looks suspicious or isn't even recognised as a URL and less people will visit it.

True but when you are multinational you probably have a .com plus a
lot of country ones, even if they just redirect, to avoid local
fakery. It gets quite messy protecting the brond and the more domains
you have the easier it is to fool someone to accept another similar
one. One definitive name can be help.

It doesn't even make your URL shorter because you'd need to include "www." in front to have any chance of it being recognised as a URL. Perhaps even "https://www." to drive home the point.

When I did ours I had a test site the.bbc up for a while until google
started pushing traffic to it. It seemed a lot nicer to say on air
than www.bbc.co.uk / .com, and nicely authoritative.

I hope it will be used eventually.

brandon