NSI bulletin 097-004 | Root Server Problems

Randy Bush writes:

Gosh, Perry. I wish I could be and make things that perfect.

Randy, we are engineers, not witch doctors.

Real engineers can and do build things to far higher levels of safety
than you seem to feel is acceptable.

I work for Wall Street firms. I've been around to witness things like
a head trader at a client order the purchase of ~$20 Billion in
securities on heavy leverage and have a couple dozen people work at
executing the trade for half the day in such a way as to avoid moving
the market.

In the sort of firms I work at, literally hundreds of millions to
billions of dollars are on the line in our systems. When failures
happen in this environment, people can lose money. If people lose
money, YOU NEVER WORK AGAIN. No excuses. No "oh, what were you
expecting, perfection?" You just don't work again.

As a result of that, WE BUILD THINGS NOT TO FAIL. We do things like
putting every machine is two nets, dual connecting all networks to
every other network. We "red/black" checkerboard the workstations
on our trading floors so that even if something really bad happens
your neighbor's machine will still be up and you can get out of your
positions. We engineer things so no single power cord, router, server
or communications link is a potential source of failure.

Its not trivial. You actually have to work to do this sort of
thing. However, it can be done, and it is done.

Moreover, we tend to build things such that dangerous infrastructure
is treated with care. We automate our management systems. We don't
have humans touch anything that could cause a network or system to go
bye-bye because we can't afford to.

Do we get failures? Sure, we get failures. Do we get catastrophic
failures? Very, very, very rarely. No, not never -- but very, very
rarely and when they happen we almost always recover very, very
fast. I've heard tales of things not recovering fast. I've never heard
tales of the people working on them after that, though.

As I said, we are engineers here, NOT WITCH DOCTORS. Saying
sarcastically "I wish I could be perfect, too" is to belittle your

When civil engineers build bridges, they almost never collapse. If
they collapse, the engineer probably isn't going to work again,
because he's just killed someone. Do collapses happen? Sure, once
every great while. Do they happen often? No.

When telephone network engineers build telephone networks, they rate
their switches at two minutes downtime per year, and they work very,
very carefully. Do failures happen? Sure, back in 1989 AT&T had a
catastrophic failure for a couple of hours. Do they happen regularly?
No, these things are goddamned rare.

When aerospace guys build fly-by-wire computer systems, people die if
they fail, so they build them so that they don't fail. Do they
sometimes fail? Sure, very rarely -- no one is perfect -- but compared
to the way the DNS has been failing of late it isn't even a contest.

Internet guys are no more stupid than areospace guys or civil
engineers, and frankly our job is no harder. We just have lots of lazy
people who spit out lots of excuses.

I'm used to an environment where excuses get you fired, though, so I
find myself less tolerant of "oh, poor us" claims from others.

There is no excuse for any system that will let you install a broken
zone file without requiring some serious levels of manual override --
there is, in fact, little excuse for building such a system so that
humans have to be involved in the first place.

I'll say this for a third time: we are engineers, not witch
doctors. These things *can* be built correctly.


Gosh, Perry. I wish I could be as important when I grow up.


You know, in light of the fact that I got bitched at a couple weeks ago
for posting a message that was a) solicited, and b) informative
(although questionably on topic; an assertion no one has _yet_
substantiated), I have to say that I think this stream of messages
from Mr Bush are entirely uncalled for.

The fact is, Randy, Perry is _right_. No amount of ad hominem attack
on his well written and thought out opinions is going to change that.

It's precisely this sort of thing that causes people like the NTIA to
decide we can't handle things "on our own".

The job is ours until we lose it, folks. Let's grow up and play nice.

-- jra

randy@psg.com (Randy Bush) writes:

Gosh, Perry. I wish I could be as important when I grow up.

when you do, please let us know how to configure our router for

"Kettle, Pot... Pot, Kettle"