Internic does it again

Mailman List Mirror (Read Only)
1

Eric Germann writes:

Well our friends at Internic apparently forgot how to update their servers

*.root-servers.net from here is showing a serial of 1997072500 with no
updates made Friday night showing (11:30am EDT July 27).

  While is is easy to blame the InterNIC and NSI for all the evils
  in the world, the InterNIC of five years ago (or even one year ago)
  is not the InterNIC that exists today. Last week I went to Network
  Solutions to meet with Dave Holtzman (SVP of Engineering) on an
  unrelated matter, and while I was there I got a chance to observe
  first hand the metamorphosis that has transpired over the past year.
  Here are just a few points that everyone should be aware of:

  * NSI completely replaced senior level management about nine
     months ago. Most of those folks were responsible for the arrogant,
     unresponsive attitude that NSI displayed for so many years. NSI
     replaced those folks with the most qualified people they could
     find. For example, Gabe Battista, NSI's CEO, was President of
     Cables and Wireless before coming to NSI last year. Dave Holtzman
     was one of IBM's most-talented senior engineers, and also has a
     substantial business background. There are dozens of other
     examples throughout the organization.

  * Since the transistion in the management structure, the call
     support area has gone from three people to 50 people, and the
     engineering support desk has gone from two to over 30 (and they
     are trying to hire more people right now). Hours have been
     extended from 5 PM EDT to 9 PM EDT, and plans are underway to
     make InterNIC support a 24x7 operation very soon.

  * The operations area is much like you would expect any large
     industrial organization to have -- a controlled environment
     computer room with raised floors, fire and water detection alarms,
     racks of equipment and systems, hot spare systems, several large
     Cisco routers including a spare Cisco 7500, three T3 links to
     separate backbone carriers, a UPS system with 200 lead batteries
     capable of running the entire facility for more than 24 hours,
     and a large deisel generator with enough fuel capacity to run
     for two weeks. The entire facility is secured with access cards,
     limiting people to those areas that they work in.

     [BTW, the entire InterNIC moved from its old location to its new
     location one month ago without any outage -- this is a true test
     of the InterNICs resources and expertise to be able to move an
     operation that large without impacting the rest of us.]

  * Effort is underway to define and document formal procedures for
     every activity that goes on in the InterNIC. Systems are being
     implemented everywhere to make the InterNIC as fault tolerant as
     possible. Based on what I saw, problems like the partial DNS
     update two weeks ago will be a thing of the past very shortly.
     I had the opportunity to talk with Richard Walsh, the Director
     of Operations, about the DNS problem and I am convinced it will
     never happen again -- there are too many automated systems and
     check procedures in place to allow a repeat of the previous
     partial update problem.

  * There is a complete focus on customer service in the InterNIC
     now. With a 1600+ percent increase in customer supprt personnel
     in just the past few months (which continues to grow as they are
     able to find and train people), the folks at NSI now talk about
     quality of service, response time, escalation procedures, and other
     customer support concepts that were anathma to the organization
     just one year ago.

     [Case in point -- before Dave Holtzman started posting on NANOG
     a few weeks ago, I can't remember anytime when an SVP at the
     InterNIC cared enough to post anything to the NANOG list.]

  Before I started US Net four years ago, I spent 16 years managing
  software engineering projects for KPMG Peat Marwick and American
  Management Systems (before that I taught CS at UofMD) -- I know
  what it takes to build, operate, and manage an industrial-strength
  operations center, and Network Solutions has one.

  I know what the response to this is -- "These folks are the InterNIC,
  they SHOULD be doing all these things!". And I agree 100%. The
  difference is they ARE doing these things now. This is a direct
  result of the new management at the InterNIC.

  Also, a lot of folks have made derogatory comments about NSI going
  public. Frankly, I think this is a good idea -- it will give them
  the financial depth to be able to support the InterNIC at a level
  that will benefit all of us. Based on what I saw, and knowing the
  skills of Battista and Holtzman, NSI is sure to be one of the best
  stock deals of the '90s.

  [FWIW, I have no connection to the InterNIC what so ever -- like
  the rest of you, I rely on them for their domain registration services
  only.]

  So if your idea of the InterNIC is two engineers, two computers, and
  a half-dozen support personnel with an attitude, forget it -- it is
  not that way anymore. Dave Holtzman wants to make the InterNIC a
  world-class support organization that puts reliability and customer
  support first. From what I have seen, he is well on his way ...

  Dave Stoddard
  US Net Incorporated
  301-572-5926
  dgs@us.net

2

And David proceeds to tell us why the InterNIC is so great.

Ok, David. Fine They are equipped technically. As you note, of
course, they _have_ to be, they're the InterNIC.

But answer this: why is it that _you_ have to be the one to tell us this?

The major complaint I hear about the InterNIC isn't that they're
arrogant, it's that their a black hole. Or worse, a roach motel:
complaints check in, but they don't check out.

That's exactly the problem I have with them. They need a PR person,
with an extensive background in the culture of the net, and the power
to get things done... and they need one _now_.

Cheers,
-- jr 'silent period or not' a

3

Exactly. If NSI doesn't have someone who head up pr doing the work of
informing the net community of what problems it's having, then rumors
start happening. Rumors spread out, and get picked up by the media.
Since the media is obviously a part of this list (we sure get quoted a
lot), these stories of NSI outages start making the news. The price of
NSI stock drops, and NSI's stockholders are not happy. Life is not good
for NSI. So, doesn't it make sense to just get someone to do the PR job?

Joe Shaw - jshaw@insync.net
NetAdmin - Insync Internet Services

4

Well, it's nice that he posted something, but why did he have to wait until
16 hours after the problem occurred [1]? Someone else noticed the problem
and sent a message to NANOG in UNDER 3 MINUTES [2]. Holtzman also made no
mention in this notice about how a sysadmin overrode the "quality
assurance" mechanisms or that they have made any effort to prevent future
occurances of the same problem(s).

Too little, too late.

Stephen

"From: David Holtzman <dholtz@internic.net>
"Subject: NSI bulletin 097-004 | Root Server Problems
"To: nanog@merit.edu
"Date: Thu, 17 Jul 1997 22:52:18 +0500 (GMT)
"
"On Wednesday night, July 16, during the computer-generation of the
"Internet top-level domain zone files, an Ingres database failure resulted
"in corrupt .COM and .NET zone files. Despite alarms raised by Network
"Solutions' quality assurance schemes, at approximately 2:30 a.m. (Eastern
"Time), a system administrator released the zone file without regenerating the
"file and verifying its integrity. Network Solutions corrected the
"problem and reissued the zone file by 6:30 a.m. (Eastern Time).
"
"Thank you.
"David H. Holtzman
"Sr VP Engineering, Network Solutions
"dholtz@internic.net

"Date: Thu, 17 Jul 1997 00:33:58 -0700 (PDT)
"From: Taner Halicioglu <taner@isi.net>
"To: NANOG <nanog@merit.edu>
"Subject: root nameservers broken again?
"
"bleh... is it me, or are the root nameservers hosed again?

5

Because of the pending IPO, the InterNIC is in a 90 day quiet period. Even
if they have hired a PR person/group/agency, that entity couldn't really do
their job until after the IPO. I believe that this is an SEC regulation.

6

Well, it's nice that he posted something, but why did he have to wait until
16 hours after the problem occurred [1]? Someone else noticed the problem
and sent a message to NANOG in UNDER 3 MINUTES [2]. Holtzman also made no
mention in this notice about how a sysadmin overrode the "quality
assurance" mechanisms or that they have made any effort to prevent future
occurances of the same problem(s).

  To join this with another thread, I think it'd be a Good Thing[TM]
  if the InterNIC were to start up an outage list...it sure would
  be popular. My only request (should any 'NIC people be reading
  this and taking me seriously, not that I think I deserve to be
  taken seriously too often) is that it be free of advertisements
  (even for NSI) or new payment schemes or /anything/ that isn't an
  outage that affects the provider community.

Too little, too late.

  But, better than most of us have grown accustomed to expect. I
  really don't think anybody here is trying to say that the InterNIC
  is now perfect, only that they've improved.

7

Sorry, Joe. Ms. Baker at DEC tells me we're off topic.

Cheers,
-- jra

8

It appears that the members of the list feel it's a worthy topic for
discussion. Since you obviously don't, and I can just feel Paul
Ferguson and Randy Baker slavering at the bit, I'm going to try asking
this just _one_ more time, and if I don't get an answer within, say, 48
hours, I'm going to start plonking people who yell "off-topic":

9

I concur. This is way off-topic. A pointer to the list charter was posted
recently and this crap wasn't part of it.

10

                              **************************
        Would an administrator of this list please compose and post a message
        defining in precise terms what topic areas are on- and off-topic for
                               the NANOG mailing list?
                              **************************

Pay careful attention, folks: the AUP is useless. It has not clearly
forbidden _any_ of the topics I've seen people get screamed at about in
the past month, with the sole exception of my bit of (you'll have to
admit: provoked) ventilation a week or so ago, for which I hereby
apologize publically to the parites involved.

I'm personally sick of all the 'off topic' messages. So, I say your off
topic message is off topic, and this reply to your off topic message is
off topic.

InterNIC dying (if it was true) is the sort of thing which could get
posted to this list. It does effect all networks and network operators. If
InterNIC goes boom, well gee... it's nice to know.

Now, the 8001 replies about how InterNIC doesn't work on Saturday's and
Sunday's (damn them for wanting TWO days off), that might be a bit
excessive, but I don't need 3 people to bitch and whine about it being off
topic, and then another 10 replying saying they agree.

If you aren't part of the solution, your part of the problem. (I'm part of
the problem for writing this, but... i'm a rebel)

Jordan