NOC communications (was Re: Process management)

Mailman List Mirror (Read Only)
1

You bring up an interesting (if unspoken) point in your prior statement,
and that is: "Communications between entities is worthwhile if the
communications facility has a high signal-to-noise rate." While I am not
certain, I have strong suspicions that the SS7 NOCs pushed their emergency
directory numbers to the general number due to useless, misdirected calls.

Most of the SS7 providers involved reported no problems with misuse of
their contact information. A couple of providers said they were changing
the contact information out of fear the information might be misused. One
provider stated they had problems, but didn't give any specifics.

As some anecdotal evidence I've found our WHOIS e-mail contact information
tends to get spammed by the small time bulk mailers, usually whose own
contact information is bogus. While our phone information tends to get
junk calls by large corporations, usually one particular major ISP's
telemarketing operation trying to sell Internet service.

Handling such calls through the general purpose number can work well,
IF and I repeat IF, the staff at the general purpose number is trained
how to handle the calls and direct the information expeditiously.
Unfortunately, it seems the person changing the emergency contact
information forgets to inform the customer service operation they
will now be getting those calls. Some companies seem to do this
well, some seem to do this poorly. As I found out, you never know
when it will be your newest, greenest tech who happens to get the
call at the exact same time the senior engineer is in the bathroom.
Even though the calls are fairly rare, it is an ongoing training process.

Of course, if the general purpose contact number was working well,
there wouldn't be a need for another contact method.

Being a secretary is not something that a NOC can afford to do.

Being a secretary is also something most NOCs don't do well. It is a
different set of skills. Dealing with a good receptionist is a pleasure.
On the other hand, a bad receptionist is really bad. How many folks
remember the recurring receptionist character sketch on Saturday Night Live.
I doubt most NOC engineers would last 30 minutes on the front desk of
a large law firm.

Therefore, to avoid this problem, one must limit the ubiquity of the
contact mechanism and increase the value of each message. Using a medium
like a phone number is of course the standard method for contact in any
emergency situation (email is great, but it lacks a rapid
question-answer-experiment ability) but phones also carry with them the
ease of use that works against them, as well as for them. A phone number
gets handed out on web sites, "emergency call" sheets, etc. and soon
people who do not have anything directly relating to operations are calling
the operations hot line. Either more staff is required to start answering
these questions, or (more likely) the "hot line" becomes not-so-hot and it
will go unanswered or not taken seriously, or simply nobody will care about
it and it will get forwarded to the Void.

Yes, I'm aware of the 9-1-1 problem. If you make something that works
well, everyone is afraid it will be overwhelmed with all sorts of calls
not relevant to its primary purpose. On the other hand, if you make a
something that is so restricted, it will never be abused, but it will also
likely never get used or important pieces of information won't get through.

How do you set things up to create a 'shared fate' among all the participants?
Do you need a strong moderator, editor or dispatcher to make such a
system work? To avoid the 'whose in charge' problem do we draw lots
each week for the moderator position? If each of the participants don't
feel some pressure from their management, shareholders, customers to
make a good faith effort; will any and every system fail of active
neglect? There have been several attempts in the past, and have not
had a sucessfull track record.

I'm going to keep throwing ideas against the wall to see what sticks.
Are any of these ideas even close to being useful? Or do people think
there is not a problem and I should just shut up.

2

[snip]

As some anecdotal evidence I've found our WHOIS e-mail contact information
tends to get spammed by the small time bulk mailers, usually whose own
contact information is bogus. While our phone information tends to get
junk calls by large corporations, usually one particular major ISP's
telemarketing operation trying to sell Internet service.

Having a different number/email address on your AS than what's on your
domain name might solve some of these problems- it might not. Being much
more aggressive and outwardly unfriendly to people who call the number on
your AS records becomes realistic if you have distinctly declared it for
NOC communications only. A brief note to your counterpart at the
offending organization (in your major ISP example) might be enough to
clear things up, or perhaps an ARIN "page of shame" in some public place
would do the trick. (ha! Try running that one up the flagpole in front of
ARIN's lawyers.)

Handling such calls through the general purpose number can work well,
IF and I repeat IF, the staff at the general purpose number is trained
how to handle the calls and direct the information expeditiously.
Unfortunately, it seems the person changing the emergency contact
information forgets to inform the customer service operation they
will now be getting those calls. Some companies seem to do this
well, some seem to do this poorly. As I found out, you never know
when it will be your newest, greenest tech who happens to get the
call at the exact same time the senior engineer is in the bathroom.
Even though the calls are fairly rare, it is an ongoing training process.

I disagree with your point that NOC-to-NOC communications should ever be
forwarded to a general number, even with adequate training. General
purpose numbers are not staffed 24x7, and the good intent of a
well-informed general purpose number gets sidetracked swiftly during
call-center grooming and re-organization of departments and other
diversions that occur during the life of a company. How many times have
you called an ISP's "hotline" to talk with someone who has no idea what a
NOC is? Or how many times have you been on hold for more than 15 minutes
only to be transferred to another queue? The best intentions are no match
for entropy if the people in charge of the system do not have direct
control over the fate of where the call lands.

Of course, if the general purpose contact number was working well,
there wouldn't be a need for another contact method.

My point exactly - we all have proof that the current system doesn't work
very well.

[snip]

Therefore, to avoid this problem, one must limit the ubiquity of the
contact mechanism and increase the value of each message.

[snip]

Yes, I'm aware of the 9-1-1 problem. If you make something that works
well, everyone is afraid it will be overwhelmed with all sorts of calls
not relevant to its primary purpose. On the other hand, if you make a
something that is so restricted, it will never be abused, but it will also
likely never get used or important pieces of information won't get through.

In the "Fantasy" proposal (low-baud satellite bi-directional group
communication), this would be solved by forcing everyone to have a
terminal. A line-printer (or thermal, if you want it to be a little more
quiet :wink: based box is easy to understand, easy to use, and can be placed in
the lights-camera-action NOC for easy access. It's slow enough that it's
not going to be used for social interaction (like an IRC channel or MUSH
would) but even a 2400 baud stream would certainly be able to print out
enough information on each provider to give an adequate exchange rate. Of
course, this scenario is overkill in my mind, and extremely difficult to
implement in the anarchy (classic use of the term) that exists in the
Internet today, even among just we members under ARIN's umbrella.

How do you set things up to create a 'shared fate' among all the

participants?

Do you need a strong moderator, editor or dispatcher to make such a
system work? To avoid the 'whose in charge' problem do we draw lots
each week for the moderator position? If each of the participants don't
feel some pressure from their management, shareholders, customers to
make a good faith effort; will any and every system fail of active
neglect? There have been several attempts in the past, and have not
had a sucessfull track record.

We already have a central authority that collects two perfectly good points
of information about each AS in North America (supposedly.) Between the
email addresses and phone numbers, there should be ONE method that gets
through to someone. The problem is information rot - phone numbers that
ring to the boiler room and email addresses for people who have not worked
at said companies for 2 years. No additional system is needed - just fix
the one we have. Fixing may take more than just good will by the
participants - it will probably take fear of policy application. (it's not
a DOS - it's "policy application" if the majority agrees to the terms.)
   The only argument against competent repair of the information database
is "fear of spam". This can be handled by swift and public flogging of
offenders, or merely excessive anti-response at the particular abuser.
Perhaps making AS records only available to members would cut down on
"farming" of such databases by un-involved third parties. I think a
username/password combination would be a good idea, provided that it was a
reliable system.

I'm going to keep throwing ideas against the wall to see what sticks.
Are any of these ideas even close to being useful? Or do people think
there is not a problem and I should just shut up.
--
Sean Donelan, Data Research Associates, Inc, St. Louis, MO
Affiliation given for identification not representation

It's a problem that we've all hit at one point or another. It's not
crisis-level all the time, though, so it's not at the forefront of people's
minds. It's only when you're looking for the right phone number of the AS
that's advertising the more specific route for your biggest customer do you
realize that something has to be done to keep the data up-to-date.

JT

3

This gives me a chance to plug my noc list again, and get
everyone else to contribute to it.

  http://puck.nether.net/netops/nocs.html

  Questions or comments about the list? Direct them to me.

  As for most people being clueless when you talk to them, education
is the only path. There's not much else that can be done, there's
not anything that prohibits people from starting a clueless company. It's
all marketing, if you can market yourselves well, and service your
market in a not-unreasonable fashion, they're not likeley to listen to
someone else anyways.

  But we don't want to do something like require people to be
certified before becoming an ISP do we? that would be worse.

  Going on a quest and attempting to educate people as best possible
is currently the only path for dealing with cluelesness.

  As for bad contact information in internet,whois databases, etc..
arin, internic, etc.. should attempt to be proactive at going after
these folks.

  Taking a look at one (you can see it's false) entry in the
internic database is: