I hate to tell you ...
In many cases, the folks in the NOC are primarily there to watch
red lights and green lights. Even if you had the secret NOC
phone number, it wouldn't do you much good. In a large provider
the NOC has soo many rules and procedures, the NOC-level techs
aren't even allowed to log onto core routers to look. And
forget about making any changes to a router configuration. Even
if the NOC folks wanted to help, they aren't allowed to.
Whether its a large provider or a small provider, they both seem
to have about the same number of people with the expertise and
authority to fix "interesting" problems. It might be called
4th level support, or backbone engineering, or senior engineers.
It has less to do with technical clue than management clue. I've
often found a person in a NOC who understood the problem, and even
understood what was required to fix it, but due to decisions by
his management he wasn't allowed to fix it.
There are management culture differences between the various
providers which affect how well their employees can fix things.
It can be annoying at times trying to get through to the proper
individual(s) who can actually fix the problem we might be having. It is
a classic case of the lesser of two evils though. I would rather have a
select few people who have authority to make changes to infrastructure and
communicate those changes to each other than have 100 people making
changes and none of them knowing what the others are doing.
Solution: Once you work your way up to the person who has authority to
make changes, NOTE their name, email address and possibly their telephone
number! You can now cut through 90% of the red tape in the future. This
has worked with amazing success for us.
You do have to take into account that while you don't have to fight your
way through voice-mail-hell or explain until you've lost your voice that
you need to escalate the issue to the magic "we can make changes" level,
there will still most likely be someone ahead of you in that persons
"to-do" pile and you can't expect them to drop whatever they're doing
for the poor schmoe who waited 4 hours to talk to them to do what you