[NOOP] 24x7 NOC..

Mailman List Mirror (Read Only)
1

We're in the process of developing a 24x7 NOC, and can agree upon about
everything except scheduling.

  We have a few ideas, all of which have really bad downsides:

- 4day, 10hour rotation. 2-hour overlap.
  - Downside is overlap, and the fact that nobody will have a regular
    schedule.. they'll be M-Th, then T-Fr, W-Sa, Th-Su, Fr-Mo.. bleh.

- M-F 9-hour shifts, Sa-Su 2x12-hour shifts
  - Downside is the Sa-Su people are just "extras".

- Su-We, We-Sa 10-hour shifts.
  Alleviates "skewing" schedules one day every week, but creates a 14.5%
  labor increase on Wednesday when you have a complete overlap.

Anyhow, we're looking for information on how other NOCs around the world
do it. We're trying to give good coverage without killing/burning out the
NOC team members.

TIA.

Jamie Rishaw

2

- Su-We, We-Sa 10-hour shifts.
  Alleviates "skewing" schedules one day every week, but creates a 14.5%
  labor increase on Wednesday when you have a complete overlap.

  How many of your NOC people willing to work that schedule? i
worked 7 months of having tues/wed off. It sucked, becuase they allways
called me even when i was OFF!. Now that i have weekends off, they only
call me when its real important (Not my definition of Important ... :().

Scott

3

  We're in the process of developing a 24x7 NOC, and can agree upon about
everything except scheduling.

  We have a few ideas, all of which have really bad downsides:

- 4day, 10hour rotation. 2-hour overlap.
  - Downside is overlap, and the fact that nobody will have a regular
    schedule.. they'll be M-Th, then T-Fr, W-Sa, Th-Su, Fr-Mo.. bleh.

  I understand AGIS used to do that. AFAIK, the two (yes,
  two) people they had on that rotation quit long ago.

- M-F 9-hour shifts, Sa-Su 2x12-hour shifts
  - Downside is the Sa-Su people are just "extras".

  You're gonna want good people on weekends, 'cause it's harder
  to find a clueful telco person during non-business hours.

- Su-We, We-Sa 10-hour shifts.
  Alleviates "skewing" schedules one day every week, but creates a 14.5%
  labor increase on Wednesday when you have a complete overlap.

  You're probably gonna have overlap at some point; either that
  or have somebody working extra hours every week.

Anyhow, we're looking for information on how other NOCs around the world
do it. We're trying to give good coverage without killing/burning out the
NOC team members.

  Can't be done. *grin*

  At a previous job, we had eight-hour shifts, no formal overlap;
  for quite a while we'd have one person in the NOC at any given
  time. Each week, one of us would work six days. There wasn't
  any overtime. It's a good thing we all liked working there; in
  fact, once we stopped enjoying the job everybody who was in the
  NOC at that time quit.

  What I'm trying at Priori is four-hour shifts. People can work
  eight or sometimes even twelve hours in a row if they want, but
  it's all in four-hour blocks. Since we're expecting that at
  least some of our part-timers will be students, that gives us a
  lot of flexibility to deal with their schedules.

  We don't yet have all the shifts covered, so Justin, Michael,
  and myself are covering as needed, and I don't yet have a good
  feeling as to how well this is going to work.

  Luckily, since we're a startup, it won't be too hard to move
  to something else as needed. The most important thing, IMHO,
  is to make sure that the schedule you have set is something that
  your staff can live with -- at that previous job I mentioned we
  were all completely insane and actually didn't mind working a
  couple day shifts, having a day and a half off, and then coming
  in for night shifts, and rotating that around every couple of
  weeks -- except for one full-time student, we preferred that to
  being stuck with the same hours every day of every week.

4

We're in the process of developing a 24x7 NOC, and can agree upon about
everything except scheduling.

Ahhh, a subject near and dear to all NOC managers.

Tip 1) Talk to the people involved, give them a voice in the decision.
They will be much less prone to gripe later on if they helped define the
schedule.

Tip 2) Overlap is *good* in a 7x24 team. No way can a group function as a
team if they never have any face to face time. I was, and still am, a firm
proponent of this. Seeing NOC people hang around and talk is a good thing;
later on when pressure mounts and tempers flare, its much better they be
friends.

Tip 3) Fixed schedules are better than fluctuating. People need some
stability to arrange their lives around.

Tip 4) Double up all shifts otherwise, when your weekend guys wife goes
into labor early (on New Years Eve), you aren't scrambling for coverage. It
happens.

To do 7x24 comfortably, you'll need 12 people and a supervisor.

We work 4/10hr days, Sun/Mon/Tue/Wed or Wed/Thur/Fri/Sat. People get three
days off in a row, including one weekend day. We have our staff meetings/
training on Wednesday afternoons. We overlap the split weeks with
Senior/Supervisory staff who work M-F and tie the two ends of the week
together. The only ones who are left out are the overnight crew. A night
supervisor is responsible for communication/training with them.

This is based upon nine years of operations center experience.
(hello to those BBN NOC alumni...)

Steve Miller
WebTV Service Operations Center

5

Keeping the night shift informed and included in what's going
  on is very important. I spent a few months as night NOC
  supervisor at a large ISP on the East Coast and was never
  sure what was going on -- not because of any failing on the
  part of my supervisor (hi Jason), but more because there
  wasn't any mechanism in place for communicating most projects
  & such with the night staff besides word of mouth from whoever
  worked late that day.

  I actually know almost as much about what's going on at night
  there now, after moving all the way across the country, as I
  did when I showed up for work every night.

  That's one of the reasons that I and my cow orkers at that
  previous job I mentioned actually liked having rotating shifts;
  it meant we'd never be stuck always working at night and never
  knowing what's going on, or always working during the busiest
  times. As long as the schedule is sent out a couple weeks or
  even months in advance, people can plan their lives around it.

6

We're in the process of developing a 24x7 NOC, and can agree upon about
everything except scheduling.

Ahhh, a subject near and dear to all NOC managers.

Steve, I'm glad you piped up and gave the benefit of your experience at BBN
and what you've learned since we worked together back then.

To do 7x24 comfortably, you'll need 12 people and a supervisor.

We work 4/10hr days, Sun/Mon/Tue/Wed or Wed/Thur/Fri/Sat. People get three
days off in a row, including one weekend day. We have our staff meetings/
training on Wednesday afternoons. We overlap the split weeks with
Senior/Supervisory staff who work M-F and tie the two ends of the week
together. The only ones who are left out are the overnight crew. A night
supervisor is responsible for communication/training with them.

Do you run three 10 hour shifts per day or two? Seems like a lot of overlap
and makes me wonder whether the minimum 12 head count rule still holds.
Three shifts per day, two crews per week, two people per shift works out to
12 people, but that leaves you without a job or else down a backup
supervisor. :slight_smile: Do you run some shifts with one person on site and others
with three?

This is based upon nine years of operations center experience.
(hello to those BBN NOC alumni...)

Steve Miller
WebTV Service Operations Center

You might as well do the whole nine yards. Is there a decent ISP network
management system or is it still OpenView and a lot of utilities?

--Kent

Kent W. England Wireline-> fon:650.596.6321
VP of Technology Receptionist-> fon:650.596.1700
GeoNet Communications, Inc. Facsimile-> fax:650.596.1701
555 Twin Dolphin Drive Email-> mailto:kwe@geo.net
Redwood City, CA 94065 Web Site-> http://www.geo.net
                                  Wireless Text-> mailto:pagekwe@geo.net
PGP Key-> http://keys.pgp.com:11371/pks/lookup?op=get&search=0x6C0CDE69

7

I've been playing with lots of 'em recently. Of the free
  ones, only MRTG and Tkined are worth a damn -- even though
  lots of people swear by NOCOL, I've been swearing at it
  'cause the configuration is hard to visually parse and not
  consistent across the various monitoring programs.

  I'd have to say that I prefer HP Openview to Tkined, but for
  what I'm doing the difference is not so great that Openview
  would be worth the money.

  Also, Tnm (the Tcl add-on that is used by Tkined) has lots
  of useful hooks for SNMP, ping, etcetera, and can be used to
  write custom stuff pretty easily. I'll be distributing some
  of what I write here once I'm happy with how it functions.

8

I've no experience in scheduling for a NOC, but have managed support
organizations that operate 24/7. My general conclussions are that not
everything works for everyone AND you will get burnout regardless of the
schedule.

However some things I've learned about 24/7 schedules --

1. People tend to be better able to deal with 10 hour shifts. Some
people can deal with 12 hours shifts but people get more surly and the
burn out is higher.

2. Some overlap is good. You need a time to relay implicit knowledge
(we all wish everything was written down, but reality dictates
otherwise). In fact a Wednesday overlap is probably really not that bad
at all - It provides a great time for any kind of triage or department
meetings that might need to occur.

Jamie Rishaw wrote:

9

It should be noted that larger NOC operations understand
that their staff members will either:

o Burn out and die.

o Learn lots and get a real job with better pay.

Thus, many NOC supervisors generally rate the expected
lifetime for a NOC staff member at 13 months. Really.

Having said that, it would behoove a good NOC supervisor
to understand how to best use the resources at their disposal.
That means not burning people our prematurely.

Or better yet, burn them out. That means that once they get
some real experience under their belts, the rest of us
can hire them. :slight_smile:

At least, that seems to be the case at the telco's.

- paul

10

Hi,

  We're in the process of developing a 24x7 NOC, and can agree upon about
everything except scheduling.

   I asked this question to the list a couple of months ago. I've been
manager of a new NOC for about 6 weeks now.

- M-F 9-hour shifts, Sa-Su 2x12-hour shifts
  - Downside is the Sa-Su people are just "extras".

   This is basically what I went with. To integrate in the weekend people
I have them do some half shifts during the week.
   I also have two of the people that willing to be a bit flexible on
short notice. I have a two level on-call setup for problems that need
extra help. Also, the more expert people can dial or ssh in as needed.
   Burnout can be a very real problem and I haven't come to that yet.
I have hired a mix of experienced and inexperienced (college age) folks. I
fully expect that people may either burnout or gain great experience and
get better jobs, but that's always a risk. It could happen to me too!

   Barry

   Barry Caplin People's Choice TV
   Network Operations Center Manager work: bcaplin@pchoicetv.com
                                      home: bc@mtiweb.com
PGP fingerprint = 9F E9 5C B6 A2 AD 85 AE 1E 4D 11 0B C2 85 60 93
finger bc@mtiweb.com for PGP public key

11

How about what I had to live with for 8 years in Navy Satellite
Communications: 2-48-2-72. It consisted of 4 watch sections, with a
rotation of 2-12 hour days, then 48 hours off, 2-12 hour mids, then 72
hours off. I enjoyed this and had plenty of time to be involved in the
projects that the day staff was involved with. You always knew your
schedule and could have a family life as well. It'll screw with your
sleeping schedule but overall it worked for the military and I'm sure it'd
work in a NOC environment.

Doug

12

> We're in the process of developing a 24x7 NOC, and can agree upon about
> everything except scheduling.
>
> - M-F 9-hour shifts, Sa-Su 2x12-hour shifts
> - Downside is the Sa-Su people are just "extras".
   This is basically what I went with. To integrate in the weekend people
I have them do some half shifts during the week.

yes it's true.. if we get the "right" schedule people will actually be
more productive...

13

I worked in the Secretary's Situation Room/Operations Center (open 24 x
7) at the US Department of State from 1983-1984. Teams rotated first
two days 8am - 4pm (briefed by outgoing team 7:15am - 8am), next two
days 4pm - midnight (brief 3:15pm - 4pm), and final two days midnight -
8am (brief 11:15pm - midnight), and had off the next 56 hours until
their rotation started anew.

The State Department Bureau of Intelligence and Research Operations
Center had to have an alternate schedule for security reasons, i.e. to
avoid unauthorized collaboration between the Secretary's Ops staff and
INR's Ops staff. They worked two-week shifts of the same hours noted
above, i.e., first two weeks 8am - 4pm, next two weeks 4pm - midnight,
and final two weeks midnight - 8am. I can't recall how much time the
INR teams had off after their sixth week.

Both schedules were brutal on families/friends, sleeping patterns and
digestive systems, but great for commuting into and out of the
District.
Debbie Kelly