If I were going to provide a 365x24x7 NOC, how many teams of personnel do I
need
to fully cover operations? I assume minimally you need 3 teams to cover the
required
24 hr coverage, but there is off time and schedule rotation?
thoughts, experience?
Mike
Although more geared up for on-call, http://blog.hinterlands.org/2010/07/running-an-oncall-rota/ is a very useful resource.
Peter
I've had it done in places where I work where you'll have 3 rotations
working 12 hour shifts.
In a 2 week pay period they get their 80 hours in a blend 36 one week
and 44 the next. It gives some nice consecutive days off time which
also doubles as a retention tool for some employees. You might have
to get creative to have all the days work out but it can be done.
I worked once in a production plant. They had 4 crews a,b,c,d and worked 6 days on 2 off. It was rotating shifts. As an example crew A worked 6
days 8-4 was off 2 days then started the next shift on 4-midnight, then midnight to 8. So at any one day 3 of the 4 crews were working and the other
crew was off.
Well, if you feel like being "nice" to your employees, or want to stop
them from making mistakes a few months/years in to shift work, (or if
you're having to set something up abroad), the Working Time Directive
can be a useful guideline.
(Full details at http://en.wikipedia.org/wiki/Working_Time_Directive
and http://www.businesslink.gov.uk/bdotg/action/layer?topicId=1073858926
if you're board, ignore the silly bits)
But basically, in general, workers aged 18 and over are entitled to:
- work no more than six days out of every seven, or 12 out of every 14
- take a 20-minute break if their shift lasts for more than six hours
- work a maximum 48-hour average week
And in general, night workers:
- should not work more than an average of eight hours in a 24-hour
period, averaged over a reference period of 17 weeks
If you're an employer, be glad you're in North America 
HTH,
Alex
Have you taken into account number of alarms per hour, inbound call volume for repairs, and how much repair is done at the first tier level? Bare minimum staffing in a busy environment?
Randy Sanders
Hours in the working year = 8 * 5 * 48 = 1920
Hours in the calendar year = 24 * 7 * 52 = 8736
Ratio = 4.55
Tony.
Guys,
Thanx alot, there is some great stuff here, also some stuff I had not
thought of.
thanx again,
Mike
For what it's worth, was part of a datacenter operations department that had a 24x7 team. 4 shifts, 4 staff on each shift (1 was supervisor who did same work as the rest, 1 'point of contact' who stayed in the office).
4 days on, 4 days off, 12 hour shifts, 8-8. Shift teams would alternate between day and night (so 4 day, 4 off, 4 night, 4 off, repeat ad infinitum). During the day that was bolstered by 6 day-staff, Monday to Friday, who would have a staggered start through the day (IIRC 2 start at 8, 2 at 9, 2 at 11)
Date: Fri, 15 Apr 2011 09:14:05 -0400
Subject: 365x24x7
From: harbor235 <harbor235@gmail.com>
To: NANOG list <nanog@nanog.org>If I were going to provide a 365x24x7 NOC, how many teams of personnel do I
need
to fully cover operations? I assume minimally you need 3 teams to cover the
required
24 hr coverage, but there is off time and schedule rotation?thoughts, experience?
You haven't done the math right. You're not even *close* to 'real world'
requirements.
3 people only provide 24-hour coverage for *five* days/week. *before*
considering any the "stuff" mentioned below.
7x24 requires _4-1/5_ persons per position, assuming 40-hr work week,
_before_ allowing for vacations, etc..
Scheduled vacation time adds another 5-8% to the manpower requirement.
Don't forget "holidays", 'sick days', and 'personal time'. that's another
5-10%, or so.
Do you intend to allow any participation in 'professional' activities,
conferences, etc. and/or any on-going training? Another 5% easily.
Now, don't foreget about 'in-house' "overhead" actitivites. e.g.
'staff meetings'. 3-5% (i.e., 1-2 hours/week) is a not-unreasonable
estimate.
These things push the total requirment to just over 5 people per position,
but 'shift scheduling' is a *bitch*.
Lastly, you have to consider meal-breaks and such _during_ the shift.
This means 'n+1' people per shift, to ensure 'n' on duty at all times.
Take your minimum required NOC positions, add one person (for the n+1
completeness), then multiply by 5 and you've got a realistic estimate of
the *minimum* manpower requirement.
Note: You'll probably want 'above minimum' staffing on 1st shift (and
possibly some on 2nd) to handle 'routine'/'non-essential' activities.
What is the work distribution? Do you have stable patterns of more-vs-less activity? If things are busy daytime and quiet nighttime, obviously you need different-sized teams.
Variable scheduling of staff is often deemed more fair, but I think it makes things less stable. People are constantly having to change their life.
A simple model has 3 teams over weekdays/nights, leaving you with weekends, holidays and vacations.
I was a part-time operator during college. Full-timer shifts were 3 people; maybe 2 for graveyard.
There were 3-5 of us covering things for that added time. But, then, the major operations were purely daytime, during the week. Graveyard shift was quiet enough that we surreptitiously bought a cot...
d/
Variable scheduling of staff is often deemed more fair, but I think it makes
things less stable. People are constantly having to change their life.
Rotating shifts between daytime and nighttime is a horrible thing to
do to your workers, both for their health and their attention span.
Full-time night work isn't great, but rotating work is even worse.
Apes are generally diurnal, not nocturnal or crepuscular. Shuffling
who has to work which days is annoying enough.
Bill Stewart wrote:
Rotating shifts between daytime and nighttime is a horrible thing to
do to your workers, both for their health and their attention span.
Full-time night work isn't great, but rotating work is even worse.
Apes are generally diurnal, not nocturnal or crepuscular. Shuffling
who has to work which days is annoying enough.
I spent several months working in a place with rotating shifts. One week 10pm to 7am, then the next week 2pm to 10pm then the week after 7am to 2pm and then repeat. I never understood why they were different lengths either. It was pretty grim and I'd much prefer to have had shifts change ever few months.
Some people claimed they'd have preferred it if we'd changed to the _following_ shift rater than the preceding shift each week but never having tried that I don't know how it would be.
Rotating shifts between daytime and nighttime is a horrible thing to
do to your workers, both for their health and their attention span.
One of the places I worked had the following pattern. It was horrible
2 days/shifts of 6am till 6pm
2 days/shifts of 6pm till 6am
4 days off
Wayne
I was offered a similar role… but more painful (Imho)
4 days 8am till 8pm
4 days off
4 days 8pm till 8am
4 days off
Rinse and repeat.
...Skeeve
If I were going to provide a 365x24x7 NOC, how many teams of personnel
do I need to fully cover operations? I assume minimally you need 3 teams to
cover the required 24 hr coverage, but there is off time and schedule rotation?thoughts, experience?
It depends a lot on how you structure your shifts; the problem is getting
everyone 40 hours without unnecessary overlap.
The TV master control facility in which I'm working presently does it
by doing overlapping 10 hour shifts; it takes 10 people to have 2 on-shift
at all times. You work 6 hours with one person, and 4 with the other.
Your 3 teams estimate is, I suspect, derived from dividing 24 hrs/day by
8/hrs shift... but that doesn't take weekends into account, and you don't
necessarily want to have 3 more teams who only work 2 days a week.
I don't think there's *any* way to do it that guarantees you'll always
have the same people working together; fortunately, I also don't think
that's all that necessary in that environment.
Cheers,
-- jra
For what it's worth, was part of a datacenter operations department
that
had a 24x7 team. 4 shifts, 4 staff on each shift (1 was supervisor who
did same work as the rest, 1 'point of contact' who stayed in the
office).
4 days on, 4 days off, 12 hour shifts, 8-8. Shift teams would
alternate
between day and night (so 4 day, 4 off, 4 night, 4 off, repeat ad
infinitum). During the day that was bolstered by 6 day-staff, Monday
to
Friday, who would have a staggered start through the day (IIRC 2 start
at 8, 2 at 9, 2 at 11)
And also for what it's worth, my understanding of body circadian rhythm
research suggests that "every 8 days" is *WAY* too short a period to
be flipping people's shifts from day to night; every 6 months is more
reasonable.
Cheers,
-- jra
You didn't work for the FAA, Dave, did you?
Cheers,
-- jra
Some people claimed they'd have preferred it if we'd changed to the
_following_ shift rater than the preceding shift each week but never
having tried that I don't know how it would be.
I've read stuff that confirms that changing to a later shift is much
easier than changing to an earlier one. It certainly matches my
experience that the jet lag flying to Europe, where I have to get up
six hours earlier, is much worse than flying back.
It also makes the obvious point that fewer shift changes are easier on
the employees than more.
R's,
John
Having done this for quite a few years my advice is that once you get
past the basic arithmetic of people-hour-equivalents etc what you need
is a middle manager who is a good "horse trader" because it quickly
becomes a market of "I can do grave shift Tuesday if you'll take my
Saturday AM, I've got a wedding I have to be at, XXX says he'll take
your Friday night if you take the Saturday but if you still want
Sunday off you're going to have to find someone to cover that and
Monday morning unless..."
And that manager has to be responsible so such shuffling really ends
up with necessary coverage because people make mistakes, often in
their favor (OOPS!), and that can get complicated.
Regular-hours life goes on despite peoples' shifts and they're forever
having to be at jury duty, doctor's appointments, social engagements,
govt offices, religion, dog kennels, kids' school meetings, etc etc
etc which are fixed in the assumption that people work roughly 9AM-5PM
M-F. Yeah some of that is actually easier for people who work odd
hours and even the attraction of odd hour work but a lot of it isn't
and comes up only once in a while per employee which means, for the
manager, once or twice a week.
Another practicality is policies about people "hanging out" during
non-work hours because you'll find overnight people are night people
even on their days off and often have nothing better to do than come
in and use your higher bandwidth etc, or just get out of the house,
they'll become friendly with overnight co-workers, which might be fine
by itself until someone brings in a couple of six-packs because hey
they're not working and there won't be a boss around after 1AM
and...again, maybe not a problem until they decide to share their fun
a little too much with people on duty who are nearby...
The other problem is food, overnight people get hungry and depending
on location and facilities for bringing and preparing food
opportunities for a 3AM lunch can be limited in many locations and if
you don't solve it somewhat employees may come up with suboptimal
solutions like deciding it's only fair that they get an extra 20
minutes travel each way for an hour lunch, or cooking mess, etc.
And of course who to call when something goes very wrong at 4AM or
they'll call you every time, for some value of "you", or worse they'll
call the police (for example) when you prefer they hadn't (or the
police or other LEO will call them)...you need some procedures and
training to anticipate such things, power problems, building
management calling at 5AM that they need whoever is on to let some
people in to knock down all the walls, they're waiting outside...
Finally, a weird thing? If there's overnight telephone support people
(some of whom are customers!) will find it, sometimes lonely people
who want to hear a human voice at 4AM and sometimes completely crazy
people who, well, want to hear a human voice at 4AM, some of them are
very good at manipulating whoever will answer the phone, "our little
secret please don't hang up please don't hang up really please!", you
need policies and training if that's a possibility. Gak, I could tell
stories...