common time-management mistake: rack & stack

Randy's P-Touch thread brings up an issue I think is worth some
discussion. I have noticed that a lot of very well-paid, sometimes
well-qualified, networking folks spend some of their time on "rack &
stack" tasks, which I feel is a very unwise use of time and talent.

Imagine if the CFO of a bank spent a big chunk of his time filling up ATMs.
Flying a sharp router jockey around to far-flung POPs to install gear
is just as foolish.

Not only does the router jockey cost a lot more to employ than a CCNA,
but if your senior-level talent is wasting time in airports and IBXes,
that is time they can't be doing things CCNAs can't.

I was once advising a client on a transit purchasing decision, and a
fairly-large, now-defunct tier-2 ISP was being considered. We needed
a few questions about their IPv6 plans answered before we were
comfortable. The CTO of that org was the only guy who was able to
answer these questions. After waiting four days for him to return our
message, he reached out to us from an airplane phone, telling us that
he had been busy racking new routers in several east-coast cities (his
office was not east-coast) and that's why he hadn't got back to us
yet.

As you might imagine, the client quickly realized that they didn't
want to deal with a vendor whose CTO spent his time doing rack & stack
instead of engineering his network or engaging with customers. If he
had simply said he was on vacation, we would never have known how
poorly the senior people at that ISP managed their time.

With apologies to Randy, let the CCNAs fight with label makers.

With apologies to Randy, let the CCNAs fight with label makers.

No, your CTO shouldn't be racking and stacking routers all the time. The fundamental concept of an organizational hierarchy dictates that. But a CTO who has lost touch with the challenges inherent in racking and stacking a router can't effectively support his team. See the TV series 'undercover boss' for a (possibly trite and clichéd) example of this.

"Your position never gives you the right to command. It only imposes on you the duty of so living your life that others can receive your orders without being humiliated."
--Dag Hammarskjold

No, your CTO shouldn't be racking and stacking routers all the time. The fundamental concept of an organizational hierarchy dictates that. But a CTO who has lost touch with the challenges inherent in racking and stacking a router can't effectively support his team. See the TV series 'undercover boss' for a (possibly trite and clichéd) example of this.

I'm not suggesting it's a crime for the CTO to put his eyes and hands
on a POP local to his office once in a while, or pay a visit to his
gear in a city where he happens to be in to conduct business that
requires the presence of the CTO, not a CCNA.

It's not a waste, it's therapeutic, breaks the monotony of a desk
job, you get a bit of exercise. Doing something mindless can help
clear your thoughts, engineering yoga.

This, however, is exactly the kind of thinking that produces bad
managers. Yes, it can be boring sitting at a desk all day. If that
is your job, don't ignore it so you can play site tech while to-do's
pile up in your absence. If your are a senior decision-maker, don't
set a bad example for everyone else in your org by blowing off your
senior-level duties to fly around the globe doing something that you
have CCNAs for.

The "my desk-job is boring so I spent the day racking gear" message is
fine if you are not blowing off real work to push boxes through the
co-lo. If that's your hobby, fine, but rack & stack is not part of
the CTO, network engineer, or whatever, job. It's a hobby and you
shouldn't ignore your actual duties to go do it. It's no better than
spending the workday at the movie theater or playing world of warcraft
at the office.

Unfortuantely, some of us work for companies with limited numbers of staff so in effect, the CTO does whatever work is necessary to get the job done and keep the client happy, whether that be rack & stack, decommission, BGP peering or even disaster mitigation...

If you don't have any junior-level employees to do the junior-level
work, yes, this is certainly true. But as soon as that "CTO," who is
also wearing a bunch of other hats, finds himself with a long to-do
list, the first thing he ought to do is delegate tasks like rack &
stack to inexpensive workers so he can use his most valuable
strengths.

Obviously we are throwing around the term "CTO" at this point in
reference to pretty small businesses, but the basic concept here is,
don't let mundane tasks get in the way of work you are specially
qualified to do. Definitely don't go out of your way to do mundane
tasks so you can play IBX tourist on your company's dime!

I was once advising a client on a transit purchasing decision, and a
fairly-large, now-defunct tier-2 ISP was being considered. We needed
a few questions about their IPv6 plans answered before we were
comfortable. The CTO of that org was the only guy who was able to
answer these questions. After waiting four days for him to return our
message, he reached out to us from an airplane phone, telling us that
he had been busy racking new routers in several east-coast cities (his
office was not east-coast) and that's why he hadn't got back to us
yet.

As you might imagine, the client quickly realized that they didn't
want to deal with a vendor whose CTO spent his time doing rack & stack
instead of engineering his network or engaging with customers. If he
had simply said he was on vacation, we would never have known how
poorly the senior people at that ISP managed their time.

on the contrary, we'd PREFER if CEO's and CTO's of our trading partners know what their company is doing and how their core network actually works. (Rather than just giving one of those stupid flyers with a world map and some lines representing their network to potential customers :wink:

no "startrek questions pls". :P.

(and rack & stack with "routers" is something else than rack & stack with serverfarms, as for servers, you can just as well have an installation company or the vendor do it for you ("clearance" issues set aside ;).. with routers its a bit more touchy which wire goes where exactly, and furthermore, they have to be individually configured during install, so its better to just be there, CTO or not CTO :stuck_out_tongue:

you might be confusing the CTO for the sales manager :stuck_out_tongue:

Hi,

     Or sometimes you don't let a hazardous task like handling a Carrier Class Router to your CCNA in case they injure themself.

     Or worst... drop it =D

     ( From an actual experience )

actually most west european countries have laws against having your employees lift up stuff heavier than 20 kilos :stuck_out_tongue:

you generally don't have insurance on your network-dude to handle such things *grin* if it drops on his foot, you're screwed. (or worse, on his hand :wink:

looking at the latest models we found units weighing 110 kilos *grin*
i'm not lifting -that- up.

IT job postings in the US often include physical qualifiers such as "must be able to lift weights of up to 50 pounds (~22.7 kilos)" and "must be able to use hand tools". The lecture about using safe lifting practices usually waits until after you accept the job and go through your new-employee orientation :slight_smile:

jms

In a message written on Fri, Feb 17, 2012 at 02:29:36AM -0500, Jeff Wheeler wrote:

Randy's P-Touch thread brings up an issue I think is worth some
discussion. I have noticed that a lot of very well-paid, sometimes
well-qualified, networking folks spend some of their time on "rack &
stack" tasks, which I feel is a very unwise use of time and talent.

At the risk of offending many folks on NANOG, our industry is more
like a trade than a profession. In many cases we would do better
to treat our people (in terms of how they are managed) like skilled
trades, electricians, plumbers, metal fitters, rather than pretend
they are white collar professionals.

Low level employees should be apprenticed by higher level employees.
Many of our skills are learned on the job; just like other trades
someone with only book knowledge is darn near useless. Not only
do those above need to teach, but they need to supervise, and
exercise standards and quality control.

To your point, if you look at skilled trades the simpler the task the
more likely it will fall to the "new guy". Rack and stack is probably
one of simplest jobs in our industry. A two man team, one senior, one
junior, showing up at a colo may see the junior guy doing the physical
work, while the senior guy works out any issues with the colo provider
brings up the interconnection to them, etc.

But key to an apprenticeship is that the senior guy does some of
the low level work some of the time, and _shows_ the junior guy how
to do it right. The senior guy might rack or stack a couple of
boxes each colo they visit, and relate concepts like how the screw
hole spacing works in the rack rails, how to plan cable management,
proper labeling, and so on.

It really accomplishes much of what everyone else is talking about,
while still being productive. The "old hat" gets the downtime and
catharsis of doing a simple, yet productive task. The new guy gets to
learn how to do the job properly. The employer knows the work has
been done right, as it was overseen by the old hat, and that they will
have someone to replace him when the old hat retires.

Maybe if we did more apprecenship style learning folks would still know
how to wrap cables with wax string. It's simple, fast, and works well.

I see this as a double-edged sword. You don't want your "C" staff out in the field actually installing gear as a general course of operations as that is a great waste of their time/talent unless the "C" role is more "honorary" than anything else. That said, you might want a senior technical person on site overseeing the installation, checking the configuration, interfacing with vendor staff, testing things, etc. And it is good to have this senior staff member present when things go sideways as is often the case with new installations and often these issues are physical and are best solved with someone senior on site who can make decisions on the spot or carry more weight with the provider to get things done quickly. This should be someone that was involved in discussions with the vendor's rep. during the planning phase. If you get too reliant on sending only the cage monkeys (a term I use with fondness) then what happens when problems turn up greatly depend on your corporate culture. Do they simply stop, report the problem and wait for direction? Is there anyone on site that has the trust of the organization to make decisions on the fly and cut through the organizational red tape? Can they authorize a configuration change to work around something unforeseen? Having someone senior enough on site to make these decisions and carries some weight with the vendor can greatly reduce the time it takes to get a data center up and running. Granted, he doesn't need to be there when the initial cables are being laid out but should be there once equipment starts being installed in racks and configured. Having that experience and authority on site at the time of installation can be quite valuable.

http://www.serverlift.com/solutions/products/sl500x-server-lift/

From: Leo Bicknell [mailto:bicknell@ufp.org]
Sent: Friday, February 17, 2012 6:46 AM
To: NANOG
Subject: Re: common time-management mistake: rack & stack

Low level employees should be apprenticed by higher level employees.
Many of our skills are learned on the job; just like other trades
someone with only book knowledge is darn near useless. Not only do
those above need to teach, but they need to supervise, and exercise
standards and quality control.

+1 I believe that can not be stressed enough. There is also another aspect to it in that about 15% of the population of people are "abstract" thinkers and 85% are "concrete" thinkers. The abstract thinkers are the ones who can come up with a vision in their head of how something should work as a system and then set out and build it. Or when they are faced with a problem, can in their head envision the work around and then apply that vision on site to do things such as rewire a portion of the network in a methodical fashion with no/little downtime. Those people are relatively rare and working with your line staff gives one an opportunity to assess the various talent sets of the people in the organization. The abstract thinkers are the ones good at being able to design a network from scratch and the concrete thinkers are the ones who will be great maintaining that network and keeping everything documented and done according to policy. You need both and it just so happens that you need more of one sort in just about the same proportion that you find them in the general population. The key is to identify which people have which talents and place them where their natural abilities more closely mesh with their job requirements. If you can do that, you can have a very powerful team. If you place people into positions simply based on the number of years in the organization or because of holes punched in the cert ticket, you might end up with people in positions that they don't really like or aren't particularly good at doing. The first step in building such an organization, though, is working closely with your people and attempting to identify whose natural abilities like in which direction. Sometimes it is more about talent than training, more about nature than nurture.

To your point, if you look at skilled trades the simpler the task the
more likely it will fall to the "new guy". Rack and stack is probably
one of simplest jobs in our industry. A two man team, one senior, one
junior, showing up at a colo may see the junior guy doing the physical
work, while the senior guy works out any issues with the colo provider
brings up the interconnection to them, etc.

But at the same time, if you have a guy who might not be so sharp at troubleshooting a very complex network but is sharp as a tack when it comes to documenting things and keeping paperwork organized, that is a vital skill in the overall effort, too. That person should be given responsibility for maintaining more of the documentation, organizing things so they are easy for other employees to find, etc. and their pay should still continue to increase as they gain wider scope across more of the organization over time. The point is that it often takes many different sorts of skills and attempting to match people's natural talents to the requirements of the organization benefits both parties provided the individual involved doesn't see their position as a dead end. A good person of the sort mentioned above can literally save hours of time for people doing other tasks such as troubleshooting a problem. There is a certain synergy involved and some organizations recognize that, and some don't. Some are better in an architectural role, some are naturally better in a sustaining role, others are better at an organizational support role and (darned) few are good at all of those tasks. Sometimes we don't have the luxury of such specialization of roles, but some organizations do, particularly if they are in a phase of reorganization and downsizing. One thing to look at might not only be "how good is this person in their current role" but also "would this person absolutely kick butt in a different role".

But key to an apprenticeship is that the senior guy does some of the
low level work some of the time, and _shows_ the junior guy how to do
it right. The senior guy might rack or stack a couple of boxes each
colo they visit, and relate concepts like how the screw hole spacing
works in the rack rails, how to plan cable management, proper labeling,
and so on.

Actually, just having the senior person assist with some tasks such as moving/installing heavy/unwieldy gear does more than just show them how to do it right, it is actually quite an important almost sort of bonding experience between employees. It says "I'm not allergic to work and not above doing the same job you are doing when it needs to get done, we are all important pieces of the big picture." It can give an employee a sense that they are respected and appreciated for the job they do, even if it is fairly low on the corporate org chart. It is still vital to the success of the overall business or they wouldn't be there to begin with. Doing things like this telegraphs that in a tangible way without having to spew a lot of corporate propaganda.

It really accomplishes much of what everyone else is talking about,
while still being productive. The "old hat" gets the downtime and
catharsis of doing a simple, yet productive task. The new guy gets to
learn how to do the job properly. The employer knows the work has been
done right, as it was overseen by the old hat, and that they will have
someone to replace him when the old hat retires.

The "old hat" still gets job satisfaction from seeing a vision come to physical life and operate as planned. Why deprive them of that? It can re-energize one's love of a particular carrier field and remind them why they are in that field to begin with.

Maybe if we did more apprecenship style learning folks would still know
how to wrap cables with wax string. It's simple, fast, and works well.

Leo, in many trades, telecommunications being one of them, the military was one source of new people with some skills and with some hands-on experience. As that scales back these days, it gets harder to find such people. We don't have trade schools and we don't have apprenticeship programs like companies used to have so I agree. People coming out of a community college or a certification program know enough to be extremely dangerous (sort of like a lieutenant with a screwdriver, the most dangerous person in the world aside from a corporal with a clipboard) and need to be mentored as they gain perspective in real world situations. I completely agree that we should be looking more at our employees in the longer term as a nurturing process and identifying where their natural interests and abilities can benefit both sides of the equation. Having that interaction with the senior staff is vital. And that senior staff member should not only be explaining WHAT he is doing, but WHY he is doing it that way.

Knowledge transfer should also include the very important WHY NOT to do
something a certain way. This part is often left out. Considering that
most bit-twiddler tasks can be performed a multitude of ways, both sides of
the argument should be presented. Perhaps this is obvious to all on the
list, but it's certainly not to junior staff.

Jeff Wheeler <jsw@inconcepts.biz> writes:

With apologies to Randy, let the CCNAs fight with label makers.

Yeah. And you need do be at last CCNP to switch a module in a router.

Had this request last year. I first thought that some troubleshooting /
configuration was involved but it was just replacing a module.

Jens

...

Imagine if the CFO of a bank spent a big chunk of his time filling up ATMs.
Flying a sharp router jockey around to far-flung POPs to install gear
is just as foolish.

There is a theory of management that says a good manager
needs to know nothing about the staff or the jobs he is managing,
because his job is about returning profit to the shareholder,
and not about what the company does. AFAIK, these
theories are made in the academic halls of the business
schools, which churn out MBAs, and, self-selected group
that they are, believe in (more) managers, and (more)
powerpoint business plans, and (more) theory.

I happen to come from a different background, and believe
that it has value to understand what the people who are
working for you actually do. That does not mean the CEO
should spend all day delivering the mail (or flipping burgers),
but she had better have done it a few times, and it is a good
idea to do it from time to time to see what has changed.
It keeps the manager grounded with the reality.

(I have been told that the reason that the commanders
in the Army are reluctant to send their people to battle
is that they have experienced it, and know it is hell.
And the reason the people will go to hell for their
commander is that the commander has the moral
authority of having done it, experienced it, know
that they are asking a lot, but it is for the common
good. People will follow a leader who has been there,
done that, and not so much when it is just an academic
business plan on a powerpoint slide.)

With all due respect, Jeff, I think you are missing several factors that come into the human equation beyond merely the most efficient use of a particular person's time.

I would go stark-raving bonkers trapped in a cubicle doing only things that CCNAs can't if I didn't get the occasional break to go out and play with real hardware in the field. Most of the well-paid well-qualified networking folks I know are the same way.

I also think that when we spend too many consecutive weeks/months/years behind a desk without going out in the real world, we become progressively more detached from the operational reality where our designs have to operate.

On the surface, it might seem an inefficient use of financial/human resources, but, I think that there is value to time in the field that doesn't necessarily show up directly on the balance sheet.

Admittedly, in my current position, I'm no longer in an operational role for the most part, but, I'm even more out in the field and spending more time in airports.

Owen

From: Gary Buhrmaster [mailto:gary.buhrmaster@gmail.com]
Sent: Friday, February 17, 2012 12:54 PM
To: Jeff Wheeler
Cc: NANOG
Subject: Re: common time-management mistake: rack & stack
...
> Imagine if the CFO of a bank spent a big chunk of his time filling up

ATMs.

> Flying a sharp router jockey around to far-flung POPs to install gear
> is just as foolish.

There is a theory of management that says a good manager needs to know

nothing about the staff or the jobs he is managing, because his job is about
returning profit to the shareholder,

and not about what the company does. AFAIK, these theories are made in

the academic halls of the business schools,

which churn out MBAs, and, self-selected group that they are, believe in

(more) managers, and (more) powerpoint business plans, and (more) theory.

I happen to come from a different background, and believe that it has

value to understand what the people who are working for you actually do.

That does not mean the CEO should spend all day delivering the mail (or

flipping burgers), but she had better have done it a few times,

and it is a good idea to do it from time to time to see what has changed.
It keeps the manager grounded with the reality.
(I have been told that the reason that the commanders in the Army are

reluctant to send their people to battle is that they have experienced it,
and know it is hell.

And the reason the people will go to hell for their commander is that the

commander has the moral authority of having done it, experienced it,

know that they are asking a lot, but it is for the common good. People

will follow a leader who has been there, done that,

and not so much when it is just an academic business plan on a powerpoint

slide.)

+1 for Gary's comment.

That is the large difference between LEADING and MANAGING.

In the context of the military scenario above, Grace Hopper comes to mind
because of her nanoseconds etc
"In her retirement speech, instead of dwelling on the past, she talked about
moving toward the future, stressing the importance of leadership."

I was lucky enough to have heard her speak once at an ACM event.

Tony Patti
CIO
S. Walter Packaging Corp.

Cue the obligatory cabling porn thread.

Cheers,
-- jr 'and aren't all the old Bell guys dead now?' a

I still have my nanosecond. Did she hand them out to the crowd there?

From: Mike Andrews [mailto:mikea@mikea.ath.cx]
Sent: Friday, February 17, 2012 1:44 PM
To: 'NANOG'
Subject: Re: common time-management mistake: rack & stack

> In the context of the military scenario above, Grace Hopper comes to
> mind because of her nanoseconds etc "In her retirement speech, instead
> of dwelling on the past, she talked about moving toward the future,
> stressing the importance of leadership."
> The Younger Years of Grace Murray Hopper
> I was lucky enough to have heard her speak once at an ACM event.

I still have my nanosecond. Did she hand them out to the crowd there?

Yes, of course!

I remember that she said they were borrowed from a phone closet in the
Pentagon...

Of course, she is also famous for "It's easier to ask for forgiveness than
it is to get permission"
Notable Quotation from her Wikipedia page at

Tony Patti
CIO
S. Walter Packaging Corp.

From: Leo Bicknell [mailto:bicknell@ufp.org]

At the risk of offending many folks on NANOG, our industry is more

like

a trade than a profession. In many cases we would do better to treat
our people (in terms of how they are managed) like skilled trades,
electricians, plumbers, metal fitters, rather than pretend they are
white collar professionals.

Low level employees should be apprenticed by higher level employees.
Many of our skills are learned on the job; just like other trades
someone with only book knowledge is darn near useless. Not only do
those above need to teach, but they need to supervise, and exercise
standards and quality control.

I disagree. The best model is - gasp - engineering, a profession which
many in "networking" claim to be a part of, but few actually are. In the
engineering world (not CS, not development - think ME and EE), there is
a strongly defined relationship between junior and senior engineers, and
real mentorship. The problem with "networking" is that TOO MANY skills
are learned on the job (poorly), rather than folks coming in with solid
fundamentals. I blame our higher education system for being ineffectual
in this regard. Most of the "book learning" that you refer to is not
true theory - it's a mix of vendor prescriptions and
overgeneralizations. In "networking", you don't learn real theory until
you're very senior - you learn practice, first.

The true problem is that most "networking" professionals came out of a
CS background or are self-taught. They might be clueful and they might
be highly adept, but they lack the structure of an engineering
educations and formal mentorship. They also lack real licensing, which
is a separate problem.

To your point, if you look at skilled trades the simpler the task the
more likely it will fall to the "new guy". Rack and stack is probably
one of simplest jobs in our industry. A two man team, one senior, one
junior, showing up at a colo may see the junior guy doing the physical
work, while the senior guy works out any issues with the colo provider
brings up the interconnection to them, etc.

Rack and stack is not a network engineer task, anymore than running a
208/3 phase circuit is an electrical engineer's task. Instead, rack and
stack is a task for a skilled telecom tradesman. I have nothing against
network engineers getting out of the office to do this, but the quality
of their work will never be up to a real telecom guy. Look at the
cabling. You can always tell which has been done by a "network engineer"
and which has been done by a real tradesman. Guess which one is better?

Dan