Spanning tree melt down ?

Unnamed Administration sources reported that Daniel Golding said:

"It was Dr. John Halamka, the former emergency-room physician who runs
Beth Israel Deaconess Medical Center's gigantic computer network"

It appears what really happened is that they put an emergency room doctor
in charge of a critical system in which he, in all likelyhood, had
limited training. In the medical system, he was trusted because of he was
a doctor. The sad thing about this is that there seems to be no
realization that having experienced networking folks in this job might
have averted a situation that could have been (almost certainly
was?) deleterious to patient care.

Did you, in fact, read Halamka's resume? He sounds to me like
he has more smarts in the networking area than many of the
RedmondWorshipers I encounter regularly.

Was he Sean Donelan or Randy Bush? No.

Yes, I read his bio. I'm sure he's quite the techie amongst his fellow
physicans, and I think thats a great thing. However, its more than just a
bad idea to put someone who isn't completely proficient in a job like this
- its bad for the patients. If you want to run a shoe company, and put a
shoe salesman with a couple linux boxes in charge of your network, more
power to you. However, if you run a huge hospital, at which, there are
numerous patient affecting IT systems, you really have an obligation to
hire a professional, rather than a talented amateur, with all due respect
to the good doctor.

As far as "Redmondworshippers" - whatever does the job. If you are running
a hospital, and microsoft products work for you, then buy them. The key is
knowing what to buy, how to keep in from breaking, and how to fix it,
quickly and efficiently when it does, be it Cisco, Microsoft, Linux, a
couple of tin cans with string, or whatever.

Some background for those not from/in Boston: This is a very large medical
center, not a community or midsized hospital

- Dan

## On 2002-11-29 15:05 -0600 Daniel Golding typed:

Yes, I read his bio. I'm sure he's quite the techie amongst his fellow
physicans, and I think thats a great thing. However, its more than just a
bad idea to put someone who isn't completely proficient in a job like this
- its bad for the patients. If you want to run a shoe company, and put a
shoe salesman with a couple linux boxes in charge of your network, more
power to you. However, if you run a huge hospital, at which, there are
numerous patient affecting IT systems, you really have an obligation to
hire a professional, rather than a talented amateur, with all due respect
to the good doctor.

Hi Daniel,

Are you suggesting that a CIO at a "huge hospital"(or any other enterprise)
Needs to be an expert at LAN/WAN networking, Systems, DBA & Security
Rather than a management expert that has a good grasp of the basic IT
issues and understands the core business needs of the enterprise ?

I would be more likely to say that, then he need be a physician with management skills. I think Dan made this point already in his post. I have tremendous respect for physicians having grown up in that field. But they tend to be so smart that they get themselves in trouble, or have trouble knowing their limitations. I say that with all respect.

I think this case showed that the IT staff was lacking some checks and balances, or just proper procedures that most networking engineers might have brought to the table.

There are a lot of physicians that get themselves in trouble flying planes too, as evident by some of the nicknames given to some of the high performance planes. At some point networks become complicated enough that experts need to be brought in. A manager knows when to delegate and I think it's important to have good people to delegate to. There are plenty of specialists in the medical field, so this is not a new concept.

A good question, Rafi. IMHO, a CIO at a hospital or other large,
technology-intensive institution should have a very solid IT background.
by preference, it is someone who has come up the ranks from development,
systems administration, or network engineering, perhaps gotten an MBA, and
gone into the management/financial side of the house. You can't be an
expert at everything. However, you should be an expert at some aspect,
preferably the one that has the greatest importance to the enterprise.
(i.e. you want your CIO at a biotech company to be very database/storage
heavy. you probably want your CIO at a bank to be very network or database
knowledgable.)

I suppose the most important thing is, hire someone who can tell when they
are being deceived by vendors, contractors, or employees. That requires a
good general knowledge of information technology concepts. This kind of
person would also know that some aspects of IT like documentation,
planning, and scalability are all constants, regardless of what type of
project is being worked on.

One of our greatest weaknesses in this field, is the belief, by those who
do not work in it, that anyone can pick up a book and quickly get up to
speed on technology. Sadly, that is not the case.

So, yes, I'm saying that a physician probably should not be the CIO of a
very large hospital.

- Dan

I find the reactions on this mailing list disturbing, to say the
least. The rush to judgement about what happened appears to be based
on speculation and assumptions about how this large facility was run,
managed and staffed.

As far as I can see, the known facts are:

There was an oversize layer 2 network and it broke.
It was hard to repair.
The CTO is a physician on the hospital board who, on first sight,
appears to have considerable qualifications in the IT area.

The unknowns are:

How many, if any, employees are there who are directly tasked with
maintaining the network?
How well trained are they?
How well documented was the network?
How well did whatever staff were responsible for maintaining it
actually understand the network?
What sort of reporting was done - did anyone raise the issues of the
size of the network?
Was any risk assessment done?
How much planning had been done, if any, to address the problems?
How did the network get to be the way it was? Did it just grow or were
changes forced upon it by external constraints?
Why was it hard to repair?

But people are speculating with no knowledge of the
actual organisation, history, planning, what risk assesment had or had
not been done, or any other information excpet guesses and prejudices
about what they think might have happened and an apparent assumption
that this is all the result of turning over a large enterprise network
to a jumped up physician whose only qualification was running a couple
of Linux boxes on a home network. None of the above unknown issues
have been addressed anywhere.

I hope the posters never pull jury service, as there seems to be a
complete disregard for the idea of gathering facts before passing
judgement.

## On 2002-11-30 15:41 +0100 Jim Segrave typed:

I find the reactions on this mailing list disturbing, to say the
least. The rush to judgement about what happened appears to be based
on speculation and assumptions about how this large facility was run,
managed and staffed.

As far as I can see, the known facts are:

There was an oversize layer 2 network and it broke.
It was hard to repair.
The CTO is a physician on the hospital board who, on first sight,
appears to have considerable qualifications in the IT area.

I agree except that it's not CTO but rather the CIO

The unknowns are:

[snipped for brevity]

Many unknowns - no argument here

But people are speculating with no knowledge of the
actual organisation, history, planning, what risk assesment had or had
not been done, or any other information excpet guesses and prejudices
about what they think might have happened and an apparent assumption
that this is all the result of turning over a large enterprise network
to a jumped up physician whose only qualification was running a couple
of Linux boxes on a home network. None of the above unknown issues
have been addressed anywhere.

## On 2002-11-29 23:43 +0200 I typed:

Are you suggesting that a CIO at a "huge hospital"(or any other enterprise)
Needs to be an expert at LAN/WAN networking, Systems, DBA & Security
Rather than a management expert that has a good grasp of the basic IT
issues and understands the core business needs of the enterprise ?

Can you please indicate the assumptions/speculations in the above question?

I hope the posters never pull jury service, as there seems to be a
complete disregard for the idea of gathering facts before passing
judgement.

1) You seem to imply *all* previous posters in this thread
(which is why I'm responding to you in public)

2) IMHO you should try having a good long look in a mirror :wink:

Thus spake "Jim Segrave" <jes@nl.demon.net>

I find the reactions on this mailing list disturbing, to say the
least. The rush to judgement about what happened appears to be based
on speculation and assumptions about how this large facility was run,
managed and staffed.

Everyone with the facts is covered by NDA.

I have tried to provide a characterization of the events based on similar
incidents at other Fortune 100 shops. I believe this to be educational even
if I can't confirm it is completely relevant to the incident at hand.

If you care about getting more details, then call the good doctor and ask
him yourself. In the meantime, you're not going to get any more information
from the press or NANOG than you have about, say, the week-long WorldCom FR
outage a year or so ago.

I hope the posters never pull jury service, as there seems to be a
complete disregard for the idea of gathering facts before passing
judgement.

I'd take a jury of NANOGers over the usual pool of people too poor or stupid
to find a way out of serving. At least most of _us_ live in the same
delusional reality.

S

Everyone with the facts is covered by NDA.

The owner(s) of the facts can always decide what facts to disclose.

If you care about getting more details, then call the good doctor and ask
him yourself. In the meantime, you're not going to get any more information
from the press or NANOG than you have about, say, the week-long WorldCom FR
outage a year or so ago.

Worldcom could disclose what happened to their network. (or what
happened to their accounting, but that's a different story).

I suspect we will learn more about what happened to Beth Israel Deaconess
Hospital's network than we've ever heard publically about any of
Worldcom's network problems. Dr. John Halamka has already publically
stated he intends to tell other hospitals what happened and how they can
avoid the same problem.

Excerpt from the Boston Globe article:
   "No other Massachusetts hospital has ever reported such a long-lasting
   or disruptive network crash, said Elliot Stone, executive director of
   the Massachusetts Health Data Consortium, a group that brings together
   chief information officers from hospitals and health plans around the
   state. He praised Beth Israel Deaconess for being open about the
   problem and sharing lessons learned, both about technology itself and
   about policy - such as the need to enforce rules against unauthorized
   additions of new software onto the network."