Is your ISP Influenza-ready?

Mailman List Mirror (Read Only)
1

In a www.washingtonpost.com article:

  http://tinyurl.com/s2jpz

It is said:

  President Bush is expected to approve soon a national pandemic
  influenza response plan that identifies more than 300 specific
  tasks for federal agencies, including [some stuff and] expanding
  Internet capacity to handle what would probably be a flood of
  people working from their home computers.

That's not a lot of detail, and the article only cites www.pandemicflu.gov
as a reference. They don't appear to have published any detailed plan
that Pres. Bush is evidently about to sign there. What is published there
feels like background information, and is vaguer still.

Anyone with more information on what they're talking about?

2

Not specifically, but i've seen various public-private groups
talking about this for at least several months. I think it's quite
interesting to have the hurricane season just a few weeks away and
having this go by. (i've also seen people talking about preparations for
the recovery efforts for any 2006 storms after what happened last year).

  I think it's important that everyone reading this realize
that the internet is now "Critical Infrastructure" for the global
economy.

  How many of the people here have heard of the NIPP or NRP?

  What preparations have you done for some sort of catastrophic
event? (facility disaster, natural disaster, or some more basic challenges
like loss of drinkable water/food for a week or two?)

  These aren't meant to necesarily be answered here, but what challenges
would you face? Do you have any "mutual-aid" like agreements with your
peers/competitors? (When there is a major ice storm or hurricane, you see
all the verizon trucks in bellsouth territory for example, helping out).

  Back to the original question, how well could you cope for such
an event? It's always challenging to think about what would happen
as sometimes it includes the unexpected.

  - Jared

3

Quite. One case of bird flu in a major city and you won't need government
intervention to make people telecommute -- large numbers will
independently and spontaneously decide to do so. (Back in the days of
dial-up, I had a lot of trouble connecting to Bell Labs on snow days. No
rule, and the place was officially open for business. But everyone just
did the rational thing.)

    --Steven M. Bellovin, http://www.cs.columbia.edu/~smb

4

In a www.washingtonpost.com article:

  http://tinyurl.com/s2jpz

It is said:

  President Bush is expected to approve soon a national pandemic
  influenza response plan that identifies more than 300 specific
  tasks for federal agencies, including [some stuff and] expanding
  Internet capacity to handle what would probably be a flood of
  people working from their home computers.

That's not a lot of detail, and the article only cites www.pandemicflu.gov
as a reference. They don't appear to have published any detailed plan
that Pres. Bush is evidently about to sign there. What is published there
feels like background information, and is vaguer still.

How about this idea... are your corporate VPN services (assuming there is
one aside fromm 'ssh to the bastion host' of course) prepared to
double/quadruple/more-uple their normal concurrent user counts? During the
fallout of Katrina we observed this being a problem for some of the
corporations in region :frowning: I know that quite a few folks plan for 50% or
less of their employees to be 'dialed in' :frowning: If 100%, or some majority,
how do the corp folks plan on supporting that? :frowning:

5

I don't know about the rest of the country, but in the northeast, there are MANY days during the winter when only a couple of people can make it to our office and a number of our clients have the same situation. On those days at Tellurian, everyone who can't make it in works from home. It is completely transparent to our clients. People in NJ may understand if we have a blizzard, but our clients in CA don't care and expect the same level of service. As an ISP/ASP, we have the bandwidth, phone lines, and VPN concentrator capacity available for our own use, but what about your clients who may only use their connection for email and web access and a few road warriors and sales folks normally. Perhaps 200-300 people can share a T1 with light to moderate use in one office, but with 200+ people connecting back in via VPN, a T1 isn't going to cut it. Scale up or down DSL to OC3 based on the client. I don't think it is something people design for and I know it isn't something most clients will pay for until they need it and don't have it. Then they will want more bandwidth installed immediately.

-Robert

Tellurian Networks - The Ultimate Internet Connection
http://www.tellurian.com | 888-TELLURIAN | 973-300-9211
"Well done is better than well said." - Benjamin Franklin

6

Vendors like it because it's a revenue boost. It obviously requires build-ahead
capacity and maintenance of overload capacity that will likely sit idle for 99%
of it's life span. Who pays?

[ ..hears ISP product managers scurrying to create "PriortyVPN" or
"priority vpn" products as a result... heh -> implied trademark here]

-M<

7

Probably sell them a product where b/w is burstable to a much higher
level - at least for short periods of time, to deal with sudden use
spikes (or to create extra capacity for those periodic trojan
outbreaks that will otherwise simply max their pipe out)

8

That works great if one customer has a trojan outbreak.

Unfortunately, it's rare that only one customer at a POP has a blizzard outbreak.

9

All the guidance suggests you're going to lose as much as 40% of your
workforce.

Well, what intrigues me, is: which 40? I don't think the virus is going
to select sales, marketing, and Tech support in that order (unless it's
an STD epidemic, har har). Were that the case we might actually look
forward to such outbreaks.

On the other hand, at *every* substantially sized network I've worked
at, the Network Engineering types that might reasonably do something
useful in such an emergency situation are generally:

1) A close-knit group, going to lunches together and cohabitating
   cubicles so as to avoid exposure to aforementioned sales, marketing,
   and tech support or customer service. Indeed, at a few places I
   worked, they even spent most every weekend together. For all
   the rest of the world decrying geeks as socially inept, they are
   highly efficient at social assimilation of their own kind.

2) Given a 'low desirability' office space. No windows, usually poor
   air circulation. It is often called "The Back Room" or similar, or
   is located in a space you wouldn't expect to find humans. This isn't
   (usually) anyone being mean: engineers seem to like dark corners,
   something about making it easier to read monitors, and locations that
   provide fewer interruptions due to unlikelyhood of foot traffic.

3) Better at taking care of their networks than themselves. Or at least,
   more willing to - too frequent is the case I see an engineer, hacking,
   coughing, and wheezing at his monitor, plucking away at the keyboard
   deep into the night.

So there you have it. They're likely to come to work even though they're
sick (presuming they don't know it's a lethal virus), where they work and
spend all their face-to-face time in close quarters with recirculated air
with the rest of the company's engineers.

It's like someone intentionally optimized this function specifically to
be the most pessimal.

So I think it's actually highly probable that a meatspace-viral vector
would take out the entire engineering staff at most service providers
I've worked at if only one of them caught the bug. I have to imagine
this is representative of other work environments. We all seem to share
the same collective experience in this sense, at least the folks I've
talked to.

And that loss would be way under 40% of the total company's staff, a
mere blip really.

So, which 40% can you afford to lose? How likely is it that the 60%
that's left behind will be able to do the job? Will they need step-by-
step instructions so that even an untrained monkey can muddle through?

10

Hello;

  Back to the original question, how well could you cope for such
an event? It's always challenging to think about what would happen
as sometimes it includes the unexpected.

All the guidance suggests you're going to lose as much as 40% of your
workforce.

Well, what intrigues me, is: which 40? I don't think the virus is going
to select sales, marketing, and Tech support in that order (unless it's
an STD epidemic, har har). Were that the case we might actually look
forward to such outbreaks.

The most likely disease vector is, from what I have heard, airline travel.
Assorted people from all over are brought together for a meal (or, at least,
bogus pretzels) in a confined space for a few hours, then released back into
the general population.

So the NANOG and IETF crowd would probably be the first to go. Since I travel
a lot, and to the same meetings, I can't say that that this seems like a good
idea to me.

If any of this actually starts happening, we all may become very interested in
video conferencing.

Regards
Marshall

11

(rest of interesting note snipped because you know how to find it)

(Warning: unnecessary and overly long speculation follows)

Studies of changes brought on by major outbreaks of the plague in
Europe tend to be surprised by the qualitative and unexpected changes
which occurred. Many make sense only in retrospect.

For example, there was recently an article floating around in the news
about how the plagues of 1666 and thereabouts may've brought on the
mini ice age thereafter which itself may've been in part responsible
for motivating the US revolution against Britain in 1776, among other
events, but that's a pretty big one in the course of modern history.

The reasoning was that the plague so reduced both the farming
population and consumption that it caused a lot of farmland to be
abandoned to second growth forest which caused widespread carbon
sequestering or something like that leading to the drop in temperature
and its subsequent effect on European civilization (I won't try to
actually argue that point here but it's intriguing.)

So if you're really expecting something as macro as 40% of the
population dropping dead I think one has to think much bigger and much
more in the realm of unexpected consequences.

As one guess, if 40% of the population dropped dead a more likely
effect than having to continue on with the other 60% of the staff is
that the company would just be unable to deal with the loss of
customers and staff not to mention the services these people are
trying to get to, they're collapsing for the same reasons, a cascade
effect. Most would be closed in short order.

Maybe all of them, kind of like the airlines trying to adjust to
higher fuel costs, many just can't even if the desire to fly (demand)
appears to be sufficient to keep them going the business models just
cease working.

Ok some airlines obviously weathered the change and even prospered but
I hope you get my point that it's way beyond Delta or UA et al just
cutting an appropriate number of flights and staff (which doesn't seem
to have worked), a linear response to a linear problem (higher fuel
costs), and required entire reworking of business models from (ahem!)
the ground up, or dissolution.

Most companies don't go under because they lose a lot of their
revenue, they're often dead due to losing a relatively small amount of
revenue (like 10-15%) due to fixed overheads. For example, do you
think your ISP's landlords are going to let them out of their office
leases just because they have so many fewer staff to seat?
Particularly in the face of a sea of bankruptcies cancelling leases?
Etc.

You'd probably be smarter just going into the casket business or
something like that, grief counseling perhaps.

12

Barry Shein wrote:
[snip]

So if you're really expecting something as macro as 40% of the
population dropping dead I think one has to think much bigger and much
more in the realm of unexpected consequences.

Uhh... I think, I _hope_ that we are talking about 40% of your
workforce NOT SHOWING UP TO THE OFFICE for days or weeks, not
dropping dead, not even necessarily getting sick.

A 40% mortality rate among otherwise healthy adults, and we have much
bigger issues to worry about.

13

Sorry! I should have said that my deadline was early this morning (EST.)

14

A slightly different aggregate: 40% of your workforce being unable to
work.

Some portion of that might be death, grieving, being sick, helping
family or friends that are sick, fighting off zombies, or searching
aimlessly for human brains to consume.

That is to say that some of the remaining 60 may be working from home.

15

As we all find out the hard way that Douglas Adams was right, and telephone
sanitizers really *are* important.....

16

Indeed. Estimates I've read on CNN in the past few days (I know, I
know) say that if H5N1 were to be approximately as virulent and deadly
as the 1918 flu, we would be
looking at 90 million infected and 1.9 million dead in the US.

-Rusty

17

According to the wikipedia's quote of WHO the weighted average
mortality rate, which would be across 50 human cases, is 66% in 2006,
and 56% across all 194 cases reported since 2004.

                  http://en.wikipedia.org/wiki/H5N1

18

According to the wikipedia's quote of WHO the weighted average
mortality rate, which would be across 50 human cases, is 66% in 2006,
and 56% across all 194 cases reported since 2004.

                  Influenza A virus subtype H5N1 - Wikipedia

is this different if you run is-is as opposed to ospf?

as is-is is not over ip, perhaps virii at the ip layer
are less of a worry to larger isps (who mostly run is-is)?

randy

19

IS-IS can carry retroviruses, which are RNA-based.

This discussion did start out with operational content....

    --Steven M. Bellovin, http://www.cs.columbia.edu/~smb

20

So there you have it. They're likely to come to work even though

they're

sick (presuming they don't know it's a lethal virus), where they work

and

spend all their face-to-face time in close quarters with recirculated

air

with the rest of the company's engineers.

That recirculated air is likely to be shared with the
rest of the buildings inhabitants, not just the engineers.

On the other hand, engineers tend to have already
perfected the art of working remotely. Continuity planning
people are likely to notice that skilled technical people
are essential to smooth operations and will kick them out
of the office before anyone gets sick.

--Michael Dillon