Vulnerbilities of Interconnection

> I suspect its a balancing act between reducancy, survival (network)
> and costs vs revenues.

> not sure I'd call it a "poor job" for not planning all possible
> failure modes, or for not having links in place for them.

It depends on your perspective and what you expect from the net and what
you see it doing for you in the future.

That's a good question. Is the net really a criticial resource? If a life
gets saved through involvement of the internet, it is news. Lifes are
saved by calling for assistence through the telephone network every day
all over the world as a matter of routine.

Try ranking how bad the following outages would be: power, gas (for
heating and cooking), water, the phone network, radio/tv, the net. I think
radio/tv and the net would have to share last place.

As we move more advanced services
to the net, we will also expect much more from it

Sure, if you run telephony over IP, you'll want your IP to be as good as
you need your voice service to be.

also in terms of crisis. Just like the net was one of the prime
sources of information during 9-11.

It's not how much something is used, but how bad it would be to go without
it. During september 11th, the phone service didn't work very well, and
the internet did a lot better. I think just about anyone would have traded
the latter for the former in a second.

In the event of a emergency, I would very much like to be as
able to reach my bank via the net as walking into their offices.

Yes, because banks are such a critical resource when there is an
emergency...

I do agree that it is a balance, but I am not so sure that everyone have
realised this. I am not even sure that all the carriers that you would
expect to have this planning have it...

When your OC3 goes down because nettles have grown into the telco's A/C
exhaust, you start getting cynical... And that was in the good old days
when business was booming. I'm sure they're cutting corners left right and
center at the moment.

That's a good question. Is the net really a criticial resource? If a life
gets saved through involvement of the internet, it is news. Lifes are
saved by calling for assistence through the telephone network every day
all over the world as a matter of routine.

Well, it's more a question of what you want the Internet to become? We can't expect to
have better quality than the most critical application that we run over it....

It's not how much something is used, but how bad it would be to go without
it. During september 11th, the phone service didn't work very well, and
the internet did a lot better. I think just about anyone would have traded
the latter for the former in a second.

I think we are using the word "emergency" with different meanings here. Assume that a nation is cut of from it's
Internet access in one way or the other due to war, natural disasters etc. In these scenarios a not functioning
Internet might actually be a problem due to the time of the outage.

In the event of a emergency, I would very much like to be as
able to reach my bank via the net as walking into their offices.

Yes, because banks are such a critical resource when there is an
emergency...

If there was a natural disaster and I could not reach my banks office I would very much like to
be able to use the on-line bank instead...

that was in the good old days
when business was booming. I'm sure they're cutting corners left right and
center at the moment.

Agreed - and this is where I think we have real problem....

- kurtis -

On the morning of 9/11 I was alone in a colo in Reston VA (3000 miles from home), and I found it very difficult to get information (other than the most basic facts, planes hitting WTC and Pentagon, crashing in PA, airplanes grounded, WTC towers collapsing) on that day. Web servers were overloaded, downloads repeatedly stalled. The big thing the 'net helped with most was sending information (via email and IM) to my friends and family back home in CA, letting them know that I was OK, and to receive information (via email and IM) from those watching TV and learn second hand what they had learned from TV. My cell phone only worked intermittently, due to heavy network congestion on the cell network surrounding DC. When I finally got done at the colo and went back to my friend's house in Vienna later that afternoon, *that* is when I was finally able to learn details about what had happened during the day and see video of the WTC etc. - via TV footage.

When I got back to the office, I learned that the big screen TV that had previously been located in the exercise room had been moved to the center of the office so that everyone could more easily see it, and everyone could hear it. Meanwhile, they all had high speed Internet connections to the computers sitting on their desks. Why bring in the TV if the 'net was "one of the prime sources of information"?

jc

Well, they probably didn't have multicast enabled, and so couldn't
get one of the many news feeds set up that day :slight_smile:

Regards
Marshall Eubanks

JC Dill wrote:

>Just like the net was one of the prime sources of information
>during 9-11.

The internet sucked as a means of getting information on 9/11. I spent
about 20 minutes hitting every news site I could think of, and they had
all tanked. I set an away msg on IM:
"Internet news sucks, I'm going to watch CNN."

I made it to the support center when the first tower fell. It took that
long for someone to come tell me that all this stuff was going on, for me
to give up on the internet and actually make it to a useful news source.

When I finally did go back to my desk to work we turned on a radio that
all of us could hear from our cube farm and tried to resume normal
operations while keeping up to date.

From a network operations perspective, anyone who has not heard William

LeFebvre's "CNN.com: Facing a World Crisis" has missed out. It talks about
how the company that hosts cnn.com handled the crises and how it affected
them from a network perspective. I've been unable to locate any decent
transcripts/recordings of this talk, but I heard it at LISA 2001.
Absolutely amazing presentation if you haven't seen it or heard it.

William said they changed a lot of the way they do things at the company
that hosts CNN.com since 9/11. I don't believe they were the only ones.

Gerald

When I finally did go back to my desk to work we turned on a radio that
all of us could hear from our cube farm and tried to resume normal
operations while keeping up to date.

>From a network operations perspective, anyone who has not heard William
LeFebvre's "CNN.com: Facing a World Crisis" has missed out. It talks about
how the company that hosts cnn.com handled the crises and how it affected
them from a network perspective. I've been unable to locate any decent
transcripts/recordings of this talk, but I heard it at LISA 2001.
Absolutely amazing presentation if you haven't seen it or heard it.

The company that "hosted" CNN demonstrated that for all the claims of their
connectivity, it was not really there. If I recall correctly, CNN came up when a certain company
from MA company-ized CNN.

Alex

> >Just like the net was one of the prime sources of information
> >during 9-11.

The internet sucked as a means of getting information on 9/11. I spent
about 20 minutes hitting every news site I could think of, and they had
all tanked. I set an away msg on IM:
"Internet news sucks, I'm going to watch CNN."

There are several ways why "internet news" wasn't as good as TV news:

1. Using an infrastructure that is built for many-to-many communication
   for few-to-many communication is problematic

2. Look at the budgets for online and TV news

3. This type of situation doesn't lend itself well to typing in the news

What the net did do, was permit people to communicate while the phone
network suffered from massive congestion.

William said they changed a lot of the way they do things at the company
that hosts CNN.com since 9/11. I don't believe they were the only ones.

Can you name a few examples of the things they changed?

I had really hoped Bill, or someone who knew Bill and this talk could give
more input on it. I found a vague summary of the whole talk:
http://www.tcsa.org/lisa2001/cnn.txt

> The internet sucked as a means of getting information on 9/11. I spent
> about 20 minutes hitting every news site I could think of, and they had
> all tanked. I set an away msg on IM:
> "Internet news sucks, I'm going to watch CNN."

3. This type of situation doesn't lend itself well to typing in the news

To answer the comment I've got off list, I was looking for images at the
least of what was going on. These were not small (middle-of-nowhere)
cities. Text on IRC or Usenet were not giving me the visual I was looking
for, and pages like slashdot were vague to begin with.

> William said they changed a lot of the way they do things at the company
> that hosts CNN.com since 9/11. I don't believe they were the only ones.

Can you name a few examples of the things they changed?

From the link above:

Aftermath
- volatility worse than ever before
- automate swing process ([it was] on todo list for a year)
- faster page reduction
- network redesign
- increased WAN bandwidth (if the servers could have handled the load,
  the WAN link would have been saturated)
- Standing phone bridge reservation
- review crisis procedures

Gerald

The news coverage on Sep 11th was unprecedented, I dont believe there is any
similar incident in which the whole of the world (not just a region or
nation) has been focused on watching the news as an event unfolds.

So to be fair (I assume) CNN hadnt asked for a service that could handle that
particular load, if they had then they probably would have not been knocked off
the air.

And now we're a year on, how many news agencies have invested in a service that
can carry 1 million streams or however many they got, I doubt any so if we have
another Sep 11th type event dont expect anything to be different in the unicast
world..

Steve

I had a good experience using the Internet for news on 9/11, because I used
it in a way that fit the model.. I didn't bother trying to load cnn.com or
whatever, but rather.. I sat in IRC, talking to people whom I trust to
various degrees, who were in turn watching every conceiveable news source
available, they transcribed, and summerized, some setup mp3 streams of the
EMS/Police radios from DC and NYC, other people read old news sources
online. People at ground zero went outside and took pictures, setup
webcams, etc..

I have to say that I doubt I missed anything... So sure, the internet
sucks as a 1:1 replacement for TV (at least without multicast)... but so
what? I think my experience was better... I wouldn't have bothered wasting
my time drooling over the TV anyways... welcome to News 2.0.

Date: Mon, 16 Sep 2002 13:15:43 -0400 (EDT)
From: alex

The company that "hosted" CNN demonstrated that for all the
claims of their connectivity, it was not really there. If I
recall correctly, CNN came up when a certain company from MA
company-ized CNN.

Which brings us back to discussions of SPOF and distributed
sources. The MA company in question helps demonstrate the
latter. The former has been discussed recently.

As network engineers/operators, we want to make things better and
more reliable. However, there are sayings to the effect of "the
trouble about doing things right the first time is people do not
appreciate how difficult it was" and "people would rather brag
about a good towing contract than to stay out of the mud".

* How many times have you heard:
  - "I don't care if I get hacked; I don't have anything
    important on my computer"
  - "Nobody would be interested in breaking in to us"
  - "I'm happy with my current provider because they don't go
     down that often"

* Ever submit a proposal for doing a job the right way, then lost
  the bid because someone else was cheaper in the short run (and
  more expensive in the long
  run)?

* Ever had to argue that maintaining RFC compliance really was
  the correct thing to do?

When a problem occurs, people get angry; only then does change
become important. Until then, it's a question of "how much does
it _cost_", *not* "what are we _investing_". Money talks, and a
pound of cure is worth an ounce of prevention, plus carries the
bragging rights of "I was a victim".

How often does a huge news even occur? The bombings a year ago?
The release of the Starr report? Do people get mad at the news
source, or just chalk it up to "the Internet is having troubles"?
Good, fast, cheap -- pick two (and make sure at least one is
"cheap").

Bottom line: When something is expected, and the lack thereof
rarely bites in a painful place, proper implementation is not
considered a valuable feature. The market for perfection is a
very small one.

Eddy

You actually more or less described what I meant, although I wasn't very clear. In principle the Internet was the place where people went for information, which is exactly what you saw - the congestion and overload. My initial point was then, that as this seems to be the case, perhaps we should engineer the network to meet these demands.

- kurtis -

William said they changed a lot of the way they do things at the company
that hosts CNN.com since 9/11. I don't believe they were the only ones.

Which was my point to start with...

- kurtis -