If your policy is not to root out every last one, then you need to
beef up your network so a single compromised host doesn't bring down the
whole network. The Internet is evidence that a network can continue to
operate even with a very large number of compromised machines on a daily
basis. On the other hand, if a single user downloading a music file on
your network can take your entire network off the air for several
days, you may have a problem.
I've often tried to explain that ISPs generally view worms as a "capacity
planning" issue. Worms change the "eco-system" of the Internet and ISPs
have to adapt. But ISPs generally can't "fix" the end-users or their
computers.
System admins were able to completely eradicate the Morris worm. But
most modern worms like Nimda, Code Red I/II, Slammer stick around.
Sometimes a new worm like Nachi supplants an older worm like Blaster.
Even if the ISP tries to be the great network firewall, we have mobile
computers with mobile code. Laptops are too common, connecting to
multiple networks.
[Sorry for responding to old mail, but I'm catching up]
I've often tried to explain that ISPs generally view worms as a "capacity
planning" issue. Worms change the "eco-system" of the Internet and ISPs
have to adapt. But ISPs generally can't "fix" the end-users or their
computers.
I'm curious to know if doing this is at all well understood?
Those of us doing research on worm spread, I don't think have a completely clear understanding of the interaction of Internet bandwidth and worm spread. Slammer, we are pretty clear became bandwidth limited (the rate of spread slowed down dramatically about 40 seconds into the spread). But we don't really know where those chokepoints live (at the edge, or in the middle).
It would seem for the Internet to reliably resist bandwidth attacks from future worms, it has to be, roughly "bigger in the middle than at the edges". If this is the case, then the worm can choke edges at the sites it infects, but the rest of the net can still function. If it's bigger at the edges than in the middle, you'd expect a big enough worm would be able to choke the core. For a given ISP, you'd want capacity to the upstream to be bigger than the capacity to downstream customers. (It would seem like this would be the reverse of what economics would tend to suggest).
Do we really know much about the capacity of the Internet to carry worm traffic? (We believe Slammer used a peak bandwidth of roughly 200 Gbps).
Stuart.
Stuart Staniford, President Tel: 707-840-9611 x 15
Silicon Defense - Worm Containment - http://www.silicondefense.com/
The Worm/Worm Containment FAQ: http://www.networm.org/faq/
Stuart Staniford writes:
It would seem for the Internet to reliably resist bandwidth attacks
from future worms, it has to be, roughly "bigger in the middle than at
the edges". If this is the case, then the worm can choke edges at the
sites it infects, but the rest of the net can still function. If it's
bigger at the edges than in the middle, you'd expect a big enough worm
would be able to choke the core. For a given ISP, you'd want capacity
to the upstream to be bigger than the capacity to downstream customers.
(It would seem like this would be the reverse of what economics would
tend to suggest).
So, essentially, you are saying that the edges (customers, presumably)
need to be bandwidth-limited to protect the core? This tends to happen
anyway due to statistical multiplexing, but is usually not what the
customers would want if they considered the question, and is not what
ISPs want if they bill by the bit.
Do we really know much about the capacity of the Internet to carry worm
traffic? (We believe Slammer used a peak bandwidth of roughly 200
Gbps).
I suspect that in the end the main backbone constaint will be peering
links, for larger ISPs.
So, essentially, you are saying that the edges (customers, presumably)
need to be bandwidth-limited to protect the core?
I wasn't advocating a solution, just observing the way things would have to be for worms to be purely a "buy a bigger box" problem (as I think Sean was suggesting if I didn't misunderstand him).
This tends to happen
anyway due to statistical multiplexing, but is usually not what the
customers would want if they considered the question, and is not what
ISPs want if they bill by the bit.
It would generally seem that ISPs would provide more downstream capacity than upstream, since this saves money and normally not all the downstream customers will use all their bandwidth at the same time. But a big worm could well break that last assumption.
So it would seem that worms are, at a minimum, not a simple or unproblematic capacity management problem.
Stuart.
Stuart Staniford, President Tel: 707-840-9611 x 15
Silicon Defense - Worm Containment - http://www.silicondefense.com/
The Worm/Worm Containment FAQ: http://www.networm.org/faq/
Stuart Staniford writes:
I wasn't advocating a solution, just observing the way things would
have to be for worms to be purely a "buy a bigger box" problem (as I
think Sean was suggesting if I didn't misunderstand him).
Ah.
It would generally seem that ISPs would provide more downstream
capacity than upstream, since this saves money and normally not all the
downstream customers will use all their bandwidth at the same time.
Right; statistical multiplexing.
But a big worm could well break that last assumption.
Yes, as could a number of events, but the response to a worm would
probably be different from the latest streaming video event, or
whatever.
So it would seem that worms are, at a minimum, not a simple or
unproblematic capacity management problem.
Well, it would seem reasonable for an ISP to minimize a worm's effect
on its non-worm customer traffic, and that might mean increasing
capacity in some places, but I don't think the goal would be to move
more worm traffic, but rather to reduce impact to other
traffic. Presumably such activity would be combined with other
anti-worm efforts.
Things are rarely as simple as they appear. Even buying a military
grade black box may not solve the worm problem.
There are some natural choke points in the Internet between ISPs and
customers. The customer may have a 1000 Mbps GigE LAN and the ISP may
have an OC192 backbone, but the link between them is normally much
smaller. Slammer, Blaster, etc had very little impact on the major ISP
backbones, but did severaly congest some of the smaller choke points. Go
ahead and ask UUNET, Sprint, AT&T, etc. what impact the worms had their
networks.
ISPs don't have (much) control over third-party computers. But they can
control their network capacity. Of course, its not a complete solution.
If you are a mid-level ISP, you may have a choke point to your customer
but are vulnerable from your upstream provider. A better designed worm
could impact even major backbones.
So you believe that the edges of the net are smaller, bandwidth-wise, than the core? So the (approximate) picture you would advocate would be that Slammer was rate limited at the customer/ISP interface? (I agree this is consistent with the fact that the tier-1s stayed up during Slammer).
(I'm not trying to be difficult here - I'm just trying to figure out if we actually have any good understanding of this issue - and therefore any ability to predict what future worms might do to the Internet).
(Blaster was not bandwidth limited so that's a whole different animal - it seems to have been limited by a slow scanning rate, and a poor transmission probability).
Stuart.
Stuart Staniford, President Tel: 707-840-9611 x 15
Silicon Defense - Worm Containment - http://www.silicondefense.com/
The Worm/Worm Containment FAQ: http://www.networm.org/faq/
Hi, Stuart.
] So you believe that the edges of the net are smaller, bandwidth-wise,
] than the core?
This was certainly the case in my previous life at a large hosting
provider. We had GigE LANs, used providers with OC192 backbones,
but had only OC3 to OC12 links to our providers. Like most edge
networks, we had CIRs on those uplinks that were considerably
lower than the pipe size. A full OC12 turned out, at the time, to
be darn expensive. 
Our choke points were always our peering or transit links. This
was the case for our (large) enterprise customers as well.
Thanks,
Rob.
Some people refer to it as the hourglass effect, but it has more than one
bump. Generally only the smallest bottleneck controls the congestion.
But worms and DDOS (but not DOS) violate some of the assumptions.
lower bandwidth<---->higher bandwidth
Local Area Network (LAN)
Campus Area Network
Customer to ISP uplink
ISP POP to Backbone
ISP Intra-Backbone
ISP to ISP transit/peer (same continent)
Intercontinental circuits
Of course, there are some exceptions like a customer with an OC192 uplink
or an ISP running a web hosting center on a ISDN link.
Hi, Sean.
] lower bandwidth<---->higher bandwidth
Great ASCII chart.
] Of course, there are some exceptions like a customer with an OC192 uplink
] or an ISP running a web hosting center on a ISDN link.
Another bit to consider is address space. Code Red discovered
a lot of folks with very small pipes (circa T1) and very large
netblocks (circa /16). These folks paid a heavy price when
hit with the "scan all IPs in the netblock" worms.
Thanks,
Rob.
Sean Donelan wrote:
ISPs don't have (much) control over third-party computers. But they can
control their network capacity. Of course, its not a complete solution.
If you are a mid-level ISP, you may have a choke point to your customer
but are vulnerable from your upstream provider. A better designed worm
could impact even major backbones.
If you are an access provider, specially in the consumer space, you can do many things
to help the "Greater Internet" by keeping your own back yard in good shape.
In the transit business, you are expected to deliver the bits regardless of the content
so there the only viable option is to drop packets where the source or destination
addresses don�t make sense.
Pete
What is the difference between a transit provider and an access provider,
specially in the consumer space? Why is a transit provider expected to
deliver the bits, but the access provider isn't? Since the bulk of
Internet access is actually provided by wholesale providers (e.g.
AOL/Earthlink buy wholesale modem access from UUNET/Level3), who is
the access provider and who is the transit provider?
And how do you handle the situation where a provider is both? UUNet, for example, sells LOTS of T-1 lines to non-ISP businesses, and sells retail dialup services to consumers. Sure they also sell wholesale bandwidth and wholesale dialup services to ISPs, but it's not their whole business.
The problem isn't "someone else's problem" for anyone.
Clearly you can apply checks to parts of your network (stub bits) such as your
dialup pool etc where can be sure about what src/dst addresses are fake, ie the
access part of your business
Steve
Sean Donelan wrote:
What is the difference between a transit provider and an access provider,
specially in the consumer space? Why is a transit provider expected to
deliver the bits, but the access provider isn't? Since the bulk of
Internet access is actually provided by wholesale providers (e.g.
AOL/Earthlink buy wholesale modem access from UUNET/Level3), who is
the access provider and who is the transit provider?
Both of these identities can and do exist within the same ISP. Their transit product would
not include active mitigation, notification and filtering but the access provider part would.
Pete