Current trends in capacity planning and oversubscription

Mailman List Mirror (Read Only)
1

While the answer is always it depends, I was wondering what the current rules of thumb university network engineers are using for capacity planning and oversubscription for resnets and admin networks?

For K-12, SETDA (http://www.setda.org/web/guest/2020/broadband) is recommending:

- An external Internet connection to the Internet Service Provider of at least 100 Mbps per 1,000 students/staff
- Internal wide area network connections from the district to each school and between schools of at least 1 Gbps per 1,000 students/staff

How does that compare with university and enterprise network rules of thumb?

2

I can tell you that no one I know in K-12 or Higher Education has the capacity for 1GE per 1000 students. Same goes for the WAN connection.

More than likely *if* they have fiber from every school to a central location it's a single pair and they couldn't afford to go 10GE. Though I have answered some RFPs where they wanted to know what it would cost. (Usually 2.5 * 1GE cost.)

I believe my Alma mater just reached 150Mbps for the 6,500 students + faculty. Sure beats the 30Mbps they started with when I was a Freshmen.
I also just finished an RFP where a campus of at least 4,000 only had 100Meg. This was in a suburban environment with at least a dozen carriers competing.

3

As someone else has said I've never seen K-12 with remotely that high
of a ratio, or, really, any educational institution. UofM here in
Missoula, MT doesn't have anywhere near those ratios for internet
services nor for the campus network. I don't have any exact details
but I'm pretty certain there's no 10 Gig-E there. I'm not even sure
if the building-to-building links are 1 Gig-E in all cases.

Actually...I'm not sure anywhere has that high of a ratio here in the
states, at least for wired connectivity. The carriers here all keep
the prices nice and high to preserve their profit margins in the face
of losing their long distance and traditional POTS cash cows as people
move more to cell phones and other non POTS carriers.

4

> - An external Internet connection to the Internet Service Provider of at
> least 100 Mbps per 1,000 students/staff

30K students here, 2x10GE to the outside world.

> - Internal wide area network connections from the district to each
> school and between schools of at least 1 Gbps per 1,000 students/staff

Not applicable when you're mostly just one campus, unless you want to discuss
per-building feeds. Most of our buildings have 1 or more 1G uplinks to a 10GE
core network, and some larger ones have their own 10G uplink. There's a few outlier
buildings that have a dozen or so people that still have 100M uplinks.

I can tell you that no one I know in K-12 or Higher Education has the
capacity for 1GE per 1000 students. Same goes for the WAN connection.

We're working on deploying 1G to the desktop. Some buildings are going faster
than others - the ones we wired first are now the hardest, because there's a
lot of Cat-3 in them. The later ones that already have Cat-6 will be easier.
With something like 120 buildings to do, it's going to take a while.

5

I am on the technology committee of the college I attended (Whitman) and they currently have a 200 mbit/sec via gigE link for a campus of just under 2000 and every building has at least 1X gigE into their backbone. They are in a rural area (walla walla, wa) but they don't generally have more than 100 or 150 mbit/sec of usage, fitting nicely in the below recommendations.

6

Michael Loftis expunged (mloftis@wgops.com):

Actually...I'm not sure anywhere has that high of a ratio here in the
states, at least for wired connectivity.

I would say that's highly dependent on your geographical location. In Montana I could see that as being true, but not in NYC, for example...

-Steve

7

From: Steve Meuse > Sent: Wednesday, November 10, 2010 9:31 AM
To: Michael Loftis
Cc: nanog
Subject: Re: Current trends in capacity planning and oversubscription

Michael Loftis expunged (mloftis@wgops.com):
>
> Actually...I'm not sure anywhere has that high of a ratio here in

the

> states, at least for wired connectivity.

I would say that's highly dependent on your geographical location. In
Montana I could see that as being true, but not in NYC, for example...

-Steve

I'd say it also depends on the nature of studies at the school, too. A
general liberal arts school might have different needs than one heavy on
science and engineering. Throw in some specialized resource like a
radio telescope or nuclear research lab and it changes even more as
those research data are shared among other collaborators at other
institutions.

8

Michael Loftis expunged (mloftis@wgops.com):

Actually...I'm not sure anywhere has that high of a ratio here in the
states, at least for wired connectivity.

I would say that's highly dependent on your geographical location. In Montana I could see that as being true, but not in NYC, for example...

It might be more dependent upon the level of competition int he
bandwidth market. Here in the mountain west (MT included) you either
pay the ILEC their exorbitant fees for your last mile, or, you pay to
trench. Only a couple years ago Qwest was quoting $8k+/mo for a DS3,
and this was where their Cisco ONS 15454 was in the same room. I
don't even want to know how much they'd charge you if you had to pay
any real line mileage. Luckily int hat particular building there
were/are other options, but almost any other place, you don't (that
building has a lot of antennae on top, a couple placed by myself)

9

Sometimes it is a hard sell, but the factor most overlooked when
designing high speed networks is that of designing for low latency.
Bandwidth and over/under subscription are only part of the network
design. Low latency networks (regional RTTs of 1-5 milliseconds; campus
RTTs in the sub millisecond range for GiGE, and microsecond range for 10
GiGE) by their nature solve a lot of QoS problems, often relegating QoS
as a method to be used in emergency cases only, such as DoS attacks.

Here is something I have been thinking is not too far over the horizon:
commodity 48 port x 10/100 GiGE switches; 10 GiG NICs on the desktop;
commodity-priced 10,40, or 100 GiGE WAN links; while user bandwidth
requirements rise at a much slower rate. Some switch vendors even today
have 48 port 10 GiGE switches in the $25K range.

My rule of thumb is that 1 GiGE link is an order of magnitude lower
latency than a 100 Mbps link, and a 10 GiGE link is an order of
magnitude lower latency than a GiGE link.

10

It also depends on how much file-sharing the students are doing...

    --Steve Bellovin, http://www.cs.columbia.edu/~smb

11

Page 10 of the presentation on the link you provided says those are the numbers they recommend "for the next 5-7 years". Perhaps they meant to say "in 5 to 7 years"?

  "for the next 2-3 years" they recommend 10 Mbps per 1,000 people to an ISP and 100 Mbps per 1,000 people between schools.

  Our campus is already using almost 3 times the ISP bandwidth recommended "for the next 2-3 years".

  If we take our current ISP bandwidth and increase it by 50% every year for 5 years it would be about twice the 100 Mbps per 1,000 students/staff recommendation.

12

Is 50% growth each year typical these days? In the dot-com boom days, people said 100% growth, other people have suggested 20% may be more reasonable now. A problem with government network capacity planning/growth forecasts is you will be stuck with whatever you choose, too high or too low, for many years because the budget cycle is so long.

It would be great if there was some actual data available. But it seems
more typical to benchmark/compare to do network capacity planning with other government agencies, so we end up with X-Mbps per Y,000 people.
Yes, I know it depends. 1,000 people downloading data from LHC experiments will be different from an administrative school office. The difference is the people using LHC data usually have someone who can figure out network capacity planning, while the people in an administrative school office may not have anyone.

So what is a reasonable network capacity for 1,000 students now and in 5 years.

13

Just as LHC people and a school are different, I'm willing to bet that bandwidth
"requirements" per student are different based on the school and its policies,
and that they're to a large extent self-fullfilling.

The "requirement" at a small liberal arts school with a fascist network usage
policy "we block all bittorrent and any protocols we don't understand (i.e.
most of them), and no network access in the dorms", will be different from a
large engineering school that says "We'll provide bandwidth so you can explore,
experiment, and learn, and we'll let you know if you're named in a copyright
complaint".

So "reasonable" bandwidth ends up depending on what the network admin thinks
"reasonable" use of the network is...

14

So what is a reasonable network capacity for 1,000 students now and in 5
years.

Just as LHC people and a school are different, I'm willing to bet that bandwidth
"requirements" per student are different based on the school and its policies,
and that they're to a large extent self-fullfilling.

There are "a number" of factors at play, but looking at a given snapshot
of network activity, some of the relevant metrics are the number of
active hosts [count inside clients if you are a content "consumer",
count outside clients of you are a content "provider"], the number of
active connections, and compare that to the bandwidth consumption. It
doesn't really matter how many students you have, it's how many are
online and active at a given time.

The "requirement" at a small liberal arts school with a fascist network usage
policy "we block all bittorrent and any protocols we don't understand (i.e.
most of them), and no network access in the dorms", will be different from a
large engineering school that says "We'll provide bandwidth so you can explore,
experiment, and learn, and we'll let you know if you're named in a copyright
complaint".

That is true to a degree in terms of your degree of network neutrality.
We do block most P2P, so I am not so certain of the bandwidth
consumption it would take these days if left open. I do, however, know
that other traffic has simply risen to take it's place. Well over half
of our bandwidth (less P2P) is streaming media, and a growing chunk of
that is Netflix. I could take 50Mbps and divide it among a thousand web
surfers, or a hundred Youtube viewers, or a few dozen Netflix folks, or
one of our Athletics Dragonfly video servers.

You can size it for a given population and a given application mix, then
be hit by the Obama inauguration, the Michael Jackson funeral, the World
Cup playoffs, or just a good healthy Patch Tuesday or Apple MacOS update.

So "reasonable" bandwidth ends up depending on what the network admin thinks
"reasonable" use of the network is...

And then there are limits to your ability to predict the "reasonable"
demands :slight_smile: Which is where things like 95th percentile numbers start to
come in handy.

Another interesting metric is looking at "per user" consumption over
time. Just as the applications have certain fringe cases that consume
an inordinate of bandwidth, you also have a few fringe case uses that
will account for an inordinate amount of bandwidth. If your goal is to
accomodate these folks equally with the rest of the population, your
bandwidth requirements will be MUCH higher to accomodate them.

So with all that said... for what it's worth... for the last 24hours,
our (sustained) inbound bandwidth (peak/avg) was 317Mbps / 107Mbps,
inside concurrently active host counts 2017/986, concurrent connection
counts 22.6K/10.8K. That is with 11K students, 1500 fac/staff, and the
numbers being considerably higher during the week. The 95th percentile
number is ~250Mbps for a typical weekday (24h), but we do "cap" dorm
traffic during classroom hours.

To backup another point brought up in this thread, the demand has
roughly doubled annually the last 3 years.

Jeff

15

I've been tracking Internet bandwidth usage for several schools (RESNET-L
reports), and I've seen it as low at 2.1 kbps/FTE (5+ years ago) to higher
than 300 kbps/FTE. Most of the schools are between 30 to 60 kbps/FTE at
this time. Very broadly speaking, rural and smaller schools are on the low
end, while state schools with tens of thousands of students are on the
higher end. Internet2 plays in the mix as well.

In terms of growth rates, while our ISP serves only one college, even though
they've been growing their bandwidth pipe at low-double digits per year,
they are maxed out most days from mid-day to 1 am. If most ISPs see
end-user traffic grow 50 to 80% per year, I can't see why schools would be
much out of that range.

Frank

16

If we take our current ISP bandwidth and increase it by 50% every
year for 5 years it would be about twice the 100 Mbps per 1,000
students/staff recommendation.

Is 50% growth each year typical these days? In the dot-com boom days,
people said 100% growth, other people have suggested 20% may be more
reasonable now.

  We did see a lower rate of growth after the dot-com boom/bust.

  However the rate of growth picked up with the popularity of video streaming sites.

  This site mentions 40 to 50% growth last year and has references to other papers that mention similar growth rates (although some of those papers may now be several years old.)

http://www.dtc.umn.edu/mints/home.php

  So to answer the question I would say that 40 to 50% growth is typical these days, it has been for us.

  I assume that it will continue for a few years but I'm less confidant speculating that it would still be 40 to 50% in 5 to 7 years. But I wouldn't bet against it either.