2008.02.18 NANOG 42 datacenter efficiencies panel notes

Cranking out the notes quickly before Beer and Gear
time. :smiley:

Apologies for typos, etc, here's the notes from Josh
and Martin's datacenter energy efficiency panel.
the slides went fast, for the content of them, check
the PDF up on the nanog site.


Josh Snowhorn, Energy Efficient Datacenter Panel

He futzes with his Mac for a while, trying to get
Google Earth to work.

Fly around the country, looking at major datacenters
around the world, where they are, how to cool.

1.5M sq ft in 111 8th. hard to get cooling there,
roof is packed.

32 ave of americas, roof is full.

165 Halsey, newark,

S&D NJ facility next to power and railroad with

seacaucus, next to RR with fiber and power lines.

Equinix Ashburn, it is the place to be.

350 Cermac in Chicago, next to RR station
Coolin on the roof, fiber in tracks.

eQuinix in CHE new facility, next to airport, RR
for fiber, power.

Infomart, Dallas, equinix, S&D, next to RR, power,

DRT in Phoenix, ots of overhead, not too far from
there to RR yard.

One wilshire in LAX, equinix has building down the
road, cooling on roof.
EQX second site, near tracks.

Westin Seattle, highrise office, cooling on roof on
garage, big hassle,

365 Main in SF, full
200 Paul, also full (DRT)
cooling on roof, everything aqueezed in, next
to tracks.

PAIX palo alto, near tracks.

3030 corvin expansion, terremark

Luncy Equinix, smart placement, building with a
lot next door; you can expan next door.

Equinix Great oaks, lots of land next door.

Space Park, DRT, fiber plant, railroad tracks
next door,

56 Marietta in Atlanta, TelX,
infrastructure packed on the roof; no power, no

District cooling, overhead, cooled from 7,000
tons of chilled water a few blocks away.

OK, now to datacenter power trends, panelists.

Tier1Research up first.
1MW datacenter takes 177millino kwh over its life.
2008, 50% of today's datecenters won't have sufficient
power and cooling.
Enterprise will outsource and use up datacenter spaces.
Humans created 161 Exabytes of data in 2006

Colo--after dotcom bust, glut of 100watt/sqft space
came on market, DRT, CRG, DupontFabros came on, bought
them, sold them off.
They're now doing metered power
squeezing power margins

Any2 free IX

New datacenters need 200-400w/sqft, which is exponential
growth in density.

Google, Yahoo, MS buying hydro power in remote areas.

Paying $400/rack to $1500/rack as density goes up,
and now utilities are making you pay for power

35 million servers worldwide, $110B spent running

datacenters are running much, much hotter.
20kw per rack, lots of space around them, the power
load can go in, but how do you cool them?

Martin Levy points out there's a wide variety of
resources in the datacenter now; servers of all ages,
disk storage, etc. The number of types of services
is growing dramatically, while network resources
haven't grown.

More cooling means more costs.
2.5 watts in, 1watt to equipment, 1.5w to infrastructure

utility load distribution; half goes to servers, and the
other chunk largely goes to cooling.

PUE = total facility power / IT equipment power

About half of the power is burned before any server is
turned on.

3% goes to lights alone. Turn the blasted lights off!

More power required per system over time.
what goes into datacenter increases as per-server load
increases as well.

What people were claiming they could do 5-7 years ago
was completely unsubstantiated; now, as people are going
in and verifying that power and cooling support what is
being promised.

Some people are doing 300watt with separated hot/cold
aisles; it's definitely becoming more mainstream.

20,000sqft ghetto colo (tier 1, 2) is $15.4M
a tier IV datacenter (2N, )
20,000 sq ft is $48.4M

Bottleneck of the internet is the colo!

Lead time to build a top tier 200w a sq ft colo is
1 year (75K sq ft and up)

drive that lead time is 50+ weeks for gensets and PDUs

US colo, go big--you can put $2B in gear in a datacenter

Google DAlles Oregon 68K pod
MS Qui9nces, 470,000 sq ft, 47MW

Yahoo wenatchee and quincy, 2mil sq ft.

Terremark, 750,000 sq ft, 100MS

NCR, 55 generators, 800watt/sq ft, turbine site.

1 wind turbine, needs thousands of acres, green power
is tough. Wave generation.
Solar tower, seville spain, great, what about night?
Green power is tough.

Flywheels instead of batteries, don't make as many

UTC turbine generator systems
10MW requiers 200micro turbines, 5000 tons of absorbtion
80MW requires 80medium turbines, 40,000 tons of absorbtion

Panel discussion
Michael K Patterson
Subodh Bapat
Martin Levy
John, CRG,
Chris Malayter
Josh Snowhorn,
Mark Bramfitt
Edmund Elizalde

The fine detail is the really tough side; Mark talked
about it from the power company side.
PGE has pushed the power companies

Martin takes it from the server level, the chip level,
and calls Mike from Intel, a senior thermal architect.

In 2002, datacenter with top 500 computing cluster,
in top 20, 128kw of power in 25 racks.
In 2007, showed progress, that same performance in
top 500 list, could be done in a single rack with
less than 40 servers, consuming 21kw in a single rack.

challenge is how do you handle 21kw in a single rack?

progression on moores law, the single chip can do more,
but it draws more too.

the server has been commoditized the equipment; but
as it's been commoditized, the building has been
The room used to be the commodity part; now, our focus
has to shift, the servers can be brought in for low cost,
but the facility is the challenge.
Takes a lot of engineering to get to 21kw/sq ft.
GreenGrid, ClimateSavers, understand what's happening
in both those organizations.

As much as we have high performance computing with
requirements have gone sky-high, what's happening with
the power on chip now that's being produced.
power has been ramping up over time; increasing density,
but with a knee; Intel and AMD quit chasing raw performance,
shifted to chasing performance per watt; was getting too
difficult to cool. CPUs dropping from 135watt CPU to 100watt
CPUs, esp. with multiple cores.
Power consumed with multiple cores is far less than half as
you split work up. among more low frequency cores.

So, let's go up a level, and talk about the system overall

So, Subodh is up next to talk about what Sun is doing in
this space.

Even though it's a "commodity" space, there's still a
lot of technology innovation still happening there; the
commodity pricing/performance hasn't been bottomed out
There's incentives being applied to nudge the commodity
market as well.
Sun was one of the first partnerships between a hardware
manufacturer and a public utility.
Goal is to use less power power rack unit, while achieving
similar performance; use gears that self-calibrate their
power utilization.
If your gear already knows what its demand is, and how
much it needs to draw, then you can size your datacenter
Need intelligent, self-calibration of power draw, and
scale back energy draw based on less utilization.

What about 80plus power supplies, and at the system
level, what can be done--how much power is wasted
by running systems that aren't the more energy
effienct models; power supplies do make a huge
difference; very likely 60-70% efficient older
power supplies, so 500watt out is 1kw in; basically
just a big transformer, fundamentally.
transformers have effiency curve, can be less
efficient at lower power levels.
Sun is aiming for 90plus power supplies, with
smoother, flatter power efficiency curve.

Can replace 7 older servers with 1 newer generation
server, which reduces power and space requirements.

Chris, S&S, John, CRG, Ed, Equinix will talk about

John addresses new datacenter builds.
Milpitas, CA; found a property, medium sized, left
over from Worldcom, DMS200 site, 75watt/sq ft,
electromechanical wouldn't support 200wsft.
Met with PGE, selected Carrier for chilled water,
Caterpillar for generators, city of milpitas,
took 3 days to beat out best way of building the
best datacenter to make sure chilled water would
not pollute the environment, run generators at
tier 2 category at the very least.
They can tell customers they can delivery 200wsft,
will enforce good practices; hold and cold aisles,
enforced blanks, high density server installations.
They've produced cutting edge facility; needs to
be cooperative effort between cities, utilities,
the building owner, and the tenants.
Milpitas is new enough there's been no legacy hardware
going into the building.

Chris, Edmund, what do you do about legacy
customers, and how do you incentivise them?
charge them more; power rates are going up
in most markets. Back to Josh's point, trying
to do environmentally friendly power in
urban environments requires incentives.
Edmund sits in as consulting engineer for equinix,
has dealt with utilities and customers.
Make sure you have as much backup power as you
have main power, so that you can keep your customers
up; make sure you don't overcommit the power available
for your customers.
Track power consumption for customers, make sure that
they're staying within their usage committment.
Free cooling, you still have to pay to run air
handlers, fan, etc.; not entirely free, but it helps.
UPS systems, move to SRD, aim to run your UPS at 75%
for highest efficiency, around 95%, but you can't
design for best efficiency, have to design for worst
power utilization efficiency.
Many types of cooling, with different mechanical
systems; liquid cool, air cool, etc.
Effiency also varies over time, so some days may be
60% and others 80%.

If Josh can provide green power to the datacenter,
10c/kwhr, vs 15c/kwhr, if he sold it, would we pay
1/3 more to get it? Nobody is willing to pay more
for green power.
Austin will charge green power rate, but will
keep it flat rate for 20 years.
half of his power is hydro, not counted as
renewable or green; but has a mandate that he
hit 20% of green power; we can't generate enough
here in California, so we have to buy it from
cheap power in pacific northwest is based on existing,
underutilized assets; once the utilitization goes
back up, there will be rate shocks in store.
Mark from PGE notes that there's problems with SF
datacenters, in that transmission gear will need
to be upgraded.

Opened up to questions.

Anton Kapella, 5nines colo, .3MW--good areal shots,
none of them have solar shields, is black body not
something they need to worry about?
120,000sqft roof flates, 2 60m and 1 11m dish, how
does he put a solar shield up that will withstand
200mph winds?

200,000sqft in sacramento, ragingwire colo, took
top of roof, no HVAC or other equipment, put a
white polymer coat on it, dropped inside temperature
by 20 degrees; payback is 9-10 months.
They had a state standard to meet, but it was a
big improvement still for them--not sexy, but
very good.

High noon, incident sunlight is about 1kw sq meter,
solar panels are about 20%; will get about 150watts
per sq meter; so max on a roof is about 2MW, and
you'd have problems at night.
Probably easier to reflect the energy, rather than
absorbing it and try to use it.

Q: Owen, 0.1MW, bought a prius, it was cheaper than
buying the gas for his LX. So his prius was free,
even without a sticker.

Q: Woody, server form factor question--he has 0MW of
datacenter, apparently. We don't seem to care about
high density anymore the way we did years ago.
If we're trying to make things green, why did we
stop trying to make servers smaller and smaller?
Intel guy responds that they're still working on
making them smaller; 64 servers in 44u on their
newer dual core boxes.
Sun used to fit 16 blades in a 3u box; now we're
back to 1 server per U. it's a cooling issue.
Sun notes that server vendors design for a number
of criteria; availability, reliability, redundancy,
The nature of applications people run has changed;
when you had a single blade 4" high that would fit
in a fedex envelope, you can't fit 16GB of DRAM in
that form factor, or fit an 70-150watt CPU; and
DIMMs put out huge amounts of heat; heat from
memory is starting to dwarf the amount of heat
put out by the CPUs; people are now pushing for
diskful blades with more memory.
Performance per watt seem to be the big metric
The more you enclose into one chassis, the more
efficient you can be about the cooling for it.
Also, look at the number of fans in each chassis,
and how much power used just in cooling.
Could be up to 40watts of power used by fans in
a 1u, and up to 100watts of fan power in a 2u.

Classic 1u or 2u server, dual proc, how much does
the average server suck down? Looking at some
trend data earlier in the week, they're pulling
250-300watts per dual proc server; so 30 watts
is 10% for the fans. 10% is still millions a

Someone from Afilias, small power, .01MW
If you look at the whole system, transmission along,
etc.; can you avoid 30% loss of power transmission,
avoid AC to DC conversions, maybe going nuclear with
direct DC feeds?
When router/switch installs big DC gear, they get
an electrian, it's a bulk feed, well handled.
208V is easier to cool, 120 is harder to cool.
servers with DC are lots of tiny feeds.
homogeneous servers are easier to handle.

Mark notes that he's got less loss, there's about
4% loss in transmission gear; from PNW, it's about
5%; never been 30%!
Lawrence Berkeley did recent study, showed that
you can get 28% improvement by going to DC power
on legacy systems.
Can get 23% improvement by getting high-efficiency
208VAC power, though, so it's more about getting
high efficiency components!

You'll still get a loss with DC plant, there's
still a rectifier from the grid power; it's just
more centralized.

The lower voltage gets higher loss. want 17400
as long as possible, and only go down to 480
when very close to fanout.

Q: Railroads going bankrupt because they thought
they were in railroad business vs transport
business. How do we get greater efficiency, how
do more people handle virtualization better;
people want to be able to shut down unused
resources but be confident that they can come
up as needed.
A later panel will address virtualization more
completely; shed load, shunt usage/shift load,
another very big topic. Some datacenters also
do chilled water reservoir at night.

Q: .25M, not sure who it is, question about moving
heat out out of server off wafer outside; server
designer just wants to get heat away from chip,
they don't think on larger scale, about moving
all the heat in a uniform direction; how do
you make sure you don't mingle cold air with
hot air?
Need server designers to come up with uniform
"cold air in here, hot air goes out there"
model. Some do side-to-side, others go front
to back; need to get HVAC people to work with
server manufacturers to standardize on that.

Q: Gilbert, small network--if pricing goes up
for green power, would we pay for it? When colos
are charging higher than list rate already, it's
more hesitant; but when it's decoupled, it's
harder to justify paying more when it could end
up being a lot more.

Q: Robert Ingram, datacenters being built next to
power industry, and along railroad tracks; but
why not use up the waste heat? It's very low
grade waste heat, though.
BiGas facility, processing waste sugar cane
into power; another person speaks up about
providing power for 30,000 homes from a
landfill as a heat source.

For more info, do a quick Yahoo Search
( http://search.yahoo.com/ )
for the following phrases:
Native Energy
Carbon offset purchases, credits,
BioMass projects.

Gotta wrap up now, few housekeeping issues,
thanks to panelists and companies who let them
come in.

Break time now.

1MW datacenter takes 177millino kwh over its life.

This doesn't seem right

1,000 kw * 24 h * 30 d = 720,000 kw-hrs/mo

177,000,000 / 720,000 = 245 months = 20 years

Lot of datacenters last 20 years?

Very interesting otherwise, I am very sorry I missed it. I'd love to be part
of the next panel.

NANOG should focus more on this, as datacenter dynamics (and all the fun
stuff to go along with it) is much more of an issue than how to put a 10g
wave on dark fiber.

alex@nac.net, 1.8 megawatts in 07927, 0.8 megawatts in 07054

(and we painted our roof white, dropped surface temp from 160 to 105 in a 90
sunny day)