WSJ: Big tech firms seeking power

Since power consumption was a topic at the last NANOG meeting.

subscription required, or buy a copy of the Wall Street Journal from
a newstand

http://online.wsj.com/article/SB115016534015978590.html
Surge in Internet Use, Energy Costs
Has Big Tech Firms Seeking Power
By KEVIN J. DELANEY and REBECCA SMITH
Wall Street Journal
June 13, 2006; Page A1

With both Internet services and power costs soaring, big technology
companies are scouring the nation to secure enough of the cheap
electricity that is vital to their growth.

The search is being led by companies including Microsoft Corp., Yahoo Inc.
and IAC/InterActiveCorp. Big Internet firms have been adding thousands of
computer servers to data centers to handle heavy customer use of their
services, including ambitious new offerings such as online video.
[...]

And, just to be fair, Google gets their own bit of news on the power
front:

http://www.iht.com/articles/2006/06/13/business/search.php

I wonder just how much power it takes to cool 450,000 servers.

Matt

I wonder just how much power it takes to cool 450,000 servers.

I've heard mumbles that the per kWh rates from Bonneville in the locations along the Columbia are in the sub-4¢ range.

Grant county is seeing a huge fiber building boom as a result. It will be more wired up than King county soon. Woody was here last night and remarked (feel free to correct me if I misquote you Bill) that it was funny that nowadays "network geeks were more interested in kilowatts than kilobits"

--chuck (in Seattle)

450,000 * 100 WT (power itself)

Cooling - I donot know, but I should estimate it as extra 70% of consumed
power.

So,

450,000 * 0.2KWT = 90,000KWT.

I wonder just how much power it takes to cool 450,000 servers.

450,000 servers * 100 Watts/Server = 45,000,000 watts / 3.413 watts/BTU = 13.1 Million BTU / 12000 BTU/Ton = 1100 Tons of cooling

A 30 Ton Liebert system runs about 80 amps @ 480 volts or 38400 watts, you'll need at least 40 or them to cool 1100 tons which is 1536 Kw * 24 hours * 7 days * 4.3 weeks = 1,110,000 KwH/month * $0.10/KwH = $111,000 /month in cooling.

I think my math is right on this...

I wonder just how much power it takes to cool 450,000 servers.

450,000 servers * 100 Watts/Server = 45,000,000 watts / 3.413 watts/BTU = 13.1 Million BTU / 12000 BTU/Ton = 1100 Tons of cooling

Error: you MULTIPLY 3.413 to go from watts to BTU, not divide. It's be more like 154,000,000 BTU, /12000 or 12,798 tons.

Also at 100 watts, you are assuming Celerons with single hard drives. We see more like 120 to 240 depending on config. 100 would be low.

A 30 Ton Liebert system runs about 80 amps @ 480 volts or 38400 watts, you'll need at least 40 or them to cool 1100 tons which is 1536 Kw * 24 hours * 7 days * 4.3 weeks = 1,110,000 KwH/month * $0.10/KwH = $111,000 /month in cooling.

80 amps @ 480 is 80 * 480 * 1.73, or 66 kw. However, they don't draw that much. A 30 ton unit, worst case (115 degrees outside across the condensor) will be about 50 kw, assuming you do not have humidification or reheats turned on.

Second issue: you are assuming 100% cooling efficiency, or, in other words, that you'd have perfect airflow, perfect air return, etc. Never happens, especially when you have customers who are idiots.

Third issue: you are assuming there is no heat loss or gain in the structure of the building. This could be very significant. Let's assume it's not.

It's likely in an environment like this, you'd have more like 14000 tons. 14000 / 30 = 466 units, @ 50 kw/unit, 23,300,000 watts, / 1000 * 24 * 30.4375 (avg days in a month) = 17,020,000 kw-hrs, @ $0.12 (more likely with todays fuel prices unless you are in Kentucky) $2,042,400/month.

Also, don't forget the original 450,000 servers at 100 watts (45 mw) would be $3,944,700/month in power. Also, 450,000 1U servers at 40/rack would be 11,250 racks, which at 10 sq-ft a rack would be 112,000 sq-ft of datacenter floor space (triple or, more likely, quadruple that for space for HVAC, generators, switchgear, UPSs, etc). That'd be 500,000 sq-ft at minimum.

Total is $5,987,000/mon, but you haven't ROIed the millions in electrical gear (think big: this is about 68 megawatts; $250k/each for a 2 mw generator (you'd need 40, $10 mm), $50k/each for a 500 kva UPS (you'd need 80 $4mm), millions in panels, breakers, piping, copper wire (700% increase in copper pricing in the last 24 months, people), etc. Oh, and 466 liebert 30 ton HVAC's, probably $25 to $40k/ea installed ($11 million). Oh, and no one has installed it yet, and you haven't paid rent on the facility that will take 2 years to build with probably 100's of workers saleries.

Take $6mm/month, divide by 450,000 servers, $13.33/month/server.

Oh, and 68 Megawatts over 112k ft of floor space is 607 watts/ft. Thats about 6 times what most centers built in the last couple years are built at.

But wait, there is more. Just a point of comparison -- Oyster Creek Nuclear Power generation plant, located here on the Jersey Shore, produces 636 megawatts. You'd take one-tenth of that capacity -- in a bulding that would sit on a 10 or 20 acre chunk of land. I put this into the 'unlikely' category. The substation alone to handle stepping 68 mwatts from transmission to 480v would be probably 4 acres. And, 68 megawatts of power at 480 volts 81,888 amps. A typicall 200,000 sq-ft multi-tenant office building has 1600 amps of service; this would be the equivalent of 50 buildings.

Having fun yet?

A 30 ton liebert takes about 30 sq-ft of floor space; 466 of them would be 13,980 sq-ft. If you use a drycooler system, they are about 100 sq-ft, and youd need 233 of them (60 ton DDNT940's), 23,300 sq-ft of roof space. Each of those weighs 2,640 pounds, for a total of 615,000 pounds, or 308 tons (of weight, not HVAC capacity). I won't even spend the CPU cycles figuring out how many gallons of glycol this would bem but probably a good guess would be about 50,000 gallons. That'd be about a quarter-million dollars in glycol.

I'm tired now, time to climb back in my hole. In other words, don't get me started on the datacenter density issue.

I happen to know that a very large power line project was just finished in that area :slight_smile: (I have family that works for the company that did the job). It's a huge amount of power that's for sure. I'm not sure what the exact route was, nor the endpoint right now, but when I did ask him at the time it didn't make sense....Now it might. I'll talk to him again.

I wonder just how much power it takes to cool 450,000 servers.

450,000 servers * 100 Watts/Server = 45,000,000 watts / 3.413

watts/BTU =

13.1 Million BTU / 12000 BTU/Ton = 1100 Tons of cooling

Error: you MULTIPLY 3.413 to go from watts to BTU, not divide. It's

be

more like 154,000,000 BTU, /12000 or 12,798 tons.

Well, the bigger problem here is that a watt is a measure of
power (engergy/time) and a BTU is a unit of energy. There is no
dimensionless conversion factor between the two.

Huh?

A Watt has no time constant. A watt is an amount of energy consumed at a moment (ie, a 60 watt light bulb), not an amount of energy over time (like a watt-hour; for instance, a 60 watt light bulb uses 60 watt-hours of power every hour, or 1.44 kwatt-hrs per day).

There is a direct correlation between watts and btu's, and that is:

   watts * 3.413 = btu

Alex Rubenstein wrote:

Huh?

A Watt has no time constant. A watt is an amount of energy consumed at a moment (ie, a 60 watt light bulb), not an amount of energy over time (like a watt-hour; for instance, a 60 watt light bulb uses 60 watt-hours of power every hour, or 1.44 kwatt-hrs per day).

There is a direct correlation between watts and btu's, and that is:

    watts * 3.413 = btu

You're confusing Watts and joules. One Watt is one joule of energy per second.

Since you like Wikipedia so much, why don't you look it up:
  http://en.wikipedia.org/wiki/Watt

Watt is not amount of power but amount of power produced during time, i.e.
its speed of energy consumption.

However kwatt-hour (I've never heard of watt-hour, but I suppose that
maybe used too..) is actually amount of energy consumed - more precisely
X kwr its how much energy device would consume if it were consuming energy at exactly the same speed of X kw for entire hour.

Once upon a time, Alex Rubenstein <alex@nac.net> said:

There is a direct correlation between watts and btu's, and that is:

  watts * 3.413 = btu

No, that's wrong.

$ units
2438 units, 71 prefixes, 32 nonlinear units

You have: watt
You want: btu
conformability error
        1 kg m^2 / s^3
        1055.0559 kg m^2 / s^2
You have: watt hour
You want: btu
        * 3.4121416
        / 0.29307107

No, that's wrong.

$ units
2438 units, 71 prefixes, 32 nonlinear units

You have: watt
You want: btu
conformability error
       1 kg m^2 / s^3
       1055.0559 kg m^2 / s^2
You have: watt hour
You want: btu
       * 3.4121416
       / 0.29307107

Agreed, my math should have said "btu/hr", which is what any HVAC system is rated in -- how many btus in an hour it can remove.

I apologize for the horrendous error, but all of the math stands.

Just sed s/btu/btu\/hr/g

(also, you can do from watt to btu/hr with the same 3.413 multiplier)

Oh lord.

a Watt is equal to one joule of energy per second. Period. a watt/hour is equal to, oh, lets see, 3600 joules consumed in that hour.

1 joule is oh, 0.00094781712 btu according to one chart, .00094845 on another.

the math is really straight forward here guys...

Error: you MULTIPLY 3.413 to go from watts to BTU, not divide. It's be
more like 154,000,000 BTU, /12000 or 12,798 tons.

Well, the bigger problem here is that a watt is a measure of
power (engergy/time) and a BTU is a unit of energy. There is no
dimensionless conversion factor between the two.

Alright, I am sorry I missed that. It should read:

Error: you MULTIPLY 3.413 to go from watts to BTU/hr, not divide. It's be more like 154,000,000 BTU/hr, /12000 or 12,798 tons.

Sorry! Sheesh.

This article talks about power and costs:

http://www.post-gazette.com/pg/06164/697875-96.stm

Interesting the power today is being used for cold storage and Aluminum plants because it is so cheap.

Christian

Actually, that's the definition of power. (Energy/time)

http://en.wikipedia.org/wiki/Electric_power

A kilowatt-hour is equivalent to 3600000 joules.

--Adam

I used very raw estimation (which is not well correct but dont make too much
of errors) - to remive 1 KW out of building, yiou spend extra 1 KW.

But anyway, 450,000 servers have a great power consumption - you can use
river or a lake to cool them, but you still need 45,000 KW of power to make
them work. So, to say 60,000KW - 100,000KW will nopt be a big mistake (total
power consumption).

Of course, if you build data center inside the the power plant dam, then you
have both, enough cooling and enough power. -:slight_smile:

Seen few data centers:
- biggerst cages are about 500 servers, may be you can pack 1,000 servers;
- ok, how many cagses in the medium size building? I''d say, 100 - 200 (may
be less).

So, 1 building can handle 50,000 - 100,000 servers. A very big building, I
guess, can handle 450,000. But what for?
You can put it al together, but how you deliver input data and ship output
data from 450,000 servers?