Install an air conditioner in your rack.
Hi David,
The thing I'm not understanding in this thread is that the last time I
checked Level 3 was a premium player not a cost player. Has that changed?
If a premium data center vendor is asking you to swallow 80F in the cold
aisle, something is very wrong. But realize I just said 80F in the *cold
aisle*. DC cooling is not about "ambient" or "sensible cooling" or similar
terms bandied about by ordinary HVAC professionals. In a data center, air
doesn't really stack up anywhere. It flows.
If you haven't physically checked your racks, it's time to do that. There
are lots of reasons for high temps in the cabinet which aren't the DC's
fault.
Is all the air flow in your cabinet correctly moving from the cold aisle to
the hot aisle? Even those side-venting Cisco switches? You're sure? If
you're looping air inside the cabinet, that's your fault.
Have you or your rack neighbors exceeded the heat density that the DC's
HVAC system supports? If you have, the air in the hot aisle may be looping
over the top of the cabinets and back in to your servers. You can't
necessarily fill a cabinet with equipment. When you reach the allowable
heat density, you have to start filling the next cabinet. I've seen DC
cabinets left half empty for exactly this reason.
Regards,
Bill Herrin
If you are using hot/cold aisles and don't fill the rack, don't forget you have to put in blank panels.
back in the arly 1990s, Tandem had a computer called "Cyclone". (these
were mission critical, fault tolerant machines).
The reason for "Cyclone" name was that the cabinets had huge fan
capacity, and that was to deal with air conditioning failure by
increasing the air flow over the electronics to still keep then "comfy"
despite high data centre air temperature. (with the aim of having the
Tandem continue to run despite HVAC failure).
With dense computers packed in 1U, you just can't have that excessive
airflow to cope with HVAC failure with tiny 1" fans.
The other difference is data centre density. Bank computer rooms were
sparse compared to today's densely packed racks. So lots of space
relative to heat sources.
The equivalent today would be the football field size data centres from
the likes of Google with high ceilings and hot air from one area with
failed HVAC to rise to ceiling and partly be taken out by the others.
But when you are talking about downdown co-lo with enclosed suites that
are packed to the brim, failure of HVAC results in quick temp increases
because the heat has nowhere to spread to, and HVACs from adjoining also
enclosed suites can't provide help.
So when a tennant agrees to rent rack space in an small enclosed suite,
it should be considerewd that the odds of failure due to heat are
greater (and perhaps consider renting rack space in different suites to
provide some redundancy).
ok old fart stories...tho maybe current.
IBM's big mainframes would repeat calculations as a way to detect
hardware errors.
Above a certain temperature they would do more repeating.
If there was any disagreement it would be reported and they had some
complex statistical formula to determine how many repetitions to try
next and what to accept.
I assume this was analogous to the various time sync game theoretic
formulas to decide which time reference to believe when they
conflict. It's not as simple as majority vote, the majority could be
wrong (e.g., same stuck bit.)
So, at least as it was explained to me, as it got warmer (e.g., A/C
failure) the machine would get slower and slower, potentially to a
crawl.
And there was no doubt a point at which it'd just shut itself off, but
before it got there. Since many mainframes were mission critical they
were trying to avoid that.
That was the kind of thing which made multi-million dollar mainframes
cost multi-millions of dollars.
Also, the IBM 3090 at least, was cooled via helium-filled pipes kind
of like today's liquid cooled systems. It was full of plumbing. If you
opened it up some chips were right on copper junction boxes (maybe
they were just sensors but it looked cool.)
There was always something amusing back then when an IBM service
person would show up with one of those typical gas tanks on wheels,
like one uses for welding, to top off your mainframe.
Once upon a time, bzs@theworld.com <bzs@theworld.com> said:
Also, the IBM 3090 at least, was cooled via helium-filled pipes kind
of like today's liquid cooled systems. It was full of plumbing. If you
opened it up some chips were right on copper junction boxes (maybe
they were just sensors but it looked cool.)
Cray supercomputers had Freon lines through them for cooling, up until
the last generation of the "old school" supercomputer. That was not
sufficient to keep it cool, so they sealed the chassis (which was huge)
and pumped it full of 4 tons of Fluorinert.
The IBM 308x and 309x series mainframes were water cooled. They did have Thermal Conduction Modules which had a helium-filled metal cap, which contains one piston per chip; the piston presses against the back of each chip to provide a heat conduction path from the chip to the cap. The cap was connected to the chilled water supply.
The bank I worked for had just installed one. A big change were noise
levels, the thing was really quiet. But servicing now required a plumber
too. (there was a separate cabinet for the water pumps as I recall.)
But in all cases, the issue is how long you can survive when your "heat
dump" is not available. If nobody is removing heat from your water loop
it will eventually fail too.
In the end, it is a lot easier to provide redundancy for HVAC in one
large room than splitting the DC into small suites that each have their
1 unit. Redundancy there would require 2 units per suite. And the
problem with having AC units that are capable of twice the load (in case
other one fails) is that it increases the on-off cycles and thus reduces
lifetime (increases likelyhood of failure).
Hi David,
80F seems ~reasonable to me. What is the inlet temp, the temperature air is
going in at? What kind of gear are operating? Routers and switches?
Servers? Disk? Is the cabinet top fan working? Most modern equipment should
be able to handle those temps. As another poster noted, are these triggers
modifiable (or have they been)? I would refer to the manufactures
guidelines. You haven't given us enough information to help. You can refer
(them) to ASHRAE standards in your conversation. I'd be surprised if they
weren't already well aware of it and practicing most of what it preaches.
They may operate safely outside of some norms.
15F-20F cooler? You might be paying too much for colo if that's true.
Best,
-M<
The separate box was a heat exchanger. In the "old" days, buildings had central systems that provided chilled water. Its similar to your house HVAC where an outside unit cools Freon and you have a heat exchanger that cools the inside air. In the case of the water cooled mainframe, the same chilled water was connected to the exchanger and not directly to the computer. The water running through the computer was a closed system.
Thanks for all the opinions and experiences with this on this on and off list. The facility in question is not one that has a cold/hot row or containment concept so ambient temp plays a greater role than in other facilities. Some folks from Level 3 reached out and are working to help me with the situation, so hopefully things are headed in the right direction now.
David
Funny how NANOG posts seem to precede actual attention from vendors isn't it.
Steven Naslund
Chicago IL
Squeaky wheel, grease. Same reason why it takes me berating companies on Twitter publicly before things actually get done.
*Stares directly at Verizon for a previous incident where rejection message from an e-mail block said to e-mail a support address to get removed, but support address has same filters and blocked unblocked request*