Temp at Level 3 data centers

Curious if anyone on here colo’s equipment at a Level 3 facility and has found the temperature unacceptably warm? I’m having that experience currently, where ambient temp is in the 80’s, but they tell me that’s perfectly fine because vented tiles have been placed in front of all equipment racks. My equipment is alarming for high temps, so obviously not fine. Trying to find my way up to whomever I can complain to that’s in a position to do something about it but it seems the support staff have been told to brush questions about temp off as much as possible. Was wondering if this is a country-wide thing for them or unique to the data center I have equipment in. I have equipment in several others from different companies and most are probably 15-20 degrees cooler.

Thanks,

David

with a former employer we had a suite at the L3 facility on Canal in
Chicago. They had this exact issue for the entire time we had the suite.
They kept blaming a failing HVAC unit on our floor, but it went on for
years no matter who we complained to, or what we said.

Good luck.

On 10/11/17, 7:31 AM, "NANOG on behalf of David Hubbard"

There are plenty of people who say 80+ is fine for equipment and data centers aren’t built for people.

However other things have to be done correctly.

Are you sure your equipment is properly oriented for airflow (hot/cold aisles if in use) and has no restrictions?

My house isnt built for moving furniture, it's built for living in. I've not
moved a bed in or out of the bedroom in 8 years now. But for the 15 minutes
I did move a bed, the door and hallway had to accomodate it.

Humans have to go into datacenters - often in an emergency. Complicating the
servicing of equipment by having sweat drip off you into the electronics is
not condusive to uptime.

/kc

I recall some evidence that 80+F temps can reduce hard drive lifetime,
though it might be outdated as it was from before SSDs were around. I
would imagine that while it may not impact the ability for a server to
handle load, it may reduce equipment lifetime. It also could be an
indication that they lack redundancy in the case of an AC failure. This
could cause equipment damage if the datacenter is unattended and
temperatures are allowed to rise.

http://www.datacenterknowledge.com/archives/2008/10/14/google-raise-your-data-center-temperature

In a message written on Wed, Oct 11, 2017 at 12:54:26PM -0400, Zachary Winnerman wrote:

I recall some evidence that 80+F temps can reduce hard drive lifetime,
though it might be outdated as it was from before SSDs were around. I

This is very much a "your infrastructure may vary" situation.

The servers we're currently buying when speced with SSD only and
the correct network card (generally meaning RJ45 only, but there
are exceptions) are waranteed for 105 degree inlet operations.
While we do not do "high temperature operations" we have seen
operations where folks run them at 90-100 degree input chasing
effiency.

Famously, Intel ran computers outside in a tent just to prove it works
fine:

It should be easy to purchase equipment that can tolerate 80-90
degree input without damage. But that's not the question here.
The question is if the temp is within the range specified in the
contract. If it is, deal with it, and if it is not, hold your
vendor to delivering what they promised.

That's a good point, though if you are running your breakers that close
I think you have bigger problems, as a power outage, however unlikely,
could cause your equipment to not come back up at all. Software updates
that reboot several servers in quick succession could also cause a
breaker to trip under those circumstances. Unfortunately, there's no way
to tell how close a breaker is to tripping without tripping it. Breakers
may have amp meteres and a rated size, but the actual load before
tripping is +-20% for common models, meaning a 20A breaker may trip as
low as 16A.

If the ambient temperature is higher is means the temperatures throughout the device would be higher and the temp at those points is what really matters. I would also be concerned because if they lose one of the a/c units what would the ambient temperature rise to? I would want them to tell me what the set point of the a/c actually is.

Bottom line 80 F input air is too hot in my opinion and apparently the equipment's opinion as well.

Steven Naslund
Chicago IL

I think the key here is that if your set point is at 80 F, you better be able to hit it with a unit down and you better be able to react instantly to any environmental failure. You just have no headroom to play with.

Steven Naslund
Chicago IL

My 0.0000041 BTC:

1) For small facilities, without separate temperature-controlled UPS zones,
the optimum temperature for lead-acid batteries may be the lower bound.
77°F is optimal, with significant reduction in battery life even 15°F above
that. Given that batteries' internal temperature will be higher than
ambient, 80° set point is not stupid. I run cooler, FWIW.

2) Headroom. I try to have documented for each facility the climb in
degrees per hour (determined empirically) as a backup so I know required
response times when AC failure occurs.

I would also be concerned because if they lose one of the a/c units
what would the ambient temperature rise to?

It doesn't matter much if the "normal" temperature in your DC is 10
or 30 degrees Celcius; if the cooling system is barely keeping up
with that, and you loose half your cooling capacity, then temperature
will rise pretty quickly, until the servers are literally cooked (i.e.
temperature reaching 100°C or more).

The spare capacity of the cooling system is the important information,
not the starting temperature. That difference of 10-20°C in starting
temperature will just give you a few minutes extra, not *save* you, if
there is not enough spare capacity in the cooling system.

Assuming a reasonably densly packed data centre, at least; with low
power density, thin unisolated walls, and winter outside, you might
survive even a full cooling failure. :slight_smile:

Also, depending on your cooling solution, a partial failure might not
be very common. We have district cooling providing us with cold water,
and if that stops pumping water, and we use up our 30m³ buffer tank
(which is enough for 30-40 minutes), *all* cooling stops. But on the
other hand, we have capacity enough to survive even if they give us
16°C water instead of the normal 9°C water.

I would want them to tell me what the set point of the a/c actually is.

That I agree with.

Bottom line 80 F input air is too hot in my opinion and apparently
the equipment's opinion as well.

Unfortunately the default settings of most servers are not very well
thought through. They will typically spin up fans *much* more than
is actually needed to protect the hardware, and often there is no way
of changing that for the user. And if you try to get the manufacturer
to tell you what the most power-efficient inlet temperature is, they
will just tell you "oh, we support anything between 5°C and 40°C" (or
whatever their actual limits are), and absolutely refuse to answer your
actual question.

Also worth noting that temperature tolerances for large scale numbers of 1U
servers, Open Compute platform type high density servers, or blade servers
is a very different thing than air intake temperatures for more sensitive
things like DWDM platforms... There's laser and physics related issues
where temperature stability is important as channel sizes get narrower in
terms of optical THz.

As temp goes up wire resistance increases too, increasing heat, increasing
resistance, etc - and I find breakers trip more easily at hotter temps too.

/kc

David

The issue has several components and is vendor agnostic.

Set Point: The systems are specifically set at a temperature
Capacity Ability: The systems can maintain a temperature
Customer Desire: What you expect from sales promises.
Sales Promise: What they might carefully avoid promising.

I suggest you review your SLA and discuss with legal asap. You could have a
document defining your question's answer already but it sits in a filing
cabinet file labeled business continuity.

If the set point is X then they likely would answer quickly that that is
the case.
If the capacity is lacking then they would likely redirect the issue.
If they don't care about the customer that alone should be an indicator
If a promise exists in the SLA then the ball is in your court

From the emails I fear that we have confirmed that this is normal. So your

question "Is the temperature at Level 3 Data Centers normally in the 80-90F
range?" sounds like a Yes.

Regardless of the situation always ask for names, titles, and ask vendors
to repeat critical information like the status of cooling in a building
designed to deal with cooling. Keep the vendors that do it well.

with a former employer we had a suite at the L3 facility on Canal in
Chicago. They had this exact issue for the entire time we had the suite.
They kept blaming a failing HVAC unit on our floor, but it went on for
years no matter who we complained to, or what we said.

Good luck.

At $dayjob-1, we had a couple cabinets at that facility, and sometime around 2007, if I recall correctly, they kicked out all of the customers that were server-farms. Only carriers were allowed to stay (with a few exceptions, I'm sure ...) I know because we picked up a few of their customers.

That facility was built out around 2000/2001, and things were a lot different back then (e.g. no one was really using 208/240 yet.) I think they just couldn't keep up when things really took off again post dot-com bust.

Hi there,

been there, done that, rocky way ahead.

In Europe the standard temperature Level3 SLA is 26C. The measurement is done on the cool aisle, at a distance of 45 cm to the equipment and a height of 100 cm. You can use a Testo 625 handheld for measurements, that is also handled by Level3 staff.

Do you guys still at least have biometric access control devices at your Level3 dc? They even removed this things at our site, because there is no budget for a successor for the failing unit. And to be consistent, they event want to remove all biometric access devices at least across Germany.

Regards
Jörg

Hi Jörg,

IMO, biometric was a gimmick in the first place and a bad idea when
carefully considered. All authenticators can be compromised. Hence, all
authenticators must be replaceable following a compromise. If one of your
DCs' palm vein databases is lost, what's your plan for replacing that hand?

Regards,
Bill Herrin

Hi Jörg,

IMO, biometric was a gimmick in the first place and a bad idea when
carefully considered. All authenticators can be compromised. Hence, all
authenticators must be replaceable following a compromise. If one of your
DCs' palm vein databases is lost, what's your plan for replacing that hand?

Basic two or three factor authentication: something that you know
(password), something that you are (biometric) and something that you have
(access card).

You can tell your password to a coworker but he can not borrow your hand.
Hence you need both. The password is the replaceable part.

My quick thoughts on the matter:

1. Above all else, know what your DC provider states in their SLA/contract.
2. It's never a bad idea to try to be on the best possible personal terms
with the DC manager(s), the better you get along the more they're inclined
to share knowledge/issues and work with you on any concerns.
3. You can't infer faults or lack of redundancy from the running
temperature - by way of example several facilities I know run at 25 degrees
celsius but if a chilled water unit in a given data hall fails there's a
number of DX units held in standby to take over. This is where point 2
comes in handy as knowing somebody on the ground they'll often be quite
happy to run through failure scenarios with you and help make sure
everybody is happy with the risk mitigation strategy.

Out of idle curiosity - I'm curious as to if the equipment that is alarming
is configurable or not? Reason I ask is I've heard users claiming
environmental parameters were out of spec before, but then it turned out it
was their own environmental monitoring they'd installed in the rack (using
default parameters out of the box, not configured to match the facility
SLA) that was complaining about a set point of 25...

Cheers,

Sam