For anyone running IP networks in the Midwest, are you having to do anything special to keep your networks up?
For the data centres, is this cold front a chance to reduce air conditioning costs, or is it actually straining the infrastructure?
I’m curious, from a +27-degree C summer’s day here in Johannesburg.
Mark.
Being a Minnesota native, I can tell you that while it is indeed cold, this is nothing new i the Great White North 
I am amaze a how consistently the media overplays the severity of Midwest cold weather as some kind of unique phenomenon. They amplify this by reporting the wind-chill factor, which is the “what it feels like” equivalent in a cold and windy environment. But equipment feels nothing, so windchill is irrelevant.
For example, Minneapolis is -20F, but the news media instead reports “-60F wind chill”, which, while dramatic, is not meaningful for most purposes. I grew up in Minnesota with -30F and lower quite common, and we walked to school in those temperatures. You just have to dress well. Minneapolis is paved with tunnels and heated skyways to eliminate most outdoor walking downtown.
As far as networks go, none of the ISPs I know of do anything different than anywhere else in the country. Everyone has backup power. It’s already common practice everywhere to exploit cooler winter ambient temperatures to reduce HVAC requirements, so that’s not new either. But it gets as hot in the Midwest in our summer as it is in SA for you now, so everyone must still build out HVAC capacity to cover the hottest days.
-mel beckman
Approximately 3 hrs ago we lost B-feed at Minneapolis Cologix.
Apparently the local utility requested that they move one side to generator due to the weather and high-utilization, and the ATS failed.
But we're up ...
And here I always figured it was bespoke knit caps for all the packets in cold-weather climes?
learn something new every day! (also, now I wonder what the people who told me they were too busy knitting caps are ACTUALLY doing??)
The main issue is infrastructure like power, cable damage, and heating/cooling systems.
Power lines tend to go down because anything weak becomes brittle and any accident involving a pole tends to cause them to break rather than absorb impact. Also, conduits and manholes that normally might be above freezing underground can have standing water freeze breaking splice cases and such.
Large chiller plants need to be run at higher temperatures and speeds to keep any outdoor components from freezing up.
Of course, repairs on anything outdoors tend to take a lot longer to resolve. Anything that requires digging might be near impossible in these conditions.
So far though in my area (Chicago) my carriers AT&T, Comcast, Cogent, and Level 3 all seem to be fine so far. Our current temp is -18. Wind chill reports at -50 to -60 depending where you are.
Steven Naslund
Chicago IL
To be fair, reporting the the wind chill factor is very meaningful for health and safety reasons almost everywhere so proper warning is given about people spending time outside. Minneapolis, and the bigger Canadian cities have those inside walkways and pedestrian pathways, but they’re not that common elsewhere. I don’t think Chicago does for example, and we don’t have that here in Buffalo. Contrary to the rumors, 0F with -40F wind chills are NOT very common around here.
People need to be warned to take this weather seriously. You might be used to it, but not everyone in a native that can say that.
To the ‘infrastructure’ question, I think the biggest concerns would be power related. Although we have a DC in Buffalo that is cooled on ambient outside air that has the opposite problem ; it’s TOO cold at the moment, so we are cycling most of the hot server exhaust back into the computer rooms to maintain temperatures.
To the 'infrastructure' question, I think the biggest concerns would >be power related. Although we have a DC in Buffalo that is cooled >on ambient outside air that has the opposite problem ; it's TOO cold >at the moment, so we are cycling most of the hot server exhaust >back into the computer rooms to maintain temperatures.
Exactly what he said. We actually run cooling and supplemental heating in extreme cold. We need to keep the chiller pulling heat into itself and pumps moving on high to keep the outdoor components from freezing up. During the summer you might run close to or slightly below freezing on the coolant loops but in these conditions you cannot run that low a temp because things will freeze up before the coolant returns. We also have to keep the room reasonably warm (50F +). You also need to watch out for fast temp excursions to keep humidity under control.
The wind speed does make some difference since it is like a fan on your evaporator pulling heat out of the cooling loops faster than still air will.
Steven Naslund
Chicago IL
Exactly what he said. We actually run cooling and supplemental heating >in extreme cold. We need to keep the chiller pulling heat into itself and >pumps moving on high to keep the outdoor components from freezing >up. During the summer you might run close to or slightly below freezing >on the coolant loops but in these conditions you cannot run that low a >temp because things will freeze up before the coolant returns. We also >have to keep the room reasonably warm (50F +). You also need to watch >out for fast temp excursions to keep humidity under control.
A good HVAC team is critical because we have noted that the building management systems often are not flexible enough to automatically deal with super extremes and require some human intervention to tell them to do things like run heat and cooling simultaneously. Other actions like closing down louvers on evaporators may or may not be automated depending on your systems. If any part of the system does fail in these conditions you have to move super quick or you could get serious damage fast. Our biggest monitoring points are flow rate/pressure (which could indicate a freeze up beginning or a pump failing), output and return temp on the loops.
Steven Naslund
Chicago IL
And apparently fire. I wasn’t going to chime in but one of my providers just alerted us to an electrical fire in a Minneapolis pop causing loads to failover to ups. Unknown whether weather conditions contributed to the incident. PZ
This was one of the things I was concerned about, fluids going smudgy or just simply freezing up… Mark.
Ironically you don’t really save a lot of energy when it’s this cold because the loops are running at high speed and the humidification coils are working overtime to keep the RH up in the room.
People think we can bring in all the outside cold we want but the issue then is humidity stability.
Steven Naslund
Chicago IL
And apparently fire. I wasn’t going to chime in but one of my >providers *just* alerted us to an electrical fire in a Minneapolis pop >causing loads to failover to ups. Unknown whether weather >conditions contributed to the incident.
Yes, in Chicago we will see an increase in home fires because heating systems are being pushed to their limits and people tend to do stupid things like run unattended space heaters and try to thaw frozen stuff in crazy ways. In a datacenter you might be pushing electrical loads while external electrical components are stressed with temperature. I have seen transformer fires caused by the oil inside not circulating correctly. You end up with hot and cold zones in them.
Steven Naslund
Chicago IL
Well said. The electrical load shifts, hydraulic systems, airflows constrained by ice cover, etc, etc, etc. All kinds of things being asked to do stuff outside or at the edge of specifications.
Hug your local facilities guys when these things happen. (Or bring them booze.)
Re: Fire
Also fire dept response.
I've ridden with the Boston Fire Dept, extreme cold is a major PITA,
hydrants freeze, you have to work in it going from the heat of the
fire to sub-zero air temps over and over, all while getting soaking
wet, and wind-chill is certainly a factor.
There were always cautionary stories about some firefighter having a
heart attack struggling to get a frozen hydrant open. Whether factual
or not I think it makes a point, no water no firefighting in general.
I could tell you about the fire boats in February...sometimes you need
them.
I just saw a news spot (I believe Chicago) where they had to raise to
multiple alarms on a fairly simple tho working house fire just so they
had enough firefighters to cycle them through a warming truck (I don't
remember any warming truck in Boston tho you could go sit in a truck
cab :-))
Which means thinner coverage for other areas.
Cold changes the transmission characteristics of fiber. At one point we were renting some old dark fiber from the local telephone company in northern Maine. When it would get below -15%-degree F the dB would get bad enough that the link using that fiber would stop working. The telephone company was selling us dark fiber because regulation required them to. They refused to give us another fiber nor inspect/repair. They took the position they were required to sell us fiber, not working fiber.
Unfortunately, they’re knitting DATA caps.
Sorry, couldn’t resist.
Matt
Excessive cold killed us once when the air exit vents froze shut.
Fletcher,
I don’t think that’s true. I find no specs on fiber dB loss being a function of ambient temperature. I do find fiber optic application data sheets for extreme temperature applications of -500F and +500F (spacecraft). You’d think if temperature affected fiber transmission characteristics, they’d see it in space.
What you likely were seeing was connector loss, owing either to improper installation, incorrect materials, or unheated regen enclosures.
Insertion loss (IL) failures, for instance, in the cold are a direct result of cable termination component shrinkage. That’s why regen and patch enclosures need to be heated as well as cooled.
All fiber termination components have stated temperature limits. As temperatures approach -40F, the thermoplastic components in a cable’s breakout, jacketing, and fiber fanout sections shrink more than the optical glass. Ruggedized connectors help somewhat, but the rule is that you can’t let optical connectors and assemblies get really cold (or really hot).
A typical spec for a single-mode OSP connector is:
Operating -30C (-22F) to +60C (+140F)
The range for the corresponding Single Mode fiber is:
Operating -55C (-67F) to +70C (+158F)
Storage -60C (-76F) to +70C (+158F)
Installation -30C (-22F) to +50C (+122F)
All professional outside plant engineers know these requirements. So if you’re seeing failures, somebody is breaking a rule.
-mel
Out here in Manitoba we use unheated/no-electricity OSP fiber patch panel pedestals in some locations, those work without issue down to the occasional -40. Note that that’s using all high-quality components.
For Fletcher’s case, it’s also possible that:
-there had been water intrusion in a splice case or cable on the way – but then that tends to cause complete failure, either on the first occasion or not long after, and not repeating temperature-dependent fade.
-there’s a bad fusion splice on the way whose characteristics are affected by temperature.
My first step in such a case would be to OTDR the line (renting an OTDR if we were a company that didn’t own one) to see approximately where the issue is and to get an idea what kind of issue it is – Fletcher, I guess that your company did not do so at that time?
Mel;
You are absolutely right. I should have been more specific in my description of the problem.