I have a pure curiosity question for the NANOG crowd here. If you run
your facility/datacenter/cage/rack on 120 volts, why?
I've been running my facility at 208 for years because I can get away
with lower amperage circuits. I'm curious about the reasons for using
high-amp 120 volt circuits to drive racks of equipment instead of
low-amp 208 or 240 volt circuits.
~Seth
I have a pure curiosity question for the NANOG crowd here. If you run
your facility/datacenter/cage/rack on 120 volts, why?
Because we are stupid.
I've been running my facility at 208 for years because I can get away
with lower amperage circuits. I'm curious about the reasons for using
high-amp 120 volt circuits to drive racks of equipment instead of
low-amp 208 or 240 volt circuits.
That makes you smarter than the average guy.
But, if we were really smart, we'd run at least 277, or maybe 347. Countless amounts of money would be saved on losses (transformation), copper (smaller wire), and many other areas. Most of the stuff we all run is already insulated for these voltage levels.
Even better would be all two pole 2 pole 480's or 2 pole 600's, then we wouldn't need neutrals.
1) Equipment used to not be dual voltage
2) For smaller scale, 120V UPS and distribution equipment is usually
cheaper
3) 120V embedded itself into operations as a result.
4) We're all lazy and hate change.
Our power is handed to us at 480v. We then deliver it to the customer at whatever they need. The nice thing about 120v is that everything uses it. No odd cords (as mentioned before) or expensive PDUs.
I've had a lot of people suggest that running our servers at 240v would save us money because we'd use less amps. Last time I looked at my bill I was being billed by the kWh, not amp and 240v at half the amps is still the same wattage. I've been told this so many times though that I'm starting to doubt myself. If anyone can present a reason for me to switch to 240v I'd like to hear it.
Aaron
Also, adding followings.
5) availability from local power provider(s)
6) local regulation such as fire department safety rules...
7) for your own safety... (120V may not kill people, but 240V can do...)
If you want better, why not just have everything to DC power ?
Something like 48V...
Alex
Wayne E. Bouchard wrote:
Also, adding followings.
5) availability from local power provider(s)
I don't know of anywhere in the US/Canada where power comes into the
building as strictly 110-120V. That is almost always delivered either as
1 leg of a 3-phase 208 service (most commercial/industrial deliveries)
or as two hots (240V across the two hots) and a neutral (120V from
either hot to neutral). Most datacenters are taking much higher voltage
feeds from their utilities and most of the readily available step-down
transformers and UPSs will produce 208 three-phase or 240V as
described above.
I am not an expert on power outside of the US, but, to the best of my
knowledge, Japan's 100V/50Hz is one of the few other countries
using less than 208V as their standard.
6) local regulation such as fire department safety rules...
I seriously question this one. Can you point to any examples?
7) for your own safety... (120V may not kill people, but 240V can do...)
It's relatively easy to kill someone with 12V, so, I don't see how 10x that
is significantly less dangerous than 20x. Sticking your fingers in a light
socket is going to hurt regardless of the voltage. Yes, 240V can hurt more
and faster, but, at the end of the day, it's not significantly more likely
to kill you than 110. Fortunately, most servers don't have light sockets
in the high voltage portion of the server.
If you want better, why not just have everything to DC power ?
Something like 48V...
There's a whole host of reasons, but, the biggest one boils down
to cost... Cost of the larger wires, cost of the increased line losses,
etc.
Owen
Aaron Wendel wrote:
Our power is handed to us at 480v. We then deliver it to the customer at whatever they need. The nice thing about 120v is that everything uses it. No odd cords (as mentioned before) or expensive PDUs.
I've had a lot of people suggest that running our servers at 240v would save us money because we'd use less amps. Last time I looked at my bill I was being billed by the kWh, not amp and 240v at half the amps is still the same wattage. I've been told this so many times though that I'm starting to doubt myself. If anyone can present a reason for me to switch to 240v I'd like to hear it.
Some servers (HP/Compaq comes to mind) and Cisco switches have limitations in terms of performance and/or capacity on 120v circuits.
Yes, it all gets crunched down to 5VDC and similar low voltages in the power supply. The limitation is likely due to the gauge of wire used and copper losses in the input circuitry. Higher current connectors and switches, larger copper conductors, etc. are costly. If you have an application that needs that kind of power, higher voltages make sense.
This is just as true if the application is a server as it is if it's an electric stove or clothes dryer.
Most of the rest of the world has 240v as conventional domestic power, and most server rooms or datacenters supporting >2KVA single devices have 208 or 240v available, so it makes sense for manufacturers of high-power gear to save the money on copper and connectors and insist on higher input voltages for full spec output.
Yes, it would be nice to be able to plug in your laptop charger, etc. And the voltage on that charger is likely compatible with anything from 100 to 240V. Wiring a NEMA 5-15 with 208V is just wrong, though. I have an IEC male to NEMA 5-15 female pigtail (old-school "monitor cord") with a big sticker saying "208V - Be very careful what you plug in here" for just that purpose.
Last time I looked at my bill I was being billed by the kWh, not amp and 240v at half the amps is still the same wattage.
Losses are I^2*R, so double the voltage, half the current and
experience a quarter of the loss...
Janet Plato
In a message written on Tue, May 26, 2009 at 12:39:10PM -0700, Seth Mattinen wrote:
I have a pure curiosity question for the NANOG crowd here. If you run
your facility/datacenter/cage/rack on 120 volts, why?
If folks are making their own choices, mainly for historical and
convenience reasons. If folks are building data centers for others,
it's that customers demand 120V power in many instances, for some
good, and many bad reasons.
However, for all the talk of power loss that's not the real issue.
The loss due to wire or amperage is a very small part of the equation.
While this paper is very much vendor produced, it's a good high
level summary none the less:
http://www.apcmedia.com/salestools/NRAN-6CN8PK_R0_EN.pdf
Note that in a 600Kw installation power loss is reduced from 8,894
W to 845 W, a savings of 1.3%. Note that they have included the
savings from additional cooling in that figure. Even at 1.3%, if
you looked at the cost of rewiring an existing data center based on
that figure you'd be nutty; return on investment would be forever.
But what you'll find in the paper is that the change allows you to
re-architect the power plant in a way that saves you money on PDU's,
transformers, and other stuff. Thus this makes the most sense to
consider in a green field deployment.
Thus, to reframe your question, in your existing, already built out data
center is it worth replacing 120V circuits with 208V/230V ones to save
power? No. Savings is likely well under 1% in that situation, and time
you add in the capital cost to do the work it makes no sense. In your
green field, new data center, does it make sense to look at power from an
entirely new point of view? Quite possibly.
Jay Hennigan wrote:
Most of the rest of the world has 240v as conventional domestic power, and most server rooms or datacenters supporting >2KVA single devices have 208 or 240v available, so it makes sense for manufacturers of high-power gear to save the money on copper and connectors and insist on higher input voltages for full spec output.
We're all 230vac here in Oz (it's a compromise between our old 240v standard and the Euro 220v one). In Oz we basically have a single style of outlet for AC for low amps and a couple of ones for higher amps.
The higher powered PSUs are much easier to deal with on that - everytime we get ready to commission a new router etc for the US or Japan we look in amazement at the endless list of NEMA plugs and voltage options and different kinds of APC power gear we need to do everything. It kind of freaks me out - locking, not locking etc. Admittedly I find the standard 2 pin US style power connector somewhat wobbly and scary - ours seems to lock in much better.
Since we get the same gear as North America mostly almost all of it copes with 90v to 240v AC 50/60hz. It's rare these days to find things without switching PSUs.
It's worth noting that despite higher voltages here there aren't more deaths or injuries - but maybe it's because people take it more seriously. Admittedly no one I know is nuts enough to use body parts for "liveness testing".
MMC
Leo Bicknell <bicknell@ufp.org> writes:
...
http://www.apcmedia.com/salestools/NRAN-6CN8PK_R0_EN.pdf
...
But what you'll find in the paper is that the change allows you to
re-architect the power plant in a way that saves you money on PDU's,
transformers, and other stuff. Thus this makes the most sense to
consider in a green field deployment.
noting also that "architect" is a noun, i find that on large plants the
cost of copper wire and circuit breakers add up, where sizes (and prices)
are based on ampherage not wattage. in the old days when a rack needed
6kW, that was 208V 30A (10 gauge wire) or it was two of 120V 30A (also 10
gauge wire). somewhere near the first hundred or so racks, the price of
the wire and breakers starts to seem high, and very much worth halving.
once in a while some crashcart CRT monitor won't run on anything but 120V
but for $50 NRC it can be replaced with an LCD. everything else that's
still worth plugging in (that is, having a power/heat cost per performance
better than that of a blow dryer) doesn't care what voltage it lives on.
Or go to Radio Shack and get one of those "international traveler" power converter packs.
I have a number of systems (ok, yes, they're old) that a) do not have autosensing power supplies (someone has to get a paperclip and flip a switch), and b) will not work on 208v -- 120 or 240, but not 208.
--Ricky
I love 208V but I have to fight almost everytime with our datacenter
provider. They got 50 or so "Colo's" which are all cookie cutter. Then
there is our datacenter, the only facility where they can deliver
3-phase and monitor actual power usage. Everytime when we ask for 3-phase
it is a fight now. Our latest circuits (50-amp although we won't use more
than 16A under normal use (A+B load)), took me 9 months to get out of
them. 
Seth Mattinen <sethm@rollernet.us> writes:
I have a pure curiosity question for the NANOG crowd here. If you run
your facility/datacenter/cage/rack on 120 volts, why?
I've spent the last several days going back and forth with salespeople,
trying to find a rack with 208v power in the south bay, or a cheap 100M
connection from market post tower to heraklesdata in Sacramento. (where
I have cheap 208v power) From what I see, most places in the bay area
just can't handle the kind of heat density that a 30a 208v circuit per rack
would bring. (they won't sell me more than 2 20A 120v circuits, either, and
many will only sell me a single 15a circuit per rack. I assume that's an
effort to keep the heat output within cooling system capabilities.) But
that still doesn't explain why they don't hand out 10a 208v circuits.
I've also seen employers pick 208v over 120v even after I pointed out
the cost per watt advantages of 208v, even without counting efficiency
gains. In one case they provisioned one rack with 208v, because the
vendor of some particularly expensive bit of equipment recommended it,
then they left all the commodity servers on 120v. Why didn't they put
everything on 208v? I pointed out that the cost per watt was lower.
Maybe I blew my credibility by wanting to research 48v power supplies for
our kit before that? (it was a telco facility, after all, and I was
young.)
30a 208v is about perfect for a rack, if you ask me. (I imagine
the guys who have to deal with cooling feel differently, but at my
scale, that's all priced into the power.)
Seth Mattinen wrote:
I have a pure curiosity question for the NANOG crowd here. If you run
your facility/datacenter/cage/rack on 120 volts, why?
I've been running my facility at 208 for years because I can get away
with lower amperage circuits. I'm curious about the reasons for using
high-amp 120 volt circuits to drive racks of equipment instead of
low-amp 208 or 240 volt circuits
And you have been doing something that is a right step in the right direction, but may well not be the best ultimate solution.
Lets half ignore codes. Not to be illegal, but they CAN be changed and you can often get specific exceptions if you are not just in the back room of an office but are in a clearly professionally managed facility with well trained staff plugging in equipment.
Every time I look at a nameplate and I see 100-250VAC I get very frustrated. If only that had been perhaps 100-300VAC, I could then run it on 277VAC and that is especailly nice for many reasons.
Most large USA buildings already have 277 and probably all their flourescent lighting is run off it (277VAC ballasts are readily available and what look like rats-ass wall switches but are higher rated ones are readily available - both even at home despot if you look hard enough), so nothing terribly new has to be learned by electricians, etc.
277 is the phase leg to NEUTRAL voltage of a 277/480 WYE system that most everything except small human plugged appliances use in any but the very largest USA office building. It is what typically comes in from the power company.
120/208 is the output of typically a delta/wye transformer that steps that down for the dumb humans to safely use and you are paying the penalty of the WHOLE LOAD having to go through a second less than 100% efficient transformer.
The beauty of 277 is that on a single breaker pole (unlike 208 where you are most likely to have 2 HOT legs and need a 2 pole simultaneous trip breaker) on reasonable size branch circuits that you are still allowed to plug MULTIPLE loads into without individual fuses or breakers (that is "allowed to" - you may chose to protect each outlet in the rack, but that is not compulsory) you get 277/120=2.31 times as much power available.
Sadly routers, servers, switches, etc. typically are rated to 250VAC, so using raw 277 won't work. But let us see how close HP/IBM/ACP and many many others are getting still using ONE breaker pole per much more efficient branch circuit. NB that as you go to larger branch circuits in AMPS, you MUST be supplying just ONE load or MUST have additioanl breakers or fuses as you split it up.
We all know 120/208 and 277/480. What about another NEW pair of voltages in WYE connection! Lets use 240/415. It is exactly twice 120/208 (well it is not stated as 240/416 I'd guess since 240 x 1.73259 = 415.82 they just truncate rather than round - though 2400/4160 is a standard designation...) and is inside the 250V max rating of the switching power supplies. It still uses a single breaker pole. Your get EXACTLY twice as much power out of a 240/415 WYE branch circuit as you would out of a 120/208 at the same AMPs. But you may save a transformer and its continuous power waste or at least part of it in between.
How do you get to 240/415 is the next issue. If you have 2400/4160 or 7,960/13,800 primary into your building, and you do all your own transformers, getting 240/415(6) can be a single transformer step for you, and you will probably have many transformers so can also create seperate 277/480 for modest size AC inits and lighting, While LARGE chillers can be ordered at the higher voltages, and for the relatively small amount of 120/208 you probably should come off 277/480 into standard 120/208 delta wye transformers because normal electricians can do that rather then the 13K gods($$$$).
But if you are a smaller building the only voltage that makes sense that the utility is supporting is 277/480. Rather than take all your rack power through another transformer step with the losses and the extra heat to eject from the building, consider instead using buck (as in the classic BOOST/BUCK transformers) to knock that 277/480 down to 240/415. It can be packaged as a 3 phase unit for less than three singles, and will be smaller and less costly to have wired up, but the three singles may be available from stock.
It is the same sort of device you must have in front of a load that needs 240 or 250 and can't handle 208, but in that case is wired BOOSTING rather than BUCKING. FWIW an electric range burner or a hot water heater element rated for 240 produces EXACTLY 75% of the heat if run on 208 (go do the math...), but you should NOT use boost bucks for such a simple situation because optional heating elements can be ordered originally OR bought as replacements for less than $10 each and easily replaced in the field to give the original 240 rated wattage on 208 supply.
In any case the 3 phase buck transformer VA rating will just need to be 3 x (277-240) x per-phase-AMPs on the load side. Or look at it this way: 37/240 = 15.4% to just buck rather than the KVA of a transformer dropping the whole load. Remember the buck transformer's secondary in this case simply is 37 volts and is wired in series but 180 degrees out of phase so drops the 277 to 240VAC But maybe that isn't wizest in the big picture but may work for you.
If you are into really big systems you really need folks that know what they are doing. Those that simply tell you to buy K rated transformers may be missing a slick opportunity to knock out power factor problems caused by triplen harmonics related to the multigrounded neutral system. So you may not want to BUCK, but instead use an exceptional transformer SYSTEM.
Look at these folks (this is just one PDF, they have plenty more, and a few dollars more spent with them can bring a rapid ROI just on power factor savings alone on the utility bill let alone the dramatically lower transformer losses) :
http://mirusinternational.com/downloads/CAT-EC01-08-F10a_w.pdf
NB that one of their tricks is to have the load split to separate outputs with different phase shifts within these special transformers so the troublesome harmonics are canceled and not reflected back into the primary. Even though newer power supplies will be more efficient and better power factor corrected that years ago, this is still an issue to be very aware of.
Anyway, GOOGLE for 415 volts or 240/415 and these days you will find many hits from big names you already know. Maybe any "issues" with local codes can be solved or already have been.
Some folks have been bucking down from 277 to more like 250 for years (to neutral, so single breaker) with the local inspector staying totally clueless at to what was "accidentally" happening. He was probably aware of plugged in PDUs with integral 277/480 to 120/208 DeltaWye transformers in use so probably assumed a lot of the higher voltage breaker panels were feeding more of that.
A 30 amp 3 phase feed at 240/415 to a big power strip with 15 or 20 amp breakers for smaller groups of single phase IEC outlets is a STANDARD product from multiple sources and at the 80% allowed loading provides 240 x 30 x 3 x .8 = 17,280 VA and just two of these feeds gets you to almost 35 KVA per cabinet that drives folks to liquid CO2 cooling systems that can easily function in a datacenter without a raised floor, and do it without CRAC created hurricanes or huge air ducts trying to cool the room and failing.
NB that such a power strip using IEC outlets uses EXACTLY the same cords as you would use for 120VAC - nothing weird other than NOONE is using the "WEIRD" North American plugs and outlets Well, almost. The power strip could be built for BOTH 120/208 AND 240/415 and would require NO outlet changes. Breakers, shoild be picked for the higher voltage. IEC outlets in these voltages are THE SAME - they just change between 15 and 20 amps and there are specials for high temperature usage.
The one thing that should be done is to be sure be to use S cord rather than SJ cord (600V class vs 300V class - well even better use the "O"il option too, simply because it lasts a lot longer - SO cord) - or use a plastic equivalent for any cords carring more than one HOT from 240/415.
The old code allowed 42 poles per breaker panel. That would be 7 cabinets worth of just these handy size 30 amp 3 phase feeds for 241920 VA total - almost 1/4 of a megawatt. Bigger branch circuit amps can give you a lot more, obviously, but short of going to Square-D I Line panels, you probably don't want much more load than this in a typical breaker panel. It would be "nice" to have the two 30 amp 240/415 WYE feeds to any cabinet be fed from the SAME breaker numbers in two totally separate breaker panels, and each server's power supplies split between the two branch circuits.
Just be VERY VERY certain you check each device as there are still some 120 volt ONLY devices some idiot will want to plug in.
Three phase isn't all that weird..
Keeping it simple, if you have three separate coils on a generator that each produce AC 120 degrees out of phase with each other, and connect one end of each coil to a single grounded point called "neutral" , you WON'T have twice the single coil voltage between any two of the other ends of the coils (the HOT wires), but have the square root of 3 times it instead because they are not a full 180 degrees out of phase the way single phase house power is, or two flash cells that when you ground the center connection between the cells and call it neutral, you will find 3 volts between the ends of the two cell battery and 1.5 from that middle to either end BUT with opposite polarity.from the center.
The sign waves coming out of that 3 phase generator peak in succession at each hot leg in turn, and a three phase motor will reverse direction if you swap any two of the three wires.
Just as on a single phase neutral where you can reduce the neutral current to ZERO if you have two identical resistive loads (ie non reactive) from each hot leg to neutral, so can you if you have three identical resistive loads each on one of the three phase hot legs to neutral.
If you put one of those resistive loads on just one phase leg, obviously the NEUTRAL has the same current as the one in use phase leg. The trick question is how much is on neutral if you had just two of the three resistors connected. Think of it this way. If all three are on, neutral has ZERO amps. cut ONE off, and you just CHANGED what is on neutrasl by one resistor's worth of current. When we remove that resistor from the common neutral wire that was reading ZERO amps, we will get a one amp reading on neutral. If we were to turn that resistor back on and shut OFF the other two, we would also get 1 amp on neutral, BUT there would be an exact 180 degree phase shift if we were watching the two currents on a scope. Together they cancel. But but but why 180 when 3 phases are all 120 degrees apart? Those two other legs ARE 120 degrees apart and each 120 apart from the third leg, but those two 120 degrees apart currents sum to being 180 degrees from the third leg (still assuming each resistor is indentical).
Anything other than resistors and current may lead or lag voltage and harmonics may add in ugly ways so neutral current can easily exceed that on any phase leg and thus we have all the power factor issues and charges from utilities and mandates for better power factor corrected supplies and codes dictating over sized neutrals where switching power supplies abound.
I was cheerily suggesting hot leg to neutral WYE (aka STAR) connections ( and I still do!). But actually those using two hot pole branch circuits ( or two hots of a three phase branch ciruuit broken into many smaller two wire 208 ones ) are NOT using the neutral at all in each of those SINGLE PHASE connections and avoid the triplen harmonic currents summing in the neutral, and can also still use the neutral for small random 120V loads but need not oversize the neutral. Actually any load including three phase DELTA stuck across there that avoids the neutral avoids the harmonics issues. I was just using the 208 single phase loads because they seemed to have been used more than I had suspected.
Using a three phase branch circuit (say 30 amps as that would be a handys size, but DOES NOT MATTER here) at 120/208 - whether you are using it as 3 120 volt circuits or 3 208 volt circuits ( as EACH of those connection options gets EXACTLY the same max power out of the circuit!!!) gives you EXACTLY 1/2 the power you could get from the same ampere circuit running 240/415 (which would normally ONLY be running phase to neutral loads at 240V, but were 415 volt capable devices available you still get the same total out - EXACTLY twice what the 120/208 can deliver).
NOTE that I carefully was refering to 4 wire 3 phase legs and a neutral for these example branch circuits. IF you are comparing two wire single phase circuits from the breaker panel to the cabinet (how silly! unless it is from a PDU in an adjacent cabinet), 208 volts gets you about 13% less power on your two wires than you could get at the same amperage on a 2 wire circuit at 240 volts. Whether those wires are both hots or a hot and a neutral makes no matter as it is simply the voltage between those two wires times max allowed amps (80% of branch ckt rating) that determines what you can get out. A switching supply simply sucks more amps at lower input voltage so gets the power it needs in either case, but you can inflict a larger load at the higher voltage.
Thus 240/415 (240/416...) DELIVERS twice what 120/208 does amp for amp with full three phase wiring, and that is why it is so great.
P=V*I
As a Holder of two different FCC licenses I can tell you voltage is not
what kills, it is amps and location that kill. Actually in certain cases
as long at you have good electrical isolation, high enough dielectric
breakdown voltage, and good grounding higher voltages can be safer and
more efficient. Also, Thomas Edison was the one that discovered that
trying to deliver DC more than a few feet was not a good idea.
The early problems with distance transmission of DC really didn't have
anything to do with the inherent properties of DC current, but with
the fact that, at the time, there was no good way to convert DC
voltages up and down in a similar fashion to the function performed by
transformers with AC.
The inability to step DC up to high voltage for distant transmission
was the real killer for early use of DC. Lately, very high voltage DC
is actually a better performer than AC for some long distance
transmission situations. In particular, DC can be used to move power
between unsynchronized grids without the usual problems, and to
transmit power through undersea cables, where AC capacitance losses
would add up. See:
http://en.wikipedia.org/wiki/High-voltage_direct_current#Advantages_of_HVDC_over_AC_transmission
The main thing that has changed since the early days is that much
better semiconductors are available to make the voltage conversion
feasible...
-Dorn
Brian Raaen wrote:
As a Holder of two different FCC licenses I can tell you voltage is not
what kills, it is amps and location that kill. Actually in certain cases
as long at you have good electrical isolation, high enough dielectric
breakdown voltage, and good grounding higher voltages can be safer and
more efficient. Also, Thomas Edison was the one that discovered that
trying to deliver DC more than a few feet was not a good idea.
Hi Brian,
as a Radio Amateur you should know AC radiates, DC not.
We did have really big trouble when when an ocean liner had to pass
under a cable bridge and for the passage, that cable bridge had to
be grounded. Eddies running through the grid brought half of the
grid down. The other half killed my computer with overvoltage.
Hydro Que'bec is running DC from the northpole down to South Florida.
Apropos, I remember a frenchman who fed his personal computer 288 Volts DC.
Theory says no matter whether the setting of the powersupply is 120 AC ord 240 AC it
should work. Try at your own risk. I haven't 
Kind regards
Peter DL2FBA
Peter Dambier wrote:>
Apropos, I remember a frenchman who fed his personal computer 288 Volts DC.
Gives a whole new meaning to "French Fries"
Mike, sorry