POE switches and lightning

We had a lightning strike nearby yesterday that looks to have come inside our facility via a feeder circuit that goes outdoors underground to our facility's gate.

What's interesting is that various POE switches throughout the entire building seemed to be affected in that some of their ports they just shut down/off. Rebooting these switches brought everything back to life. It didn't impact anything non-POE, and even then, only impacted some devices. But it was spread across the whole building, across multiple switches.

I was just curious if anyone had seen anything similar to this before? Our incoming electrical power has surge suppression, and the power to the switches is all through double conversion UPS, so I'm not quite sure why any of them would have been impacted at all. I'm guessing that the strike had some impact on the electrical ground, but I don't know what we can do to prevent future strikes from causing the same issues. Thoughts?

My first guess would be the lightning was close enough/powerful enough,
to send out an EM Pulse which got picked up by the copper going to the
devices. This EM Pulse may have been interpreted at the switchport as
the device relinquishing power?

Had you tried just unplugging one of the devices from Ethernet, and
plugging it back in to reset the PoE exchange?

Ken Matlock
Network Analyst
Exempla Healthcare
(303) 467-4671
matlockk@exempla.org

I don't know how to account for this in a PoE world, but when I last
managed a campus network, we had major issues (particularly in an
active-thunder-storm environment) of severe difference in
ground-potential between buildings.

The only way we could survive was to connect buildings (including
free-standing kiosks) with their own "grounds" using glass.

Does anybody make a CAT 5 1-to-1 isolation transformer?

Cat 5 has isolation transformers in or just behind each jack. However,
in most equipment the grounds aren't really isolated, and in the case of
POE they (mostly) aren't at all.

Lightning likes to do "interesting" things. It can induce a 20kv per
few feet gradient (or more) across the ground mesh of a power substation
(like 4/0 wire in a mesh of 4 foot squares or so; normally more
complicated than that since it has to clear equipment etc...). It likes
to eat power supplies in well-grounded equipment and leave cheaper stuff
alone. It can hit an antenna, leave the receiver completely intact, and
fry the power supply of the next box over. We tended to lose either
fluorescent ballasts or the thermostat transformer in our furnace when I
lived in an active ham's house in Alabama, the radios tended to live.
(you should have seen his coax entry panel (1/4 inch copper sheet,
grounded outside)), and stuff got manually disconnected from both
antennas and power when a storm was expected (every afternoon :-).

It wouldn't surprise me if the first answer was right and either the
ground pulse or EMP reset the safety switches in the POE feeders.

-- Pete

While the equipment may well be affected by an EM pulse, if the gear returns to normal after a power cycle, then the equipment vendor didn't do their job fully developing the product. A product should be tested to take such pulses and should recover provided it has not suffered a catastrophic failure (and in fact it should contain sufficient protection to avoid such in most cases).

In working on one particular router in the lab some years ago, I was verifying some software functionality and the hardware engineer I was working with reached over my shoulder and used a device that delivered a high voltage spike (simulated lightning) to a 10BaseT network port. After I peeled myself off the ceiling (and he stopped laughing), we set to work figuring out how to get the device to self-reset after such a strike. One component, an Ethernet hub chip, got into a confused state. I was able to detect this in software, so we adjusted the product design so that the software could yank the hub chip's reset line.

It's unfortunate that products, both hardware and software, receive minimal quality testing these days. Guess it's not a surprise, since buyers seemed to prefer products that were quick to market, with lots of bugs, rather than reliability and resilience.

About a month ago, we had a lightning strike near our main campus. We lost one POE Cisco 3560 completely (apparently blown power supply), and in a separate but nearby building, another 3560 lost the ability to deliver POE, but continued to operate as a switch. Both had to be replaced. Both were on wiring closet type UPS'es with surge suppression, and those were unaffected.

Mark

Caleb Tennis wrote:

We had a lightning strike nearby yesterday that looks to have come inside our facility via a feeder circuit that goes outdoors underground to our facility's gate.

What's interesting is that various POE switches throughout the entire building seemed to be affected in that some of their ports they just shut down/off. Rebooting these switches brought everything back to life. It didn't impact anything non-POE, and even then, only impacted some devices. But it was spread across the whole building, across multiple switches.

I was just curious if anyone had seen anything similar to this before? Our incoming electrical power has surge suppression, and the power to the switches is all through double conversion UPS, so I'm not quite sure why any of them would have been impacted at all. I'm guessing that the strike had some impact on the electrical ground, but I don't know what we can do to prevent future strikes from causing the same issues. Thoughts?

I use these on any cable that leaves my building.

http://www.amazon.com/APC-PNET1GB-ProtectNet-Standalone-Protector/dp/B000BKUSS8

It seems to play well with PoE (I put mine before the injector), and also works well with T1s and POTS.

-Paul

It's not just a matter of "these days" -- lightning is awfully hard to deal with, because of how quirky the real-world behavior can be. I had to deal with this a lot in the 1970s on RS-232 lines -- we could never predict what would get fried. Of course, there was also a ground strikes very near my apartment, where the induced current tripped a circuit breaker, blew out a couple of lightbulbs, and and came in through the cable TV line to fry the cable box, fry the impedance-matching transformer, and fry the RF input stage on the television...

    --Steve Bellovin, http://www.cs.columbia.edu/~smb

While the equipment may well be affected by an EM pulse, if the gear returns to normal after a power cycle, then the equipment vendor didn't do their job fully developing the product. A product should be tested to take such pulses and should recover provided it has not suffered a catastrophic failure (and in fact it should contain sufficient protection to avoid such in most cases).

In working on one particular router in the lab some years ago, I was verifying some software functionality and the hardware engineer I was working with reached over my shoulder and used a device that delivered a high voltage spike (simulated lightning) to a 10BaseT network port. After I peeled myself off the ceiling (and he stopped laughing), we set to work figuring out how to get the device to self-reset after such a strike. One component, an Ethernet hub chip, got into a confused state. I was able to detect this in software, so we adjusted the product design so that the software could yank the hub chip's reset line.
    

Luck. I've needed that kind of reset a few times...

It's unfortunate that products, both hardware and software, receive minimal quality testing these days. Guess it's not a surprise, since buyers seemed to prefer products that were quick to market, with lots of bugs, rather than reliability and resilience.
    

That is certainly true (and not entirely modern; you can read about that
problem in old roman literature. When was "Zen and the art of
motorcycle maintainance" written? - 1970's); however it is nearly
impossible to protect well against close-by lightning.

    

It's not just a matter of "these days" -- lightning is awfully hard to deal with, because of how quirky the real-world behavior can be. I had to deal with this a lot in the 1970s on RS-232 lines -- we could never predict what would get fried. Of course, there was also a ground strikes very near my apartment, where the induced current tripped a circuit breaker, blew out a couple of lightbulbs, and and came in through the cable TV line to fry the cable box, fry the impedance-matching transformer, and fry the RF input stage on the television...
  

I can second Steve in spades; I used to work for the power company in
Alabama... There you learn a LOT more than you ever wanted to know
about lightning. Consider that one hit can destroy the inside of a

10Mw 66kv->12kv distribution transformer (I actually saw the strike

involved; it was less than a mile from my apartment at the time, and
dropped power to me; the apt was fed from an entirely different
company... My power came back in a few minutes; the other load took
almost a week (they had a redundant feed; it was a hospital, but they
ran in a low-power mode till a BIG crane and big lo-boy truck came with
another transformer)); how are you going to protect any computer from
*that*...

-- Pete

Inductively coupled EMP onto the CAT5. I've seen ethernet port chips
vaporized on switches. I've even seen holes blown in port interface chips,
and the switch continue working (have a DC powered Catalyst 2900XL switch with
the center 8 ports in a nonworking state due to EMP from a close strike; the
2900XL is still running fine, just can't use those center eight ports anymore).
The building it is installed in is on solar power, and at the time was off-
grid. A Siteplayer Telnet was blown, and the eight ports were fried (one of
which was connected to the Siteplayer Telnet that got blown) on the switch,
but that was the extent of the damage.

I'm from a broadcast engineering background, and have seen lightning's effects
in many many devices, including vaporized PC traces, etc. Virtually all
damage I've seen has been due to either EMP or improperly bonded grounding
systems. In particular, if your telecom ground isn't bonded to the electrical
NEC safety ground, you will get a voltage difference between the grounds,
depending upon the voltage gradient in the ground. Whole books have been
written on this subject; I've got one by Polyphaser about nuclear EMP (same
concept, larger scale) protection for radio stations.

Imagine the lightning bolt's ionization conduction channel as the primary side
of many transformers, with every single conductor within many meters being
potential secondaries. The closer the secondary, the more coupling. It's a
1:1 turns ratio, too, and so a 100% coupled secondary would give an equal
amperage through the secondary. Air-core transformers are loosely coupled at
best, but even a tenth of one percent coupling of a 100kiloampere lightning
stroke is 100 amps in magnitude. Loosely coupled current transformers, like
this, tend to generate large open circuit voltages, too.

The most graphic evidence I've seen of the power of lightning-created EMP was
made during a strike I saw in June of 1998 at a radio station's studios. The
studios were in an old, 1950's vintage school building, built to 1950's civil
defense standards for EMP resistance (rebar in a Faraday cage arrangement,
metal roof, lightning rods on the roof). There is a 100 foot studio-
transmitter link (STL) tower at one end of the building. The STL tower took a
direct hit. The Faraday cage rebar verticals embedded in the walls became
coupled secondaries, and large currents flowed.

Every single CRT monitor in the entire 300 foot long building was left with a
rainbow effect on the screen due to the residual magnetism from the EMP. Even
monitors that weren't plugged in were rainbowed. Many PC's died that day, but
I resurrected several hard drives where I could find identical control boards;
no hard drive was unreadable due to magnetic issues, but only electrical (no
bad heads or erased sections on the platters; every one I found a compatible
replacement control board for was recovered).

Made some good money degaussing CRT's that week. (used a bulk tape eraser;
turned on the eraser, brought it close to the CRT, worked it over all
surfaces, then slowing drew the eraser away from the CRT, and turned it off).

The EMP was strong enough that there were a couple of pieces of spare
equipment, located in a room less than 30 feet from the tower, that had
lightning damage even though they weren't plugged in or connected to anything.
One 250MCM ground wire from the tower was vaporized; there were three, and the
other two survived, but with noticeable heat-induced discoloration (they were
replaced, and the glassed-up ground rods were as well). Engineering estimates
of the stroke current were that it was somewhat greater than 200kiloamperes.

One of the STL transmitters was damaged, but on the audio side. Neither of
the two STL transmitters sustained any RF output damage thanks to the sacrifice
of the two daisy-chained Polyphaser arrestors (the arrestors acted as fuses,
and had to be replaced, but they're a lot cheaper than a 950MHz Marti
STL-10!). One of the two four foot Marti STL dishes had a melted feed, but
the other one, which was lower on the tower (about 85 feet up) was undamaged.
Fortunately, neither of the two half-inch heliax runs from the dishes were
damaged.

The 10base-2 LAN took extensive damage, but not every NIC. The most
interesting damage was to the RG-58 cable itself, which had holes blown in it
every 30 feet or so. Made a good argument to upgrade to 10Base-T at the time.

At my current employer, which is a lightning magnet, we use Altelicon AL-
CAT5HPW lightning arrestors on all cat5 installations that go outside a
building. At any building known to have lighting issues, we put one of those
on every cat5 going to the switch (Altelicon also makes four-port versions).
Tripplite also makes cat5 PoE compatible arrestors.

Lightning damage is completely predictable, if you have all the information,
as it's all physics: we just never have complete information, like the
coupling percentage from the primary ion channel to the various potential
secondary conductors. Lightning will take the path of least resistance (which
may not be the path you think), and it will generate EMP, which will create
induced currents. Proper single-point star grounding and bonding of ground
conductors and electrode fields is a must to reduce damage; multiple electrodes
or electrode fields must be bonded, or you will get damage. You may get damage
anyway; depends entirely on the physics of that particular stroke.

Fun stuff, that's for sure.

IMHO,
Long runs of UTP (unshielded twisted pair) make wonderful antenna systems for EMI and EMP which is why they are matched to differential drivers and receivers to reject as much common noise as they are designed to. Older and larger Ethernet interfaces have drivers separated from the logic components that can handle higher over currents and voltages that are induced on the cable. Newer, smaller integral designs cannot usually handle as much power as the older designs.
UTP systems in industrial environments require higher performance drivers that handle higher currents and voltages since they are induced by large motors, HVAC and florescent lighting systems.
EMP (lightening) is a very wide band, high number of frequency bands signal with large amounts of power being induced into the systems they are impinging very rapidly. ITGOD (In the Good Old Days) we used to run everything through conduits which when properly grounded protected both power and signal circuits against lightning very well.

There is a very large wifi network in multiple mile long structures connected by underground tunnels, in the most active thunder storm zone in the country, some of the issues were:

• There were a number of power and grounding zones in the buildings
• There were local grounds at each of the wiring closets that all equipment in that zone was tied to. (Have the earth ground checked and if it they are corroded or no longer working, have a new earth ground field dug or tie to the “building steel” if it has one or more grounding points.)
• All inter zone runs were fiber
• Both PoE switches and local PoE bricks were used to power the remote access points to keep power drops over the utp to within design parameters.
• Some zones had switches with both fiber and PoE ports with the PoE ports handling local access points and the fiber ports running to smaller remote switches with PoE from there to the edge devices.
• If the power runs were too long and their was no local power available, custom cables were manufactured to increase power conductor sizes to lessen voltage drops
• All outside runs were in conduit and would preferably be fiber.

When I installed my first Ethernet with RG-9, I had to ground the cable at the center of the run and tape each end of the cable since it had almost lethal voltage at the either end of the Ethernet cable.

My analog circuits professor said forget this digital design stuff, it as an analog signal in a transmission medi(a)um.

Regards,

John (ISDN) Lee

About a month ago, we had a lightning strike near our main campus. We lost one POE Cisco 3560 completely (apparently blown power supply), and in a separate but nearby building, another 3560 lost the ability to deliver POE, but continued to operate as a switch. Both had to be replaced. Both were on wiring closet type UPS'es with surge suppression, and those were unaffected.

Mark

From: Caleb Tennis [mailto:caleb.tennis@gmail.com]
Sent: Thursday, May 13, 2010 10:37 AM
To: North American Network Operators Group
Subject: POE switches and lightning

We had a lightning strike nearby yesterday that looks to have come inside our facility via a feeder circuit that goes outdoors underground to our facility's gate.

What's interesting is that various POE switches throughout the entire building seemed to be affected in that some of their ports they just shut down/off. Rebooting these switches brought everything back to life. It didn't impact anything non-POE, and even then, only impacted some devices. But it was spread across the whole building, across multiple switches.

I was just curious if anyone had seen anything similar to this before? Our incoming electrical power has surge suppression, and the power to the switches is all through double conversion UPS, so I'm not quite sure why any of them would have been impacted at all. I'm guessing that the strike had some impact on the electrical ground, but I don't know what we can do to prevent future strikes from causing the same issues. Thoughts?

It is not clear to me from the above if there are copper circuits coming into the building, but lightning can certainly zap those as well. In very high impact areas (such as mountaintops or Miami) it is a good idea to mandate that all incoming / outgoing circuits are on fiber, without exception.

Marshall

We had a lightning strike nearby yesterday that looks to have come inside our facility via a feeder circuit that goes outdoors underground to our facility's gate.

Perhaps there was a "move" of the earth-level relative to the neutral line.
I have no idea how neutral-line to earth potential is handled in us, but here in austria we use a so called "nullung".
That means that the earth-ground potential line of the building (which includes also the lightning conductor) is connected to the neutral power line where it enters the building, keeping this potential-difference low.

Theres also a potential between earth ground and the neutral-phase of the online-ups.

The ethernet-cables; utp or stp?
pannels correctly earthed?

Perhaps a electrician should check the earthing.

Also all copper lines that enter the building should be protected by lightning protectors.

Kind regards,
   Ingo Flaschberger

In the US neutral and earth ground are supposed to be bonded only once
at the service entrance. A separate ground from the neutral conductor is
carried to sub-panels where is it not bonded. Additional bonding can
cause weirdness and will turn the ground into a current carrying
conductor. However, an older building I used to be in (built 1978) only
gave me a neutral with bonded subs, so you'll run into all kinds of
stuff depending on the age of the building. Working at a university was
particularly interesting with of the vast range of building ages.

~Seth

In my experience, each building has a building ground-point at the
service entrance, as outlined.

I the problem in a campus on some soils is that building grounds might
be several volts apart--except during thunder storms when the voltage
difference might be (it appears) thousands of volts, and with a
lightning strike to one of them many thousands of volts.

That is why I argue for glass only between buildings. I don't care how
much PoE saves.