PoE, Comcast Modems, and Service Outages

Mailman List Mirror (Read Only)
1

I just got off the phone with a Comcast tech, and wanted to double-check my sanity.

Somehow in the last 6 months I’ve managed to reach the exact same rep twice when dealing with an outage or a degraded service event.

I asked him to remotely reboot the modem because there was high packet loss.

Both times I’ve talked with him, he noted the high packet loss, started to reboot the modem, and then asked me point-blank if we had any PoE switches on our network.

When I said “yes”, he said I needed to disable PoE because it messes with the Comcast modems and he can see “buildups” in his graphs that show power is “leaking” to the Comcast modem every 24 hours.

For reference, our setup is:

Internal Network ←→ PoE Switch ←→ My Router (FreeBSD Box) ←→ Comcast Modem

I told him the Comcast modem isn’t plugged into the PoE Switch, it’s plugged into My Router (FreeBSD box) and My Router does not negotiate PoE+ and the switch shows PoE isn’t being send to My Router’s LAN port. While the switch is capable of outputting old-school 24v PoE, it must be specifically turned on for a port, and it’s not enabled or used anywhere on the networks I manage.

When provided with that information, the Comcast tech still insisted that the switch was sending PoE to My Router and it was “leaking through” to the Comcast modem and that’s why every 4-6 weeks the Comcast modem needs to be reset. The tech insisted that switches that are PoE-capable always send PoE even if the device doesn’t request it or negotiate it. Attempts to explain the difference between the old 24-volt PoE and PoE+/++ were met with arguing that he’s been in the industry for decades and I don’t know what I’m talking about…and that all my problems would go away if I just disabled PoE everywhere on the switch.

Again, I double-checked the port and said “It’s not sending PoE to my router, but even if I were, I highly doubt PoE would leak through a PCI card to the opposite side of the chassis to the on-board NIC and out to your modem”.

He insisted it happened “all the time” and he had previously fried equipment by plugging it into a PoE switch. He insisted that he’s also handled quite a few calls relating to this magic PoE problem over the years and Comcast has internal tools that show graphs of how much PoE power “builds up” inside their modems and he “can see a buildup in my router that resets every 24 hours”.

I didn’t have the heart to tell him that I manage about 40 networks that have Comcast connections…and they all have identical FreeBSD boxes acting as their router, and they are all using the exact same PoE switches at every location with all ports set to PoE+…and we only have degraded service or outages after ~30 days at ~3 locations.

Slightly off-topic, but if I call Comcast about outages or degraded service and any other tech but this guy answers, they all say “you need to unplug your Comcast modem and plug it back in once every 3-4 weeks” and they act like it’s normal to reboot the modems every few weeks. In fact, last week I wanted Comcast to check on a modem setting at one location and they said the modem had been up for over 127 days and it should be rebooted. I said “it’s up and working fine, why would I reboot it?”.

Anyways, am I insane for thinking the tech was flat-out wrong? I mean…occasionally some really bizarre stuff happens in IT…but this seems extremely far-fetched and contrary to everything I know about the PoE standard.

-A

2

That's ridiculous, as you already know.

Their crappy equipment needing rebooting every few weeks, not ridiculous.

Their purchasing gear from incompetent vendors who cannot be standards
compliant for PoE PD negotiation, tragically plausible.

Ethernet electronic differential signalling not being handled properly
with respect to grounding or other issues, not unheard-of.

Ghosts of PoE floating around a network through other devices, causing
weird problems on the far side of properly installed and standards compliant
gear, ah, super-unlikely, I'll go so far as to say nah, but in this audience
I am positive someone has a counterexample SOMEWHERE, cuz, you know, the
world's a strange place and there's broken stuff out there.

He's got graphs showing it every 24 hours? Liar, liar, pants on fire,
lazy SOB is looking for an excuse to clear you off the line. Where the
heck does this "24 hour" cycle even come from? What SNMP OID is there
for "ghostly PoE build-up"? What crontab is there that would clear out
such buildups in the router's daily run? What capacitor would store up
juice for precisely 24 hours? What's the mechanism here? CURIOUS MINDS
WANT TO KNOW!

Been doing PoE everywhere for years and this is the stupidest thing I've
heard this year so far in the networking category.

Next time, play dumb. People who can, do. People who can't, tech support.
Worth remembering.

... JG

3

We’ve routinely seen where lines not even connected to the same circuit in any way (ie an OTA antenna coax line and cat5 POE) cause issues with one another. As much as we would all love to have a perfect line in the sand, there isn’t. Don’t rule anything out until the issue is resolved.

As someone that sees this in the field and watches people simply hate on someone because there’s a frustrating situation, it’s worth taking a breath before too upset.

4

Both times I've talked with him, he noted the high packet loss, started to reboot the modem, and then asked me point-blank if we had any PoE switches on our network.

This sounds like a guy who has created his own script for 'improving' his call resolution stats. Lucky you!

When I said "yes", he said I needed to disable PoE because it messes with the Comcast modems and he can see "buildups" in his graphs that show power is "leaking" to the Comcast modem every 24 hours.

I just choked on my ice tea reading that.

For reference, our setup is:

Internal Network ←→ PoE Switch ←→ My Router (FreeBSD Box) ←→ Comcast Modem

I told him the Comcast modem isn't plugged into the PoE Switch, it's plugged into My Router (FreeBSD box) and My Router does not negotiate PoE+ and the switch shows PoE isn't being send to My Router's LAN port. While the switch is capable of outputting old-school 24v PoE, it must be specifically turned on for a port, and it's not enabled or used anywhere on the networks I manage.

Unifi/EdgeSwitch?

Yeah, you know when 24v passive POE is turned on because it kills the port on the other end that aren't designed to handle it. Your router would likely have a dead eth port on it.

When provided with that information, the Comcast tech still insisted that the switch was sending PoE to My Router and it was "leaking through" to the Comcast modem and that's why every 4-6 weeks the Comcast modem needs to be reset. The tech insisted that switches that /are/PoE-capable /always/send PoE even if the device doesn't request it or negotiate it. Attempts to explain the difference between the old 24-volt PoE and PoE+/++ were met with arguing that he's been in the industry for decades and I don't know what I'm talking about...and that all my problems would go away if I just disabled PoE everywhere on the switch.

Last time I checked, ethernet ports require magnetics for normal operation, so they're isolated anyway.

And, you are right on the POE negotiation - there is no POE until certain conditions are met on the line to signal to the switch that the device requires POE.

Yes, in theory there could be interference if POE was enabled, it's not likely these days, and there's no interference from POE if its not been negotiated.

Anyways, am I insane for thinking the tech was flat-out wrong? I mean...occasionally some really bizarre stuff happens in IT...but this seems extremely far-fetched and contrary to everything I know about the PoE standard.

Nope, not insane. Tech is full of it and really needs to be retrained properly by a supervisor. I'd be asking for said supervisor next time you call in and get him.

5

You can run cable lines next to A/C wiring and get problems too. Or
ethernet lines next to A/C wiring. That does not justify wild claims
about PoE such as what this tech was making, and until someone shows
me a graph of "PoE buildups" observable via SNMP or whatever the
cable company is using to graph trends, it seems pretty clear that
this is a bogus answer.

There's a lot of difference between "we observed this very specific kind
of interference related to PoE in a particular circumstance" and the
crazy generalizations being made by the tech. Asking to please make sure
your switch is grounded properly? That'd be good. Asking for PoE to be
disabled on the port? Yeah fine. Suggesting separation of cables?
Sure. Checking for proper grounding of the ground block (on the cable
inlet)? Sure. There's room for things to happen.

I'm all for investigating with an open mind, but I draw the line at crazy.

Given that so much of the world works on PoE, it seems like the other
potential resolution would be to note that there's an implication here
by the tech that Comcast's hardware is standards noncompliant and ask
them what they plan to replace their cheap CPE with.

... JG

6

There’s a certain manufacturer of TDD radio where the CPU clock is at the same frequency as what Verizon’s enodeB will transmit. Even at miles away, it can and will cause PIM issues. Again, don’t rule it out.

Maybe he’s just looking for a simple answer that 99% of callers will accept and it makes them happy. When a customer of mine tells me they think it’s something and I know it’s off, I just let them believe in their statement. There’s no reason to go after this tech and insult him, all that’s doing is making everyone miserable.

7

There's a certain manufacturer of TDD radio where the CPU clock is at the
same frequency as what Verizon's enodeB will transmit. Even at miles away,
it can and will cause PIM issues. Again, don't rule it out.

I'm not ruling anything out, but on the flip side, here in this group
of professional networkers, you'd think lots of people would have piped
up by now with "me too"'s if PoE ghosts killing cable CPE on a 24 hour
cycle were a common thing.

Maybe he's just looking for a simple answer that 99% of callers will accept
and it makes them happy. When a customer of mine tells me they think it's
something and I know it's off, I just let them believe in their statement.

I'm unclear on how this is making the caller happy.

I'm trying to envision under what circumstances a customer site that has
purchased PoE switches, presumably to power PoE gear, would be delighted
to hear that their not-directly-connected PoE gear would need to be
removed, presumably replaced, and then, what? Run extension cords and
bricks to all the access points, IP phones, cameras, door terminals,
and other PoE-powered gear?

There's no reason to go after this tech and insult him,

I'd agree it isn't sporting, but on the other hand, a poster here asked
for an evaluation. I did not immediately blow it off, but instead
tossed out some thoughts for consideration. Then blew it off. But I
am still pondering the issue.

all that's doing is making everyone miserable.

I am guessing lots of people laughed. It's an El Reg grade tale of
woe. I have to assume the poster who asked is frustrated but trying
to resolve a real issue.

So if you want the $100 test to eliminate PoE electrical effects, get
a pair of media converters and run fiber between them. Put the CPE on
the far end. Optimize as appropriate if you have SFP-capable switches.

... JG

8

Just to be clear Josh, I’m not insulting him.

I find the situation extremely difficult to believe based on my (possibly incorrect) understanding of how PoE works and very (very!) basic knowledge of things like RF interference—especially when it comes to Cable networks.

I mean, the call literally went like this:
“Thank you for calling Comcast this is , how can I help you?”
“Hey, can you remotely reboot the modem on account 12345? We’re seeing high packet loss and latency starting about 10 minutes ago.”
“Yeah…uh…do you have a PoE switch at that location?”

When you hear hoof beats, look for horses, not zebras. As a first troubleshooting step, I certainly wouldn’t jump to “it’s PoE”. Granted, I have no idea if Comcast has “PoE Buildup” graphs in their internal tools, but based on my conversations with tons of other Comcast reps about tons of other Comcast connections and never hearing one of them mention those graphs, I’m leaning towards him lying through his teeth.

Lastly, the reboot of the Comcast modem “fixed” the issue.

I saw one of the IT guys from another office in the complex a few minutes ago and he said their internet had problems at the same time. Comcast has been out to the equipment room in the facility ~5 times over the last few years to “adjust” things…so I’m still leaning towards this being something more common like faulty equipment, bad signal levels, etc…and not “It’s because you have a PoE switch”.

-A

9

Unifi/EdgeSwitch?

Yeah. Unfortunately. USW-24-250.

Yeah, you know when 24v passive POE is turned on because it kills the
port on the other end that aren’t designed to handle it. Your router
would likely have a dead eth port on it.

I’ve never tested it with one of my routers, but a tech did accidentally test it on a UniFi AP. I’m still not sure how he ignored the warning about sending 24 volts down the line, but the WAP didn’t like it and decided to refuse to work with anyone ever again. :wink:

-A

10

Obligatory relevant xkcd: xkcd: Tech Support

11

Sure–that would shoot down the “leaking non-existent PoE across a motherboard and out another NIC” theory, but I was more thinking along the lines of something like PoE causing RF interference or something.
I mean it’s DC not AC so…it wouldn’t be putting out a modulating signal that interferes? …honestly that’s outside my knowledge domain.

-A

12

As a small DOCSIS operator, I suppose my sample set isn't large enough to be significant but I can tell you PoE has never been an issue with our modems.

We've had a significantly higher failure rate due to things such as customers driving nails through the modem in an attempt to mount them to walls, and concerned citizens shooting birdshot into our overhead distribution in an attempt to curb the rodent population.

K

13

When I said "yes", he said I needed to disable PoE because it messes with the Comcast modems and he can see "buildups" in his graphs that show power is "leaking" to the Comcast modem every 24 hours.

This guy must have a side business selling crystals, or those Faraday cages to contain the bad type of WiFi or 3/4/5G radiation, or some other scam.

* jgreco@ns.sol.net (Joe Greco) [Tue 29 Mar 2022, 20:52 CEST]:

People who can, do. People who can't, tech support. Worth remembering.

I think this smearing of an entire line of work is uncalled for.

  -- Niels.

14

Bingo ^^^

As a few others have said, I’d be more inclined to feel like this was a real thing if the tech had mentioned checking grounding, etc and not just said all the PoE has to go away for the modem to function correctly. If it is truly a power issue there are a bunch of ways to isolate/troubleshoot/fix it.
Thanks,
Matt

Matt Ertle
Manager - Network Operations
Eastern Shore of Virginia Broadband Authority
(w) 757 414.0304
(f) 757 656.7066
mertle@esvba.com

15

    Unifi/EdgeSwitch?

Yeah. Unfortunately. USW-24-250.

Oh, I have quite a few of them in service. They work great in my experience as long as you don't shove them in hot closets or cabinets and cause them to overheat. Also, firmware has gotten a bit better over the years.

I have a lot of background in Ubnt stuff, so I'm a bit biased here.

    Yeah, you know when 24v passive POE is turned on because it kills the
    port on the other end that aren't designed to handle it. Your router
    would likely have a dead eth port on it.

I've never tested it with one of my routers, but a tech did accidentally test it on a UniFi AP. I'm still not sure how he ignored the warning about sending 24 volts down the line, but the WAP didn't like it and decided to refuse to work with anyone ever again. :wink:

Yup, that's normal if its a more recent AP that wasn't designed for 24V passive (pretty much anything recent AP wise).

I've seen techs destroy ethernet ports on brand new laptops doing that. Very expensive lesson :slight_smile:

16

Taken at face value, I assume the tech would be looking at historical signal graphs (we keep them for cable networks for each CM) that record stats like FEC, SNR, and signal strength. For aerial runs it's common to see some change throughout the day due to warming and cooling. These look like waves with peaks and valleys around 4PM/4AM and generally affect all customers in a service area equally. Sometimes there will be a device at a customer premise that causes interference with a CM, something like a motor or tool. These could absolutely be on a 24hr cycle (think of a programmable thermostat kicking on the blower fan in your HVAC at the same time every day).

As Joe said, there's no SNMP MIB for PoE buildup. There are well documented MIBs for DOCSIS to cover standard signal level, quality, or similar. The cause of that signal strength or quality can be myriad. This Comcast tech has likely climbed the ladder of inference several steps too far.

17

Thanks Blake,

As I understand it all that stuff is on the “cable provider” side of the CPE and (within reason) it’s up to the provider to deal with the signals arriving on the cable side of the modem.
i.e. if it was a blower or something in our suite that was causing RF interference, the provider might work with us to move the modem or the cable run.

-A

18

Generally anything inside the customer premise (including wiring) is the customer’s responsibility. If your coax runs across a fluorescent light fixture, that’s on you. If your coax is RG59, it’s on you to replace with RG6 quad shield.

Maybe the cable operator will work with you, maybe not. I’ve had some techs replace splitters or make custom length user cables.

And yes, the RF stats I mentioned are all on the HFC interface of the cable modem (or the CMTS). Customer facing Ethernet stats may not even be tracked by Comcast.

–Blake

19

Hi,

So if you want the $100 test to eliminate PoE electrical effects, get
a pair of media converters and run fiber between them. Put the CPE on
the far end. Optimize as appropriate if you have SFP-capable switches.

But now the modem will suffer from excessive gas...

"Did you hear that?" -"What?" "That's your modem, farting. Time to reboot again!"

Thanks,

Sabri

20

he said I needed to disable PoE because it messes with the Comcast
modems and he can see "buildups" in his graphs that show power is
"leaking" to the Comcast modem every 24 hours.

revealing the critical failure with comcast support; they do not share
what they are smoking

randy