PoE, Comcast Modems, and Service Outages

As someone with a family member that is a zebra from a health perspective, I follow up with:

Yes, /look/ for horses when you hear hoof beats, but -- and this is the important thing -- do not rule out zebras.

We've run into too many people who hear hoof beats, assume horses, and proceed as if zebras are absolutely not a possibility.

I second this.

I'd also be willing to get approval for the expense to procure and install the optical isolation equipment in the spirit of good client relations with many of the clients that I've had in the past.

Me: $BOSS, I want to spend $200 as a troubleshooting diagnostic effort to test to see if the problem that $CLIENT is having is related. I think it's not, but I want to rule it out.

Boss: Why haven't we done this yet? Please make it happen.

I'll never forget my first major escalation as a young engineer, on a phone call with multiple angry executives and CCIEs, asking them to try replacing their patch cables.

The problem went away.

K

Their crappy equipment needing rebooting every few weeks, not ridiculous.
Their purchasing gear from incompetent vendors who cannot be standards

    compliant for PoE PD negotiation, tragically plausible.

Many customers buy their own cable modem. You can lease an Xfinity device as well and those function pretty nicely these days but YMMV. But typically a device reboot is a way to quickly solve a few different kinds of problems, which is why techs will often recommend it as an initial step (you can generally assume that there's data behind what occurs when any one of tens of thousands of support reps suggesting something to a customer - support at scale is data-driven).

   He's got graphs showing it every 24 hours? Liar, liar, pants on fire,

    lazy SOB is looking for an excuse to clear you off the line.

Could well be from noise ingress - lots of work goes into finding & fixing ingress issues. Hard to say unless we look in detail at the connection in question and the neighborhood node.

JL

I asked him to remotely reboot the modem because there was high packet loss.

FWIW, as a customer (assuming residential), you can login to the website and check for area outages/impairments at https://www.xfinity.com/support/status-map. You can also use the Xfinity app to remotely reboot your cable modem, run diagnostics/check for outages, etc. See https://www.xfinity.com/support/articles/check-service-outage

Both times I’ve talked with him, he noted the high packet loss, started to reboot the modem, and then asked me point-blank if we had any PoE switches on our network.

High packet loss typically suggests an RF impairment of some type. I don’t know how to explain the PoE comment but am happy to look at your connection if you want to email me off-list.

I said “it’s up and working fine, why would I reboot it?”.

In some cases a reboot will trigger a pull of the latest firmware, which might include security fixes, performance improvements, and other changes.

Jason

I asked him to remotely reboot the modem because there was high packet loss.

FWIW, as a customer (assuming residential), you can login to the website and check for area outages/impairments at https://www.xfinity.com/support/status-map. You can also use the Xfinity app to remotely reboot your cable modem, run diagnostics/check for outages, etc. See https://www.xfinity.com/support/articles/check-service-outage

Both times I’ve talked with him, he noted the high packet loss, started to reboot the modem, and then asked me point-blank if we had any PoE switches on our network.

High packet loss typically suggests an RF impairment of some type. I don’t know how to explain the PoE comment but am happy to look at your connection if you want to email me off-list.

I said “it’s up and working fine, why would I reboot it?”.

In some cases a reboot will trigger a pull of the latest firmware, which might include security fixes, performance improvements, and other changes.

Jason

Packet loss absolutely could be noise ingress. That's measurable in terms of s/n ratio and it's on the outside plant HFC side.

But noise doesn't "build up", and PoE is DC, and PoE is on the LAN side, separated from Comcast's equipment by a router. Level 1 Comcast guy is simply wrong.

> Their crappy equipment needing rebooting every few weeks, not ridiculous.
> Their purchasing gear from incompetent vendors who cannot be standards
    compliant for PoE PD negotiation, tragically plausible.

Many customers buy their own cable modem. You can lease an Xfinity
device as well and those function pretty nicely these days but YMMV.
But typically a device reboot is a way to quickly solve a few
different kinds of problems, which is why techs will often recommend
it as an initial step (you can generally assume that there's data
behind what occurs when any one of tens of thousands of support reps
suggesting something to a customer - support at scale is data-driven).

No one's doubting all of that -- support is best when data-driven, scale
or otherwise. But that's actually the issue here. There's no data that
I know of to suggest widespread PoE ghost current buildups, and, given
the audience here, no one else has popped up with a clear "me too".

PoE is typically negotiated by modern switches, 24v Unifi special
jobbies aside, so it's all DC on cables that are already handling
differential signalling.

> He's got graphs showing it every 24 hours? Liar, liar, pants on fire,
    lazy SOB is looking for an excuse to clear you off the line.

Could well be from noise ingress - lots of work goes into finding &
fixing ingress issues. Hard to say unless we look in detail at the
connection in question and the neighborhood node.

No doubt. There's huge amounts of room for problems to be introduced
into last mile networks. But, again, this isn't about general problems.
This is about a tech claiming it's due to PoE, and that he's seen it
often before.

I certainly have a lot of sympathy for cable techs, but that doesn't
mean I want to swallow any random garbage they want to blame issues on.
Please just tell me it's the chipmunks getting feisty and nibbling on
the copper if you want to feed me a line...

... JG

Hi, Colleagues:

0) I would like to share a personal experience of a different setting to offer an angle for looking into this puzzling topic.

1) During my graduate study, I was doing microwave experiments in the laboratory. On a six foot bench, I had a series (maybe a dozen or so) of waveguide components (made of metal, either solid copper or silver plated) connected together as the test bed. With a half dozen or so equipment attached to them at various points to serve as the energy source or sink as well as detection instruments. The setup exhibited randomly and unpredictable behaviors. After awhile, my advisor brought up the topic of "Ground Loops" that could introduce the interference in mysterious manners. This is a phenomenon whereby minuscule electric current flowing along metallic parts (even though they may appear to have no "resistance" in between) that are connected to the system common potential reference (the "ground") via slightly different paths. These could be very small resistance path between two points or high resistance leakage source. The resultant electric potentials among the subsystems of interest could be significant enough to affect the outputs of sensitive instruments.

2) After much cut-&-try, including plugging all instruments into the same electric extension strip with no avail, I finally floated the AC power cords of all instruments (using three-to-two prong adapters) but kept only the most sensitive node in the system connected to the AC power source "ground". Although this was against the electric safety code, I finally got consistent results. With clear record of the configuration, I even could take the waveguide setup apart and then reassembled days later to repeat the same results. Years later, I applied the same philosophy to a smart-meter PCB design which got a better precision than the chip manufacturer's demon board could.

3) So, it is possible that the site with the reported "PoE induced" issues may be somehow experiencing the above related phenomena. This kind of situations are almost impossible to duplicate at another site. It has to be diagnosed with pains-taking detective efforts, such as inserting isolation subsystems suggested by one colleague. Since this phenomenon takes a month or so to show up, discipline and patience are the virtue.

4) On the other hand, a product that can build up certain "memory" of the disturbances like the above and to the point of requiring periodical power cycling to flush clear the issue is definitely a sign of defect design, based on my old school training.

Regards,

Abe (2022-03-31 10:40)

I fully acknowledge that such things /are/ possible. However I believe they are somewhere between quite and extremely rare.

My belief is that the support technician was using something that's just true enough or possible enough as an excuse to justify something and make it someone else's problem.

This is another reason why I'd go for the $100 optical coupling. Mostly so that I could politely pat said technician on the cheek saying "Yes, that's possible, but it's not the problem we're discussing in this case because of optical coupling / electrical isolation."