Comcast outages continue even in areas with PG&E power restored

Mailman List Mirror (Read Only)
1

The FCC asked a half-dozen carriers about their network resilience plans last month. Comcast was not one of the service providers askedd about their plans.

The FCC should have looked closer at Comcast in California. While it was expected many people would loose home Internet, voice, video service when their Customer Premise Equipment lost power. The FCC no longer requires battery backup for CPE. That is now a customer responsibility.

It turns out, Comcast's outside plant was woefully unprepared to handle long, i.e. 24 hour, power outages. And even when power is restored to people's homes, Comcast service is often still down.

https://www.ktvu.com/news/still-no-relief-pge-shutoffs-disrupt-comcast-services

Maybe the FCC needs to ask about cable system Outside Service Plant network reslience of Comcast. Are other cable OSP resilience similar?

2

So I knew that telcos are required to battery backup pots, but are isp's too? I have a dinky little provider who also provides pots, but i have never been clear whether dsl stays up too in a blackout.

Mike

3

First of all DSL is not pots.

Traditional voice services run on a subscriber loop which is a pair of copper lines running from the central office to the customer end point.

This analog voice service is almost always backed up with a bank of batteries so that the service continues to run in the event of an emergency.

DSL is a data service that runs on the subscriber loop at the same time as the voice service. This service is not required to be battery backed and will invariably stop working when power is cut at the customer end point.

Ted

4

Why you don't have Comcast service during a power outage:

Throughout the state, Comcast equipment was knocked offline by PG&E’s power shutdown, Hammel said. The cable company was “only using generators in very discrete and specific cases where there’s a demonstrated need,” such as a request from the Federal Emergency Management Agency, she said.

5

And this is why the distributed nature of small node’s is detrimental in an extended power outage.

There is no practical way to back them up with power for an extended period of time.

6

How distributed is the power on a typical HFC system in practice? I'm sure I'm missing some of them, but having walked out most of a small-ish (~2000 residences) city recently for a FTTx deployment, I think I only saw 2-3 power nodes on Comcast's plant. There were several times as many fiber-coax nodes being line-powered off the coax plant though still surprisingly few (the plant is ancient and hasn't seen a lot of fiber overbuild).

That's comparable to how many powered RTUs the LEC had in town and many fewer utility-powered field nodes than would have been present if it had been in AT&T Lightspeed territory.

Now, I have no idea what the backup line power is in practice on that Comcast plant. I know that Bright House/Spectrum, in an another area I've supported, has very little backup on many residential-only parts of their plant. I've observed that they have, in practice, maybe 15-30 minutes of hold-up before DOCSIS nodes start dropping.

7

Once upon a time, Sean Donelan <sean@donelan.com> said:

It turns out, Comcast's outside plant was woefully unprepared to
handle long, i.e. 24 hour, power outages. And even when power is
restored to people's homes, Comcast service is often still down.

When I had Comcast in Huntsville, AL, there appeared to be no backup
power in their plant. Any power blink and my Internet and TV both
dropped (my equipment is on UPS).

8

It’s very difficult to properly build a resilient infrastructure when those shareholders must get their value!

9

Of course generalizing all service providers isn't fair... From my experience tracking telecommunication during disasters for the last 20-30 years... Let me generalize (ignoring special goverment priority systems).

Generally, during natural (and man-made) disasters:

Telecommunication service providers historically failed in this order

1. VSAT/DTH/satellite (during weather events)
2. Cable
3. Cellular/wireless
4. Telco/Wireline
5. Broadcast radio/TV (less than 20% over-the-air stations operating)
6. Network backbone systems (inter-city and toll offices)

There are too few WISPs for reliable predictions. I'd guess WISPs reliability is similar to cellular/wireless systems.

Restoration order is a bit different. Telecommunications network service historically recovers in this order, assuming customer premise isn't damaged:

1. VSAT/DTH/satellite (after weather clears)
2. Network backbone (inter-city and toll offices)
3. Cellular/wireless (COWs and COLTs deployed)
4. Broadcast radio/TV (20% over-the-air stations operating)
5. Telco/Wireline
5. Cable

Cable systems tend to be the first to fail, and the last to be restored.
Telco systems tend to fail later, but take a long time to be restored.

Network backbones can take a while to repair, but generally nothing else works until they are repaired, so they get repaired first or second.

Note: During even the worst catastrophes, there is almost always one or two broadcast radio stations still operating. I set 20% radio/TV stations operating as an arbitrary minimum level. Likewise, COWs and COLTs don't provide full cellular service, but do provide minimumal cell services.

In the last 10 years, cellular/wireless system resiliance has been improving while telco/wireline system resiliance has been getting noticablly worse. I assume this a flywheel affect as telco companies have been shifting infrastructure investement to wireless networks and away from wireline networks for the last 20 years.

10

This is why I’m concerned about a future with micro-cell sites, we will have a few fortresses with cell service but the day-to-day service won’t look anything like the outage service and is likely to fail catastrophically as a result.

- jared

11

In article <c4faaa6b-b6d0-0452-470f-cd37076a91fe@monmotha.net> you write:

How distributed is the power on a typical HFC system in practice? I'm
sure I'm missing some of them, but having walked out most of a small-ish
(~2000 residences) city recently for a FTTx deployment, I think I only
saw 2-3 power nodes on Comcast's plant.

I spend too much time looking up at the power lines while walking the
dog and around here, Spectrum ex-Roadrunner, on just about every block
I see something on the cable plant with an electric meter. I can't
tell how many are amplifiers and how many are fiber to coax adapters.
None have any evident batteries although I suppose there might be some
in the cabinets.

My current phone and Internet service is FTTH from the local RLEC.
The box they installed here is powered from a 12v UPS and I'm
reasonably sure there are no active components between here and the CO
since it's only 1/4 mile away, so I'll be interested to see how it
does when the power goes out.

12

First of all DSL is not pots.

[....]

DSL is a data service that runs on the subscriber loop at the same time as the voice service. This service is not required to be battery backed and will invariably stop working when power is cut at the customer end point.

That is not why people are surprised. When the house doesn't have power, and doesn't have home generator or UPS, (most) people are less surprised their DSL or Cable modem and VOIP doesn't work anymore.

The reasons I saw people angry on twitter was no Comcast service even when they had power at the house (utility, generator, UPS). Their Comcast service died quickly, even when the home had power but the Comcast outside plant didn't seem to have any backup power.

DSL modems also need power at the home, but the telco providers seem to have more backup power in the outside plant or central offices. That meant DSL worked as long as the house had power (or a home generator or UPS).

I know, rich people problems. Rich people can afford backup generators and got upset when their Internet and TV didn't work.

13

So it turns out that our local telco/isp does keep dsl running via the same mechanism as they keep pots power backed up (i'm guessing it's a diesel generator at the co, but am not sure). It seems that a lot of the pedestals terminating the local loop these days do the conversion to IP right there with sip/h.248/mgcp/rtp. I'm not sure how they get power to the pedestal, but these were all a home run to the co at one time so it probably wasn't hard to power them from the co. For all i know, that's how they're all powered all the time, with a transfer switch at the co, rather than tapping the local grid next to the pedestal.

Of course this is a lot of conjecture on my part... be glad to be clued in by folks in know.

Mike

14

An old news story, but telco's usually have backup batteries in their outside plant, cell towers, etc. During power outages, they shuttle small generators between outside cabinets to re-charge the batteries. Remote Terminal Units (RTUs) use local power, i.e. look for the utility meter nearby. There is often a generator plug and battery cabinet next to the RTU. They aren't powered from the central office.

Some cable systems have battery and/or generator backup on their "I-Net" cable plant serving government and major businesses, but not on their residential cable plant. I don't know Comcast's business practices.

Old news story:

https://www.multichannel.com/news/att-will-replace-batteries-after-fires-130936
ORIGINAL: JAN 18, 2008

AT&T Will Replace Batteries After Fires

City officials, long critical of the size and placement of powering cabinets needed to back up AT&T’s U-verse TV video service, now have concerns beyond aesthetics. Sometimes, the cabinets explode.

AT&T acknowledged the problem and said it would replace 17,000 lithium batteries in outdoor cabinets around the country.

[...]

The steel cabinets house controls and backup power supplies for the video network.

“They’ve been pretty cooperative,” Kesner said of AT&T. “We’re in a holding pattern” regarding the video deployment, he said.

[...]

15

Interesting! And so primitive! So they go to all of the expense of laying fiber, but not power too?

Mike

16

Legacy carriers with existing copper plant being overbuilt by fiber often use "span power" at 48V or ~190VDC from the nearest central facility (which may be an old-school CO or RTU with utility power) to power their distributed plant that's deep into the network near the customer edge. There's usually then also some local battery as well but not much. The assumption is that the legacy copper, now being used simply for power, follows essentially the same routes as the fiber and will drop at the same time. In a cable cut situation this is often true, but in a "cable rot" situation it's obviously not, and fiber paths aren't always the same as power paths.

The CO and older RTUs from the POTS and early ADSL era will have utility power as primary with somewhat extensive battery facilities. Old-school CO will have a lot more battery capacity than an RTU. An RTU will usually end up with a portable genset being delivered during an extended power outage as mentioned. These RTUs still host legacy POTS and TDM services that either have serious SLAs on them or regulatory uptime considerations whereas distributed peds are normally best-effort, non-regulated data services only in many cases precisely to keep the reliability requirements (and therefore cost) on them down.

SBC/AT&T's pedestals they built for Lightspeed (U-Verse) do usually have local utility with a few hours of backup. I'm not sure why the went that route rather than span power. Might be that the early VDSL DSLAMs just used too much power for that. They seem to only have maybe a day or so of battery before they need a portable generator brought around which AT&T at least has procedures for (whether they are executed or not is another matter). They're something of a hybrid between a conventional RTU and modern distributed pedestal.

All the conventional telcos are far more focused on keeping voice service alive since they get raked over the coals by the FCC if it drops due to lack of 911. That includes wireless if they are both a wireline and wireless operator. Interestingly, VoIP service delivered to the customer as such (even if there's an ATA built into the customer's "modem") often gets a pass on this since it's not considered POTS. Same goes for SLA'd business services, too especially T1s since those may host regulated voice.

Coax operators have historically had less need for reliability as they were originally built purely for convenience services (cable television) and have been pressed into service for more modern data needs. Those "Alpha boxes" you may see around providing line power do usually have some batteries in them, though they're often ill-maintained and only provide maybe an hour of hold-up at most.

Exclusively residential areas sometimes have zero hold-up ability at all. They'll drop at least outside node based digital services (e.g. DOCSIS) as soon as power falls along with anything being distributed into the field on AM fiber carriers. It's not unheard of for conventional linear TV to still be delivered into the field from a head-end at baseband on coax and sometimes that'll stay up longer depending on RF power/split budget as long as the local RF head end still has power.

Pure-play fiber carriers, especially PON-based, get to turn what is often a curse in terms of design into a blessing, here. They usually have almost no active outside plant or, if they do, it's less distributed and can afford reasonable backup power infrastructure. While it's annoying to have no power available as you approach customer prem, that forces you to make choices such that there's no worry about backup until you can (usually) just make it the customer's responsibility right at the demarc. Your CO/head-end needs backup, of course, but that's usually a facility that can afford it.

17

All true telco equipment is powered by batteries. Commerical power or generators just recharge the batteries. No switch over when commerical power is lost. Except when the generators(where equiped) switch over to recharge the batteries.

Comcast and telcos do not put batteries in all remote powered terminals. I have an Enterprise grade Ethernet over coax connection. The headend it's distributed from doesn't have batteries. If it loses power, doesn't matter if I have power or a generator or ups to take over. This Internet connection goes down.

For telcos(when I worked there), they usually had batteries that would last 4 to 8 hrs at remote terminals with powered equipment. And a connection for a splice crew to come out and connect their generator to it for power in case of an extended outage. Back then that was also how most cell phone towers were outfited.

I also have some knowledge of the commerical power grid in my local area. It's not unheard of for the Comcast headend to lose power but my office doesn't.

Lyle Giese
LCR Computer Services, Inc.

18

So when we were working on this 20 years ago at Cisco, there was a tremendous amount of effort to deal with the issue of e911 and generally battery backup. I'm really surprised to hear that though we went through a lot of effort to deal with the CPE, that the cable plant was the actual problem. The cable companies should, imo, be held to the same standard as the telcos. Maybe even moreso these days since IP has taken over everything. The need for reliable e911 hasn't gone away just because the bits have turned into IP bit these days.

Mike

19

They get around it, at least in part, by selling it as a "VoIP" service rather than "phone service".

AT&T does the same with U-Verse voice. You can still buy POTS from AT&T, but it's a separate product with a completely different pricing structure from the U-Verse voice product.

Voice over HFC networks is sometimes sold as a POTS-like service. I've only heard of this happening in places where the LEC and cable provider happen to end up being one-in-the-same. In those cases, yeah uptime is a big deal.

I think what happens is that the standards get written and equipment designed with the assumption that everybody will be deploying all sorts of SLA'd, guaranteed services, then 99% of deployments end up being exclusively best-effort because it's so much easier and cheaper to deploy.

GPON seems to be an interesting case of this since it's commonly deployed by telcos rather than cable MSOs, and, in greenfield applications, is often deployed exclusive to copper plant at all. It's pretty common to find GPON ONTs with inbuild UPS monitoring and communications as well as ATAs designed to deliver POTS-like service, but then a lot of SPs who are NOT the LEC of record just use that infrastructure to deliver VoIP-like services and push the UPS responsibility off onto the customer.

20

Oh, but they are equal. The telco's went to the regulators and got the FCC and state PUCs to reduce or make backup power a customerresponsbility...
Just like the cable companies.

So now they are equal -- in the race to the bottom.

Service providers must "OFFER" customers an OPTION for 8/24-hour standby backup power. The decision to puchase backup power is up to the customer.
I assume you read the fine print on the back of your bill or the order terms (subject to change at anytime, without notice).

The FCC is looking at standby power for cellular towers, but hasn't been paying attention to wireline and cable systems outside plant power.

As I mentioned a few postings ago, cellular/wireless systems have been getting more resiliant. Wireline/cable systems have been getting less reliable.