power to the internet

https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678

This article details some of the issues with California's "new reality" of planned blackouts. One of the big things that came to light with these blackouts is that our network infrastructure's resilience is pretty lacking. While I was (surprisingly to me) ok with my DSL connection out in the boonies, lots and lots of people with cable weren't so lucky. And I'm not sure how bad the situation is with cellular infrastructure, but I assume it's not much better than cable. And I wouldn't doubt that other DSL deployments go dark when power is down. I have no clue with fiber.

So I guess what I'm wondering is what can we do about this? What should we do about this? These days IP access is not just convenience, it's the way we go about our lives, just like electricity itself. At base, it seems to me that network operators should be required to keep the lights on in blackouts just like POTS operators do now. If I have power to light my modem or charge in my phone, I should be able to get onto the net. That seems like table stakes.

One of the things we learned also is that the blackouts seem to last between 2-3 days apiece. I happen to have a generator since I'm out in the boonies and our power gets cut regularly because of snow, but not everyone has that luxury. I kind of want to think that my router+modem use about 20 watts, so powering it up would take about 1.5kwh for 3 days. a quick google look shows that I'd probably need to shell out $500 or so for a battery of that capacity, and that's doesn't include your phones, laptops, tv's, etc power needs. What does that mean? That is a major expense for a lot of people.

On the bright side, I hear that power generator companies stocks have gone through the roof.

On the dark side, this is probably coming to a lot more states and countries due to climate change. Australia. Sigh.

Mike

Do you have a source for this? It would seem that these power issues are rather unique to California not because of some “climate change” bogeyman, but rather because of a failed public policy at the state level.

It would also seem that these issues of rolling blackouts aren’t even new to California, either, as, apparently, it’s already been the norm during 2000/2001:

C.

On the dark side, this is probably coming to a lot more states and
countries due to climate change. Australia. Sigh.

Do you have a source for this? It would seem that these power issues are rather unique to California not because of some “climate change” bogeyman, but rather because of a failed public policy at the state level.

It would also seem that these issues of rolling blackouts aren’t even new to California, either, as, apparently, it’s already been the norm during 2000/2001:

Having lived through the blackouts that was entirely different. 90% Enron manipulating the markets. There was plenty of capacity both in transmission and generation, but Enron manipulated prices and apparent supply to make money and screwed the whole state over. There was just about 2x the generating capacity, no real shortage.

This time it’s PG&E all alone, but still fallout from back then. Too much liability and they’ve not maintained the infrastructure and so they decided that to reduce the liability costs it’s cheaper to blackout. Same story again different colors. PG&E making a mint while people get screwed (PG&E was mostly at the getting screwed end in 2000-2001)

There's no point in engaging climate deniers. This is what's happening to us right now regardless of what science you choose to call "bogeymen". We are living in fear of becoming the next Paradise.

Mike

Yes, this is exactly right. My point here isn't to assign blame, but to ask what the hell we're going to do about it. Trying to score political points is disgusting.

Mike

Do you live in California? Do you have your business in California? Take a look at neighboring states. I did. California madness is why I now live in Nevada.

Our ecology doesn't have the Austrailian plant Eucalyptus. We do have tumbleweeds which pose their own fire risk. Place like Lake Tahoe is heavily forested -- the difference is that in Nevada there is active fuel control and controlled burns, so we have fewer burn-to-the-ground fires in populated areas.

I used to make a living as a freelance writer. A GOOD living. In Nevada I'm outside the reach of CA AB 5 should I want to give up $DAYJOB. When I have to subcontract freelance work, I won't hire a California resident, because I don't want to be controlled by any "long-reach" laws. Because the law and Calufornia court decisions are currently silent about C, S, and LLC writers, I avoid them until the climate becomes clearer.

Do I experience power outages? Yes. Longest duration? Several hours, when high winds cause damage (but that damage doesn't start wildfires). One very nice thing is that where I am, we have a geothermal power plant close by (on the order of 15 miles). No pre-emptive wide-area shutdowns, though.

NV Energy has photoelectric arrays that are part of the utility, not only on private roof-tops.

The same thing we’ve always done and recommended — Vote With Your Wallet. Move to state that takes care of its infrastructure and doesn’t have such a gridlock. Or remain in California if you think “climate deniers” (whatever that term may mean) are “disgusting”.

As a consumer and internet infrastructure operator, I don’t particularly see or care about the difference between PG&E getting screwed or doing the screwing. It’s the populace of the state that gets the resulting fallout in terms of the rolling blackouts. Which other state has had this in the last 20 years?

I’m an ex-California resident myself here — voted with my wallet already. Love the idea and implementation of an independent power grid of my new home state.

C.

Yes.

Mike

Good riddance. This has nothing to do with the climate change that is actually happening here.

Mike

Well, enjoy your climate change! Surely has nothing to do with the ban of preemptive burning that works in adjacent states just fine!

C

This is a very short term problem.

The market is going to fill with battery storage sooner rather than later.

Solar is just exploding.

Your car will “house tie”.

6G will solve your data problem.

D

Unless telecom infrastructure has been diligently changing out the lead acid battery approach at all their remote terminals, powered gpon, hfc and antennae plants will never last more than minutes. If at all.

A traditional car has between a 100-200amp alternator @12volts

How much generating capacity can you get out of a typical hybrid?

Self-isolating and re-tieing inverters. Economic household ATS systems. Do those exist?

Enough independent distributed capacity and now comes the ability to create grid islands. How might that look?

Electric grid shortage is likely coming to NYC, courtesy of folk of certain political persuasion and their love of stone age era living. IP decommissioning.

If you have CO loop copper, keep it.

Joe

Don Gould wrote:

Same story again different colors. PG&E making a mint while people get screwed

I’m not quite sure that’s an accurate statement.

In 2000-2001, PG&E got screwed by Enron’s market manipulation. ( Good job those who pushed so hard for deregulation of public utility services! )

PG&E is currently in bankruptcy proceedings, largely as a result of liabilities from wildfires in 2017 and 2018. Under California’s application of inverse condemnation, a power utility is responsible for any damage caused by a wildfire if it was determined that their equipment was part of the cause. This applies even if the utility was in 100% compliance with all laws and regulations.

So you have a terrible combination where housing prices in the state are driving more and more people to build in wildfire prone areas, climate change is increasing the frequency of weather conditions favorable to wildfire ignition, and the utility company that is being held financially liable for damages while at the same time not being allowed by the PUC to raise capital for infrastructure changes to reduce the chances of electrical equipment starting such things.

The answer is easy. Money. If people want a power grid that is safe and reliable, then the utility should be given the funds to do it via rates and appropriate tax revenues. They should not be expected to turn profits like private enterprise. The power grid is for the benefit of all, not just the financial benefit of those who have equity stakes.

This situation is the logical extension of 40+ years of America’s only real product ; financial engineering.

I’m pretty sure political bickering is well beyond the scope of the mailing list. Is anyone moderating this?

  • Mike Bolitho

It certainly wasn't my intent or desire to have this turn political, and shame on the person who did. This is a serious networking related issue for California *right* *now*. It may become a serious networking related issue for a lot of other places too -- California is hardly unique in its wildland - urban interface issues, and lots of places burn just like California. And definitely lots of places have a 100+ years of fire suppression which is a policy thing, not a political thing.

The question is what are network operators going to do? If the answer is "nothing", don't be surprised to get legislation shoved down your throats. Don't expect the bay area of all places to passively put up with all of this. If your network fails because of power going out and I can't call 911, you've got a big, big problem.

Mike

As a small WISP operator in Northern California and well into the urban interface we fell victim to the PSPS this year. Thousands was spent on upgrading battery plants that would normally hold during a short outage and generator purchases, whether it be small inverter style generators for small sites to permanent standby site generators for those sites that are larger or a PITA to get to. We still have more work to do and hope to be better prepared for next summers rounds of shut offs. I am currently developing a portable trailer mounted solar/battery plant to replace the portable generators just for fuel cost savings since I spent just about $500/week in generator fuel alone for the largest outage.

If that was a reference to my comments, it was certainly not my intention. I was striving to avoid it being seen as that, but apparently fell short.

To reanswer the question posed though, is still the same ; $$$. If network operators take the position that the electric utility supply should be more reliable than it is, then they need to start influencing and lobbying for ways for that to happen. If not, they will have to increase investments into local generation or storage capacity to bridge those gaps.

You seem to imply that regulation is inherently bad; however the scenario that you describe (power failures impacting 911 service) is only a concern to an operator if there is a legislatively define deterrent.

Yeah, that's the biggest change: the outages are 2-3 days each. Maybe they'll get better but their stated goal is to inspect the entire distribution system before turning power back on, so that is going to take some time no matter what.

And the fuel cost is definitely a consideration. Unfortunately October and November is pretty lousy for solar. Wind might be better since that is the reason they're doing the PSPS in the first place :slight_smile:

Mike

Exactly. And we will build it all.

The power stuff is serious people. We’ve gotten letters from the FCC over it. There is additional regulation coming down when people can’t call 911!

You need at minimum 8 hours (or your CRT response time with a generator trailer, or a standby generator or two) of battery on your telecom equipment. All of it. Everywhere.

Comcast is the worst about this, they never replace and often don’t even place batteries in their RTs at all - and they are going to get fined over it mark my words.

If that was a reference to my comments, it was certainly not my intention. I was striving to avoid it being seen as that, but apparently fell short.

Not directed at you at all.

To reanswer the question posed though, is still the same ; $$$. If network operators take the position that the electric utility supply should be more reliable than it is, then they need to start influencing and lobbying for ways for that to happen. If not, they will have to increase investments into local generation or storage capacity to bridge those gaps.

You seem to imply that regulation is inherently bad; however the scenario that you describe (power failures impacting 911 service) is only a concern to an operator if there is a legislatively define deterrent.

Not at all. I'm saying that this problem will be solved one way or the other. Frankly it's surprising that anybody offering telephony service has gotten away with not fulfilling the battery backup mandate. I guess there must have been some wiggle room that the carriers took advantage of. And if so, legislation to fix that will be immanent.

Mike