The FCC has released its report and analysis of Hurricane Michael impact on communications: preparation, effect and recovery.
https://www.fcc.gov/document/fcc-releases-report-communication-impacts-hurricane-michael-0
Conclusions and Recommendations
51. Backhaul outages loomed large as an impediment to communications recovery. Uncoordinated post-storm recovery efforts between and among communications, utility, and debris removal teams created unnecessary delays to a speedy return to service. Customers who had communications service restored – only to lose it again almost immediately because of a fiber cut – provide a clear example of how better cross-sector coordination could have improved the restoration process.
Trying not to get political, here goes…
Something important to keep in mind: The current administration has been getting slammed for their lack of response in the aftermath of Michael since the hurricane hit. A lot of that criticism revolves around communications infrastructure and FEMA’s lack of assistance. The current administration has, time and time again, used federal agencies (specifically their presidential appointees) to defend the administration’s actions or inactions. I have read the full report and it is more or less a thinly veiled hit piece. I’m not going to link them here (they are easy enough to find via Google) but there are several very good articles written by reputable tech journalists that go into greater detail responding to the report. Worth checking out.
I say all of that because most of us like to hate on telecom companies (many times rightly so) but I don’t think they are entirely to blame here. There’s nothing Verizon or AT&T can do if their backhaul is cut by a tree or some third party clean up crew. The report is a gross oversimplification of how telecommunication infrastructure works. I think anyone here that has ever worked a storm like this can attest to the complexity and difficulty you run into during recovery. Hanlon’s Razor and all but this is the FCC and I would hope they would know better.
Speaking specifically to point 51, it’s impossible to coordinate between the thousands of crews working to clean things up and repair physical infrastructure after a massive storm like this. Many of the people doing physical cleanup are volunteers that are fully independent of any governing body or company. It is not a telco’s responsibility to know when and where those crews are working. Further, even if those crews we’re calling in and letting each telco know exactly where they were, what does that provide other than an impossibly large and fluid dataset to parse for any meaningful information.
This is what I tell outage complainers during natural disasters, such as the fires in California that recently took out a lot of power and communications:
“Stop whining about how long it is taking to repair your Internet, your cell phone service, or your cable TV. You didn’t pay anything extra to recover from natural disasters, and none of us in the field are getting paid anything extra to restore your services.
No, we don’t know how long it will take. It takes what it takes. That you don’t get instant gratification doesn’t make us incompetent. It makes you ungrateful.
It’s a natural disaster. These are not scheduled. Your outage is nobody’s fault. We don’t have a duty to mitigate all conceivable failures.
It takes time to repair. We’re not cheating you, or loafing around. We don’t owe you any special attention because of your status or reputation.
So quit whining and be thankful you’re alive, and hopefully you haven’t lost too much. Maybe pitch in and help those who have.“
I also send this to ignorant journalists and grandstanding politicians.
-mel via cell
This webinar may be of some interest to those in this group:
https://www.fcc.gov/small-rural-communications-provider-network-resiliency-webinar
Here’s some additional color commentary on the FCC’s concerns:
https://urgentcomm.com/2019/05/10/backhaul-problems-disjointed-recovery-efforts-key-causes-of-unacceptable-extended-wireless-outage-after-hurricane-michael-fcc-report-says/
"“Uniti Fiber (Uniti) provides backhaul services to Verizon Wireless in Bay and Gulf Counties. Uniti indicates it experienced at least 33 separate fiber cuts during the recovery effort. These fiber cuts included damage to sections that already had been repaired. Commenters attributed fiber cuts to debris-removal crews, power-company restorations, and returning homeowners clearing their property.”
One of my takeaways from that article was that burying fiber underground could likely have avoided many/most of these fiber cuts, though I’m not familiar enough with the terrain to know how feasible that is.
Frank
In Florida, especially the panhandle, it’s not possible to bury it. The water table is way too high.
I suspect that may not be possible in (parts of) Florida.
However, even in places where it's possible, fiber installation is
sometimes miserably executed. Like my neighborhood. A couple of
years ago, Verizon decided to finally bring FIOS in. They put in the
appropriate calls to utility services, who dutifully marked all the
existing power/cable/gas/etc. lines and then their contractors (or
sub-sub-contractors) showed up.
The principle outcome of their efforts quickly became clear, as one
Comcast cable line after another was severed. Not a handful, not even
dozens: well over a hundred. They managed to cut mine in three places,
which was truly impressive. (Thanks for the extended outage, Verizon.)
After this had gone on for a month, Comcast caught on and took the
expedient route of just rolling a truck every morning. They'd park at
the end of the road and just wait for the service calls that they knew
were coming. Of course Comcast's lines were not the only victims of
this incompetence and negligence. Amusingly, sometimes Verizon had to
send its own repair crews for their copper lines.
There's a lot more but let me skip to the end result. After inflicting
months of outages on everyone, after tearing up lots of lawns, after all
of this, many of the fiber conduits that are allegedly underground: aren't.
---rsk
There’s more to it than this too. I was down there (I have sites I’m responsible for in Panama City Beach) in February and I was talking to a bunch of folks in the area as a result. This storm was fairly unusual for the area for a number of reasons. One, it normally doesn’t hit the panhandle at anywhere near a category 5, and two, it was still a high category 3 by the time it hit Georgia. The amount of damage done was immense, is still not cleaned up (I drove past multiple buildings that were still piles of rubble, 4 months after the storm), and I was seeing forests full of damaged and destroyed trees all the way to I-10.
All in all, the vast majority of Panama City looked much more like 4 months after a tornado rather than a hurricane, and all that damage continued all the way into Georgia. Thinking this was just like any other hurricane to hit the area is the absolute wrong tack to take - from what I heard there was some discussion of whether it was worth it to reopen Tyndall AFB, because the only thing left standing was some WWII era bomb-proof concrete hangars.
On the flip side, improvements in response are a good thing - as long as people aren’t beating up on the people who did the responding in the first place without cause.
Can everyone on this list that is using Gmail please complain to Google as
there is a STUPID default to set font-size:small. It make the text unreadable
on smaller device with already small fonts. The message I’m replying to
does this. font-size:small should be verboten in email even for footnotes.
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">The=
re's more to it than this too. I was down there (I have sites I'm r=
esponsible for in Panama City Beach) in February and I was talking to a bun=
ch of folks in the area as a result. This storm was fairly unusual for the =
area for a number of reasons. One, it normally doesn't hit the panhandl=
e at anywhere near a category 5, and two, it was still a high category 3 by=
the time it hit Georgia. The amount of damage done was immense, is still n=
ot cleaned up (I drove past multiple buildings that were still piles of rub=
ble, 4 months after the storm), and I was seeing forests full of damaged an=
d destroyed trees all the way to I-10.</div><div class=3D"gmail_default" st=
yle=3D"font-size:small"><br></div><div class=3D"gmail_default" style=3D"fon=
The majority of people doing locates are terrible at their job. (Un)fortunately, people doing the conduit installations are often terrible at their job as well. It’s about a 50/50 split if the line was located correctly and the installation crew was careless or the line wasn’t located correctly in the first places. Sometimes lines can be off by 10 feet.
Nature is more powerful than humans.
In Puerto Rico and US Virgin Islands, the hurricanes severely damaged essentially every type of infrastructure. Buried cables, aerial cables, microwave towers, cellular towers, satellite dishes, solar panels, primary and backup power stations, access roads, accesss airports, access seaports, broadcast radio/tv, cable TV systems, etc. etc. etc.
All the island backbone ring buried cable systems in PR experienced multiple cuts due to mudslides, bridge failures, and other restoration activites.
Immediately after the hurricanes, essentially every infrastructure damage assessment was 75% or worse, with most communications outside plant infrastructure 95% to 100% damaged.
When there is only light to moderate damage, independent repair efforts are faster because you avoid lots coodination meetings. But with the major damage in PR and USVI, the lack of coodination caused conflicting repair efforts and re-work for the first several months. It wasn't patch and move on, it was rebuild from scratch.
We had some issues around here where the locates were literally on the wrong side of the street. Crews were hitting gas lines left and right. Apparently in some cases they didn't actually LOCATE at all. They just sprayed and flagged based on some ancient (in many cases 50+ year old) "as-built" drawings.
Didn't stop the county and cities from putting a stop to it after a while, though. Probably because, as you say, it was maybe 50/50 whether the locates were off vs. whether the boring crew just wasn't paying attention. How you can't know where an active pinger on your drill head is to within a few inches let alone several feet is beyond me.
I dunno how the big guys get away with it. If I hit something, you can darn well bet someone's going to be on my neck immediately to shut the job down and pull my bond if possible.
It helps when the people in the field are like 3 subcontractors removed.