And your upstream(s) to work. And their upstream(s) to work. etc. If 90% of the stations in the EAS web are down you may end up with nothing working.
The telecommunications damage in PR and USVI will be a good test how well the EAS works during extreme telecommunications damage.
This seems to indicate that it will be 4-6 months until things get back to normal, if there indeed is a huge effort to do so.
"But as first responders on the ground in Puerto Rico told Fernández Campbell, this isn’t enough. Trump should also ask Congress to pass a relief package for Puerto Rico to give FEMA and the island more money to rebuild. He could deploy more military resources to help with search and rescue operations."
I hope this happens.
After a week without power, all the stationary batteries throughout the telecommunications network are likely completely drained. This makes restoration even more difficult, like a dead car battery needing a jump start.
I am focusing on U.S. territories, but there is also disaster response from Hurricanes Irma and Maria on Antigua and Barbuda, Cuba, Dominica, Montserrat, Saint Martin, and St. Kitts and Nevis.
Fatalities, including deaths attributed to post-hurricane recovery:
Hurricane Iram: 72 - Florida; 40 - Caribbean
Hurricane Maria: 16 - Puerto Rico; 2 - U.S. Virigin Islands; 15 - Dominica, 3 - Haiti; 2 - Guadeloupe
Department of Defense:
Supporting FEMA, the Department of Defense has deployed USNORTHCOM Brigadier General Rich Kim to Puerto Rico to manage the Title 10 (military) response efforts in Puerto Rico and U.S. Virgin Islands. USSOUTHCOM continues to support relief activities elsewhere in the Caribbean.
Airports and sea ports:
Puerto Rico: 3 sea ports open; 5 sea ports open with restrictions, daylight hours only. 9 airports are open. Only San Juan Airport open to commercial air traffic, approximately 15-20 commercial flights. All other flights reserved for priority military and relief activities.
U.S Virgin Islands: 4 sea ports open with restrictions, daylight hours only. U.S. VI airports closed except military and relief flights.
Electricity:
Puerto Rico: 1.57 million customers out of service. An estimate of 4% has been restored. Restoring power to airports, hospitals, sea ports and water treatment plants are still critical priorities. 80% of transmission lines damaged, power generation plants appear intact.
U.S. Virgin Islands: 55,000 customers out of service, most of the islands. St. Thomas has five feeders partially energised. St. Croix has three feeders partially energized. Restoring power to airports, hospitals, sea ports and water treatment plants are still critical priorities.
Telecommunications:
Pictures posted on twitter of joint restoration meeting between telecommunications providers, FEMA and Puerto Rico Telecommunications Regulatory Board. From the logos & colors on shirts: Claro, T-Mobile, Sprint, and many other company logos I couldn't make out (estimate 20 people in the room).
Reports of generators and fuel stolen from cell sites and remote telecommunications locations. This is not unusual during disasters. The Puerto Rico Telecommunications Industry Alliance, which appears to be a lobbying group of communication companies in Puerto Rico, has sent a letter about the need for FEMA to coordinate logistics and prioritize access to fuel and security. PRTIA (or APT in Spanish) has existed for a few years, but I can't judge if its letter represents telecommunication companies in Puerto Rico.
Puerto Rico:
2,432 of 2,671 cell sites (91%) out of service.
No update/change to cable and wireline systems, about 55% of central offices with voice, data and long-distance. The rest with only local voice, no inter-office connections. No clear description about status of local loops or subscribers with service.
Pictures of Liberty Cable PR repair crews posted on twitter. I still haven't found a public statement about LibertyPR's status.
Approximately 450-500 out of 1200 Internet networks and 35-38 out of 48 ASNs are present in the global Internet routing table, with occasional up/down changes due to restoration activity.
U.S. Virigin Islands:
70 of 106 cell sites (66%) out of service.
No update/change to cable and wireline systems.
U.S. Virgin Islands Internet routes have nearly returned to normal, with occasional up/down blips due to restoration activity.
I'm not ignoring the status competitive and smaller USVI and PR communication providers, its just difficult to find official statements from them. If you have status about them, let me know.
After a week without power, all the stationary batteries throughout the
telecommunications network are likely completely drained.
from the point of view of cell sites, wouldn't battery autonomy be
measured in hours rather than days? I could see some site having
autonomy in days due to permanent generator, and when fuel runs out so
does the cell site.
I'm not ignoring the status competitive and smaller USVI and PR
communication providers, its just difficult to find official statements
from them. If you have status about them, let me know.
One aspect often forgotten is that people have homes (or what is left of
them) families and the need to find food/water which can involve
standing in line for hours in a day and they may not be able to show up
for work. larger companies can usually find enough employees not so
hindered, but smaller outfits may not be able to remain functional due
to not enough staff able to work.
Smaller outfits may not have the ability to get petrol for their trucks
to go out oand fix things. (whereas the big guys have the credentials to
get petrol form authorities/army.
Yes, long-term power is generators. But there is always a catch.
What happens during disaster recovery is the batteries are damaged by being drained repeatedly, dirty power from generators, and enviromental conditions. After too many deep-discharge cycles during the disaster, the batteries won't hold a charge any more. The battery failure rate, requiring replacement, goes through the roof after about a week in a disaster. Even those 10-year telco batteries don't last 10-years during disaster conditions.
Since a lot of telecommunications gear actually runs off -48 volt battery string, and the generators recharge the batteries; when the batteries completely fail even with a generator, no more telecom. You have to replace the battery string or run the telecom gear on raw generator power (which then damages the telecom gear even more).
Sometimes even the battery starter on the generator fail to start after too many refueling stops. Most backup generators are only rated for "stand-by" service, not continuous operation for weeks. Generators need more maintenance, and fail more often.
Disaster logistics is a string of dominos. If they start being knocked over, it just gets worse. Stuff that works great during normal conditions doesn't anymore. Simple fixes are all complicated now.
Telecommunications:
Pictures posted on twitter of joint restoration meeting between..........
What twitter feed was this?
I didn't catch it.
FYI:
White House announces that the US Army Corp of Engineers is in charge of
power in Puerto Rico, and were given priorities to hospitals and other
emergency services. No mention of telecom being part of those
priorities. Initial push is installing temporary power generation.
They are not yet working on fixing the electrical grid.
(44 of 69 hospitals now have power).
(Note: FEMA has decided to stick to road deliveries, not air drops for
supplies).
The WISP I'm getting updates from is having thefts as well. Still having logistics issues. The leading idea at the moment is tower-mounted solar panels and batteries. Nothing is foolproof without armed guards, but it's better than nothing.
Career federal employees are taught to write situation reports in very boring language with just the facts known. Nevertheless, after reading lots of situation reports, you start to notice when the bubureaucratic language changes. Perhaps the most famous was the commander of Apollo 13's report "Houston, We have a problem."
Puerto Rico has announced a new web site with current status:
However, in the last 24 hours I've noticed some agency situation reports used different statistics to report "happy, happy, joy, joy" stuff. In the bureaucratic world, this is very concerning, such as when the Veterans Administration was misreporting appointment waiting times to look better.
You can't fix problems, if the real situation isn't being reported accurately to senior leadership even if its bad news.
Sean, thank you for all the excellent updates you have been providing. Status.pr is disturbing since there is no context to the stats offered on this page. 49% of supermarkets may be open, but with nothing on their shelves. And 11k refugees? Who are they trying to kid with a number like that.
-Barb
+1.808.385.1677
mauigrrl@earthlink.net
Written on the move, apologies for any errors.
The situation reports from Puerto Rico seems to be getting passed through public relations, so I'll try to add some context.
Public Safety
Primary Public Safety Answering Point (9-1-1) center generator ran out of diesel fuel. Switched to alternate PSAP.
San Juan Police Department has restored its radio repeaters and police radio communications metro-wide. (translated from spanish, so I think I understood the technical translation).
Landline Central Offices
813,546 subscribers (CIA World Factbook)
390,000 subscribers in 52 municipalities with voice, data and long distance (Claro)
Repaired fiber optic cable conntecting CO's in Fajardo and Rio Grande.
65% of inter-office Central Office connections restored island-wide.
Remaining CO's have only local voice calling.
Optico Fiber reports most of its infrastructure is intact, and has open WiFi hotspots outside its offices.
Wireless services
3,227,281 subscribers (CIA World Factbook)
29 municipalities have 0% working cell sites. It appears carriers are repairing one tower in each county/municipality to improve island-wide coverage. Several municipalities going from 0 to 1 cell site working.
310,000 subscribers in 28 municipalities with working cell towers (Claro)
34% of San Juan has working cell tower coverage (Claro)
Cell on Wheels in Ponce (4 mile radius) serving 6,000 calls per hour, 35,000 texts per hour (AT&T)
Dorado, Tao Baja and Toa Alta have T-Mobile service (T-Mobile)
I don't know what FCC and PRTRB are counting:
286 working cell sites out of 2671 (according to FCC report)
96 working cell sites out of 1600 (according to PR Telecommunications Regulatory Board report)
For context, the number of cell sites repaired each day since the end of Hurricane Maria is improving slowly - average less than 20 sites a day, but some days its negative, i.e. more cell towers failing than repaired.
On U.S. Virigin Islands, the number of cell sites out of service decreased initially, but has slowly increased for the last 5 days.
I created a spreadsheet of the FCC wirelss outage data from hurricane Harvey, Irma and Maria.
https://www.donelan.com/FCC-Wireless-Outages.xlsx
There is no consistent pattern between states, territories or hurricanes. Florida had the fatest wireless restoration, average 500 cell sites restored a day; while U.S. Virgin Islands averaged less than 1 cell site restored. But Florida was mostly restoring the electrical grid, which restored lots of cell sites. Harvey was slow to start restoring cell sites, the tropical storm lasted for days; but less than 6% of cell sites were out of service.
Cable systems
First official report from Liberty Cable Puerto Rico
Most cable headends or in good condition, with backup generators. Internet connection to international circuits reconnected. Main fiber trunk between San Juan and Luquillo completed. Working to repair infrastructure and primary services such as physical plant, main repeater bases, fiber optic ring and fiber to distribution stations in neighborhoods. (LibertyPR)
Satellite Services and Satellite Phones
As more satellite phones are distributed, social media and news reporters are saying satellite capacity is getting worse. It may be user issues and lack of training, or running out of satellite bandwidth in the area.
American Red Cross driving a VSAT station between shelters, and setting up temporary hotspots for an hour at each shelter so people can contact family members.
I don't know what FCC and PRTRB are counting:
286 working cell sites out of 2671 (according to FCC report)
96 working cell sites out of 1600 (according to PR Telecommunications
Regulatory Board report)
I had noticed the different numbers too. My speculation:
The 1600 may refer to antenna sites, whereas the 2671 may be the sum of
the number of sites reported by each carrier (think a mast supporting
antennas from multiple carriers).
Assuming my logic is correct, the 96/1600 statistic may be of more use
in a "can I dial 911" point of view. Having multiple carriers "up" at
the same tower doesn't increase geographic footprint where some coverage
exists.
From a disaster management point of view, in a town where each carrier
has its own tower, deciding which one to light up first could be
interesting. (aka carriers getting together to compare state of antennas
in town and somehow elevating that info to whoever controls the
generators (army corps of engineers who are "foreigners" with no local
knowledge).
Has anyone heard anything about Liberty Cablevision / AS14638?
Our Netflow stats show a traffic drop to zero at the moment of landfall of Maria, late on 9/19, and a continued flat line at zero until now. Almost 11 days without a single packet exchanged. This is (as far as I am aware), the #2 largest ISP in Puerto Rico.
By comparison, Claro’s traffic certainly has dropped by a large degree, but it always stayed at least slightly above zero, and is roughly at 10% of normal traffic levels today.
-Phil
The whole thing is a disgrace.
The Government of Puerto Rico has created a map of working cell sites in puerto Rico. I'm not certain about the source of the information. Cellular carriers usually object/refuse to release details about their operations.
The map shows most working cell sites are in metro areas around San Juan. As I guessed, one or two cell sites in each county/municipality around the island. There are almost no working cell sites covering the interior of the island.
Comparing the map to census bureau population maps indicates the working cell sites are in high population areas, which is necessary for disaster triage. Satellite phones are being distributed to mayors in the other counties/municipalities.
The first public statement I've seen from LibertyPR was yesterday. Their
network was completely down. They've restored some of their main infrastructure, i.e. cable headends and main fiber connections.
100% of subscribers are out of service.
I've seen pictures on twitter of LibertyPR crews fixing cables and poles on the island.
Liberty cable Puerto Rico has put out a press release today.
LibertyPR is opening one public WiFi hot spot in Bahia Urbana in San Juan from 3pm to 7pm Saturday, and 8am to 7pm daily starting Sunday.
Additional hot spots will be announced by LibertyPR via press release in the future.
I guess this is a sign LibertyPR's public relations office is back in operation.
At this point, I wouldn't trust status.pr and any media reports without
verifying information. As far as LibertyPR is concerned my cousin who lives
in Carolina, PR told me thieves were stealing fiber optic cable after the
storm. I trust the Seon Donelan, FCC, US Military, FEMA reports in that
order. There was a report that 33% of cell phone service was reported. That
is BS. We know from FCC reports it is still at ~90% out as far as number of
operational cell sites.
The media here in the states is no better. I have multiple confirmations
and am looking for hard proof but the Teamsters Puerto Rico trucking union
is refusing to move containers out of the port. Only 20% of truckers showed
up for work. Perhaps someone who works at Crowley can give us more concrete
info but if you can't even move supplies out of the port, how the heck are
you supposed to replace wires/fiber/fuel etc?
Here is a CNBC report: https://www.youtube.com/watch?v=f4Z01o4tBlI
- Javier
The more I read about this, the more disturbed I get. On the one hand, we
keep hearing that the trucks aren't moving because roads are impassable.
Then I read that government officials are driving from their remote areas
to San Juan to ask why no aid is coming, disputing the claims about the
roads. We hear that there isn't fuel for the trucks, then a reporter from
CNBC disputes that claim as well. The only thing that seems to be a common
thread is that there are massive amounts of supplies sitting in San Juan
and that they can't get truck drivers to deliver them.
Do FEMA and the National Guard have the authority to commandeer the trucks
and deliver the containers themselves? The telcom companies aren't going to
be able to do much by way of repairs without supplies.
I haven't seen any reports of a Teamster union refusal. I *have* seen
reports that only 10-30% of truck drivers are operational, because of
one or more of:
1) Their rigs are stuck behind a highway outage due to washout
or downed trees.
2) Their rigs are stuck due to lack of diesel.
3) Rigs are fine, but drivers are stuck due to 1) or 2), or they
are too busy trying to save their families/etc to show up to work.
I've seen too many reports of "I have been waiting in line for 5 hours for
water / gasoline / ice for <relative>'s insulin" to get too irate at people who
fail to show up for work.