Hurricane Maria: Summary of communication status - and lack of

Mailman List Mirror (Read Only)
1

This summary status is a bit different. Instead it is a report about what we don't know, and estimate how much we don't know based on official reporting.

The FCC DIRS report is based on outages reported by service providers. Almost no local providers have been able to file either a positive or negative report on the DIRS website. So the data is very incomplete.

Electric Services
    Puerto Rico: 1,569,796 customers out of service (100%)

    U.S. Virgin Islands: 90% of grid on St. Thomas and St. John destroyed by Hurricane Irma, a few critical facilities re-energized. 25,000 customers on St. Croix out of service after Hurricane Maria, but may be able to re-energize most of the St. Croix grid on Friday.

Internet Services

    All submarine cables and landing stations appear operational. 2 colocation data centers on Puerto Rico appear to be operational. I couldn't tell if there any of the old Internet Exchange Points were still operating, or had ceased before the hurricanes.

    Puerto Rico:
       Approximately 880 networks out of 1200 not reachable (24 out 48 ASN).

     U.S. Virgin Islands:
       Approximately 13 networks out of 70 not reachable (2 out of 6 ASN)

Public Safety Services

   Public Safety Answering Points (9-1-1 centers)
     Puerto Rico: 2 out of 2 PSAPs reporting. 2 operating normally.
     US Virgin Islands: 2 out of 2 PSAPs reporting. 2 operating without automatic location identifier

    NOAA Weather Forecast Office
       Puerto Rico: Office on backup generation, weather RADAR offline, 1 of 2 Weather Radio transmitters offline

       US Virgin Islands: Served by San Juan, PR WFO, 1 of 1 Weather Radio transmitter offline

Wireless Services

    Puerto Rico: (1703 cell sites out of 1789)
      95% of cell sites out of service. 48 out of 78 counties with 100% of cell sites out of service.

    U.S. Virgin Islands: (82 cell sites out of 107)
      77% of cell sites out of service.

Cable and Wireline Systems

     Puerto Rico (est. 11 companies-ILEC, CLECs and Cable in LATA)
        FCC summary implies no companies have reported yet.
        Large percentages of consumers without cable or wireline service

     U.S. Virgin Islands (est. 3 companies-ILEC, CLECs and Cable in LATA)
        FCC summary implies no companies have reported yet.
        Large percentage of consumers without cable or wireline service

Broadcast facilities

   Puerto Rico
     34 TV stations-not including repeaters, translators, boosters
        1 TV station reporting - 1 station out of service
        33 TV stations not reporting (public reports no TV stations operating on air)

     141 radio stations-not including repeaters, translators
         141 radio stations not reporting (public reports estimate a dozen radio stations operating on air)

     U.S. Virgin Islands
       5 TV stations- not including repeaters, translaters, boosters
          5 TV stations not reporting

        26 radio stations-not including repeaters, translators
          26 radio stations not reporting

2

Following up - there are three cable landing stations and 9 submarine cable systems connecting Puerto Rico.

One of the cable landing stations experienced flooding, and shutdown its power system affecting some circuits. I haven't been able to determine how many submarine cable systems are affected, since they share cable landing stations.

3

And that shutdown affected Internet capacity throughout South America.

Rubens

4

T-Mobile PR on twitter just posted that two of it's submarine cables are
out of service.

Claro PR Wireless (this is the ILEC in PR) website can't even be reached.

I am assuming this is due to power and submarine cable issues since I'm
sure t-mobile and many other providers are using the same cables.

Link to the post on twitter:
https://twitter.com/tmobilepr/status/911644083155869697

- Javier

5

Reportedly most (All?) operational cellular carriers on Puerto Rico have activated "universal roaming" service. All working towers will accept roaming connections from any phone from any carrier (or no service provider). You may need to turn the phone off & on so it scans for a working signal.

Roaming still requires a working cell tower. 48 counties and county-equivalents in Puerto Rico have 0% cell sites working. Less than 25% of cell sites in the remaining counties are working. Capacity is extremely limited, so use SMS/Text rather than voice or data.

A side-effect of universal roaming is lack of billing, so expect carriers to announce they are waiving charges and overages in Puerto Rico.

6

According to PREPA Net, the fiber subsidary of the Electric Power Authority, the power system for the Punta las Mar�as submarine cable station is back in service after flooding. I think Isla Verde and Punta las Marias refer to the same landing point.

I don't know the status of individual submarine cable systems using that landing station.

The Miramar and San Juan cable landing stations are also in service.

7

The ILEC, Claro, is reporting all 24 central offices in Puerto Rico are now operating on generators, and maintaining re-fueling operations.

The CO's in the (San Juan?) metro area have voice, data and long distance service including to the mainland.

The CO's elsewhere in Puerto Rico have only local voice service. The offices are isolated, with no long distance or inter-office data service.

Although the CO's are operational, substantial outside plant is damaged.
Which means most subscribers do not have service. Inter-office facilities outside the (San Juan) metro area are damaged, which means people with service in those areas can only make local calls.

Wireless sites are still being evaluated. The Puerto Rico Transportation Department is providing road crews to clear/rebuild roads and escort cellular providers repair convoys to remote cell sites.

The Puerto Rican government has not re-established communications with officials in the following municipalities: Aibonito, Jayuya, Lajas, Mayaguez, Quebradillas, Rinc�n, Sabana Grande, Vieques and Villalba.

8

From one of my colleages that has a decent sized WISP in Puerto Rico.

" Guys, we are ok, network hurt pretty bad… will need help "

There are a bunch of WISPs waiting to go rebuild, but waiting for the clearance to do so.

https://radar.qrator.net/as14979/providers#startDate=2017-08-09&endDate=2017-09-23&tab=current

It looks like they're still online via Critical Hub Networks and Columbus Networks, but not Liberty.

9

I'm not sure what clearances they are waiting for. If they are already in Puerto Rico, self-sufficient, and respect curfews and other emergency responders, they should be able to start local restoration and recovery activities.

Several local ISPs and communication providers have announced open public WiFi hotspots outside their Puerto Rico offices during non-curfew hours. I've also seen reports from individuals volunteering on the Virigin Islands setting up internet access.

If they are not already on the island, most Puerto Rican airports and ports are still closed to non-military or relief activities. There is no U.S. mail or freight service. Only one airport was open for limited commercial flights. They will need to bring everything neccessary to support themselves, including food, water, shelter, etc.

Managing volunteers who want to help is difficult in all disasters. Unless they have training how to survive and take care of themselves in such a situation, letting in outside well-meaning volunteers sometimes become additional people who need to rescue.

WISPs already on Puerto Rico or U.S. Virigin Islands, with resources for recovery and restoration of communications; can contact the FCC Operations Center, (202) 418-1122, FCCOperationCenter@fcc.gov

http://transition.fcc.gov/Daily_Releases/Daily_Business/2017/db0920/DA-17-913A1.pdf

10

Sorry, WISPs in the US48 to go to PR to help rebuild downed WISPs.

Yes, they need to be able to get there first. Those already on the island are doing what they can until more supplies arrive.

11

As of this morning, the ILEC Claro is reporting

8 central offices have voice, data and long distance service operating, mostly in metro areas. This does not include outside plant or local loops serving customers.

Central offices serving 55 of 78 municipalities have local voice service, no inter-office or long-distance service. Again, not including the local loop to a customer.

27% of cell sites in service, mostly in the north and east parts of the island, operating. I'm not sure if this is 27% of all cell sites on the island, or 27% of cell sites only in the north and east.

Other providers say they are working to restore service, but are not releasing specific data about their network status (AT&T, Open Mobile, Sprint, T-Mobile). Cable provider, LibertyPR, hasn't said anything to local reporters that I could find in any of the PR newspaper websites; and appears to be completely out of service.

There are several competitive providers and small providers I don't have information about in PR. I just don't know the market. If anyone has status about any small providers operating, let me know their status.

12

Priority is to restore communications to emergency responders, restore
power to hospitals and other critical infrastructure). So workers that
clear roads, remove dangling electrical wires would prioritize fixing of
that critical infrastructure. That road you need cleared to get to your
fixed wireless antenna will wait.

Similarly, I get the impression that all cargo capacity into the island
is still controlled to prioritize essentials. So those spare circuit
board you need to fix a router have to wait.

Also, with residences overwhelmingly without power, fixing the "normal"
ISP business won't do much when nobody can use it. It is best to focus
on wi-fi in central locations such as shelters, and cellular for first
responders and others.

There are good reasons local governments work out disaster plans because
they need to identify in advance what gets priority after a disaster.

13

You're assuming the WISP isn't providing infrastructure to critical facilities.

14

It looks like someone kicked the cellular carriers public relations people into gear. Today, instead of the normal "we care" messages; they released statements providing more concrete details about their restoration activity in PR and USVI.

Overall, 91.2% cell sites out of service in Puerto Rico. 34 of 78 counties have 100% cell sites out of service. This will continue to change up and down, as sites are restored and circuits are damaged by cleanup activity.

There are over 2,671 cell sites on Puerto Rico and 106 cell sites in U.S. Virgin Islands. As carriers bring in tens of generators and repair equipment at a time, gives you some idea how long restoration will take.

In alphabetical order...

ATT:
"We continue to send aircraft with essential supplies and network resources as we help the people of Puerto Rico. These flights include portable temporary cell sites, high capacity generators to provide temporary power, and other larger network equipment on cargo planes and barges to help restore services on the island. We planning to set up a number of portable cell sites in the San Juan area as soon as possible.

So far, we’ve sent multiple flights carrying the following supplies:
More than 30 generators
5,000+ gallons of water
We are also focused on network restoration in the U.S. Virgin Islands are bringing additional resources there."

Claro (google translate from Spanish):
They reported that in the metropolitan area specifically, Claro's signal was already reaching 31 percent of customers in San Juan, 22 percent in Guaynabo and 18 percent in Carolina and Bayamón.

At the island level, the Claro signal is up in 14 municipalities today, covering an average of 20 percent of the clients in Aguada, Manatí, Mayaguez, San Germán, Cabo Rojo, Trujillo Alto, Dorado, Camuy, Quebradillas, Humacao, Juncos , Caguas, Aguadilla and Toa Baja.

That number will increase in the coming days.

Sprint:
"A vessel has already arrived in Puerto Rico with the generators and parts required to begin the work. In turn, a body of over 40 Sprint engineers and technicians in the United States were sent to the Island to join the local technical staff, coordinate the delivery of the equipment received and continue work to speed up the communication.
A second shipment will arrive on the island this Wednesday, September 27 with additional spare parts and materials."

T-Mobile:
"The damage to the infrastructure is unprecedented, but equally it is the support we are receiving from T-Mobile US. Between Saturday and Sunday, six MD11 cargo planes and one AM124 (second largest cargo plane in the world) arrived with 80 generators, 16 trucks, equipment to build 100 communication facilities. More cargo planes will arrive today with more equipment and personnel."

T-Mobile also mentions while T-Mobile's field engineering crew was at the Luis Muñoz Marín Airport, they were drafted to help install a generator for the FAA Control Tower. That's one way to help get your supplies on the island.

If you have information about other telecommunication providers in Puerto Rico or U.S. Virgin Islands, let me know.

Due to damage to the FAA communications and guidance systems, only a dozen or so commercial flights can land during daylight hours each day. Airlines report over 20,000 people on standby lists, and nearly 1,000 people waiting at the airport for any flight.

The Port of San Juan is open, daylight hours only, and receiving freight barges. While there is a plenty of fuel, food and supplies at the port; getting truck drivers to the port and damage/blocked roads is slowing distribution of supplies to the rest of the island. U.S. Mail and other express delivery companies still do not have service in Puerto Rico. Limited U.S. Mail hand-out service is available at a few post offices in U.S. Virgin Islands.

15

You know, that's a really good point. In such situations, the sooner
you can get the basic infrastructure operational again and
transportation, electrical systems, and fuel distribution (generators
have to run on something...) in particular, the faster everything can
start coming back together. First and foremost, this means making the
place habitable again so you actually have customers to serve. So any
time spent doing something like what is related above is extremely
worth while and can only serve to facilitate future work for everyone
on the island.

16

What is the US government role in all of this? It sounds like a few https://en.wikipedia.org/wiki/Lockheed_C-5_Galaxy could be of use here to airlift in lots of gear.

17

I doubt the runway is stable enough to hold the weight of a loaded c5.

18

Keep on posting this great info Sean. It is being passed along. Just wanted
you to be aware.

19

Things are better and worse in Puerto Rico and other Caribbean islands. Help is needed, but anyone wanting to help in the field, be certain you understand what you would be doing, and whether you are actually helping or hindering on the ground efforts.

From Washington Post:

[U.S. FEMA Director] Long also warned people not involved with the relief effort to stay away.

“If you’re going to Puerto Rico right now, it should be for only a life-sustaining, life-support mission,” he said. “Because everybody that’s trying to get in that’s not supporting that is getting in the way.”

According to reports, the major (but not named) telecommunication companies met today with the Puerto Rico Telecommunications Regulatory Bureau about coordinating restoration efforts. Several companies have agreed to joint repairts. Instead of each company sending multiple crews to the shared cell sites, they will agree to divide the work among all the companies.

This will distribute more repair crews from all participating companies to more cell sites from different companies around the island.

Claro, the ILEC, is the only company that has publically confirmed the joint repair agreement. AT&T, T-Mobile and Sprint also have repair crews on the island, but I haven't been able to confirm which companies have signed the joint repair agreement.

Claro also said they've re-connected 55% of its Central Offices, including voice, data and long distance. Once again, I'm guessing this is inter-office trunks, and not local subscriber loops.

The FCC reports 2,429 of 2,671 cell sites (90.9%) are out of service in Puerto Rico. And 65 out of 106 cell sites (61.3%) are out of service in the U.S. Virgin Islands.

Broadcast Radio and Television

14 AM stations on the air on Puerto Rico

8 FM stations on the air on Puerto Rico

2 TV stations on the air on Puerto Rico

Special notice: On Wednesday, September 27, 2017 at 2:20pm Easter Time, FEMA will be conducting a scheduled national test of the Emergency Alert System. This national test was scheduled in July, 2017. The test will take about a minute, and sound like a typical monthly EAS test "This is a national test of the Emergency Alert System. This is only a test."

Most people probably won't pay attention to the national EAS test on Wednesday. But there are always few news stories about some people being alarmed by the national test.

If there is an *new* emergency or severe weather at the time, the national test will be rescheduled for October 4, 2017. Although the disasters in Puerto Rico and U.S. Virgin Islands are continuing, the national test will be a very brief interruption on radio and TV on the islands. The telecommunications damage in PR and USVI will be a good test how well the EAS works during extreme telecommunications damage.

20

The telecommunications damage in PR and USVI will be a good test how well the EAS works during extreme telecommunications damage.