Hurricane Irma: U.S. Virgin Islands and Puerto Rico

Mailman List Mirror (Read Only)
1

5 fatalities in U.S. Virgin Islands and Puerto Rico

FCC doesn't have reliable reporting on Wireline and Cable service providers in either USVI or Puerto Rico. The assumption is large numbers of customers are without cable or wireline service.

870,000 customers (55% of the island) without power in Puerto Rico

22,900 customers without power U.S. Vigin Islands

1 hospital closed, 25 hospitals on backup generators

29.3% of cell sites in Puerto Rico and 60.7% of cell sites in the U.S. Virgin Islands are out of service.

1 radio station and 2 TV stations in USVI are out of service

4 radio stations in Puerto Rico are out of service

2

I have talked(briefly sms/fb) to several friends of mine who operate
critical datacenter infrastructure. What is not shot down mostlly on diesel
generators but actual impact if power isnt restored in next 5-7 days is yet
to come.

USVI and Puerto Rico will get the restoration it needs in critical
infrastrucure, no doubt I am more concerned about othet caribbean who were
impacted.

Having lived in Caribbean for years and been on many islands, i can only
imagine recovery Of services on smaller islands taking long time.

There is an effort by Martin hannigan announced on twitter and a
coordination call. Martin is active member of NANOG, Martin how can we
support/join this great initivative?

3

Electric Power
     Florida: 6,117,0247 customer outages (59% of state)

     Puerto Rico: 368,682 customer outages (23.5% of territory)

     U.S. Virgin Islands:
   St. John: 100% customers without power
   St. Thomas: 99% customers without power (airport, hospital, and some areas on those substations re-energized)
   St. Croix: 31% customers without power

     Georgia: 866,682 customer outages (9% of state)

     South Carolina: 184,471 customer outages (7.2% of state

Public Safety:
    29 Public Safety Answering Points (9-1-1 centers) impacted
      Florida
   14 PSAPs out-of-service, no re-route
   11 PSAPs out-of-service, rerouted to other answering points
   2 PSAPs with partial service, no automatic location
      U.S. Virgin Islands
   2 PSAPs (out of 2) with partial service, no automatic location

Wireless Services:
     Florida
   27.4% cell sites out of service (6 counties with more than 50% out of service)

     Puerto Rico
   19.4% cell sites out of service (8 areas with more than 50% out of service)

     U.S. Virgin Islands
   55% cell sites out of service (90% out of service St. John, 72% out of service St. Thomas, 22.5% out of service St. Croix)

Cable and Wireline

     Florida
   7,597,945 customers out of service
   390 switching centers (central offices) out of service

     Puerto Rico and U.S. Virgin Islands
   Reporting still spotty, assume customers out of service in
areas without electric power. Some VSAT service restored on U.S. VI.

Broadcast
     8 TV stations out of service
   Florida: 2 stations
   Puerto Rico: 4 stations
   U.S. Vigin Islands: 2 stations

     26 radio stations out of service
   Florida: 25 stations
   U.S. Virgin Islands: 1 station reporting (but other stations are likely out of service, but can't report)

4

A couple of additional items

Telecommunications - Free WiFi hotspots (where power available)

     Comcast Xfinity WiFi
     Charter Spectrum WiFi

I expect other commercial WiFi hotspot service providers in Florida have (or will) have free access in the disaster areas.

AT&T and Verizon are bringing temporary Cells on Wheels trucks to the disaster areas. AT&T, Sprint and Verizon are providing extra mobile data or waiving usage surcharges in the disaster areas.

Check the AT&T, Comcast, Charter, Sprint and Verizon corporate sites for details. T-Mobile is probably doing something too, but the T-mobile web-site makes it very difficult to find disaster information.

5

T-Mobile is providing free calling text and data.

6

A couple of folks asked about commercial colocation and data centers...

There are about 100 major colocation and datacenters in Florida and southern Georgia. I've check the usual suspects, and none have posted outages. Directv and Dishtv "local-in-local" TV stations appear to be uninterrupted, which indicates the major fiber hubs they use for TV signal aggregation are working in the local areas.

I can't tell how many data centers are on backup generators.

Today's update from various official sources (FEMA, Dept. of Energy, FCC, NOAA, etc).

Electric Power (DOE)
    Florida: 4,788,277 customer outages (48% of total state customers)
    Georgia: 932,587 customer outages (22% of total state customers)
    South Carolina: 140,759 customer outages (6% of total state customers)
    North Carolina: 56,834 customer outages (1% of total state customers)
    Alabama: 20,050 customers outages (1% of total state customers)
    Puerto Rico: 303,998 customers (19% of total customers)

    U.S. Virgin Islands
   Water and Power Authority (WAPA) reported several feeders on St. Thomas are re-energized. There are currently two generators (#14 and #25) online with a maximum capacity of 39 MW. An 800 kW generator has arrived on St. Thomas September 12.

Water (FEMA)

     U.S. Virgin Islands: 341,000 people without potable water
     Puerto Rico: 61,980 people without potable water

Public Safety
    Hospitals (FEMA)
   Florida: 33 closed, 204 healthcare facilities evacuated
   Puerto Rico: 1 closed, 6 on generator power
   U.S. VI: 1 closed and evacuated

    NOAA Weather Radio (NOAA)
   Florida: 9 out of 32 stations (28%) out of service
   Georgia: 4 out of 29 stations (13%) out of service
   U.S. VI: 1 out of 1 station (100%) out of service

    Public Safety Answering Points (9-1-1 centers) (FCC)
   Florida: 27 impacted (3 out of service, 9 partial service, 9 re-routed with ALI, 8 re-routed without ALI)
   U.S. VI: 2 impacted, without ALI/ANI

Cable and Wireline systems (FCC)

819 switching centers (cable headends and central offices) out of service. Unknown how many are isolated, damaged or just without power.

7,184,909 subscribers out of service in Alabama, Florida and Georgia; not including Puerto Rico and U.S. Virgin Islands. Because of the limited number of service providers, I think FCC is not releasing the information.

From other reporting, Puerto Rico has about 50% out of service and U.S. VI

has about 33% out of service not including St. John and St. Thomas.

Wireless Service (FCC)

     Alabama: less 1% cell sites out of service
     Florida: 24.6% cell sites out of service (5 counties over 50% OOS)
     Georgia: 10.5% cell sites out of service (1 county over 50% OOS)
     Puerto Rico: 14.5% cell sites out of service (3 counties over 50% OOS)
     U.S. VI: 53% cell sites out of service (St. John - 9 out of 10 OOS, St. Thomas 38 out of 57 OOS)

Broadcast (FCC)

    Television: 9 stations out of service
    Radio: 51 stations out of service

7

Just to clarify, since several people wanted corrections. There is substantial outside plant damage in Florida, and many local loop circuits are out of service. I was only referring to data services inside major colocation and data centers, not in offices and other locations.

One person did report they had servers reboot in a Florida colocation facility during the hurricane, likely due to power distubances. The power came back quickly, I assume an issue switching to backup generators or a glitchy UPS.

Another person reported they lost connectivity to their systems in a colocation facility for 24 hours. I'm still checking on that one.

8

Disclosure note: AT&T and Comcast public relations folks have been sending information about what they are doing for disaster recovery. I've included some of their information.

From various official sources (FEMA, Dept. of Energy, FCC, NOAA, etc).

Fatalities (FEMA)
    Georgia: 2
    Florida: 12
    South Carolina: 2
    Puerto Rico: 3
    U.S. Virgin Islands: 4
Note: FEMA is slower than media reports about U.S. fatalities

Non-US fatalities (AP/Reuters)
    Caribbean fatalities: Anguilla (4), Barbuda (1), British Virgin
    Islands (5), Cuba (10), French Territories (10), St. Maarten (4),
    Haiti (1)

Electric Power (DOE)

Florida: 3,568,499 customer outages (35% of total state customers)
Georgia: 451,033 customer outages (11% of total state customers)
South Carolina: 58,972 customer outages (2% of total state customers)
North Carolina: 24,445 customer outages (<1% of total state customers)
Puerto Rico: 117,244 customers (8% of total customers)
U.S. Virgin Islands:
   The airport and hospital are still energized. Besides a few smaller
   areas, most customers on St. John and St. Thomas are without
   power. Restoration efforts will continue as USVI WAPA works to get
   critical facilities reenergized on the two islands.

Water (FEMA)

      U.S. Virgin Islands: 341,000 people without potable water
      Puerto Rico: 61,980 people without potable water

Public Safety
     Hospitals (FEMA)
    Florida: 11 closed, 204 healthcare facilities evacuated
    Puerto Rico: 1 closed, 6 on generator power
    U.S. VI: 1 closed and evacuated

     NOAA Weather Radio (NOAA)
    Florida: 6 out of 32 stations (18%) out of service
    Georgia: 7 out of 29 stations (24%) out of service
    U.S. VI: 1 out of 1 station (100%) out of service

     Public Safety Answering Points (9-1-1 centers) (FCC)
    Florida: 29 impacted (4 out of service, 9 partial service, 7 re-routed with ALI, 8 re-routed without ALI)
   Georgia: 5 impacted (1 re-routed with ALI, 3 re-routed without ALI)
    U.S. VI: 2 impacted, without ALI/ANI

Cable and Wireline systems (FCC)

1,040 switching centers (cable headends and central offices) out of service. Unknown how many are isolated, damaged or just without power.

8,190,407 subscribers out of service in Alabama, Florida and Georgia; not including Puerto Rico and U.S. Virgin Islands.

According to Comcast: All comcast's miami-dade and broward facilities are on generator power. Comcast is deploy portable generators in neighborhoods to re-charge outside plant. Comcast has no network access beyond Marthon in the Florida Keys, but has crews ready when the area is accessible.

Wireless Service (FCC)

      Alabama: less 1% cell sites out of service
      Florida: 18.1% cell sites out of service (3 counties over 50% OOS)
      Georgia: 5.3% cell sites out of service
      Puerto Rico: 10.1% cell sites out of service
      U.S. VI: 55% cell sites out of service (St. John - 9 out of 10 OOS, St. Thomas 38 out of 57 OOS)

According to AT&T: deployed 6 portable, satellite connected cell on trucks in the Florida Keys (Stock Island, Key West and Marathon and 2 satellite connected cell on trucks in Naples, Florida. The AT&T National Disaster Recovery Team has over 20 units deployed throughout Florida (don't know what that means, but sounded good).

Broadcast (FCC)

     Television: 10 stations out of service
     Radio: 39 stations out of service

9

Sean - I think I speak for all of us when I say thank you very much for
these updates! The concise nature of them is super helpful.

-Dave

10

In my last summary, I made a comment I didn't know what the network disaster recovery team meant.

AT&T recovery efforts
    3000 recovery team members
    14 Satellite Cell on Light Truck (SatCOLT)
   - 1 in Tallahassee, 2 in Naples, 4 in Florida Keys
   - 1 portable cell site St. Thomas, U.S. Virgin Islands
    3 Emergency Communications Vehicles
    50 Drones
    Command trailers, hazardous materials response equipment

Verizon recovery efforts
    2 Satellite Picocels on Trailers (SPOTs) in the Florida Keys
    Refueling and generators at cell towers througout Florida
    Hundreds of portable generators
    2 Wireless Emergency Communication Centers in Naples (provide charging stations, phones and computers for the public to contact family and friends)

Sprint recovery efforts
    Sprint reports over 75% of its network is repaired in the southeast and Puerto Rico.

AT&T, Sprint, T-Mobile, Verizon continue to extend their wireless data, text and voice plans waiving overage charges in the disaster areas. Details are different for each carrier, so check their web sites or customer service.

Federal Aviation Administration recovery efforts
    Deployed emergency mobile air traffic control tower to the international airport on St. Thomas, U.S. Virgin Islands

    Drone authorizations
   138 commercial drone operator authorizations for Hurricane Harvey
   80 commercial drone operator authorizations for Hurricane Irma

Federal Communicatiosn Commission recovery efforts
     Reports at least 8,258,789 cable and wireline subscribers out of service in Alabama, Florida and Georgia.

     Strangely, the number of non-mobile switching centers increased from 1,040 (Sept 13) to 2,188 (Sept 14). I believe this exceeds the number of cable headends and central offices in Florida. So I don't know what this number represents or why it increased so much.

The situation is still dire in some locations, but generally the disaster operations have moved to recovery and restoration. Unless something significant happens, this is the last summary report about Hurricane Irma from me.

11

any word on cuba?

randy

12

Cuba status: (United Nations, Office of the Resident Coordinator)

10 fatalities.

The country has recovered 70% of its power service.

The seven provinces most affected are being assisted by other territories. It is expected that full services could take up to a month to be restored.

62% of the 320 Wi-Fi areas affected have been restored.

Cellular coverage remains and 43% of the service was recovered, even though there were 176 transmitters affected.

50% of telephone services (fixed lines, collective phones, public pay phones and internet lines) have been restored, with 79,000 cases pending.

https://reliefweb.int/report/cuba/response-hurricane-irma-cuba-situation-report-no-7-office-resident-coordinator-14092017

13

any word on cuba?

...
Response to Hurricane Irma: Cuba Situation Report No. 7 - Office of the Resident Coordinator (14/09/2017) - Cuba | ReliefWeb

bad, but more like florida than their neighbors to the east, some of
whom are almost standing on bedrock. thanks for the info.

randy

14

I said it would be the last summary report, but an important correction.

FCC found a problem with the data statistics:

[NOTE: The number of cable systems and wireline subscribers out of service is reduced significantly from yesterday because, in addition to removing from the total states/territories deactivated in DIRS, the FCC found that one or more providers had previously filed multiple entries, resulting in double/triple counting number of subscribers and inaccurate percentages per day. Those duplicative entries have now been removed.]

There are at least 1,691,484 subscribers out of service in the affected areas in Florida, down from 8,258,789.