FYI Netflix is down

Seems that they are unreachable at the moment. Called and theres a recorded
message stating they are aware of an issue, no details.

-Joe

Seeing some reports of Pinterest and Instagram down as well. Amazon cloud services being implicated.

From Amazon

Amazon Elastic Compute Cloud (N. Virginia) (http://status.aws.amazon.com/)
8:21 PM PDT We are investigating connectivity issues for a number of
instances in the US-EAST-1 Region.
8:31 PM PDT We are investigating elevated errors rates for APIs in the
US-EAST-1 (Northern Virginia) region, as well as connectivity issues to
instances in a single availability zone.

-Grant

To further expand:

8:21 PM PDT We are investigating connectivity issues for a number of instances in the US-EAST-1 Region.

8:31 PM PDT We are investigating elevated errors rates for APIs in the US-EAST-1 (Northern Virginia) region, as well as connectivity issues to instances in a single availability zone.

8:40 PM PDT We can confirm that a large number of instances in a single Availability Zone have lost power due to electrical storms in the area. We are actively working to restore power.

I have an instance in zone C and it is up and fine, so it must be A, B, or
D that is down.

Nature is such a PITA.

Whatever happened to UPSs and generators?

It is my understanding that instance zones are randomized between
customers -- so your zone C may be my zone A.

Ian

I was wondering the same thing! Also, Reddit appears to be really slow right now and I keep getting "reddit is under heavy load right now, sorry. Try again in a few minutes."

I wonder if it's related. I believe they use Amazon for some of their stuff.

Derek

Yes, although, when you launch an instance, you do have the option of
selecting a zone if you want. However, once the instance is started it
stays in that zone and does not switch.

You don't need them with The Cloud!

But seriously, this is something like the third or fourth time AWS fell
over flat in recent memory.

~Seth

They may use it for content, but reddit.com resolves to IPs own by quest

8:49 PM PDT Power has been restored to the impacted Availability Zone and
we are working to bring impacted instances and volumes back online

Major storm:

http://www.washingtonpost.com/blogs/capital-weather-gang/post/severe-thunderstorm-watch-through-1-am-for-washington-dc-area/2012/06/29/gJQAY04LCW_blog.html

"Storms packing wind gusts of nearly 80 mph have just blown through
the D.C.-Baltimore region"

https://www.dom.com/storm-center/dominion-electric-outage-summary.jsp

Right around 50% of northern Virginia is without power right now.

Regards,
Bill Herrin

Whose generator worked although the first five gas stations I passed
had no power.

Streaming services and web; just tried my Roku and it failed to connect.

~Seth

Whatever happened to UPSs and generators?

They can and do fail. See list archives for numerous reports and examples :slight_smile:

Generators are capable of not starting.
ATSs can get into a situation where they don't transfer loads properly, or they can't start the generator(s)
UPSs can fail, drain out, or be left in bypass.
Breakers can trip and need a manual reset
etc...

jms

you know what's happening even more?

..Amazon not learning their lesson.

they just had an outage quite similar.. they "performed a full audit" on
electrical systems worldwide, according to the rfo/post mortem.

looks like they need to perform a "full and we mean it" audit, and like
I've been doing/participating in at dot coms for a decade plus: Actually Do
Regular Load tests..

Related/equally to blame: companies that rely heavily on one aws zone, or
arguably "one cloud" (period), are asking for it.

Please stop these crappy practices, people. Do real world DR testing.
Play "What If This City Dropped Off The Map" games, because tonight, parts
of VA infact did.

Down: Instagram, Pinterest, Netflix, Heroku, Woot. Pocket(Read It Later),
and on and on. A bunch of openID sites. A bunch of DNS sites (think
zoneedit et al). Infact, probably nearly a /12 if not more of space..

Blame lies both with AWS (again) and with these services providers.

They all should know better.

-j

Zoneedit doesn't seem to be down . I can both use the website and
resolve my domains.

I am not a computer science guy but been around a long time. Data centers and clouds are like software. Once they reach a certain size, its impossible to keep the bugs out. You can test and test your heart out and something will slip by. You can say the same thing about nuclear reactors, Apollo moon missions, the NorthEast power grid, and most other technology disasters.

well one would think that they could at least get power redundancy right...