As an FYI, it looks like Amazon is doing a mass reboot of the physical
hosts in us-west-2 across all AZ's and it is scheduled to start tomorrow
and take a couple days.
Go to *https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Events
<https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Events>:* to
see what instances are affected when.
-Grant
Bash related?
Likely not, since it's affecting Windows instances as well.
Also not just us-west-2 -- we have tons of instances scheduled for downtime in us-east-1 and eu-west-1 as well.
-Peter
Doubt it since a bash patch shouldn't require a reboot
Just got the same email. Not just US. Servers in Sydney we have also. Why
such short notice?
Unless you have a long-running bash script in the background providing
a vital system service, and that service is so important in your
environment that you might as well reboot rather than kill and
respawn it.
Likely some sort of potentially serious bug or flaw in EC2 or Xen. AWS
Security is really on the ball on such things and do everything they can to
make invisible fixes with no customer impact, but sometimes a reboot is
required in order to apply the changes necessary to keep customer instances
safe from attacks and vulnerabilities.
Another possibility: getting rid of older hardware. A reboot will keep you
in the same class of service but may move you to a new physical machine.
Unlikely though at this reported scale.
Same thing happened in December 2011 [1].
Beckman
Rumor mill is that it's XSA-108, embargoed until 2014-10-01 12:00
(http://xenbits.xen.org/xsa/). Just somebody's guess, though, afaik.
~reed
For those interested, this is the Xen bug they were fixing with the reboots
http://xenbits.xen.org/xsa/advisory-108.html
-Grant
Ouch. Good thing Bashpocalypse is still capturing everyone's attention...
Interestingly, Amazon *didn't* discover this bug, which makes one wonder why
they, out of all the big Xen-based providers out there, got a heads-up in
advance of the embargo end. If I was a big provider who didn't get advance
notice, I'd be somewhat miffed.
- Matt
Rackspace did reboots over the weekend for this as well - http://www.rackspace.com/blog/an-apology/
Bryan
read: http://www.xenproject.org/security-policy.html
they have a sensible, commonly used security policy that involves private
notification to large customers in advance where it is practical and there
is not evidence of ongoing exploits in the wild.
this is kind of incident handling 101 and shouldn't be surprising to anyone.
t
You don't have to be that large to get on the list.
There’s always people who feel “left out of the loop” when these things occur. I’ve found
there’s no one location for centralized data after many years of doing this from the
ASN.1/ILMI days to present. It requires being professional and engaging when most people
just want to consume the derived data.
Having found a few of these issues myself over the years, the best bugs are the ones
where the advisory comes out after the fixed software is broadly available and
deployed. Nothing will be perfect as people always like their legacy system
that requires no work, but in reality, there is no such thing.
- Jared