RightScale: How to Prepare for Amazon's EC2 Instance Reboot
Amazon Web Services notified customers on Wednesday that it would be conducting a big reboot of a substantial number of Elastic Compute Cloud (EC2) instances between Sept. 26 (yes, today) and Sept. 30.
September 26, 2014
Amazon Web Services notified customers on Wednesday that it would be conducting a big reboot of a substantial number of Elastic Compute Cloud (EC2) instances between Sept. 26 (yes, today) and Sept. 30.
As Amazon reboots many of its EC2 instances to install patches, RightScale CTO Thorsten von Eicken noted in a blog post that this reboot of the instances is different from past reboots, and provided a list of recommendations to ensure customer EC2 instances continue to run smoothly.
According to von Eicken, the key differences between this reboot and past ones, such as the December 2011 instance reboot, are twofold:
"A substantial number of instances will be rebooted," von Eicken noted. But, "AWS has said that not all instances of the impacted instance types will be rebooted."
More importantly, it's important not to relaunch an instance during the maintenance upgrade, as it could result in the instance not getting the critical patches.
"The second point is really the critical one," von Eicken wrote. "Normally, whenever our Ops team receives a maintenance notice regarding a specific set of instances, we relaunch them as soon as possible at our convenience so that by the time the maintenance windows arrives, our instances are already on hosts that have had the maintenance done."
This time, though, because of the lack of patched capacity and the sheer scale of the EC2 instance reboot, there's no guarantee for instances that are relaunched during the maintenance window.
For customers and partners facing this issue, von Eicken provided this list of recommendations to follow:
Read the details of the maintenance notice you receive from AWS.
Check the AWS console "Events" page for affected instances. The AWS console will be your most up-to-date source, so don't depend on email notifications.
Relaunch these instances as soon as possible in a controlled manner to "snatch up" patched host capacity before others get them. You can also try to relaunch on instance types that AWS says will not be affected.
Wait awhile, then check the AWS console to verify that you indeed got patched hosts. Note: AWS has a script running to update the AWS console with maintenance notices on newly launched instances. By midday Thursday Pacific time, AWS hopes to have these notices updating every one to two hours.
If you didn’t, try again a bit later.
Double-check periodically to make sure no instance is left subject to maintenance.
Make that you have the appropriate alerts set up.
Plan to monitor closely your AWS-based applications through the maintenance window.
Not all instance types will be affected. Included in those not to be touched are T1, T2, M2, R3 and HS1 instance types. The reboot will occur in all regions, but even instances on the list to be rebooted won't necessarily be affected.
"As usual, AWS is totally tight-lipped about the underlying cause. It seems obvious that the company is patching a security vulnerability, but it will not disclose which one until October 1—that is, after they have patched all hosts," von Eicken wrote.
About the Author
You May Also Like