Reboot Command Typo Takes Down Joyent US-East-1

Who says spelling isn't important these days? Or at least the ability to type without mistakes? Students, take note, because a simple typo took down the entire Joyent US-East-1 data center this week. According to Joyent, an "operator error" mistyped the command to reboot a select set of new systems, instead specifying a reboot of all systems.

Mental note to all future admins: Double-check your typing before executing. All told, the downtime wasn't severe, but certainly longer than it should have been. Joyent's Eastern data center was down for at total of one hour, 17 minutes.

"Unfortunately, the tool in question does not have enough input validation to prevent this from happening without extra steps/confirmation, and went ahead and issued a reboot command to every server in us-east-1 availability zone without delay," the Joyent team wrote on the company blog.

Joyent goes into detail about the technical aspects of the downtime cause, but in the end, it was essentially a typo, showing that more often than not, human error is the biggest cause of downtime. And such disasters can be avoided.

Joyent noted in its blog post that it "will be taking several steps to prevent this failure mode from happening again, and ensuring that other business disaster scenarios are able to recover more quickly." Joyent's plans are three-fold:

The company apologized for the downtime, and it looks like it has a sound plan in place to deal with any future potential issues.