Rackspace CEO Apologizes for Server Downtime

Rackspace (RAX) President and CEO Taylor Rhodes yesterday sent a letter to customers apologizing for any downtime during server maintenance this past weekend.

Rhodes, who was appointed as the company’s CEO in September, said some of the San Antonio, Texas-based managed cloud hosting company’s customers were affected by the Xen bug, a recent security vulnerability in software used by several major cloud services providers.

He said the maintenance affected approximately 50,000 of Rackspace’s customers. According to him, “this issue has been fully remediated, without any reports of compromised data among our customers.”

In his email to customers, Rhodes wrote:

“I’m writing to apologize for the downtime and inconvenience that you and others of our customers have suffered in recent days.  Like other major cloud providers, we were forced to reboot some of our customers’ servers to patch a security vulnerability affecting certain versions of XenServer, a popular open source hypervisor.  This maintenance was especially difficult for many of you because it had to be performed on short notice, and over the weekend.”

Rhodes said Rackspace worked with Xen partners last week to develop and test a patch, and create a reboot plan, but the patch wasn’t ready until Friday evening. Instead of waiting for the technical details of the vulnerability to become public, Rackspace began server reboots over the weekend to avoid exposing its customers to any additional risk.

Rhodes explained the dilemma companies have with patching security vulnerabilities:

“We decided the lesser evil was to proceed immediately, at which time we notified you, and our partners in the Xen community, of the need for an urgent server reboot.  Even then, to avoid alerting cyber criminals, we didn’t mention Xen as the reason for the reboot. Another major cloud provider did attribute its reboot to security problems with Xen, which put all users of the affected versions of that hypervisor at heightened risk.”

He also noted that some of the server reboots took much longer than they should have and that Rackspace’s notifications weren’t as clear as they should have been. But it’s something he plans to work on going forward.

“As a veteran Racker who is proud of our commitment to our customers and their businesses, I am personally sorry for any inconvenience or downtime that we caused you during this incident,” he said.

Follow CJ Arlotta on Twitter @cjarlotta and Google+ for further updates on the story above — or if you just want to say hello.