The Rackspace Fallout: Details Scarce After Weekend-Long Email Outage

The Rackspace outage, which put thousands of email users offline over the weekend due to a cybersecurity event, is a lesson for partners in disaster planning and customer support.

The yet-to-be-disclosed security incident took down Rackspace’s Microsoft Hosted Exchange environments. The cloud services provider early Monday morning announced that it had resorted email services “to thousands” of Microsoft 365 customers, but partners and customers continue to voice their concerns about the vendor’s response to the crisis.

StratusDial’s Bill McClain

“We’re looking at it they’re never going to be able to recover on that platform. We’ve moved on,” said Bill McClain, founder and president of MSP Networking Solutions and voice provider StratusDial.

Rackspace early on Friday announced that it was investigating connectivity and login issues in its Hosted Exchange environments. The company turned off the environment to further analyze the problem.

Rackspace continued to describe the problem as “connectivity issues” in multiple updates throughout the morning. A day later on Saturday, Rackspace concluded that a “security incident” had impacted the environment. It added that the Rackspace email product line and platform had not experienced any problems. Only Hosted Exchange accounts had suffered the incident, according to the company.

Rackspace encouraged customers to open new Microsoft 365 accounts and offered them free Microsoft Exchange Plan 1 licenses. It added on Saturday that it had dispatched 1,000 people to support customers. The company said employees made contact with customers either by phone or an alternative email address, although many customers complained on social media about long waits on call support lines.

Rackspace when contacted by Channel Futures pointed to the last update it made on its website.

“We continue to make progress on restoring email service to every affected customer,” the company wrote. “At this time, moving to Microsoft 365 is the best solution for customers who can now also implement temporary forwarding. To assist customers through options, Rackers are contacting every Hosted Exchange customer by phone. Customers will also be contacted via alternate email addresses. This outreach is being performed in addition to chat, phone, and ticketing.”

Customer Impact

Nevertheless, the outage drew complaints from customers on Twitter.

Tiffany Dowd, president and founder of the social media consulting company Luxe Social Media, took to Twitter to air her frustration.

“There has been a major outage affecting small businesses worldwide – including mine! No emails and I’m traveling on business for several weeks. Unacceptable,” she wrote.

Attorney Ronald Richards criticized how Rackspace communicated throughout the weekend.

Iman Jalali, managing partner at Bear Peak Capital, pointed to layoffs as a reason for the delayed restoration and subpar support.

Security Culpability?

Details remain to be seen about the actual security incident and any security posture from Rackspace that may have led to the problem. However, partners have pointed to systemic failings.

Lawrence Technology Systems’ Tom Lawrence

“Just from the standpoint of them not being able to get the systems back up in a reasonable amount of time and running shows there is some type of failure in their disaster recovery process,” said Tom Lawrence, who runs the Michigan-based MSP Lawrence Technology Services.

That said, Lawrence said he isn’t jumping to too many conclusions.

“I do try to avoid victim blaming because the reality of these attacks is that threat actors are doing these things for profit, and until there is a debrief we don’t know if they had an adequate level of protections or not. Was this a new sophisticated attack or were their servers left unpatched? We just don’t know yet,” he told Channel Futures.

Lawrence added that Microsoft’s slowness to respond with a method for patching issue compounded the problem.

Peter Radizeski, founder and president of the advisory firm Rad-Info, emphasized the importance of how a vendor responds to a disruption like this.

Rad-Info’s Peter Radizeski

“Outages happen. Period. How the vendor responds to that outage is the difference,” Radizeski told Channel Futures. “In my 23 years of experience, most outages are treated like a nuisance. They should be treated the way Tylenol reacted to poisoning in the 1980s. It is crisis management. It is reputation management. Being pro-active and forthcoming in communications is the key to maintaining trust. Everyone understands today that outages happen – every provider has had a major one at this point: AT&T, Rogers, Lumen, Verizon, Netflix, AWS, Rackspace. How the company reacts is what partners are looking at – especially for the coveted enterprise accounts. No one will risk moving a large account to a provider who fails during an outage. And the funny thing is, communicating during this time isn’t that difficult,” Radizeski told Channel Futures.

Partner Response

McClain said members of his team were working over the weekend to restore email access to customers. McClain said the first ticket from a customer appeared at 6:30 in the morning on Friday followed by a steady stream of tickets. Because Networking Solutions uses hybrid domains from Rackspace, it needed to migrate all of its Rackspace users to new accounts – even those not using Hosted Exchange.

McClain said his team has moved approximately 150 users and about 600 mailboxes and is nearing completion. They called Rackspace on Friday and didn’t get an answer. They called again on Saturday, also to no avail.

“I pretty much told my team, ‘We have to do whatever we have to and assume we’re not going to have any help whatsoever from Rackspace,’ and that was the case,” McClain told Channel Futures..

Part of the challenge was the uncertainty about..-… what exactly had caused the Rackspace outage and how badly the outage had compromised the system. As a result of not knowing the full scope of the problem, McClain’s firm chose to pick an alternative.

“Saturday morning, we went with the decision to not wait – to rip the Band-Aid off. We don’t know what we’re waiting for. We don’t want it to be Thursday and then decide, ‘Hey, we need to rip this off.’ So we had them up and running Sunday.”

Netgain’s Sumeet Sabharwal

Sumeet Sabharwal, CEO at the MSP Netgain Technology, said the incident serves as a reminder for all MSP owners about their value.

“In crises like these, our approach makes the world of a difference. This includes leading with transparency, prioritizing and communicating effectively across all media, committing leadership to be visible and on the frontlines, keeping internal teams motivated via ongoing outreach, and coordinating efforts across all internal functions including Billing. These moments of crisis reveal a great deal about the company and the character of the leaders operating it,” Sabharwal wrote on LinkedIn.

McClain agreed.

“Companies like mine get their best accolades during emergencies. That’s when the cream rises to the top,” he said.

Takeaways

Tom Lawrence shared takeaways from the Rackspace outage. First, he argued that customers should establish an email backup. Moreover, they should put a plan in place for any extended outage from their cloud provider. He also argued that letting someone else host and maintain your applications helps customers cut down on risks and costs.

He also offered advice for Exchange users.

“Anyone still using Exchange on-prem or hosted should be looking to get off of that platform as Microsoft has made it clear they only care about charging you for licensing fees, but don’t care about the security of the product,” Lawrence said.

McClain there exists in technology, and especially in the SaaS world,two types of crises: emergencies and planned emergencies. The best providers, he said, build out contingencies for when things go wrong.

“What I tell my clients is, ‘We will do our best to protect from the bad guys. But we will also simultaneously prepare for when they do get through,'” he said.