Intermedia CEO Apologizes for Cloud, Hosted Exchange Outage

By samdizzy

Intermedia is working hard to earn forgiveness from customers and partners this evening. After suffering a hosted Exhange and cloud services outage that rocked many VARs and MSPs, Intermedia CEO Phil Koen on Sept. 3 published a blog explaining the outage and corrective measures — and asking customers to hopefully accept his apology.

In it Koen wrote:

To our customers and partners,

As you know, Intermedia experienced a significant network interruption today. As CEO of Intermedia, I want to offer a profound apology.

I also want to explain what happened, what we’ve learned, and what we’re going to change.

First, you should know that our services are now stable. There has been no data loss and there were no security breaches. And all of the emails that were sent to you during this issue will be delivered as the backlog is processed.

Here’s what we know.

Around midnight on Wednesday night, August 28th, we experienced an issue with our core routers. We implemented a fix to this issue, and we believed that it was completely resolved by 3 AM EST on Thursday.

At 7 AM EST on Tuesday, September 3, we observed some further anomalies on our core routing equipment. Our attempts at intervention were unsuccessful, and corruption in our core routing table propagated to the edge routers in all our US datacenters. This created significant packet loss between the edge and core in each datacenter and it prevented delivery of service to our customers. In addition, as our communication systems reside in the same datacenters, our ability to communicate with customers and partners was disrupted.

At 8 AM EST on Tuesday, we started restoring network connectivity by systematically rebooting each of the affected network devices. Our network and services were brought back online throughout the day, with complete network restoration by 3:30 PM EST.

As connectivity was restored to each datacenter, our system administration team focused on resolving any residual server issues caused by the network outage. The final server-related issues were resolved by approximately 6:30 PM EST. Our support team was able to take phone calls beginning around 3 PM EST.

What comes next.

As of now, all our services are back online. We now have three tasks ahead of us.

The first task is to complete our RFO (Reason For Outage) report to fully identify the root causes. We’re working with our network equipment vendor to complete this report, and we will share it with our customers and partners as soon as we can.

Second, we will take the recommendations from this RFO and make any changes necessary to improve our stability and resiliency.

Third, we will also improve the responsiveness and robustness of our customer notification tools and systems. Although we were successful in notifying many of our customers about the issues via alternate email addresses, text messages and HostPilot, not all customers were reached. Going forward, we will make more timely use of our website and social media—especially Twitter and Facebook.

I hope you accept my apology.

On behalf of all of us at Intermedia, I offer a profound apology. We understand how essential our services are to your business. We fully appreciate the breadth of our responsibility. We live and breathe it every day.

I’d appreciate any feedback you can share. Please email [email protected]. This mailbox is monitored by Intermedia’s senior management team.

Sincerely,

Phil Koen

CEO, Intermedia

Extremely Bad Timing

There's never a good time for a cloud services outage. But Intermedia's setback on Sept. 3 was especially painful. The reasons:

Still, the only proper thing to do in an event such as this is to eat humble pie, apologize and communicate future corrective actions. It took Intermedia a few hours on Sept. 3 to sort through the challenges but Koen's blog entry was a welcome note later in the day.