How Managed Service Providers Handled CrowdStrike Outage
Lessons learned from CrowdStrike outage: use layered security and constantly communicate.
The CrowdStrike outage last week that caused a great deal of pain for IT teams around the world, also offered an opportunity for managed service providers to prove their value.
Regardless of how much harm the CrowdStike outage caused their customers, leading MSPs moved quickly to address problems and concerns. That included getting clients who experienced the Windows blue screen of death back online, and keeping customers informed about the situation.
Meriplex's Neil Medwed
“We had a major success story – a company that was not a customer was affected and called us to help them,” said Meriplex vice president of corporate development Neil Medwed. “We were able to dispatch to their location within one-and-a-half hours of their initial ask for help and we engaged a team of four techs that worked all weekend to get them back up and running so their business was fully operational for Monday morning.”
Houston-based Meriplex (No. 50 on the 2024 MSP 501 list) works with enterprises and other large customers, and Medwed said some were affected by the outage.
“Our team worked quickly to restore their environment,” he said.
Mann Consulting's Harold Mann
As an MSP with an Apple-heavy business, San Francisco-based Mann Consulting had fewer than 1% of its client devices impacted, according to Mann president Harold Mann. He said the total came to about 50 to 60 impacted devices. Mann said his company quickly contacted all his CrowdStrike-impacted clients to alert them of the outage.
“While it's easy for the Mac community to gloat, I think any responsible IT person knows that you can substitute CrowdStrike for whatever their core application stack is and it could be just as challenging,” said Mann, whose company is also on the MSP 501 list. “It's a strong wake-up call for MSPs to regularly do tabletop exercises for black swan events and to use this crisis to have strong conversations not only internally but also with customers."
“Of course it does underscore the challenges with some of the Microsoft OS architecture, and it also highlights the importance of rolling out updates more gradually," Mann said. "Any Mac MSP that brags that this is yet another reason not to use Windows should be wary. The entire computing world can be much stronger as a result of this, if they choose to change how they do things.”
Marc Umstead, president of Plus 1 Technology, said his Pottstown, Pennsylvania-based MSP does not use CrowdStrike, but he reached out to his clients as soon as he heard of the outage to keep them informed.
“I learned over the last couple of years the most important factor during these events is communication,” Umstead said. “Anytime there is anything technology related that makes national news, we make sure to get communication out to all of our clients as soon as possible. When the CrowdStrike news hit, we had an email out to our clients by 5 a.m. to let them know that while they are not utilizing CrowdStrike they may still experience issues with some of the SaaS applications and websites they use daily.”
Plus 1 sent out a video over the weekend, summarizing the CrowdStrike issue and providing tips for avoiding scammers looking to take advantage of the situation.
“We have found that keeping clients informed with the information they need helps reduce their anxiety and helps eliminate 100s of ‘What is going on?’ phone calls,” Umstead said.
MSPs: Don’t Be Too Hard on CrowdStrike after Outage
Even MSPs who do not use CrowdStrike said partners and customers should not be too tough on the cybersecurity vendor. Several pointed out similar outages have happened previously and will likely happen again with other vendors and cloud services.
“Companies make mistakes, it's how they handle this that will change my perspective on them, in either direction,” Mann said. “I would imagine CrowdStrike has averted far more BSODs (blue screen of deaths) than they've caused.”
“The key for MSPs is to not play the blame game or try to use these events as some selling tool,” Umstead said. “It was CrowdStrike today. It could be Sentinel One, Citrix, Cisco or any other vendor tomorrow. The key is communication and having a plan for how to deal with these events.”
Jason Rinker, chief revenue officer of Joplin, Missouri-based Stronghold Data (No. 138 on the MSP 501), noted how CrowdStrike’s stock fell 13% Monday after an 11% dip on Friday.
“I am sure they will bounce back, but they have work ahead of them,” Rinker said of CrowdStrike. “My heart goes out to all of them as they work through this issue.
One lesson for MSPs from the incident is to not rely on one vendor for security. Ron Lovern, executive vice president of Dallas-based Triton Networks, said Triton does not use CrowdStrike, and his clients were not impacted. Lovern said Triton (No. 32 on the MSP 501 list) takes a layered approach to cybersecurity with tools from vendors including Auvik, Cisco, Fortinet, Microsoft and Todyl. This allows for controlled management of network, end user devices, application services, updates, and service renewals.
“Some of our vendors had issues that did cause some problems on the backend but did not directly impact our end users,” he said.
“We utilize a combination of security partners and network tools) to manage end users and devices through Triton One Solutions platform. What I think we will see going forward is a similar approach from other MSPs and large enterprise clients.”
Len DiCostanzo, CEO of MSP Toolkit, said the CrowdStrike incident creates an opening for competing vendors and MSPs to show customers and potential customers how they can bolster security.
“In working with my vendor clients, I do see an opportunity for those in cyber,” he said. “One [vendor] has built a cadence to offer help to CrowdStrike affected partners and their clients and free use of their cyber platform for 30 days while they work to come back online. As always, after an incident like this, there are angry people and there will be lift and shift/rip and replace opportunities for sure.”
Read more about:
MSPsAbout the Author
You May Also Like