The Gately Report: One MSP's Account of Devastating Ransomware Attack
Plus, why the hackers behind the high-profile SolarWinds attack targeted Microsoft.
![Ransomware attack long-term impact Ransomware attack long-term impact](https://eu-images.contentstack.com/v3/assets/blt10e444bce2d36aa8/blt81e9b3d9f8878477/6567889c1bfe7d0407ac7f5d/Ransomware.jpg?width=700&auto=webp&quality=80&disable=upscale)
Carlos Amarillo/Shutterstock
Channel Futures: How did this ransomware attack start?
Robert Cioffi: I think we were just one of the unfortunate, unlucky ones that were early in the process of this attack. They gained unauthorized access to our system. They defeated security by exploiting a zero-day exploit that Kaseya had known about and was working on behind the scenes. We did not know. Had we known, we would have taken other preventative measures. The attackers uploaded REvil Sodinokibi ransomware to our Kaseya VSA server and pushed a script to install it on all 2,500 of the endpoints that we managed on that day. So that number included about 250 servers and the balance of which were laptops and desktops.
CF: Did REvil make a ransom demand?
RC: I believe the ransom demand was $70 million from Kaseya for a universal decryption key for all of their customers, or individual customers like us could pay $6 million or $45,000 per file type. Now, the last option was almost meaningless. Why would I only want to recover Word documents? So we had the option to pay $6 million and our response was. "Go pound sand."
CF: How did you respond to this? How did it cripple your business?
RC: It did cripple us in the most horrific way because we had every single one of our accounts that we were servicing, some of which for decades, all their systems were completely encrypted. So it didn't just affect us. It affected the 80 customers that we were servicing as of that day. We had 80 paying customers that had Kaseya VSA installed on their systems and all of their systems had now become crippled. So it wasn't just a matter of recovering my own business, but now 80 other entities as well. So how the hell do you do that? We had a very big math problem on our hands.
So over that weekend, because this was Friday when the attack took place and July 4th was observed on Monday, we had a little bit of cushion there to kind of catch our breath. So the timing, even though it was awful in some ways, was actually a little OK in others because forensically we were able to determine the nature of the attack, when it took place, and that it was a smash-and-grab style attack, meaning there was no data exfiltration, or the overwhelming evidence suggested that there was no data exfiltration, so we had a recovery time objective.
We knew that as of the morning of July 2, we could restore from backups on those days. And we were very confident in our backup technology that we were using and that we would be able to recover those servers. Now desktops and laptops, it was a completely different story. Those machines needed to be wiped from scratch and reinstalled. And if there were any data, any files, any settings on them, first of all those files were lost permanently and we would have to just deal with rebuilding machines from scratch. And by the nature of the way we manage our systems for our customers, the vast majority of customers didn't really have anything all that important saved on their local machines anyway. They shouldn't be working that way. It should either be in the cloud or on the server.
CF: Did you get help from anyone during the recovery process?
RC: Our friends, MSPs who showed up en masse from across the country, came on airplanes and stuck it out with us for some time, some of them upward of two weeks. That augmented our staff so that we had the numbers to be able to do the recovery. So it took us about 17 calendar days. By July 19, we had about 95% of systems restored, and I'm including all desktops and laptops, not just servers.
None of the MSPs that I know were victims of any other cyber crime, and certainly not in that particular moment because if they were affected in that moment, they would have been dealing with their own apocalypse. If you're a Lord of the Rings fan, it was like the Riders of Rohan showing up just in the nick of time to turn the tide of the battle. And that's exactly what happened. Someone lit the fire, that first beacon, to summon the riders. One friend told two friends, those two told two others and sure enough, we had about 30 different companies responding to help us at various levels.
CF: Did your company incur a big financial impact from this attack?
RC: Sure. I mean, who doesn't? You're going to have customers that leave you because they've lost their trust. There were all sorts of reasons. Let me give you one crazy reason. One customer left us because their largest customer that represented 10% of their revenue said fire your MSP. They didn't want to; we had a good relationship with them and they were probably in our top five in terms of annual recurring revenue. It was a big account, and they fired us because their largest customer said get rid of them. We were tainted. Our reputation was damaged, even though there was no wrongdoing on our part, that we had followed every single security protocol that was recommended by Kaseya. We were actually even patched to the very latest patch version, and have actual documentation and proof of this. Yet we were still held liable in the eyes of the public in some ways. So you suffer reputational damage, which then leads to economic damage.
CF: Were there any lawsuits?
RC: We had a customer sue us. They didn't get very far, but they tried. We had a number of accounts that ended up walking away from us simply because they just felt that we were no longer their trusted IT company. But we also had most of our customers say the opposite. I had comments from one customer who said, "We thought about firing you guys, but then we thought about it, and the way you guys led us through that war and the way you guys walked through hell, we can't think of a better IT company to help us in our time of need should something like this ever happen again."
Others said. "It was terrible for us, but we empathize how much more horrible it was for you guys." We had gift baskets and people showing up at our door, sending food. They knew and understood that a) this was not our fault, and b) we were in a terrible position and that our very existence was at risk. Those are the customers that felt the true partnership. When you have a partner, you care about the other side. Just because they're the vendor and providing you the services, they're a partner, they're an extension of your business. And a lot of our customers saw that and I think responded in kind. Others elected not to.
CF: In the aftermath, was there anything you’ve done or were able to do to prevent something like this from happening again?
RC: In the month of August of that year, about 30-45 days after the attack, I began to reflect on what we had just gone through. Most of the war was over, but there was still a lot of clean-up going on. One of those reflections was the community response that we received and it dawned on me that there are probably a lot of MSPs out there that were not as well connected or well invested in the MSP community like we were. So just instinctively I registered the domain name MSP911.org because I thought about those first few moments of panic, the sheer terror that I felt and the frozen state that I was in, that I didn't know what to do or who to call, or how to even begin to even think about how to fix this. The problem was so overwhelming it just runs over any of the best of us and puts us in a state of pure panic.
So I thought, wouldn't it be wonderful if there was a website that I could have gone to and hit the giant red button to say I need help, and have people in the industry … that would be willing to respond and say, "I don't know you from a hole in the wall, but you have a problem and I'm going to see what I can do to help you because you're part of this community and there's a common enemy that just attacked you." So long story short, that was the vision of MSP911.
CF: Did MSP911 become a reality?
RC: I have since donated it to CompTIA, and CompTIA has branded it as the CompTIA Emergency Response Team, of which I am the chairperson. It has a bunch of people who have also been victims and or have direct experience in dealing with incident response. Just overnight, we had someone submit a form on the website saying, "I've been attacked and I need some help."
A week and a half ago, we got another one where I am providing I'll say friendly coaching and counseling to someone who owns an MSP, and who's at her wits' end with the emotional and financial toll it’s taking on her. And I'm just simply there to converse with her, to be a friendly voice. If at minimum this is what we can do, then we're helping change the world because I don't want anybody to ever go through a situation like what we went through alone. It was hard enough to do it with the amount of help that we had, and it almost killed us anyway. Thankfully, it did not, but I can't imagine somebody going through this alone. I don't see how anyone can mentally get through an attack like this without having at least a friendly voice to talk to. We're still working on the legal aspects of this. But we want to be able to have volunteers who are willing to show up and pitch in, just like all those volunteers helped us. So that's the vision of what we're working on. It's mostly built. It's there and it's a resource that's available to the community.
CF: Any lessons learned from this experience?
RC: No. 1, you've got to have a cyber liability insurance policy. You need to have one for yourself. It's non-negotiable. It's permission to play these days. And I would also encourage them to demand the same of their customers. Now that's a hard road because not everybody is going to want to buy one. But you’ve got to protect yourself as well and make sure that they understand that cyber incidents are not covered as part of your standard managed services agreement. And that's the purpose of having a cyber policy in place, because there's nothing guaranteed in this world in terms of security, and if somebody wants to get into your systems, they're probably going to find a way, despite all the wonderful protection that you supply as the MSP. So this is a cost of doing business and it's the way we need to look at it. It's about risk management.
On that afternoon of July 2, it took my director of finance to tap me on the shoulder while I was in that chaotic state to say, "Should we call our cyber policy?" And I said, "Oh, yeah, that's a brilliant idea' let's do that."
It's surprising how many people will have a policy and then forget to activate it. It's really important to make that phone call early because of the resources that are in that policy, including the legal protection, forensics and crisis management services. These are things that you're going to need very early before you start making decisions.
CF: Did you contact the federal government about the attack?
RC: One of the early things that I did as well was file an FBI report, and within an hour or two, I was speaking to an FBI agent. And I think in this particular case, because it was such a widespread global attack, and started to really hit the news hard and heavy that afternoon, I was invited into different calls with various FBI agents and conference calls with other victims, and those are some of the same agents that were there in Dallas, in the courtroom, because these are people that had been working the case since the hours after it had unfolded.
It's important to get law enforcement involved, at least for their awareness. A lot of people think, "What are they going to do?" For instance, the FBI … can recover assets quickly. If you're involved in a case where there's a legal wire transfer, for instance, the earlier you call them, the quicker they can get that money back. But if you wait too long, that money may be gone forever. So do those two phone calls first, insurance and law enforcement.
In other cybersecurity news …
Last week, we reported that Nobelium, the Russian nation-state hacking group behind the massive SolarWinds attack in 2020, has targeted Microsoft, compromising a small number of email accounts, including those belonging to senior staff.
In 2021, Nobelium targeted Microsoft customers, with some of them successfully compromised.
Yoad Fekete, co-founder and CEO of Myrror Security, a software supply chain security provider, was part of the Microsoft incident response team for the SolarWinds attack back in 2020. We spoke with him about why Microsoft once again found itself in the crosshairs of Nobelium.
For starters, Fekete isn’t surprised that Nobelium has once again targeted Microsoft.
“Unfortunately if there's a will, there's a way,” he said. “In the case of APT29 (a Russian state-sponsored hacking group) and Microsoft, we're speaking about nation-state actors that are trying to gain access to an enterprise with many legacy environments that unfortunately always have a door open or a misconfiguration left hanging."
Microsoft is a great organization with amazing security teams across the board, but “we can't expect a company that's been around for as long as Microsoft has to not miss a vulnerability from time to time,” Fekete said.
“In this specific attack, there's nothing fancy at play,” he said. “The hackers seemingly exploited a misconfigured account where multifactor authentication (MFA) was missing, enabling them to gain access to corporate resources from a test environment. Detecting an MFA problem is generally easy. However, the ability to access corporate resources from the test environment indicates an architecture/networking problem, which is harder to pre-detect with existing solutions. So, to defend your organization from these attacks, we go back to the usual cliché ideas: make sure you have visibility into all of your assets, perform continuous threat modeling and adjust your security controls accordingly.”
Going back to the 2020 SolarWinds attack, organizations need to expand the scope of their security measures to newer attack vectors, Fekete said. SolarWinds needed a solution to verify their building integrity, same for 3CX.
“Organizations should be monitoring all open-source components, third-party dependencies and vendor systems within their software supply chain for vulnerabilities and abnormalities while prioritizing any genuinely exploitable and reachable findings,” he said.
When it comes to nation-state attackers, they usually have predefined targets, such as U.S. government assets, Fekete said.
“It's not a secret that Microsoft has a U.S. government cloud, which is the crown jewel for such actors if compromised,” he said. “Whether that was the case here I can’t say, but it’s a good question to ask.”
Cybersecurity vendors announced fewer, but larger acquisitions in 2023 compared to the long-term trend, according to Canalys.
For the full year, 30 acquisitions were announced by the 41 cybersecurity vendors tracked by Canalys. The value of the 30 announced deals totaling $32 billion.
Cisco acquiring Splunk, at $28 billion, was the largest announced deal, followed by Palo Alto Networks acquiring Talon Security for $625 million.
Cisco was the most active vendor, announcing seven acquisitions, including Armorblox, Lightspin, Valtix, Oort and more. Check Point Software Technologies announced three acquisitions, and IBM, Okta, Palo Alto Networks and SonicWall each made two acquisitions.
Matthew Ball, chief analyst at Canalys, which shares a parent company with Channel Futures (Informa), said startup valuations remain higher than pre-2020 levels, which led to fewer, but larger deals being made compared to the long-term average.
“Cisco-Splunk also skewed the numbers,” he said. “Vendors made acquisitions across a broad range of technologies, but expanding secure access service edge (SASE) platforms was the most frequent driver alongside cloud security. We expect current volumes of deals to continue in 2024, as cybersecurity vendors expand their platform strategies further.”
In the meantime, for the fourth quarter of 2023, new investment secured by pre-IPO cybersecurity vendors declined by 49% to $1.68 billion and was down by 52% in 2023, according to Canalys.
High costs of credit has led to a sustained pullback of venture capital, resulting in startup and growth-stage cybersecurity vendors scaling back expansion plans and cutting operating costs, it said.
Emerging inflationary pressures have increased the uncertainty of the timing and frequency of interest rate cuts. This will maintain the current constrained activity, at least for the next two to three quarters.
Identity and access management (IAM) and fraud prevention were key areas of funding in the fourth quarter, according to Canalys. Funding for vendors developing security for artificial intelligence (AI) large language models (LLMs) and secure adoption of LLMs, as well as compliance and governance was prominent.
Cybersecurity vendors announced fewer, but larger acquisitions in 2023 compared to the long-term trend, according to Canalys.
For the full year, 30 acquisitions were announced by the 41 cybersecurity vendors tracked by Canalys. The value of the 30 announced deals totaling $32 billion.
Cisco acquiring Splunk, at $28 billion, was the largest announced deal, followed by Palo Alto Networks acquiring Talon Security for $625 million.
Cisco was the most active vendor, announcing seven acquisitions, including Armorblox, Lightspin, Valtix, Oort and more. Check Point Software Technologies announced three acquisitions, and IBM, Okta, Palo Alto Networks and SonicWall each made two acquisitions.
Matthew Ball, chief analyst at Canalys, which shares a parent company with Channel Futures (Informa), said startup valuations remain higher than pre-2020 levels, which led to fewer, but larger deals being made compared to the long-term average.
“Cisco-Splunk also skewed the numbers,” he said. “Vendors made acquisitions across a broad range of technologies, but expanding secure access service edge (SASE) platforms was the most frequent driver alongside cloud security. We expect current volumes of deals to continue in 2024, as cybersecurity vendors expand their platform strategies further.”
In the meantime, for the fourth quarter of 2023, new investment secured by pre-IPO cybersecurity vendors declined by 49% to $1.68 billion and was down by 52% in 2023, according to Canalys.
High costs of credit has led to a sustained pullback of venture capital, resulting in startup and growth-stage cybersecurity vendors scaling back expansion plans and cutting operating costs, it said.
Emerging inflationary pressures have increased the uncertainty of the timing and frequency of interest rate cuts. This will maintain the current constrained activity, at least for the next two to three quarters.
Identity and access management (IAM) and fraud prevention were key areas of funding in the fourth quarter, according to Canalys. Funding for vendors developing security for artificial intelligence (AI) large language models (LLMs) and secure adoption of LLMs, as well as compliance and governance was prominent.
Robert Cioffi knows what it’s like to have your entire life turned upside-down and left permanently scarred by a ransomware attack.
He’s CTO and co-founder of New York-based Progressive Computing, one of 60 MSPs hit with ransomware attacks just hours before the July 4th weekend in 2021. Affiliates of the REvil ransomware gang launched cyberattacks on Kaseya and customers using Kaseya’s VSA product, including Cioffi’s business.
Earlier this month, Cioffi stood in a federal courtroom in Dallas to give a victim statement during the sentencing of Yaroslav Vasinskyi, a Ukrainian national, charged with conducting the ransomware attacks. Sentencing will be in March.
“My work is not finished,” Cioffi said. “Even when Yaroslav Vasinskyi is finally in a jail cell living out his sentence, the work is not done. We need to continue to fight the good fight here and make sure that we are helping each other as an industry be better, to help all those that we interface with. Law enforcement, the U.S. Department of Justice and our politicians, they need to be hearing from us. We need to do things differently than what we have been doing. And if we don't stand up and take that voice, then we're just going to be right where we are every day. And everyone who's reading this is going to say, 'Good Lord, I dread that nightmare that Robert went through, and I just hope it's not me next time.' Well, it's coming for you. And if you don't do something about it, then you're next.”
Lasting Trauma from Ransomware Attack
During his victim impact statement, Cioffi said he doesn’t have a medical diagnosis of PTSD, “but it sure as hell feels like I do.”
![Progressive Computing's Robert Cioffi Progressive Computing's Robert Cioffi](https://eu-images.contentstack.com/v3/assets/blt10e444bce2d36aa8/bltb664cd1534ba7ea4/65b41d882a9c7d040ac4f5d4/Cioffi_Robert_Progressive_Computing.jpg?width=700&auto=webp&quality=80&disable=upscale)
Progressive Computing's Robert Cioffi
“So will I ever be whole? “I don't know," he said. "Are there others here that suffer some of that feeling as well? I can shake my head and nod 'yes.' We are incredibly proud of our accomplishments and what we were able to do in the short order that we were able to do it in. It was a monumental effort to get this thing done and we did it. And we did it because of our strong company culture and the way we feel for each other as a team. So for us, it was a massive victory in some ways that we were able to kind of come through this, singed and all, yet made whole because we demanded it of ourselves. And the help that we received from the community, it lifted our spirits in a way that helped overcome a lot of things. But did we suffer financially? You bet. Did we have lots of costs that we still haven't recouped? You bet. Do I feel PTSD, although not medically diagnosed with such an ailment? You bet; the scars are permanent.”
See our slideshow above for more from Cioffi and more cybersecurity news.
About the Author(s)
You May Also Like