Indie Security Product Testing Is Really Hard — and More Important than Ever

Lorna Garey

Customers are doing more of their own purchasing research, so it’s likely you’ve had a call asking about lofty promises by security vendors. A little hyperbole is to be expected. This is a crowded field — the 2017 version of Gartner’s Magic Quadrant for endpoint protection alone includes 22 providers. Customers are rightly worried about ransomware and other threats, and are looking for a magic bullet. How do you as trusted adviser vet vendor claims?

“I talk to a lot of partners who say, ‘Oh, we tested it ourselves,'” Dan Schiappa, senior vice president and general manager of Sophos’ end-user and network security groups, told me at the company’s recent partner summit. When asked how: “We went to VirusTotal and downloaded a hundred pieces of malware.”

The problem with partners simply throwing a few hundred known exploits at a firewall or AV engine is twofold. First, you may end up with a false sense of how well the product will perform in the real world, and not just because of zero-day attacks. These tests often lack the proper context, including where the malware came from.

“Did it come from a browser? Did someone click on it? Did someone download it?” said Schiappa. “If you just throw it in a folder and run it, you’ll lose a lot of that context … it’s a terrible way to test real-world scenarios.”

And, next-gen products that depend on behavioral heuristics and post-execution detection could underperform. Schiappa points out that simply asking, “Did [product] catch an executable before it ran?” isn’t going to evaluate the efficacy of software from companies such as CrowdStrike, Cylance or SentinelOne, whose primary value proposition is behavior-based detection. That is, the systems watch an executable run, evaluate its actions and then decide whether to block it. As security vendors use more machine learning to spot malware trying to exploit something, rather than just shutting down known attacks or carpet bombing all suspect processes, the need for sophisticated, independent testing gets more urgent.

I spent several years as a reviews editor, so I get how complex any product evaluation is. Vendors will argue endlessly over every tiny detail of a testbed setup — as they should, because the stakes are high. Few partners, or even master agents or distributors, are equipped or, frankly, willing to spend the money and time to do in-depth testing of technology, especially a tech as complex and fast-changing as security. Yet customers depend on you for informed advice.

What about depending on tests the vendor did internally? That’s sort of like a take-home midterm exam, if you let the student also write the questions. Instead, when evaluating a new security provider, ask whether it has results from reputable third-party testers such as MRG Effitas, NSS Labs, AV-Test and SE Labs.

“MRG doesn’t have a standard published test, but man, when they test your stuff, they go to the dark web, they do some real testing,” said Schiappa. “We hired them to do the first Intercept X test and they’re like ‘We’re cashing your check and whatever comes out, comes out. If it comes out terrible, we’re going to …

… publish a test that you paid for that says your product stinks.’ We’re like, ‘Bring it on.'”

That review involved MRG actually purchasing zero-day exploits. This level of testing is incredibly expensive to perform, but results for many providers are published and available to download for free. Partners should have their their technical teams watch for comparative results, and ask your masters and distributors what they’re taking away from independent reviews.

The more complex security gets, the more difficult it is to do any sort of apples-to-apples comparison, but that doesn’t mean take vendor claims at face value. Demos are tuned to deliver the results the company doing the demo wants to show. If a vendor never submits its products to a third-party testing firm or actively resists these evals, ask why.

NSS just published a useful guide to DDoS protection services, and AV-Test just put 17 free and paid protection tools to the test to see how well they clean up and recover a Windows system after a successful malware attack. SE Labs has test reports grouped for home, SMB and enterprise products, including a review of email security suites from Proofpoint, Mimecast, Forcepoint and Microsoft. And, testing organizations’ blogs are great resources for more than just reviews.

Now, you do still need to put products through their paces in customer sites. NSS found that real-world throughput of some next-gen firewalls could be significantly less than advertised and recommends running short-listed systems in the actual environment. But when making that short list for customers, do your due diligence.

More news from around the channel.

… an 11 percent rise in distributor revenues, 140,000 new deals being registered per quarter and $43 billion in business driven by global channels. Byrne reiterated Dell EMC’s commitment to a new deal reg team under vice president of global business operations, channel, Stephanie Mims, saying of poaching, “We absolutely have a zero tolerance policy — ask around.” Vice president of global channel programs Kimberly Deleon announced a new tool to help partners track training, competencies, and service and overall revenue as well as the addition of VMware VXrail to MDI eligibility.

Byrne, who likes to call services “a true pot of gold to make your business stickier,” left no doubt that Dell EMC wants its partners to attach support options to every box and add storage to every deal.

“Not attaching services jeopardizes your program status,” said Erica Lambert, vice president, global channel services, who previously discussed a lucrative rebate for partners who resell Dell EMC services — up to 1 percent of overall revenue in the partner program, not of the services themselves.

Dell EMC also unveiled this week a new IoT strategy.