lab malware

This week Cylance’s Chad Skipper published an article called “Security Testing Houses: Know the Truth!” that all people interested in security solutions testing should read. There are some serious accusations against some testing houses and vendors (without naming them) such as:

–          “vendors who pay so that their test results will show 100% efficacy”

–          “bribing the testing house to hide the negative results of their tests.”

Even though I have been involved in this industry for more than 17 years, I am not aware of any case like those described above. That being said, I do agree with most of the article. To name a few: outdated testing methodologies, not enough samples being used, having to pay to challenge the test results… that happens. And it has to be fixed, that’s why organizations like AMTSO exist, and the first thing that came to my mind after reading the blog was “we need to have Chad in the next AMTSO meeting”. Guess what, when I asked AMTSO about it they told me he had already registered for the next meeting we’ll have next month in Malaga. Awesome!

Chad ends the article saying “Test for Yourself”. I also agree with this, and in fact it is something that has been happening for a long time. The largest customers we have in different areas (Governments, Telecommunications, Financial, Health, Facilities industries) have selected our EDR solution (Adaptive Defense 360) after several months of intensive and deep testing of different solutions.

The truth is that this kind of “do-it-yourself” testing is only available for big corporations. Small and medium companies lack the resources to do it properly, and that’s why they trust professional testing companies’ results to make decisions. Security Week’s Kevin Townsend wrote an article a few months ago about this topic in this fantastic article: “Inside The Competitive Testing Battlefield of Endpoint Security”.

Out of all the regular tests performed by the biggest testing companies one of the tests I like the most is the Real-World Protection Test performed by AV-Comparatives. In the aggregated February-June 2016 test with 1,868 test cases (PDF), how many vendors obtained 100% accuracy with 0 false positives? None of them. It is clear that Chad cannot be referring to AV-Comparatives when he is talking about vendors that pay to obtain a 100% efficacy.

This is the same AV-Comparatives I talked to last year to test our EDR solution, Adaptive Defense 360, with a number of other similar solutions. Have you seen that test? No, that’s because even though Panda offered to pay for each product included in that test, the other vendors (Cylance was NOT one of them) didn’t want to.

In 3 weeks I will be in Denver to discuss these topics at the 26th Virus Bulletin conference with ESET’s Righard Zwienenberg in our talk “Anti-malware Testing Undercover”.