Internet Security Suites fail to block exploits and do little to protect users against exploits, according to a recently released "test" [here] by Secunia, a Danish vulnerability notification firm. I quoted the word "test" as it's very common to see vulnerability companies use close-to-unethical tactics to oversell problems with the AV industry in order to promote their own services [another example here].
Now its Secunia's turn. In their "test" they assume that anti-virus products have poor performance in detecting vulnerability exploits because of their limited focus on traditional AV signatures. So along comes Secunia's Chief Technology Officer (CTO) Thomas Kristensen with the bright idea of testing 12 different Internet Security Suites from McAfee, Norton, Kaspersky, Panda and others against a testbed of exploit files. So far so good, it’s an interesting idea for comparing technologies and I believe it should be performed.
However when testing exploits one very important aspect is that these products don't just rely on traditional signature detection. Yet Secunia's "test methodology" only takes into consideration manually scanning 144 different inactive exploit files. This is very much like saying that you're going to test a car’s ABS breaks by throwing it down a 200 meter cliff. Absurd, sensationalist and misleading at best.
Just to clarify, if you only test 1 part of a product against exploits, which by the way is the part of the product which IS NOT designed to deal with exploits, and leave out of the test the part of the product that DOES deal with exploits and vulnerabilities, there's a very good chance the results will be misleading. Mr. Kristensen, as a Chief Technology Officer, should know this and should be very well aware of the consequences of a faulty methodology. So the question remains, why did he ignore it and just go for the yellow sensationalist approach?
But the absurd doesn't stop with Secunia's flawed testing methodology. Mr. Kristensen concludes that "… major security vendors do not focus on vulnerabilities. Instead, they have a much more traditional approach, which leaves their customers exposed to new malware exploiting vulnerabilities." Well duh, if you only test traditional signatures and neglect the other technologies included in the product which ARE designed to block exploits, what do you expect? Oh, wait, I just saw on their website that Secunia actually sells a vulnerability scanner! Hmmm, I wonder if that has something to do with the flawed conclusions of this test… Internet Security Suites do not rely on signature detection alone since many years ago. Panda's (and other) products integrate behavioral analysis, context-based heuristics, security policies, vulnerability detection, etc. However none of these technologies were tested by Secunia.
Let's just take 1 of the many protection technologies included in Panda Internet Security 2009 which DOES deal with prevention of vulnerability exploitation and see how it behaves against these exploits if tested correctly. I'm talking about Kernel Rules Engine, a security policy technology incorporated in 2004 to all Panda products which effectively prevents zero-day exploits of PDF, DOC, XLS, PPT and many other vulnerable applications. While Secunia's test grants Panda a lowly 1.59% detection rate of the important threats, if they would have tested correctly they would have found out that just with Kernel Rules Engine Panda's product is able to generically and proactively block 56% of the important threats. And this just with KRE technology. But Panda's products also include other technologies such as TruPrevent's Behavioral Analysis, URL Filters and the Vulnerability Detection module which would prevent other exploits if Secunia cared to run their tests with a minimum level of professionalism.
Note to Secunia:
The following exploits (at least), which in your study are marked as "not detected by Panda", are actually detected generically with the correct testing methodology. Hint: have you tried actually "running" the exploits?
SA14896 CVE-2005-0944 PoC.mdb SA20748#1 CVE-2006-3086 PoC.xls SA21061 CVE-2006-3655 POC1.ppt SA21061 CVE-2006-3656 POC2.ppt SA21061 CVE-2006-3660 POC3.ppt SA22127#1 CVE-2006-4694 PoC.ppt SA23540 CVE-2007-0015 PoC.qtl SA23676#2 CVE-2007-0028 Exploit1.xls SA23676#2 CVE-2007-0028 exploit2.xls SA23676#2 CVE-2007-0028 PoC.xls SA23676#3 CVE-2007-0029 PoC.xls SA23676#4 CVE-2007-0030 PoC.xls SA23676#5 CVE-2007-0031 PoC.xls SA24152 CVE-2006-1311 PoC.rtf SA24359#1 CVE-2007-0711 PoC.3gp SA24359#3 CVE-2007-0713 PoC.mov SA24359#4 CVE-2007-0714 PoC.mov SA24359#8 CVE-2007-0718 PoC.qtif SA24359#9 CVE-NOMATCH PoC.jp2 SA24659 CVE-2007-0038 GameOver.ani SA24664 CVE-2007-1735 PoC.wpd SA24725 CVE-2007-1867 GameOver.ani SA24784 CVE-2007-1942 Exploit.bmp SA24784 CVE-2007-1942 PoC.bmp SA24884 CVE-2007-2062 GameOver.cue SA24973 CVE-2007-2194 GameOver.xpm SA25023 CVE-2007-2244 PoC.bmp SA25034 CVE-2007-2366 GameOver.png SA25044 CVE-2007-2365 GameOver.png SA25052 CVE-2007-2363 GameOver.iff SA25089 CVE-2007-2498 PoC.mp4 SA25150#1 CVE-2007-0215 PoC1.xls SA25150#1 CVE-2007-0215 PoC2.xls SA25150#3 CVE-2007-1214 PoC.xls SA25178 CVE-2007-1747 PoC.xls SA25278 CVE-2007-2809 GameOver.torrent SA25426 CVE-2007-2966 PoC.lzh SA25619#1 CVE-2007-0934 PoC.vsd SA25619#2 CVE-2007-0936 GameOver.vsd SA25619#2 CVE-2007-0936 PoC.vsd SA25826 CVE-2007-3375 PoC.lzh SA25952 CVE-2007-6007 PoC1.psp SA25952 CVE-2007-6007 PoC2.psp SA25952 CVE-2007-6007 PoC3.psp SA25988 CVE-2007-1754 PoC.pub SA25995#1 CVE-2007-1756 PoC.xls SA25995#2 CVE-2007-3029 PoC1.xls SA25995#2 CVE-2007-3029 PoC2.xls SA25995#3 CVE-2007-3030 PoC.xlw SA26034#4 CVE-2007-2394 PoC.mov SA26145 CVE-2007-3890 PoC1.xlw SA26145 CVE-2007-3890 PoC2.xlw SA26433 CVE-2007-3037 PoC.wmz SA26619 CVE-2007-4343 Exploit.pal SA26619 CVE-2007-4343 GameOver.pal SA27000 CVE-2007-5279 PoC.bh SA27151 CVE-2007-3899 GameOver.doc SA27151 CVE-2007-3899 PoC.doc SA27270 CVE-2007-5709 GameOver.m3u SA27304#1 CVE-2007-5909 GameOver1.rtf SA27304#1 CVE-2007-5909 GameOver2.rtf SA27304#1 CVE-2007-5909 PoC1.rtf SA27304#2 CVE-2007-6008 PoC1.eml SA27304#2 CVE-2007-6008 PoC2.eml SA27361#4 CVE-2007-2263 PoC.swf SA27849 CVE-2007-6593 GameOver1.123 SA27849 CVE-2007-6593 GameOver2.123 SA27849 CVE-2007-6593 GameOver3.123 SA28034 CVE-2007-0064 PoC1.asf SA28034 CVE-2007-0064 PoC2.asf SA28034 CVE-2007-0064 PoC3.asf SA28034 CVE-2007-0064 PoC4.asf SA28083#2 CVE-2007-0071 PoC.swf SA28092#1 CVE-2007-4706 PoC.mov SA28209#10 CVE-2007-5399 PoCbcc.eml SA28209#10 CVE-2007-5399 _PoC_cc.eml SA28209#10 CVE-2007-5399 PoC_date.eml SA28209#10 CVE-2007-5399 PoC_from.eml SA28209#10 CVE-2007-5399 PoC_imp.eml SA28209#10 CVE-2007-5399 PoC_prio.eml SA28209#10 CVE-2007-5399 PoC_to.eml SA28209#10 CVE-2007-5399 PoC_xmsmail.eml SA28209#11 CVE-2007-5399 PoC.eml SA28209#12 CVE-2007-5399 PoC.eml SA28209#13 CVE-2007-5399 PoC.eml SA28326 CVE-2008-0064 GameOver1.hdr SA28326 CVE-2008-0064 GameOver2.hdr SA28506#1 CVE-2008-0081 Exploit.xls SA28506#1 CVE-2008-0081 PoC.xls SA28506#2 CVE-2008-0111 PoC1.xls SA28506#2 CVE-2008-0111 PoC2.xls SA28506#2 CVE-2008-0111 PoC3.xls SA28506#4 CVE-2008-0114 PoC.xls SA28506#7 CVE-2008-0117 Exploit.xls SA28506#7 CVE-2008-0117 GameOver.xls SA28506#7 CVE-2008-0117 PoC.xls SA28563 CVE-2008-0392 Exploit_CommandName.dsr SA28563 CVE-2008-0392 GameOver_CommandName.dsr SA28765 CVE-2008-0619 PoC.m3u SA28765 CVE-2008-0619 PoC.pls SA28802#1 CVE-2007-5659 GameOver.pdf SA28802#1 CVE-2007-5659 PoC.pdf SA28904#2 CVE-2008-0105 PoC1.wps SA28904#2 CVE-2008-0105 PoC2.wps SA28904#3 CVE-2007-0108 GameOver.wps SA29293#1 CVE-2008-1581 PoC.pct SA29321#2a CVE-2008-0118 PoC.ppt SA29321#2b CVE-2008-0118 GameOver.ppt SA29321#2b CVE-2008-0118 PoC.ppt SA29620 CVE-2008-0069 GameOver.sld SA29650#5 CVE-2008-1017 crgn_PoC.mov SA29704#1 CVE-2008-1083 PoC.emf SA29704#2 CVE-2008-1087 PoC.emf SA29838 CVE-2008-1765 Exploit.bmp SA29838 CVE-2008-1765 GameOver.bmp SA29934 CVE-2008-1942 PoC_ExtGState.pdf SA29934 CVE-2008-1942 PoC_Height.pdf SA29934 CVE-2008-1942 PoC_MediaBox.pdf SA29934 CVE-2008-1942 PoC_Width.pdf SA29941 CVE-2008-1104 Exploit.pdf SA29941 CVE-2008-1104 PoC.pdf SA29972 CVE-2008-2021 PoC.ZOO SA30143#1 CVE-2008-1091 PoC.rtf SA30953 CVE-2008-1435 PoC.search-ms SA30975 CVE-2008-2244 PoC1.doc SA30975 CVE-2008-2244 PoC2.doc SA31336#2 CVE-2008-3018 PoC.pict SA31336#4 CVE-2008-3020 PoC.bmp SA31336#5 CVE-2008-3460 PoC1.wpg SA31336#5 CVE-2008-3460 PoC2.wpg SA31336#5 CVE-2008-3460 PoC3.wpg SA31385 CVE-2008-2245 PoC.emf SA31441 CVE-2008-4434 PoC.torrent SA31454#X CVE-NOMATCH PoC.xls SA31454#2 CVE-2008-3005 Exploit.xls SA31454#2 CVE-2008-3005 PoC.xls SA31675#3 CVE-2008-3013 PoC.gif SA31675#4 CVE-2008-3014 PoC.wmf SA31675#X CVE-NOMATCH PoC.emf SA31675#X CVE-NOMATCH PoC.wmf SA31675#5 CVE-2008-3015 PoC.ppt SA31821#6 CVE-2008-3626 PoC1.mp4 SA31821#6 CVE-2008-3626 PoC2.mp4