The folks over at Consumer Reports have published a review of the various antivirus solutions. Now they tackled this from a much different angle for their testing. What these folks did, and managed to seriously piss off the industry in the process, was to use mutations of known viruses. And (insert deity) bless them for doing it. And thanks to Brian Krebs on his blog Security Fix I was able to learn about this story.
Well, the Consumer Reports pissed off the industry…and they wrote a letter.
More than 100 security experts and executives from companies like Microsoft and HP as well as anti-virus vendors F-Secure, Kaspersky, McAfee, Sophos, Symantec and Trend Micro signed their names to a declaration denouncing Consumer Reports’ methods, stating that it is “not necessary and … not useful to write computer viruses to learn how to protect against them.”
Well, here is a link to the letter. Hmmm, ok. So they’re pissed off because the testing managed to demonstrate the failings of the antivirus products? OK, I can see that. I don’t agree with it but, I can see why they’re pissed.
From Krebs piece again,
As I have noted here before, many malware authors are increasingly outpacing the security vendors by automagically updating the genetic makeup of their creations before anti-virus companies have time to ship updates. As a result, we have an industry whose business is predicated on 10 percent to 20 percent of its customers being successfully attacked before it can even begin to respond, according to some estimates.
Now, these numbers are only going to grow.
I can see why they’d be pissed. Being shown to be sub par would do that to most people. The arguement that you should only test with known viruses seems to me to be inherently flawed. I would want to know that my antivirus product can respond to and if possible protect ungainst the unknown threats.
And finally,
The most innovative idea I’ve seen so far came in a presentation from Paul Vixie and David Dagon at the DefCon hacker conference in Las Vegas this year. Vixie and Dagon proposed creating a massive malware repository to which all of the anti-virus vendors would automatically submit new samples.
Well, this has already been done in fact. The guys at Offensive Computing also released this idea at Defcon, and in fact it is already up and running. Check it out.
[tags]Antivirus, Offensive Computing, Consumer Reports, Defcon[/tags]
“So they’re pissed off because the testing managed to demonstrate the failings of the antivirus products?”
they aren’t pissed off because of that… they’re pissed off because consumer reports contracted to have viruses made for the test…
it’s not necessary to create new viruses in order to test how well an anti-virus product deals with new viruses – you only need to use a somewhat out of date (by a few months) anti-virus product and the viruses that were discovered since that last update occurred… this is called retrospective testing and it already displays how bad anti-virus products are against new viruses (some have a detection rates that barely make it into the double digits)…
the controversy is over the fact that consumer reports is being part of the problem rather than part of the solution when they make viruses or have others make viruses for them…
Kurt, you are right about the AV objection, but there’s a problem with your (and the AV companies’) argument of back-tracking AV products for accurate testing.
It’s this: Retrospective testing has an inherent sampling error. You can’t really test against a random sample of all viruses created in the last few months, you can only test against a sample of all viruses *discovered* in the past few months. Therefore, viruses that are undiscovered (because, among other things, AV products aren’t good enough) can’t be checked. Retrospective testing, therefore, can give a rough relative comparison of products, but cannot establish an absolute value for how good they are.
The Consumer Reports test was a controlled experiment, so it didn’t suffer from this sampling problem.
However, the AV companies *still* have a good point regarding ethics. At Black Hat Europe this year I had dinner with an AV researcher from one of the prominent companies, who told me that they had been talking to an external researcher about theories on developing a new kind of malware, but when he actually went ahead and wrote a proof of concept, they had to cut all contact with him.
That makes sense. Like a lot of the security world, AV vendors have an inherent perverted incentive in that they are trying to reduce the number and severity of viruses but their value-add only increases when these things get worse. Therefore, to protect their integrity (or at least the perception of it), they have to take a hard line against all virus creation. The creation of a virus, any virus, even in a lab, has to be harshly condemned. There’s a perceived slippery slope from AV vendor to extortion operation if they soften that line.
So, as it happens, Consumer Reports and ISE are right. Their technique has definite advantages over restrospective testing. But, on the other hand, the AV companies are right too. They have to condemn that technique, they cannot practice it themselves, and there is some risk associated with other companies practicing it (the size of that risk I don’t know).
@ken
“It’s this: Retrospective testing has an inherent sampling error. You can’t really test against a random sample of all viruses created in the last few months, you can only test against a sample of all viruses *discovered* in the past few months.”
ken, it goes beyond that… no only is there no statistically representative sample to be had in the wild, in the context of virus detection there is no such thing as a statistically representative sample (that’s why testers generally try hard to use as much of the entire population or some significant subset thereof as they can)… without the possibility of a statistically representative sample the selection bias argument falls apart…
“Therefore, viruses that are undiscovered (because, among other things, AV products aren’t good enough) can’t be checked.”
viruses get discovered eventually so if they don’t make it into this quarter’s retrospective test then they’ll make it into next quarter’s or the quarter after that…
“Retrospective testing, therefore, can give a rough relative comparison of products, but cannot establish an absolute value for how good they are.”
nothing can establish an absolute value for how good they are… everything is an approximation, even if we did have statistically representative samples…
“The Consumer Reports test was a controlled experiment, so it didn’t suffer from this sampling problem.”
no, it suffered from worse sampling problems… first and foremost that the viruses it used were not from the real world and therefore a test using them can’t represent real world performance…
second, creating 5500 new viruses is an herculean task even if they were just variants of existing viruses – the probability is high that they were generated algorithmically, which is not how variants are created in the wild… therefore they are even more unrepresentative of real world viruses…
third, verifying all 5500 new viruses output were really viruses (just because you start with a virus doesn’t mean your output is still viral) is also a herculean task and there’s no word on how or even if they did this… without verification for each and every virus sample their testbed cannot be trusted and neither can their results…
best case scenario for their test is that it measures av performance against lab created viruses rather than viruses from the real world… worst case scenario is that it measures av performance against garbage samples…
@kurt
“viruses get discovered eventually…”
I call this the talking tortoise problem.
From Terry Pratchett’s ‘Small Gods’:
Tortoise: “How many talking tortoises have you met?”
Brutha: “I don’t know.”
Tortoise: “What d’you mean, you don’t know?”
Brutha: “Well, they might all talk. They just might not say anything when I’m there.”
It’s a general security problem. It’s hard for a company to be really certain its network security isn’t compromised; perhaps the attacker was more sophisticated than the network’s detection techniques. Similarly, there could be a good number of viruses that never get discovered because they are sufficiently well-written and have small target populations.
“first and foremost that the viruses it used were not from the real world and therefore a test using them can’t represent real world performance…”
It’s not a definitive comparison, because a real-world malicious virus writer could easily be either more or less sophisticated than ISR in generating viruses, but nonetheless was a perfectly valid test. There is no reason that a virus writer *couldn’t* have used an equivalent algorithm.
“the probability is high that they were generated algorithmically, which is not how variants are created in the wild…”
But they could be. If anti-virus products can’t detect new viruses that are algorithmic derivatives of known viruses, then this is extremely worrisome.
“verifying all 5500 new viruses output were really viruses (just because you start with a virus doesn’t mean your output is still viral) is also a herculean task and there’s no word on how or even if they did this…”
We don’t know for sure, but ISR is a reputable company with some very notable security researchers working for it. If Avi Rubin says he has an algorithm to morph viruses without destroying their viral characteristics, I believe him.
@ken
“Similarly, there could be a good number of viruses that never get discovered because they are sufficiently well-written and have small target populations.”
ok, fair enough, i was overgeneralizing when i said viruses get discovered eventually… viruses that pose any sort of real problem for the computer user population get discovered eventually… if a virus affects so few computers that it never crosses paths with a virus analyst, or someone who notices it’s effects on their computer, or someone who is able to determine it’s a virus because they employ generic technologies, or someone who just finds it suspicious enough to submit it for good reason or just because it coincidentally happens to be there while they’re having unrelated computer problems then it’s affecting a very small population indeed and doesn’t represent a tangible problem for computer users as a whole…
“It’s not a definitive comparison, because a real-world malicious virus writer could easily be either more or less sophisticated than ISR in generating viruses, but nonetheless was a perfectly valid test. There is no reason that a virus writer *couldn’t* have used an equivalent algorithm.”
there is no reason they couldn’t have gotten assistance from little green men, either… there are many techniques they *could* use, many more than are testable, many more than an anti-virus product can reasonably be made to account for, so singling out one and testing it instead of sticking to what what they actually use is arbitrary and not particularly valid…
“But they could be. If anti-virus products can’t detect new viruses that are algorithmic derivatives of known viruses, then this is extremely worrisome.”
there are a countably infinite number of derivation algorithms, an anti-virus that tested against them all would not halt… perhaps we can keep our expectations about what an anti-virus should be able to do in the realm of the computationally feasible… if the derivation algorithm is known (ie. for a polymorph) then it’s reasonable to expect anti-virus products to detect all the derivatives, otherwise you might as well be criticising them for simply not being able to detect all possible viruses…
“We don’t know for sure, but ISR is a reputable company with some very notable security researchers working for it.”
security researchers are not virus researchers… over and over again people underestimate the level of specialization present (and required) in the anti-virus field…
“If Avi Rubin says he has an algorithm to morph viruses without destroying their viral characteristics, I believe him.”
and if he says that then i’ll call him on his false authority syndrome and is failure to account for the impact decidability plays in such things…
Gents,
There are few things that I enjoy more than an intelligent public discourse. I whole heartedly thank you for sharing your points of view on this subject. It has been a true pleasure following this thread today.
I’ve spent much of the day on the sofa reading a copy of the Cliff Stoll’s “The Cuckoo’s Egg” and drinking way too much coffee. Basically, thanks for brightening my Ernesto imposed day in the house.
cheers!