The Worldwide Malware Signature Counter Lives On

At the bottom of the Triumfant home page is the Worldwide Malware Signature Counter, a fixture on the site since May of 2009.  The Counter was designed, according to the associated blog post marking its debut, “to graphically reinforce what many in the IT security industry believe is a growing problem that is being largely ignored – that the reliance on signatures to protect endpoints and servers against malicious attack is simply unsustainable”.  My only regret is that I never found a way to add the hard clunking sound from the timer on “24” to add emphasis.

I periodically check the Counter against reported malware counts to ensure that it is an accurate and fair representation of the signature story.  Truthfully, the Counter was designed to err on the side of understatement to avoid the impression of FUD or sensationalism, so I normally have to correct it up instead of down. Yes, IT security folks, there are actually marketing people with restraint.  Go figure.

Last week I updated the Counter to track to the signature counts reported by Symantec at the close of 2011.   Doing so led to a time of reflection on the genesis and objective of the Counter, and the changes in the threat landscape between then and now.

When Triumfant introduced the Counter three years ago, the world was still coming to terms with the evolution of malicious attacks and the hard realization that signature based protections could no longer be their primary shield. I would hope that there are very few serious members of the IT security community who need further convincing today.

Ironically, in the past three years the large vendors that owe their market presence largely on selling AV software have shifted their messaging.  Most dropped signature counts from their annual threat reports in spite of such counts being a featured staple in years past.  I noted in one blog post that one such vendor dropped any mention of the word “signature” completely.  In an interesting twist, some of these vendors now use the large malware sample numbers to sell other products and solutions in their portfolio.  The flood of annual reports that are the precursor for the RSA Conference scream numbers such as 75 million and 250 million for malware samples.  You have to feel for signature software: it made these vendors market leaders and it is now being dismissively kicked to the curb. Think Sunset Boulevard for security software.

Meanwhile, the battle to protect sensitive data and intellectual property continues to rapidly evolve.  The first malware sprung to life when sensitive information moved from corporate systems to the first personal computers.  Those early attacks now seem laughable against the volume and sophistication of the threats we face today, and things will only get more complicated when you consider the flood of mobile devices and BYOD machines that will soon be accessing corporate systems.  Furthermore, the adversary has changed from basement hackers to well organized, well funded, and highly motivated groups driven by monetary gain or political motives.  The sum total of this evolution creates a gap between signature based protections and the current reality that grows faster than a simple signature counter can capture.

The counter was a great visual to help people grasp the shift in the IT security world and helped bring attention to Triumfant’s ability to detect malware without signatures.  The counter often provoked people to ask if we were a replacement for signature based protections, and we always said no.  Signature based protections are a logical brick in the wall around IT assets, but they are just a brick, not the entire wall.  I should add that the Counter now serves as a symbol for all solutions that based their detection capability on some form of prior knowledge, not just AV.

My next thoughts went to the Counter itself and its continued existence on the Triumfant site.   After some consideration, I decided to keep it around because while the thinking of the IT security world has evolved there are still plenty of other business people outside of security that are still coming to terms with the concept.  Truth be told, I have an emotional fondness for the Counter and it is still a place for people to discover Triumfant and the uniqueness of our approach.

The Triumfant Worldwide Malware Signature Counter will live on.  Maybe I will finally add that sound effect.  Clunk…Clunk…Clunk…

Malware Counts – Shock, Yawn, or a Useful Reminder of Today’s IT Security Reality?

5 million new threats in Q3 2011!

This was one of the hot lead statistics from the Q3 2011 PandaLabs Report released at the beginning of this month.  Instead of pondering that number, I found myself pondering how the market reacts to that number as we move toward the end of 2011.  Shock? Knowing nod of the head? Yawn?

When I joined Triumfant in November of 2008, the world had entered that year with less than 1 million signatures according to Symantec’s Internet Threat Report series.  Those were simpler times.   In 2009, the number of new signatures exceeded the number of total signatures reported in 2008.  The statistics were sobering and captured the attention of the market as organizations began to internalize that the malware game had changed dramatically across multiple dimensions – volume, velocity, and sophistication.  Threats were also shifting from broad, opportunistic blunt instruments to targeted attacks, some written for a single target.  The term Advanced Persistent Threat moved from the MIC into the broader consciousness.

As we close out 2011, my impression is that the 5 million number by PandaLabs generates very little response and such numbers numbers no longer resonate.  Maybe these numbers have gotten large enough where they loose a sense of connection.  Maybe the numbers have been overused to the point that they no longer have any impact (the marketing bashers so prevalent in IT security will quickly form a line here).  Or maybe most right thinking people have seen the weight of evidence and have accepted the new threat reality.  Regardless, they appear to no longer capture the imagination.

What the numbers continue to say is that the world of IT security has changed dramatically and continues to rapidly evolve.  The numbers dictate that organizations need to be open-minded to new solutions and must stay nimble to keep up with this evolution.  For example, I think organizations now academically understand that the notion of the 100% shield is obsolete, but far too many have to emotionally accept that reality and take action accordingly.

The numbers also remind us of the relentless nature of the adversary, who never stop trying to broaden the always-present gap between offense and defense.  The numbers indicate that your defenses have plenty to do, so make sure that they are stood up and properly configured on every machine so as not to give the bad guys a beachhead.  There is no 100% shield, but you should ensure that your shields stop what they can.

The numbers reinforce the fact that you should expect to be breached.  Accept that there will be attacks written specifically to evade your shields and get to your sensitive data and IP.  Think beyond shields and have rapid detection and response software in place for those times when you are breached.

In the end, the only real number that is truly significant is how many breaches that go undetected and result in loss of revenue, loss of customer confidence, or loss of intellectual property.  All you have to do is read this very frank assessment of the cost of the RSA breach to know that the number “1” may be far more impactful than 5 million.

The Worldwide Malware Signature Counter – A One Year Report Card

About a year ago we had the idea of the Worldwide Malware Signature Counter as a graphical representation of how the reliance on previous knowledge of attacks for detection was no longer a serviceable approach for protecting endpoint machines.  Much of the actual data used to build the math behind the filter (yes, it has thoughtfully constructed, sound mathematical principles behind it) was taken from the Symantec Internet Security Threat Report (ISTR).  Since Symantec just released the annual update to that report, it seemed an appropriate time to look back at the counter and see how well our analysis held up a year later.

All things considered, the counter was remarkably accurate.  Charting both the year-over-year and cumulative signature counts through 2008, we concluded that new signatures were growing at 60% of the cumulative rate on an annual basis.  This proved to be a bit aggressive, as Symantec’s actual numbers showed 2009 growth to be 51% of the cumulative number, or just under 2.9M new signatures.   But because we conceived the counter to be instructional and not hyperbole, we built the calculations on the conservative side and the counter in fact lagged just slightly behind the actual numbers reported by Symantec throughout the year and eventually in the ISTR.

When I first did the math on the numbers from the ISTR in 2009, I was struck on how the signature numbers broke down as a practical drag on the resources of the AV companies.   Of course a 51% increase year-over-year only exacerbates the problem.  Using the numbers from the recent ISTR, that burden translates to 241,316 signatures per month, roughly 7,934 signatures per day, 5.5 signatures per minute, and ultimately one signature every 12 seconds.  It is a model that is simply not sustainable, and by every indication, it will only get worse.

The bigger question after a year is “so what?”.  Well, the language from the AV vendors has certainly changed.  In fact, the following is a direct quote from the Symantec document:

Signature-based detection is lagging behind the creation of malicious threats—something which makes newer antivirus technologies and techniques, such as behavioral-based detection, increasingly important. …. This trend suggests that security technologies that rely on signatures should be complemented with additional heuristics, behavioral monitoring techniques, and reputation-based security. (page 48, Symantec Global Internet Threat Report – Trends for 2009,  Volume XV, Published April 2010)

During his keynote at this year’s RSA, Symantec CEO’s Enrique Salem was quoted as saying “Traditional signature-based approaches to security are not keeping up.”  Of course, such admissions are directly colored by the alternative technologies the AV companies recently introduced to the market after ignoring calls from the rest of the industry for alternative detection methods.  But at least they have stepped away from their defense of signature based AV in the face of all evidence to the contrary.  I am not claiming they were driven to such mea culpa’s by our signature counter, but I do think we helped point out the issue.

Unfortunately, while organizations have also come to terms with the limitations of signature based AV, many are adopting the alternatives provided by the AV vendors instead of looking to more promising technologies.  Symantec brought Quorum to the market, so reputation based security is their answer.  McAfee bought a whitelisting technology so – surprise! – whitelisting is their answer.  I guess I was hoping that organizations would see the past the entrenched vendors for alternatives given that these vendors were so slow to come to terms with the signature problem, but factors such as risk avoidance have suppressed some innovative alternatives from getting play.

Meanwhile, the counter continues to increment, and recently passed 7 million signatures on pace to add over 4 million signatures for 2010.  I was recently asked if we planned on retiring the counter given the shift in sentiment toward signature based AV.  We still see enough executives and security people that don’t yet understand the problem, so the counter will live on to help us make the point.

So in regards to a grade, how about an gold star for creativity, an “A” for the math, and an “I” (incomplete) for changing the world.

Lessons Learned from the McAfee DAT Fail

When I worked at Information Builders, founder and CEO Gerry Cohen would pass by my office in the evening and stop in and ask simply: what did we learn today?  While simple, that question forced you to take a look at the day and see what lessons could be learned from the experiences.

Last week, the security market had quite the experience as McAfee inadvertently disabled thousands of PC’s with an update to their signature files that knocked out a file critical to the XP operating system.  Now a week later, it is prudent to ask: what did we learn?

This was inevitable. The velocity and volume at which malicious attacks have been growing simply overwhelmed the process of writing and updating signatures to keep pace.  The signature counts are now over 7 million, with half of those signatures coming in 2009.  I have been shouting this from the mountaintop for over a year now – the process is not sustainable.  That is why we started the Worldwide Malware Counter to provide a visual representation of the problem.  Those who have chosen to look the other way can no longer ignore the evidence as this problem interrupted business, infrastructure, and healthcare.

This is an industry problem, not a McAfee problem. I don’t blame McAfee, I think the law of averages simply kicked in and they were the unlucky target.  The other vendors will likely jump on McAfee, but they in fact owe them a debt because deep down they all know it could have been their number that came up first.  Trading McAfee AV for some other AV software is not the answer.

This problem is not going away. Now the AV vendors will be under increasing scrutiny, and the relentless burden of writing signatures will only worsen.  They are being strangled on both ends, and similar problems are sure to follow.  Yes, they will all tighten their QA processes, but the forces at work will only grow stronger and the process will buckle again.  And by the way, have you ever stopped to think of the load on the network and the endpoints to continuously deliver and process ever larger DAT files? Or the performance hit of having to check 7M signatures constantly?

Malware writers will leverage the “Tony Stark Effect”. In Ironman, Tony Stark cannot have the shrapnel removed from his chest because it is too close to his heart.  In the same way, malware writers were already pushing attacks closer to the critical files at the heart of the operating system.  This pushed McAfee to extend some generic signatures too close to one of these files and it backfired.  Now the AV vendors will be skittish about signatures that get close to other files like SVCHOST, which is a gap that the malware writers will exploit.

The LINUX, Mac, and anti-AV forces will be in full voice. This event will feed the fires of those who either tout their OS as a malware free environment or those on the fringe that advocate running without AV software.  While we can detect the attacks that evade AV software, we never advocate going without AV and believe it has a place in the defensive strategy for the endpoint.  But it does need help, as antivirus detection rates demonstrate the holes.  I am also a believer that if everyone shifted to LINUX or the Mac, then the malware writers would follow.  Remember the answer to the famous question when Willie Sutton, the prolific bank robber, was asked “why do you rob banks?” – his response “because that is where they keep the money”.   If business moves to these OS’s, the malware will follow.

While I don’t blame McAfee, they really dropped the ball in responding to the crisis. McAfee is a partner and I normally find them pretty savvy with their marketing and their handling of the media.  But they flat out crashed and burned in handling this problem, starting with initial denials and following with near radio silence over the first 48 hours.  While this could have happened to any AV vendor, I do have to call out McAfee for the weak response.

It will be interesting to watch as this problem continues to play out.

Antivirus Detection Rates – Undetected Attacks Are Still Attacks

I came across an article in The Business Times this morning that contained a quote that caught my eye.  The article was called “Singapore a growing platform for cyber attacks on region” which talked about the growing number of cyber attacks originating in Singapore.  In the article there was a definition attributed to Symantec:

“By Symantec’s definition, an attack denotes any malicious activity carried out over a network that has been detected by a firewall, intrusion detection or prevention systems.”

Obviously, the word that stuck out in this definition was “detected”.  Why?  Because I have news for you – malicious activity that goes undetected is also an attack.  In fact, I would say that undetected attacks would be placed in a higher tier of the definition, because Rule One of criminal behavior is Don’t Get Caught.  Attacks that would fall under the characterization of an Advanced Persistent Threat are engineered to evade detection and are very much an attack.

(This reminds me of one of my favorite movie scenes.  In Stripes, Harold Ramis and Bill Murray are sitting in the Army recruitment office and the recruiter asks them if they have “ever been convicted of a felony?”.  Bill Murray’s response: “Convicted?”.)

In fairness to Symantec, I am not sure if this quote from the article was paraphrased or misquoted, and I am not out to pick on Symantec.  What I do want to point out is a huge flaw in how in the industry measures malicious activity.  Let me explain.

Both AV software vendors and internal security groups often report on what was detected.  Makes sense, right?  If you could count undetected attacks they would instantly be now detected.  But according to the Symantec Internet Security Threat Report: “Symantec created 2,895,802 new malicious code signatures in 2009, a 71 percent increase over 2008”.  It therefore makes sense that the number of detected attacks would go up proportionately with the number of identified signatures.  An organization could be doing a worse job year over year detecting attacks but their raw volume of detected attacks would still go up, giving a perception of success.

Executives look at the bulk score and are mollified that the organization is protected.  But if the number of attacks grew by 71%, the number of attacks detected by the organization better track to that same 71% or the organization is losing ground.  If you think it through, that 71% may be deceiving because what Symantec and the other AV vendors don’t tell you is how long your organization was exposed between when the attack actually was first introduced and when they finally detected it and wrote a signature. It could have been six hours, but it could have also been six months.

In short, gauging success from bulk detection numbers is a quick way to obfuscate the real risk to any organization.  But if you are selling a shield that has known flaws, it is a great way to use the steadily growing malware volume to present either software or organizational effectiveness in a successful light.

Because Triumfant uses change detection to identify malicious attacks, we have always been open about our ability to see attacks that are resident prior to our installation.  That being said, we inevitably see anomalies that are artifacts of attacks that have passed through the organization’s shields soon after we are installed.  Once installed, we can readily detect what does make it through the organization’s shields or attacks being done by maliciously intended insiders.  It is eye opening to the organization just how many attacks have and are getting through.

Don’t let yourself be lulled to sleep by bulk detection rate numbers.  A lot of attacks are getting through, so counting detected attacks is potentially a false gauge of success.

RSA Shocker (Not): Symantec Admits Traditional Signature Based Tools are “Not Keeping Up”

“Traditional signature-based approaches to security are not keeping up.  What we’ve had to do is come up with a new approach. The idea is it has to be able to deal with attacks that we’ve never seen.”

Words from some maverick security company?  Hardly.  These are the words of Symantec CEO’s Enrique Salem from his Tuesday RSA Conference keynote.  And he is about to tell the assembled RSA crowd that Symantec’s prevalence technology is the answer to the vexing problem of rapidly emerging and constantly evolving threats.  I can’t fault his message – his company paid handsomely for that keynote spot so he can proclaim his new technology as the 2010 silver bullet.  But in my opinion, Salem and Symantec’s new found honesty regarding the efficacy of AV is late, awkward, and does little to provide real leadership to the market.  The industry leaders should not feel all self congratulatory in finally admitting a problem they have ignored for far too long.

I had a similar experience listening to a CEO in denial say something equally late and awkward before at the 1999 Sapphire Conference (SAP user conference) in Philadelphia.  SAP was acting like the World Wide Web was simply not happening all around them because it was so foreign to their core technology.  In his keynote, then SAP CEO (or COB) Hasso Plattner grudgingly referenced the internet as an “emerging technology” but was still ultimately dismissive.  I remember thinking “sir, I think the internet has already emerged and no dismissal from you can change that fact”.  Actually, I think my exact thought was “Emerging? Dude, internet done emerged!”

What confounds me is that companies still somehow either believe or want to believe that companies like Symantec can solve this problem.   Not one person in a company or government agency that fights what has been called the advanced persistent threat tells me that they believe that prevalence technology is a viable solution for what Salem calls “the attacks that we’ve never seen”.  Same with whitelisting, which is the proposed answer for companies like McAfee and Lumension.

(As a complete aside, one vendor actually touted “intelligent whitelisting” at RSA, I assume implying that somehow intelligence had been left out of previous whitelisting attempts.  I could see people everywhere saying “AH! I was supposed to be intelligent about whitelisting!  Now I get it.”)

I think it is disingenuous for companies that have been at the front of the A/V wave to feign public shock that signatures are no longer viable when their own customers have been pleading with them for years and years to step up and make the jump to newer technology.   We of course have been pointing out the problem for some time, with our Worldwide Malware Signature Counter providing a visual for the problem.  I also think it odd that a company like Symantec would post a reports showing that 100% of the enterprises they polled for a recent study had been attacked (see an interesting view of FUD surveys in John Pescatore’s blog here).  The math is simple: if Symantec represents 40% market share and 100% were attacked, aren’t they saying that they failed to protect 40% of the enterprises represented in the survey? Seriously, am I missing something here?

Let me be clear.  The answers to the problem Salem raises do exist.  You and your organization are simply going to have to look outside of your AV suite vendor to find it.

Grading the Worldwide Malware Signature Counter

One of the fun things we did at Triumfant in 2009 was introduce the Worldwide Malware Signature Counter as a visual representation of the number of signatures being produced to address the growing number of malicious attacks.  The counter is visible from our home page and automatically increments to keep pace with the reported growth of signatures as reported by the antivirus companies. 

While the counter was meant to be illustrative of the problem, we did perform due diligence in an attempt to make it as accurate as possible given the historical signature counts at our disposal, going so far as to bring in an MIT graduate to help create the formula.  The counter is designed to take into account that a plot of historical data shows a geometric progression as the rate of growth accelerates.  I am also told it takes into account factors such as general humidity, global warming, and the falling net worth of Tiger Woods, but I digress.

 The bottom line is that while yes, it was a marketing construct to draw attention to the ability of our product to detect malware without the need for signatures; we made a considered attempt at being accurate.  And with the start of the New Year, it is an obvious time to see how we performed. 

On December 31, Symantec showed 5,853,273 signatures, and our counter was at roughly 6,050,000.   So we were pretty close in our predictions, and charting the historical numbers explains why the counter was a bit high by year end. The actual growth rate of new signatures is below 100% (94%) for the year, in contrast to a 165% growth rate for 2008.  While this was slower than the previous year, there were still 3.2 million new signatures in 2009.

We will keep the counter running in 2010 and I made the proper adjustment to start the year at the 12/31/09 Symantec count because we want the counter to be fair and accurate.  Using the 2009 rates to extrapolate for 2010, we are looking at over 6 million new signatures and nearly 12 million total signatures by year end.  When you feed that data into the antivirus detection rate studies such as the one recently posted by Cisco, and the Signature Counter remains effective in placing the problem into perspective. 

The bottom line is that the combined weight of the growing threats and the challenges with A/V detection rates leaves a gap in endpoint protection that cannot be ignored.  There has been a lot of hype around prevalence data (Symantec Quorum) and whitelisting, but my discussions with the industry analysts all indicate that organizations are quickly finding that these technologies do not close that gap.  That is where Triumfant enters the picture, as we can detect the attacks that evade other protections.  And not only can we see the attacks, we can create a situational remediation to stop the attacks and address all of its collateral damage in five minutes or less.  Maybe you should have a look.