The Readers Speak! – Top 10 Posts for 2010

The Triumfant blog has been up and running for two years now and I am always flattered that anyone would take time from their day to read a post.  As we end the year, I thought I would post a list of the top 10 posts for the year, as determined by the number of views.

Advanced Persistent Threat: Solution – No, Effective Detection – Yes

This post is about how Triumfant uses its unique approach – change detection and contextual analysis to see the attacks characterized by the Advanced Persistent Threat.

Antivirus Detection Rates – Undetected Attacks Are Still Attacks

This is one of my favorites and addresses a critical concept – the reporting from your current defenses will obviously not tell you what attacks are getting through.  The see no evil approach does not mean that you are not getting attacked.

Antivirus Detection Rates – It is Clear You Need a Plan B

There are any number of reports and studies that clearly show that AV detection rates are bad and getting worse.  So what are organizations doing about that fact (if anything)?

Tired of the Term Advanced Persistent Threat – How About Cold Harsh Reality?

This post followed a spirited exchange in the blogosphere and twitterverse about the term Advanced Persistent Threat and whether APT is more about the adversary or the attacks.  This post was my entry into the conversation.

Intel Acquires McAfee, IBM Acquires BigFix – What Does It Mean to You?

2010 was a tumultuous year for the security industry and these two acquisitions are at the front of that tumult.  This post is my take on what these acquisitions mean and what happens to smaller companies when subsumed by larger ones.

Antivirus Detection Rates Study Shows the Real Exposure to Your Organization

Another post that follows yet another study on AV detection rates.  The goal was simple: there are lots of these reports and studies published, but very little pragmatic assessment about what that means in regards to risks for the organization.

Triumfant and Operation Aurora – Detecting the Advanced Persistent Threat

Remember back before Stuxnet?  When Operation Aurora hit, I got lots of inquiries of whether Triumfant would have detected the attack.  Because none of our customers were hit by the attack, our CTO Dave hooks broke down all of the data on Aurora and created this in depth case study.

Oh the Animals You Will See at the RSA Zoo (Conference)

This was written as a bit of a joke but reflects my many years of exhibiting at the RSA show.  It was one of those posts that sounded good when written, but gives pause before you post because of the fear that it will be funny to no one else but you.  I was pleased with the spirit in which it was received.

Security Configuration Management – Plugging the Holes in Your Endpoint Security

This post dug into the concepts of security configuration management in depth and provided a pragmatic conversation about the approach of Triumfant that includes our normative baseline and our automated remediation capabilities.

The Yin and Yang of Triumfant – Agent Based Precision With Network Level Analytical Context

This very recent post grabbed a significant quantity of views faster than just about any post.  The post discusses the ability of Triumfant to deliver agent level precision with the power and context of server based analysis.

So there you have the top ten as voted by you, the readers.  Thank you for reading and the feedback you provide.  Have a great holiday and a Happy New Year.

Security Configuration Management – Don’t Fall for the Old Saw of Patch Management

Yesterday I attended a customer event for one of the larger IT security firms and one of our partners.  During one of the sessions, another partner gave a presentation on security configuration management that nearly drove me to a Kanye West “grab the microphone” moment.

The partner in question regaled the attendees with a story of automated configuration management that involved long intrusive scans, followed by analysis to identify problems, followed by the issuance of what are essentially patches to correct non-compliant machines.  This process seemed horribly cumbersome and certainly did not meet my definition of automated.  And the remediations for the detected problems were pulled from a list of pre-written remediations in a one-size-fits-all approach.

Worse of all, the process happened infrequently – monthly at best, perhaps quarterly or twice yearly.

The saw blades are a visual of configuration drift over time. The length of the saw teeth represents the amount of drift and the size of the gap between them represents the time between corrections.  If the height of each saw tooth indicates how much configuration drift you will experience with large gaps between configuration corrections, which do you think represents the most secure environment?  The bigger the teeth, the higher the organizational risk.  You want the hacksaw blade.

My negative reaction comes from knowing there is a better way to deliver security configuration management Triumfant will continuously scan for changes to endpoint machines and detect when the machine is in a non-compliant state.  Triumfant’s analytics will evaluate the changes to the machine, create a remediation for that problem, and return the machine to compliance.  Remediations are created on the fly to address specific detected problems on each machine.  The remediations are surgical, contextual, and situational.  The remediation is delivered to the machine and executed by the agent.  All of this can be set to a one-touch confirm from the administrative console or fully automated.  And we will open a touble ticket, populate the ticket, and close it to track the process.

The result – your organization starts every day in an audit ready state.  The maximum drift is 23 hours and 59 minutes.  Not a month, 3 months, or 6 months.  No need for heavy, obtrusive scans, no human intervention needed to write remediation scripts, no additional patching activity.

Folks, patching is a big part of the problem, so why would you get excited about any so-called solution that is essentially a patch management process?  Patching is hard and rarely done well.  Why do you think there is so large a time gap between correction cycles with this technique?  Take a hard look at many of the companies pushing configuration management – they often have their roots in patch management and that is how they address the problem.

Security configuration management effectively reduces the attack surface on each machine, but this is achieved only when the configurations are continuously enforced.  When you can detect and remediate problems every day, you create what we call persistent security readiness.  Don’t settle for old school techniques and large gaps between corrections because monthly or quarterly is not persistent.  There is a better way.

A Condensed Guide to the Security Fails of 2009

The past several weeks I have been posting a series I called the Security Fails of 2009.  It was designed to be a look at stories that illustrated the challenges faced in IT security as well as some of the broader issues shaping the industry. 

For your convenience, here is a recap with links:

12/10 – The Marine One Breach – illustrates the threats created by unauthorized applications.

12/14 – The Strange Case of the Missing Cyber Czar – a look at the seven months that had passed since the announcement of the position in May.  Obviously the position has been subsequently filled.  Coincidence?

12/16 – Conficker Becomes a Media Darling.

12/18 – Adobe Takes the Exploit Crown from Microsoft.

12/21 – The Heartland Payment Systems Breach – Lessons learned form the largest breach of customer data to-date.

Security Fails of 2009 – The Heartland Payment Systems Breach

This is the fifth in the series of Security Fails of 2009.  As 2009 draws to a close I think no one would argue that this has been an extremely eventful year for IT security.  While others will soon be trotting out their “best of 2009” lists, I thought I would instead visit some of the prominent fails of 2009. 

In January of 2009, it was disclosed that Heartland Payment Systems had experienced an intrusion into their computers that may have compromised over 100 million customer records.  After the dust settled, the breach was found to involve 130 million customer records, pushing this breach well past the previous record represented by the 2007 TJX breach that compromised 94 million records.  Heartland processes 100 million payment card transactions per month for 175,000 merchants.

By December the attack was traced to admitted TJX intruder Albert Gonzalez who eventually entered into a plea agreement on the Heartland breach and additional charges that he hacked into Hannaford Brothers, 7-Eleven and two other unnamed national retailers.  Heartland has allocated $12.6M for the clean-up, and as of today Heartland was still settling with American Express ($3.6M) and resolving other class action suits.

The scope of the breach re-energized conversations about the efficacy of the PCI standards and the general state of fraud protection for card based transactions.  The dialogue became more interesting when Heartland CEO Robert Carr did an interview with Bill Brenner of CSO Magazine where Carr laid the blame squarely on the audits done by their Qualified Security Assessors (QSAs).  Carr’s comments were viewed by many in the security community as “disingenuous” as most believe that the source of the breach could have been eliminated if Heartland had applied some generally accepted security controls. 

PCI has long been an industry hot button, and the Heartland attack was illustrative of the issues at hand.  Heartland appeared to be in full compliance with the PCI standards, but was attacked by essentially a “garden variety” SQL injection.  In an interesting twist, Heartland’s traditional signature based tools missed the attack, but the attackers actually used antivirus software to cover their tracks and avoid detection. 

So what are the lessons learned?  Heartland demonstrates that even the most sophisticated companies in regards to IT security are still far too reliant on signature based tools and must look to new and evolved technologies to close security gaps that allow long known vectors such as SQL injection to breach their perimeters.  Heartland is also a great “exhibit A” that compliance does not equal security; it is only a temporary measure that certain standards were in place at a point in time.  Finally, in spite of calls to action to rid the card processing industry of fraud, there is not much evidence that anything other than rhetoric came from the attack, so we can fully expect to see another Heartland in 2010.

Why Bad Things Happen to Good Endpoints

I was with a prospect the other day and was asked what, at least for me, was a very thought provoking question.  We were discussing the two major areas of application for Triumfant – continuous enforcement of security configurations and real-time malware detection and remediation – and he asked why you would need the latter if the former was done properly.  In other words, if all of my endpoint protections are in place and everything is properly configured, why am I still getting attacked?

Simple and logical question, right?  But it led me to think long and hard why attacks happen at a very elemental level.  We in security face this question from the powers that be because they cannot understand that attacks still come even though we have added multiple layers of defense. 

After consideration I came up with three reasons.  For perspective, my reasons are very much endpoint centric and presume the attacks have already made their way through protections on the network level, so this is not a cloud to ground holistic view.  Each reason is based on the assumption that the preceding reason(s) have been fully addressed and the represented gap is closed – each reason stands on its own as a problem.  And I will resist the urge to plug in how Triumfant addresses each gap, but I have noted blog entries that do if you care to read on.

Here are my three reasons:

  1. Attacks get through because the machines and the protection software deployed to protect them are not configured to be secure.  The analogy is simple: the most well designed and secure deadbolt lock only secures a door when the deadbolt is engaged.  Too frequently, endpoint protection tools are either improperly installed or improperly configured to perform the tasks for which they are intended, so attacks make it through.  For how Triumfant addresses the configuration management gap see “A New Approach to Configuration Management”.
  2. Attacks get through because traditional endpoint protection tools miss known attacks even when there is a signature for that attack and the protection is properly configured.  The failure rate depends on whose statistics you chose to use, but Gartner puts the detection failure rate at two to ten percent while other studies show failure rates exceeding fifty percent.  Given there will be well over 5M signatures by the end of 2009, ten percent is non-trivial.  See “Antivirus Detection Rates – It is Clear You Need a Plan B”.
  3. Attacks get through because they have been carefully designed and engineered to evade traditional endpoint protections.  These include zero day attacks, rootkits, targeted attacks and the work of the maliciously intended insider.  Zero day attacks are more generic in nature and broker on the fact that most tools require prior knowledge to detect an attack.  Targeted attacks are designed to specifically infiltrate networks and applications to retrieve sensitive information or financial data.  See “It is Raining and You Will Get Wet”.

I am not saying this is groundbreaking thinking here, but if you put things into this perspective, it clearly defines the gaps in protection and subsequently provides a roadmap of what must be done to protect your endpoints.  Reducing the attack surface is clearly not enough.  Antivirus is not getting it done – even the AV vendors say so.  And the bad guys are relentless in their pursuit to exploit any crack in the defenses. 

So what do you think? Too simple or simply brilliant?

A New Approach to Security Configuration Management

In my last post  called If We Know How Breaches Happen, Then Why Aren’t We Doing Something?, I addressed a recent post by Rich Mogull in the Securosis blog called “We Know How Breaches Happen” where Mogull rightfully asked if we know that security configuration problems are at the root of a significant number of breaches, then why more isn’t being done to enforce essential security configuration practices. 

In my post I ended by noting that a better method of automating enforcement of security configurations was needed as most security configuration management tools on the market today are a derivation of patch management techniques and processes.  They rely on the application of pre-written scripts for remediation that are pushed out like patches.  Some tools literally have tens of thousands of these pre-written remediations.   If a new problem is detected, someone – either internally or the vendor – has to write a new script which is then pushed out.  This effort can be non-trivial as new scripts must be handled the same way that a new patch must be tested and managed.  The application of these scripts is very brute force and non-specific.  If someone gets sick (one computer) then everyone has to take the pill (the script). 

That is why an alternative is required.  patch management, or the lack thereof, is constantly cited as a contributing problem to security and specifically, security configuration management.  If security configuration management tools rely on what essentially is the patch management process, why would we expect those solutions to solve the problem? 

Triumfant Resolution Manager takes a decidedly different approach to security configuration management.  It continuously enforces security configurations and policies by scanning every computer, every day, detecting when a machine is non-compliant, building a remediation on the fly for the specific problem for each specific machine and applying these remediations to return non-compliant machines to compliance.   There are no scripts to write, no scripts to be pushed out to every machine.  Triumfant has automated the process of detection, analysis and remediation.  You start every day with every machine audit ready and at the highest state of security readiness.

So how does Triumfant learn the desired security configurations to enforce?  The first way is that it learns the rules from the environment by taking thousands of attributes from every machine and correlating them into the rules that compose what we call the Adaptive Reference Model.  The model is a normative baseline of all of the correlated attributes across the endpoint environment expressed as rules for specific configuration settings.  Essentially, Resolution Manager will learn what is normal and then enforce that normal.  For more insight into the grouping and correlation performed by the Adaptive Reference Model please see this previous post.

Obviously many environments are in a state where there is no satisfactory normal.   The learned option still works, but requires some finesse and is dependent on having a set of machines that can serve as a golden image of your desired configuration.  In this case, you would simply start by pointing Resolution Manager at these machines first, allowing the Adaptive Reference Model to be built with the rules learned from these machines.  Then as you add other machines to the model, they will be assessed against the rules learned from the golden image machines and remediated to be brought into compliance with those rules.    

Seldom is the world as tidy as a golden image.  There are the “except fors” and special cases that keep IT security teams busy.  So the second way to establish security configurations is through explicit rules and policies that are created within a simple wizard driven interface.  These can apply to all machines in the endpoint population or to specific groups, making it simple to accommodate the requirements of groups or job functions.   Rules are extremely flexible and can be extremely precise and specific or employ generic filters to cover broader subject matter.  Using this approach, organizations can deploy security configurations that are both practical and appropriate.

The beauty of Triumfant Resolution Manager is that you can employ a hybrid model.  You can explicitly state the configuration rules that are most important, and Resolution Manager will extend the explicit rules with learned rules to cover the rest.  No other tool can provide this two phase approach, effectively extending the reach of security configuration management well beyond that which is explicitly defined.   

The ability of Triumfant Resolution Manager to continuously and automatically enforce security configurations creates enormous benefit for our customers.  This capability provides all of the advantages of security configuration management with less human interaction and extends the capability far beyond the explicit configuration rules.  The end result is lower risk and increased security readiness.   While it is the ability of Resolution Manager to detect the malicious attacks that evade other endpoint security software certainly garners a lot of attention, the ability to ensure that the endpoint population begins every day properly configured is equally critical in securing any organization.

If We Know How Breaches Happen, Then Why Aren’t We Doing Something?

In his August 26 blog post on the Securosis Blog called “We Know How Breaches Happen”, Rich Mogull made some very good observations about the cause of data breaches. According to Mogull:

“If we look across all our sources, we see a consistent picture emerging. The vast majority of cybercrime still seems to take advantage of known vulnerabilities that are can be addressed using common practices. The Verizon report certainly calls out unpatched systems, configuration errors, and default passwords as the most common breach sources.”

When I was with Cybertrust, Peter Tippett, one of the early pioneers in antivirus software now with Verizon Business, would make the impassioned case (he still is) that following a relatively small number of essential practices would lower an organization’s risk significantly.  Tippett’s researchers from Cybertrust are at the core of the Verizon team that publish the Verizon Data Breach Investigations Report which notes that “2008 continued a downward trend in attacks that exploit patchable vulnerabilities versus those that exploit configuration weaknesses or functionality.”

In other words, a little discipline in security configuration management would go a long way toward making organizations more secure and eliminating the low hanging fruit used by hackers.  Hackers are people too, and like the rest of us they will take the path of least resistance.  They could choose the difficult path of engineering a new zero day attack. Or they could take the far more simple approach of using an exploit that leverages a common misconfiguration known to exist in a significant number of endpoint machines and build an attack with enough variation to evade the signatures in place for earlier versions of that attack. 

GGGR1You can almost picture a Glengarry Glen Ross hacker’s boiler room full of hackers under quota and a boss telling the crew that “Coffee is for hackers that Hack!”  It is just too simple to spin up an exploit that picks off the multitude of unpatched and misconfigured endpoints.

In a separate post, Mogull talks about the Data Breach Triangle in the context of fire triangle (oxygen, heat, fuel – take away any one of the three and the fire goes out): the sides of the triangle are the three components needed for a breach to occur, so removing any one side prevents the breach.  Good security configuration management should help eliminate the triangle side Mogull calls the exploit, which is the vulnerability or flaw that provides the hacker the path to the data.

Given the obvious linkage between breaches and configuration problems, why hasn’t security configuration management become an essential component of endpoint security strategies?  The answer is steeped in irony given that configuration management has replaced patchable vulnerabilities as the exploit of choice.  Many of the companies that offer security configuration management use what are essentially patch management tools as the technical implementation.  Think of the number of patch management vendors that now offer security configuration management. These solutions essentially push out configurations and remediations the same way that they would push out a patch. 

Why is his ironic? While this is a proven and technically sound approach, patching is somewhat universally recognized as problematic.  It is therefore sensible and logical to ask why we would expect that using the same type of tool and underlying processes would prove to be successful in addressing configuration management.  Patching tends to be a brute force process, while configuration management requires much more flexibility and finesse.  And creating a patch (call it any name the vendor gives it – it is still a patch) is a human resource intensive process that requires someone to write, test and deploy a script.  

It would be fair of the patch management vendors to note that a lack of institutional commitment and sound process are also contributing factors and I would agree.  But the parallel between patch management and configuration management is too obvious to ignore.  Mogull’s observations are backed by numerous studies and analysis that cite unpatched systems and security configurations errors as a problem, so the evidence would indicate that these tools are not getting the job done. 

Clearly a different and more automated approach is appropriate, and in my next post I will tell you how Triumfant addresses this problem.  Specifically, I will detail how Triumfant continuously enforces security configurations by detecting machines that are out of compliance with desired configurations and automatically build a remediation to return the machine to compliance.

What is clear is that we need to address these foundational problems to become more secure and reduce organizational risk.  Perhaps now the tools have caught up with the problem and we can make the road of that hacker a bit more difficult.