Breach Counts: We Don’t Know What We Don’t Know (Foghorn Leghorn Edition)

I asked a question last week on Twitter that provoked some interesting discussion and even a slap on the hand.  I thought my question was relatively simple and sensible:

Is it reasonable to wonder if the breaches we know about – the adversary was caught for lack of a better term – might we only be viewing a sample that represents the less well conceived and/or constructed attacks?

Seemed reasonable.  I asked the question because I use the various breach reports for statistics, and they of course report on breaches that are discovered. Think back to the hide and seek of your childhood.  In my experience, the worst hiders were very likely the first caught.  I even mentioned the old Monty Python “How to Hide” sketch.  So it seemed sensible to ask if the reports were skewed to the worst hiders of the attack population.  Or to quote that great security analyst and philosopher Foghorn Leghorn: “that breach is about as sharp as a bowling ball”.

I try very far to stay away from fear, uncertainty and doubt (FUD), but my question pushed the FUD detector of Pete Lindstrom (@SpireSec), a security analyst and founder of Spire Security, past his tolerance point.  Pete’s contention was that raising the question without supporting evidence was a form of FUD, because I was raising a level of uncertainty and perhaps fear.  Point taken, but that does not stop my intellectual curiosity because I still believe there is a bit of Gordian Knot at play here.  I raised the question because I really study the reports and use the presented statistics to support my points about Triumfant so I am not spreading FUD.  Foghorn would likely say that I am “more mixed up than a feather in a whirlwind”.  But the more I look at the statistics, the more I see unanswered questions that lie beyond the available evidence.

Which takes me back to the point of my original question: it is impossible to gauge the problem we collectively face in IT security because we do not know what we do not know.  And what we do not know is the proportion between detected and undetected breaches.  I raised a similar question in a blog post about malware detection rates tow years ago and noted that an undetected attack is still an attack, even if we can’t count it.

The breach counts in the collective reports actually rely on two things: detection and disclosure.  The Verizon Business report is based on the Verizon caseload and cooperation from law enforcement agencies from several countries.  How many breaches are detected that do not show up on the Verizon report or the others? How many breaches are not reported to the authorities?  There are regulatory mandates that require an organization to disclose breaches that involve the loss of certain types of data, but what happens when those regulatory lines are not crossed?  The Verizon Report is actually called the Data Breach Investigations Report.

I go back to what we don’t know.  How many breaches go undiscovered?  How many breaches are discovered and not disclosed?  Are the detected and disclosed breaches representative of the broader population or are they representative of the less well written and less well executed breaches? Are the breaches in the report 99% of the breaches? 50%? The tip of the proverbial iceberg?

These questions have ramifications, particularly when we put them in the context of what evidence we do have.  For example, if we discover that the discovered breaches are not exactly, as Foghorn would note, the sharpest knives in the drawer, what does it say about the ability of organizations to detect breaches when the average time from infiltration to detection is 173.5 days as reported by the Trustwave report?

I agree with Pete – we need evidence.  Unfortunately, a reasonable conclusion that can be drawn from the collective evidence of these studies is that most organizations are not equipped to detect breaches.  Which of course adds to the conundrum the evidence points to the fact that we will struggle to gather the proper evidence.

I don’t think the collective industry will answer these questions, because they are the uncomfortable detritus of years of placing so much emphasis on prevention. The “2011: Year of the Breach” declarations have been an uncomfortable public realization for the industry and for organizations.  Even if we were better at detecting breaches, organizations will not self-disclose unless required to do so for a variety of valid reasons.

So, FUD accusations aside, I stand by my question.  Of course, Foghorn would likely say that I “Got a mouth like a cannon. Always shooting it off”.

In 10 Days, the Mac Safe Haven Becomes a Botnet Spewing, APT Vulnerable OS

In rapid succession, the IT security world, not to mention the perceived cocoon of safety for Mac users, was rocked by two announcements.  On April 4, Russian antivirus company Dr. Web announced that they had discovered a Mac Botnet, called Flashback, and that the bot had infected 600,000 machines.  About ten days later, Kaspersky announced the discovery of a backdoor trojan called Backdoor.OSX.SabPub.  This attack leverages an exploit that uses malformed Word documents to deliver malware that opens a backdoor that can be used for advanced, persistent attacks.  Holy APT Batman!  Perceived safety to botnet to advanced persistent threat in 10 days!

Oh the shame.  The Mac went from safe haven to botnet spewing, APT exploitable platform tied to three-year old vulnerabilities before our very eyes.  As I tweeted, the heads of the Mac fanboys and the APT crew were simultaneously exploding.  Mac users were sent to various sites to download software to check their machines for Flashback like common Windows XP users.  I could not help but wonder if some enterprising bad guys had set up malware delivery disguised as Flashback checkers – wouldn’t that have been ironic.

I am really just having some fun here.  I take no joy in the Mac becoming a target, although it is good for business.  I am also not on some war against “smug” Mac owners because I have made the jump myself.

For me, the folklore/mythology of the Mac world as a safe haven from malicious attack reminds me of a scene from the classic movie and personal favorite, Butch Cassidy and the Sundance Kid.  In this scene, Butch and Sundance have fled to Bolivia and have taken a legitimate job guarding the payroll for a mining company.  At the beginning of the scene they are riding with the old, hardened mine boss (played perfectly by the great character actor Strother Martin) and begin to argue where the inevitable ambush will occur.  The mine boss responds disdainfully: “Morons. I’ve got morons on my team. Nobody is going to rob us going down the mountain. We have got no money going down the mountain. When we have got the money, on the way back, then you can sweat.”

Mac users, I hope you have enjoyed the ride down the mountain.  The recent Mac malware news just means that the downward portion is over, and now that there is a critical mass of Macs plugged into the networks and systems where the money lies. It is time for Mac users to sweat.

We could engage in what I am sure will be an animated conversation about the superiority of the Mac OS and the inherent vulnerabilities of Windows, but I contend this was all about opportunity.  Sure Windows machines were likely the road of least resistance, but malware writers have proven to be a resilient and industrious bunch and repeatedly rise to find a way around every barrier put in their path.  So now that the opportunity has arrived – what the adversary wants is on or accessible via the Mac – the Mac OS barriers will also be breached.

I should point out that Mac users are not finished with their journey into the seedy underbelly of IT security.  Not surprisingly, the sales of Mac AV software has gone way up.  Wait until the Mac people connect the dots that the same crew that discovered the malware also sells them AV software.  Of course, that AV software will at least partially return their cocoon of safety, until they find out that motivated adversaries will drive around their new shiny AV software like a traffic cone on the interstate.

I hope they enjoyed the ride down the mountain.

The Reader’s Speak – the Top Ten Posts of 2011

The year is rolling to its inexorable end and it is time to look back fondly on the top blog posts from Exceptional Security in 2011.  The selection process is generally scientific, using the site stats to gauge reader interest.  But personal bias and self-indulgence are also a factor.  At least I am honest, and I refrain from clichéd predictions.

Advanced Persistent Threat: Solution – No, Effective Detection – Yes.  This post was actually written in January of 2010 but has been the most-read post on the blog.  The post addresses the qualifications of Triumfant as a viable and effective tool for detecting targeting attacks, including APT.

The UC Berkeley Breach – You Don’t Know What You Don’t Know. Another post written before 2011 that continues to resonate.  In fact, this post is a very early expression of what I now call Rapid Detection and Response – the ability to quickly detect the attacks that evade preventative software and quickly respond to the breach.

Trojan Horses, Payloads and Flamethrowers.  This post turns the most overused cliché in IT security – the Trojan Horse – on its ear to illustrate rapid detection and response and the folly of relying solely on perimeter defenses.  Not to mention gross misuse of literary license as I insert flamethrowers into classical mythology.

Sayano-Shushenskaya Accident A Model for What a Duqu/Stuxnet Combo Could Mean.  This post uses the incident at a Russian hydroelectric facility to illustrate what kind of terrorism could be performed with a Stuxnet style attack.  The images from a 900 ton turbine unit tearing free of its moorings seemed to provide readers a visual reference point for the potential of such attacks.

Purely Commercial Espionage – The Advanced Persistent Threat Targets Businesses.  The exact definition of APT is hotly debated, but most see it as cyber warfare at the nation state level and not an issue of commerce.  Regardless of definitions, this post explores the burden that commercial organizations are bearing from targeted attacks that extract intellectual property from U.S. companies, negatively affecting the economy.

Certificate Authorities Hacked – So Who Can You Trust? This post speaks to the corruption of the chain of trust caused by the hacking of several certificate authorities.  The important takeaway is that prevention mechanisms can be fail along a variety of vectors, so adding rapid detection and response is necessary and prudent.

The Emotional Barriers to Embracing the Presumption of Breach Doctrine.  Why, in the face of all statistics and other forms of evidence to the contrary, do people cling to the notion of the 100% effective preventative shield?  This post looks at the emotional component that prevents highly rational people from admitting that they are getting breached and taking the appropriate action. I think it is a concept worth exploring more broadly.

Finding a Needle in a Haystack – Child’s Play! Another alternate take on a treasured IT security cliché – the needle in the haystack.  Specifically that finding a known thing – the needle – in a homogenous population – the haystack – was a far easier proposition than locating malware without a signature in the vast IT world. Too big to do in one post, it turned into a series of posts.

Virus Attacks U.S. Drone Fleet and the Need for Rapid Detection and Response.  Sometimes when you are trying to get some traction around a concept or term, the world throws you a bone.  As I was introducing the concept of Rapid Detection and Response, the story broke about the attacks on the C&C center for the U.S. drone fleet and how that was a perfect scenario for the concept.

Time to Put Your Antivirus Software on a Diet.  This was posted in late 2010 but got a lot of reader momentum in 2011.  The post is an answer to the question frequently asked when we present Triumfant: “Are you saying you replace antivirus tools?”.   As a bonus, it contains my favorite phrase of 2011: fusillade of FUD.

Well, that wraps 2011 for Exceptional Security unless something big happens that requires comment.  Otherwise, thank you for reading – it is always humbling to know that someone takes the time to read.

See you in 2012.

Making the Case for Rapid Detection and Response

In my post “You Need a Plan B for Security“, I cited two numbers from the Verizon Business 2011 Data Breach Investigations Report (published May 2011): 60 and 86.  These two numbers jumped out at me from the report because they are subjective numbers that emphatically support the need for rapid detection and response to identify those attacks that get through preventative IT security software. The attacks that either evade perimeter and endpoint shields, or the attacks that the shields simply fail to detect.

“60” represents the percentage of attacks in the study that went undiscovered for a month or more.  Three out of five attacks that got past the organization’s shields were free to do damage on the host machine and the network for an extended period.  Free to establish command and control, spread to critical systems, and exfiltrate sensitive data and intellectual property.  By the way, there is nothing to indicate that these attacks were super sophisticated zero days or the advanced persistent threat.  The lack of rapid detection and response makes such sophistication unnecessary.

Organizations rest in the false security of security suite reports that show a steady increase in malware detection rates artificially inflated by the always-increasing number of attacks.  Or they are willing to take a gamble that the number of attacks that do get through will be minimal.  Ask Sony how many attacks it takes to cause an enormous amount of seemingly endless headaches and public relations hits.  Better yet, ask their CEO who is under pressure to resign because of the incident.

“86” represents the percentage of reported attacks that were discovered by a third party.  Conversely, this means the attacked organization found the problem only one out of eight times.  If a third party had not brought the attack to their attention, it may have never been discovered.  One could easily surmise that if left to the attacked organization to detect the problem, the 60% number above could have been much worse.

It is clear that organizations are not prepared to detect and respond to successful attacks.  One out of eight is a horrible rate given the accelerating pace that attacks are getting through the shields.  They most certainly are not prepared to detect these attacks rapidly before they can cause significant damage.

There is another component to consider.  Detection of the attack by a third party means that the attacked organization’s dirty laundry is now public.  At a minimum this erodes public and consumer trust and at its worse can negatively impact the organization’s brand and potentially affect valuation.

Budgets are tight, the economy staggering.  Rather than spend more money on yet another shield that will get compromised, organizations may want to take the numbers 60 and 86 to heart and take a hard look at their rapid detection and response capability.  Because by ignoring the need for rapid detection and response, organizations are enabling the adversary to establish a long term and highly destructive presence in their environments.

Attacks are getting through.  You must have a way to effectively identify successful attacks and provide the actionable information to make an informed and rapid response.

Trojan Horses, Payloads and Flamethrowers

Today I will use perhaps the single most overused symbol/metaphor used by the IT Security Industry: the Trojan Horse. I pride myself in avoiding clichés and overused metaphors, but in this case I think I have a slightly different spin.  What if Paris had a flamethrower?

The account of the Trojan Horse is all about shields and the reliance on those shields.  Unlike the shields we deploy today in IT security, the walls of Troy actually worked and had repelled attackers since anyone could remember, including a multiple year siege by the largest invading army ever amassed.

Eventually the relentless, highly trained, and well funded adversary (ring a bell?) built the wooden horse and filled it with 30 men.  They presented the horse as tribute at the gates of Troy and pretended to withdraw their army.  The Trojans willfully brought the horse into the walls of the city, forever consigning us to the Trojan Horse as the metaphor for a user willfully letting malicious software into their systems and launching hundreds of hackneyed advertisements, web site graphics and booth backdrops.

However, I would suggest that too much emphasis is placed on the delivery mechanism rather than on the payload.  The wooden horse did not bring down Troy, it fell because of what was in the wooden horse.  It was the payload – the 30 Greek soldiers that crept out and opened the gates to the city – that ultimately spelled the demise of the city.  Had the horse been filled with tradeshow booth giveaways, there is no real story.  Sometimes in IT security we get so swept up in how something made it to a machine that we forget that it is the payload that ultimately does the damage.

I would also call to attention the emphasis on the perimeter. The people of Troy became so dependent of their perimeter defenses that I suspect they rarely thought of the defense of an attacker that had successfully breached those defenses. Once the horse and its payload were inside the city walls, the perimeter defenses were irrelevant.  And from the story there appears to be no protections inside the city. Today we focus on a disproportionate amount of our diligence on the perimeter, when the attackers are after what is on individual machines.

Now to the flamethrower.  If you read the various accounts of the Trojan Horse, there were one or more people in Troy that had serious misgivings about the Trojan horse.  Something did not feel right – it was anomalous.  What if Paris, son of the King of Troy, had taken the warnings to heart, picked up his new flamethrower given to him by Apollo, and reduced the Trojan Horse and its payload to embers?  We all might be speaking Trojan today! Paris was an accomplished archer so one flaming arrow would have done the trick.

Less dramatically, what if Paris had assigned someone to watch the horse?  Given it was anomalous, that would make sense.  Once the Greek soldiers began to emerge, the sentry could have initiated a rapid detection and response protocol and quickly acted to stop the Greeks before they could open the gates.

The story of the Trojan Horse is a contemporary warning about the over-reliance on perimeter defenses and the lack of tools for rapid detection and response when the perimeter defenses fail.  Because unlike Troy, your perimeter defenses are failing and things are getting through.  To make matters worse your defenses don’t see the wooden horses that are making it to the host machines, so they most certainly are not seeing the harmful payloads.

The new lesson of the Trojan Horse: now is the time to implement a rapid detection and response tool to complement those lovely walls you keep trying to make breach-proof in the face of all evidence to the contrary.  At least invest in a flamethrower.

You Need a Plan B for Endpoint Security

You need a Plan B.

Plan A in endpoint security is to prevent malicious software from infiltrating a machine.  Most of the software on the exhibit floor of any IT security show is Plan A software with the remainder aimed at identity management.  As the number and complexity of attacks steadily increase, the amount of Plan A tools deployed at any given site has gone up proportionately.  Every year brings out a new “it” Plan A product and another layer of shields.

In spite of all of this Plan A activity, the number of successful infiltrations is on the rise.  Malware detection rates vary from study to study, but if you are RSA, NASDAQ, Sony, or any of the scores of recent breaches you realize that the bickering over the numbers on these studies is meaningless once you are attacked.  Add targeted attacks and the Advanced Persistent Threat to the mix, and the picture is less than rosy.

You need a Plan B.  Plan B is not a difficult concept to grasp or justify.  It simply says that there are no 100% shields, no fool-proof Plan A.  It accepts the hard truth that motivated, well-funded attackers will infiltrate your systems.  Therefore, you need a Plan B to detect the attacks that evade your Plan A software and so you can take informed action based on that knowledge.

The “Verizon Business 2011 Data Breach Investigations Report”, Published May 2011 had two interesting facts that scream for the need for a Plan B:

  • 60% of the breaches they studied went undetected for over a month.  The bad guys had free access to internal systems for extended periods.
  • 86% of the breaches were discovered by an external party.  The organizations would have never known they had been breached if someone from the outside had not told them.

Don’t take for granted that you have not been infiltrated because your Plan A software has not detected the presence of an attack.  That is self-deceiving logic.  If the attack gets past the protection of Plan A it has already evaded the detection capabilities of Plan A.

Here is something else to consider:  most of the Plan A software are shields to defend the increasingly porous perimeter.  Successful infiltrations are obviously at the endpoint.  Furthermore, the shields are often concerned with the attack vector and not the payload.  Once an attack makes it to the machine, it is all about the payload.  So again, we are back to the need for a Plan B that has a different focus and methodology than Plan A.

Having a Plan B is not an admittance of failure or running up a white flag on the idea of prevention.  It is a prudent, pragmatic and necessary response to the current threat environment.  You need a Plan B that focuses on detecting successful attacks and provides the analysis necessary to take immediate and informed action.  You need a Plan B that is not tied to traditional techniques that rely on prior knowledge such as signatures.  Finally, you need a Plan B that lives where the attacks happen – the endpoint.

It all goes back to the opening line: You need a Plan B.

Einstein Could Smell the Coffee – Can You?

We cannot solve our problems with the same thinking we used when we created them” Albert Einstein

The past weeks have been on a Headline-per-day rate of high-profile hacks (today it is NATO). What makes these hacks stand out is that they are mostly organizations that you would expect to be secure – Citigroup, Booz Allen, and MIT are a short representative list. Sony is still reeling from their PS3 hack, and shareholders are calling for the resignation of the CEO.

Yet even at the companies that have been hacked, old thinking is being employed as the solution. Or as Einstein put it: the same thinking we used when we created the problem. The bigger tragedy may be that companies continue to look toward vendors who have rested on old technologies and have pushed half-hearted spackle jobs to cover the holes in their products as “innovation”.

Wake up. Smell the coffee. Because it is brewing and it is strong. The adversary has moved on to new techniques and is operating with a new boldness because of tepid defenses we put in their path. Heck, they don’t even have to come up with new techniques or reach the lofty designation of Advanced Persistent Threat, because we don’t adequately defend against the well-known vulnerabilities and attack vectors. We either forego locks on the door or install them only to not bother with turning the dead bolt and engaging the lock. And in the face of enormous evidence that the locks are no longer effective, we continue to install new, shiny versions of the same old technology, only to scratch our heads when tomorrow’s headline is revealed.

How many more lead headline hacks will companies need to see before looking to innovative approaches? Has everyone become so numb that they look the other way until they are the headline? The recent attacks have been characterized as “unsophisticated” as if that should be somehow comforting. I think it is the opposite, because it is likely the sophisticated attacks are working undetected, busily extracting confidential data and corporate IP.

I believe Einstein would tell us that it does not take a genius to see the problem, and it does not take a genius to determine that it is time to embrace new approaches to security. Funny thing is that I discovered this Einstein quote while verifying the attribution of another quote that also applies here:

Insanity: doing the same thing over and over again and expecting different results.” Albert Einstein