Sayano-Shushenskaya Accident A Model for What a Duqu/Stuxnet Combo Could Mean

On August 17, 2009, a 900 ton hydroelectric turbine was torn from its moorings at the Sayano-Shushenskaya hydroelectric power plant and dam in Russia.   The 900-ton unit actually lifted high enough (40-50 feet) to crash into the ceiling of the turbine facility.  The accident ultimately cost 75 people their lives.  Every one of the ten power generation units in the plant was damaged, some irreparably.  The 6,500 MW power station will not return to full capacity until 2014.  40 tons of transformer oil was released into the surrounding ecosystem, killing an estimated 400 tons of trout in two fisheries.  Untold hours of production capacity of surrounding businesses were lost due to the interruption of power to the area.  You can get a full picture of the event from a DOE presentation.

Before picture of the turbine generator hall of the Syano-Shusenskaya Plant

Before picture of the turbine generator hall of the Syano-Shusenskaya Plant

In my last post I defined how Duqu is a notable shift in the malware game, most notably as a precursor to carrying out Stuxnet level complexity attacks without the need for human intelligence gathering.   The ability to potentially affect and disrupt industrial controls in turn creates the potential for industrial blackmail and potentially cyber-terrorism.

For the record, I abhor peddling fear and in my time in the IT security space I have never used that tactic, and I am not using it now.  I do think what I described is a very real threat that is at our doorstep right now.  Duqu rang the doorbell.

I spent Wednesday at the SINET 2011 Showcase put on by the Security Innovation Network.  Triumfant was honored to be recognized as a SINET 2011 Innovator at the event, and General Keith Alexander, the Commander of the U.S. Cyber Command gave the closing keynote.   General Alexander used the Sayano-Shushenskaya accident in his talk, and it immediately struck me that the General had provided me with the example I needed for this conversation.

Post-event view of the turbine generator hall.

The Sayano-Shushenskaya plant had been a place of historical operational problems, and the specific turbine (Turbine 2) at ground zero of the accident was particularly problematic.  The turbine had a history of vibration issues that kept it from safely operating at capacity, and a new vibration controller had been installed in 2009.  This controller was offline on the fateful day when another plant experienced problems and Sayano-Shushenskaya was asked to raise capacity to make up for the shortfall.  When the load on Turbine 2 was increased, vibrations steadily increased to over 5 times the load limit, and the structural integrity of the unit ultimately failed.

Back to Duqu.  Introduction of Duqu into the Sayano-Shushenskaya would gather the data needed on not only how to infiltrate the plant systems, but where the plant was most easily compromised.  Hacking into maintenance records would readily pinpoint Turbine 2 as the weakest physical link.  The keylogging capabilities could gather the necessary access to the industrial controls of the plant, including the vibration control process.  The bad guys do not need human intelligence from the plant – Duqu provides all the data they need.

The information to disable energy production in hand, a Stuxnet level attack can be written to infiltrate the industrial control systems of the plant.   The low effort approach would be to disable the vibration control system for Turbine 2 at a time when peak capacity was required and wait for the failure.  A more aggressive approach could actually manipulate the demand on Turbine 2 to force it to run beyond established limits and, with the disabling of the vibration control system, guarantee an event on demand.  This is no different than what Stuxnet did to the centrifuges in the Iranian nuclear sites – it made them spin beyond operational tolerances and destroyed the devices.  The difference is that this attack sends a 900-ton turbine structure 50 feet into the air.

NOTE added 11/18/2011:  Since this post went public on 10/28/2011, it was reported that a water plant in Springfield, Illinois was impaired when the SCADA industrial controller from a water pump was hacked and manipulated to damage the pump and render it operational.  The hackers simply turned the system on and off until the pump overheated and burnt itself out.  Details can be found at Krebs on Security and on Wired.

Too much physical destruction for you to consider?  How about infiltrating the industrial controls of a pharmaceutical company and changing the machines that control the flow of ingredients.  No explosions, no floods, no fires.  But a disturbing bit of potential terrorism.  And not just an Advanced Persistent Threat stealing intellectual property.

Now do you see the connection? Was that the doorbell?

(Triumfant has gone on record as saying we would detect Duqu and would be able to stop the attack before it collected the data it seeks.)


Duqu Enables Stuxnet Level Complexity Against Commercial Targets

In my opinion, the recently discovered Duqu attack is more significant on a broad scale than the discovery of its predecessor, Stuxnet.  I think Duqu will force a much broader re-examination of IT security philosophies, particularly those commercial organizations that felt removed from Stuxnet grade attacks.  Duqu is the clear wake-up call that no one should feel immune from the Advanced Persistent Threat or well-crafted, targeted attacks.

Those who have investigated Stuxnet have noted that there was a tremendous amount of intelligence required to make Stuxnet so effective in disabling the Iranian nuclear program. The nature of this intelligence led investigators to conclude that humans were the source for this intelligence.  This finding was significant because the need for human intelligence gathering was obviously a limiting factor for even the best organized and well-funded cyber criminals to successfully launch an attack of that level of complexity and reach.

Duqu is evidence that the adversary is developing tools to automate the intelligence gathering process.  Duqu is designed to infiltrate the same industrial control systems as Stuxnet, gather information for 36 days, exfiltrate the gathered data, and then destroy itself.  One of the programs eventually activated by Duqu is an infostealer that is used by the adversary for “enumerating the network, recording keystrokes, and gathering system information” (according to the Symantec analysis). Even the 36-day window is designed to bridge the 30-day password change requirements in place for many organizations.

All of this information is collected and sent to the command and control site before the attack destroys itself.  The collected data provides the adversary the information needed to penetrate the intended victim’s network and the insight needed to build the malware to perform the intended purpose. It demonstrates a patient, deliberate, and calculating adversary that is willing to write a sophisticated piece of code to gather information to maximize the second wave of the attack to reach the attacker’s ultimate end game.

If attacks like Duqu succeed in eliminating the need for human intelligence gathering activity, then the game changes.   While Stuxnet was a leap forward in malware technology, you essentially had to be a nation state with human intelligence operations or at least an organization capable of somehow accessing such information to implement the attack.  If attacks like Duqu can gather that intelligence through automated processes, then the barrier for entry (human intelligence gathering) is greatly reduced or eliminated, allowing a lot more bad guys into the game.

I don’t deal in fear – never have.  But with a Duqu to gather intelligence, it is conceivable that cyber criminals could gather the information necessary to blackmail industrial organizations by building attacks aimed at the industrial controllers at the core of their operations or production. Remember that Stuxnet caused industrial devices to operate in a way that was destructive to those devices.  The physical destruction set back the Iranian nuclear program three to five years by most assessments.  It is therefore conceivable that similar damage could be wrought at assembly plants and operational locations, impairing the ability of companies to do business.

With the discovery of Duqu, commercial organizations no longer have the luxury of viewing Stuxnet as an aberration played out on the world political stage by nation states.   Duqu brings Stuxnet to their back door.  The Advanced Persistent Threat is no longer about the loss of confidential information or intellectual property, but about actual interruption of operations.  And I can assure you these businesses can put a hard dollar figure to interruptions of operation.

Organizations that were slow to adopt the assumption of breach doctrine and put tools in place for rapid detection and response, must now rethink their approach.  Organization will need to detect attacks that get through their defenses and be able to rapidly respond to address those attacks.  I think we will look back in several years and see the discovery of Duqu as a significant milestone when attacks like Stuxnet stopped being the concern of the NSA or the DoD and became a threat to the broader commercial realm.

(Triumfant has gone on record as saying we would detect Duqu and would be able to stop the attack before it collected the data it seeks.)

Yes, Triumfant Will Detect Duqu

The Duqu attack has been gaining a lot of attention this week, and when an Advanced Persistent Threat like Duqu is announced, I get the inevitable questions of “Would Triumfant detect this Advanced Persistent Threat?”  Based on a review of the research presented by Symantec, the answer would be yes.

For those not familiar, Symantec researchers discovered the Duqu attack and released the details in a bulletin “Symantec Security Response: W32.Duqu“.  Duqu is being called a precursor attack for Stuxnet, because it was written to gather information about the applications and networks of an organization to provide the data necessary to execute a future attack on an industrial control facility.  The report notes that Duqu uses much of the same logic as Stuxnet without the destructive capabilities.  The attack exists for 36 days and then destroys itself.

In the Symantec Security Response is the following description of how Duqu infiltrates the targeted machine:

“Duqu consists of a driver file, a DLL (that contains many embedded files), and a configuration file. These files must be installed by another executable (the installer), which has not yet been recovered. The installer registers the driver file as a service so it starts at system initialization. The driver then injects the main DLL into services.exe. From here, the main DLL begins extracting other components and these components are injected into other processes.”

After reviewing this with Dave Hooks, our CTO, I can tell you that Triumfant will detect this attack and present it to the attention of the administrators as an anomalous event.  There are several triggers that will initiate the Triumfant analysis process, but I will use the one we can present with the most certainty based on the information available.

The Symantec report cites that  “The installer registers the driver file as a service so it starts at system initialization”.  A new service will be detected by the Triumfant agent running on the attacked machine the next time the machine is restarted, and detection of this service will trigger Triumfant’s real-time analysis.  When the agent contacts the Triumfant server to begin the real-time analysis, the Triumfant’s server will in turn initiate probe requests to the agent on the attacked machine.  These probes are sophisticated algorithms designed to correlate changes to the infected machine for the purposes of identifying all of the damage from the attack.  These probes would identify the injection of the main DLL into services .exe, and the other DLLs injected into the other processes.  Triumfant would also correlate the internet traffic tied to the attack with any affected ports and IP addresses.

Triumfant will perform this analysis and return a comprehensive report that shows an anomalous application with the new service and the related services that had been corrupted.  We also suspect that the installer would likely have been an autostart mechanism which would trigger the same analysis, but since the report gave no details about the installer we can not make that claim with certainty.

In summary, based on the Symantec analysis we believe that Triumfant will see Duqu and will build a remediation to stop the attack and repair all of the associated damage to the affected machine.  We think this is notable given that most articles I have encountered about Duqu say that there is no tool that will detect and/or stop the attack.  The ability to detect and remediate Duqu is also a great example of what we call Rapid Detection and Response.

Will Triumfant detect Duqu? Yes.

USB Drives – Cool Tool or Malware Delivery Device

Behold the USB drive. Simple. Functional. Efficient. The USB device is also a symbol of all that makes IT security so difficult. But take heart, because the USB device is also illustrative of the functions and benefits of Triumfant.

Why does the USB key represent the difficulties with IT security? Because a USB device
is an infiltration and exfiltration method wrapped into one tidy package. The bad guys are using USB devices to deliver malicious payload to host machines because this vector readily evades perimeter network defenses that use techniques like deep packet inspection and sandboxing. Unfortunately, techniques require that the attack come across the wire to work, so the attacks delivered by a USB device easily fly under their radar. The USB device has become a very effective mechanism for delivering the targeted and sophisticated zero day attacks and advanced persistent threats that are becoming increasingly difficult to detect.For an example, start with Stuxnet, the malicious attack that grabbed more headlines than a Britney Spears midnight trip for a haircut. Stuxnet evaded protection by using USB drives for transport to the host machines from which the attack spawned.

In regards to exfiltration, there is no simpler tool for offloading data than a USB device. While this has great utility, it is a major problem in the context of data loss prevention (DLP) activities, as once data is loaded onto a device there is absolutely no control of where that data may land. All bets are off.

You would think that USB devices would be the bane of every IT security person on the planet, yet security vendors give them away at industry tradeshows. Most people will pop in a USB key with little thought of the risk, so a “just say no” approach is not effective. Our CTO was at a customer recently and was told that USB devices were not allowed at the site. Minutes later he produced a report that showed that USB devices had been used in over 20% of the machines in the past two weeks. So much for strongly worded guidelines.

The problems surrounding USB devices are useful in pointing out the value of Triumfant:

Malware detection and remediation. Triumfant will detect attacks that are delivered to a machine via a USB device, analyze the attack, and build a remediation to stop the attack and repair all of the damage to the machine. Infection to remediation in minutes. Remember, Triumfant detects attacks by identifying and analyzing changes to the machine, and is therefore attack vector agnostic.

Continuous enforcement of policies and configurations. With Triumfant you can build and enforce policies that disables the use of removable media like USB devices. Triumfant will set the policy and remediate any machine found to be out of compliance.

Continuous monitoring/situational awareness. Your organization may choose to not disable USB devices. Triumfant can provide information about what machines have had a USB device inserted and can identify machines with unusually high levels of data movement. Alternately, if you do disable the devices you may also have users with Admin rights to their machines, enabling them to change the configuration of the machine to override the policies. Triumfant can provide information about what machines have had a USB device inserted and identify those machines where the policy has been altered. Triumfant is not a data loss prevention (DLP) tool and therefore cannot tell you what, if any, data was exfiltrated, but we can tell you that such an exfiltration was possible.

In summary, Triumfant is able to protect machines from attacks delivered by USB devices,
is able to enforce configurations that disable the use of USB devices, and provide insight into usage patterns of USB devices.

If only Triumfant could help me find the numerous USB devices my teenagers borrow and never return. Of course, once they have them, perhaps it is best I don’t plug them into my machine.

Is Security Tech Failing Us or Are We Failing to See the Light?

I read a very, very good article yesterday on the Information Week site called “Outgunned: How Security Tech Is Failing Us”.  The article takes a hard look at why organizations are losing the battle against the evolving threats and makes some very good recommendations.

I only have two issues with the article.  First, I think the title is a bit misleading, as I don’t necessarily believe that the tech is failing anyone.  There is lots of good tech available, some of which is noted in the article.  In my opinion, the failure comes from the dogged persistence of organizations that will not let go of traditional technologies and the “usual suspect” mega-vendors.  Which leads me to my second issue when the author ends a great piece by referring back to Symantec and McAfee, thereby making my first point.  I have nothing against these two companies and we in fact partner with both, but the default of falling back on the AV mega-vendors is one of the key factors that kills innovation.

The article does a great job of describing how the world has changed so there is no need to rehash it here.  What is inexplicable to me is that organizations will not let go of old tech or the mega-vendors that have trotted out all manner of add-on tech to try and fill the gaps.  No organization in their right minds would still build mainframe centric systems with block mode 3278 dumb terminals.  Yet these same sane-thinking companies refuse to let go of protection technology that relies on prior knowledge of an attack such as signatures.

The mega-vendors knew AV was in trouble so they started layering all manner of new tech (most acquired tech, actually) onto the agents.  Host based IDS, heuristics, behavioral analysis, and now prevalence technologies and whitelisting.  The result is an agent that forces users to abandon their machines for periods of heavy scanning.  Don’t laugh – I know a large company in Northern Virginia where the Wednesday long lunch is a tradition that is predicated on the fact that their AV scan starts at noon on Wednesdays.

And still the attacks come.  And just like at the end of the article, organizations still look to their mega-vendor for answers.  Hoping against hope that a 100% shield is not a mirage but one new feature away.  Good luck with that strategy.

I will give Mr. Shipley his due.  He is the first I have seen in print that essentially says that traditional protections are essentially a commodity where there is no clearly differentiated product.  Specifically, Shipley notes: “We’re pouring billions of dollars–literally–into security products that are gaining us very little.  We don’t retire anything but rather pile on more layers, leading to increased complexity, expense, and exposure.”

So back to my original issue – for new tech to be effective, there must be significant changes in thinking.  Organizations must accept that they are being attacked.  Not “will be” – “are”.  They must accept that the days of a shield have gone.  Forever.  We will never get that toothpaste back into the tube.  They must understand that attacks come from everywhere, not just the network.  Stuxnet travelled by USB key, malicious insiders are everywhere.  So all of the deep packet inspection in the world will not offer 100% detection.  They must accept that money has to be put toward tech that detects when an attack happens and then works quickly to stop the attack.

Here is another part of this equation that no one wants to hear – organizations must be willing to accept that fighting back against the threats we see today is hard work.  The days of installing AV and keeping the signature files updated are gone.  Do not discount this factor – many companies would rather take a low-friction path and bet against the odds than do the hard work to protect their confidential data and IP.  The hard work extends to going to management and explaining that hard, sweeping change is necessary and that while the tech decisions made five years ago were sound the world has changed and IT security must keep pace.  Change scares everyone, but better to be scared than on the front page of the NY Times.

If you have read this blog before you know this is not new subject matter for me.  Simply look one entry back to this post about what I call the Blue Pill Stupor.  Or here or here (“why innovation gets throttled”) or here (“denial of innovation attack”).  Do I have a horse in this race?  You bet. Triumfant offers some very innovative technology that solves a significant number of hard problems, but I am quite sure that companies sometimes pass on us because we are not a mega-vendor and because we represent a departure in the perceptions and assumptions that have shaped IT security for far too long.

Triumfant requires no prior knowledge to detect an attack, and will detect attacks regardless of how the malware got to the machine.  In addition, we thrive on detecting the work of malicious insiders.  But wait, there is more – Triumfant will build a situational and contextual remediation for every attack it detects.  Infection to remediation in five minutes.  I firmly believe that installing the Triumfant product would allow organizations to pull multiple (five or more) products off their endpoints.  It would be like an instant CPU upgrade.  In addition, I am quite sure the organization would come out ahead in regards to expense.

Compelling story?  Yeap.  Innovative?  Check.  Able to detect the attacks that evade traditional protections?  Absolutely.  Detects the Advanced Persistent Threat?  Roger that.  Unfortunately, I leave companies knowing we can help but also knowing that they are not willing to put aside their biases to let go of old thinking.

So while I applaud Mr. Shipley’s article I do disagree that the lack of tech is the problem.  If organizations are willing to change their perceptions and assumptions, there is tech out there that can help.  Some of it may well be from Symantec and McAfee, but much of it is likely from someone else.  There is no more overused term than “think outside the box”, but I am afraid it fits here, especially since the box seems to be self-imposed.  Industry analyst Rich Mogull notes in the Securosis blog “There is No Market for Security Innovation“ that buyers don’t consider innovative products until they believe “existing tools are failing so badly that you can’t keep the business running”.   All of the new tech in the world can’t fight that line of thinking.

Or think or it this way – according to Mr. Shipley, things are bad and will get worse.  So what do you have to lose? Or gain?

Stuxnet is the Latest Wake-up Call from the The Blue Pill Stupor

“You take the blue pill and the story ends. You wake in your bed and believe whatever you want to believe.” Morpheus in “the Matrix”

I was at a reception
two weeks ago, listening intently to Maj. Gen. Suzanne M. Vautrinot, the Director of Plans and Policy for the U.S. Cyber Command, when I had an epiphany.  Gen. Vautrinot was speaking to the need for public support to fight the growing cyber threats and used the “red pill, blue pill” metaphor from “The Matrix” to describe her belief that people are choosing to ignore the problem rather than face the facts in front of them.

I nearly sprang from my chair.  I had been searching for a metaphor to explain why rational and smart people continue to ignore the evidence around them concerning IT security in general and endpoint protection specifically.  While Gen. Vautrinot used the reference in a somewhat different way, she had shown me the light.

I call the phenomenon “the blue pill stupor” – the process of ignoring the evidence around you in the hope of maintaining the status quo you choose to believe because, frankly, it is an easier path.  You simply take the blue pill, return to the reality of your making, and hope there are no consequences.

Let me tell you why you’re here. You’re here because you know something. What you know, you can’t explain. But you feel it. You felt it your entire life. That there’s something wrong with the world. You don’t know what it is, but it’s there.

Before Morpheus offers Neo the two pills, he speaks to what has drawn Neo to that moment.  Neo has the sense that there is something wrong but Neo, not knowing he is living a dream, cannot see beyond the reality created for him.  That is what makes the IT security version of the blue pill stupor so confounding – we see the evidence that the world has changed all around us.   Currently, everyone is all concerned about Stuxnet  and rightfully so – it is a scary example of what we face today.  But the Stuxnet frenzy will wane, and unless it happens directly to them, no one will really act differently.  Just think back to how gorked everyone was about “operation Aurora” and how that now feels like a distant, hazy past.  Unlike Neo, we make a conscious decision to ignore what we know and hope against hope that it will all be okay.

You take the red pill and you stay in Wonderland and I show you how deep the rabbit-hole goes.

A second character in The Matrix, Cypher, is offered the same choice by Morpheus, chooses the red pill, and later regrets his choice.  That is because the red pill knocks you completely and totally out of your existing comfort zone.  The red pill is the harder oil to swallow, because it forces us to face reality and when we do, we have to choice but to change.

The red pill requires those who have made choices in regards to protecting organizational information assets to face management and tell them that the game has changed, so the protections must change.  That it is in fact no longer possible or practical to block every piece of malware, that attention must be paid to detecting what makes it through the shields and stopping those attacks as quickly as possible.  That the reports management gets about increasing numbers of antivirus detections are a mirage, because they reflect increasing attack volume but not the falling antivirus detection rates.  They don’t show just how much is actually getting through.  The advanced persistent threat – that is a problem for NASA and the NSA, not their organization.

For their part, management takes their own blue pill.  They are not on the cover of the New York Times with a public breach.  They don’t feel any intellectual property leaking out, they don’t feel personal information about customers and employees being secretly exfiltrated, and no one in accounting is telling them they have a cash shortfall because financial transactions were interrupted or intercepted.  Yet.

IT Security and management tacitly adopt a similar position.  They speak to the large embedded antivirus vendors that tell them they have it all covered and they want to believe it because admitting to the truth means hard decisions have to be made and even harder work will have to be done.  They are quietly playing a game of risk management hoping that they can get by in the blue pill stupor and never get an attack that erodes their customer’s confidence, forfeits their leadership in intellectual property, or degrades market valuation.

But the odds grow thinner as attacks increase in complexity and volume by the day.  Unfortunately the security markets does not help matters because the large, imbedded companies are the leading distributors of blue pills and don’t want their big cash cow customers suddenly getting a red pill dose of reality and looking at new, innovative solutions.

History has not been kind to cultures that choose the blue pill route.  Normally the awakening from the stupor only comes after some form of spectacular incident.  But it does not have to be so.  Innovative solutions exist today that will make the red pill much easier to swallow if organizations are willing to reach for the red pill.  If these organizations are willing to set aside some well established predispositions they may find that they can shut off as many or more of the old protections built for a threat environment that has long since passed by as they will have to add new and innovative protections to address the current and future threats.  Change always brings some level of discomfort, but in this case the change will be less painful than anticipated and certainly worth the effort.

Remember — all I am offering is the truth, nothing more.