Validation among insecurities
Industrial controls systems need a secure patch management approach, as threats can target outdated systems or careless networking errors. Industrial sector cyber security has a unique set of complexities that differ from business datacenter cyber security.
As media headlines have demonstrated, cyber security breaches across industries inflict heavy costs, loss of trust from customers, and damage to an organization’s credibility. In the past year, hackers stole 76 million records from financial services firm JP Morgan Chase and more than 80 million customer records from healthcare provider Anthem. The Target data breach cost $162 million in cyber security-related expenses across 2013 and 2014. While the financial costs related to a breach can be staggering for any industry, in the energy sector, where just an hour of downtime can correlate to millions of dollars in lost production, the potential implications can be extremely detrimental to business success.
Cyber security risk: Operations, reputation
Energy companies are accelerating implementation of full-scale cyber security programs and policies. In particular, oil and gas organizations, which are regulated differently than the power generation industry, face huge risks associated with industrial control system (ICS) vulnerabilities.
For example, an oil and gas company calculated that the failure of one of its control system’s human machine interfaces (HMIs) and the resulting downtime of two days would cost the organization an estimated $12 million in lost production. This calculation weighed the risks of waiting another year to install cyber security protection rather than immediate installation. Given the operational and reputational costs of not implementing security programs, there should be no delay.
In general, the energy industry relies on more outdated equipment than the commercial sector. In the business environment systems have updated software and hardware components needed to effectively combat today’s threats, which often target outdated systems or careless errors on the network. While the Sony breach raised greater awareness and concern about nation states as perpetrators of cyber threats, there are numerous vulnerabilities closer to home.
Patching ICS vulnerabilities
The root cause of many large-scale breaches is often an unpatched device lingering on the network and creating vulnerabilities for the entire organization. With an ICS, patching the network involves much more than just updating an app on an iPhone or updating software on a personal computer.
Patch management can be a time-intensive exercise, and to be truly effective it requires thorough operational testing. Validated patch management in an environment that mimics the plant environment helps energy organizations manage patches in an efficient and effective way to keep systems online and protected against the latest known vulnerabilities.
Do-it-yourself patch management
Approaches for executing patch management differ in the energy sector today.
"Self-performers" use anti-virus solutions and complete the process on their own time. If they are on a Microsoft operating system (OS), they rely on patches from Microsoft and its anti-virus provider, then test, validate, and deploy the updates themselves. Depending on plant size, this can be an onerous task. A plant with five computers on the control network requires manual implementation on each computer individually; some industrial facilities in certain regions have as many as 85 computers on the controls network. Not only is the process more time consuming when managing individually, it is also more of a liability for the organization as it introduces risk to the process.
Self-performers take 100% of the risk themselves to manage patches and ensure all computer systems are updated. If the patch unexpectedly requires system shutdown, the organization loses revenue due to unplanned downtime. Additionally, many "do-it-yourself" approaches still require a USB drive to transfer the patches to each computer system.
According to a recent study of security behaviors from Raytheon and the Ponemon Institute, more than half (52%) of survey respondents plugged in a USB device that was given to them by another person in the last three months. The potential for insider threats is a concern that reverberates through the oil and gas and power generation sectors. Using someone else’s USB or leaving it in an unsecure location may allow internal actors to manipulate the portable device. Connecting a compromised USB device to the controls network could introduce malware to the network and wreak havoc on the system and operations.
When organizations rely on trusted security services and providers who take all necessary precautions to secure their supply chain through update delivery, they substantially mitigate risks and transfer liability away from the self-performer.
Validated patch management
Organizations can select a packaged solution or a comprehensive service when choosing a third-party patch management approach.
With a packaged solution, a vendor gathers the necessary patches for a plant facility based on the operating system (OS) software and hardware and shares the packaged updates with plant operators to implement themselves. This option reduces the work involved in identifying patches and combining them for one monthly update. However, the organization is still responsible for testing these patches and mitigating any adverse impacts prior to deployment.
Untested patches are unreliable in the ICS environment because, without testing, operators cannot predict how a particular system will respond to the updates in its "as is" state. In 2014, 10 of 12 OS patches issued for ICSs had to be modified for the live industrial environment to avoid complications, downtime, or vulnerabilities. With validated patch management, the patch is run in a virtual environment on-site or a lab environment that mimics the plant environment to identify any incompatibilities that may exist before the patch is applied. This allows operators to determine what alterations need to be made to ensure uptime and protection against cyber security threats.
Virtual testing with vendor training is more effective than not testing, but it is limited in its capability to fully address all factors within the plant environment. A secure external lab environment that has the physical hardware and software in place for testing is the best method to guarantee industrial controls receive tailored patches monthly. This process helps keep ICSs secure and up-to-date so malware and malicious attacks cannot manipulate system vulnerabilities.
Failed patch strategies
Energy companies already have felt the repercussions of failed patch management strategies. In one energy company, an HMI was using an obsolete OS no longer supported by vendors for continued patches. Malware was introduced on this system, which, in turn, affected production. In addition to the lost production and forced downtime, the company had to launch an investigation to identify the cause of the shutdown, using more resources and lowering productivity. To avoid this scenario, organizations should regularly patch systems using a validated patch management system, while also managing outdated hardware and software; proactive lifecycle management can avoid costly forced downtime.
Securing connected machines
As more technology is introduced into the plant environment, there is greater need to validate secure configurations and patches to maintain a reliable control system network. Securing connected machines in the industrial sector has a unique set of complexities that are very different from those related to protecting a business data center.
Additionally, industrial organizations must understand the challenges of and differences between information technology (IT) and operational technology (OT) to execute a strong cyber security strategy. The world of OT security needs to be foundationally different from traditional IT detection systems in existence today. As with other industries, ICS cyber security will likely transition into a managed service practice rather than an in-house or vendor-generated solution. As energy organizations place greater emphasis on cyber security, they must remember that successful patch management is the foundation for a secure and productive enterprise.
– Dana Pasquali is a product management leader at GE Measurement & Control, a division of GE Oil & Gas; edited by Mark T. Hoske, content manager, Control Engineering, firstname.lastname@example.org.
- Industrial control systems (ICS) need a different approach to patch management than IT systems.
- Cyber security risk assessment should include cost of operational downtime and to reputation.
- A validated patch management system while managing outdated hardware and software can be part of a proactive lifecycle management plan to avoid costly forced downtime.
Are you waiting for an industrial control system cyber security incident before seeking a budget to match the risk?
In this online version of this April Control Engineering article, see links to additional, related coverage below.