A Field Study of Digital Forensics of Intrusions in the Electrical Power Grid Eli Sohl, Curtis Fielding, Tyler Hanlon, and Julian Rrushi
Hassan Farhangi, Clay Howey, Kelly Carmichael, Joey Dabell
Department of Computer Science Western Washington University 516 High Street Bellingham, WA, 98225, United States
Department of Applied Research British Columbia Institute of Technology 3700 Willingdon Avenue British Columbia, V5G 3H2, Canada
{sohle, fieldic, hanlont2}@students.wwu.edu
[email protected] ABSTRACT The paper describes the findings of a field study that we conducted to practically determine the digital forensics processes, tools, and technical considerations that apply to investigations of computer intrusions into the electrical power grid. Our findings are based on technical interactions with industry and law enforcement, as well as on actual applied research in the lab. In the paper, we discuss the most widespread vulnerabilities of industrial control systems, and thus build on those preliminaries to describe the inner workings of control system forensics in the real world. Several advanced works on control systems forensics abstract from lowlevel fundamentals, assuming that those are easily doable. The main contribution made by this paper is the focus on those fundamentals, with the goal of laying the foundation for further developments of practical and hence usable digital forensics in the electrical power grid.
Categories and Subject Descriptors D.4.6 [Operating Systems]: Security and Protection
General Terms Security
Keywords Industrial control systems; digital forensics; vulnerabilities and exploits
1.
INTRODUCTION
We performed a field study of the specific digital forensics thought processes and tools that are currently used by digital forensics examiners in industry and the law enforcement
c 2015 Association for Computing Machinery, Inc.
ACM acknowledges that this contribution was co-authored by an affiliate of the Canadian National Government. As such, the Crown in Right of Canada retains an equal interest in the copyright. Reprint requests should be forwarded to ACM, and reprints must include clear attribution to ACM and Defence R&D Canada. CPS-SPC’15, October 16, 2015, Denver, Colorado, USA. c 2015 ACM. ISBN 978-1-4503-3827-1/15/10 ...$15.00.
DOI: http://dx.doi.org/10.1145/2808705.2808716.
{Hassan_Farhangi, Clay_Howey, Kelly_Carmichael, Joey_Dabell}@bcit.ca
to investigate attacks and intrusions into the electrical power grid. We worked alongside several digital forensics industry, and also had several fruitful interactions with law enforcement. One of the question that we asked is the following: what digital forensics frameworks and tools would you use if a report of a cyber incident in the electrical substations and the smart grid came in at this moment ? We received generous contributions and indicators in response. In the paper, we describe the main steps and the thought processes that law enforcement and industry go through when investigating a cyber incident in the electrical power grid. We then pinpoint some of the digital forensics tools that are used in those thought processes, and also describe the salient characteristics of those tools. Those digital forensics tools are made by a variety of vendors, from research divisions to open source developers, to commercial industry. A major part of the findings described in this paper were obtained through practical work done in the lab. We had several controllers available, and thus analyzed concrete vulnerabilities and forensics tools. The contribution of this paper is structured as follows. Description at a low level of technicality of the vulnerabilities that affect control systems in the power grid, in Section 2. Our work in the lab found that controllers are much easier to exploit than current general-purpose computing machines, since most controllers do not have any hardening at all. The thought processes that govern investigations of intrusions into the electrical power grid are discussed in Section 3. Description of the typical evidence that is left behind by cyber intrusions in the electrical power grid is given in Section 4. A discussion of the digital forensics tools that were found in practice to work on control systems, with varying degrees of applicability, is given in Section 5. We discuss some of the related work in Section 6, and conclude the paper in Section 7. The source of public information on vulnerabilities that affect industrial control systems in electrical substations and the smart grid was the ICS-CERT [17].
2. 2.1
LOW-LEVEL VULNERABILITIES IN CONTROL SYSTEMS Memory Errors and their Reverse Engineering
A large portion of system vulnerabilities that affect industrial control systems in electrical substations and the smart grid pertains to code that suffers from memory errors. The industrial control systems that are affected by memory errors include smart meters and relays in the advanced metering infrastructure, as badly as intelligent electronic devices (IEDs), bay controllers, and station controllers in electrical substations. The security posture of the code of most of the legacy control systems is comparable to that of general-purpose computing in the 90’s. That is a major root cause of computer intrusions into the power grid, which is there to stay, taking into account the decades-long lifespan of control systems. Modern control systems are affected as well, although to varying degrees. Some control system vendors are revamping their cybersecurity capabilities, which may address memory vulnerabilities in their products. Siemens, for example, is adding cybersecurity mechanisms to their power systems, in addition to establishing an industrial control system cyber emergency response team. Another example is Emerson, which is integrating cybersecurity measures in their new control system designs. One of the key characteristics of a memory vulnerability is a channeling problem, which allows input data to force vulnerable code to attempt to write an arbitrary value to an arbitrary address. The input data are often received over the network. In a typical exploit, the arbitrary value is a memory address. The memory address where that value is written refers to a memory location that stores the so-called control data. Control data include saved instruction pointers, i.e., return addresses, function pointers, or any other pointers that can redirect the execution flow of the code. The memory address that is written in that memory location may refer to another memory location, which stores the beginning of attacker-provided data. Those data may be machine executable code, and are commonly referred to as shellcode in hacker nomenclature. The shellcode typically spawns a shell, or creates a server socket which spawns a shell upon receiving a connection. The alternatives are really left to the hacker’s creativity. The shellcode is typically stored on an executable memory segment, which includes the stack and the heap. Table 1 provides examples of public cases of stack overflow vulnerabilities in control systems deployed in electrical substations and the smart grid. Current general-purpose operating systems set the stack to nonexecutable by default. Our experience however suggests that is not the case of many legacy control system brands. A variant of the exploit leverages return oriented programming (ROP). Instead of forcing vulnerable code to take the execution flow to shellcode stored somewhere in memory, the attacker manipulates the stack such as to take the execution flow to existing blocks of instructions. Since the stack is changed, one block of existing instructions returns to another block of existing instructions chosen by the attacker. The various existing blocks of instructions reached
Vendor
Rockwell Automation
Schneider Electric
ABB Yokogawa
Control System Product ControlLogix, CompactLogix, GuardLogix, SoftLogix, MicroLogix 1100 and 1400 TwidoSuite, PowerSuite, PL7, SFT2841, etc. ABB AC500
Affected Component
CVE
EtherNet/IP
CVE2012-6438, CVE2012-6436
Modbus
CVE2013-0662
Web server
CENTUM, Exaopc, B/M9000CS, B/M9000 VP, etc.
Proprietary protocol
CVE2011-5007 CVE2014-0783, CVE2014-0784, CVE2014-0782
Table 1: Public cases of stack overflows in control systems Vendor
Schneider Electric Sunway Mitsubishi Electric Automation Control Microsystems
Control System Product Accutech Manager pNetPower, ForceControl MX, CitectFacilities, CitectSCADA
Affected Component
CVE
Proprietary protocol Web server
CVE2013-0658 None assigned CVE2013-3075
ClearSCADA
DNP3, IEC 60870, and Modbus
Proprietary protocol
None assigned
Table 2: Public cases of heap overflows in control systems that way produce the behavior that the attacker would have otherwise attained through crafted shellcode. An exploit of a memory vulnerability may overwrite non-control data, i.e., pure data, as well. Typical pure data of interest include cryptographic keys, controller or human-machine interface (HMI) software passwords, and generally any data that are used in conditional statements and that can take the execution flow to authenticated code. Table 2 provides examples of public cases of heap overflow vulnerabilities in control systems deployed in electrical substations and the smart grid.
Integer overflows appear to be widespread in control systems. An integer overflow is not exploitable directly as no buffers are overwritten. In other words, there is no actual memory overflow. Nevertheless, integer overflows can be serious vulnerabilities in that can cause other kinds of actual
Vendor
Rockwell Automation
General Electric
Trihedral Engineering ICONICS
Control System Product FactoryTalk
Affected Component
CVE
Proprietary protocol
PROFICY HMI/SCADA CIMPLICITY VTScada
Web server
CVE2012-4713, CVE2012-4714, CVE2013-2805, CVE2013-2807, CVE2013-2806 CVE2012-4689
Proprietary protocol
CVE2014-9192
GENESIS
Web
None assigned
Table 3: Public cases of integer overflows in control systems buffer overflows. Erroneous integers can lead to buffer overflows in memory operations such as the following: 1. The overflowed integer is used as the size of a destination buffer 2. The overflowed integer is used to determine how far into an array to index 3. The overflowed integer is used to check whether the amount of bytes in a source buffer is safe to copy into a destination buffer Table 3 provides examples of public cases of integer overflow vulnerabilities in control systems deployed in electrical substations and the smart grid.
While the exploit techniques discussed so far pertain to the Intel family of central processing units (CPU), the underlying concepts apply similarly to other CPU architectures. Intel CPUs are widespread amongst control systems. The M1 controller from Bachmann Electronics [1], for example, runs on an Intel x86 CPU, and so does the OCR-400 control system from Emerson Process Management. Other kinds of embedded CPUs such as ARM and PowerPC are widespread as well. The authors of this paper had discovered and analyzed numerous memory vulnerabilities in control systems during his tenure with industry. They had ethically developed proof of concept exploits of most of those vulnerabilities, mostly on the VxWorks [6] and Windows CE [3] operating systems. While that vulnerability research work was conducted under nondisclosure agreements with control system vendors in North America, some of the technical details can be anonymously recalled and correlated to the work done in this project. The electric power industry is moving fast to deploy cryptographic solutions that provide for confidentiality, integrity,
and authentication of network communications. With regard to electrical substations, the deployment of cryptographic modules is more challenging due to hard real-time constraints. At any case, some of the memory vulnerabilities in control systems can be exploited to read from arbitrary memory locations. Exploits of this kind can read cryptographic material such as premaster and master secrets, as well as encryption keys and passwords, from the memory of a control system. The are cryptographic solutions such as SEL-3025 from Schweitzer Engineering Labs [7] that are implemented as external stand-alone modules rather than as software or firmware that runs on a control system. Exploits of memory vulnerabilities apply to those solutions as well, although the attack surface differs from product to product. An approach that is commonly used by the hacker community to discover and analyze vulnerabilities in control systems consists of extracting the firmware from their flash memory and reverse engineering it. Firmware extraction is often done by accessing the flash memory chip through the Joint Test Action Group (JTAG) port on the motherboard of the control system [2]. Common other intended uses of a JTAG port include testing printed circuit boards and debugging firmware. We point out, however, that it is a common practice in the electric power industry to disconnect the JTAG pins and thus erase the JTAG interface on control systems prior to putting those controllers into production. To evade that layer of defense, the hacker community commonly uses a digital multi-meter to pinpoint the JTAG pins, and thus solder a hand–made JTAG interface. It should be noted that the authors of this paper have in fact worked on control systems in production that have the JTAG pins enabled. In such cases, firmware extraction does not require any hardware instrumentation. Device programmer is also a frequently used tool by the hacker community. A device programmer can extract the contents of flash memory chips as well as those of the main memory chips. Once the firmware is extracted from flash memory, hackers typically analyze it with tools such as Binwalk [8]. The analysis includes finding and extracting executables and data files, finding and extracting compressed contents, identifying opcodes, and performing data entropy analysis. The extracted executables are analyzed further, with the goal of discovering blocks of code that are vulnerable to memory bugs. The hacker community uses tools such as IDA Pro [9] to analyze the firmware instructions. The code is analyzed statically with the help of the graph modeling capability of IDA Pro. Static analysis of firmware is, at times, combined with dynamic analysis, if a debugger is available on the control system. The kind of debugger that is commonly used to perform dynamic analysis of controller firmware is a JTAG debugger [4], such as ICE 2 by Windriver [5]. Software debuggers are found infrequently on control systems. The authors of this paper had utilized a software debugger to analyze the firmware of a control system that was running VxWorks. That however was a limited case. The exploitation of a memory vulnerability can cause a denial of service on a control system as well. That is when the vulnerable code does not reside in a child process forked off a parent process. If it is a child process to be disrupted, the crash is only a transient failure, which leaves the parent process intact. Other connections would be accepted and handled as if the transient failure never happened. On the other
hand, the authors of this paper had analyzed memory vulnerabilities which, when exploited, corrupted memory used by the kernel. As a result, the entire control system would become entirely dysfunctional. The LEDs on the control system panel were blinking to indicate severe system error. A system reboot was required to bring the control system back to functional. Thus, denial of service conditions in a control system caused by exploitations of memory vulnerabilities can range from temporary disruption to a persistent system failure.
Vendor
Siemens
Control System Product S7-300
Emerson
DeltaV
Schneider Electric
Quantum Ethernet Module S7-400
Siemens
2.1.1
Assessment of the level of hardening on control systems
We performed an assessment on a sample of control systems to determine whether or not their operating systems are hardened with respect to the vulnerabilities described earlier in this section. Our key finding is that controllers are much easier to exploit than current general-purpose computing machines. We found that most controllers do not have any hardening at all. Address space randomization is missing. That is an import defense mechanism which randomizes the base address of each process (or task in VxWorks). The attacker would have difficulties getting the addresses all right in an exploit. Stack protection mechanisms such as StackGuard are absent as well. StackGuard places a random word, i.e., a canary, right before the saved frame pointer and the saved instruction pointer. If the overflow of a buffer stored on stack occurs, the canary would get overwritten before the overflow overwrites the saved control data. The code checks the integrity of the canary prior to following the saved instruction pointer. If the canary results to have been changed, the code simply raises an error and thus avoids loading the saved instruction pointer onto the program counter register. As we wrote earlier in this paper, the stack is executable. An executable stack is a flaw of its own, in that a nonexecutable stack can prevent shellcode injected onto the stack from executing. Heap protection mechanisms such as HeapShield are missing as well. HeapShield introduces an alternative to freelist-based chunk allocation, and thus makes heap overflows harder to exploit. No format bug prevention mechanism such as FormatGuard is deployed on controllers. FormatGuard extracts the count of the actual arguments to a format function, which is then passed to a safe format function wrapper. Unlike controllers, control system technology that runs on general-purpose operating systems such as Windows benefits from most of the hardening mechanisms that come with it. Windows is hardened by default, and thus can protect industrial control applications. Mounting an exploit against such applications involves the evasion of the defense mechanisms, in addition to exploiting the target vulnerabilities.
2.2
Hardcoded Credentials
Several control systems have default passwords or cryptographic keys hardcoded in their firmware. Those credentials are trivial to identify and extract. The common approach used by the hacker community is similar to the one used to extract firmware to search for memory vulnerabilities. Once firmware is available, an attacker can look at the strings throughout the firmware image. Passwords will be in there quite visible. To identify cryptographic keys, an attacker searches for sequences of bytes with a high level of entropy.
Affected Component Controller firmware Controller firmware Controller firmware
CVE
Controller firmware
None assigned
None assigned CVE2014-2350 None assigned
Table 4: Public cases of hardcoded credentials in control systems Binwalk is one of the tools that are used for that purpose. Table 4 provides examples of public cases of hardcoded credentials in control systems deployed in electrical substations and the smart grid. All control systems in the electrical power grid, including smart meters and relays, IEDs, bay controllers, and station controllers, are equally affected.
2.3
ActiveX and CSS Vulnerabilities
ActiveX is software developed by Microsoft. It is commonly used by Internet Explorer to build rich controls. ActiveX controls can be invoked through client-side scripts written in a language such as Javascript or VBScript. ActiveX controls are susceptible to the memory vulnerabilities described earlier in paper. A vulnerability in an ActiveX control is triggered by calling vulnerable methods with possibly malicious parameters from within a client-side script. The shellcode is injected via the client-side script as well. The exploit is delivered when a target system operator visits a malicious web site. No other interaction is required by the user for the attack to work. A real world vulnerability in an ActiveX control is the one that affects MICROSYS PROMOTIC. The vulnerability resides in an ActiveX method called GetPromoticSite(), which the attacker invokes via Javascript to cause a null pointer, as shown in [10]. Table 5 provides examples of public ActiveX vulnerabilities in control technology deployed in electrical substations and the smart grid. A vulnerability of this kind affects engineering workstations and laptops used by system operators. The attack compromises those systems first, and then leverages them as a bridge towards smart meters and relays in the advanced metering infrastructure, or towards IEDs, bay controllers, and station controllers in electrical substations.
Cross-site scripting (CSS) vulnerabilities occur when a web application gathers unsanitized or insufficiently validated data from users, which can then be displayed to other users. Like ActiveX vulnerabilities, CSS vulnerabilities are exploited via malicious scripts written in a client-side language such as Javascript. Those attacks exploit the trust that system operators place on a web application, and typically intercept web authentication cookies, which are then used to access controllers or web-based HMI systems. Several CSS vulnerabilities in control systems are public. We will have to omit examples in this paper due to lack of space.
Vendor
Honeywell Rockwell Automation Advantech Siemens
Control System Product ScanServer
Affected Component ActiveX
CCW
ActiveX
Broadwin WebAccess Tecnomatix FactoryLink
ActiveX ActiveX
CVE
None assigned CVE2014-5424 None assigned CVE2011-4055, CVE2011-4056
Table 5: Public cases of ActiveX vulnerabilities in control systems
2.4
Supply Chain Attacks
Due to economic factors, a large number of fabrication facilities, which produce field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), integrated circuits in general, have been migrating offshore. There is a supply chain threat that has been known to the cybersecurity community for quite a few years now, namely the possibility that adversaries maliciously manipulate computer chips produced and assembled in offshore foundries and packaging facilities, respectively. Malicious manipulation of computer chips includes insertion of backdoor features into chip designs, weakening or total neutralization of security mechanisms, insertion of chip primitives carrying resident malicious code such as worms and trojan horses, etc. Since computer chips are widely used in control systems, adversaries could cause disruption of our electrical power grid. There is also a known possibility that hardware chips get intercepted and replaced with trojanized versions prior to the control system equipment being delivered to a power utility. Similar threats apply to the software and firmware that runs on control systems. The validation of entire software and firmware packages is seldom done. The trustworthiness of such packages relies on the amount of trust that power companies place on the code developers and software companies. Clearly the threat of inserting backdoor functionalities into the code that runs on control systems is feasible and real. That forms the basis of our approach in this project, which is to develop a digital forensics capability that works under the assumption that a control system might have been compromised from the moment it was deployed on the field and connected to the power equipment. We are not aware of public cases of supply chain attacks at the hardware or software/firmware level against our electrical power grid. Nevertheless, it is worth emphasizing in this paper that there are several weak points in the supply chain of our electrical power grid that could be exploited maliciously. Malicious integrated circuits or trojanized firmware or software enable an attacker to take over entire segments of the advanced metering infrastructure by the means of smart grid worms. An entire electrical substation could fall as badly. Supply chain attacks typically require the involvement of nation state actors, and as of this writing are not generally made public.
2.5
Weak or Absent Cryptographic Protection
We found that several control system vendors are developing and deploying cryptographic protections of power grid communications. Nevertheless, numerous segments of the electrical power grid are known not to have any cryptographic protection at all. Old protocols such as DNP3 communicate in plaintext. Commands and replies are entirely unauthenticated, which opens that way to the socalled poisoning attacks. Such attacks spoof sensor data in order to make the control algorithms on control systems take actions that could cause physical damage to the power equipment. Furthermore, commands such as those that trip circuit breakers could be spoofed and hence cause physical damage to power equipment as bad. Yet another obvious, but dangerous, vulnerability is the lack of authentication on control systems with regard to accepting ladder logic or code in any of the other IEC 61131-3 languages. Any network misconfigurations or other vulnerabilities that enable an attacker to get into a substation or smart grid network would clearly result in the possibility of the attacker uploading logic into the control systems, and thus controlling the power grid equipment.
3.
DIGITAL FORENSICS THOUGHT PROCESSES
The digital forensics operation is divided into two main phases, namely data acquisition from a compromised control system, and digital forensics processing of those data. With regard to field devices, i.e., programmable logic controllers (PLCs), intelligent electronic devices (IEDs), phasor measurement units, and smart meters, typically the data processing is not done on the device itself. The acquisition of data from a control system is firstly attempted by running code on the control system to do the so-called memory imaging. There appear to be no memory imaging tools that are specific to control systems as of this writing. Digital forensic analysts typically attempt to adapt Linux-based tools such as LiME when working with Unix-based control systems, including those that run VxWorks. One of the tricks seems to be the static compilation of the source code, which produces an executable that has no reliance on shared libraries. Digital forensic analysts attempt to upload and run the command line version of the Forensic Toolkit (FTK) imager on control systems that run Windows CE or embedded. The real-time operating systems that run on control systems are typically highly customized. The technical details of those customizations are vendor specific, and vendors hardly provide those details to the digital forensics analyst, especially if the analyst is with industry. Because of the various customizations, the memory imaging tools often fail to run on the control system. Furthermore, uploading a memory imaging tool to a control system is not always a feasible task. Some control systems do not provide any interfaces to their operating system, besides a utility for the upgrade of existing firmware modules. Consequently, a frequently used option consists of accessing the hardware such to retrieve the contents of main memory and flash memory. The access to hardware is often attempted via the JTAG interface. Flash memory chips are often mapped into specific addresses of the main memory, therefore are easy to read. So is main memory once the access to the JTAG pins is attained. The tools that are commonly used to do the
memory imaging over JTAG pins include ICE 2, UrJTAG, and OpenOCD. The challenge here is that control system vendors often remove the JTAG interface, or entirely remove the JTAG pins prior to putting the control system in production. The reason is that hackers often use the JTAG pins to dump the control system firmware, which they reverse engineer to discover vulnerabilities. The absence of a JTAG interface is not so much of a problem since using a digital multi-meter along with the electrical specification of the CPU often suffices to locate the JTAG pins. A handmade JTAG interface is then soldered on the motherboard of the control system. The complete disabling of JTAG pins can however be quite challenging. If the JTAG pins are not usable to do memory imaging, an alternative is any universal asynchronous receiver/transmitter (UART) port on the control system hardware. The capabilities provided by an UART port are vendor specific. Some allow for complete access to main memory and flash memory chips. Others provide only a limited set of functions, which do not allow for reading memory. An in circuit serial programming (ICSP) interface, if present on the control system hardware, may also be used to read the contents of memory. If neither an UART port nor an ICSP port is usable to do memory imaging, then the final chance is the so-called device programmer. An example of a device programmer that is used to do memory imaging over hardware buses, which is otherwise referred to as snarfing, is the Dataman 848Pro2 Super Fast Stand-Alone Programmer. We did not get indications of specific device programmers that are the most frequently used in digital forensics operations in the electrical power grid. Virtually all device programmers are used for that purpose. The device programmer is attached to the outer pins of the memory chip, and operates over the standard 1149.1 buses. It uses data de-serialization algorithms to minimize the number of scans. The support by device programmers for most memory chips in existence does not appear to be an issue. The main limitation of the first phase of a digital forensics operation in the electrical power grid defined as above, is that if the intrusion hits the actual power equipment through the compromised control systems, the latter are typically damaged. The degree of damage may include burning of the control system hardware, and thus may prohibit any memory imaging. Once the memory imaging is done, the preprocessing tool of choice with regard to main memory analysis appears to be Volatility. This tool is comprised of plugins that can dissect the memory image such as to extract executable files, data files, kernel modules, operating system data structures, etc. Volatility is able to process mainly Windows and Linux-based memory images. All the other control systems require separate tools, which we have not been able to identify in this project, assuming those tools exist. The contents of flash memory are often organized into a file system, therefore are easy to identify and access. The control system applications that run on general-purpose computers do not represent any of the challenges described above. Most of those computers are Windows machines, and thus can be analyzed through digital forensics tools such as EnCase and Forensics Toolkit (FTK). Those tools come with their own memory imaging components, and thus can do the forensics data acquisition in addition to forensics data processing.
We now discuss the typical evidence that is left behind by cyber intrusions in the electrical power grid.
4.
EVIDENCE BEING SOUGHT
Exploitations of memory vulnerabilities in the embedded operating system of an industrial control system may leave several types of evidence on the compromised machine, including the following: • Injected shellcode • Rogue OS processes • Additional malware code • Code injected into the address space of existing OS processes • Rogue data, such as malware configuration files • Modifications to various kinds of data on the industrial control system • New client or server sockets • File creation and file access data The shellcode is injected in the heap, in the stack in certain cases, and in any other data section that is executable. Injected shellcode is the primary instrument of gaining access into the industrial control system in the first stages of the attack. The forensics analyst can locate and identify shellcode by extracting and analyzing the RAM of the compromised industrial control system. The contents of RAM are first analyzed to define the boundaries of each OS process that was loaded in memory at the time the image was taken. Within each OS process then, the forensics analyst dissects the executable image to identify the stack, heap, and data sections. Those regions of memory are then searched for signs of shellcode. Shellcode is typically executable code, but could be a sequence of addresses as well in the case the return oriented programming (ROP) technique was applied to run code taken selectively from the existing instructions of the vulnerable OS process. Injected shellcode may be stored on the flash memory of the industrial control system as well. That is due to the swapping out of memory pages of OS processes by the memory scheduler. If the compromised OS process is swapped out, its data structures along with the injected shellcode are placed on the flash memory. Thus, when searching for injected shellcode, the forensics analyst extracts and analyzes the contents of the flash memory as well. Some exploits might spawn new OS processes on the compromised industrial control system. The forensics analyst examines the contents of RAM in order to identify malicious OS processes, and the flash memory as well just in case a malicious OS process was swapped out. Some exploits download additional malware executable files to perform more complex operations on the compromised industrial control system. The forensics analyst can identify such malware modules by extracting the contents of the flash memory and analyzing the file system. If those malware executable files are running, they will be visible as malicious processes in RAM or flash, and thus can be identified as we wrote earlier in this section. Some cyber attacks are crafted such as to inject
malware modules into the address space of existing OS processes on the compromised industrial control system, rather than deploying that malware as separate executable files and hence processes. The forensics analyst can identify those code injections by dissecting the images of legitimate processes, and searching for machine code in all data sections that were originally executable. Some malware operate in accordance with configuration files, which are downloaded on the compromised industrial control system along with the malware code. Those malware configuration files may be stored directly on the flash memory, or be mapped into the address space of a malicious or legitimate OS process. The forensics analyst searches for those files by examining the file system that is stored on the flash memory, or by dissecting OS processes taken from RAM or flash memory. Malware may cause malicious modifications to various data on the compromised industrial control system. An example is the overwriting of sensor data, which form the basis of the operator’s understanding of the state of the power system. The forensics analyst can search for discrepancies or inconsistencies in data by extracting them from the flash memory. The act of gaining control of an industrial control system over the network, by definition, leaves behind socket data structures. Those are found in RAM, and can be identified by reverse engineering the various data structures of the OS kernel. The forensics analyst can also link socket data structures to OS processes that were making use of them, and hence following the leads to identify malicious OS processes in memory. In addition to configuration files, malware can create other data files on the compromised industrial control system. The forensics analyst may extract those files from flash memory, and thus apply file signatures to assess whether or not those files can be recognized and were originally on the industrial control system. Since file handles reside in RAM as well, the forensics analyst can determine which files a malicious process was accessing, and in which mode. An exploitation of hardcoded credentials leaves behind socket data structures, which are identical to those created by legitimate network access to the industrial control system and hence challenging to differentiate. Those socket data structures, however, are kept in RAM for the duration of the network session. Once the attacker retreats, those data are lost. The forensics analyst is required to take dumps of RAM memory at regular intervals for being able to catch those socket data structures. Proper logging and auditing could inform the forensics analyst as well. In their current state, however, the industrial control systems that operate the power grid have limited or at times entirely absent logging capabilities. Exploitations of ActiveX and cross site scripting vulnerabilities target the operator’s browser so that to intercept authentication material from web-based HMI tools. The evidence that those attacks leave behind on an industrial control system is similar to that of attacks that exploit hardcoded credentials. Exploitations of the compression side channel intercept authentication material as well, namely cookies and in some cases passwords. Since such exploitations rely on landing malicious scripts on a system operator’s web browser, in terms of evidence those are similar to the other attacks discussed in this paragraph. Supply chain attacks on industrial control systems may leave behind malicious processes and/or kernel threads,
which in turn may create network sockets, download or upload files, modify data, or access existing files. The forensics analyst may analyze all those data by extracting and dissecting the contents of RAM and flash memory. Exploitations of the lack of cryptographic protection vary from injecting power system commands into industrial control systems, to uploading logic code on them. In their current state, some of the industrial control systems that are deployed in the power grid may record some of the actions that they take on the power system. An example is a log entry that indicates that a circuit breaker was tripped at a given time. The forensics analyst may retrieve such logs by dumping and dissecting the contents of the flash memory. Such analyses typically navigate the file system, however raw data on flash may be found and analyzed as well. Almost all of the attacks on an industrial control system may upload malicious logic files, or overwrite existing logic files. Dumping RAM and flash contents enables a forensics analyst to locate and extract logic files. The analysis of the logic code itself is a separate operation, which follows the digital forensics process. We now discuss the main tools that are used to implement the thought processes of a digital forensics operation as conducted in the electrical power grid.
5.
APPLICABLE TOOLS
LiME is a Loadable Kernel Module (LKM) that dumps the contents of volatile memory from Linux and Linux-based devices [18]. Data can be written either to memory or else over a network, the latter option being very useful when dealing with specialized hardware. The forensics analyst can use LiME to extract and analyze evidence that is stored in RAM, including OS processes, injected code, modified data structures, files created and files accessed in reading or writing, socket data structures, and mapped files. Being a forensics tool that is tightly coupled with the kernel of Linux, LiME’s use may be limited to industrial control systems that run embedded Linux, and could also be extended to VxWorks. FTK imager is a standalone memory imaging program included in the more general FTK tool [19]. The FTK imager has the ability to dump memory in one single file or alternatively in several segments that can be reconstructed later on. The tool calculates the hash of data to validate their integrity before closing the files. The tool can recover deleted files, and can also export files and directories from the memory image. In addition, it allows for mounting an already created image. FTK imager comes in two forms, namely one as a command line utility, and one as a GUI application. Control system forensics typically involves attempting to run a tool of this kind on the control system. FTK could only run on control systems that run Windows CE or embedded. That is the reason for which the command line version is frequently considered in a digital forensics operation in the electrical power grid. FTK imager supports a variety of file systems. The FTK imager tool can be used by the forensics analyst to acquire the contents of RAM and flash, which he can dissect by the means of other tools, namely FTK [20] or EnCase Forensics [21], in search of the kind of evidence that we discussed earlier in this paper. The forensics analyst can determine whether FTK imager is applicable to the investigation of an intrusion in the power grid based on the operating system that is running on the compromised
industrial control system, and on whether there are ways to run the tool on the compromised industrial control system. FTK imager can be thought of as the Windows embedded counterpart of the Linux LiME forensics tool. Volatility is a digital forensics framework that analyzes dumps of main memory from Windows and Linux-based machines, including control systems [22]. The tool as a modular design, which makes it suitable for future plugins designed especially for control systems. The tool is written in Python, and has scriptable APIs. The folk wisdom is that no other digital forensics framework can perform tasks that Volatility cannot or cannot be programmed to perform. Because of that reason, we see Volatility as a framework that is viable to be extended to cover control systems to detail. Currently the tool is generally usable in only a limited subset of control system forensics cases. Some of the analytical functions of the Volatility framework are the following: • Identify the type of system that the memory image came from • Identify and locate data structures, executables, data • List the processes or tasks of the system using multiple techniques • Display the shared libraries loaded by a process or task • Extract a shared library from the main memory image • Display open file handles • Enumerate imported and exported functions of a process or task • List the kernel drivers loaded into the system • Scan memory for symbolic link objects • List active network connections • Detect listening sockets In addition to serving as a digital forensics framework, Volatility is quite usable as a malware analysis tool. That is yet another aspect that could be extended to analyze control system malware. The forensics analyst can use Volatility in conjunction with the LiME tool. As we wrote earlier in this paper, LiME only dumps the contents of RAM memory of an industrial control system. LiME does not have any capabilities in dissecting and analyzing those memory contents. It is Volatility that can be used to locate and extract the evidence from the memory dump produced by LiME. Dshell is a highly extensible framework for packet capture analysis, written almost entirely in Python [23]. It provides a flexible command-line interface to its functionality through a standard, POSIX-compliant shell. Dshell is designed with a focus on modularity. Dshell commands take packet-capture files (including pcap and pcapng formats) as input, and can perform various analyses on them. Its extensibility makes it a powerful tool in a wide range of situations, including those which require the ability to rapidly develop very specific sets of tools. It is possible to identify and trace information at any level on the networking technology stack, finding all SYN packets which received an RST/ACK response, for example, or extracting all files
transferred during an HTTP session, or identifying all connections established across the course of the packet capture, or grep-ing for a given string within messages’ data sections. Dshell refers to each of its utilities as a decoder, and one of its main selling points is the ability to chain decoders, which is a very powerful tool when working with large datasets. The forensics analyst can use Dshell to search for evidence in live network packets that flow in a smart meter network or electrical substation network. The evidence that is left behind by the use of hardcoded credentials, the use of credentials captured by ActiveX or cross site scripting attacks, or sometimes by compromised supply chains, appears to be within reach to Dshell. Redline provides a comprehensive suite of utilities for analyzing Windows-based systems [24]. It is capable of performing forensics analysis both in live situations and on previously obtained capture files. For obtaining capture files for later analysis, Redline provides three different types of what it calls Collectors. These are scripts which acquire the same forensics data that Redline would analyze live, but which save it to file instead of performing analysis on it. These files can later be imported into Redline for analysis. It is possible to configure and customize the collectors so that they can perform better in specialized situations. The forensics analyst uses the collector programs to dump the contents of RAM and flash memory. Thus, those collector programs are similar to the FTK imager and LiME forensics tools in terms of provided functionality. The forensics analyst can then use the Redline tool to dissect the memory image produced by the collector programs, and thus search for the kind of evidence that we discussed earlier in this paper. EnCase is a digital forensics tool that places its focus on flexibility, simplicity, and ease of use more than on advanced functionality. It appears to be designed for maximum marketability to private investigators with limited technical competence. Its features include the ability to find image files and run keyword searches. They do appear to have certain powerful features for picking out and searching over different types of information, for example identifying all saved e-mail messages and running a keyword search only across those. One curiosity-piquing feature is the EnCase Decryption Suite (EDS), which enables the decryption of encrypted files and folders. It appears that EDS performs decryption based on known passwords, and it may crack any system which stores its passwords on disk. EnCase provides the functionality of distributing the analysis of retrieved forensic data. A forensics analyst can easily set up queues defining the order in which to give attention to different pieces of potential evidence on the computer, storage medium, or other device in question. This queue can be shared across multiple machines. In conclusion, the forensics analyst can use the EnCase Forensics tool to both collect dumps of RAM and flash memory from an industrial control system, and to dissect and analyze those memory contents. FTK offers digital forensics capabilities that are comparable to those of EnCase. FTK performs well in file system forensics. It can recover deleted files from a myriad of file system types, and also can work with multiple sources of evidence data. The tool enables a digital forensics analyst to work with static evidence as well as with remote evidence. The tool uses hashing, data carving, and can examine evidence in volume shadow copies. FTK can examine a dump of the
main memory as well, which is very useful in a control system forensics case. Its capability includes cluster analysis and data visualization. Overall, FTK is a very useful tool with regard to digital forensics of intrusions into electrical substations and the smart grid, if data acquisition from a compromised control system is successful.
6.
RELATED WORK
Volatile system memory represents a valuable medium for the collection of evidence of computer intrusion. A common operation in digital forensics is to dump at specific points in time the main memory for off-line analysis. It is like taking screenshots of the entire main memory contents, and then analyzing those contents for evidence. Petroni et al. have developed tools that extract data from a memory dump, and tools that aid a forensics examiner to analyze those data to discover evidence [11]. Breeuwsma explored in [2] the use of Joint Test Action Group (JTAG) ports to acquire the contents of flash memory of an embedded system. Common other uses of a JTAG port include testing printed circuit boards and debugging embedded code [4]. Kilpatrick et al. [12, 13] and Chandia et al. [14] designed a network forensics capability specifically for industrial control systems. The capability is comprised of software agents and a data warehouse. The software agents run at key points in the industrial communication network such as to capture network packets. The software agents preprocess network packets, and thus send the resulting data to the data warehouse. The data warehouse is based on a relational database along with query mechanisms, which support digital forensics of intrusions into industrial control systems. The authors’ prototype implementation is based on Modbus TCP, and involves two industrial controllers and one HMI server. Valli developed a set of rules for the SNORT intrusion detection system [15], which enables the tracing of exploits that leverage known vulnerabilities [16]. Valli collected vulnerability reports from CERT websites, control system vendor websites, etc., and thus examined vulnerabilities that affect industrial communication protocols such as Modbus and DNP3.
7.
CONCLUSIONS
The security posture of industrial control systems in the power grid appears to be much less mature than that of the general-purpose computing. The vulnerabilities that affect control systems bring memories of what we had in generalpurpose computing decades ago. We explored and assessed the thought processes, techniques, and tools that can be used to investigate exploitations of such vulnerabilities in the real world. Although we found no general framework that can be used to perform digital forensics of cyber attacks on the electrical power grid, the though processes, tools and techniques that we described in this paper are usable for that purpose. The degree to which control systems lend themselves to those tools varies from controller to controller. The main advantage of those tools is their rich functionality and coverage of most of the important categories of digital evidence in industrial control systems. Their main disadvantage is that they appear to be designed for operation in general-purpose computing, and are only adapted for operation in the special-purpose computing that is performed in the electrical power grid. Clearly
there is a need for digital forensics tools that can cope with the specific interfaces, networking, and operating systems of control systems.
8.
REFERENCES
[1] Bachmann electronic GmbH, Available online at http://www.bachmann.info [2] Breeuwsma, M.F. (2006) Forensic Imaging of Embedded Systems Using JTAG (boundary-scan), Digital Investigation, vol. 3, issue 1. [3] Windows Embedded, Available online at http://www.microsoft.com/windowsembedded/ en-us/products-solutions-overview.aspx [4] Stollon, N. (2010) On-Chip Instrumentation: Design and Debug for Systems on Chip, Springer. [5] Wind River ICE 2, Available online at http://www.windriver.com/products/ product-notes/wind-riverICE-2product-note.pdf [6] Industrial Profile for VxWorks, Available online at http://www.windriver.com/products/platforms/ industrialdevices/ [7] Secure industrial communications, Available online at https://www.selinc.com/securecommunications/ [8] Binwalk firmware analysis tool, Available online at http://binwalk.org [9] Interactive Disassembler, Available online at https://www.hex-rays.com/products/ida/ [10] Auriemma, L. ActiveX GetPromoticSite Unitialized Pointer, Available online at http: //aluigi.altervista.org/adv/promotic_2-adv.txt [11] Petroni, N.L., Walters, A., Fraser, T., and Arbaugh, W.A. Fatkit: A Framework for the Extraction and Analysis of Digital Forensic Data from Volatile System Memory, Digital Investigation, vol. 3, issue 4. [12] Kilpatrick, T., Gonzalez, J., Chandra, R., Papa, M., and Shenoi, S. An Architecture for SCADA Network Forensics, In International Federation for Information Processing, vol. 222, Advances in Digital Forensic II, pp. 273–285. [13] Kilpatrick, T., Gonzalez, J., Chandia, R., Papa, M., and Shenoi, S. Forensic Analysis of SCADA Systems and Networks, In International Journal of Security and Networks, Inderscience, vol. 3, issue 2, pp. 95–102. [14] Chandia, R., Gonzalez, J., Kilpatrick, T., Papa, M., and Shenoi, S. Security Strategies for SCADA Networks, In IFIP International Federation for Information Processing, vol. 253, Critical Infrastructure Protection, pp. 117–131. [15] SNORT, Available online at https://www.snort.org [16] Valli, C. SCADA Forensics with Snort IDS, In Proceedings of World Congress in Computer Science, Computer Engineering, and Applied Computing, pp. 618–621, Las Vegas, USA. [17] ICS-CERT, Available online at https://ics-cert.us-cert.gov [18] LiME, Available online at https://github.com/504ensicsLabs/LiME [19] FTK Imager, Available online at http://accessdata.com/product-download/ digital-forensics/ftk-imager-version-3.2.0
[20] FTK, Available online at http://accessdata.com/solutions/ digital-forensics/forensic-toolkit-ftk [21] EnCase Forensic, Available online at https://www.guidancesoftware.com/products/ Pages/encase-forensic/overview.aspx
[22] The Volatility Framework, Available online at https://code.google.com/p/volatility/ [23] Dshell, Available online at https://github.com/USArmyResearchLab/Dshell [24] Redline, Available online at https: //www.mandiant.com/resources/download/redline
