Category Archives: Ics

Cyber Security Week in Review (Feb. 15, 2019)


Welcome to this week's Cyber Security Week in Review, where Cisco Talos runs down all of the news we think you need to know in the security world. For more news delivered to your inbox every week, sign up for our Threat Source newsletter here.

Top headlines this week


  • Email provider VFEmail says it suffered a “catastrophic” cyber attack. The company warned that about 18 years’ worth of customers’ emails may be permanently gone. “Every file server is lost, every backup server is lost. Strangely, not all VMs shared the same authentication, but all were destroyed. This was more than a multi-password via ssh exploit, and there was no ransom. Just attack and destroy,” VFEmail representatives said in a statement. 
  • Russia is considering isolating itself from the global internet. The Kremlin is experimenting with a new practice of only routing the country’s web requests through the country and not internationally. The country will run a test later this year in an effort to test its cyber defenses.
  • Apple released fixes for multiple security flaws in iOS. Two of the vulnerabilities, which were discovered by Google’s threat research team, were being exploited in the wild. The bugs could allow an attacker to escalate their privileges and eventually completely take over a device. 

From Talos


  • Microsoft released its monthly security update this week, disclosing a variety of vulnerabilities in several of its products. The latest Patch Tuesday covers 69 vulnerabilities, 20 of which are rated “critical,” 46 that are considered “important” and three that are “moderate.” This release also includes a critical security advisory regarding a security update to Adobe Flash Player. 
  • Adobe released security updates for several of its products, including Flash and Acrobat Reader. Cisco Talos specifically discovered a critical remote code execution vulnerability in Adobe Acrobat Reader DC. An attacker could cause a heap overflow by tricking the user into opening a specially crafted PDF, which would allow the attacker to gain code execution privileges. 
  • A new tool from Talos can allow you to study the effect of cyber attacks on oil pump jacks. We released a 3-D printed, small-scale model of a pump jack that can be “hacked” from a smartphone, causing it to eventually overheat. We’ll also be taking this exhibit on the road over the course of the year. 

Malware roundup


  • A new variant of the Astaroth trojan is targeting Brazil via multiple spam campaigns. Once infected, the malware can steal users’ personal information and uses several deobfuscation techniques to make it more difficult to detect. The spam emails are also hitting users in parts of Europe.
  • Credit unions across the U.S. received phishing emails last week targeting anti-money laundering efforts. The phony emails claim to have information on unauthorized wire transfers and ask them to open a PDF that displays the alleged transaction and contains a link to a malicious web page. The attackers used information that’s believed to only be available to the National Credit Union Administration.
  • Google removed a cryptocurrency-stealing malware from its store. The malicious app disguised itself as the legitimate MetaMask service. Once downloaded, it would steal login credentials to steal users’ Ethereum funds. 

The rest of the news


  • Blockchain technology could be useful in detecting deepfake videos, specifically in police body cameras. A new tool called Amber Authenticate runs in the background of cameras to record the hashes of the video, which would appear different a second time if the user had edited the video. All of these results are recorded on the public blockchain.
  • India requested Facebook give its government a backdoor into the WhatsApp messaging app. This would require Facebook to give the government access to users’ encrypted messages that were originally secret.
  • Two U.S. senators are requesting an investigation into foreign VPN services. The senators say the companies could pose a national security risk.  


What you can learn from Cisco Talos’ new oil pumpjack workshop

Paul Rascagneres wrote this blog post with contributions from Patrick DeSantis from Cisco Talos ARES (Advanced Research/Embedded Systems).

Executive summary


Every day, more industrial control systems (ICS) become vulnerable to cyber attacks. As these massive, critical machines become more interconnected to networks, it increases the ways in which attackers could disrupt their operations and makes it tougher for those who protect organizations' networks to cover all possible attack vectors. To demonstrate how these ICSs interact with a network, we are releasing a model of a 3-D printed oil pumpjack connected to a simulated programmable logic controller (PLC) supporting two industrial protocols. Throughout the year, Talos will have this model at several workshops where attendees can try it out for themselves. For convenience, we are also providing the blueprints and code to even test this out for yourself at home.


We are releasing the 3-D printed model of the pumpjack, the Arduino source code (including the Modbus over TCP and the EtherNet/IP protocols), as well as the code for the human-machine interface (HMI) to control the pump over a network.


To show how serious of a problem it could be if an attacker were to gain control of this device in the real world, here's a GIF that shows how the pump reacts when the motor speed is increased beyond its natural pace.

Hardware description


Global architecture


The project is divided into seven parts:

  • The 3-D printed parts.
  • The pump, which is controlled by a motor.
  • A gauge to show the speed of the motor, activated by a servo-motor.
  • An Arduino UNO board, which is the brain of the pump and simulates a PLC.
  • An Ethernet Arduino shield for Ethernet support.
  • A motor shield to manage the motors and the smoke generator.
  • An HMI developed in Python with Flask to monitor and control the pump remotely.

Here are some additional details on these components.

Oil pumpjack 3-D objects


Here is the model of the pump:
  • Object 1 is the pump.
  • Object 2 is the motor, which activates the pump.
  • Object 3 is where the three boards are located. It also contains the gauge that shows the speed level.

The .stl files can be downloaded on our GitHub.

Electronic components


The electronic components, which function as the systems controller, is composed by one Arduino board (Arduino UNO), two shields (Ethernet and Motor), one servo motor, one motor and a smoke generator.

The Arduino UNO and the Ethernet shield (on the left) will be connected directly together. The VMA03 shield (of the right) could be directly connected to. However, during our test, we identified that this component generates a lot of electromagnetic noise. This noise impacted the Ethernet signal. So we decided to separate them:

The Arduino and the Ethernet shield are powered via a USB port. The VM03 shield is powered externally by a 12-volt adapter.

Software


Arduino


The Arduino source code can be downloaded on our GitHub. You can find two projects: The first project supports Modbus over TCP protocol and the second is the EtherNet/IP protocol. For each project, we used Python scripts to test the communication, an HMI, protocol scanner or PCAP of the communication.

Here is an explanation of the Arduino's GPIO pins.

  • Motor A (main pump motor) is controlled by the PIN 6 (PWM) and 7 (direction).
  • Motor B (the smoke generator) is controlled by the PIN 9 (PWM) and 12 (direction).
  • PIN 8 is used by the speed gauge.
  • The speed of the motor is defined by arbitrary values between 5,000 and 15,000 — it's set to 8,000 by default. Be careful with this. If the speed gets too high, it can destroy the pump.
  • The IP of the pump controller is statically set to 10.10.10.1, you can easily change it in the setup() function.
  • In the Modbus over TCP protocol, the speed is located in the register 6 and the gauge value in the register 7. A register is a 16 bits object in the Modbus protocol.
  • In the Ethernet/IP protocol, the speed is located in B1:1 and the gauge in B1:7. The Bx:y are tags used by Ethernet/IP to store values.


You can use the serial port in the Arduino IDE to get the debug while the pump is running.

Implemented protocols


Modbus over TCP/IP


The first protocol we implemented is in the Modbus protocol. It was implemented in 1979 by Modicon (now Schneider Electric). It often communicates between controllers and other systems or devices on a TCP network. We recommend reading this page to understand it. We based our implementation of Mudbus library, which supports the following commands:

  • Read coil 0x01
  • Read register 0x03
  • Write coil 0x05
  • Write register 0x06
  • Write multiple coils, 0x0f
  • Write multiple registers 0x10
  • Get device information 0x43


The Coils values are stored in the C[] arrays and the registers values in the R[]. The pump only uses the Register to store the speed of the pump motor and the gauge value.

The device can be queried by using the PyModbus API. Here is an example of the output:
from pymodbus.client.sync import ModbusTcpClient
import sys

client = ModbusTcpClient("10.10.10.1")
result = client.read_holding_registers(6, 1)
print(result.registers)
result2 = client.read_holding_registers(7, 1)
print(result2.registers)


from pymodbus import mei_message
rq = mei_message.ReadDeviceInformationRequest()
result3=client.execute(rq)
print(result3.information)

client.close()
user@lab:~/pumpjack_project/arduino_modbus/python$ ./test.py 
[8000]
[73]
{0: 'Talos PLC', 1: 'pumpjack', 2: '0.1'}

EtherNet/IP


We implemented a second protocol used in industrial infrastructure: EtherNet/IP. This protocol is the adaptation of the Common Industrial Protocol (CIP) to Ethernet. We did not fully implement the protocol; we only support read and write tags. We support the Micrologix or SLC PLCs protocol. The protocol is pretty similar to the Modbus protocol, but also includes the notion of the session ID. The session is included in our implementation.

The current version supports Bx:x and Nx:x tags, where x is between 0 and 9.

The device can be queried by using the pycomm API. Here's an example:
from pycomm.ab_comm.slc import Driver as SlcDriver
import logging

c = SlcDriver()
def read_val(num):
print c.read_tag('B1:%d' % num)[3]

if c.open('10.10.10.1'):
read_val(1)
user@lab:~/pumpjack_project/arduino_ENIPCIP/python$ ./test.py 
8000

Human-machine interface (HMI)


Finally, we provide an HMI to manage the pump. It is developed in Python by using Flask to create the web server and PyModbus to communicate with the pump. Here is a screenshot of the interface:

The web server will first retrieve the pump device name and version via the Modbus over TCP protocol using the "get device information" (0x43) command. It will then retrieve the value of the motor speed and the gauge. The value will determine the gauge level on the web page. By clicking on the increase or decrease button, the motor speed will be increased or decreased by using the Modbus protocol and changing the register 6 value.

Examples of workshop


There are a lot of ways in which researchers could utilize this system to research potential attack vectors on an oil pumpjack. For example, it could be used to understand the two ICS protocols, which are not encountered in traditional IT networks. We can create a packet capture of network traffic on the HMI to perform additional analysis in Wireshark. The built-in Wireshark dissector perfectly parses the Modbus over TCP. The EtherNet/IP protocol dissector is less robust but is able to be partially decoded manually, which is a great exercise. Jared Rittle from Cisco Talos has some of his previous work with the Wireshark dissector available here.

Another scenario could be to scan the local network to identify the Modbus systems and try to modify the pump behaviour by enumerating and then modifying the values stored in the coils and registers. If you are interested in ICS attacks such as Stuxnet — a malicious worm that attacks SCADA systems — you can see the impact of compromising the HMI system and modifying the information provided by the HMI to the operator.

All the scenarios are offensive. We recommend playing the defensive part. For example, SNORT® provides Modbus over TCP support, which you can read the details of here. With this module, you can monitor the traffic, block requests from unauthorized IPs, identify large scans or avoid putting larger values in the registers/coils — and avoid breaking your pumpjack.

Conclusion


We hope these materials help researchers better understand industrial protocols, more particularly Modbus over TCP and EtherNet/IP. These two protocols are unauthenticated protocols always used in production. In the examples, we saw how to modify the internal value of a PLC (simulated by the Arduino UNO). But the PLC programming is performed by this protocol, too. On a real PLC, we can use the same protocol to program it, replace the original code, or patch it.o For more information about a real-life PLC attack, we recommend our whitepaper "Process Control through counterfeit comms: Using and abusing built-in functionality to own a PLC," which is available here.

We decided to publicly release our project to make it accessible to the largest audience possible.. Do not hesitate to contribute to this project,by adding features to the implemented protocols or adding new ones. We would be happy to handle pull requests on our GitHub.

Omron addressed multiple flaws in its CX-Supervisor product

The electronics firm Omron released a security update to address flaws in its CX-Supervisor product that can be exploited DoS attacks and remote code execution.

CX-Supervisor allows to rapidly create human-machine interfaces (HMIs) for supervisory control and data acquisition (SCADA) systems thanks to the availability of a large number of predefined functions and libraries. The software is widely adopted in multiple industries, mainly in the energy sector.

The vulnerabilities were reported through Trend Micro’s Zero Day Initiative (ZDI). by the security expert Esteban Ruiz of Source Incite. One of the vulnerabilities, tracked as CVE-2018-19027 received a “high” severity rating.

The CVE-2018-19027 flaw affects the CX-One products, the flaw was reported to the vendor on 2018-07-02 while it was publicly disclosed on 2019-01-14.

“This vulnerability allows remote attackers to execute arbitrary code on vulnerable installations of OMRON CX-One CX-Protocol. User interaction is required to exploit this vulnerability in that the target must visit a malicious page or open a malicious file.” reported the advisory published by ZDI.

“The specific flaw exists within the handling of PSW files. The issue results from the lack of proper validation of user-supplied data, which can result in a type confusion condition. An attacker can leverage this vulnerability to execute code in the context of the current process.”

CX-Supervisor Omron

The ICS-CERT published an advisory that includes details for all the vulnerabilities discovered by the expert, The addressed issues include a use-after-free, lack of proper validation for user-supplied input, and type confusion issues that can be exploited by attackers to execute arbitrary code on the vulnerable systems.

The “IMPROPER NEUTRALIZATION OF SPECIAL ELEMENTS USED IN A COMMAND (‘COMMAND INJECTION’) CWE-77″ could allow and attacker to inject commands to delete files and/or delete the contents of a file on the system by using a specially crafted project file. The exploitation of this bug, tracked as CVE-2018-19013 can cause a DoS condition, the issue has received a CVSS v3 base score of 5.0.

The vulnerabilities have been addressed with the release of version 3.5.0.11 of CX-Supervisor.
The ICS-CERT also suggest to upgrade development projects and save them in the new format, then rebuilt in the latest 3.5.0.11 format.

Pierluigi Paganini

CX-Supervisor, ICS (SecurityAffairs – CX-Supervisor, ICS)

The post Omron addressed multiple flaws in its CX-Supervisor product appeared first on Security Affairs.

TRITON Attribution: Russian Government-Owned Lab Most Likely Built Custom Intrusion Tools for TRITON Attackers

Overview

In a previous blog post we detailed the TRITON intrusion that impacted industrial control systems (ICS) at a critical infrastructure facility. We now track this activity set as TEMP.Veles. In this blog post we provide additional information linking TEMP.Veles and their activity surrounding the TRITON intrusion to a Russian government-owned research institute.

TRITON Intrusion Demonstrates Russian Links; Likely Backed by Russian Research Institute

FireEye Intelligence assesses with high confidence that intrusion activity that led to deployment of TRITON was supported by the Central Scientific Research Institute of Chemistry and Mechanics (CNIIHM; a.k.a. ЦНИИХМ), a Russian government-owned technical research institution located in Moscow. The following factors supporting this assessment are further detailed in this post. We present as much public information as possible to support this assessment, but withheld sensitive information that further contributes to our high confidence assessment.

  1. FireEye uncovered malware development activity that is very likely supporting TEMP.Veles activity. This includes testing multiple versions of malicious software, some of which were used by TEMP.Veles during the TRITON intrusion.
  2. Investigation of this testing activity reveals multiple independent ties to Russia, CNIIHM, and a specific person in Moscow. This person’s online activity shows significant links to CNIIHM.
  3. An IP address registered to CNIIHM has been employed by TEMP.Veles for multiple purposes, including monitoring open-source coverage of TRITON, network reconnaissance, and malicious activity in support of the TRITON intrusion.
  4. Behavior patterns observed in TEMP.Veles activity are consistent with the Moscow time zone, where CNIIHM is located.
  5. We judge that CNIIHM likely possesses the necessary institutional knowledge and personnel to assist in the orchestration and development of TRITON and TEMP.Veles operations.

While we cannot rule out the possibility that one or more CNIIHM employees could have conducted TEMP.Veles activity without their employer’s approval, the details shared in this post demonstrate that this explanation is less plausible than TEMP.Veles operating with the support of the institute.

Detail

Malware Testing Activity Suggests Links between TEMP.Veles and CNIIHM

During our investigation of TEMP.Veles activity, we found multiple unique tools that the group deployed in the target environment. Some of these same tools, identified by hash, were evaluated in a malware testing environment by a single user.

Malware Testing Environment Tied to TEMP.Veles

We identified a malware testing environment that we assess with high confidence was used to refine some TEMP.Veles tools.

  • At times, the use of this malware testing environment correlates to in-network activities of TEMP.Veles, demonstrating direct operational support for intrusion activity.
    • Four files tested in 2014 are based on the open-source project, cryptcat. Analysis of these cryptcat binaries indicates that the actor continually modified them to decrease AV detection rates. One of these files was deployed in a TEMP.Veles target’s network. The compiled version with the least detections was later re-tested in 2017 and deployed less than a week later during TEMP.Veles activities in the target environment.
    • TEMP.Veles’ lateral movement activities used a publicly-available PowerShell-based tool, WMImplant. On multiple dates in 2017, TEMP.Veles struggled to execute this utility on multiple victim systems, potentially due to AV detection. Soon after, the customized utility was again evaluated in the malware testing environment. The following day, TEMP.Veles again tried the utility on a compromised system.
  • The user has been active in the malware testing environment since at least 2013, testing customized versions of multiple open-source frameworks, including Metasploit, Cobalt Strike, PowerSploit, and other projects. The user’s development patterns appear to pay particular attention to AV evasion and alternative code execution techniques.
  • Custom payloads utilized by TEMP.Veles in investigations conducted by Mandiant are typically weaponized versions of legitimate open-source software, retrofitted with code used for command and control.

Testing, Malware Artifacts, and Malicious Activity Suggests Tie to CNIIHM

Multiple factors suggest that this activity is Russian in origin and associated with CNIIHM.

  • A PDB path contained in a tested file contained a string that appears to be a unique handle or user name. This moniker is linked to a Russia-based person active in Russian information security communities since at least 2011.
    • The handle has been credited with vulnerability research contributions to the Russian version of Hacker Magazine (хакер).
    • According to a now-defunct social media profile, the same individual was a professor at CNIIHM, which is located near Nagatinskaya Street in the Nagatino-Sadovniki district of Moscow.
    • Another profile using the handle on a Russian social network currently shows multiple photos of the user in proximity to Moscow for the entire history of the profile.
  • Suspected TEMP.Veles incidents include malicious activity originating from 87.245.143.140, which is registered to CNIIHM.
    • This IP address has been used to monitor open-source coverage of TRITON, heightening the probability of an interest by unknown subjects, originating from this network, in TEMP.Veles-related activities.
    • It also has engaged in network reconnaissance against targets of interest to TEMP.Veles.
    • The IP address has been tied to additional malicious activity in support of the TRITON intrusion.
  • Multiple files have Cyrillic names and artifacts.


Figure 1: Heatmap of TRITON attacker operating hours, represented in UTC time

Behavior Patterns Consistent with Moscow Time Zone

Adversary behavioral artifacts further suggest the TEMP.Veles operators are based in Moscow, lending some further support to the scenario that CNIIHM, a Russian research organization in Moscow, has been involved in TEMP.Veles activity.

  • We identified file creation times for numerous files that TEMP.Veles created during lateral movement on a target’s network. These file creation times conform to a work schedule typical of an actor operating within a UTC+3 time zone (Figure 1) supporting a proximity to Moscow.


Figure 2: Modified service config

  • Additional language artifacts recovered from TEMP.Veles toolsets are also consistent with such a regional nexus.
    • A ZIP archive recovered during our investigations, schtasks.zip, contained an installer and uninstaller of CATRUNNER that includes two versions of an XML scheduled task definitions for a masquerading service ‘ProgramDataUpdater.’
    • The malicious installation version has a task name and description in English, and the clean uninstall version has a task name and description in Cyrillic. The timeline of modification dates within the ZIP also suggest the actor changed the Russian version to English in sequential order, heightening the possibility of a deliberate effort to mask its origins (Figure 2).


Figure 3: Central Research Institute of Chemistry and Mechanics (CNIIHM) (Google Maps)

CNIIHM Likely Possesses Necessary Institutional Knowledge and Personnel to Create TRITON and Support TEMP.Veles Operations

While we know that TEMP.Veles deployed the TRITON attack framework, we do not have specific evidence to prove that CNIIHM did (or did not) develop the tool. We infer that CNIIHM likely maintains the institutional expertise needed to develop and prototype TRITON based on the institute’s self-described mission and other public information.

  • CNIIHM has at least two research divisions that are experienced in critical infrastructure, enterprise safety, and the development of weapons/military equipment:
    • The Center for Applied Research creates means and methods for protecting critical infrastructure from destructive information and technological impacts.
    • The Center for Experimental Mechanical Engineering develops weapons as well as military and special equipment. It also researches methods for enabling enterprise safety in emergency situations.
  • CNIIHM officially collaborates with other national technology and development organizations, including:
    • The Moscow Institute of Physics and Technology (PsyTech), which specializes in applied physics, computing science, chemistry, and biology.
    • The Association of State Scientific Centers “Nauka,” which coordinates 43 Scientific Centers of the Russian Federation (SSC RF). Some of its main areas of interest include nuclear physics, computer science and instrumentation, robotics and engineering, and electrical engineering, among others.
    • The Federal Service for Technical and Export Control (FTEC) which is responsible for export control, intellectual property, and protecting confidential information.
    • The Russian Academy of Missile and Artillery Sciences (PAPAH) which specializes in research and development for strengthening Russia’s defense industrial complex.
  • Information from a Russian recruitment website, linked to CNIIHM’s official domain, indicates that CNIIHM is also dedicated to the development of intelligent systems for computer-aided design and control, and the creation of new information technologies (Figure 4).


Figure 4: CNIIHM website homepage

Primary Alternative Explanation Unlikely

Some possibility remains that one or more CNIIHM employees could have conducted the activity linking TEMP.Veles to CNIIHM without their employer’s approval. However, this scenario is highly unlikely.

  • In this scenario, one or more persons – likely including at least one CNIIHM employee, based on the moniker discussed above – would have had to conduct extensive, high-risk malware development and intrusion activity from CNIIHM’s address space without CNIIHM’s knowledge and approval over multiple years.
  • CNIIHM’s characteristics are consistent with what we might expect of an organization responsible for TEMP.Veles activity. TRITON is a highly specialized framework whose development would be within the capability of a low percentage of intrusion operators.

A Totally Tubular Treatise on TRITON and TriStation

Introduction

In December 2017, FireEye's Mandiant discussed an incident response involving the TRITON framework. The TRITON attack and many of the publicly discussed ICS intrusions involved routine techniques where the threat actors used only what is necessary to succeed in their mission. For both INDUSTROYER and TRITON, the attackers moved from the IT network to the OT (operational technology) network through systems that were accessible to both environments. Traditional malware backdoors, Mimikatz distillates, remote desktop sessions, and other well-documented, easily-detected attack methods were used throughout these intrusions.

Despite the routine techniques employed to gain access to an OT environment, the threat actors behind the TRITON malware framework invested significant time learning about the Triconex Safety Instrumented System (SIS) controllers and TriStation, a proprietary network communications protocol. The investment and purpose of the Triconex SIS controllers leads Mandiant to assess the attacker's objective was likely to build the capability to cause physical consequences.

TriStation remains closed source and there is no official public information detailing the structure of the protocol, raising several questions about how the TRITON framework was developed. Did the actor have access to a Triconex controller and TriStation 1131 software suite? When did development first start? How did the threat actor reverse engineer the protocol, and to what extent? What is the protocol structure?

FireEye’s Advanced Practices Team was born to investigate adversary methodologies, and to answer these types of questions, so we started with a deeper look at the TRITON’s own Python scripts.

Glossary:

  • TRITON – Malware framework designed to operate Triconex SIS controllers via the TriStation protocol.
  • TriStation – UDP network protocol specific to Triconex controllers.
  • TRITON threat actor – The human beings who developed, deployed and/or operated TRITON.

Diving into TRITON's Implementation of TriStation

TriStation is a proprietary network protocol and there is no public documentation detailing its structure or how to create software applications that use TriStation. The current TriStation UDP/IP protocol is little understood, but natively implemented through the TriStation 1131 software suite. TriStation operates by UDP over port 1502 and allows for communications between designated masters (PCs with the software that are “engineering workstations”) and slaves (Triconex controllers with special communications modules) over a network.

To us, the Triconex systems, software and associated terminology sound foreign and complicated, and the TriStation protocol is no different. Attempting to understand the protocol from ground zero would take a considerable amount of time and reverse engineering effort – so why not learn from TRITON itself? With the TRITON framework containing TriStation communication functionality, we pursued studying the framework to better understand this mysterious protocol. Work smarter, not harder, amirite?

The TRITON framework has a multitude of functionalities, but we started with the basic components:

  • TS_cnames.pyc # Compiled at: 2017-08-03 10:52:33
  • TsBase.pyc # Compiled at: 2017-08-03 10:52:33
  • TsHi.pyc # Compiled at: 2017-08-04 02:04:01
  • TsLow.pyc # Compiled at: 2017-08-03 10:46:51

TsLow.pyc (Figure 1) contains several pieces of code for error handling, but these also present some cues to the protocol structure.


Figure 1: TsLow.pyc function print_last_error()

In the TsLow.pyc’s function for print_last_error we see error handling for “TCM Error”. This compares the TriStation packet value at offset 0 with a value in a corresponding array from TS_cnames.pyc (Figure 2), which is largely used as a “dictionary” for the protocol.


Figure 2: TS_cnames.pyc TS_cst array

From this we can infer that offset 0 of the TriStation protocol contains message types. This is supported by an additional function, tcm_result, which declares type, size = struct.unpack('<HH', data_received[0:4]), stating that the first two bytes should be handled as integer type and the second two bytes are integer size of the TriStation message. This is our first glimpse into what the threat actor(s) understood about the TriStation protocol.

Since there are only 11 defined message types, it really doesn't matter much if the type is one byte or two because the second byte will always be 0x00.

We also have indications that message type 5 is for all Execution Command Requests and Responses, so it is curious to observe that the TRITON developers called this “Command Reply.” (We won’t understand this naming convention until later.)

Next we examine TsLow.pyc’s print_last_error function (Figure 3) to look at “TS Error” and “TS_names.” We begin by looking at the ts_err variable and see that it references ts_result.


Figure 3: TsLow.pyc function print_last_error() with ts_err highlighted

We follow that thread to ts_result, which defines a few variables in the next 10 bytes (Figure 4): dir, cid, cmd, cnt, unk, cks, siz = struct.unpack('<, ts_packet[0:10]). Now things are heating up. What fun. There’s a lot to unpack here, but the most interesting thing is how this piece script breaks down 10 bytes from ts_packet into different variables.


Figure 4: ts_result with ts_packet header variables highlighted


Figure 5: tcm_result

Referencing tcm_result (Figure 5) we see that it defines type and size as the first four bytes (offset 0 – 3) and tcm_result returns the packet bytes 4:-2 (offset 4 to the end minus 2, because the last two bytes are the CRC-16 checksum). Now that we know where tcm_result leaves off, we know that the ts_reply “cmd” is a single byte at offset 6, and corresponds to the values in the TS_cnames.pyc array and TS_names (Figure 6). The TRITON script also tells us that any integer value over 100 is a likely “command reply.” Sweet.

When looking back at the ts_result packet header definitions, we begin to see some gaps in the TRITON developer's knowledge: dir, cid, cmd, cnt, unk, cks, siz = struct.unpack('<, ts_packet[0:10]). We're clearly speculating based on naming conventions, but we get an impression that offsets 4, 5 and 6 could be "direction", "controller ID" and "command", respectively. Values such as "unk" show that the developer either did not know or did not care to identify this value. We suspect it is a constant, but this value is still unknown to us.


Figure 6: Excerpt TS_cnames.pyc TS_names array, which contain TRITON actor’s notes for execution command function codes

TriStation Protocol Packet Structure

The TRITON threat actor’s knowledge and reverse engineering effort provides us a better understanding of the protocol. From here we can start to form a more complete picture and document the basic functionality of TriStation. We are primarily interested in message type 5, Execution Command, which best illustrates the overall structure of the protocol. Other, smaller message types will have varying structure.


Figure 7: Sample TriStation "Allocate Program" Execution Command, with color annotation and protocol legend

Corroborating the TriStation Analysis

Minute discrepancies aside, the TriStation structure detailed in Figure 7 is supported by other public analyses. Foremost, researchers from the Coordinated Science Laboratory (CSL) at University of Illinois at Urbana-Champaign published a 2017 paper titled "Attack Induced Common-Mode Failures on PLC-based Safety System in a Nuclear Power Plant". The CSL team mentions that they used the Triconex System Access Application (TSAA) protocol to reverse engineer elements of the TriStation protocol. TSAA is a protocol developed by the same company as TriStation. Unlike TriStation, the TSAA protocol structure is described within official documentation. CSL assessed similarities between the two protocols would exist and they leveraged TSAA to better understand TriStation. The team's overall research and analysis of the general packet structure aligns with our TRITON-sourced packet structure.

There are some awesome blog posts and whitepapers out there that support our findings in one way or another. Writeups by Midnight Blue Labs, Accenture, and US-CERT each explain how the TRITON framework relates to the TriStation protocol in superb detail.

TriStation's Reverse Engineering and TRITON's Development

When TRITON was discovered, we began to wonder how the TRITON actor reverse engineered TriStation and implemented it into the framework. We have a lot of theories, all of which seemed plausible: Did they build, buy, borrow, or steal? Or some combination thereof?

Our initial theory was that the threat actor purchased a Triconex controller and software for their own testing and reverse engineering from the "ground up", although if this was the case we do not believe they had a controller with the exact vulnerable firmware version, else they would have had fewer problems with TRITON in practice at the victim site. They may have bought or used a demo version of the TriStation 1131 software, allowing them to reverse engineer enough of TriStation for the framework. They may have stolen TriStation Python libraries from ICS companies, subsidiaries or system integrators and used the stolen material as a base for TriStation and TRITON development. But then again, it is possible that they borrowed TriStation software, Triconex hardware and Python connectors from government-owned utility that was using them legitimately.

Looking at the raw TRITON code, some of the comments may appear oddly phrased, but we do get a sense that the developer is clearly using many of the right vernacular and acronyms, showing smarts on PLC programming. The TS_cnames.pyc script contains interesting typos such as 'Set lable', 'Alocate network accepted', 'Symbol table ccepted' and 'Set program information reponse'. These appear to be normal human error and reflect neither poor written English nor laziness in coding. The significant amount of annotation, cascading logic, and robust error handling throughout the code suggests thoughtful development and testing of the framework. This complicates the theory of "ground up" development, so did they base their code on something else?

While learning from the TriStation functionality within TRITON, we continued to explore legitimate TriStation software. We began our search for "TS1131.exe" and hit dead ends sorting through TriStation DLLs until we came across a variety of TriStation utilities in MSI form. We ultimately stumbled across a juicy archive containing "Trilog v4." Upon further inspection, this file installed "TriLog.exe," which the original TRITON executable mimicked, and a couple of supporting DLLs, all of which were timestamped around August 2006.

When we saw the DLL file description "Tricon Communications Interface" and original file name "TricCom.DLL", we knew we were in the right place. With a simple look at the file strings, "BAZINGA!" We struck gold.

File Name

tr1com40.dll

MD5

069247DF527A96A0E048732CA57E7D3D

Size

110592

Compile Date

2006-08-23

File Description

Tricon Communications Interface

Product Name

TricCom Dynamic Link Library

File Version

4.2.441

Original File Name

TricCom.DLL

Copyright

Copyright © 1993-2006 Triconex Corporation

The tr1com40.DLL is exactly what you would expect to see in a custom application package. It is a library that helps support the communications for a Triconex controller. If you've pored over TRITON as much as we have, the moment you look at strings you can see the obvious overlaps between the legitimate DLL and TRITON's own TS_cnames.pyc.


Figure 8: Strings excerpt from tr1com40.DLL

Each of the execution command "error codes" from TS_cnames.pyc are in the strings of tr1com40.DLL (Figure 8). We see "An MP has re-educated" and "Invalid Tristation I command". Even misspelled command strings verbatim such as "Non-existant data item" and "Alocate network accepted". We also see many of the same unknown values. What is obvious from this discovery is that some of the strings in TRITON are likely based on code used in communications libraries for Trident and Tricon controllers.

In our brief survey of the legitimate Triconex Corporation binaries, we observed a few samples with related string tables.

Pe:dllname

Compile Date

Reference CPP Strings Code

Lagcom40.dll

2004/11/19

$Workfile:   LAGSTRS.CPP  $ $Modtime:   Jul 21 1999 17:17:26  $ $Revision:   1.0

Tr1com40.dll

2006/08/23

$Workfile:   TR1STRS.CPP  $ $Modtime:   May 16 2006 09:55:20  $ $Revision:   1.4

Tridcom.dll

2008/07/23

$Workfile:   LAGSTRS.CPP  $ $Modtime:   Jul 21 1999 17:17:26  $ $Revision:   1.0

Triccom.dll

2008/07/23

$Workfile:   TR1STRS.CPP  $ $Modtime:   May 16 2006 09:55:20  $ $Revision:   1.4

Tridcom.dll

2010/09/29

$Workfile:   LAGSTRS.CPP  $ $Modtime:   Jul 21 1999 17:17:26  $ $Revision:   1.0 

Tr1com.dll

2011/04/27

$Workfile:   TR1STRS.CPP  $ $Modtime:   May 16 2006 09:55:20  $ $Revision:   1.4

Lagcom.dll

2011/04/27

$Workfile:   LAGSTRS.CPP  $ $Modtime:   Jul 21 1999 17:17:26  $ $Revision:   1.0

Triccom.dll

2011/04/27

$Workfile:   TR1STRS.CPP  $ $Modtime:   May 16 2006 09:55:20  $ $Revision:   1.4

We extracted the CPP string tables in TR1STRS and LAGSTRS and the TS_cnames.pyc TS_names array from TRITON, and compared the 210, 204, and 212 relevant strings from each respective file.

TS_cnames.pyc TS_names and tr1com40.dll share 202 of 220 combined table strings. The remaining strings are unique to each, as seen here:

TS_cnames.TS_names (2017 pyc)

Tr1com40.dll (2006 CPP)

Go to DOWNLOAD mode

<200>

Not set

<209>

Unk75

Bad message from module

Unk76

Bad message type

Unk77

Bad TMI version number

Unk78

Module did not respond

Unk79

Open Connection: Invalid SAP %d

Unk81

Unsupported message for this TMI version

Unk83

 

Wrong command

 

TS_cnames.pyc TS_names and Tridcom.dll (1999 CPP) shared only 151 of 268 combined table strings, showing a much smaller overlap with the seemingly older CPP library. This makes sense based on the context that Tridcom.dll is meant for a Trident controller, not a Tricon controller. It does seem as though Tr1com40.dll and TR1STRS.CPP code was based on older work.

We are not shocked to find that the threat actor reversed legitimate code to bolster development of the TRITON framework. They want to work smarter, not harder, too. But after reverse engineering legitimate software and implementing the basics of the TriStation, the threat actors still had an incomplete understanding of the protocol. In TRITON's TS_cnames.pyc we saw "Unk75", "Unk76", "Unk83" and other values that were not present in the tr1com40.DLL strings, indicating that the TRITON threat actor may have explored the protocol and annotated their findings beyond what they reverse engineered from the DLL. The gaps in TriStation implementation show us why the actors encountered problems interacting with the Triconex controllers when using TRITON in the wild.

You can see more of the Trilog and Triconex DLL files on VirusTotal.

Item Name

MD5

Description

Tr1com40.dll

069247df527a96a0e048732ca57e7d3d

Tricom Communcations DLL

Data1.cab

e6a3c93a6d433cbaf6f573b6c09d76c4

Parent of Tr1com40.dll

Trilog v4.1.360R

13a3b83ba2c4236ca59aba679941c8a5

RAR Archive of TriLog

TridCom.dll

5c2ed617fdec4779cb33c89082a43100

Trident Communications DLL

Afterthoughts

Seeing Triconex systems targeted with malicious intent was new to the world six months ago. Moving forward it would be reasonable to anticipate additional frameworks, such as TRITON, designed for usage against other SIS controllers and associated technologies. If Triconex was within scope, we may see similar attacker methodologies affecting the dominant industrial safety technologies.

Basic security measures do little to thwart truly persistent threat actors and monitoring only IT networks is not an ideal situation. Visibility into both the IT and OT environments is critical for detecting the various stages of an ICS intrusion. Simple detection concepts such as baseline deviation can provide insight into abnormal activity.

While the TRITON framework was actively in use, how many traditional ICS “alarms” were set off while the actors tested their exploits and backdoors on the Triconex controller? How many times did the TriStation protocol, as implemented in their Python scripts, fail or cause errors because of non-standard traffic? How many TriStation UDP pings were sent and how many Connection Requests? How did these statistics compare to the baseline for TriStation traffic? There are no answers to these questions for now. We believe that we can identify these anomalies in the long run if we strive for increased visibility into ICS technologies.

We hope that by holding public discussions about ICS technologies, the Infosec community can cultivate closer relationships with ICS vendors and give the world better insight into how attackers move from the IT to the OT space. We want to foster more conversations like this and generally share good techniques for finding evil. Since most of all ICS attacks involve standard IT intrusions, we should probably come together to invent and improve any guidelines for how to monitor PCs and engineering workstations that bridge the IT and OT networks. We envision a world where attacking or disrupting ICS operations costs the threat actor their cover, their toolkits, their time, and their freedom. It's an ideal world, but something nice to shoot for.

Thanks and Future Work

There is still much to do for TRITON and TriStation. There are many more sub-message types and nuances for parsing out the nitty gritty details, which is hard to do without a controller of our own. And although we’ve published much of what we learned about the TriStation here on the blog, our work will continue as we continue our study of the protocol.

Thanks to everyone who did so much public research on TRITON and TriStation. We have cited a few individuals in this blog post, but there is a lot more community-sourced information that gave us clues and leads for our research and testing of the framework and protocol. We also have to acknowledge the research performed by the TRITON attackers. We borrowed a lot of your knowledge about TriStation from the TRITON framework itself.

Finally, remember that we're here to collaborate. We think most of our research is right, but if you notice any errors or omissions, or have ideas for improvements, please spear phish contact: smiller@fireeye.com.

Recommended Reading

Appendix A: TriStation Message Type Codes

The following table consists of hex values at offset 0 in the TriStation UDP packets and the associated dictionary definitions, extracted verbatim from the TRITON framework in library TS_cnames.pyc.

Value at 0x0

Message Type

1

Connection Request

2

Connection Response

3

Disconnect Request

4

Disconnect Response

5

Execution Command

6

Ping Command

7

Connection Limit Reached

8

Not Connected

9

MPS Are Dead

10

Access Denied

11

Connection Failed

Appendix B: TriStation Execution Command Function Codes

The following table consists of hex values at offset 6 in the TriStation UDP packets and the associated dictionary definitions, extracted verbatim from the TRITON framework in library TS_cnames.pyc.

Value at 0x6

TS_cnames String

0

0: 'Start download all',

1

1: 'Start download change',

2

2: 'Update configuration',

3

3: 'Upload configuration',

4

4: 'Set I/O addresses',

5

5: 'Allocate network',

6

6: 'Load vector table',

7

7: 'Set calendar',

8

8: 'Get calendar',

9

9: 'Set scan time',

A

10: 'End download all',

B

11: 'End download change',

C

12: 'Cancel download change',

D

13: 'Attach TRICON',

E

14: 'Set I/O address limits',

F

15: 'Configure module',

10

16: 'Set multiple point values',

11

17: 'Enable all points',

12

18: 'Upload vector table',

13

19: 'Get CP status ',

14

20: 'Run program',

15

21: 'Halt program',

16

22: 'Pause program',

17

23: 'Do single scan',

18

24: 'Get chassis status',

19

25: 'Get minimum scan time',

1A

26: 'Set node number',

1B

27: 'Set I/O point values',

1C

28: 'Get I/O point values',

1D

29: 'Get MP status',

1E

30: 'Set retentive values',

1F

31: 'Adjust clock calendar',

20

32: 'Clear module alarms',

21

33: 'Get event log',

22

34: 'Set SOE block',

23

35: 'Record event log',

24

36: 'Get SOE data',

25

37: 'Enable OVD',

26

38: 'Disable OVD',

27

39: 'Enable all OVDs',

28

40: 'Disable all OVDs',

29

41: 'Process MODBUS',

2A

42: 'Upload network',

2B

43: 'Set lable',

2C

44: 'Configure system variables',

2D

45: 'Deconfigure module',

2E

46: 'Get system variables',

2F

47: 'Get module types',

30

48: 'Begin conversion table download',

31

49: 'Continue conversion table download',

32

50: 'End conversion table download',

33

51: 'Get conversion table',

34

52: 'Set ICM status',

35

53: 'Broadcast SOE data available',

36

54: 'Get module versions',

37

55: 'Allocate program',

38

56: 'Allocate function',

39

57: 'Clear retentives',

3A

58: 'Set initial values',

3B

59: 'Start TS2 program download',

3C

60: 'Set TS2 data area',

3D

61: 'Get TS2 data',

3E

62: 'Set TS2 data',

3F

63: 'Set program information',

40

64: 'Get program information',

41

65: 'Upload program',

42

66: 'Upload function',

43

67: 'Get point groups',

44

68: 'Allocate symbol table',

45

69: 'Get I/O address',

46

70: 'Resend I/O address',

47

71: 'Get program timing',

48

72: 'Allocate multiple functions',

49

73: 'Get node number',

4A

74: 'Get symbol table',

4B

75: 'Unk75',

4C

76: 'Unk76',

4D

77: 'Unk77',

4E

78: 'Unk78',

4F

79: 'Unk79',

50

80: 'Go to DOWNLOAD mode',

51

81: 'Unk81',

52

 

53

83: 'Unk83',

54

 

55

 

56

 

57

 

58

 

59

 

5A

 

5B

 

5C

 

5D

 

5E

 

5F

 

60

 

61

 

62

 

63

 

64

100: 'Command rejected',

65

101: 'Download all permitted',

66

102: 'Download change permitted',

67

103: 'Modification accepted',

68

104: 'Download cancelled',

69

105: 'Program accepted',

6A

106: 'TRICON attached',

6B

107: 'I/O addresses set',

6C

108: 'Get CP status response',

6D

109: 'Program is running',

6E

110: 'Program is halted',

6F

111: 'Program is paused',

70

112: 'End of single scan',

71

113: 'Get chassis configuration response',

72

114: 'Scan period modified',

73

115: '<115>',

74

116: '<116>',

75

117: 'Module configured',

76

118: '<118>',

77

119: 'Get chassis status response',

78

120: 'Vectors response',

79

121: 'Get I/O point values response',

7A

122: 'Calendar changed',

7B

123: 'Configuration updated',

7C

124: 'Get minimum scan time response',

7D

125: '<125>',

7E

126: 'Node number set',

7F

127: 'Get MP status response',

80

128: 'Retentive values set',

81

129: 'SOE block set',

82

130: 'Module alarms cleared',

83

131: 'Get event log response',

84

132: 'Symbol table ccepted',

85

133: 'OVD enable accepted',

86

134: 'OVD disable accepted',

87

135: 'Record event log response',

88

136: 'Upload network response',

89

137: 'Get SOE data response',

8A

138: 'Alocate network accepted',

8B

139: 'Load vector table accepted',

8C

140: 'Get calendar response',

8D

141: 'Label set',

8E

142: 'Get module types response',

8F

143: 'System variables configured',

90

144: 'Module deconfigured',

91

145: '<145>',

92

146: '<146>',

93

147: 'Get conversion table response',

94

148: 'ICM print data sent',

95

149: 'Set ICM status response',

96

150: 'Get system variables response',

97

151: 'Get module versions response',

98

152: 'Process MODBUS response',

99

153: 'Allocate program response',

9A

154: 'Allocate function response',

9B

155: 'Clear retentives response',

9C

156: 'Set initial values response',

9D

157: 'Set TS2 data area response',

9E

158: 'Get TS2 data response',

9F

159: 'Set TS2 data response',

A0

160: 'Set program information reponse',

A1

161: 'Get program information response',

A2

162: 'Upload program response',

A3

163: 'Upload function response',

A4

164: 'Get point groups response',

A5

165: 'Allocate symbol table response',

A6

166: 'Program timing response',

A7

167: 'Disable points full',

A8

168: 'Allocate multiple functions response',

A9

169: 'Get node number response',

AA

170: 'Symbol table response',

AB

 

AC

 

AD

 

AE

 

AF

 

B0

 

B1

 

B2

 

B3

 

B4

 

B5

 

B6

 

B7

 

B8

 

B9

 

BA

 

BB

 

BC

 

BD

 

BE

 

BF

 

C0

 

C1

 

C2

 

C3

 

C4

 

C5

 

C6

 

C7

 

C8

200: 'Wrong command',

C9

201: 'Load is in progress',

CA

202: 'Bad clock calendar data',

CB

203: 'Control program not halted',

CC

204: 'Control program checksum error',

CD

205: 'No memory available',

CE

206: 'Control program not valid',

CF

207: 'Not loading a control program',

D0

208: 'Network is out of range',

D1

209: 'Not enough arguments',

D2

210: 'A Network is missing',

D3

211: 'The download time mismatches',

D4

212: 'Key setting prohibits this operation',

D5

213: 'Bad control program version',

D6

214: 'Command not in correct sequence',

D7

215: '<215>',

D8

216: 'Bad Index for a module',

D9

217: 'Module address is invalid',

DA

218: '<218>',

DB

219: '<219>',

DC

220: 'Bad offset for an I/O point',

DD

221: 'Invalid point type',

DE

222: 'Invalid Point Location',

DF

223: 'Program name is invalid',

E0

224: '<224>',

E1

225: '<225>',

E2

226: '<226>',

E3

227: 'Invalid module type',

E4

228: '<228>',

E5

229: 'Invalid table type',

E6

230: '<230>',

E7

231: 'Invalid network continuation',

E8

232: 'Invalid scan time',

E9

233: 'Load is busy',

EA

234: 'An MP has re-educated',

EB

235: 'Invalid chassis or slot',

EC

236: 'Invalid SOE number',

ED

237: 'Invalid SOE type',

EE

238: 'Invalid SOE state',

EF

239: 'The variable is write protected',

F0

240: 'Node number mismatch',

F1

241: 'Command not allowed',

F2

242: 'Invalid sequence number',

F3

243: 'Time change on non-master TRICON',

F4

244: 'No free Tristation ports',

F5

245: 'Invalid Tristation I command',

F6

246: 'Invalid TriStation 1131 command',

F7

247: 'Only one chassis allowed',

F8

248: 'Bad variable address',

F9

249: 'Response overflow',

FA

250: 'Invalid bus',

FB

251: 'Disable is not allowed',

FC

252: 'Invalid length',

FD

253: 'Point cannot be disabled',

FE

254: 'Too many retentive variables',

FF

255: 'LOADER_CONNECT',

 

256: 'Unknown reject code'