Automating Defense: Lessons from Reinforcement Learning in Military OT

TLDR: Researchers at Black Hat 2024 demonstrated how reinforcement learning (RL) agents can autonomously defend military industrial control systems against network-based attacks. By training agents in simulated environments and using a middleware layer to translate abstract actions into real-world commands, they successfully mitigated RDP-based denial-of-service attacks in under one second. This research highlights a shift toward autonomous, machine-speed response capabilities for critical infrastructure where human operators are often overwhelmed or unavailable.

Defending operational technology (OT) environments is a losing game when you rely solely on human intervention. By the time an operator notices an anomaly in a maritime control system or a power grid, the adversary has already achieved their objective. The research presented at Black Hat 2024 on Autonomous Resilient Cyber Defence (ARCD) moves past the theoretical, showing how reinforcement learning agents can be deployed to provide machine-speed responses to network-based threats.

The Mechanics of Autonomous Defense

At the core of this research is the use of multi-agent reinforcement learning (MARL) to manage complex, interconnected systems. Unlike traditional rule-based intrusion detection systems that trigger static alerts, these RL agents are trained to observe the environment, detect malicious patterns, and execute defensive actions. The researchers utilized PyTorch and Gymnasium to build high-fidelity simulations of maritime systems, specifically focusing on the Integrated Platform Management System (IPMS).

The agents operate in a loop: they observe the state of the network, receive a reward based on the success of their defensive actions, and update their policy accordingly. The challenge, as any researcher who has worked with RL knows, is the "sim-to-real" gap. A model that performs perfectly in a clean, simulated environment often fails when faced with the noise and uncertainty of a production network. To bridge this, the team implemented a middleware layer that acts as a translator. It takes the abstract "isolate node" command from the RL agent and maps it to the specific API calls required by the target hardware, such as a PLC or a network switch.

Addressing the Generalization Problem

One of the most significant hurdles in applying machine learning to security is the tendency for models to overfit to specific training scenarios. If you train an agent to defend against a specific Metasploit module, it will be blind to a slightly modified payload or a different attack vector. The researchers tackled this by using curriculum learning. Instead of throwing the agent into a complex, chaotic environment, they started with simple tasks and gradually increased the difficulty.

They also employed action masking to prevent the agent from attempting nonsensical or dangerous actions that could inadvertently cause a self-inflicted denial-of-service. By masking out invalid actions during the training phase, the agent learns to focus on effective defensive strategies. This is critical for OT environments where a misconfigured firewall rule can shut down a propulsion system or a navigation sensor.

Real-World Applicability for Pentesters

For those of us conducting penetration tests on industrial control systems, this research provides a glimpse into the future of our target environments. We are moving toward systems that can actively fight back. During a test, you might find that your standard Cobalt Strike beacon or Metasploit exploit is suddenly blocked or isolated by an autonomous agent before you can even move laterally.

The impact of this is clear: the "smash and grab" approach to OT exploitation will become increasingly difficult. Pentesters will need to focus on identifying the logic gaps in these autonomous systems. If an agent is trained to block RDP-based DoS attacks, how does it handle a more subtle, low-and-slow data exfiltration attempt? Understanding the defensive policy of these agents will become a new phase of the engagement, similar to how we currently map out EDR configurations.

The Defensive Reality

Defenders should view this as a force multiplier rather than a replacement for security operations. The goal of the ARCD program is to provide "cyber first aid." It is about buying time for a human expert to assess the situation and perform a full recovery. If an agent can contain a Denial of Service attack in under a second, it prevents the system from crashing, which is the primary objective in an OT environment.

However, the risk of adversarial machine learning remains. If an attacker can identify the features the agent uses to make its decisions, they can craft adversarial inputs to bypass the defense. We are already seeing research into how these agents can be manipulated, and it is only a matter of time before these techniques are weaponized against autonomous defensive systems.

The path forward requires a focus on explainability. If an autonomous agent isolates a critical system, the human operator needs to know why. Without that context, the defense is just as dangerous as the attack. As we continue to integrate AI into our defensive stacks, we must ensure that we are not just building faster systems, but smarter, more transparent ones that can be audited and understood by the people responsible for the safety of the infrastructure. Keep an eye on the CAGE Challenge repositories; the datasets and environments being released there are the best starting point for anyone looking to experiment with these autonomous defense concepts.