Detecting EDR-Evading Malware Through Memory Forensics

TLDR: Modern EDRs are increasingly blind to sophisticated malware that uses module unhooking, process hollowing, and direct system calls to operate in memory. By analyzing memory structures like thread states and system call implementations across processes, researchers can identify these stealthy techniques. This post details how to use new Volatility 3 plugins to automate the detection of these evasions in enterprise environments.

Security researchers often operate under the assumption that if an EDR is active and reporting as healthy, the endpoint is secure. This is a dangerous misconception. The latest wave of malware doesn't just avoid disk-based detection; it actively subverts the monitoring mechanisms that EDRs rely on to maintain visibility. When you are performing a red team engagement or conducting incident response, you cannot trust the EDR's dashboard. You need to look at the raw state of physical memory to see what is actually happening.

The Reality of EDR Evasion

Most EDRs monitor system activity by hooking functions within ntdll.dll. When a process makes a system call, the EDR intercepts it, inspects the parameters, and decides whether to allow or block the action. Malware authors have responded by implementing techniques to bypass these hooks.

One common method is module unhooking. The malware reads a clean copy of ntdll.dll from disk, finds the hooked functions, and overwrites the EDR's hooks with the original, clean bytes. Once the hooks are removed, the malware can execute system calls directly without the EDR ever knowing. Another technique involves process hollowing, where a legitimate process is started in a suspended state, its memory is replaced with malicious code, and it is then resumed. Because the EDR often hooks the process creation event, it misses the subsequent injection and execution.

Detecting the Unhooking

The key to detecting module unhooking is realizing that the EDR's hooks are usually only removed from the specific processes where the malware is active. In a typical enterprise environment, you might have hundreds of processes running. If you compare the implementation of system call handlers across all of them, you will find that most share the same hooked state. The outliers—the processes where the hooks have been removed—are your primary targets for investigation.

We have developed a new plugin for Volatility 3 that automates this comparison. It enumerates all running processes, reads the bytes of critical system call handlers, and flags any process that deviates from the expected, hooked implementation. This approach is agnostic to the specific EDR vendor because it focuses on the state of the memory rather than the specific hooking technique used.

Identifying Process Hollowing and Suspicious Threads

Process hollowing is another persistent threat that often evades standard telemetry. When a process is created in a suspended state, it is assigned a thread with a specific suspend count. A legitimate process created by the user or the system will have a suspend count of zero. A process created by malware for hollowing will have a suspend count of one.

Our research shows that by checking the KTHREAD.SuspendCount field within the ETHREAD structure, we can identify these hollowed processes instantly. We integrated this check into a Volatility 3 plugin that scans all threads across all processes. If a thread has a suspend count greater than zero, it is flagged. While there are rare false positives, such as certain browser processes, this method is highly effective at surfacing malicious activity that would otherwise remain hidden.

Direct System Calls and Call Stack Spoofing

Advanced malware has moved beyond simple unhooking to using direct system calls. Instead of calling the function in ntdll.dll, the malware executes the syscall instruction directly. This bypasses the hooked function entirely. To counter this, EDRs have started performing call stack examination. They inspect the return address on the stack to ensure the system call originated from a legitimate, signed module.

Malware authors have countered this with call stack spoofing, as seen in projects like DodgeBox. By manipulating the stack, the malware makes it appear as though the system call originated from a trusted binary rather than the malicious code. However, even with stack spoofing, the malware must still set up the registers correctly before executing the syscall instruction.

Our approach to detecting this is to look for the specific code patterns required to set up these registers—specifically the RAX or EAX register for the system call index and the R10 register for the parameters—outside of the expected memory regions. By scanning memory for these patterns, we can identify direct system call usage regardless of how well the call stack is spoofed.

Moving Forward

Memory forensics is not just a post-mortem tool; it is a critical component of proactive threat hunting. The techniques described here allow you to cut through the noise and identify the presence of sophisticated threats that have successfully bypassed your primary security controls. If you are not already incorporating memory analysis into your standard testing or response workflow, you are missing a significant portion of the attack surface. Start by auditing your environment for these anomalies and you will likely find that your visibility into endpoint activity is far lower than you previously believed.