How to Hide Malicious Containers in Kubernetes Using Kernel Hooking

TLDR: This research demonstrates how a custom rootkit, Kubekit, can hide malicious containers and processes from standard Kubernetes and Docker monitoring tools. By utilizing kernel-level hooking and user-space API manipulation, the tool effectively blinds administrators to unauthorized activity. Pentesters can use this technique to maintain persistence during red team engagements, while defenders must prioritize eBPF-based monitoring to detect such stealthy modifications.

Container orchestration platforms like Kubernetes are often treated as black boxes by security teams. We assume that if kubectl get pods doesn't show a container, it isn't running. This assumption is dangerous. The reality is that if an attacker gains root access to a node, they can manipulate the very tools we rely on for visibility. The research presented at Black Hat 2023 by Nicholas Lang and Andrew Hughes proves that hiding a container is trivial if you know how to hook the right system calls.

The Mechanics of Container Stealth

At its core, a rootkit is just code running where it shouldn't be, doing things it shouldn't do. In the context of Kubernetes, the goal is to intercept the communication between the container runtime and the API server. When you run a command like kubectl get pods, the API server queries the container runtime (like Docker or containerd) to list the active containers. If you can hook the functions responsible for returning that list, you can simply filter out your malicious payload before the API server ever sees it.

The researchers developed Kubekit to achieve this by targeting two distinct planes: the control plane and the data plane. By setting breakpoints in arbitrary files across namespaces, the tool can edit memory to hide files, processes, and even the rootkit itself. This is not just about deleting a log entry; it is about modifying the runtime's perception of reality.

Technical Implementation

The power of this technique lies in its ability to manipulate the JSON responses sent by the Kubernetes API. By hooking into the pre-TLS response-sending function, the rootkit can modify the data on the fly. If you are a researcher looking to replicate this, the process involves identifying the correct offsets for the target binary. Since Kubernetes binaries are often stripped, you might need to resolve symbols manually or use an offset if you are targeting a specific version.

Consider this simplified logic for hooking a process-listing function:

// Conceptual hook for process hiding
int hook_get_processes(void) {
    if (getenv("KUBEKIT_SECRET")) {
        return original_get_processes();
    }
    // Filter out the malicious process from the list
    return filter_malicious_process(original_get_processes());
}

The tool also uses a virtual device to enable logging and further interaction. If you are performing a penetration test, you would typically deploy this after achieving initial access via a vulnerable application or a misconfigured service account. Once you have the ability to load a kernel module or inject into a user-space process, you can effectively make your presence invisible to standard tools like ps or top within the container.

Real-World Applicability for Pentesters

During a red team engagement, the biggest challenge is maintaining persistence without triggering alerts. Most security monitoring in Kubernetes relies on agents that run as privileged containers or sidecars. These agents often query the same APIs that the rootkit is intercepting. If you can successfully hide your pod from the API server, you have effectively bypassed the primary detection mechanism for most organizations.

You will encounter this most often in environments where developers have excessive permissions or where the underlying node OS is not properly hardened. If a pod is running with privileged: true or has access to the host's process namespace, the barrier to entry for this kind of rootkit is non-existent. The impact is total control: you can run your C2 infrastructure, exfiltrate data, or pivot through the cluster, all while the dashboard shows a clean, healthy environment.

Defensive Strategies

Defending against this requires moving away from API-based monitoring and toward kernel-level observability. If you are relying solely on kubectl or standard logs, you are already behind. You need to implement tools that use eBPF to monitor system calls directly at the kernel level, bypassing any user-space hooks the attacker might have installed.

Falco is the industry standard for this. By writing custom rules that detect the loading of unauthorized kernel modules or suspicious system call patterns, you can catch the rootkit before it has a chance to hide itself. Pairing Falco with Falcosidekick allows you to trigger automated response actions, such as killing the offending pod or isolating the node, the moment a rule is violated.

Moving Forward

The lack of a publicly known, widely used Kubernetes rootkit does not mean they don't exist; it means we haven't been looking in the right places. As container security matures, we will see more sophisticated threats that target the orchestration layer rather than the application code. If you are responsible for securing a cluster, stop trusting the output of your management tools. Start verifying the state of your nodes at the kernel level. The next time you see a "clean" cluster, ask yourself if you are seeing the truth or just what the runtime wants you to see.