How Malicious Models on Hugging Face Turn Into Self-Propagating Worms

TLDR: Researchers at Black Hat 2024 demonstrated that the Hugging Face hub is vulnerable to remote code execution (RCE) via insecure deserialization of pickle files. By exploiting third-party model libraries that lack secure coding practices, attackers can bypass existing platform-level scanning to execute arbitrary code. This research highlights a critical supply chain risk where compromised models can be weaponized into self-propagating worms that infect downstream users.

Machine learning model hubs have become the central repository for AI development, but they are currently operating with a massive, unaddressed security blind spot. The industry has spent years hardening web applications against insecure deserialization, yet we are now blindly pulling serialized Python objects—specifically pickle files—from public hubs and executing them in our local environments. This research from Black Hat 2024 proves that this practice is a recipe for disaster, turning the convenience of pre-trained models into a vector for remote code execution and worm-like propagation.

The Mechanics of the Pickle Vulnerability

At the heart of this issue is the pickle module in Python. It is not a secure format, and it was never intended to be. When you call pickle.load(), the interpreter executes arbitrary code defined within the serialized stream. While Hugging Face has implemented scanning mechanisms to detect known malicious patterns, these defenses are fundamentally flawed because they rely on blacklists or incomplete whitelists that fail to account for the sheer variety of third-party libraries integrated into the hub.

The vulnerability arises because many popular machine learning libraries, such as those found in the Transformers ecosystem, do not always use secure loading practices. When a researcher or developer downloads a model, they often execute code that implicitly calls pickle.load() on files provided by the model author. If an attacker uploads a model containing a crafted pickle payload, they gain execution on the victim's machine the moment the model is loaded.

Bypassing the Scanners

The research identifies three primary techniques to bypass the platform's current security controls:

1. Code Reuse: Attackers can identify "orange" functions—functions that are not explicitly blacklisted but can be chained together to achieve RCE. By reusing existing, benign-looking code paths within the model's dependencies, an attacker can construct a payload that the scanner ignores.
2. Configuration Abuse: Many models load configuration files that dictate how the model is initialized. By manipulating these files, an attacker can force the library to load a malicious pickle file from a different, attacker-controlled repository, effectively redirecting the execution flow.
3. Format Encoding: Scanners often look for specific file signatures. By encoding the malicious pickle data using Base64 or other formats, attackers can hide the payload from simple pattern-matching scanners, only for the model library to decode and execute it during the initialization phase.

Weaponizing the Worm

The most alarming aspect of this research is the demonstration of a self-propagating worm. An attacker can create a model that, upon execution, performs a search for other models on the victim's local machine, modifies them to include the same malicious payload, and then uses the victim's credentials to push the infected models back to the Hugging Face hub.

This creates a cycle of infection. A user downloads a "useful" model, their environment is compromised, and their own account is then used to distribute the infection to every other user who downloads their models. This is a classic T1203 exploitation scenario, where the client-side execution is triggered by the very tools intended to accelerate development.

Practical Implications for Pentesters

If you are performing a security assessment on an organization that relies heavily on AI, stop looking only at their web APIs. Start auditing their model ingestion pipeline. Ask the following questions:

• Where are the models coming from?
• Are they being scanned with tools like Picklescan or Modelscan before they hit the production environment?
• Is there a sandbox in place for model evaluation?

The impact of a successful exploit is total system compromise. Once you have RCE, you have access to the environment's secrets, API keys, and the ability to exfiltrate sensitive training data.

Defensive Realities

Defending against this is difficult because the problem is architectural. The industry needs to move away from pickle entirely. OWASP has long warned about the dangers of insecure deserialization, and the AI community is finally learning this lesson the hard way.

In the short term, organizations must treat every pre-trained model as untrusted code. Do not run load_from_hub or similar functions in a privileged environment. Use a dedicated, isolated sandbox for model testing, and implement strict egress filtering to prevent the model from "phoning home" or reaching out to the hub to propagate itself. If you must use these models, perform a manual code review of the loading logic. If the library uses pickle.load(), assume it is vulnerable. The convenience of a pre-trained model is not worth the risk of a persistent, self-replicating backdoor in your infrastructure.