How AI-as-a-Service Platforms Are Accidentally Exposing Your Private Models and Cloud Secrets

TLDR: Researchers at Black Hat 2024 demonstrated that AI-as-a-Service (AIaaS) platforms like Hugging Face, Replicate, and SAP AI Core often fail to properly isolate user-provided models, leading to Remote Code Execution (RCE) and lateral movement. By crafting malicious model files or containers, attackers can bypass sandbox restrictions to access private datasets, source code, and cloud credentials. This research proves that AI models must be treated as untrusted code, and infrastructure providers must implement strict, defense-in-depth isolation to prevent cross-tenant compromise.

The industry is currently obsessed with prompt injection, but while everyone is busy trying to trick LLMs into saying something offensive, the real danger is hiding in the infrastructure itself. If you are a researcher or a developer, you likely treat AI models as static data files. This is a dangerous assumption. As demonstrated by the Wiz Research team at Black Hat 2024, many AI-as-a-Service platforms treat these models as executable code, creating a massive, often overlooked attack surface for Remote Code Execution (RCE) and lateral movement.

The "Pickle" Problem and Beyond

The core issue is that AI models are rarely just raw data. They are often serialized objects. The most common format, the Python Pickle, is inherently insecure because it allows for arbitrary code execution during the deserialization process. The official documentation explicitly warns that you should never unpickle data from an untrusted source, yet AI platforms frequently do exactly that to load user-uploaded models.

The researchers showed that by crafting a malicious pickle file, they could gain RCE on the underlying infrastructure. The attack flow is straightforward: you create a model that contains a payload, upload it to a platform like Hugging Face, and wait for the platform’s inference engine to load it. Once the engine deserializes the model, your code executes with the privileges of the service account running the container.

This is not limited to pickle files. The researchers found similar issues with other formats:

• TensorFlow Lambda Layers: These can contain arbitrary Python functions that execute when the model is loaded.
• Replicate’s Cog: This format uses Docker containers. If the platform doesn't properly restrict the container's capabilities, you are essentially running a full-blown container escape scenario.
• Mozilla’s Llamafile: This is a self-contained executable, which makes it an obvious vector for direct code execution.

Lateral Movement in Managed Kubernetes

Gaining RCE is only the first step. The real impact comes from what you can reach once you are inside the platform's environment. In many cases, these AIaaS providers run their inference engines on managed Kubernetes clusters, such as AWS EKS.

The researchers demonstrated a clever privilege escalation technique using the Instance Metadata Service (IMDS). If a pod is misconfigured or lacks proper network policies, an attacker can query the IMDS at 169.254.169.254 to retrieve the node's IAM role credentials. By leveraging these credentials, they can often move laterally to access other services, such as S3 buckets containing private datasets or EFS file shares holding sensitive configuration files.

In one instance, they discovered an exposed Istio service mesh configuration. Istio is designed to manage traffic, but it can also act as a firewall. The researchers found that Istio’s traffic rules explicitly excluded its own user ID (1337) from certain restrictions. By setting their process's user ID to 1337, they bypassed the service mesh's network controls, allowing them to scan the internal network and reach services that should have been isolated.

Why This Matters for Pentesters

If you are performing a penetration test on an organization that uses AI, stop looking only at the web application layer. Start looking at the model supply chain. Ask the following questions:

1. Where are the models stored?
2. What format are they in?
3. Does the platform perform any static analysis or sandboxing before loading the model?
4. What permissions does the inference service account have within the cloud environment?

During an engagement, you can test for these vulnerabilities by attempting to upload a "benign" model that performs a simple outbound network request (like a DNS lookup or an HTTP call to a Burp Collaborator instance). If you get a hit, you have confirmed that the platform is executing your code. From there, you can attempt to enumerate the environment, look for mounted secrets, or test for network egress.

Defensive Strategies

Defending against these attacks requires a shift in mindset. You must treat every AI model as untrusted code.

• Isolation: Use dedicated, hardened environments for model inference. Ensure that these environments have no access to sensitive cloud metadata or internal network resources.
• Format Selection: Move away from insecure formats like Pickle. Prioritize safer, data-only formats like SafeTensors or GGUF, which do not support arbitrary code execution.
• Least Privilege: Ensure that the service accounts running your inference engines have the absolute minimum permissions required. Use Kubernetes Network Policies to restrict traffic between pods and prevent lateral movement.

The era of "AI security" being limited to prompt injection is over. As these platforms become more integrated into enterprise workflows, the infrastructure supporting them becomes a high-value target. If you are building or testing these systems, assume the model is the exploit.