Weaponizing LLMs to Build Deceptive Web Honeypots

TLDR: Traditional static honeypots are easily fingerprinted and bypassed by automated scanners, limiting their effectiveness in modern threat detection. By integrating Large Language Models (LLMs) into a web honeypot, researchers can now generate dynamic, context-aware responses that mimic real vulnerabilities and services. This approach forces attackers to waste time and resources interacting with fake endpoints, providing defenders with high-fidelity telemetry on attacker behavior.

Static honeypots are a relic of a simpler era. If you have spent any time running a listener on a public-facing IP, you know that within minutes, your logs are flooded with automated noise from scanners like Shodan or Censys. Most of these tools identify a honeypot by its predictable, static responses. Once an attacker realizes they are interacting with a fake service, they move on, and you lose the chance to observe their actual TTPs. The game has changed, and we need to stop relying on hardcoded responses that any half-decent script can detect.

Moving Beyond Static Responses

The research presented at DEF CON 2024 on Galah shifts the paradigm from static emulation to dynamic, LLM-powered deception. Instead of serving a pre-written file, Galah uses an LLM to analyze incoming HTTP requests and generate a response that appears to be a legitimate, vulnerable application.

When an attacker sends a request, Galah processes it through a rule-based engine. If the request matches a known pattern, it can serve a static response to save on API costs or latency. If the request is novel or requires more complex deception, it is passed to an LLM. The model is instructed to act as a specific target application, such as a vulnerable instance of Palo Alto GlobalProtect, which recently saw significant exploitation in the wild.

The power of this approach lies in the realism. Because the LLM generates the response on the fly, it can handle unexpected inputs, path traversal attempts, or even bizarre payloads that a static script would simply reject or ignore. For a pentester, this means the "target" feels alive. It responds to your directory brute-forcing with plausible 404s or even fake sensitive files, keeping you engaged long enough to reveal your intent.

The Mechanics of LLM Deception

Implementing this requires careful prompt engineering. You cannot just feed raw requests into a model and hope for the best. The Galah project demonstrates that you must explicitly instruct the model to ignore instructions contained within the attacker's input to prevent prompt injection, which is a critical OWASP Top 10 concern for any LLM-integrated application.

A typical interaction flow looks like this:

curl -i http://<honeypot-ip>/etc/passwd

The honeypot receives this request, wraps it in a system prompt that defines the persona, and sends it to the LLM. The prompt includes instructions to:

1. Analyze the HTTP request.
2. Emulate the target application.
3. Generate a realistic response without revealing that it is a honeypot.

The challenge, as the research highlights, is getting the LLM to output valid, clean JSON or HTML. Models often hallucinate extra characters or include conversational filler that breaks the illusion. Galah addresses this by using a post-processing layer that validates the JSON output and strips away any unwanted artifacts before sending the response back to the attacker.

Real-World Implications for Pentesters

For those of us conducting red team engagements or bug bounty research, this technology introduces a new layer of uncertainty. When you encounter a service that seems to have a high-severity vulnerability, you usually jump straight to exploitation. If that service is actually an LLM-powered honeypot, your "exploit" might just be feeding data into a model that is logging your every move.

During the talk, the researcher demonstrated how they successfully tricked attackers into believing they had found an open HTTP proxy. The honeypot returned a "connection established" message, leading the attackers to route their traffic through the fake proxy. This is a massive win for defenders. By observing the traffic routed through the proxy, they can identify the attacker's true source IP, their user agents, and the specific targets they are trying to reach.

Defensive Considerations

Defenders should view this as a way to increase the cost of attack. If an attacker has to spend time determining whether a target is real or a sophisticated honeypot, they are less efficient. However, this is not a silver bullet. LLM-based honeypots are resource-intensive and require careful monitoring of API usage to avoid massive bills. If you are deploying this, you must implement strict API usage limits and ensure the honeypot itself is isolated from your internal network.

The most interesting takeaway from this research is the "honesty" of different models. When asked directly if they were a honeypot, some models were surprisingly candid, while others doubled down on the deception. This suggests that the choice of model—whether you use a commercial provider or a local model via Ollama—significantly impacts the quality of your deception.

If you are building your own detection infrastructure, stop relying on static signatures. Start looking at how you can use these models to create environments that are not just hard to break, but hard to read. The next time you are on an engagement, take a second look at that "vulnerable" server. It might be a parrot, and it might be watching you back.