Automating Bug Bounties with LLM Alloy Models

TLDR: Researchers at DEF CON 2025 demonstrated an autonomous penetration testing framework that uses a coordinator-agent architecture to scale vulnerability discovery across thousands of targets. By implementing "alloy models"—a technique for dynamically mixing LLMs based on target context—they significantly reduced false positives and improved detection rates for complex bugs. This approach shifts the paradigm from manual, tool-heavy testing to intelligent, agent-driven research that can handle everything from XXE to business logic flaws at scale.

Security research is hitting a wall of scale. When you are staring down a massive bug bounty program with thousands of endpoints, the traditional "scan and pray" approach using standard vulnerability scanners is no longer enough. You end up with a mountain of noise, a sea of false positives, and very little time to actually dig into the high-impact business logic flaws that pay the bills. The recent research presented at DEF CON 2025 on autonomous penetration testing shows exactly how we can move past this.

The Architecture of an Autonomous Pentester

The core of this research is a coordinator-agent architecture. Think of the coordinator as the lead pentester on an engagement. It manages the high-level strategy, analyzes the target application, performs initial reconnaissance, and assigns specific tasks to specialized agents. These agents are essentially LLMs fine-tuned or prompted to look for specific vulnerability classes, such as XXE, SQL Injection, or Server-Side Request Forgery.

What makes this setup effective is the inclusion of "validators." In any automated system, the biggest enemy is the false positive. If your agent reports a vulnerability every time it sees a generic error message, you are wasting your time. The validators act as a second pair of eyes, cross-referencing the findings and filtering out the noise. This is how the researchers managed to successfully identify and report a critical CVE-2025-30220 in GeoServer. They didn't just throw a payload at the wall; they used a structured, agent-based workflow to confirm the vulnerability before ever touching a human-readable report.

Why Alloy Models Matter

One of the most interesting technical takeaways is the concept of "alloy models." If you have ever worked with LLMs for security tasks, you know that no single model is the silver bullet. One model might be brilliant at spotting Cross-Site Scripting but completely blind to complex Business Logic flaws.

Instead of forcing one model to do everything, the researchers implemented a system that dynamically selects and mixes models based on the context of the target. If the coordinator identifies a potential injection point, it might route the task to a model known for its prowess in payload generation. If it detects a complex authentication flow, it switches to a model better suited for logic analysis. This mixing—or "alloying"—of models increased their detection rates from roughly 25% to 55%. It is a simple but powerful way to get more out of the current generation of LLMs without needing to retrain them from scratch.

Practical Application for Pentesters

For those of us in the trenches, this isn't just academic. The real-world application here is about resource management. When you are on a time-boxed engagement, you cannot manually fuzz every single parameter on every single endpoint. By using an autonomous framework, you can offload the repetitive, low-level discovery tasks to these agents.

For example, when testing a web application, you can point the framework at the target and let it handle the initial discovery of Path Traversal or Server-Side Template Injection while you focus on the deeper, more creative exploitation paths that require a human touch. The framework uses tools like Puppeteer to interact with the application, allowing it to handle modern, JavaScript-heavy frontends that often break traditional scanners.

The Defensive Reality

Defenders should take note: the era of automated, intelligent exploitation is here. If your security controls rely on static signatures or simple rate limiting, you are going to be bypassed by these agents. The researchers emphasized that their framework is designed to be "traceable," meaning it includes custom HTTP headers in its traffic. This is a courtesy for bug bounty programs, but in a real-world attack, you won't get that. To defend against this, you need to focus on behavioral analysis and anomaly detection that can identify the patterns of an agent trying to "reason" through your application's logic, rather than just looking for known malicious payloads.

What Comes Next

The most impressive part of this research is the honesty about its limitations. The team explicitly stated that they do not perform post-exploitation. Once they confirm a vulnerability, they stop. This is the right move for a research project, but it leaves the door open for the next generation of tools. If you are looking to build your own agentic security tools, start by focusing on the coordinator logic. The ability to parse a target's scope and prioritize the "juiciest" endpoints is where the real value lies.

Stop trying to build a single, all-knowing model. Start building systems that can orchestrate specialized agents and validate their findings. The future of offensive security isn't in bigger models; it is in better orchestration. Go look at the OWASP Top 10 and ask yourself which of those categories could be handled by a specialized agent today. That is your starting point.