Beyond the Hype: Why LLMs Are Currently Failing Security Research

TLDR: This panel discussion at Black Hat 2024 cuts through the noise surrounding AI in security, highlighting that while LLMs can accelerate basic tasks, they are currently prone to hallucinations and data poisoning. The experts argue that AI is a tool for augmentation rather than a replacement for human expertise, especially in complex hardware and software analysis. Pentesters should treat AI-generated code and analysis as untrusted input that requires rigorous manual verification.

Security research is currently obsessed with finding ways to force LLMs to do the heavy lifting. We see the same pattern every few years: a new technology arrives, the industry collectively loses its mind, and then we spend the next decade figuring out how to actually use it without breaking everything. The current obsession with using AI for vulnerability discovery and code analysis is no different. While the potential for automation is high, the reality of using these models for high-stakes security work is far more nuanced than the marketing suggests.

The Hallucination Problem in Security Work

One of the most significant hurdles identified by the panel is the inherent unreliability of LLMs when tasked with complex security analysis. When you ask a model to analyze a piece of code for a vulnerability, you are essentially asking a probabilistic engine to perform a deterministic task. The result is often a convincing, yet technically flawed, explanation.

For a researcher, this is dangerous. If you are already deep in the weeds of a complex Linux kernel exploit or a proprietary hardware interface, you cannot afford to waste time chasing ghosts generated by a hallucinating model. The panel noted that these models are trained on vast datasets that include outdated or insecure coding practices. Consequently, they often suggest "fixes" that introduce new vulnerabilities or fail to address the underlying issue. If you are using ChatGPT or similar tools to assist in your workflow, you must treat every output as a potential OWASP A03:2021-Injection risk or a logic error waiting to happen.

Data Poisoning and the Trust Gap

The panel raised a critical point regarding the data used to train these models. If an attacker can influence the training data, they can effectively "poison" the model's understanding of what constitutes secure code. This is not just a theoretical concern. As we rely more on AI to generate boilerplate code or analyze dependencies, we are creating a massive, centralized attack surface.

Consider the implications for a bug bounty hunter. If you are using an AI-powered tool to scan a target, and that tool has been trained on poisoned data, it might intentionally ignore specific patterns or misclassify critical vulnerabilities as false positives. This creates a false sense of security that is far worse than having no tool at all. The panel emphasized that the lack of transparency in how these models are trained and updated makes it nearly impossible to verify their integrity in a professional engagement.

Hardware Research: Where AI Hits a Wall

Hardware security remains the final frontier where human expertise is still vastly superior to current AI capabilities. Analyzing a chip for side-channel vulnerabilities or reverse-engineering a proprietary protocol requires a deep understanding of physical constraints and low-level logic that LLMs simply do not possess.

The speakers pointed out that even when AI is used to assist in tasks like Whisper-based transcription of technical documentation or basic script generation, the final verification must always be done by a human. There is no "AI-only" path to finding a zero-day in a modern SoC. The complexity of the hardware, combined with the lack of accessible, high-quality training data for specific architectures, means that AI will remain a secondary tool for the foreseeable future. If you are working on hardware security, your time is better spent mastering your oscilloscope and logic analyzer than trying to prompt-engineer your way to an exploit.

The Future of Security Tooling

We are moving toward a model where AI acts as a force multiplier for specific, well-defined tasks. The panel suggested that the most effective use of AI in security is in the "boring" parts of the job: summarizing logs, generating test cases for known vulnerability patterns, or translating documentation.

However, the industry needs to be realistic about the limitations. We are not at the point where we can hand off a target to an AI and expect a comprehensive report. The risk of T1190-Exploit Public-Facing Application or T1566-Phishing remains a human-centric problem that requires human-centric solutions.

If you are a pentester, start by building your own validation pipelines. Never copy-paste code from an LLM directly into your exploit chain without a full audit. The most successful researchers are those who use AI to clear the path of low-level noise, allowing them to focus their limited time on the complex, high-value targets that require genuine human intuition. The hype will eventually fade, but the need for rigorous, manual verification will remain the cornerstone of our profession. Don't let the convenience of a chat interface compromise your methodology.