Fuzzing the WebKit JIT: How Custom Compilers Uncover Memory Corruption

TLDR: Researchers at Black Hat 2023 demonstrated a novel approach to finding memory corruption in WebKit by building a custom fuzzer, Ruby, which extends AFL++ to generate semi-well-formed WebAssembly modules. This technique successfully identified critical vulnerabilities, including type confusion and stack overflow bugs, in the WebAssembly compiler components. Pentesters and researchers should adopt these targeted, structure-aware fuzzing strategies to uncover deep-seated flaws in complex JIT environments.

Modern browser security relies on the assumption that the Just-In-Time (JIT) compiler is a black box that correctly translates high-level code into machine instructions. When that assumption fails, the result is usually a memory corruption vulnerability that leads to arbitrary code execution. The research presented at Black Hat 2023 on the WebAssembly compiler within WebKit proves that these components are not just complex, but highly susceptible to bugs that standard black-box fuzzing misses.

The Challenge of Fuzzing JIT Compilers

Standard fuzzing often fails against JIT compilers because the input must be syntactically valid enough to pass the initial parser, yet malformed enough to trigger edge cases in the optimization phases. If the fuzzer generates completely random bytes, the parser rejects the input immediately. If it generates perfectly valid code, it never reaches the deep optimization logic where the most interesting bugs reside.

The researchers addressed this by creating Ruby, a custom fuzzer built on top of AFL++. By integrating wasm-smith, a tool for generating valid WebAssembly modules, they created a pipeline that produces "semi-well-formed" samples. These samples are structured enough to reach the compiler's optimization phases but contain specific mutations designed to stress-test the compiler's internal logic.

Deep Dive: Type Confusion and Stack Overflow

The research uncovered several high-impact vulnerabilities, specifically CVE-2022-32863, CVE-2022-32885, and CVE-2022-32886. These bugs illustrate the danger of improper state management during the compilation process.

Take CVE-2022-32863 as an example. The vulnerability exists in the BBQ (Build Bytecode Quickly) compiler tier. The root cause was an improper implementation of the stack overflow check. When the compiler generates the prologue for a function, it uses a temporary register to calculate the stack pointer. If the stack limit is exceeded, the compiler triggers an exception handler. However, the researchers found that the handler was invoked before the object instance was fully initialized. This led to a type confusion scenario where the engine operated on uninitialized memory, allowing an attacker to manipulate the execution flow.

The fix, while seemingly straightforward, highlights the complexity of these systems. The patch required ensuring that the stack overflow check occurs only after the object instance is properly initialized, preventing the engine from ever reaching the exception handler with a "dirty" state.

Why This Matters for Pentesters

For those of us performing browser-based research or red team engagements, this work is a masterclass in target-specific fuzzing. If you are looking for bugs in a browser engine, you cannot rely on generic tools. You must understand the internal IR (Intermediate Representation) generation and the specific optimization passes of the target compiler.

The researchers demonstrated that by logically dividing the fuzzer into a generator and a harness, they could reuse the generator across different compiler tiers like LLInt, BBQ, and OMG. This modularity is key. If you are auditing a browser, don't just throw payloads at the DOM. Look at how the engine handles complex WebAssembly modules. The OWASP Memory Corruption documentation provides a good baseline for understanding the mechanics, but the real work happens in the compiler's optimization phases.

Defensive Considerations

From a defensive perspective, the takeaway is that JIT compilers are an enormous, high-risk attack surface. Browser vendors are increasingly moving toward more memory-safe languages for these components, but the legacy C++ codebases in engines like WebKit remain a primary target. Organizations should prioritize keeping browser versions updated, as these vulnerabilities are often exploited in the wild before a patch is even fully understood by the broader community.

If you are building a security program, ensure that your threat modeling accounts for the browser as a primary execution environment. The complexity of JIT compilation means that bugs are inevitable. The goal is to reduce the window of opportunity for an attacker to turn a compiler bug into a full sandbox escape.

The next time you are looking at a browser-based target, stop treating the JIT as a black box. Start looking at the compiler's IR generation and the way it handles edge cases in the language specification. The bugs are there, hidden in the optimization passes, waiting for someone to generate the right kind of semi-well-formed input.