Hardening HSMs: Why Your Banking-Grade Crypto Wallet Might Be Less Secure Than You Think

TLDR: Hardware Security Modules (HSMs) are often treated as black-box "get out of jail free" cards for cryptographic security, but they are far from impenetrable. Research presented at Black Hat 2024 highlights how flaws in PKCS#11 implementations and the risks of custom firmware modules can lead to full key compromise. Pentesters should stop treating HSMs as out-of-scope and start auditing the integration layer, specifically looking for insecure API usage and weak access control logic.

Hardware Security Modules are the industry standard for protecting high-value cryptographic keys. When a bank or a crypto custodian needs to sign transactions or manage millions in assets, they drop an HSM into the rack and assume the "banking-grade" label handles the security. This assumption is dangerous. Recent research into YubiHSM and broader PKCS#11 implementations shows that the security of these devices is only as strong as the code interacting with them. If you are performing a red team engagement or a deep-dive security assessment, the HSM is no longer a "do not touch" zone. It is a critical target.

The Illusion of the Black Box

Most security teams view the HSM as a monolithic, tamper-proof vault. The reality is that these devices are essentially specialized computers running a minimal Linux distribution, often with a PowerPC core and dedicated crypto accelerators. The attack surface is not just the physical hardware; it is the software stack, the API, and the custom modules that developers load onto the device to handle business logic.

When developers need to perform complex operations—like enforcing multi-signature quorum rules or verifying transaction payloads—they often write custom firmware modules. These modules run directly on the HSM. If your code has a buffer overflow or a logic error, you are not just crashing a web server; you are potentially exposing the raw memory space where cryptographic keys reside. The CVE-2023-39908 vulnerability in the YubiHSM PKCS#11 library is a perfect example. It demonstrated that even widely used, vetted libraries can have implementation flaws that allow an attacker to bypass intended security constraints.

Auditing the Integration Layer

For a pentester, the most fruitful path is rarely trying to physically bypass the HSM's anti-tampering sensors. Instead, focus on the PKCS#11 interface. This API is notoriously complex and prone to Broken Access Control issues.

When you are testing an application that talks to an HSM, look for how it handles key attributes. Developers often fail to properly set the CKA_SENSITIVE and CKA_EXTRACTABLE flags. If a key is marked as CKA_EXTRACTABLE, an attacker with sufficient privileges can pull the key material out of the HSM. Even if the key is not extractable, an attacker might be able to use the HSM as an "oracle" to sign arbitrary data if the application does not strictly validate the transaction payload before sending it to the device.

Consider this common pattern in insecure integrations:

// Insecure: The HSM signs whatever it is told to sign
CK_RV rv = C_Sign(hSession, pData, ulDataLen, pSignature, &ulSigLen);

If the application does not implement a custom module to validate that pData is a legitimate, authorized transaction, the HSM becomes a high-speed signing machine for an attacker. The fix is to move business logic into the HSM itself, ensuring that the device only signs data that meets specific, hard-coded criteria.

State Management and Merkle Trees

One of the most significant challenges in hardening HSMs is state management. If you need to track millions of users or manage a dynamic list of authorized signers, you cannot store that state in the HSM's limited memory. Developers often offload this state to an external database, which introduces a massive security gap. If the database is compromised, the HSM's security is effectively nullified.

The research suggests a more robust approach: using Merkle trees to manage state. By encoding the state as a tree and storing only the root hash inside the HSM, you can verify the integrity of any state component with a simple Merkle proof. This allows the HSM to remain stateless while still providing cryptographic assurance that the data it is processing has not been tampered with. This technique is particularly effective for high-assurance systems where you need to perform state transitions on large datasets without exceeding the HSM's memory constraints.

Moving Beyond the "Banking-Grade" Label

Defenders must stop relying on the vendor's certification as a substitute for a secure architecture. If you are deploying an HSM, your primary focus should be on minimizing the attack surface. This means disabling all unused PKCS#11 functions, enforcing strict access control at the application level, and ensuring that any custom firmware is built using a secure, reproducible pipeline.

For those of you on the offensive side, the next time you encounter an HSM in an environment, do not assume it is secure. Audit the API calls, look for logic flaws in the custom modules, and investigate how the application manages state. The most critical vulnerabilities are rarely in the crypto itself; they are in the messy, human-written code that sits between the application and the vault.