Why WeChat’s Custom Encryption Protocol is a Masterclass in Security Through Obscurity

TLDR: Researchers at the Citizen Lab analyzed WeChat’s proprietary MMTLS encryption protocol and discovered a double-layered scheme that lacks forward secrecy and relies on a forgeable checksum. By reverse-engineering the underlying Mars library, the team exposed how these custom implementations create significant blind spots for traffic analysis and interception. This research serves as a stark warning for developers and security auditors: proprietary crypto is almost always a liability compared to industry-standard implementations like TLS 1.3.

Security researchers often talk about the dangers of "rolling your own crypto," but rarely do we get such a clear, high-profile example of why it fails. The recent analysis of WeChat’s MMTLS protocol presented at Black Hat Asia 2025 confirms what many of us have suspected for years: when massive applications bypass standard protocols like TLS 1.3 in favor of custom, proprietary stacks, they aren't just adding complexity. They are actively introducing vulnerabilities that would have been mitigated by using battle-tested, open-source libraries.

The Anatomy of a Double-Layered Failure

WeChat’s network traffic is not just encrypted once; it is wrapped in two distinct layers. The first layer, which the researchers dubbed "business-layer encryption," handles the initial data serialization using Protocol Buffers. When a user is logged out, this layer uses static Diffie-Hellman to establish a session. When logged in, it switches to a fixed key delivered by the server, utilizing AES-CBC with a checksum that is fundamentally flawed.

The second layer, MMTLS, was added in 2016. It acts as a wrapper around the business-layer traffic. While it mimics the structure of standard TLS—using records for handshakes, data, and alerts—it is a proprietary implementation. The researchers found that MMTLS lacks forward secrecy. If an attacker manages to compromise the static private key used in the initial handshake, they can retroactively decrypt all captured traffic for that session. This is a massive departure from the security guarantees provided by modern protocols that enforce ephemeral key exchanges.

Reverse Engineering the Mars Library

The core of this research relied on deconstructing Tencent’s Mars library, a cross-platform infrastructure component written in C++. Because the library is only partially open-source, the researchers had to perform extensive reverse engineering on the "Mars-wechat" module, which contains the proprietary MMTLS code.

By mapping the API endpoints—referred to internally as "Scenes"—the team identified how the application serializes data into Protobuf fields before passing it to the encryption routines. The following pseudo-code illustrates the logic flow they uncovered:

// Simplified representation of the MMTLS encryption trigger
void SomeAPI::doScene(APIObject* obj) {
    // 1. Serialize Protobuf data
    ByteArray data = obj->serialize();
    
    // 2. Apply Business-layer encryption
    ByteArray encryptedData = BusinessLayer::encrypt(data, sessionKey);
    
    // 3. Wrap in MMTLS and send via NetCore
    NetCore::send(MMTLS::wrap(encryptedData));
}

This flow is problematic because it creates a "front door" for interception. Because the business-layer encryption uses a forgeable checksum, an attacker performing an Adversary-in-the-Middle (AitM) attack doesn't need to break the AES encryption to manipulate the traffic. They can simply modify the payload and recalculate the checksum, effectively bypassing the integrity checks that a standard HMAC would provide.

Real-World Impact for Pentesters

For those of us conducting mobile application penetration tests, this research is a goldmine. If you are tasked with auditing a mobile app that uses a custom network stack, stop looking for standard TLS interception points. You need to identify the underlying native libraries—often found in the /lib/ directory of the APK—and begin hooking the encryption functions.

The impact here is not just theoretical. The researchers noted that the Five Eyes intelligence alliance has historically exploited vulnerabilities in these types of proprietary protocols to conduct mass surveillance. When an app uses a custom protocol, it creates a unique fingerprint that is trivial for state-level actors to identify and decrypt. If you find an app using a custom protocol, your report should highlight the Cryptographic Failures as a critical finding. The lack of forward secrecy and the use of forgeable integrity checks are not just "bugs"; they are architectural failures that render the encryption effectively useless against a motivated adversary.

Moving Beyond Proprietary Stacks

Defenders and developers must understand that obfuscation is not security. Tencent’s response to the researchers—promising to upgrade to AES-GCM—misses the point. The issue isn't the choice of cipher; it's the decision to build a custom protocol stack that lacks the rigorous, peer-reviewed scrutiny of established standards.

If you are building a mobile application, the path forward is clear. Use standard, well-maintained libraries like BoringSSL or OpenSSL. Do not attempt to implement your own handshake, your own session resumption, or your own integrity checks. The complexity of modern cryptography is too high for any single team to get right, and as this research shows, the cost of getting it wrong is the total loss of user privacy. The next time you are testing an app, look for the tell-tale signs of custom binary protocols. If you find them, you have likely found the most vulnerable part of the entire system.