Kuboid
Open Luck·Kuboid.in
Black Hat2023
Open in YouTube ↗

Evading Logging in the Cloud: Bypassing AWS CloudTrail

Black Hat4,680 views26:02about 2 years ago

This talk demonstrates three distinct techniques for bypassing AWS CloudTrail logging: protocol mutation, the use of undocumented APIs, and the exploitation of non-production endpoints. These methods allow an attacker to perform unauthorized actions within an AWS environment without generating detectable audit logs. The research highlights how these techniques can be used to silently enumerate permissions and perform administrative tasks while remaining invisible to security monitoring tools. The speaker also introduces an automated discovery pipeline using CertSpotter to identify additional vulnerable non-production endpoints.

How to Bypass AWS CloudTrail Logging Using Undocumented APIs and Protocol Mutation

TLDR: Security researchers have identified three distinct methods to perform unauthorized actions in AWS environments without triggering CloudTrail logs: protocol mutation, abusing undocumented APIs, and exploiting non-production endpoints. These techniques allow attackers to perform reconnaissance and administrative tasks while remaining invisible to standard monitoring tools. Pentesters and bug bounty hunters should prioritize testing these non-standard entry points to identify gaps in their current detection coverage.

CloudTrail is the bedrock of AWS security monitoring. Most security teams treat it as the single source of truth for auditing API activity. If an action happens in an AWS environment, the assumption is that it leaves a footprint. This research proves that assumption is dangerous. By manipulating the way requests are sent to the AWS API, an attacker can effectively blind security operations centers and perform sensitive operations without leaving a trace.

The Mechanics of Protocol Mutation

AWS APIs are not monolithic. They support multiple protocols, including JSON and XML, and different versions of these formats. The core of the protocol mutation technique lies in the rigid, model-first design of AWS services. When a service is built, AWS defines a strict model for how requests must be structured.

During a penetration test, you can exploit this by sending malformed inputs or using incorrect protocol headers. For example, by sending a 1.0 JSON header to a 1.1 API endpoint, you can trigger different HTTP response codes based on whether the identity has the necessary permissions. If the identity is authorized, the API returns a 404. If it is unauthorized, it returns a 403. Because the request is malformed or uses an unexpected protocol version, the AWS logging infrastructure fails to record the event. You are essentially performing permission enumeration without the audit trail. This is a classic case of Broken Access Control where the logging mechanism itself is the point of failure.

Abusing Undocumented APIs

Finding undocumented APIs is often the most effective way to bypass security controls. These are endpoints that exist within the AWS infrastructure but are absent from the official AWS SDK documentation. Because they are not part of the standard SDK, they often lack the same level of scrutiny and logging integration as their documented counterparts.

You can discover these by crawling the AWS Management Console. When you interact with the console, your browser must use these same internal APIs to function. By intercepting these requests, you can extract the underlying API operations. In one notable case, the iamadmin service provided operations that mirrored standard IAM functions but operated entirely outside the scope of CloudTrail. By using these undocumented methods as a proxy, you can perform administrative tasks—like listing access keys—without generating a single log entry. This technique highlights a critical Identification and Authentication Failure where the lack of visibility into internal API surfaces creates a massive blind spot for defenders.

Exploiting Non-Production Endpoints

Non-production endpoints—such as those labeled beta, gamma, or staging—are frequently overlooked during security assessments. These endpoints are often connected to production resources but are managed under different security policies. The danger here is that these endpoints may not be configured to report to the same CloudTrail logging buckets as production endpoints.

Using the AWS CLI, you can target these endpoints directly by using the --endpoint-url argument. If you find a non-production endpoint that has access to production data but lacks logging, you have found a high-value target for defense evasion.

To scale this discovery, you can use CertSpotter to monitor certificate transparency logs. By automating the process of checking these new subdomains against AWS API responses, you can identify which endpoints are active and potentially vulnerable. This is not just about finding a bug; it is about mapping the shadow infrastructure that exists alongside the production environment.

The Defensive Reality

Blue teams are currently relying on tools that assume CloudTrail is comprehensive. If your SIEM or Managed Threat Detection service is only looking at standard CloudTrail logs, you are missing these activities entirely.

Defenders need to move beyond simple log analysis. You must implement network-level monitoring to detect traffic directed at non-standard AWS endpoints. Additionally, you should audit your IAM policies to ensure that identities do not have permissions to interact with internal or non-production service namespaces. If an identity doesn't need to talk to a gamma endpoint, it shouldn't have the permission to do so.

For those of us on the offensive side, these techniques are a reminder that the cloud is not a black box. It is a complex, distributed system with thousands of undocumented interfaces. The next time you are on an engagement, don't just look at the documented API calls. Look at the headers, look at the subdomains, and look at the protocols that the SDKs are using under the hood. There is almost always a way to interact with the service that the developers didn't intend for you to see. Keep digging into the internals, and you will find the gaps that everyone else is missing.

Talk Type
research presentation
Difficulty
advanced
Has Demo Has Code Tool Released


Black Hat USA 2023

118 talks · 2023
Browse conference →
Premium Security Audit

We break your app before they do.

Professional penetration testing and vulnerability assessments by the Kuboid Secure Layer team. Securing your infrastructure at every layer.

Get in Touch
Official Security Partner
kuboid.in