The Debug Filter That Read Masked Apigee Secrets

Apigee's Debug feature has an awkward job. It must show enough of a live API transaction to explain why a proxy behaved the way it did, without turning every person who can troubleshoot that proxy into a reader of its secrets.

Apigee handles that tension with two controls. A Debug session can use a filter to decide which transactions are worth recording, and data masking removes sensitive fields and variables from the result. Variables prefixed with private. receive an even stronger-looking treatment: they do not appear in Debug at all.

That sounds like a clean boundary. The debugger can inspect the execution, while values such as API keys, backend credentials, and private variables stay hidden. There was just one problem: the filter made its decision before those values disappeared.

On runtime 1-14-0-apigee-3, I could create a session whose filter asked whether private.token began with a chosen prefix. A match captured a transaction; a miss captured nothing. The token never appeared in the returned trace, but the presence of a transaction answered the question. Repeating that question recovered the value one character at a time.

What a Debug session records

An Apigee API proxy is a pipeline of policies: verify a credential, read configuration, transform a message, call a backend, and so on. When that pipeline misbehaves, a Debug session records the steps taken by a small number of requests and the flow variables available along the way.

Sessions are deliberately bounded. They run for a limited time and collect only a limited number of transactions. On a busy proxy even that can be noisy, so the API accepts a condition in its filter field:

POST /v1/organizations/ORG/environments/ENV/apis/API/revisions/REV/debugsessions
Authorization: Bearer ACCESS_TOKEN
Content-Type: application/json

{
  "filter": "request.verb = \"POST\""
}

Only matching requests enter the session. The filter is therefore not a search over an already-produced trace; it is part of the capture decision itself.

Masking acts on the other side of that decision. An environment can mask selected JSON paths, XML paths, headers, and variables. Proxy code can also create a variable whose name begins with private.. Apigee documents that prefix as a way to prevent the variable from appearing in Debug output. A common pattern is to load a credential from a Key Value Map into a private variable:

<KeyValueMapOperations name="LoadToken" mapIdentifier="secrets">
  <Get assignTo="private.token">
    <Key>
      <Parameter>backend-token</Parameter>
    </Key>
  </Get>
  <Scope>environment</Scope>
</KeyValueMapOperations>

The useful security question was not whether private.token appeared in the response. It did not. The question was whether the filter could still refer to it.

The filter could question a hidden value

I started with a known random token and two otherwise identical sessions. Their filters differed by one guessed character:

private.token StartsWith "a"
private.token StartsWith "b"

After sending the same request through the proxy, one session contained a transaction ID and the other was empty. Changing the first character of the stored token changed which session captured the request. Masking was working—the private variable was absent from the trace—but it was working too late to hide the filter's verdict.

The effective order looked like this:

flowchart TD
  request["Request enters the proxy"] --> variables["Policies populate raw flow variables"]
  variables --> filter{"Debug session filter matches?"}
  filter -->|No| ignored["Ignore transaction"]
  filter -->|Yes| selected["Select transaction"]
  selected --> masking["Mask fields and omit private.* variables"]
  masking --> result["Return Debug transaction"]

That capture-or-ignore branch is a boolean oracle. It does not reveal the secret directly; it answers a series of yes-or-no questions about it. StartsWith made those questions especially convenient because every successful answer preserved the prefix for the next round.

The decisive API interactions were small. Create a filtered session, generate a request carrying the target value, then list the transaction IDs recorded by the session:

GET /v1/organizations/ORG/environments/ENV/apis/API/revisions/REV/debugsessions/SESSION/data
Authorization: Bearer ACCESS_TOKEN

A match returned an ID; a miss returned an empty list. The response never needed to include private.token:

// matching prefix
["0"]

// non-matching prefix
[]

One bit at a time

I wrapped those calls in a prefix search. The following is reduced from the proof of concept to show the extraction logic; authentication, polling, cleanup, and retry handling are omitted:

def matches(prefix):
    session_id = create_debug_session(
        filter=f'private.token StartsWith "{prefix}"'
    )
    send_request_through_proxy()
    return bool(list_transaction_ids(session_id))


prefix = ""
for position in range(EXPECTED_LENGTH):
    for char in ALPHABET:
        if matches(prefix + char):
            prefix += char
            print(prefix)
            break
    else:
        raise RuntimeError(f"no candidate matched at offset {position}")

If the alphabet contains 64 characters, recovering a character takes 32 guesses on average. That makes the technique much slower and noisier than a direct disclosure. It also needs repeatable traffic: every guess has to reach the code path that populates the same variable with the same value. A rapidly changing nonce is a poor target; a long-lived API key is not.

The KVM token was a deterministic test fixture, not a special property of the bug. The filter operated on flow variables, so other stable values available during proxy execution were candidates too. Examples included a masked incoming header such as request.header.X-API-Key or a credential exposed to the proxy by an authentication policy. Whether a particular value was practical to recover depended on knowing its variable name and being able to make it recur while the sessions were active.

The account extracting the token could not read the KVM

My first script used one Environment Admin identity to prepare the fixture and run the attack. That made the demo easy to reproduce, but it also made the result look circular: an Environment Admin could already manage the KVM.

So I split the experiment into two identities.

The Debug account had only these permissions:

apigee.tracesessions.create
apigee.tracesessions.get
apigee.tracesessions.list
apigee.tracesessions.delete

The setup account stopped participating as soon as it had inserted the token. A separate traffic loop kept requests moving through the deployed proxy. The Debug account then reconstructed the token solely from empty and non-empty session results; a direct KVM read with that identity failed.

This separation was the point of the finding. It was not interesting that a KVM administrator could learn a KVM value. It was interesting that a role intended to operate Debug sessions could query information that Debug deliberately refused to display.

There were still meaningful limits. The account needed access to the target proxy's trace sessions, a useful guess at the variable name, and enough recurring traffic to ask many questions. I did not demonstrate cross-organization access, nor did the primitive magically enumerate every secret in an environment. It turned a stable, known variable into a queryable one.

A queryable blank field

I reported the issue on Christmas Day. The initial reproduction was accepted only after I separated the setup identity from the Debug identity, which was fair—the cleaner experiment made the permission boundary impossible to miss. Google awarded $1,337 and later marked the issue fixed.

We disagreed about the severity. Google's view was that masking primarily prevents accidental exposure and is not a strong authorization boundary for someone who can debug a proxy. I thought that undersold environments where engineers can inspect production behavior but cannot change and redeploy production code. In that model, “may run a trace” and “may read every secret processed by this proxy” are very different permissions.

After the fixed notice, I went back to the public Apigee Hybrid runtime images to see whether the patch was visible there. In Hybrid 1.14.0, the relevant implementation made the separation unusually clear. FlowDebugSessionManager parsed the submitted filter into a normal Apigee expression, while the masking configuration travelled separately into the Debug service:

LogicExpression filter = session.getFilter()
    .map(ExpressionParser::parseLogicExpression)
    .orElse(null);

debugService.createSession(
    session.getName(),
    session.getOriginalAuthority(),
    session.getMaskConfiguration(),
    ...
);
registerFiltersMarkers(session, filter);

That split explains the oracle: the expression tree made the capture decision from the live message context, and masking shaped the trace collected after that decision.

The public images did not reveal the fix. FlowDebugSessionManager, MessageDebugInterest, and the Debug masking utility were byte-for-byte identical in Hybrid 1.14.0, 1.14.1, and 1.14.2. There was no new check for private.* variables in those classes and no sign that masking had moved ahead of the filter. Because my report concerned the managed runtime, the product team could have fixed the issue in the control plane—for example, by rejecting protected variable names when a session is created—or in managed-only code that is absent from the Hybrid image. The static artifacts narrow the possibilities, but they do not identify which of those paths Google chose.

What stayed with me was not the prefix loop; that part was almost mechanical. It was how convincing the empty field looked. The UI showed no token, the API returned no token, and the private-variable feature appeared to have done its job. But confidentiality had already been lost one stage earlier, when a selector was allowed to branch on the raw value. Redaction is only as strong as the earliest observable decision made with the data.