Beyond the Basics

Introduction

A proxy can have TLS, input validation, and authentication configured correctly while still exposing data through a cache or losing the logs needed to investigate a request. Those problems often come from how policies interact, rather than from a missing policy.

The previous post covered the first five checks: encryption, input validation, sensitive data, rate limiting, and access control. This continuation looks at logging, caching, fault handling, flow variables, and automated review. The examples show where a configuration can look reasonable in isolation but behave differently once it is part of a request flow.

Step 6: Monitor API Activity Without Leaking Data

Logs help explain failed requests and suspicious activity, but collecting every request and response can create another store of sensitive information. API keys, access tokens, and personal data can end up in a logging system with different retention and access rules from the API itself. Decide which fields are needed for investigation and exclude the rest, rather than starting with complete payloads.

External Logging Systems

Apigee’s Trace (Debug) Tool shows runtime execution and is useful for debugging individual calls. Persistent logging goes to an external system through facilities such as the MessageLogging policy. Destinations can include Google Cloud Logging, Splunk, Datadog, Dynatrace, or Loggly.

The documentation includes MessageLogging examples, but successful flow execution is not proof of successful log delivery. MessageLogging runs independently of the flow, so a connection or validation failure may not appear as a failed policy execution in Trace. When investigating missing records, check the destination as well as the proxy.

Detecting Logging Failures with ServiceCallout

If the flow needs to react to a failed logging request, ServiceCallout provides a synchronous HTTPS call whose result can be checked. That gives the proxy more control over how it handles delivery failures. The connection to the logging service also needs certificate validation; sending logs over HTTPS is not enough if validation errors are ignored.

<ServiceCallout name="SC-LogRequest">
    <Request variable="logRequest">
        <Set>
            <Verb>POST</Verb>
            <Headers>
                <Header name="Content-Type">application/json</Header>
            </Headers>
            <Payload contentType="application/json">{...}</Payload>
        </Set>
    </Request>
    <HTTPTargetConnection>
        <URL>https://logging-service.example.com/logs</URL>
        <SSLInfo>
            <Enabled>true</Enabled>
            <TrustStore>ref://myTrustStore</TrustStore>
            <IgnoreValidationErrors>false</IgnoreValidationErrors>
            <Protocols>
                <Protocol>TLSv1.2</Protocol>
                <Protocol>TLSv1.3</Protocol>
            </Protocols>
        </SSLInfo>
    </HTTPTargetConnection>
</ServiceCallout>

This example enables TLS, uses a TrustStore, and sets IgnoreValidationErrors to false. The trust store defines the certificates accepted for the connection, while the protocol list restricts it to TLSv1.2 and TLSv1.3. These settings protect the connection carrying the logs; they do not decide which request fields are appropriate to log.

A User-Agent Header That Broke Logging

During one assessment, I found a MessageLogging policy that generated JSON for an external logging service such as Datadog. The payload contained metadata including the timestamp, request method, and User-Agent header. That header was copied directly from {request.header.user-agent}, without escaping it for the JSON string.

A double quote in the supplied value broke the generated JSON. The following simplified entry shows the result:

{
    "timestamp": "2024-09-21T14:20:00Z",
    "method": "GET",
    "user-agent": "malicious-user-agent""
}

The logging service rejected the entry because it could not parse it, while the API request still completed. The missing record was therefore a consequence of the requester’s input, rather than a general outage of the logging system. A requester who knew about the behavior could cause their requests to disappear from that log.

The value needs encoding for the format it is being inserted into. Apigee’s escapeJSON(string) function escapes characters such as double quotes inside a JSON string. Applied to the User-Agent field, the template becomes:

<MessageLogging name="Log-Request">
    <Syslog>
        <Message>
            {private.datadogKey} {"timestamp": "{system.timestamp}", "method": "{request.verb}", "user-agent": "{escapeJSON(request.header.user-agent)}"}
        </Message>
        <Host><site_intake_endpoint></Host>
        <Port><site_port></Port>
        <Protocol>TCP</Protocol>
    </Syslog>
</MessageLogging>

Using escapeJSON prevents this header from breaking the JSON string. It fixes the formatting problem shown here, rather than guaranteeing delivery: the logging connection and destination can still fail for other reasons. Both valid formatting and failed delivery need attention when reviewing the logging configuration.

Step 7: Keep Access Checks Around Cached Responses

Apigee’s PopulateCache, LookupCache, and InvalidateCache policies write, read, and remove cached entries. ResponseCache stores whole responses. Caching avoids repeated backend work, but also introduces a second route by which the proxy can return data. That route needs the same access checks as a response retrieved from the backend.

Keep Sensitive Values Out of Cache Keys

A cache key is stored data too. In the Cassandra-backed cache described here, keys are stored as supplied, so a token or personal information placed in a key is exposed to anyone who can read the relevant database records. Protecting the cached response while overlooking its key leaves another copy of that information behind.

The likelihood of someone gaining database access may be low in a particular deployment. Avoiding secrets in keys is still a useful precaution: it reduces what can be exposed if that access occurs, without relying entirely on the database boundary. Review the identifiers used in a key alongside the value and its expiry.

How a Cache Hit Can Skip Authorization

Consider a proxy that accepts an order number in X-Order-ID: 123456789. On the first request, it authenticates the caller, checks whether they can access that order, and retrieves its details from the backend. To speed up subsequent requests, PopulateCache stores part of the response under that order number for ten minutes:

<PopulateCache async="false" continueOnError="false" enabled="true" name="Populate-Order-Cache">
    <CacheKey>
        <KeyFragment>order</KeyFragment>
        <KeyFragment ref="request.header.X-Order-ID"/>
    </CacheKey>
    <Scope>Exclusive</Scope>
    <ExpirySettings>
        <TimeoutInSeconds>600</TimeoutInSeconds> <!-- Cache expires in 10 minutes -->
    </ExpirySettings>
    <Source>flow.addressDetails</Source>
</PopulateCache>

Now suppose a later request runs LookupCache before authentication and authorization. The lookup uses the same order number supplied in the header:

<LookupCache async="false" continueOnError="false" enabled="true" name="Lookup-Order-Cache">
    <CacheKey>
        <KeyFragment>order</KeyFragment>
        <KeyFragment ref="request.header.X-Order-ID"/>
    </CacheKey>
    <Scope>Exclusive</Scope>
    <AssignTo>flow.addressDetails</AssignTo>
</LookupCache>

If finding a cached entry causes the proxy to return immediately, the access checks never run. A caller who knows or guesses the order number receives the cached details, even though the request that originally populated the cache was authorized. Checking the first caller does not establish permission for everyone who requests the same key later.

The ten-minute expiry limits how long that entry remains available, but does not determine who may read it. Predictable or sequential order numbers make the problem easier to encounter because a caller can supply numbers other than their own. The access decision must happen on the cache-hit path as well as the backend path.

Put the Checks Before LookupCache

Run authentication and the relevant resource authorization checks before returning cached data. Policies such as OAuthV2 or VerifyJWT can establish part of that decision, but a valid credential alone does not establish permission to read a particular order. The example below places JWT processing before LookupCache, and populates the cache only when the lookup did not find an entry:

<Flow name="GetOrderDetails">
    <Description>Retrieve order details from cache or backend.</Description>
    <Condition>(proxy.pathsuffix == "/orders") and (request.verb = "GET")</Condition>
    
    <!-- Request handling steps -->
    <Request>
        <Step>
            <Name>Decode-User-JWT</Name>
        </Step>
        <Step>
            <Name>Verify-User-JWT</Name>
        </Step>
        <Step>
            <Name>Lookup-Order-Cache</Name>
        </Step>
    </Request>
    
    <!-- Response handling steps -->
    <Response>
        <Step>
            <Name>Populate-Order-Cache</Name>
            <Condition>lookupcache.Lookup-Order-Cache.cachehit == false</Condition>
        </Step>
    </Response>
</Flow>

Match Keys and Expiry to the Data

For user-specific or session-specific data, use non-sensitive identifiers or hashes that distinguish that scope. An order number identifies an object; it does not identify everyone permitted to access it. Key design and authorization serve different purposes, so changing the key should not replace the access check.

Entries associated with a session also need an invalidation policy when the user logs out or the session expires. Invalidation limits how long the entry persists. Authentication and authorization must still run for requests while the entry exists.

Step 8: Make Fault Handling Part of the Flow

Failures come from several places: a client may send an invalid key, exceed a rate limit, or supply malformed input, while a backend may be unavailable. The response should make the failure understandable without exposing sensitive data or implementation details. The proxy also needs to stop or change its processing when a policy has reported a failure.

VerifyAPIKey, Quota, and SpikeArrest can raise faults handled through FaultRules. When a condition needs to be detected explicitly, such as a failure indicated by a backend response, RaiseFault can move execution into fault handling.

Give the Client a Useful Error

Default error messages may be unclear, or may reveal more internal detail than the caller needs. For an invalid API key, a message such as ‘Your API key is invalid. Please ensure that it is correct and has not expired’ gives the client something to check. There is no need to include internal variable names or backend details to explain that failure.

FaultRules and DefaultFaultRule

A FaultRule handles a known fault and invokes the policies needed to build its response. For a quota violation, an AssignMessage policy can return a message explaining that the usage limit has been exceeded:

<FaultRule name="QuotaViolationHandler">
    <Condition>(fault.name = "QuotaViolation")</Condition>
    <Step>
        <Name>Assign-QuotaExceeded-Message</Name>
    </Step>
</FaultRule>

A DefaultFaultRule covers faults without a specific handler. The response can be generic, such as ‘An unexpected error occurred. Please contact support.’ This gives unanticipated errors an intentional response rather than leaving the client to interpret whatever the failing component returned:

<DefaultFaultRule name="DefaultFault">
    <Step>
        <Name>Assign-Message-503</Name>
    </Step>
    <AlwaysEnforce>true</AlwaysEnforce>
</DefaultFaultRule>

Check Error Variables Set by JavaScript

JavaScript policies often communicate a problem through a flow variable. For example, a script might set an error state after a validation failure:

context.setVariable("errorState", true);

Setting a variable is not the same as raising a fault. Unless another step reads it, subsequent policies continue processing the request. Place a check after the JavaScript policy so that the reported error is handled before the flow uses an invalid or missing result:

<Step>
    <Name>RaiseFault.HandleJSError</Name>
    <Condition>errorState == true</Condition>
</Step>

That step turns the script’s reported condition into a fault the proxy can handle. The same review applies to other error variables: identify where each one is set and where the flow acts on it.

Review continueOnError

The continueOnError attribute allows a flow to continue after a policy fails. There are reasons to use it, but later steps cannot assume that the failed policy produced its normal result. If a ServiceCallout fails, for example, its failure variable needs to be checked before the flow proceeds as though the call succeeded:

<Step>
    <Name>RaiseFault.HandleServiceCalloutFailure</Name>
    <Condition>servicecallout.ServiceCallout-policy.failed == true</Condition>
</Step>

Reviewing only the policy configuration misses half of this behavior. Follow the flow after the failure and check whether it raises a fault, logs the error, or otherwise handles the failed result. A failure variable that nobody reads does not affect processing.

Step 9: Verify the Variables Policies Actually Populate

Flow variables connect policies to one another. Clear names and readable code help during review, but a small mismatch can still leave a policy referencing a value that was never set. Undefined variables can alter behavior in ways that are difficult to diagnose; unused ones add noise and make the useful references harder to follow.

The following SpikeArrest policy takes its rate from a flow variable:

<SpikeArrest continueOnError="false" enabled="true" name="SpikeArrest.ArrestByClientID">
    <DisplayName>SpikeArrest.ArrestByClientID</DisplayName>
    <Identifier ref="apigee.client_id"/>
    <Rate ref="verifyapikey.VerifyAPIKey.spike.arrest.value">500ps</Rate>
</SpikeArrest>

It appears after VerifyAPIKey in the PreFlow, so the ordering looks reasonable:

<PreFlow name="PreFlow">
    <Request>
        <Step>
            <Name>VerifyApiKey.VerifyClientID</Name>
        </Step>
        <Step>
            <Name>SpikeArrest.ArrestByClientID</Name>
        </Step>
        ...
    </Request>
    ...
</PreFlow>

The mismatch is in the name. The Apigee documentation describes the verifyapikey.{policy_name} pattern for the variables populated by VerifyAPIKey. In this configuration, the intended rate variable is verifyapikey.VerifyClientID.spike.arrest.value, while SpikeArrest references verifyapikey.VerifyAPIKey.spike.arrest.value.

The preceding policy does not populate that reference. Its position in the flow does not fix the name, and the rate therefore may not behave as intended. I have seen this kind of mismatch in deployed proxies, where the expected throttling was absent. Verify the exact variable names and runtime values rather than assuming that an earlier policy supplied them.

Step 10: Automate Repeated Checks

Manual review can explain how a proxy works, but it is easy to overlook a familiar configuration mistake in a large set of proxies. Automated checks can apply the same naming, configuration, and security rules across that set and flag problems before deployment. That consistency is useful even when each individual check is simple.

Automation also changes what a reviewer has time to examine. Repeated variable mismatches and configuration errors can be flagged mechanically, leaving more attention for questions that depend on the flow: who may receive a cached response, what happens after a failed call, and where sensitive data persists. Those are the connections worth following beyond the presence of individual policies.