Poisoning Rhino, Then Waiting for the Callback

I had spent enough time with Apigee's code-execution policies to notice that the interesting paths often appeared where two policies met. This time I wanted to know whether one policy could leave Rhino in a dangerous state and a different JavaScript execution path could find that state later.

That led to a chain between PythonScript and an asynchronous JavaScript callback. In short, PythonScript could modify a Rhino JSContext and leave it attached to a worker thread. A JavaScript policy could separately register an httpClient callback. If that callback later landed on the prepared worker, it reused the modified context while Apigee's per-thread security mark was absent. The function could reach java.lang.Runtime and execute a host command.

I reproduced the chain on runtime 1-17-0-apigee-4. The prerequisite was permission to modify and deploy API proxy code containing PythonScript and JavaScript policies. Once the policies were deployed, later HTTP requests supplied the worker-pool activity needed to trigger the chain.

What was left to try in JavaScript?

After my earlier sandbox bypasses, I was running out of ideas for the ordinary JavaScript policy. The direct routes into Java were already restricted, and getting past one check would still leave another.

Rhino can expose Java classes to JavaScript. Apigee limits that feature with a class shutter. In the restricted mode, its filter admits configured classes and packages: basic value types and the Java implementation of Apigee's script-facing objects. A class such as java.lang.Runtime stays outside that surface.

An exposed Java object might suggest another route through getClass(). Apigee's wrapper explicitly hides that property. It returns NOT_FOUND when a script asks for getClass, closing that obvious reflection entry point.

There was also the JVM permission boundary. The normal JavaScript execution path set a thread-local security mark before calling the script shell and cleared it afterwards. Even resolving a dangerous Java class would not by itself grant permission to start a process.

The walkthrough below shows those three checks separately. These are cleaned decompilations from the tested Apigee runtime. The Rhino excerpts later in the article are pinned to version 1.7.13.

setClassShutter(new ClassShutter() {
    public boolean visibleToScripts(String className) {
        if (JSContext.this.allowClassAccess
                || JSContext.allowedClasses.contains(className)) {
            return true;
        }
        for (String p : JSContext.allowedPackages) {
            if (className.startsWith(p)) {
                return true;
            }
        }
        return false;
    }
});

public Object get(String name, Scriptable start) {
        if (name.equals("getClass")) {
            return NOT_FOUND;
        }
        return super.get(name, start);
    }

private void shellExecute(Shell shell, Binder binder,
        MessageContext messageContext) throws FlowException {
    // Service lookup, timing and exception handling omitted.
    securityPolicy.setSecurityManagerOnThread(
        true, this.javascriptStep.getContextInfo());
    shell.execute(binder);
    securityPolicy.setSecurityManagerOnThread(false, "");
}

I did not have a useful new way through that synchronous path. The httpClient callback caught my attention because it ran the JavaScript function later, through a different executor. I wanted to see what happened to the same restrictions there.

What httpClient saved for later

The JavaScript object model exposes httpClient for outbound HTTP requests. A policy can send a request and supply a function to handle its response or error. For example, a proxy can fetch backend data and put part of the result into a flow variable.

Using a callback releases the JavaScript worker while the HTTP operation is pending. It can return to the pool and service other work; a queued task invokes the function when the response is ready. This is the non-blocking pattern described in Apigee's callback guidance.

The proxy flow itself does not simply run past every following policy during that wait. In the inspected implementation, the JavaScript step installs a resume handler and returns ExecutionResult.PAUSE. The flow resumes after the callback executor finishes its work. Releasing a worker and advancing the flow are separate operations.

I followed registration into ScriptableHttpClient.buildCallBackEvent(). It creates a CallBackEvent containing the supplied function and its scope. The constructor also retains thisObj, a copied Binder containing Apigee bindings, the execution deadline and callback depth. It reads the current JSContext to collect some of those values, but does not store that parent context for later execution.

private CallBackEvent buildCallBackEvent(Object callBack) {
    CallBackEvent callBackHolder = null;
    if (callBack instanceof BaseFunction) {
        BaseFunction function = (BaseFunction) callBack;
        Scriptable scope = JSContext.getCurrentContext().getTopLevelScope();
        callBackHolder = new CallBackEvent(function, scope, this, getCallBackExecutor());
        this.hasCallBacks = true;
    }
    return callBackHolder;
}

public CallBackEvent(BaseFunction function, Scriptable scope, Scriptable thisObj, CallBackExecutor callBackExecutor) {
    this.function = function;
    this.scope = scope;
    this.thisObj = thisObj;
    JSContext parenJsContext = JSContext.getCurrentContext();
    this.binder = new Binder(parenJsContext.getBinder());
    this.endTimeMS = parenJsContext.getEndTimeMS();
    this.callBackExecutor = callBackExecutor;
    this.depth = callBackExecutor.getCurrentStackDepth() + 1;
}

CallBackExecutor callBackExecutor = httpClient.getCallBackExecutor();
if (httpClient.hasCallBacks()) {
    ThreadPoolManager threadPoolManager = ThreadPoolManager.getInstance();
    Objects.requireNonNull(threadPoolManager);
    callBackExecutor.setTaskSubmitHandler(threadPoolManager::submitAsMetricsAwareTask);
    callBackExecutor.setResumeHandlerAndStart(throwable -> {
        handleResume(throwable, binder, messageContext, executionContext);
    });
    return ExecutionResult.PAUSE;
}
binder.unBindAll();
return ExecutionResult.SUCCESS;

public void prepareToExecute(CallBackEvent callBackHolder) {
    this.callBackQueue.add(callBackHolder);
    int inFlight = this.inflightCallBackCount.decrementAndGet();
    logger.atFine().log("Adding call back callBackQueue size is %d inflight is %d", this.callBackQueue.size(), inFlight);
    this.numberOfCall.incrementAndGet();
    if (inFlight == 0 && this.resumeHandler != null) {
        submitPendingCallbacks();
    }
}

private void submitPendingCallbacks() {
    if (readyToExecute() && !this.shutDown.get()) {
        logger.atFine().log(" submitPendingCallbacks");
        Runnable r = () -> {
            Throwable error = null;
            boolean finished = true;
            try {
                finished = executeJSCallBacksCode();
            } catch (Throwable e) {
                error = e;
            }
            if (finished) {
                if (this.readyToResumeFlag.compareAndSet(false, true)) {
                    this.resumeHandler.accept(error);
                } else {
                    logger.atWarning().log("Attempt to call resume handler more than once");
                }
            }
        };
        if (this.submitHandler != null) {
            this.submitHandler.accept(r);
        } else {
            r.run();
        }
    }
}

The saved function therefore keeps its JavaScript environment and the Apigee variables needed to continue. Its interpreter context comes from the worker that eventually runs the event.

That did not automatically expose Java classes in a callback. CallBackExecutorImpl explicitly enters the restricted factory. On a clean worker, that still gives the function a restricted context. I needed to follow the other boundary as well: which thread would carry the permission checks?

The security mark did not follow the callback

The security mark around shell.execute() belongs to the thread executing that synchronous call. When the shell returns, Apigee clears the mark. The JavaScript step schedules the later callback task separately through ThreadPoolManager.submitAsMetricsAwareTask.

On the worker selected for that task, the callback executor enters Rhino and drains its event queue. CallBackEvent.execute() restores the saved binder and deadline, then calls the function directly. I found no matching installation of the JavaScript security-policy mark around that invocation.

private void shellExecute(Shell shell, Binder binder,
        MessageContext messageContext) throws FlowException {
    // Service lookup, timing and exception handling omitted.
    securityPolicy.setSecurityManagerOnThread(
        true, this.javascriptStep.getContextInfo());
    shell.execute(binder);
    securityPolicy.setSecurityManagerOnThread(false, "");
}

private synchronized boolean executeJSCallBacksCode() throws Throwable {
  int inFlight = 0;
  if (!this.shutDown.get()) {
    JSContextFactory factory = JSContextFactory.getRestrictedInstance();
    try {
      factory.enterContext();
      while (!this.callBackQueue.isEmpty()) {
        this.current = this.callBackQueue.poll();
        this.current.execute();
      }
      inFlight = this.inflightCallBackCount.get();
      // error handling omitted
    } finally {
      this.current = null;
      factory.exit();
    }
  }
  return inFlight == 0 && this.callBackQueue.isEmpty();
}

public void execute() {
  if (this.depth > 10) {
    throw new IllegalStateException("Too many nested calls to Callbacks.");
  }
  // null-function and deadline checks omitted
  JSContext cx = JSContext.getCurrentContext();
  cx.setBinder(this.binder);
  cx.setEndTimeMS(this.endTimeMS);
  this.function.call(cx, this.scope, this.thisObj, this.args);
  clear();
}

Apigee's internal security manager uses thread-local state to decide whether to enforce its JVM permission checks. Saving the function, variables and deadline does not carry that mark to another worker. The callback could resume its application state without restoring the restriction used around an ordinary JavaScript policy.

That left one of the two relevant checks absent on the callback path. The class shutter still prevented a clean callback from resolving Runtime. I now had a specific question: could another policy leave the callback worker with class access already enabled?

A public setter could open the class shutter

The class shutter's first condition reads JSContext.allowClassAccess. When the flag is true, the filter immediately allows the requested class. The shutter remains installed, but its allowlists no longer restrict the result.

The flag was mutable through a public method:

public boolean setAllowClassAccess(boolean set) {
    boolean old = this.allowClassAccess;
    this.allowClassAccess = set;
    return old;
}

That setter changes the live object whose flag the shutter reads. It does not bind the change to one script invocation or arrange to restore the old value when that script finishes.

The implementation supported both open and restricted context creation. JSContextFactory.makeContext() left a restricted object's flag at its constructor default of false; the open factory set it to true:

protected Context makeContext() {
    JSContext ctx = new JSContext(this);
    if (this.restrictPackageAccess) {
        ctx.setInstructionObserverThreshold(this.INSTRUCTION_OBSERVER_THRESHOLD);
    } else {
        ctx.setAllowClassAccess(true);
    }
    ctx.setWrapFactory(new JSWrapFactory());
    return ctx;
}

My earlier PythonScript research had already shown a broader Java-access path, including a legacy route that accepted JavaScript resources inside that policy. The code suggests that the policy may once have been intended to support both languages, although I could not confirm that design history.

PythonScript uses Jython. From the Python seed in this test, I could reach Apigee's JSContextFactory and call the public setter on the returned context. The tested runtime's Java permission policy permitted the calls needed to obtain the factory and change the flag.

That suggested a short chain: use PythonScript to open the shutter, then call Runtime.exec() from an asynchronous JavaScript callback where the thread mark was absent. One problem remained. allowClassAccess belonged to a JSContext, so the modified object had to survive long enough for a callback on the same worker to reuse it.

Two entries outlived a single cleanup exit

A Rhino Context is associated with a Java thread. Entering one does not necessarily create a new interpreter object: Context.enter(...) first checks whether the thread already has a current context. If it does, Rhino returns that same object and increments its nesting counter.

The JavaScript step tried to clean up a prior context before preparing a fresh policy execution. Its clearJSContext() helper performed one Context.exit(), which decremented the counter. Rhino cleared the thread association only when that counter reached zero.

I entered the restricted factory twice from PythonScript, changed the context returned by the first entry through setAllowClassAccess(True), and deliberately omitted both exits. On an initially clean worker, the two entries raised the counter from zero to two. Returning from PythonScript left that modified object attached to the worker.

When JavaScript's clearJSContext() helper later performed one Context.exit(), the two outstanding entries became one. The walkthrough compares the seed with this policy-start cleanup.

factory = JSContextFactory.getRestrictedInstance()
ctx = factory.enterContext()
factory.enterContext()
ctx.setAllowClassAccess(True)

protected void clearJSContext() {
    Context context = Context.getCurrentContext();
    if (context != null) {
        try {
            Context.exit();
        } catch (Exception e) {
            logger.atSevere().withCause(e).log("%s Failed to clear JS context from thread.", this.javascriptStep);
        }
    }
}

public static void exit()
{
    Object helper = VMBridge.instance.getThreadContextHelper();
    Context cx = VMBridge.instance.getContext(helper);
    if (cx == null) {
        throw new IllegalStateException(
            "Calling Context.exit without previous Context.enter");
    }
    if (cx.enterCount < 1) Kit.codeBug();
    if (--cx.enterCount == 0) {
        VMBridge.instance.setContext(helper, null);
        cx.factory.onContextReleased(cx);
    }
}

Context old = VMBridge.instance.getContext(helper);
if (old != null) {
  cx = old;
} else {
  if (cx == null) {
    cx = factory.makeContext();
    // initialization omitted
  }
  VMBridge.instance.setContext(helper, cx);
}
cx.enterCount++;

The existing-context branch also explained why selecting the restricted factory did not repair the flag. Rhino returned the attached object without calling makeContext(). The factory's restricted default applied when constructing a missing object, while this worker already had one with allowClassAccess=true.

This gave me a persistent state change across tasks. It still did not give me working JavaScript execution.

A fresh policy failed before its script could run

I sent ordinary JavaScript policies through the pool and waited for one to reuse a prepared worker. I expected the retained context to expose Runtime. Instead, Apigee threw IllegalStateException while building the JavaScript environment, before my script body started.

The Python seed had attached a context without leaving a JavaScript invocation active. Rhino tracks that invocation separately as a top call. Helpers need its scope when converting Java values into objects in the correct JavaScript environment.

RhinoExecutor.runScript() called Binder.bindAll(...) before script.exec(...). During that preparation, a content wrapper asked ScriptRuntime.getTopCallScope(cx) for the current scope. The reused context had no active top call, so the helper threw before the policy body could establish one.

This was the second reason the callback mattered. Its function and scope already existed. CallBackEvent.execute() restored the saved state and invoked that function directly, avoiding the fresh-policy binding phase that had just failed.

For an interpreted callback, InterpretedFunction.call() checks whether a top call is active. If none exists, it routes the function through ScriptRuntime.doTopCall(), which establishes the saved scope. Apigee's factory hook records it on JSContext, and the function is invoked again inside that active top call. This time Rhino reaches the interpreter.

The next walkthrough follows those calls and their return. The Rhino helpers come from the 1.7.13 ContextFactory, InterpretedFunction and ScriptRuntime sources; their upstream line numbers are preserved.

private synchronized boolean executeJSCallBacksCode() throws Throwable {
  int inFlight = 0;
  if (!this.shutDown.get()) {
    JSContextFactory factory = JSContextFactory.getRestrictedInstance();
    try {
      factory.enterContext();
      while (!this.callBackQueue.isEmpty()) {
        this.current = this.callBackQueue.poll();
        this.current.execute();
      }
      inFlight = this.inflightCallBackCount.get();
      // error handling omitted
    } finally {
      this.current = null;
      factory.exit();
    }
  }
  return inFlight == 0 && this.callBackQueue.isEmpty();
}

public void execute() {
  if (this.depth > 10) {
    throw new IllegalStateException("Too many nested calls to Callbacks.");
  }
  // null-function and deadline checks omitted
  JSContext cx = JSContext.getCurrentContext();
  cx.setBinder(this.binder);
  cx.setEndTimeMS(this.endTimeMS);
  this.function.call(cx, this.scope, this.thisObj, this.args);
  clear();
}

public Object call(Context cx, Scriptable scope, Scriptable thisObj,
          Object[] args)
{
  if (!ScriptRuntime.hasTopCall(cx)) {
    return ScriptRuntime.doTopCall(this, cx, scope, thisObj, args, idata.isStrict);
  }
  return Interpreter.interpret(this, cx, scope, thisObj, args);
}

public static Object doTopCall(Callable callable,
                Context cx, Scriptable scope,
                Scriptable thisObj, Object[] args, boolean isTopLevelStrict)
{
  if (scope == null)
    throw new IllegalArgumentException();
  if (cx.topCallScope != null) throw new IllegalStateException();

  Object result;
  cx.topCallScope = ScriptableObject.getTopLevelScope(scope);
  cx.useDynamicScope = cx.hasFeature(Context.FEATURE_DYNAMIC_SCOPE);
  boolean previousTopLevelStrict = cx.isTopLevelStrict;
  cx.isTopLevelStrict = isTopLevelStrict;
  ContextFactory f = cx.getFactory();
  try {
    result = f.doTopCall(callable, cx, scope, thisObj, args);
  } finally {
    cx.topCallScope = null;
    // Cleanup cached references
    cx.cachedXMLLib = null;
    cx.isTopLevelStrict = previousTopLevelStrict;

    if (cx.currentActivationCall != null) {
      // Function should always call exitActivationFunction
      // if it creates activation record
      throw new IllegalStateException();
    }
  }
  return result;
}

protected Object doTopCall(Callable callable, Context cx,
    Scriptable scope, Scriptable thisObj, Object[] args) {
  if (cx instanceof JSContext) {
    JSContext jcx = (JSContext) cx;
    jcx.setTopLevelScope(scope);
    if (args != null) {
      notifyArgs(args, jcx);
    }
  }
  return super.doTopCall(callable, cx, scope, thisObj, args);
}

protected Object doTopCall(Callable callable,
              Context cx, Scriptable scope,
              Scriptable thisObj, Object[] args)
{
  Object result = callable.call(cx, scope, thisObj, args);
  return result instanceof ConsString ? result.toString() : result;
}

The callback could now run in the context modified by PythonScript, through the path that lacked the synchronous security mark. The class shutter exposed Runtime, and the missing mark left Apigee's normal JVM permission boundary inactive for that callback invocation. I had the two conditions needed to put the worker-pool chain together.

Reproducing the worker-pool race

I split the proof into two proxies because the chain needed two different scheduling roles.

Proxy A ran a normal JavaScript policy. At startup it called Context.exit() until its current worker had no Rhino context, allowing the policy to initialize normally even after earlier attempts. It then issued a delayed outbound request through httpClient. Registration saved the callback function, scope, binder, deadline, and depth while the response remained pending.

Proxy B ran the PythonScript seed. While Proxy A waited, repeated requests to Proxy B visited workers in the same pool. Each successful seed entered the restricted factory twice, set allowClassAccess=true, and returned without matching the entries. A worker could finish the request and return to the pool with the modified context still attached.

When Proxy A's outbound response arrived, Apigee queued the saved CallBackEvent as new work. The callback had no affinity to the thread that registered it. A clean worker kept Runtime hidden. A contaminated worker reused the open context, and the saved function supplied the missing top call.

Worker selection made the proof nondeterministic. In a deployment with several message processors, both proxies also had to reach the same JVM because the leaked state could not cross a process boundary. I controlled the policies, callback body, delay, and request volume; repeated requests handled the two scheduling decisions I could not control.

The model predicted three outcomes, which made useful negative controls:

1. A callback on a clean worker could not resolve the blocked Java class.
2. A fresh JavaScript policy on a contaminated worker failed during global binding, before its body.
3. A saved callback on a contaminated worker entered a top call and reached the host operation.

What the shell proved

For the end-to-end check, the callback opened a reverse shell. Inside it, I checked the operating-system identity of the Apigee message processor:

bash-5.1$ whoami
apigee

The prerequisite narrowed who could exploit the issue: an attacker first needed the ability to deploy both policy flows. But the result still crossed a real boundary. Apigee allowed a proxy developer to supply JavaScript and PythonScript under constrained runtimes; the callback reached host command execution as the message processor's operating-system user.

The worker state also crossed proxy and request boundaries without travelling through a documented flow variable. Proxy B changed a Rhino object attached to a Java thread. Proxy A's callback later consumed that state because the pool happened to select the same thread. The policy model presented those executions as separate, while the runtime let one inherit the other's interpreter authority.

Once the decompiled path and the host-side result agreed, I reported the finding.

The inspected repair closes both lifecycle gaps

My later review of the Apigee hybrid 1.17.1 runtime image found two changes aimed at the state used by this chain. I did not rerun the proof against that image or a managed-runtime build, so this evidence does not establish the first fixed version or a managed-service remediation date.

Old contexts are drained before JavaScript starts

JSContext gained clearPriorContexts(), which repeatedly calls Context.exit() until the worker has no current Rhino context, with a guard against an endless loop:

public static void clearPriorContexts() {
    int maxLoop = 1000;
    while (Context.getCurrentContext() != null) {
        try {
            if (maxLoop-- <= 0) {
                logger.atSevere().log(
                    "Exceeded max loop iterations clearing prior JS contexts.");
                return;
            }
            Context.exit();
        } catch (Throwable e) {
            logger.atFine().withCause(e).log("Failed to close Context");
        }
    }
}

Both JavascriptStepExecution.execute(...) and RhinoExecutor.execute(...) call this cleanup before preparing a policy. The older cleanup could remove one entry; the loop drains all outstanding entries before normal JavaScript initialization. A worker carrying Proxy B's modified context should therefore detach it before a fresh policy begins.

Callbacks run through an untrusted executor

ScriptableHttpClient now passes InternalMessageContext::callUntrusted into each CallBackEvent. The event uses that executor around the captured function instead of calling the function directly. The call eventually reaches SecurityPolicyServiceImpl.callUntrusted(...), which installs the thread mark when necessary, executes the callback, and clears only the mark it added:

InternalSecurityManager securityManager = getSecurityManager();
boolean wasMarked = false;
if (securityManager != null) {
    wasMarked = securityManager.markedThread();
    if (!wasMarked) {
        securityManager.markThread("callUntrusted");
    }
}

V result = AccessController.doPrivileged(() -> callable.call());

if (securityManager != null && !wasMarked) {
    securityManager.clearThread();
}
return result;

This change establishes the JVM permission boundary on the worker that actually executes the callback. Its authority no longer depends on the mark left on that thread by unrelated work.

The bundled Rhino version remained 1.7.13, setAllowClassAccess(...) remained mutable, and the callback executor still entered a restricted context in much the same way. The repair instead enforced clean state before policy execution and explicit untrusted authority around the asynchronous function. Static inspection of hybrid 1.17.1 therefore indicates that the complete callback chain is closed.

Why I still keep looking?

Although Google no longer pays bounties for Apigee sandbox bypasses. But I keep looking anyway. These remain some of my favourite bugs because the interesting part is rarely one missing check. This time every individual control looked reasonable in isolation; the failure appeared only after I followed their lifetimes across a callback and a reusable worker.

The shell is a satisfying proof, but the best moment comes earlier: when an apparently exhausted attack surface becomes interesting again because two components disagree about when security state begins and ends.