Evaluate your application

SDK preview. This path calls your application, unlike the prompt benchmark started with Start benching. You supply the entry point and test cases. It is not an automatic hosted runner for any connected repository.

Run a small suite

Complete SDK setup, then run this on a test server, never a live customer request path. Use synthetic inputs and safe tool dependencies. The SDK does not sandbox your application or prevent its external side effects.

const report = await bench.evaluateSystem({
  sourceRevision: process.env.BENCH_SOURCE_COMMIT!, // Full 40-character SHA
  contextRevision: 'refund-policy-v1',
  cases: [
    {
      id: 'refund-after-deadline',
      split: 'incident',
      input: { orderId: 'fixture-order', days: 31 },
      expectedOutput: { refunded: false },
      requiredTools: ['load-policy'],
      forbiddenTools: ['issue-refund'],
      maxModelCalls: 3,
    },
  ],
  run: (input, { signal }) => runYourActualApplication(input, { signal }),
})

// Optional. Saving a report does not run a judge or use evaluation credits.
await bench.publishSystemEvaluation(systemId, report)
await bench.shutdown()

runYourActualApplication is your application’s entry point, not a prompt assembled by Bench. Instrument its nested model, agent and tool calls with bench.trace. Await every call, including completion of streams, before returning the final result. Return the business outcome you want to check, such as a fixture database’s refund state. A model saying “refunded” is not evidence that a refund actually happened.

Cases, criteria and coverage

An expected output is an exact typed comparison. Required tools must have a successful recorded call. Forbidden tools must have no recorded call. Tool/model call budgets catch unexpected loops. These are deterministic checks, not semantic judgments. Use suite partitions deliberately:

Partition	Purpose
Capability	Can the application perform a supported task?
Regression	Does established behavior still work?
Incident	Does a reproduced production failure remain fixed?
Holdout	Does the change work on independent cases not used to build it?

The SDK accepts 1 to 100 cases. Each case has at most 90 tool assertions and 100 captured spans. Missing assertions, missing root evidence, capture limits, timeout, or unfinished child work produce an incomplete result. Timeouts are cooperative; untrusted or non-cooperative code needs an isolated process or sandbox. The suite stops after a timeout or unfinished work rather than overlapping the next case. Reports pin source revision, context revision, suite hash and planned case count. They retain redacted inputs, expected outcomes, checks and observed trajectories. Redaction can remove detail needed for a repair; inspect it before sharing a brief. Only explicitly instrumented behavior is covered. Unrecorded calls are unknown.

View results

Open AI systems → your system → Runtime. Browse cases and their tool, harness or quality findings. Review shows checks and recorded execution. Copy fix prompt includes the case and pinned revisions for your approved coding agent. Saved SDK reports are labeled Client-reported execution. They are not a server certification, a hosted Bench score, or proof that a production issue was fixed. Reports are limited to 500 KB, deduplicated, and retained for 30 days. The preview accepts up to 100 reports per system during that window; keep additional reports locally. A repository-scoped key may access only its permitted systems. MCP exposes these reports through bench_get_runtime_evaluations, without an evaluation charge. It does not execute the customer’s application itself.

Isolated E2B execution

The pipeline includes an operator-only CLI for a clean, pinned source checkout:

python -m pipeline.system_runtime \
  --repo /absolute/path/to/clean/fixture \
  --revision FULL_40_CHARACTER_COMMIT_SHA \
  --suite /outside-the-repository/cases.json \
  --command '["python3","bench_entry.py"]' \
  --confirm-cloud-run

Install the pinned E2B Python dependency e2b==2.5.0 in the operator environment and provide E2B_API_KEY there. The entry script reads BENCH_CASE_PATH and writes one JSON object containing output and the completed spans tree. Expected answers remain outside the sandbox. SDK-generated local reports are a different interface. Each case gets a fresh sandbox with internet access disabled and no ambient Bench credentials. Dependencies must be in the reviewed template or source snapshot. Commands and combined stdout/stderr are bounded; cleanup is verified. This mode uses fixture dependencies, not arbitrary live provider credentials. Its local JSON output may contain your application data: redact it before retaining or sharing it. compare_system_runs compares identical case suites and context versions. A candidate must fix an incident, retain previous passes and pass regression and holdout cases. Incomplete evidence remains review-required. This validates only the configured cases, not every possible behavior. The CLI does not push, merge or deploy. Next: production checks, test library, and reviewing code fixes.

Start here

Evaluate and improve

MCP

SDK

Development

Evaluate your application

Run a small suite

Cases, criteria and coverage

View results

Isolated E2B execution

​Run a small suite

​Cases, criteria and coverage

​View results

​Isolated E2B execution

Run a small suite

Cases, criteria and coverage

View results

Isolated E2B execution