> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usebench.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluate your application

> Run the actual entry point, including model decisions, tools and orchestration.

<Note>
  SDK preview. This path calls your application, unlike the prompt benchmark started
  with **Start benching**. You supply the entry point and test cases. It is not an
  automatic hosted runner for any connected repository.
</Note>

## Run a small suite

Complete [SDK setup](/sdk/quickstart), then run this on a test server, never a live
customer request path. Use synthetic inputs and safe tool dependencies. The SDK
does not sandbox your application or prevent its external side effects.

```ts theme={null}
const report = await bench.evaluateSystem({
  sourceRevision: process.env.BENCH_SOURCE_COMMIT!, // Full 40-character SHA
  contextRevision: 'refund-policy-v1',
  cases: [
    {
      id: 'refund-after-deadline',
      split: 'incident',
      input: { orderId: 'fixture-order', days: 31 },
      expectedOutput: { refunded: false },
      requiredTools: ['load-policy'],
      forbiddenTools: ['issue-refund'],
      maxModelCalls: 3,
    },
  ],
  run: (input, { signal }) => runYourActualApplication(input, { signal }),
})

// Optional. Saving a report does not run a judge or use evaluation credits.
await bench.publishSystemEvaluation(systemId, report)
await bench.shutdown()
```

`runYourActualApplication` is your application's entry point, not a prompt assembled
by Bench. Instrument its nested model, agent and tool calls with `bench.trace`.
Await every call, including completion of streams, before returning the final result.
Return the business outcome you want to check, such as a fixture database's refund
state. A model saying “refunded” is not evidence that a refund actually happened.

## Cases, criteria and coverage

An expected output is an exact typed comparison. Required tools must have a successful
recorded call. Forbidden tools must have no recorded call. Tool/model call budgets
catch unexpected loops. These are deterministic checks, not semantic judgments.

Use suite partitions deliberately:

| Partition  | Purpose                                                         |
| ---------- | --------------------------------------------------------------- |
| Capability | Can the application perform a supported task?                   |
| Regression | Does established behavior still work?                           |
| Incident   | Does a reproduced production failure remain fixed?              |
| Holdout    | Does the change work on independent cases not used to build it? |

The SDK accepts 1 to 100 cases. Each case has at most 90 tool assertions and 100
captured spans. Missing assertions, missing root evidence, capture limits, timeout,
or unfinished child work produce an incomplete result. Timeouts are cooperative;
untrusted or non-cooperative code needs an isolated process or sandbox. The suite
stops after a timeout or unfinished work rather than overlapping the next case.

Reports pin source revision, context revision, suite hash and planned case count.
They retain redacted inputs, expected outcomes, checks and observed trajectories.
Redaction can remove detail needed for a repair; inspect it before sharing a brief.
Only explicitly instrumented behavior is covered. Unrecorded calls are unknown.

## View results

Open **AI systems → your system → Runtime**. Browse cases and their tool, harness or
quality findings. **Review** shows checks and recorded execution. **Copy fix prompt**
includes the case and pinned revisions for your approved coding agent.

Saved SDK reports are labeled **Client-reported execution**. They are not a server
certification, a hosted Bench score, or proof that a production issue was fixed.
Reports are limited to 500 KB, deduplicated, and retained for 30 days. The preview
accepts up to 100 reports per system during that window; keep additional reports
locally. A repository-scoped key may access only its permitted systems.

MCP exposes these reports through `bench_get_runtime_evaluations`, without an
evaluation charge. It does not execute the customer's application itself.

## Isolated E2B execution

The pipeline includes an operator-only CLI for a clean, pinned source checkout:

```sh theme={null}
python -m pipeline.system_runtime \
  --repo /absolute/path/to/clean/fixture \
  --revision FULL_40_CHARACTER_COMMIT_SHA \
  --suite /outside-the-repository/cases.json \
  --command '["python3","bench_entry.py"]' \
  --confirm-cloud-run
```

Install the pinned E2B Python dependency `e2b==2.5.0` in the operator environment
and provide `E2B_API_KEY` there. The entry script reads `BENCH_CASE_PATH` and writes
one JSON object containing `output` and the completed `spans` tree. Expected answers
remain outside the sandbox. SDK-generated local reports are a different interface.

Each case gets a fresh sandbox with internet access disabled and no ambient Bench
credentials. Dependencies must be in the reviewed template or source snapshot.
Commands and combined stdout/stderr are bounded; cleanup is verified. This mode
uses fixture dependencies, not arbitrary live provider credentials. Its local JSON
output may contain your application data: redact it before retaining or sharing it.

`compare_system_runs` compares identical case suites and context versions. A candidate
must fix an incident, retain previous passes and pass regression and holdout cases.
Incomplete evidence remains review-required. This validates only the configured
cases, not every possible behavior. The CLI does not push, merge or deploy.

Next: [production checks](/sdk/production-checks), [test library](/guides/test-library),
and [reviewing code fixes](/guides/automatic-fixes).