> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usebench.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run tests in CI

> Check application outcomes on pull requests and stop regressions before merging.

## Copy this into your coding agent

Paste into Cursor, Codex, Claude Code, or another coding agent in your project.

```text theme={null}
Set up Bench in CI using https://docs.usebench.ai/sdk/ci.
Use my existing SDK installation and repository-scoped key. Keep keys in CI secrets.
Use the application-test or simulation helper for this project's language. Run
the real application with isolated test dependencies. Include capability, regression
and incident cases. Assert actual tool effects, not just the final answer.
Fail the job for a failed, errored, unscored, missing or incomplete case. Save the
redacted report even when checks fail and flush telemetry before exit.
Keep the existing test runner and make failed or incomplete Bench reports fail
the job. Use the native helper names from the matching language guide.
Use synthetic data and show a passing run, a deliberately failing run, and the
setup still required in my CI provider. Do not expose secrets to forked PR code.
```

CI runs your tests automatically when someone opens or updates a pull request.
Bench can record these runs and check application outcomes in all four languages.
Installing the SDK does not create a CI workflow or a required PR check.

| Language                | Available today                                      |
| ----------------------- | ---------------------------------------------------- |
| JavaScript / TypeScript | Tracing, application tests and scripted simulations. |
| Python                  | Tracing, application tests and scripted simulations. |
| Go                      | Tracing, application tests and scripted simulations. |
| Rust                    | Tracing, application tests and scripted simulations. |

## Prepare the test suite

Complete your [language setup](/sdk/quickstart) and commit its dependency lockfile.
Use the public installation command from your language guide. Keep the same SDK
version locally and in CI.

For JavaScript and TypeScript, create the cases and application adapter described
in [application testing](/sdk/system-evaluation). Keep the same cases, business
criteria and test data when comparing a change. Include at least one normal task,
one reproduced failure and one previously working behavior.

The following ESM script expects your `tests/bench-suite.mjs` to export
`createBenchSuite(bench)`. That is your application adapter, not an SDK export.
It returns `contextRevision`, `cases`, `run`, and, when needed, `observe` and
`timeoutMs`. Pass the same Bench instance to your instrumented application.

Save as `scripts/bench-ci.mjs`:

```js theme={null}
import { mkdir, writeFile } from 'node:fs/promises'
import { execFileSync } from 'node:child_process'
import { Bench } from '@benchai/sdk'
import { createBenchSuite } from '../tests/bench-suite.mjs'

for (const name of ['BENCH_API_KEY', 'BENCH_REPOSITORY', 'BENCH_BRANCH']) {
  if (!process.env[name]) throw new Error(`Set ${name} in CI`)
}

const bench = new Bench({
  apiKey: process.env.BENCH_API_KEY,
  repository: process.env.BENCH_REPOSITORY,
  branch: process.env.BENCH_BRANCH,
  endpoint: process.env.BENCH_API_BASE_URL,
  environment: 'ci',
  captureContent: false,
})

try {
  const suite = await createBenchSuite(bench)
  const report = await bench.evaluateSystem({
    ...suite,
    sourceRevision: execFileSync('git', ['rev-parse', 'HEAD'], {
      encoding: 'utf8',
    }).trim(),
  })

  await mkdir('bench-results', { recursive: true })
  await writeFile('bench-results/report.json', JSON.stringify(report, null, 2))

  const passed = report.summary.status === 'completed'
    && report.planned_case_count > 0
    && report.cases.length === report.planned_case_count
    && report.cases.every((item) => item.status === 'passed')

  console.log(JSON.stringify(report.summary))
  if (!passed) process.exitCode = 1
} finally {
  await bench.shutdown()
}
```

For scripted conversations, call `bench.simulateSystem` with your session adapter
instead. The report and pass/fail gate have the same shape. A failed check can
still have `summary.status: 'completed'`, so checking that field alone is not enough.

## Add a GitHub Actions job

Set `BENCH_API_KEY` as a repository secret and `BENCH_API_BASE_URL` as a repository
variable using the values from Bench. The default SDK key is sufficient for tracing
and saving application reports. These operations do not consume Bench evaluations.
Your application's model calls can still have provider costs.

Save as `.github/workflows/bench.yml`, alongside your existing tests:

```yaml theme={null}
name: Bench application tests
on:
  pull_request:
  workflow_dispatch:

permissions:
  contents: read

jobs:
  application-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      # Add your app's build and test-dependency setup here if needed.
      - name: Check application outcomes
        run: node scripts/bench-ci.mjs
        env:
          BENCH_API_KEY: ${{ secrets.BENCH_API_KEY }}
          BENCH_API_BASE_URL: ${{ vars.BENCH_API_BASE_URL }}
          BENCH_REPOSITORY: ${{ github.repository }}
          BENCH_BRANCH: ${{ github.head_ref || github.ref_name }}
      - name: Save the report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: bench-results
          path: bench-results/report.json
          if-no-files-found: warn
          retention-days: 7
```

The source revision records the commit actually checked out by CI. On a pull
request, that can be GitHub's test merge commit. Add this job as a required check
in your repository rules if failed outcomes should block merging.

GitHub does not supply repository secrets to forked pull requests or Dependabot
runs. This script fails clearly when the key is missing. Use a reviewed, trusted
run for those contributions; never switch to `pull_request_target` to execute
untrusted PR code with secrets. See [GitHub's secrets documentation](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/use-secrets).

## Python, Go and Rust

Call the native application-test or simulation helper from your existing test
runner. Read actual test database or tool state, assert that the report passed,
and save the redacted report as a CI artifact. Use the same cases and revisions
when comparing a change. Set the client environment to `ci`.

| Language | Test command                                                    | Setup                 |
| -------- | --------------------------------------------------------------- | --------------------- |
| Python   | `python -m unittest discover` or your existing `pytest` command | [Python](/sdk/python) |
| Go       | `go test -race ./...`                                           | [Go](/sdk/go)         |
| Rust     | `cargo test --locked`                                           | [Rust](/sdk/rust)     |

For Python, require `report["summary"]["status"] == "completed"` and every case
to have `status == "passed"`. In Go, require `report.Passed()`; in Rust, require
`report.passed()`. Check errors returned by the helper before accepting a report.
A failed, cancelled, timed-out or incomplete suite must fail the job.

You can also keep ordinary tracing around existing tests. Flush those traces
before exit and inspect delivery statistics separately from test assertions.

## Verify the setup

1. Run a correct fixture and confirm the job passes.
2. Change one expected outcome and confirm the job fails with a saved report.
3. Remove an assertion or force a timeout and confirm incomplete work cannot pass.
4. Restore the case, rerun, and inspect the saved report artifact.

To display a report under **Application tests**, explicitly publish it with your
language's report helper and the correct system ID. Uploading an Actions artifact
alone does not publish it to Bench. See [application testing](/sdk/system-evaluation)
for the language guides, report limits and examples.

Application-test helpers capture spans inside the report in all four languages.
Flushing does not upload those spans as production events. Ordinary tracing outside
the helper sends events you can find in Bench's `ci` environment.