> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usebench.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Python

> Record Python agent, model and tool activity in Bench.

## Copy this into your coding agent

Paste into Cursor, Codex, Claude Code, or another coding agent in your project.

```text theme={null}
Add Bench using https://docs.usebench.ai/sdk/python and the setup key from my Bench SDK page. Keep the key server-side. Label the
environment and start with metadata-only capture. Wrap one real request and
its tools, including async context propagation. Flush a synthetic interaction
and confirm it arrived in Bench. Show changed files and remaining setup steps.
```

<Note>Beta release. All four languages support tracing, application tests and scripted simulations.</Note>

## Install

[Package on PyPI](https://pypi.org/project/trybench-sdk/0.1.0/) · [Source and examples](https://github.com/trybench/bench-sdk/tree/v0.1.0/python)

```sh theme={null}
python -m pip install trybench-sdk==0.1.0
```

## Record one interaction

Choose **Python** on the [SDK setup page](/sdk/quickstart). Create a setup key and
use its installation command. Python 3.10 or later is required.

```python theme={null}
import os
from bench_sdk import Bench

bench = Bench(
    api_key=os.environ["BENCH_API_KEY"],
    repository=os.environ["BENCH_REPOSITORY"],
    branch=os.environ["BENCH_BRANCH"],
    endpoint=os.environ["BENCH_API_BASE_URL"],
    system_name="Support agent",
    environment="staging",
)

with bench.trace("connection-check", kind="AGENT") as span:
    span.set_output({"ok": True})

bench.shutdown()
```

Open the system's **Production** tab. Setup changes from **Waiting for first
event** to **SDK events received** after the event is accepted.

## Wrap your application

Use `with bench.trace(...)` around a Deep Agents, LangGraph, LangChain or custom
application call. It also works around `await agent.ainvoke(...)`. Wrap tools
inside the same request with `kind="TOOL"`; nested async tasks inherit their
parent trace. Keep streaming spans open until the stream finishes.

Inputs and outputs are omitted by default. With content capture enabled, pass
`input=...` and call `span.set_output(result)`. Built-in filtering runs before
delivery. Add a `redact(value)` callback for your application's extra fields.
See [privacy and redaction](/guides/security-and-privacy#how-redaction-works).

Call `bench.flush()` at request boundaries, or `await bench.aflush()` in async
code. Call `shutdown()` after active requests finish. Delivery retries once;
`bench.stats` shows queued and dropped spans. A delivery failure does not change
your application's result or exception.

Set `component_id` to a real prompt component from Bench to connect the event to
its saved criteria. [Production checks](/sdk/production-checks) require recorded
content and a key with evaluation allowance.

Automatic framework adapters are coming soon. Use explicit wrappers with your current framework.

## Test your application

`await bench.evaluate_system(...)` calls your application's request handler with
pinned cases. It captures the real nested tool/model traces, compares the final
output and independently observed state, and returns a redacted report locally.
Use a test database and test service credentials.

```python theme={null}
report = await bench.evaluate_system(
    source_revision=os.environ["GIT_COMMIT_SHA"],  # Full 40-character commit SHA
    context_revision="refund-policy-v1",
    cases=[{
        "id": "outside-refund-policy", "split": "regression",
        "input": {"days": 45}, "expected_output": {"refunded": False},
        "forbidden_tools": ["issue-refund"],
    }],
    run=lambda request, context: handle_request(request),
)
assert report["summary"]["status"] == "completed"
assert all(case["status"] == "passed" for case in report["cases"])
# Explicit upload, only when you want this report saved in Bench:
await bench.publish_system_evaluation(int(os.environ["BENCH_SYSTEM_ID"]), report)
```

`run(input, context)` and `observe(context)` may be synchronous or asynchronous.
When a case has `expected_state`, supply `observe` to read the authoritative test
state. Context provides `case_id`, `cancelled`, `signal`, `deadline` and
`raise_if_cancelled()`. Inputs are JSON snapshots; changes inside the application
do not change the case's assertions.

`await bench.simulate_system(...)` accepts the same revisions and cases, plus
`create_session(initial_state, context)`. Cases use
`input={"initial_state": {...}, "turns": [...]}` and require `expected_state`.
Return an object with `turn(message, context)`, `observe()` and `close()` methods.
A fresh session receives 1 to 20 scripted customer turns. Bench snapshots observed
state before closing the session. `context.turn_index` identifies the turn.

Both helpers use a 30-second timeout per case, configurable with `timeout` up to
300 seconds. Pass a `threading.Event` as `cancel_event` to stop the suite. Missing
assertions, missing state, unfinished traces and timeouts remain incomplete.
Callbacks must honor cancellation; Python cannot forcibly stop a synchronous
thread. Await all child work and isolate external side effects. These helpers
are local execution, not a process sandbox or a hosted verification claim.

Tests record redacted content even when production capture is metadata-only, so
use synthetic inputs. Reports are not uploaded and paid checks are not started
unless you take a separate explicit action. Automatic framework adapters are
coming soon.

Run the complete [refund simulation example](https://github.com/trybench/bench-sdk/blob/v0.1.0/python/examples/simulate_refund.py). It sends two refund requests, checks the number of refunds written, and demonstrates the failing implementation and its fix.

## Run in CI

Use Bench with your existing pull-request checks. See [Run tests in CI](/sdk/ci)
for setup, report handling and the features available in each language.