Skip to main content

​
1. Connect your system

Install the Bench SDK or attach its OpenTelemetry bridge, then send one real request. Bench draws the whole system from that trace: the runtime, each agent, the model calls and tools per agent, and hand-offs between agents. Frameworks that emit OpenTelemetry GenAI spans work with the exporter alone. Open the system’s System overview to see what ran and correct its name or scope. Traces do not carry editable prompt text, so your coding agent registers each model call it finds in the repository (one entry per request the code sends to a model: a code-derived key, its messages, the model configuration and the conditions that decide which pieces are sent) with register_calls. Each registered call is an evaluable component, shown as Call #id. The Set up using a coding agent prompt does both steps for you; see Bench from coding agent. Connecting GitHub is the fallback: it scans code for prompts, models, tools and workflows, and adds background rescans and pull requests. A quick scan is available as soon as a system exists. It lists prompts whose quality is not measured yet, tools and hand-offs not tested yet, and each model’s estimated cost next to a lower-priced model. It does not run your application, and every finding has a Bench button.

​
2. Tell Bench what good behavior means

Open Understanding to review the context Bench collected. Add your product’s purpose, rules and examples. For a refund assistant, that might include the refund window, exceptions and when to ask a person for help. Bench keeps context with the system it describes. Your stated rules take priority over patterns in recorded behavior. Adding context does not start a paid evaluation.

​
3. Start a bench

Choose Start benching. Bench creates test cases and criteria, includes enabled cases from your test library, measures the current prompt and model, then tests possible improvements against the same cases. A green pulsing dot means the bench is active. You can leave the page and return. Use Stop bench on the system page to stop its active and queued prompts. Completed results stay available. To check the application’s tool calls and final outcome together, use Real app testing. A prompt comparison alone cannot tell you whether a tool changed a database or whether a workflow finished successfully. See real app testing and simulation testing for a side-by-side comparison and setup links.

​
4. Understand the result

Open History, select a bench and choose View details. Review the cases, criteria, failures, quality change and estimated model cost. Missing evidence is shown as missing; it is not counted as a passing result. A cheaper model may give worse answers. Compare both quality and cost before changing your application. Previous results keep the context and cases used at the time, so you can understand why results changed.

​
5. Review a fix

In Fixes, compare the original prompt with the proposed change. Copy a fix prompt for your coding agent, or download a fix brief with the relevant evidence. Review changes and rerun important cases before merging or deploying them. An opened pull request is a proposed fix, not a change to your live application.

​
6. Learn from production

The SDK records real interactions in Production. Production checks and automatic re-benching are on by default for every system; the owner can turn them off. An event is checked at ingest when its span is linked to a prompt, comes from the watched environment, carries captured input and output, and the prompt has a completed bench. Otherwise it shows Not checked and can be checked from its drawer once content is captured. Failed checks feed an automatic re-bench that uses the failing interactions as new cases; those benches appear in Results and History. New interactions do not rewrite your rules or turn an incorrect response into an expected answer. Continue with SDK setup, test cases, production checks or code fixes.