Skip to main content

​
Copy this into your coding agent

Paste into Cursor, Codex, Claude Code, or another coding agent in your project.
Open an AI system’s Production tab. It opens with an attention strip (failures first, then proposed fixes, quality, cost and a cheaper model to bench) and a table with Quality and Cost columns. An event opens in a right-hand drawer with a timeline naming the actual tools and models per span, the models and tools used, and the recorded error message for failed spans. The Set up SDK and Quickstart actions help you connect application activity. Reading the short monitoring introduction does not enable capture. Recorded input/output token counts appear when the SDK sends them; when the SDK did not report bench.cost.usd, cost is estimated from tokens and labelled ~.

​
Capture first, evaluate separately

Metadata-only capture is free of evaluation-credit charges. Inputs and outputs are omitted by default. To evaluate real behavior, explicitly enable content capture, supply the span’s input, and add a project-specific redact function. In TypeScript, the traced callback result becomes the output. In Python and manual span APIs, record the result explicitly using the language guide. Only send data you have permission to process.
Built-in redaction catches common secrets and identifiers, but does not guarantee anonymization of free text. Raw traces and their linked production-check payloads are deleted from the application database after 30 days. This is not a claim about backup deletion or every other category of Bench data. Intentionally saved datasets and cases have their own lifecycle.

​
Run a check

  1. Generate a rubric by benching a connected prompt.
  2. Open a recorded interaction under Production and link it to that prompt if its span did not already carry bench.component_id or a matching code location.
  3. In the drawer, choose Check interaction, then confirm the use of one evaluation.
The API responds with a queued job immediately. Workers pin the rubric content and digest, reserve one shared account credit, run the judge, then save results. You can leave the page and return. Results show failed checks and their reasons. Identical evidence is deduplicated. Failed checks can be retried explicitly. Interrupted jobs are failed after their lease expires; reserved credits are reconciled. Failed execution returns the account credit, but attempted execution still counts toward a key’s safety cap. Automatic checks are on by default for every system, for its production environment; the system owner can change the environment or turn them off in the Production tab. An event is checked at ingest only when all of these hold:
  • the span is linked to a prompt, through bench.component_id or a matching code location;
  • the event comes from the watched environment (production by default);
  • the SDK captured input and output (metadata-only spans cannot be judged);
  • the prompt has a completed bench whose rubric supplies the criteria.
Otherwise the event shows Not checked. It can still be checked manually from its drawer once content is captured. The sending key needs evaluation allowance; the default SDK setup key cannot spend evaluations. Existing continuous checks for a linked prompt also apply.

​
Turn feedback into better benchmarks

Passed checks provide regression examples. Failed checks provide incident cases. Bench uses their recorded inputs and existing criteria. A recorded answer never becomes the expected answer just because the application produced it. Incomplete or uncertain checks do not become passing examples. Run Bench when a check finds a failure is also on by default: a failed check starts a new Bench with the failing interactions as new cases, alongside passing examples from the same environment. Those benches appear in Results and History like manual ones. The owner can turn it off in the Production tab to review failures manually. This is limited to one automatic Bench per system per day, requires available account allowance, and waits while another Bench is running. The previous rubric, business context revision and original cases stay pinned for the comparison. The new Bench compares prompts and models using fixed simulated tool replies. Available recorded tool results preserve the incident’s conditions for that replay. Real app tests are a separate step that runs your actual app to verify effects. Original regression cases stay in the suite; new production examples fit the remaining case allowance, with the triggering failure first. It does not deploy code, publish a pull request, or silently rewrite business policy. Review tool and harness suggestions with real app tests before shipping. You can also save reviewed cases or criteria through the test library.

​
Check a recorded execution

API and MCP callers can explicitly set evidence_scope: "trace" and select the completed root span. Bench validates one connected, finished tree and includes its recorded model/tool relationships, inputs, outputs, status and attributes. Incomplete trees or evidence over 45 KB are rejected, not silently shortened. The default scope remains span. This checks the recorded interaction against the linked prompt’s criteria. It does not run the application again. Calls that were not recorded cannot be checked. Use real app tests to repeat a scenario with your app.

​
Delivery and storage

Exports are buffered and bounded, retried once, and dropped if delivery fails. Use onError and bench.stats to monitor drops. In serverless applications, await bench.flush() in a supported lifecycle hook before the process freezes. Use shutdown() after in-flight requests finish at process exit, not per request on a reusable instance. The SDK is not a durable spool. Bench stores the traces you send. Monitor export errors in your application so you can tell when an event did not reach Bench. See retention and deletion.