Skip to main content
Open an AI system’s Production tab. The centered Set up SDK and Quickstart actions help you connect runtime activity. Reading the short monitoring introduction does not enable capture. Recorded input/output token counts appear when the SDK sends them; missing usage is not treated as zero or a verified cost saving.

​
Capture first, evaluate separately

Metadata-only capture is free of evaluation-credit charges. Inputs and outputs are omitted by default. To evaluate real behavior, explicitly enable content capture, supply the span’s input, and add a project-specific redact function. The callback result becomes the output unless you provide a deliberate override. Only send data you have permission to process.
Built-in redaction catches common secrets and identifiers, but does not guarantee anonymization of free text. Raw traces and their linked production-check payloads are deleted from the application database after 30 days. This is not a claim about backup deletion or every other category of Bench data. Intentionally saved datasets and cases have their own lifecycle.

​
Run a check

  1. Generate a rubric by benching a connected prompt.
  2. Open a recorded interaction under Production and link it to that prompt.
  3. Choose Check interaction, then confirm the use of one evaluation.
The API responds with a queued job immediately. Workers pin the rubric content and digest, reserve one shared account credit, run the judge, then save results. You can leave the page and return. Results show failed checks and their reasons. Identical evidence is deduplicated. Failed checks can be retried explicitly. Interrupted jobs are failed after their lease expires; reserved credits are reconciled. Failed execution returns the account credit, but attempted execution still counts toward a key’s safety cap. Automatic checks are off by default. The component’s continuous-evaluation flag and a key with available evaluation allowance are both required. The default SDK setup key cannot spend evaluations, even if that component flag is enabled.

​
Turn feedback into better benchmarks

Review unexpected behavior, then save the intended behavior as a case or a criterion in the system’s test library. Enabled user cases and criteria join future benches; previous scores retain their original evidence. Production data does not silently rewrite business policies or become a verified golden answer. This preview does not autonomously generate approved labels, re-bench the whole system, open fixes or deploy code on every new trace.

​
Optional TypeSafe checks

An operator may configure TYPESAFE_API_KEY on the API server. No TypeSafe key is placed in the SDK or browser. The optional API/MCP check takes a narrow question plus explicit match/no-match criteria and requires share_with_typesafe: true. It returns a typed probability from pinned model jev-1.13.0, marked for review. A value near 0.5 means uncertain, not passing. TypeSafe checks consume one Bench evaluation just like rubric checks. TypeSafe receives the selected evidence in the US. Review applicable privacy, transfer, retention and commercial terms before enabling it. It remains disabled unless the server has been explicitly configured. No private customer data was used for live provider tests.

​
Check a recorded execution

API and MCP callers can explicitly set evidence_scope: "trace" and select the completed root span. Bench validates one connected, finished tree and includes its recorded model/tool relationships, inputs, outputs, status and attributes. Incomplete trees or evidence over 45 KB are rejected, not silently shortened. The default scope remains span. This judges an observed execution. It does not rerun the application. The rubric provider uses the linked prompt’s exact-system rubric. Jev uses the supplied narrow question over the trace. Neither proves that uninstrumented actions were captured. Use application-runtime evaluation for repeatable cases.

​
Delivery and storage

Exports are buffered and bounded, retried once, and dropped if delivery fails. Use onError and bench.stats to monitor drops. In serverless applications, await bench.flush() in a supported lifecycle hook before the process freezes. Use shutdown() after in-flight requests finish at process exit, not per request on a reusable instance. The SDK is not a durable spool. Postgres stores this preview’s customer traces. You do not need an Axiom account or OpenTelemetry exporter. Bench accepts its own JSON payload, not native OTLP. Bench’s internal operational telemetry separately uses OpenTelemetry and Axiom; it does not forward customer SDK content there.