> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usebench.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Production checks

> Compare recorded interactions with saved criteria.

Open an AI system's **Production** tab. The centered **Set up SDK** and **Quickstart**
actions help you connect runtime activity. Reading the short monitoring introduction
does not enable capture. Recorded input/output token counts appear when the
SDK sends them; missing usage is not treated as zero or a verified cost saving.

## Capture first, evaluate separately

Metadata-only capture is free of evaluation-credit charges. Inputs and outputs
are omitted by default. To evaluate real behavior, explicitly enable content
capture, supply the span's `input`, and add a project-specific `redact` function.
The callback result becomes the output unless you provide a deliberate override.
Only send data you have permission to process.

```ts theme={null}
const bench = new Bench({
  apiKey: process.env.BENCH_API_KEY!,
  repository: 'your-team/your-app',
  branch: 'main',
  captureContent: true,
  redact: removeYourDomainIdentifiers,
})
```

Built-in redaction catches common secrets and identifiers, but does not guarantee
anonymization of free text. Raw traces and their linked production-check payloads
are deleted from the application database after 30 days. This is not a claim
about backup deletion or every other category of Bench data. Intentionally saved
datasets and cases have their own lifecycle.

## Run a check

1. Generate a rubric by benching a connected prompt.
2. Open a recorded interaction under **Production** and link it to that prompt.
3. Choose **Check interaction**, then confirm the use of one evaluation.

The API responds with a queued job immediately. Workers pin the rubric content
and digest, reserve one shared account credit, run the judge, then save results.
You can leave the page and return. Results show failed checks and their reasons.
Identical evidence is deduplicated. Failed checks can be retried explicitly.
Interrupted jobs are failed after their lease expires; reserved credits are
reconciled. Failed execution returns the account credit, but attempted execution
still counts toward a key's safety cap.

Automatic checks are off by default. The component's continuous-evaluation flag
and a key with available evaluation allowance are both required. The default
SDK setup key cannot spend evaluations, even if that component flag is enabled.

## Turn feedback into better benchmarks

Review unexpected behavior, then save the intended behavior as a case or a
criterion in the system's test library. Enabled user cases and criteria join
future benches; previous scores retain their original evidence.

Production data does not silently rewrite business policies or become a verified
golden answer. This preview does not autonomously generate approved labels,
re-bench the whole system, open fixes or deploy code on every new trace.

## Optional TypeSafe checks

An operator may configure `TYPESAFE_API_KEY` on the API server. No TypeSafe key is
placed in the SDK or browser. The optional API/MCP check takes a narrow question
plus explicit match/no-match criteria and requires `share_with_typesafe: true`.
It returns a typed probability from pinned model `jev-1.13.0`, marked for review.
A value near 0.5 means uncertain, not passing. TypeSafe checks consume one Bench
evaluation just like rubric checks.

TypeSafe receives the selected evidence in the US. Review applicable privacy,
transfer, retention and commercial terms before enabling it. It remains disabled
unless the server has been explicitly configured. No private customer data was used for live provider tests.

### Check a recorded execution

API and MCP callers can explicitly set `evidence_scope: "trace"` and select the
completed root span. Bench validates one connected, finished tree and includes
its recorded model/tool relationships, inputs, outputs, status and attributes.
Incomplete trees or evidence over 45 KB are rejected, not silently shortened.
The default scope remains `span`.

This judges an observed execution. It does not rerun the application. The rubric
provider uses the linked prompt's exact-system rubric. Jev uses the supplied narrow
question over the trace. Neither proves that uninstrumented actions were captured.
Use [application-runtime evaluation](/sdk/system-evaluation) for repeatable cases.

## Delivery and storage

Exports are buffered and bounded, retried once, and dropped if delivery fails.
Use `onError` and `bench.stats` to monitor drops. In serverless applications,
`await bench.flush()` in a supported lifecycle hook before the process freezes.
Use `shutdown()` after in-flight requests finish at process exit, not per request
on a reusable instance. The SDK is not a durable spool.

Postgres stores this preview's customer traces. You do not need an Axiom account or
OpenTelemetry exporter. Bench accepts its own JSON payload, not native OTLP. Bench's
internal operational telemetry separately uses OpenTelemetry and Axiom; it does not
forward customer SDK content there.