Skip to main content

​
1. Validate the SDK without paid evaluations

  1. Build and install the local SDK tarball. Open SDK in Bench, choose the repository and branch, create a setup key, and copy the agent prompt.
  2. Send a synthetic metadata-only interaction. Confirm the runtime system and its nested spans appear under AI systems → Production.
  3. Verify the application receives the original callback value and error. No prompt, response or key should appear in metadata. Capture must not reduce evaluation allowance.
  4. Test revoked credentials, API downtime, timeout, queue overflow, duplicate delivery, serverless flush() and process shutdown(). Check stats and onError for drops.
  5. Only then opt into synthetic content capture with a domain-specific redactor. Plant fake sensitive values and verify they do not reach stored traces.
The initial setup key has a zero evaluation cap. Keep it that way while testing capture. Follow SDK quick start and production checks.

​
2. Validate production checks with a small budget

  1. Connect or upload a prompt and complete one evaluation to generate its rubric.
  2. Link a content-bearing synthetic span to the actual prompt component.
  3. Use a separate staging credential with a small, explicit nonzero cap. Read bench_evaluation_allowance before starting work.
  4. Start one check. Confirm the immediate queued response, navigate away, and return to the saved result. Verify one reservation and no duplicate charge for the same evidence.
  5. Exercise provider failure, worker restart and explicit retry. Verify failure settlement and the key’s attempted-use cap. A failed provider call must not become a passing score.
  6. Verify cross-tenant/repository denial, deleted sources and 30-day raw-trace expiry.
  7. Review a real failure and save its intended behavior as a case. Confirm a later bench uses the new case and criteria without changing historical results.

​
3. Validate live providers

Passing mocked tests does not establish model availability, billing or judge accuracy.
  • Rotate keys exposed in chat. Put fresh credentials only in an approved server-side environment or secret store, never in a document or SDK/browser bundle.
  • Confirm data-processing and transfer requirements before sending customer content.
  • Pin provider, model and judge settings. Start with synthetic labeled cases covering correct, incorrect, ambiguous and adversarial behavior.
  • Set a call limit, deadline and provider-side spending limit. Bench credits are not the same as the provider’s billed usage.
  • For TypeSafe, configure TYPESAFE_API_KEY on the API worker, check its capabilities, and explicitly set share_with_typesafe: true for the chosen check. Use a narrow question with concrete yes/no criteria. The default local launcher leaves it disabled.
  • Measure false positives and false negatives on human-reviewed holdouts. Select thresholds per domain and risk. A typed probability is not proof of correctness.
  • Compare baseline and candidate on the same independent inputs. Repeat stochastic cases. Track quality, cost and latency separately. Offline replay is not a live A/B test.
Suggested slices include refund-policy boundaries, customer-support escalation, role swaps across multi-turn conversations, grounded claims, tool failures and prompt injection. Keep expected behavior independent of the system’s actual answer.

​
Run the synthetic Jev connectivity test

From the API repository, with TYPESAFE_API_KEY already supplied through an approved server-side environment:
This opt-in test checks two synthetic refund-policy replies using pinned jev-1.13.0. It logs model, probability and token usage, not the key. The client uses bounded retries, so provider usage can exceed two attempts if a retryable error occurs. It is skipped in ordinary tests and does not enable TypeSafe in the application. This is a direct adapter test, outside Bench’s credit ledger. It proves connectivity and behavior on two examples, not calibration or the full production-check job path.

​
4. Validate automatic repairs

  1. Use a disposable, deliberately broken fixture with known expected behavior.
  2. Export its fix brief, pin source and checks, then generate a candidate.
  3. Require targeted fail-to-pass, existing pass-to-pass and unseen holdout success.
  4. Reject stale hashes, protected-file edits, invalid paths, credentials, timeout and excessive output. Test isolation and network denial separately in the hosted runtime.
  5. Review the resulting diff and report. Explicitly authorize any later external PR.
  6. Deploy a limited canary after human approval, retain rollback, and measure actual outcomes before marking a production issue resolved.
See Code fixes for the local runner contract.

​
Run the synthetic E2B smoke test

From the pipeline repository:
Enter the key at the non-echoing prompt, or supply E2B_API_KEY through an approved environment. The script creates at most two 120-second sandboxes. It sends only a hard-coded refund fixture, never your repository or the provider key into the guest. It requires the broken and corrected versions to produce their expected results, tests outbound TLS denial to two public endpoints, and confirms both sandboxes stopped. This consumes E2B compute, not Bench evaluation credits. Use an application-level network probe. E2B documents that its firewall can accept a TCP connection before blocking forwarding, so a successful socket connect alone does not prove outbound access. See E2B network controls. A passing fixture does not verify arbitrary customer repositories, all network routes, hosted agent orchestration or production isolation. Those remain separate release gates.

​
Release checklist

Each gate is separate. A locally passing test suite does not certify production readiness, compliance, deployment or business-outcome improvement.