Skip to main content
In staging, failed or canceled evaluations return their reserved allowance. Automatic retries share the original reservation. A Free whole-system bench uses one allowance only after every prompt finishes successfully. A completed score of zero still counts as a successful evaluation, because Bench measured a failure. Reading results and fully cached replays do not use another allowance.

​
The evaluation path

  1. Pin inputs. New benches save a redacted source-prompt snapshot before any model call, alongside the selected context revision and evaluation configuration. Dynamic values remain marked unresolved; the source preview is not a recording of every rendered runtime message.
  2. Build criteria. Generated criteria describe observable good and bad behavior. Explicit library criteria are included without asking the model to rewrite them.
  3. Route checks. Checks use deterministic evaluation or an appropriate judge. Unsupported or insufficiently grounded checks are excluded from scoring.
  4. Build cases. Selected golden cases seed the suite. Generated cases cover remaining needs within the plan budget.
  5. Measure the baseline. Bench executes the prompt and evaluates the response against the suite. Provider failures and unsupported replay inputs remain unscored.
  6. Compare candidates. Candidate prompt/model combinations are measured against the same suite. Primary and secondary criteria both count. Breaking an individually passing baseline check blocks a recommendation, even if an average improves. Reported quality and cost come from those results.
  7. Review a recommendation. The suggested change is available for investigation. Applying a patch, opening a PR, merging and deploying are distinct actions.

​
Investigate failures

History lists one row per bench. Reports shows benches with completed prompt results. Each row shows quality change, estimated cost change and proposed models, with ten benches per page. Open View details to inspect a prompt, then switch between prompts in the same bench without leaving the panel. The investigation panel has five tabs: Older runs without a source snapshot say so; Bench does not substitute today’s source for a historical prompt. Deleted evidence stays unavailable. Close the panel to return to the results or history you were viewing. Reports can export the selected prompt’s saved result as JSON. Observations use a paginated table. Filter by Needs context or To review, then use View more for the supporting explanation. These observations are not confirmed quality failures. Saved cases separate Input, Expected result and Checks with clear labels. Add feedback when a test or judgment is wrong. Specify whether a correction belongs to that case or to a general system rule. To edit future cases, use Test library. History charts show quality, failing checks or model cost. They join only comparable runs with matching prompt and evaluation provenance. A candidate is not a production improvement. Changes to criteria or context can make scores incomparable.
Testing a candidate on the suite used to optimize it is not held-out validation or proof of causality in production. Use independent cases and runtime evidence before rolling out a consequential change.

​
Quality and cost

The Bench score in a system’s Results is the average baseline quality score across prompts in a completed bench, shown out of 100. Quality and estimated cost changes are shown separately. No score is invented when scoring failed or is incomplete.
  • Quick scan: source-backed risks and price comparisons, not executed behavioral tests. “Quality unverified” is not a failing quality score. “Potentially lower model cost” assumes the same input and output volume, not measured overpayment. Published model-price comparisons expire after 30 days unless refreshed. See quick-scan impact.
  • Completed bench: the candidate is compared with the original setup on the same suite. Quality changes are percentage-point differences, not a claim about production improvement.
  • Cost impact: estimated model cost per test call. A multi-prompt bench uses an unweighted average across its prompts, not their production traffic mix or monthly bill. Missing or zero price coverage does not become a 100% saving. A cheaper candidate that regresses a prompt is flagged for quality review.
  • Proposed change: a recommendation, not a merged fix. No production issue is marked solved just because a candidate scored better.
Filter findings by quality or cost impact, or by component: prompts, models, tools, harness/workflow or system. The Quality and cost checks card toggles scan findings on or off. Scan findings, evaluation findings, criteria and cases are paginated so large suites remain navigable. The Feed’s Quality increase and Cost decrease cards summarize measured comparisons from the latest completed bench for each system. Missing comparisons show a quiet empty state rather than an invented score. These are candidate test results, not deployed improvements or monthly savings.

​
While Bench works

One Bench is working panel contains the stages for the whole system. Expand a stage to see the prompts being tested, their activity and saved details. Concurrent prompts may occupy different stages. Prompt rows are paginated. Leaving the browser does not stop the server-side evaluation.

​
If a run stops

The Results page shows one warning for the latest stopped bench, with one Retry bench action when allowance remains. Close the notice to hide it for the current browser session, including page navigation and refresh. No restore button replaces it. When the allowance is exhausted, Upgrade to bench more opens the plan offer before any new evaluation starts. Older failures remain in History. A stopped run is not a quality failure and does not have a measured candidate impact. A provider credit or billing error needs the model-provider account resolved before retrying. Upgrading a Bench plan does not replenish a model provider’s balance. Saved source evidence remains visible even if scoring never starts. The app keeps operational diagnostics private and offers a concise error and Contact us action. In the Feed, close an attention notice to hide it for the current browser session. When no notices remain, the attention section disappears. This does not resolve or delete the issue, and a new stopped run produces a new notice. Worker leases prevent stale workers from overwriting terminal results and recover stale reservations. An interrupted execution can still require a new run. Context-summary and production-check jobs have separate durable queues.

​
Whole-application tests

Use the application-runtime evaluator to execute your actual entry point, tools and orchestration against capability, regression, incident and independent holdout cases. SDK results are explicitly client-reported and appear under Runtime, separate from prompt Bench scores. The normal Start benching action does not yet configure a hosted runner for arbitrary repositories.