> ## Documentation Index > Fetch the complete documentation index at: https://docs.usebench.ai/llms.txt > Use this file to discover all available pages before exploring further. # How Bench works > Understand the path from an AI system to evidence, evaluation and a reviewed fix. ## One system, an ongoing loop ```text theme={null} Connect a source -> recognize systems -> collect context | criteria + test cases | evaluate -> investigate failures | propose -> validate -> review PR | merge and deploy with your controls | observe production -> improve future tests ``` The diagram describes the intended lifecycle. The local preview supports evaluation, production capture, background checks and a separate code-fix validation workflow. Hosted automatic code generation and a validation-gated PR publisher are not yet connected. ## 1. Connect and understand Connect GitHub, upload prompts, or instrument your application with the SDK. MCP lets your coding agent work with Bench's API. GitHub discovery identifies supported prompt, model and tool references. SDK capture can declare runtime systems without a repository scan. Each system has its own versioned context. Code describes structure, documents and user rules describe intended behavior, and traces describe observed behavior. A repeated mistake is not a new policy. Add optional context or correct Bench's understanding at any time. The recognition catalog covers 18 framework and SDK families, with different levels of structural support. See [framework and runtime support](/guides/framework-support) for the exact distinction between identifying a system and executing it. ### How context becomes an evaluation | Evidence | How Bench uses it | | -------------------------------------- | --------------------------------------------------------------------------------- | | Business outcome and explicit policies | Define the success criteria and constraints | | Prompt, model and tool definitions | Identify the current setup and the behavior it claims to support | | Documents and repository background | Ground criteria in cited evidence, scoped to the system | | Expected answers and labeled examples | Supply test cases without putting the answers into the model's input | | Production traces and conversations | Reveal realistic inputs and possible failure patterns, not automatic ground truth | | User corrections | Update a case or system rule for future benches, preserving previous results | The API stores versioned context and source references in Postgres. It retrieves relevant evidence for the prompt being evaluated. It does not require a graph database or vector database for this preview. Shared repository documentation is background, not automatically a policy for every system. Conflicting evidence remains a discrepancy to investigate. Adding evidence queues a background summary update. It does not block navigation or silently start a paid evaluation. If summarization fails, the saved evidence remains. The [quick repository review](/guides/quick-scan) checks quality risks and model prices. It is not a measured runtime score or proof of savings. ## 2. Define and test good behavior Click **Start benching** to run an evaluation within your allowance, including your Free plan allowance. Bench freezes the selected inputs and context revision, combines generated and enabled user criteria, and builds a suite including selected test-library cases. Expected answers stay out of the tested model's input. For example, a refund assistant may need to apply a 30-day policy, handle missing order details, and escalate exceptions. Boundary cases test day 30 and day 31. Conversation cases test whether a later message changes the rules or causes a role swap. Checks need evidence of the relevant behavior; a topology diagram alone cannot prove a tool executed. Bench measures the current prompt/model, compares candidates, and saves the rubric, cases and results for investigation. The current execution unit is primarily a prompt/model, not arbitrary end-to-end multi-agent application code. The separate [application-runtime evaluator](/sdk/system-evaluation) calls your actual entry point with explicitly configured cases, instrumentation and final-outcome checks. Reports appear in the system's **Runtime** tab. Optimization-suite results need independent holdout validation before a consequential rollout. For a grounding check, Bench needs the source that an answer is meant to follow and observable claims to compare. For tool-use checks, it needs an executable harness or recorded tool evidence appropriate to the check. Missing evidence is a coverage gap, not an invented passing score. A model judge can be wrong, so important criteria need human-reviewed reference cases and false-positive/false-negative testing. ## 3. Investigate and propose a fix Open **History** to inspect a run. The investigation panel separates **Activity**, **Test results**, **Observations**, **Errors** and **Fixes**. Activity shows saved stages and cases, not a model's hidden reasoning. Repeated execution errors are grouped, with the affected case count. From **Fixes**, compare the source prompt with the proposed prompt and model. Choose **Prepare fix prompt** to copy the evidence into a trusted coding agent, or download **Fix brief**. A coding agent can propose a source change; it cannot certify its own success. The local validator pins a clean source commit, accepts bounded allowlisted replacements, and independently checks that the incident fails on baseline and passes on candidate. Regression and holdout checks must pass. It also replays the emitted patch to confirm it produces the exact validated bytes. The result is a patch and report for review. ## 4. Open a PR, then measure the deployed result In a GitHub-connected fix workflow, publishing a proposed change means writing a feature branch and opening a pull request in the connected repository. It does not mean merging or deploying. Branch protection and your review process still apply. Bench already has a prompt PR endpoint and MCP tool. That endpoint does not enforce the new code validator's report, so it must not be used as an autonomous arbitrary-code publisher. The local validator itself never pushes or opens a PR. A hosted publisher must verify validation evidence and the current base before a separately authorized write. After an approved merge and deployment, production evidence can show whether behavior actually improved. An offline higher score or opened PR is not a resolved production issue. ## What the SDK, Jev, E2B and Axiom each do The web app and MCP call the authenticated Bench API. The API coordinates repository discovery, Postgres records and evaluation requests to the Python pipeline over gRPC. Provider calls happen server-side. The browser polls saved progress and can close without stopping a run. Full evaluations currently use an in-process worker, so they cannot resume arbitrary interrupted calls. Worker leases fence stale writes and recover reservations; summary and production-check jobs use durable database queues. | Component | Responsibility | | --------------------- | -------------------------------------------------------------------------------------------------------- | | SDK | Capture runtime interactions and explicitly evaluate the actual application entry point | | Bench API and workers | Scope data, preserve context versions, enforce budgets, schedule checks and retain evidence | | Evaluation pipeline | Generate criteria/cases, measure prompt/model behavior and compare candidates | | Jev | Optionally judge a narrow semantic question against selected evidence | | Coding agent | Propose a bounded source-code change | | E2B | Run pinned application adapters in opt-in, network-disabled sandboxes, not provide a correctness verdict | | Axiom | Receive Bench's internal operational traces, including latency, usage, stage and errors | | GitHub | Review and merge a separately authorized proposed change | Jev's Noul primitive returns a probability for a yes/no question, such as whether a reply violates a supplied refund policy. Bench keeps the result review-required; it does not convert a model probability into ground truth. The TypeSafe skill shaped this narrow-question design. See [TypeSafe's Noul reference](https://docs.typesafe.ai/primitives/noul). The E2B synthetic smoke test creates short-lived sandboxes, runs a deliberately broken fixture and its corrected version, checks outbound TLS denial, and verifies cleanup. It does not upload customer code, generate a fix with an agent, or certify all isolation properties. Full hosted execution needs separate author/validator environments, secret boundaries, controlled networking and operational limits. Axiom is a separate observability destination, not an evaluator or a model-price source. The API and pipeline have bounded, asynchronous OpenTelemetry exporters. The internal exporter records operational metadata, not prompts, responses or customer tool payloads. Failure to export must not stop a bench. See [Axiom's OpenTelemetry documentation](https://axiom.co/docs/send-data/opentelemetry). Customer SDK traces currently go to Bench's Postgres store, not automatically to Axiom. Native OTLP ingestion and forwarding customer traces are separate integrations. Jev is currently opt-in for selected production interactions; it is not called by the static repository review or used to generate every rubric. ## Keep improving without rewriting history The SDK defaults to metadata-only capture. Content capture requires a deliberate choice. Recording telemetry does not consume evaluation credits; executing production checks does. Checks run in background jobs, and failed infrastructure calls remain errors rather than scores. Review a production interaction, supply the intended answer, and save it as a test case. Future evaluations can then use it. New traces do not silently become golden answers, rewrite policies, trigger unrestricted spending or authorize deployment. Next: [SDK setup](/sdk/quickstart), [production checks](/sdk/production-checks), [code fixes](/guides/automatic-fixes), and [validation steps](/guides/validation).