One system, an ongoing loop
1. Connect and understand
Connect GitHub, upload prompts, or instrument your application with the SDK. MCP lets your coding agent work with Bench’s API. GitHub discovery identifies supported prompt, model and tool references. SDK capture can declare runtime systems without a repository scan. Each system has its own versioned context. Code describes structure, documents and user rules describe intended behavior, and traces describe observed behavior. A repeated mistake is not a new policy. Add optional context or correct Bench’s understanding at any time. The recognition catalog covers 18 framework and SDK families, with different levels of structural support. See framework and runtime support for the exact distinction between identifying a system and executing it.How context becomes an evaluation
The API stores versioned context and source references in Postgres. It retrieves relevant
evidence for the prompt being evaluated. It does not require a graph database or vector
database for this preview. Shared repository documentation is background, not automatically
a policy for every system. Conflicting evidence remains a discrepancy to investigate.
Adding evidence queues a background summary update. It does not block navigation or
silently start a paid evaluation. If summarization fails, the saved evidence remains.
The quick repository review checks quality risks and model
prices. It is not a measured runtime score or proof of savings.
2. Define and test good behavior
Click Start benching to run an evaluation within your allowance, including your Free plan allowance. Bench freezes the selected inputs and context revision, combines generated and enabled user criteria, and builds a suite including selected test-library cases. Expected answers stay out of the tested model’s input. For example, a refund assistant may need to apply a 30-day policy, handle missing order details, and escalate exceptions. Boundary cases test day 30 and day 31. Conversation cases test whether a later message changes the rules or causes a role swap. Checks need evidence of the relevant behavior; a topology diagram alone cannot prove a tool executed. Bench measures the current prompt/model, compares candidates, and saves the rubric, cases and results for investigation. The current execution unit is primarily a prompt/model, not arbitrary end-to-end multi-agent application code. The separate application-runtime evaluator calls your actual entry point with explicitly configured cases, instrumentation and final-outcome checks. Reports appear in the system’s Runtime tab. Optimization-suite results need independent holdout validation before a consequential rollout. For a grounding check, Bench needs the source that an answer is meant to follow and observable claims to compare. For tool-use checks, it needs an executable harness or recorded tool evidence appropriate to the check. Missing evidence is a coverage gap, not an invented passing score. A model judge can be wrong, so important criteria need human-reviewed reference cases and false-positive/false-negative testing.3. Investigate and propose a fix
Open History to inspect a run. The investigation panel separates Activity, Test results, Observations, Errors and Fixes. Activity shows saved stages and cases, not a model’s hidden reasoning. Repeated execution errors are grouped, with the affected case count. From Fixes, compare the source prompt with the proposed prompt and model. Choose Prepare fix prompt to copy the evidence into a trusted coding agent, or download Fix brief. A coding agent can propose a source change; it cannot certify its own success. The local validator pins a clean source commit, accepts bounded allowlisted replacements, and independently checks that the incident fails on baseline and passes on candidate. Regression and holdout checks must pass. It also replays the emitted patch to confirm it produces the exact validated bytes. The result is a patch and report for review.4. Open a PR, then measure the deployed result
In a GitHub-connected fix workflow, publishing a proposed change means writing a feature branch and opening a pull request in the connected repository. It does not mean merging or deploying. Branch protection and your review process still apply. Bench already has a prompt PR endpoint and MCP tool. That endpoint does not enforce the new code validator’s report, so it must not be used as an autonomous arbitrary-code publisher. The local validator itself never pushes or opens a PR. A hosted publisher must verify validation evidence and the current base before a separately authorized write. After an approved merge and deployment, production evidence can show whether behavior actually improved. An offline higher score or opened PR is not a resolved production issue.What the SDK, Jev, E2B and Axiom each do
The web app and MCP call the authenticated Bench API. The API coordinates repository discovery, Postgres records and evaluation requests to the Python pipeline over gRPC. Provider calls happen server-side. The browser polls saved progress and can close without stopping a run. Full evaluations currently use an in-process worker, so they cannot resume arbitrary interrupted calls. Worker leases fence stale writes and recover reservations; summary and production-check jobs use durable database queues.
Jev’s Noul primitive returns a probability for a yes/no question, such as whether a
reply violates a supplied refund policy. Bench keeps the result review-required; it
does not convert a model probability into ground truth. The TypeSafe skill shaped this
narrow-question design. See TypeSafe’s Noul reference.
The E2B synthetic smoke test creates short-lived sandboxes, runs a deliberately broken
fixture and its corrected version, checks outbound TLS denial, and verifies cleanup.
It does not upload customer code, generate a fix with an agent, or certify all isolation
properties. Full hosted execution needs separate author/validator environments, secret
boundaries, controlled networking and operational limits.
Axiom is a separate observability destination, not an evaluator or a model-price
source. The API and pipeline have bounded, asynchronous OpenTelemetry exporters.
The internal exporter records operational metadata, not prompts, responses or
customer tool payloads. Failure to export must not stop a bench. See
Axiom’s OpenTelemetry documentation.
Customer SDK traces currently go to Bench’s Postgres store, not automatically to
Axiom. Native OTLP ingestion and forwarding customer traces are separate integrations.
Jev is currently opt-in for selected production interactions; it is not called by
the static repository review or used to generate every rubric.