Bench from coding agent

​

What headless means

​

Start from your agent

https://mcp.usebench.ai/mcp

Set up Bench for this project through the connected Bench MCP server.
Start with bench_capabilities, bench_whoami and bench_get_setup_status.
Detect the project language and install the appropriate Bench SDK if missing, or attach
its OpenTelemetry bridge if the framework already emits GenAI spans.
Send one real request so Bench recognizes the system, then check bench_list_systems.
Find every model call this system makes and register it with bench_register_calls:
a code-derived key, its messages as fragments with the resolved text, the model
configuration, the conditions that decide which pieces are sent, and how well you verified it.
Record the system's understanding with bench_put_context_source: purpose, rules,
examples, and architecture with models, expected volume and typical request sizes.
Show me Bench's quick scan findings and explain what to bench first.
Set up real app tests in this codebase, using synthetic data.
Keep credentials in my local secret environment and out of source and chat replies.
Ask before spending evaluations, changing billing, inviting people or publishing code.
Check the results and setup status, and tell me what is still incomplete.

​

Copy a setup prompt from Bench

​

Discover the system from your coding agent

1. Instrument and run once. Install the Bench SDK (TypeScript @benchai/sdk, Python trybench-sdk, Go github.com/trybench/bench-sdk/go, Rust trybench-sdk) and attach it to what the application already emits: the OpenTelemetry bridge (BenchSpanExporter in TypeScript, bench_sdk.otel.attach(bench) in Python) for frameworks that emit GenAI spans (Pydantic AI, Strands, Mastra, LangChain, OpenAI Agents SDK, Vercel AI SDK and similar), record_external_span for frameworks with their own tracing events, or nested AGENT, LLM and TOOL spans in custom code. One real request creates the system with the runtime, agents, model calls per agent, tools per agent and hand-offs that actually ran, classified by the OpenTelemetry GenAI attributes. Bench makes no model call for this.
2. Register the model calls. Traces never carry the editable template, so the agent reads the repository, finds every prompt the system sends (system prompts, instructions, templates in code, YAML, Markdown or JSON, prompts assembled across files), groups them by the model call that sends them and calls bench_register_calls on the system with one entry per call: a code-derived key, its messages as fragments holding the resolved text it got by running the code that builds them, the model configuration, the conditions that decide which pieces are sent, and how well it verified the text. A tool may also list an output_schema, a JSON Schema for what it returns; Bench checks the replies it simulates for that tool against it, and a schema Bench cannot use is ignored with a warning on that call. Bench stores each call as an evaluable component under the real repository and branch, shown as Call #id, and returns component ids. Registering again with the same keys keeps the same calls; calls missing from a later registration are kept and labelled not seen.
3. Link runtime evidence. The agent puts each component id on the spans of the model call that sends that prompt (bench.component_id), so production evidence and evaluations refer to the same prompt.
4. Write down what the system is for. From the README, docs, prompts and code, the agent saves understanding with bench_put_context_source, each as its own source with the files it came from: a short purpose (business_intent: who the system serves, what it does, what a good result looks like), policies and constraints (rules_constraints), examples (examples_feedback) and system structure (system_structure: architecture plus the models each prompt runs on, expected request volume and typical request sizes). Bench judges results against these, and a bench needs at least a purpose and rules or it stops with benchmark_needs_context. You can edit or add to them at any time on the system’s Understanding page.
5. Read the quick scan. As soon as the system exists, its Results tab, Feed, Findings and the AI systems list show quality findings (prompts whose quality is not measured yet, tools and hand-offs not tested yet) and cost findings (each model’s estimated cost per 1,000 typical requests next to a lower-priced model in the same family). No bench is needed for this. See quick scan.
6. Prepare real app tests and bench. Check bench_evaluation_allowance, ask for approval, then start the first bench with bench_start_system_evaluation.

​

Connect GitHub

1. Read bench_get_processing_notice. After the user authorizes the notice, call bench_acknowledge_processing with body containing its exact version, source: "github" and accepted: true.
2. Call bench_connect_github. Open the returned GitHub URL for the user. Retain the returned state; it expires after ten minutes.
3. The user selects the repositories on GitHub. The callback says to return to the coding agent; it does not create a Bench browser session.
4. Call bench_finish_github_connection with body: { "state": "..." }. If the result is pending, wait the returned poll_interval before polling. Only the account that initiated the flow can complete it.
5. Use bench_connection_status, bench_activate_installation, bench_list_repos and bench_list_branches to select the repository and branch. Run bench_scan_repo, then read bench_get_scan and bench_list_systems.

​

Prompts, systems and understanding

​

Cases, criteria and files

• bench_save_test_case and bench_save_criterion add, update or pause entries for a selected prompt component. Preserve expected values and their JSON types.
• bench_preview_dataset previews CSV, XLSX, JSON or JSONL up to 4 MB.
• bench_upload_dataset saves the approved mapping and content. Use bench_remap_dataset to create a corrected version.
• bench_import_dataset_cases adds a specific dataset version to a prompt’s test library. Uploading a dataset alone does not create an evaluation suite.
• bench_compare_dataset compares recorded outputs; it does not run a model or replay the live app. Results remain available through bench_list_dataset_reports.

​

Run, inspect and improve

​

Set up real app testing

​

Plans and credentials