Skip to main content
For setup and management entirely from a coding agent, see the headless workflow and SDK platform clients.

​
Copy this into your coding agent

Paste into Cursor, Codex, Claude Code, or another coding agent in your project.
Bench MCP connects your coding agent to Bench. From the agent you define what a good result means for your business, register the system’s prompts, record its rules, read the quick scan, start a bench of the whole AI system (prompts, tools, hand-offs and model configuration) and review the recommended changes, without switching tools. You can also run the server locally over stdio with an API key instead of the hosted OAuth server: npx -y @benchai/mcp (npm package @benchai/mcp, set BENCH_API_KEY).
Your plan determines the access and evaluation allowance shown in your Bench account. Connecting does not start an evaluation or apply changes.

​
Connect your coding agent

The Set up using a coding agent prompt in onboarding and Getting started already embeds the connect command with a server-only key, so Claude Code or Codex connects the Bench MCP itself as its first step and then installs the SDK, registers prompts, records understanding and runs the quick scan. See Bench from coding agent. To connect by hand, open MCP in the Bench app and select your client, or use the terminal command or configuration shown below. Other clients shows the server configuration for compatible MCP clients. The starter prompt asks for approval before using an evaluation or applying a fix.
Run this in your terminal:
Open Claude Code, run /mcp, and follow the sign-in prompts. Use the same account you use in the Bench app.To inspect the configuration:
The default scope is the current project. Add --scope user if you want the connection available across projects.

​
Check the connection

Ask your coding agent:
Use Bench to show my account and connected repositories. Do not start an evaluation yet.
The agent can use bench_whoami and bench_list_repos. If the repository you need is missing, connect GitHub in Bench and confirm that the GitHub App has access to it.

​
Run your first bench

1

Recognize the system and register its model calls

Ask: “Install the Bench SDK, send one real request, then register every model call this system makes with bench_register_calls and show me the system.”One trace draws the runtime, agents, model calls, tools and hand-offs. Registered calls (a code-derived key, messages as fragments, model configuration, conditions, verification level) become evaluable components, shown as Call #id, without a GitHub connection. A repository scan (bench_scan_repo) is the fallback for connected repositories, and the prompt-upload flow works without either.
2

Record understanding and read the quick scan

Ask: “Record the purpose, rules, examples and system structure with bench_put_context_source, then show the quick scan findings.”A bench needs a recorded purpose and rules. The quick scan lists prompts whose quality is not measured yet, tools and hand-offs not tested yet, and each model’s estimated cost next to a cheaper model, before any bench runs.
3

Confirm the evaluation

Ask: “Bench this system. Explain the input and context first, then ask me before using an evaluation.”Bench creates or reuses criteria and test cases, scores each registered prompt, and evaluates candidate improvements within your plan’s limits. Each model-heavy stage may take up to one hour.
4

Follow progress

Your agent can poll bench_get_evaluation. Current system-first runs do not require a rubric-review step. An older run may return awaiting_review; inspect its saved cases before using bench_submit_run_review to continue.
5

Investigate the result

Ask: “Show the failed checks, explain why they failed, and compare the proposed prompt and model with the baseline. Include cost and any unscored checks.”Use the actual saved benchmark and optimization results. A missing result or failed provider call is not evidence that the AI behavior is bad.

​
Review a proposed fix

Use bench_get_fix_brief with a completed run ID to export its pinned repair evidence. An approved coding agent can propose changes, then an independent validator checks them. See Code fixes. This read-only tool does not start a paid evaluation, execute code or publish a PR. A recommendation is not a deployed fix. Inspect the proposed prompt, selected model, measured quality, cost, and limitations before changing your application. bench_open_prompt_pr is an external write action. Ask explicitly before using it. It requires a connected repository and a supported prompt location. Uploaded prompts do not have a repository file to change. Creating a PR does not merge or deploy it.
Prompt evaluation does not execute every part of a multi-agent application. Tool side effects, orchestration, real-time voice behavior, and production outcomes need runtime evidence and their own execution checks. Do not describe a prompt-level result as a validated end-to-end system fix.

​
Useful requests

The system-first preview adds system context, test-library and production tools. They require the matching API and MCP deployment.
  • bench_start_system_evaluation: bench all recognized prompts in a system after confirming allowance use. This is the Free whole-system path. bench_start_evaluation keeps its explicit prompt-selection behavior.
  • bench_register_calls: register the model calls your agent found in the repository, one entry per call (code-derived key, messages as fragments, model configuration, conditions, verification level), as evaluable components of a system, without a GitHub scan. Returns one component id per call (shown as Call #id) for bench.component_id on the matching spans. bench_register_prompts is deprecated.
  • bench_put_context_source: record understanding by category: business_intent, rules_constraints, examples_feedback, system_structure (architecture, models, expected volume, typical request sizes).
  • bench_list_systems / bench_get_system: find a system and its prompt IDs.
  • bench_get_system_context / bench_list_context_sources: inspect versioned evidence.
  • bench_save_system_context: save an explicit correction for future runs.
  • bench_list_test_library / bench_save_test_case / bench_save_criterion: customize cases and criteria.
  • bench_list_datasets / bench_import_dataset_cases: use an existing mapped dataset.
  • bench_evaluation_allowance: check the account and key limits before spending.
  • bench_get_evaluation_artifacts: inspect the exact saved run evidence.
  • bench_list_production_traces / bench_get_production_trace: inspect recorded behavior.
  • bench_check_production_span / bench_get_production_check / bench_retry_production_check: run and follow a background check.
Read Production checks for capture, consent and retention. Repository-restricted keys use the system-scoped tools above. Legacy per-prompt result tools cannot verify all the repositories behind old evidence and reject restricted keys. A shared prompt does not share another system’s private cases.
  • “Find which prompts are responsible for this behavior.”
  • “Explain this failed check and the evidence used to judge it.”
  • “Compare the recommended model’s cost and quality with the baseline.”
  • “Prepare a prompt change for my review. Do not open a PR yet.”

​
Troubleshooting

Check the configured URL, inspect the client’s MCP status, and complete sign-in again. For a local environment, copy the configuration from that environment’s MCP page instead of mixing production endpoints with local credentials.
Confirm the Bench account, GitHub account, and repository access granted to the Bench GitHub App. Scoped API keys may intentionally hide other repositories.
MCP connection is free, but evaluations share your plan allowance. Check Billing for the current balance and plan options.
Read the run’s status and error. A legacy review pause needs a case review. An execution failure needs a retry or configuration correction, not a fabricated quality score.
Read Security and limits before sharing access or enabling write tools.