Skip to main content
MCP and API workflows are available in production and development. The new SDK platform clients require the current SDK source build; the published 0.1.0 packages provide tracing, real app tests and simulations. See installing platform clients.

​
What headless means

Your coding agent can manage every Bench workflow: setup, GitHub repositories, prompts and systems, context, files and datasets, tests and criteria, evaluations, quality results, history, real app testing, production feedback, workspaces and billing. MCP and the SDK call the same API as the dashboard. Your plan and credential permissions apply to every request. Sign-in, GitHub repository grants and payment confirmation still happen with their identity or payment provider. The agent starts those flows and resumes after approval. It cannot grant itself repository access or silently buy a plan. You do not need to complete Bench’s browser onboarding wizard.

​
Start from your agent

Configure a remote HTTP MCP server using your coding agent’s MCP settings:
For development use https://mcp.staging.usebench.ai/mcp with a development account. Complete the connector’s OAuth sign-in. A first-time user receives a normal Free Bench account. Reconnecting never replenishes spent evaluations. Paste this into the coding agent:

​
Copy a setup prompt from Bench

Use Copy prompt beside the coding-agent prompt in Getting started or onboarding. Repository setup, Real app testing, Production, Understanding and the test library also show a short, scrollable prompt preview with a green Copy prompt button. Copy the full prompt into your agent. The copied scope contains identifiers for the selected system and components; it does not copy saved prompts or credentials. Choose Open in coding agent beside Copy prompt to open the Codex app, Cursor or Claude Code in your terminal. Each option shows its agent’s icon. The prompt is also copied so you can paste it if the installed app does not open. Choose Copy for another agent or terminal to use a different coding agent. The same menu is available in Fixes. In the test library, Cases and Criteria remain visible above the compact agent card. Ask the agent to read the existing library before adding cases or criteria. In Real app testing, the prompt explicitly covers SDK installation, a safe app adapter, execution and report publication. Installing the SDK alone does not prove that an application test ran.

​
Connect GitHub

  1. Read bench_get_processing_notice. After the user authorizes the notice, call bench_acknowledge_processing with body containing its exact version, source: "github" and accepted: true.
  2. Call bench_connect_github. Open the returned GitHub URL for the user. Retain the returned state; it expires after ten minutes.
  3. The user selects the repositories on GitHub. The callback says to return to the coding agent; it does not create a Bench browser session.
  4. Call bench_finish_github_connection with body: { "state": "..." }. If the result is pending, wait the returned poll_interval before polling. Only the account that initiated the flow can complete it.
  5. Use bench_connection_status, bench_activate_installation, bench_list_repos and bench_list_branches to select the repository and branch. Run bench_scan_repo, then read bench_get_scan and bench_list_systems.
An expired or consumed state requires checking connection status before restarting. If a GitHub organization requires administrator approval, wait for that approval. Installing the Bench SDK before connecting GitHub is supported. To add another repository later, ask the agent to list existing installations and repositories first. If GitHub has not granted access, repeat the authorization flow and select the additional repository on GitHub. Then select its branch, scan it and review the discovered systems. Existing connected repositories remain available. For a project without GitHub, upload explicit prompt content instead.

​
Prompts, systems and understanding

Use bench_upload_prompts for prompt files or pasted text. Authorize the upload source first. It returns an upload repository and branch that other tools can use. Repository prompts are updated by editing source and rescanning, or by creating an approved prompt PR. A saved recommendation does not change deployed code. Use bench_create_system, bench_update_system, bench_add_system_component and the system-connection tools to organize prompts, tools and harnesses. Obtain component and call-site IDs from real scan results. Read the current context with bench_get_system_context before saving a correction with bench_save_system_context. Context writes use expected_version; a context_changed error means re-read and reconcile instead of overwriting. bench_upload_context_document accepts text, Markdown, JSON and JSONL up to 60 KB. Use bench_seed_system_context and bench_summarize_system_context to refresh the saved understanding. Provider connections can import selected Langfuse or LangSmith evidence with explicit content consent.

​
Cases, criteria and files

  • bench_save_test_case and bench_save_criterion add, update or pause entries for a selected prompt component. Preserve expected values and their JSON types.
  • bench_preview_dataset previews CSV, XLSX, JSON or JSONL up to 4 MB.
  • bench_upload_dataset saves the approved mapping and content. Use bench_remap_dataset to create a corrected version.
  • bench_import_dataset_cases adds a specific dataset version to a prompt’s test library. Uploading a dataset alone does not create an evaluation suite.
  • bench_compare_dataset compares recorded outputs; it does not run a model or replay the live app. Results remain available through bench_list_dataset_reports.
MCP upload tools take file content, not a path on your computer. The coding agent reads the selected file locally, then supplies UTF-8 content or binary content_base64. The SDK clients accept text or bytes directly.

​
Run, inspect and improve

Check bench_evaluation_allowance, then get approval before starting bench_start_system_evaluation. Poll bench_get_evaluation. Read exact saved criteria, cases, scores and recommendations with bench_get_evaluation_artifacts. Use bench_list_evaluations and bench_list_system_runs for history; cancellation, reruns, archival and restoration are available through the corresponding tools. A rerun evaluates current inputs; it does not rewrite historical scores. Use bench_get_fix_brief to prepare a source change. Run independent checks before publishing it. bench_open_prompt_pr creates a PR after approval and requires a paid plan. It does not merge or deploy the change.

​
Set up real app testing

First install the runtime SDK for the project’s language, unless a compatible version is already installed: Use the language-specific SDK docs for the installed version. Existing provider SDKs, Vercel AI SDK and custom provider abstractions can stay in place; instrument the actual application entry point and model/tool calls. The coding agent can obtain an SDK key through OAuth using bench_create_api_key or bench_get_setup_key, store it in your environment and instrument the app’s actual entry point. A scoped key can publish traces and reports without account management authority. Set an explicit application environment such as staging. Use the SDK’s evaluateSystem / evaluate_system / EvaluateSystem to execute real app cases, or its simulation API for scripted conversations. Instrument model, agent and tool calls, check the final result and tool effects, and publish the report explicitly. See Real app testing for adapters and examples. Use bench_sdk_status, bench_list_runtime_evaluations and bench_get_production_health to verify received evidence. Creating a key or writing an adapter does not prove that a request ran. Uploaded reports are labeled as client-reported evidence. Hosted execution can be configured through bench_get_runtime_settings and bench_set_runtime_settings when it is enabled in that environment. A missing hosted runner does not prevent local SDK real app tests. Production feedback, automatic checks and runtime-fix publication have explicit tools and retain their existing limits and approval requirements.

​
Plans and credentials

bench_capabilities, bench_evaluation_allowance and bench_list_plans return pricing_url and upgrade_url for the current environment. Plan-limit errors also return these links directly, so the agent can immediately show where to upgrade. Dev links open https://stg.usebench.ai/plans; production links open the production platform. A key-specific cap points to manage_key_url instead. Use bench_list_plans, bench_create_checkout or bench_get_billing_portal to prepare approved subscription changes. Return the Stripe URL for coupon entry and payment confirmation. Never ask the developer to paste card details into the agent. Read the operation reference for every input contract and SDK platform clients to automate the same workflows in code.