> ## Documentation Index > Fetch the complete documentation index at: https://docs.usebench.ai/llms.txt > Use this file to discover all available pages before exploring further. # Bench from coding agent > Set up, test and manage Bench through MCP and the SDK without using the dashboard. MCP and API workflows are available in production and development. The new SDK platform clients require the current SDK source build; the published 0.1.0 packages provide tracing, real app tests and simulations. See [installing platform clients](/sdk/platform#install-the-platform-clients). ## What headless means Your coding agent can manage every Bench workflow: setup, GitHub repositories, prompts and systems, context, files and datasets, tests and criteria, evaluations, quality results, history, real app testing, production feedback, workspaces and billing. MCP and the SDK call the same API as the dashboard. Your plan and credential permissions apply to every request. Sign-in, GitHub repository grants and payment confirmation still happen with their identity or payment provider. The agent starts those flows and resumes after approval. It cannot grant itself repository access or silently buy a plan. You do not need to complete Bench's browser onboarding wizard. ## Start from your agent Configure a remote HTTP MCP server using your coding agent's MCP settings: ```text theme={null} https://mcp.usebench.ai/mcp ``` For development use `https://mcp.staging.usebench.ai/mcp` with a development account. Complete the connector's OAuth sign-in. A first-time user receives a normal Free Bench account. Reconnecting never replenishes spent evaluations. Paste this into the coding agent: ```text theme={null} Set up Bench for this project through the connected Bench MCP server. Start with bench_capabilities, bench_whoami and bench_get_setup_status. Show me the processing notice and ask before acknowledging a new data source. Connect GitHub, let me approve repository access, then finish the connection. Find this repository and branch, scan them, and show the recognized systems and prompts. Help me add the system's purpose, test cases and pass/fail criteria. Detect the project language and install the appropriate Bench SDK if missing. Set up SDK tracing and real app tests in this codebase, using synthetic data. Run an approved test, explicitly publish its report, and verify Bench received it. Keep credentials in my local secret environment and out of source and chat replies. Ask before spending evaluations, changing billing, inviting people or publishing code. Check the results and setup status, and tell me what is still incomplete. ``` ## Copy a setup prompt from Bench Use **Copy prompt** beside the coding-agent prompt in Getting started or onboarding. Repository setup, Real app testing, Production, Understanding and the test library also show a short, scrollable prompt preview with a green **Copy prompt** button. Copy the full prompt into your agent. The copied scope contains identifiers for the selected system and components; it does not copy saved prompts or credentials. Choose **Open in coding agent** beside **Copy prompt** to open the Codex app, Cursor or Claude Code in your terminal. Each option shows its agent's icon. The prompt is also copied so you can paste it if the installed app does not open. Choose **Copy for another agent or terminal** to use a different coding agent. The same menu is available in Fixes. In the test library, Cases and Criteria remain visible above the compact agent card. Ask the agent to read the existing library before adding cases or criteria. In Real app testing, the prompt explicitly covers SDK installation, a safe app adapter, execution and report publication. Installing the SDK alone does not prove that an application test ran. ## Connect GitHub 1. Read `bench_get_processing_notice`. After the user authorizes the notice, call `bench_acknowledge_processing` with `body` containing its exact `version`, `source: "github"` and `accepted: true`. 2. Call `bench_connect_github`. Open the returned GitHub URL for the user. Retain the returned `state`; it expires after ten minutes. 3. The user selects the repositories on GitHub. The callback says to return to the coding agent; it does not create a Bench browser session. 4. Call `bench_finish_github_connection` with `body: { "state": "..." }`. If the result is `pending`, wait the returned `poll_interval` before polling. Only the account that initiated the flow can complete it. 5. Use `bench_connection_status`, `bench_activate_installation`, `bench_list_repos` and `bench_list_branches` to select the repository and branch. Run `bench_scan_repo`, then read `bench_get_scan` and `bench_list_systems`. An expired or consumed state requires checking connection status before restarting. If a GitHub organization requires administrator approval, wait for that approval. Installing the Bench SDK before connecting GitHub is supported. To add another repository later, ask the agent to list existing installations and repositories first. If GitHub has not granted access, repeat the authorization flow and select the additional repository on GitHub. Then select its branch, scan it and review the discovered systems. Existing connected repositories remain available. For a project without GitHub, upload explicit prompt content instead. ## Prompts, systems and understanding Use `bench_upload_prompts` for prompt files or pasted text. Authorize the `upload` source first. It returns an upload repository and branch that other tools can use. Repository prompts are updated by editing source and rescanning, or by creating an approved prompt PR. A saved recommendation does not change deployed code. Use `bench_create_system`, `bench_update_system`, `bench_add_system_component` and the system-connection tools to organize prompts, tools and harnesses. Obtain component and call-site IDs from real scan results. Read the current context with `bench_get_system_context` before saving a correction with `bench_save_system_context`. Context writes use `expected_version`; a `context_changed` error means re-read and reconcile instead of overwriting. `bench_upload_context_document` accepts text, Markdown, JSON and JSONL up to 60 KB. Use `bench_seed_system_context` and `bench_summarize_system_context` to refresh the saved understanding. Provider connections can import selected Langfuse or LangSmith evidence with explicit content consent. ## Cases, criteria and files * `bench_save_test_case` and `bench_save_criterion` add, update or pause entries for a selected prompt component. Preserve expected values and their JSON types. * `bench_preview_dataset` previews CSV, XLSX, JSON or JSONL up to 4 MB. * `bench_upload_dataset` saves the approved mapping and content. Use `bench_remap_dataset` to create a corrected version. * `bench_import_dataset_cases` adds a specific dataset version to a prompt's test library. Uploading a dataset alone does not create an evaluation suite. * `bench_compare_dataset` compares recorded outputs; it does not run a model or replay the live app. Results remain available through `bench_list_dataset_reports`. MCP upload tools take file **content**, not a path on your computer. The coding agent reads the selected file locally, then supplies UTF-8 `content` or binary `content_base64`. The SDK clients accept text or bytes directly. ## Run, inspect and improve Check `bench_evaluation_allowance`, then get approval before starting `bench_start_system_evaluation`. Poll `bench_get_evaluation`. Read exact saved criteria, cases, scores and recommendations with `bench_get_evaluation_artifacts`. Use `bench_list_evaluations` and `bench_list_system_runs` for history; cancellation, reruns, archival and restoration are available through the corresponding tools. A rerun evaluates current inputs; it does not rewrite historical scores. Use `bench_get_fix_brief` to prepare a source change. Run independent checks before publishing it. `bench_open_prompt_pr` creates a PR after approval and requires a paid plan. It does not merge or deploy the change. ## Set up real app testing First install the runtime SDK for the project's language, unless a compatible version is already installed: | Language | Installation | | ----------------------- | ------------------------------------------------ | | JavaScript / TypeScript | `npm install @benchai/sdk` | | Python | `python -m pip install trybench-sdk` | | Go | `go get github.com/trybench/bench-sdk/go@latest` | | Rust | `cargo add trybench-sdk` | Use the language-specific SDK docs for the installed version. Existing provider SDKs, Vercel AI SDK and custom provider abstractions can stay in place; instrument the actual application entry point and model/tool calls. The coding agent can obtain an SDK key through OAuth using `bench_create_api_key` or `bench_get_setup_key`, store it in your environment and instrument the app's actual entry point. A scoped key can publish traces and reports without account management authority. Set an explicit application environment such as `staging`. Use the SDK's `evaluateSystem` / `evaluate_system` / `EvaluateSystem` to execute real app cases, or its simulation API for scripted conversations. Instrument model, agent and tool calls, check the final result and tool effects, and publish the report explicitly. See [Real app testing](/sdk/system-evaluation) for adapters and examples. Use `bench_sdk_status`, `bench_list_runtime_evaluations` and `bench_get_production_health` to verify received evidence. Creating a key or writing an adapter does not prove that a request ran. Uploaded reports are labeled as client-reported evidence. Hosted execution can be configured through `bench_get_runtime_settings` and `bench_set_runtime_settings` when it is enabled in that environment. A missing hosted runner does not prevent local SDK real app tests. Production feedback, automatic checks and runtime-fix publication have explicit tools and retain their existing limits and approval requirements. ## Plans and credentials | Capability | Rules | | ------------------------------------------------------------------- | ----------------------------------------------------------------------------- | | Connect MCP and manage authorized data | Available on all plans | | Bench evaluations | Share the account allowance and the API key's spending cap | | Choose models/providers | Active Growth or Enterprise required; Free/Builder use defaults | | Test-case ceilings | Enforced by the API for each plan | | Account setup, API-key management, workspace membership and billing | OAuth account authority required; organization admin rules still apply | | Repository-scoped key | Cannot access another repository or a system containing excluded repositories | | Automatic evaluations | Require account authorization and explicit approval to enable | `bench_capabilities`, `bench_evaluation_allowance` and `bench_list_plans` return `pricing_url` and `upgrade_url` for the current environment. Plan-limit errors also return these links directly, so the agent can immediately show where to upgrade. Dev links open `https://stg.usebench.ai/plans`; production links open the production platform. A key-specific cap points to `manage_key_url` instead. Use `bench_list_plans`, `bench_create_checkout` or `bench_get_billing_portal` to prepare approved subscription changes. Return the Stripe URL for coupon entry and payment confirmation. Never ask the developer to paste card details into the agent. Read the [operation reference](/reference/headless) for every input contract and [SDK platform clients](/sdk/platform) to automate the same workflows in code.