Set up using a coding agent
Open Getting started in Bench and choose Set up using a coding agent. Copy the single prompt into Claude Code or Codex inside your application repository. The prompt embeds the Bench MCP connect command with a server-only key, so the agent connects itself; you do not paste keys by hand. The agent then:- Installs the Bench SDK for the repository’s language, or attaches the OpenTelemetry bridge to a framework that already emits GenAI spans.
- Sends one real request so Bench recognizes the system: runtime, agents, model calls and tools per agent, and hand-offs.
- Registers every model call with
bench_register_calls: a code-derived key, its messages as fragments holding the resolved text, the model configuration, the conditions that decide which pieces are sent, and how well the agent verified it. - Records understanding: purpose, policies, examples, and architecture with the models, expected volume and typical request sizes.
- Runs Bench’s quick scan: potential failures, quality improvements in prompts, tools, hand-offs and model configuration, and model-cost reductions.
- Prepares real app tests, and starts the first bench if you approved the spend.
Manual setup
Open Manual setup for a stepper that covers the SDK, GitHub and MCP separately. Use it when you prefer to do each step yourself or cannot run a coding agent in the repository.1
Connect your application
Install the SDK or attach the OpenTelemetry bridge, choose an environment and
send one real request. The step completes after Bench receives an event;
creating a key alone is not evidence that instrumentation works. Follow the
SDK quickstart. Then register the system’s prompts from your
coding agent or, as a fallback, connect GitHub and scan the repository.
2
Check your system’s understanding
Open an AI system’s Understanding page. Add or correct what success means:
for example, “refund an eligible order exactly once” and the relevant refund
policy. A bench needs a recorded purpose and rules; without them it stops with
benchmark_needs_context. See context.3
Read the quick scan
The system’s Results tab lists quality and cost findings as soon as the
system exists. Each finding has a Bench button. See quick scan.
4
Run and inspect a Bench
Select Start benching on a system to compare prompts and models. Inspect
failed cases and the evidence behind recommendations. Use Real app testing
to inspect tests of application behavior and tool effects. See
real app testing.
5
Connect GitHub, optionally
Connect GitHub when you want background rescans on new commits and pull
requests with proposed prompt changes. It is not required for discovery or
prompts. See repositories.
6
Connect your coding agent, optionally
Connect MCP with sign-in, then ask your agent to list your Bench
systems. The checklist confirms a successful authenticated MCP request. An API
key created for an agent does not by itself confirm MCP connection.