Skip to main content

Copy this into your coding agent

Paste into Cursor, Codex, Claude Code, or another coding agent in your project.
Beta release. All four languages support tracing, real app tests and scripted simulations.

Install

Package and API reference · Source and examples

Record one interaction

Choose Go on the SDK setup page. Go 1.22 or later is required. This client uses the standard library.
Open the system’s Production tab to confirm receipt.

Forward spans your framework already records

If the application already produces spans (its own tracing hooks or an OpenTelemetry pipeline), forward each finished span with client.RecordExternalSpan, keeping the framework’s trace, span and parent identifiers, name, start and end times and attributes. Bench infers the kind from the gen_ai.* attributes (gen_ai.operation.name first, then gen_ai.agent.name, gen_ai.tool.name, gen_ai.request.model) and draws the agents, model calls, tools and hand-offs from one real request. Failed spans keep error.message and exception.type / exception.message, never the stack trace.

Register the model calls

Traces never carry the editable template. After the first request, register the model calls the system makes from the repository, one entry per call, using the register_calls operation (POST /api/ai-systems/{id}/calls) or the bench_register_calls MCP tool, then put the returned component ids on the matching model-call spans with ComponentID. Your coding agent does this for you when set up from Bench’s setup prompt.

Wrap your application

Replace the callback with your request handler. Pass its ctx into nested bench.Trace calls for model and tool operations, including goroutines. Use Kind: "TOOL" for tools. The original result and error are returned unchanged. For streams, use StartSpan, keep the span open, then call SetOutput or SetError and End when the stream finishes. Explicit wrappers work with Google ADK and custom Go applications. Framework versions may require a newer Go version than the Bench client. Content capture defaults to off. To enable it, set CaptureContent: true and provide SpanInput.Input. Built-in filters run before delivery; add a Redact callback for extra fields. See privacy and redaction. Call Flush(ctx) at request boundaries and Shutdown(ctx) after active requests finish. If the request was canceled, use a fresh bounded context for flushing. Stats() reports dropped events; delivery errors do not replace application errors. Set ComponentID to a real Bench prompt component to link events to its criteria; it is sent as bench.component_id. Production checks work with accepted, linked events. Report bench.cost.usd when your provider gives you the price; otherwise Bench estimates cost from tokens and labels it ~.

Test your application

client.EvaluateSystem(ctx, options) calls your real application with pinned JSON cases and returns a redacted report without uploading it.
Use bench.Expect(value) to assert an output/state, including explicit JSON null with bench.Expect(nil). Omitted pointers mean no assertion. For state checks, set ExpectedState and Observe func(context.Context, string) (any, error). It must read the authoritative test state independently of the final reply. JSON input numbers arrive as json.Number to preserve large identifiers. Pass the provided context into every nested tool/model trace and await all goroutines. client.SimulateSystem(ctx, bench.SimulationOptions{...}) takes the same cases and revisions plus CreateSession func(context.Context, any) (bench.SimulationSession, error). Use bench.SimulationInput{InitialState: ..., Turns: []any{...}} as case input and require ExpectedState. The returned session implements Turn, Observe and Close, each accepting context.Context. A fresh session handles 1 to 20 scripted turns; observed state is snapshotted before Close resets it. bench.EvaluationCaseID(ctx) identifies the current case. Timeout defaults to 30 seconds per case; set Timeout up to five minutes. Cancel the supplied context to stop. Missing assertions/observations, unfinished spans and timeouts are incomplete and fail Passed(). Go cannot forcibly stop an uncooperative goroutine. Use isolated test dependencies and cooperative callbacks. Cleanup gets a fresh five-second context, which it must honor. Real app tests record redacted content locally, even when production tracing is metadata-only. They do not upload reports or spend evaluation credits by implicitly running a judge. Use synthetic test data. Run the complete refund simulation example. It sends two refund requests, checks the number of refunds written, and demonstrates the failing implementation and its fix.

Run in CI

Use Bench with your existing pull-request checks. See Run tests in CI for setup, report handling and the features available in each language.