Skip to main content
Checked September 19, 2026. Start with Mastra, then add the other fixtures for coverage. Connecting GitHub tests static discovery. Production monitoring requires running an instrumented application with synthetic traffic. Neither step proves that Bench can execute an entire agent graph.

​
Choose a small fixture

Pin the upstream commit and record it with your results. Fork into an account where the Bench GitHub App has access. For an SDK monorepo, prepare a sample-only repository with its dependencies and upstream license/attribution. A GitHub blob/ or tree/ URL is not a repository-picker subdirectory setting.

​
1. Verify discovery

  1. Connect the fixture’s repository and branch in staging. Wait for the scan to complete and inspect warnings before accepting its inventory.
  2. Check the source file/line, framework evidence, candidate boundary and the fixture-specific expectations above. A dependency alone must not invent a system. Discovery remains runtime_verified: false.
  3. Add one sentence of business context and one expected behavior. Re-scan the same commit: the system identity and your context should survive without a duplicate system.
  4. Review a discovered prompt’s eligibility separately. An unresolved model, prompt or input is a visible limitation, not a successful full-agent bench.
See framework support for recognition limits. Use small repositories: traversal is limited to 5,000 visited files, 2 MB per file and 20 MB total; tests are excluded by default.

​
2. Verify production monitoring

  1. Confirm the staging API includes the preview SDK/key/trace endpoints. Open SDK, create a repository-scoped setup key with zero evaluation allowance, and keep it server-side. Explicitly set BENCH_API_BASE_URL to the intended staging API, not the SDK’s default production origin.
  2. Follow the SDK quick start. In this development workspace, the preview source is bench-pipeline/.context/implementation/bench-sdk. Build/package that checkout or use the supplied preview artifact. Do not assume npm install @trybench/sdk is available from the public registry.
  3. In a Node.js 20+ server, wrap an existing application call with bench.trace. Set the repository, branch and an explicit systemName; start with captureContent: false. Instrument nested tool/agent calls explicitly if testing parent-child spans. There is no automatic framework patching.
  4. Execute a synthetic successful request and a controlled failing request. Verify that application return values/errors are unchanged. For streaming, keep the wrapper open through completion; returning the initial stream object does not measure the finished interaction. Use a small final-output projection if content capture is later enabled.
  5. Flush before process exit or serverless freeze. Open AI systems → your runtime system → Production. Confirm repository/branch association, success/error status and any explicitly instrumented parent-child spans. Confirm inputs/outputs are absent and no evaluation was charged. Check bench.stats and onError for delivery failures; buffering is not durable.
A provider stub can verify SDK delivery without paid model calls, but record that result as a transport smoke test. A real framework/runtime test must actually execute that framework with synthetic data and approved provider credentials. A declared runtime system can appear before evaluable prompt components are connected.

​
3. Optional, separately approved evaluation

Only after capture works, follow production checks: enable content capture for synthetic inputs, link a real prompt component, generate its rubric and explicitly approve a check’s evaluation usage. A default setup key cannot spend evaluations. Verify pass/fail evidence, then manually save intended behavior into the test library; captured output is not a golden answer. Skip this section when paid calls are not authorized.

​
Record the result

Before sign-off, also verify Test library → Criteria shows the generated rules after a benchmark, and Add criterion persists a new scoped rule. Re-bench and check the included-case manifest: prior selected passing checks must remain passing for a candidate to be recommended. A plan-bounded suite does not replay all history. Follow application-runtime evaluation for the real-entrypoint path. Use a synthetic app whose model answer is correct but whose harness invokes a forbidden tool. Expect a failed check in Runtime, then fix the harness and rerun the unchanged suite. Confirm the copied brief includes expected outcomes and revisions. Test a timeout and unfinished tool: neither should become a complete score. The operator E2B fixture was executed in six fresh sandboxes: baseline reproduced two harness failures, candidate passed incident, regression and holdout cases. This validates fixture execution and cleanup, not live model-provider behavior. Save the staging build, fixture commit, repository/branch, system ID, trace ID and pass/fail notes. Keep secrets and captured content out of tickets. The Mastra, OpenAI and LangGraph counts above were rechecked using file-level declaration probes against pipeline/stage_1_extract/system_discovery.py; Vercel coverage was checked against its source and analyzer_js.py. These are not completed repository/UI or live-runtime tests. SDK behavior was checked against the local preview’s README and source. No sample dependencies were installed, application code executed or paid services invoked for this guide.