Headless platform clients

The published 0.1.0 registry packages do not include platform clients. Install the current SDK source using the commands below. Existing tracing, real app testing and simulation APIs remain compatible. Development and production credentials are separate.

BenchPlatform (Platform in Go) manages Bench accounts and saved resources. It is separate from the existing Bench tracing and application-test client. Each call makes one API request; it does not automatically retry writes, start an evaluation, or execute application code. All 135 API operations are listed in the operation reference.

Install the platform clients

Use the tested main source until a registry release containing platform clients is published. Pin the resolved commit in your project’s lockfile:

TypeScript
Python
Go
Rust

npm install git+https://github.com/trybench/bench-sdk.git#main

python -m pip install 'trybench-sdk @ git+https://github.com/trybench/bench-sdk.git@main#subdirectory=python'

go get github.com/trybench/bench-sdk/go@main

cargo add trybench-sdk --git https://github.com/trybench/bench-sdk --branch main

Use https://api.usebench.ai in production or https://api.staging.usebench.ai for development. The examples below use development to keep test data separate from your production account.

Authenticate

Use a Bench API key for scoped data access. For GitHub setup, API-key management, workspace membership and billing, use a Bench OAuth account credential obtained by the authenticated MCP connector, or a normal authenticated Bench session. API keys cannot mint broader credentials or gain account authority. An AuthKit token is exchanged through the exchange_oauth_token operation and must have the configured MCP resource audience. Do not forward it as a bearer credential for ordinary Bench API operations. TypeScript, Python and Go accept a token supplier for refresh; Rust supports set_credential after refresh. Never put token values in source or logs. Public authentication operations are available without a Bench credential.

Read systems and results

TypeScript
Python
Go
Rust

import { BenchPlatform } from '@benchai/sdk'

const platform = new BenchPlatform({
  credential: process.env.BENCH_API_KEY!,
  endpoint: 'https://api.staging.usebench.ai',
})
const capabilities = await platform.call('capabilities')
const systems = await platform.call('list_systems')
const system = await platform.call('get_system', { path: { id: 123 } })
const history = await platform.call('list_evaluations')

import os
from bench_sdk import BenchPlatform

platform = BenchPlatform(
    os.environ['BENCH_API_KEY'], endpoint='https://api.staging.usebench.ai'
)
capabilities = platform.call('capabilities')
systems = platform.call('list_systems')
system = platform.call('get_system', path={'id': 123})
history = platform.call('list_evaluations')

platform, err := bench.NewPlatform(bench.PlatformOptions{
    Credential: os.Getenv("BENCH_API_KEY"),
    Endpoint: "https://api.staging.usebench.ai",
})
if err != nil { return err }
systems, err := platform.Call(ctx, "list_systems", bench.PlatformRequest{})
system, err := platform.Call(ctx, "get_system", bench.PlatformRequest{
    Path: map[string]string{"id": "123"},
})

use trybench_sdk::{BenchPlatform, PlatformRequest};
use std::collections::HashMap;

let platform = BenchPlatform::new(
    std::env::var("BENCH_API_KEY")?, "https://api.staging.usebench.ai"
)?;
let systems = platform.call("list_systems", PlatformRequest::default()).await?;
let system = platform.call("get_system", PlatformRequest {
    path: HashMap::from([("id".into(), "123".into())]),
    ..Default::default()
}).await?;

Replace sample IDs with values from the list/scan operations. The local operations() / Operations() method returns the pinned catalog without a network call. list_operations retrieves the API’s current catalog.

Request shape

Field	Meaning
`path`	Named resource identifiers such as `id`, `owner`, `repo`, `datasetID`
`query`	Branch, environment, version, paging or other URL query values
`body`	JSON request body with the API’s field names
`form`	Text or structured fields for multipart uploads; objects serialize as JSON
`files`	Explicit file names and content; only for upload operations

MCP tools added with this release use the same path, query, body, form shape. Earlier tools keep their existing input shape, such as system_id or run_id; inspect the MCP tool schema before calling one.

Upload a dataset

Preview before choosing and approving the column mapping:

import { readFile } from 'node:fs/promises'

const preview = await platform.call('preview_dataset', {
  path: { id: 123 },
  form: { mode: 'recorded_outputs' },
  files: [{ name: 'cases.csv', content: await readFile('./cases.csv') }],
})

// After checking the preview and obtaining content-storage consent:
const dataset = await platform.call('upload_dataset', {
  path: { id: 123 },
  form: {
    mode: 'recorded_outputs',
    content_consent: true,
    mapping: { input: 'question', expected: 'expected', actual: 'actual' },
  },
  files: [{ name: 'cases.csv', content: await readFile('./cases.csv') }],
})

Python files use {'name': 'cases.csv', 'content': bytes_or_text}. Go uses bench.PlatformFile{Name: "cases.csv", Content: data}; Rust uses PlatformFile { name: "cases.csv".into(), content: data }. Binary XLSX uploads retain their bytes. Context documents are limited to 60 KB; datasets to 4 MB; prompt uploads to 2 MB in total.

Save understanding and run a bench

const current = await platform.call<{ context: { version: number } }>(
  'get_system_context', { path: { id: 123 } },
)
await platform.call('put_context_source', {
  path: { id: 123 },
  body: {
    expected_version: current.context.version,
    source: {
      id: 'manual-refund-policy', kind: 'user', category: 'rules_constraints',
      scope: 'system', text: 'Refund only within 30 days of purchase.',
    },
  },
})

const allowance = await platform.call('evaluation_allowance')
// After the user approves spending:
const started = await platform.call<{ runs: Array<{ id: number }> }>('start_system_evaluation', {
  body: { ai_system_id: 123 },
})
const runId = started.runs[0]?.id
if (!runId) throw new Error('Bench did not return an evaluation run')
// Poll get_evaluation with this ID; inspect artifacts when it finishes.
const result = await platform.call('get_evaluation_artifacts', { path: { id: runId } })

A context_changed error requires re-reading and reconciling the saved version. Do not blindly retry a write. Use the test-library MCP tools or the documented context-source contract to add golden cases and criteria. Historical reports retain the evidence used at the time of evaluation.

Add test cases and criteria

Save each entry as a versioned context source. Use a component ID returned by get_system, preserve structured expected values, and re-read the context version before each write. For example:

const context = await platform.call<{ context: { version: number } }>(
  'get_system_context', { path: { id: 123 } },
)
await platform.call('put_context_source', {
  path: { id: 123 },
  body: {
    expected_version: context.context.version,
    source: {
      id: 'manual-library-refund', title: 'Refund within 30 days',
      kind: 'user', category: 'examples_feedback', scope: 'case',
      case_id: 'manual-library-refund',
      text: JSON.stringify({
        type: 'case', component_id: 456, enabled: true,
        input: 'Can I refund an order delivered 12 days ago?',
        expected: { decision: 'approve' },
        resolved_prompt: 'Approve refunds within 30 days. Return a JSON decision.',
      }),
    },
  },
})

For a criterion, use another manual-library-… ID, category rules_constraints, scope system, and JSON text with type: 'criterion', component_id, enabled, statement, pass_condition and fail_condition. Reuse the same source ID to edit or pause it. Setting enabled: false pauses an entry; delete_context_source removes it. These edits affect future evaluations, not historical reports.

Publish real app results

Execute your real app with the existing SDK evaluation API, then call publishSystemEvaluation / publish_system_evaluation on the tracing client or publish_runtime_evaluation on the platform client with the complete report as body. See Real app testing. Reading or publishing a report does not execute the application or start a paid Bench evaluation.

Errors, streams and cancellation

Clients preserve API error status, code, message and reference, plus the complete server error in details (Details in Go). Plan-limit errors include pricing_url, upgrade_url, billing_url and payment_confirmation_required. Show the upgrade link to the developer; completing payment remains their choice. API-key cap errors instead include manage_key_url, since a plan upgrade does not change that cap. A plan, credential or repository error needs an account/configuration change. A response larger than 16 MB requires a smaller page. HTTP redirects are not followed with credentials. Endpoints must use HTTPS, except loopback HTTP during local tests. Legacy stage endpoints return NDJSON: the platform client returns the full event array and raises an error for a terminal error event. Prefer background evaluation runs and explicit polling for long work. TypeScript accepts an AbortSignal, Go accepts a context, and dropping the Rust future cancels its request. Python calls are synchronous. The default request timeout is 120 seconds.

import { PlatformError } from '@benchai/sdk'

try {
  await platform.call('set_model_providers', { body: { providers: ['openai'] } })
} catch (error) {
  if (error instanceof PlatformError && error.details.upgrade_url) {
    console.log(`Choose a plan: ${error.details.upgrade_url}`)
  } else {
    throw error
  }
}

Start here

Evaluate and improve

MCP

Bench SDK

Data controls

Reference

Headless platform clients

Install the platform clients

Authenticate

Read systems and results

Request shape

Upload a dataset

Save understanding and run a bench

Add test cases and criteria

Publish real app results

Errors, streams and cancellation

​Install the platform clients

​Authenticate

​Read systems and results

​Request shape

​Upload a dataset

​Save understanding and run a bench

​Add test cases and criteria

​Publish real app results

​Errors, streams and cancellation

Install the platform clients

Authenticate

Read systems and results

Request shape

Upload a dataset

Save understanding and run a bench

Add test cases and criteria

Publish real app results

Errors, streams and cancellation