> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usebench.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Headless operation reference
> Every Bench API operation, input schema, and required credential.
Use the API for your environment and the current SDK source build for platform clients. Account authority means Bench OAuth or a signed-in session; an API key retains its narrower permissions.
Use these operation IDs with SDK `call`. MCP uses `bench_` names; existing MCP tools retain their own input schemas. Public sign-in operations are handled by the MCP connector, not exposed as tools.
[Download the complete machine-readable catalog](/reference/headless-operations.json). The API serves it at `GET /api/headless/operations`.
| Operation | Method and path | Credential |
| ---------------------------- | --------------------------------------------------------------------- | ---------- |
| `get_setup_status` | `GET /api/getting-started` | account |
| `list_operations` | `GET /api/headless/operations` | public |
| `capabilities` | `GET /api/headless/capabilities` | credential |
| `evaluation_allowance` | `GET /api/evaluation-allowance` | credential |
| `get_processing_notice` | `GET /api/legal/processing-notice` | public |
| `acknowledge_processing` | `POST /api/legal/processing-acknowledgements` | account |
| `get_login_url` | `GET /api/auth/login-url` | public |
| `complete_login` | `POST /api/auth/callback` | public |
| `verify_email` | `POST /api/auth/verify-email` | public |
| `login_with_password` | `POST /api/auth/password` | public |
| `logout` | `POST /api/auth/logout` | public |
| `exchange_oauth_token` | `POST /api/auth/mcp-token` | public |
| `whoami` | `GET /api/auth/me` | credential |
| `complete_onboarding` | `POST /api/auth/onboarding/complete` | account |
| `update_profile` | `PATCH /api/auth/me` | credential |
| `set_model_providers` | `PATCH /api/auth/me/model-providers` | credential |
| `receive_github_callback` | `POST /api/github/headless-callback` | public |
| `connect_github` | `GET /api/github/install-url` | account |
| `complete_github_callback` | `POST /api/github/callback` | account |
| `finish_github_connection` | `POST /api/github/connection` | account |
| `connection_status` | `GET /api/github/status` | credential |
| `activate_installation` | `POST /api/github/installations/{installationID}/activate` | account |
| `list_repos` | `GET /api/github/repos` | credential |
| `list_branches` | `GET /api/github/repos/{owner}/{repo}/branches` | credential |
| `upload_prompts` | `POST /api/prompt-scans` | credential |
| `scan_repo` | `POST /api/repos/{owner}/{repo}/scan` | credential |
| `get_scan` | `GET /api/repos/{owner}/{repo}/scan/latest` | credential |
| `get_repository_refresh` | `GET /api/repos/{owner}/{repo}/scan/refresh` | credential |
| `set_repository_refresh` | `PATCH /api/repos/{owner}/{repo}/scan/refresh` | credential |
| `set_prompt_models` | `POST /api/repos/{owner}/{repo}/scan/call-sites/model` | credential |
| `open_prompt_pr` | `POST /api/repos/{owner}/{repo}/prompt-pr` | credential |
| `generate_business_context` | `POST /api/repos/{owner}/{repo}/business-context` | credential |
| `get_business_context` | `GET /api/repos/{owner}/{repo}/business-context/latest` | credential |
| `generate_eval_benchmark` | `POST /api/repos/{owner}/{repo}/eval-benchmark` | credential |
| `get_eval_benchmark` | `GET /api/repos/{owner}/{repo}/eval-benchmark/latest` | credential |
| `review_eval_benchmark` | `POST /api/repos/{owner}/{repo}/eval-benchmark/review` | credential |
| `run_baseline` | `POST /api/repos/{owner}/{repo}/baseline` | credential |
| `get_baseline` | `GET /api/repos/{owner}/{repo}/baseline/latest` | credential |
| `run_search_optimize` | `POST /api/repos/{owner}/{repo}/search-optimize` | credential |
| `get_search_optimize` | `GET /api/repos/{owner}/{repo}/search-optimize/latest` | credential |
| `get_recommendation` | `GET /api/repos/{owner}/{repo}/recommend/latest` | credential |
| `list_golden_cases` | `GET /api/repos/{owner}/{repo}/golden-cases` | credential |
| `label_golden_case` | `PATCH /api/golden-cases/{id}` | credential |
| `fetch_website_text` | `POST /api/website-text` | credential |
| `get_link_metadata` | `POST /api/link-metadata` | credential |
| `start_evaluation` | `POST /api/evaluation-runs` | credential |
| `start_system_evaluation` | `POST /api/evaluation-runs/system` | credential |
| `list_evaluations` | `GET /api/evaluation-runs` | credential |
| `rerun_evaluation` | `POST /api/evaluation-runs/{id}/rerun` | credential |
| `get_evaluation` | `GET /api/evaluation-runs/{id}` | credential |
| `get_evaluation_artifacts` | `GET /api/evaluation-runs/{id}/artifacts` | credential |
| `get_fix_brief` | `GET /api/evaluation-runs/{id}/fix-brief` | credential |
| `cancel_evaluation` | `POST /api/evaluation-runs/{id}/cancel` | credential |
| `cancel_bench` | `POST /api/evaluation-runs/{id}/cancel-bench` | credential |
| `submit_run_review` | `POST /api/evaluation-runs/{id}/review` | credential |
| `archive_evaluation` | `POST /api/evaluation-runs/{id}/archive` | credential |
| `unarchive_evaluation` | `POST /api/evaluation-runs/{id}/unarchive` | credential |
| `create_system` | `POST /api/ai-systems` | account |
| `list_systems` | `GET /api/ai-systems` | credential |
| `get_system` | `GET /api/ai-systems/{id}` | credential |
| `list_runtime_evaluations` | `GET /api/ai-systems/{id}/runtime-evaluations` | credential |
| `publish_runtime_evaluation` | `POST /api/ai-systems/{id}/runtime-evaluations` | credential |
| `get_runtime_settings` | `GET /api/ai-systems/{id}/runtime-settings` | credential |
| `set_runtime_settings` | `PUT /api/ai-systems/{id}/runtime-settings` | account |
| `publish_runtime_fix` | `POST /api/ai-systems/{id}/runtime-jobs/{jobID}/publish` | account |
| `retry_runtime_job` | `POST /api/ai-systems/{id}/runtime-jobs/{jobID}/retry` | account |
| `preview_dataset` | `POST /api/ai-systems/{id}/datasets/preview` | credential |
| `list_test_library` | `GET /api/ai-systems/{id}/test-library` | credential |
| `import_dataset_cases` | `POST /api/ai-systems/{id}/test-library/import` | credential |
| `upload_dataset` | `POST /api/ai-systems/{id}/datasets` | credential |
| `list_datasets` | `GET /api/ai-systems/{id}/datasets` | credential |
| `get_dataset` | `GET /api/ai-systems/{id}/datasets/{datasetID}` | credential |
| `remap_dataset` | `PUT /api/ai-systems/{id}/datasets/{datasetID}/mapping` | credential |
| `compare_dataset` | `POST /api/ai-systems/{id}/datasets/{datasetID}/compare` | credential |
| `list_dataset_reports` | `GET /api/ai-systems/{id}/datasets/{datasetID}/reports` | credential |
| `delete_dataset` | `DELETE /api/ai-systems/{id}/datasets/{datasetID}` | credential |
| `get_system_context` | `GET /api/ai-systems/{id}/context` | credential |
| `list_context_sources` | `GET /api/ai-systems/{id}/context/sources` | credential |
| `put_context_source` | `PUT /api/ai-systems/{id}/context/sources` | credential |
| `upload_context_document` | `POST /api/ai-systems/{id}/context/upload` | credential |
| `seed_system_context` | `POST /api/ai-systems/{id}/context/seed` | credential |
| `summarize_system_context` | `POST /api/ai-systems/{id}/context/summarize` | credential |
| `delete_context_source` | `DELETE /api/ai-systems/{id}/context/sources/{sourceID}` | credential |
| `list_context_connections` | `GET /api/ai-systems/{id}/context/connections` | credential |
| `create_context_connection` | `POST /api/ai-systems/{id}/context/connections` | credential |
| `sync_context_connection` | `POST /api/ai-systems/{id}/context/connections/{connectionID}/sync` | credential |
| `delete_context_connection` | `DELETE /api/ai-systems/{id}/context/connections/{connectionID}` | credential |
| `update_system` | `PATCH /api/ai-systems/{id}` | credential |
| `resolve_finding` | `PUT /api/ai-systems/{id}/finding-resolution` | credential |
| `delete_system` | `DELETE /api/ai-systems/{id}` | credential |
| `add_system_component` | `POST /api/ai-systems/{id}/components` | credential |
| `remove_system_component` | `DELETE /api/ai-systems/{id}/components/{componentID}` | credential |
| `set_continuous_evaluation` | `PATCH /api/ai-systems/{id}/components/{componentID}/continuous-eval` | account |
| `add_system_connection` | `POST /api/ai-systems/{id}/edges` | credential |
| `remove_system_connection` | `DELETE /api/ai-systems/{id}/edges/{edgeID}` | credential |
| `create_system_run` | `POST /api/ai-systems/{id}/runs` | credential |
| `list_system_runs` | `GET /api/ai-systems/{id}/runs` | credential |
| `get_system_run` | `GET /api/ai-systems/{id}/runs/{runID}` | credential |
| `link_system_run` | `POST /api/ai-systems/{id}/runs/{runID}/components` | credential |
| `complete_system_run` | `POST /api/ai-systems/{id}/runs/{runID}/complete` | credential |
| `get_production_health` | `GET /api/ai-systems/{id}/production-health` | credential |
| `get_production_feedback` | `GET /api/ai-systems/{id}/production-feedback` | credential |
| `set_production_feedback` | `PUT /api/ai-systems/{id}/production-feedback` | account |
| `production_capabilities` | `GET /api/traces/capabilities` | credential |
| `sdk_status` | `GET /api/traces/sdk-status` | credential |
| `get_trace_privacy` | `GET /api/traces/privacy` | account |
| `set_trace_privacy` | `PUT /api/traces/privacy` | account |
| `get_production_check` | `GET /api/traces/checks/{jobID}` | credential |
| `retry_production_check` | `POST /api/traces/checks/{jobID}/retry` | credential |
| `ingest_traces` | `POST /api/traces` | credential |
| `list_traces` | `GET /api/traces` | credential |
| `get_trace` | `GET /api/traces/{id}` | credential |
| `delete_trace` | `DELETE /api/traces/{id}` | credential |
| `link_span` | `POST /api/traces/{id}/spans/{spanID}/link` | credential |
| `unlink_span` | `DELETE /api/traces/{id}/links/{linkID}` | credential |
| `evaluate_span` | `POST /api/traces/{id}/spans/{spanID}/evaluate` | credential |
| `get_workspace` | `GET /api/org` | credential |
| `create_workspace` | `POST /api/org` | credential |
| `rename_workspace` | `PATCH /api/org` | credential |
| `invite_workspace_member` | `POST /api/org/invites` | account |
| `accept_workspace_invite` | `POST /api/org/invites/accept` | account |
| `set_workspace_member_role` | `PATCH /api/org/members/{userID}` | account |
| `remove_workspace_member` | `DELETE /api/org/members/{userID}` | account |
| `create_api_key` | `POST /api/account/api-keys` | account |
| `get_setup_key` | `POST /api/account/api-keys/setup` | account |
| `list_api_keys` | `GET /api/account/api-keys` | account |
| `revoke_api_key` | `DELETE /api/account/api-keys/{id}` | account |
| `list_plans` | `GET /api/billing/plans` | public |
| `get_payment_method` | `GET /api/billing/payment-method` | credential |
| `list_invoices` | `GET /api/billing/invoices` | credential |
| `create_checkout` | `POST /api/billing/checkout-session` | account |
| `change_plan` | `POST /api/billing/change-plan` | account |
| `get_billing_portal` | `GET /api/billing/portal-session` | account |
| `cancel_subscription` | `POST /api/billing/cancel` | account |
| `reactivate_subscription` | `POST /api/billing/reactivate` | account |
## Inputs
Each contract lists path identifiers, query parameters, JSON body or multipart form fields. Required values are validated by the API; inspect the returned errors when a field depends on account state.
Get setup status. Does not start an evaluation. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {
"environment": {
"type": "string"
},
"repo_full_name": {
"type": "string"
}
},
"type": "object"
}
}
```
List operations. Does not start an evaluation.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Read credential authority, plan, model-selection entitlement, repository scope and pricing and upgrade links, and links to the current operation catalog and evaluation allowance. Does not grant permissions. Does not start an evaluation.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Evaluation allowance. Does not start an evaluation.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Get processing notice. Does not start an evaluation.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Record the user's explicit authorization for the current processing notice and selected source. Read get\_processing\_notice and show the notice first; never accept it on the user's behalf without permission. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"accepted": {
"type": "boolean"
},
"source": {
"type": "string"
},
"version": {
"type": "string"
}
},
"type": "object"
}
}
```
Get login url. Does not start an evaluation.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {
"challenge": {
"type": "string"
},
"provider": {
"type": "string"
}
},
"type": "object"
}
}
```
Complete login. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"code": {
"type": "string"
},
"newsletter_consent": {
"type": "boolean"
},
"state": {
"type": "string"
},
"verifier": {
"type": "string"
}
},
"type": "object"
}
}
```
Verify email. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"code": {
"type": "string"
},
"newsletter_consent": {
"type": "boolean"
},
"pending_authentication_token": {
"type": "string"
},
"provider": {
"type": "string"
}
},
"type": "object"
}
}
```
Login with password. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"email": {
"type": "string"
},
"newsletter_consent": {
"type": "boolean"
},
"password": {
"type": "string"
}
},
"type": "object"
}
}
```
Logout. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Exchange oauth token. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"token": {
"type": "string"
}
},
"required": [
"token"
],
"type": "object"
}
}
```
Whoami. Does not start an evaluation.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Record setup completion for the current account. Use onboarding\_generation from whoami; a stale generation is rejected. No browser wizard is required. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"generation": {
"type": "integer"
}
},
"type": "object"
}
}
```
Update profile. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"name": {
"type": "string"
}
},
"type": "object"
}
}
```
Choose allowed model providers for future Bench evaluations. Requires an active Growth or Enterprise plan. Free and Builder use Bench defaults and cannot change this preference.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"providers": {
"items": {
"type": "string"
},
"type": "array"
}
},
"type": "object"
}
}
```
Receive github callback. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"installation_id": {
"type": "integer"
},
"state": {
"type": "string"
}
},
"type": "object"
}
}
```
Start GitHub repository authorization. Set headless=true to return to the coding agent after the user approves access on GitHub. Keep the returned state and poll finish\_github\_connection. GitHub installation approval is required; no Bench onboarding is needed.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {
"headless": {
"type": "string"
}
},
"type": "object"
}
}
```
Complete a manual GitHub callback using the original account-bound state and installation\_id. Prefer connect\_github with headless=true and finish\_github\_connection. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"installation_id": {
"type": "integer"
},
"state": {
"type": "string"
}
},
"type": "object"
}
}
```
Finish the connection started with connect\_github(headless=true). Pass its state. A pending response means wait the returned poll\_interval before calling again. On success, list\_repos and choose a branch. This binds the installation to the signed-in user. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"state": {
"type": "string"
}
},
"type": "object"
}
}
```
Connection status. Does not start an evaluation.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Activate installation. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"path_parameters": {
"installationID": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
List repos. Does not start an evaluation.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {
"all_accounts": {
"type": "string"
}
},
"type": "object"
}
}
```
List branches. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"owner": {
"minLength": 1,
"type": "string"
},
"repo": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Upload prompts. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"form": {
"additionalProperties": false,
"properties": {
"model": {
"type": "string"
},
"pasted_text": {
"type": "string"
}
},
"type": "object"
},
"file_field": "files",
"max_file_bytes": 2097152
}
```
Scan repo. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"owner": {
"minLength": 1,
"type": "string"
},
"repo": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "Repository branch, required for branch-scoped operations.",
"type": "string"
}
},
"type": "object"
}
}
```
Get scan. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"owner": {
"minLength": 1,
"type": "string"
},
"repo": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "Repository branch, required for branch-scoped operations.",
"type": "string"
}
},
"type": "object"
}
}
```
Get repository refresh. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"owner": {
"minLength": 1,
"type": "string"
},
"repo": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "Repository branch, required for branch-scoped operations.",
"type": "string"
}
},
"type": "object"
}
}
```
Set repository refresh. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"owner": {
"minLength": 1,
"type": "string"
},
"repo": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "Repository branch, required for branch-scoped operations.",
"type": "string"
}
},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"enabled": {
"type": "boolean"
}
},
"type": "object"
}
}
```
Record the model used by uploaded prompt call sites. This changes source metadata, not the allowed-provider setting. Requires model-selection entitlement.
```json theme={null}
{
"path_parameters": {
"owner": {
"minLength": 1,
"type": "string"
},
"repo": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "Repository branch, required for branch-scoped operations.",
"type": "string"
}
},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"call_site_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"model": {
"type": "string"
}
},
"type": "object"
}
}
```
Open prompt pr. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"owner": {
"minLength": 1,
"type": "string"
},
"repo": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "Repository branch, required for branch-scoped operations.",
"type": "string"
}
},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"base_branch": {
"type": "string"
},
"body": {
"type": "string"
},
"branch": {
"type": "string"
},
"changes": {
"items": {
"additionalProperties": false,
"properties": {
"content": {
"type": "string"
},
"path": {
"type": "string"
}
},
"type": "object"
},
"type": "array"
},
"title": {
"type": "string"
}
},
"type": "object"
}
}
```
Generate business context. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"owner": {
"minLength": 1,
"type": "string"
},
"repo": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "Repository branch, required for branch-scoped operations.",
"type": "string"
}
},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"business_doc_text": {
"type": "string"
}
},
"type": "object"
}
}
```
Get business context. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"owner": {
"minLength": 1,
"type": "string"
},
"repo": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "Repository branch, required for branch-scoped operations.",
"type": "string"
}
},
"type": "object"
}
}
```
Generate eval benchmark. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"owner": {
"minLength": 1,
"type": "string"
},
"repo": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "Repository branch, required for branch-scoped operations.",
"type": "string"
},
"call_site_id": {
"type": "string"
}
},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"previous_routing_json": {
"type": "string"
},
"reference_today": {
"type": "string"
},
"suite_name": {
"type": "string"
}
},
"type": "object"
}
}
```
Get eval benchmark. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"owner": {
"minLength": 1,
"type": "string"
},
"repo": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "Repository branch, required for branch-scoped operations.",
"type": "string"
},
"call_site_id": {
"type": "string"
}
},
"type": "object"
}
}
```
Review eval benchmark. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"owner": {
"minLength": 1,
"type": "string"
},
"repo": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "Repository branch, required for branch-scoped operations.",
"type": "string"
}
},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"branch": {
"type": "string"
},
"call_site_id": {
"type": "string"
},
"test_cases": {}
},
"type": "object"
}
}
```
Run baseline scoring for an existing reviewed benchmark. This can consume an evaluation credit; confirm spending first. Streamed events preserve progress and terminal errors. A model override requires Growth or Enterprise.
```json theme={null}
{
"path_parameters": {
"owner": {
"minLength": 1,
"type": "string"
},
"repo": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "Repository branch, required for branch-scoped operations.",
"type": "string"
},
"call_site_id": {
"type": "string"
}
},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"max_concurrency": {
"type": "integer"
},
"model": {
"type": "string"
},
"suite_name": {
"type": "string"
}
},
"type": "object"
}
}
```
Get baseline. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"owner": {
"minLength": 1,
"type": "string"
},
"repo": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "Repository branch, required for branch-scoped operations.",
"type": "string"
},
"call_site_id": {
"type": "string"
}
},
"type": "object"
}
}
```
Evaluate prompt improvements after a baseline. Confirm evaluation spending first. Candidate lists are server controlled on all plans; use set\_model\_providers for eligible provider preferences.
```json theme={null}
{
"path_parameters": {
"owner": {
"minLength": 1,
"type": "string"
},
"repo": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "Repository branch, required for branch-scoped operations.",
"type": "string"
},
"call_site_id": {
"type": "string"
}
},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"candidate_models": {
"items": {
"additionalProperties": false,
"properties": {
"model_id": {
"type": "string"
},
"provider": {
"type": "string"
},
"tier": {
"type": "string"
}
},
"type": "object"
},
"type": "array"
},
"max_case_concurrency": {
"type": "integer"
},
"max_combo_concurrency": {
"type": "integer"
},
"max_rounds": {
"type": "integer"
},
"min_quality_score": {
"type": "number"
},
"suite_name": {
"type": "string"
},
"variant_count": {
"type": "integer"
}
},
"type": "object"
}
}
```
Get search optimize. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"owner": {
"minLength": 1,
"type": "string"
},
"repo": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "Repository branch, required for branch-scoped operations.",
"type": "string"
},
"call_site_id": {
"type": "string"
}
},
"type": "object"
}
}
```
Get recommendation. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"owner": {
"minLength": 1,
"type": "string"
},
"repo": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "Repository branch, required for branch-scoped operations.",
"type": "string"
},
"call_site_id": {
"type": "string"
}
},
"type": "object"
}
}
```
List golden cases. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"owner": {
"minLength": 1,
"type": "string"
},
"repo": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "Repository branch, required for branch-scoped operations.",
"type": "string"
},
"call_site_id": {
"type": "string"
}
},
"type": "object"
}
}
```
Label golden case. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"label": {
"type": "string"
},
"note": {
"type": "string"
}
},
"type": "object"
}
}
```
Fetch website text. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"url": {
"type": "string"
}
},
"type": "object"
}
}
```
Get link metadata. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"url": {
"type": "string"
}
},
"type": "object"
}
}
```
Start evaluation. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"ai_system_id": {
"type": "integer"
},
"branch": {
"type": "string"
},
"context_doc": {
"type": "string"
},
"generate_context": {
"type": "boolean"
},
"max_rounds": {
"type": "integer"
},
"min_quality_score": {
"type": "number"
},
"prompts": {
"items": {
"additionalProperties": false,
"properties": {
"call_site_id": {
"type": "string"
},
"workflow_name": {
"type": "string"
}
},
"type": "object"
},
"type": "array"
},
"repo_full_name": {
"type": "string"
}
},
"type": "object"
}
}
```
Start system evaluation. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"ai_system_id": {
"type": "integer"
}
},
"type": "object"
}
}
```
List evaluations. Does not start an evaluation.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {
"include_active": {
"type": "string"
}
},
"type": "object"
}
}
```
Rerun evaluation. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"context_doc": {
"type": "string"
},
"generate_context": {
"type": "boolean"
},
"single_prompt": {
"type": "boolean"
}
},
"type": "object"
}
}
```
Get evaluation. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Get evaluation artifacts. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Get fix brief. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Cancel evaluation. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Cancel bench. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Submit run review. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"test_cases": {}
},
"type": "object"
}
}
```
Archive evaluation. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Unarchive evaluation. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Create an AI system with a name and optional description. Add prompt, tool or harness components after scanning a repository or uploading prompts. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"description": {
"type": "string"
},
"name": {
"type": "string"
}
},
"type": "object"
}
}
```
List systems. Does not start an evaluation.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Get system. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
List runtime evaluations. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"environment": {
"type": "string"
},
"offset": {
"type": "string"
}
},
"type": "object"
}
}
```
Publish an SDK real app test report (maximum 500 KB). Execute the actual application with the SDK evaluateSystem/evaluate\_system API first. Uploaded results remain client-reported evidence, not hosted verification.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": true,
"properties": {},
"type": "object"
}
}
```
Read hosted runtime availability, settings and jobs. For local real app tests, use the SDK even when hosted execution is unavailable. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Configure hosted real app execution for a connected system: repository, branch, enabled, daily\_limit and config. Check get\_runtime\_settings for environment availability. Enabling it can execute application code and consume evaluation allowance; obtain user approval first. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"branch": {
"type": "string"
},
"config": {
"additionalProperties": true,
"properties": {},
"type": "object"
},
"daily_limit": {
"type": "integer"
},
"enabled": {
"type": "boolean"
},
"repository": {
"type": "string"
}
},
"required": [
"repository",
"branch",
"enabled",
"daily_limit",
"config"
],
"type": "object"
}
}
```
Publish runtime fix. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
},
"jobID": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Retry runtime job. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
},
"jobID": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Preview CSV, XLSX, JSON or JSONL and suggested column mapping without storing it. Upload the file bytes, optional sheet, mode and mapping. Inspect mapping\_error before import. No model runs.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"form": {
"additionalProperties": false,
"properties": {
"content_consent": {
"type": "boolean"
},
"mapping": {
"additionalProperties": false,
"properties": {
"actual": {
"type": "string"
},
"expected": {
"type": "string"
},
"history": {
"type": "string"
},
"input": {
"type": "string"
},
"notes": {
"type": "string"
},
"resolved_prompt": {
"type": "string"
},
"template": {
"type": "string"
},
"variables": {
"type": "string"
}
},
"type": "object"
},
"mode": {
"enum": [
"classifier",
"recorded_outputs",
"replay"
],
"type": "string"
},
"name": {
"type": "string"
},
"sheet": {
"type": "string"
}
},
"type": "object"
},
"file_field": "file",
"max_file_bytes": 4194304
}
```
List test library. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"kind": {
"type": "string"
},
"offset": {
"type": "string"
}
},
"type": "object"
}
}
```
Import dataset cases. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"component_id": {
"type": "integer"
},
"dataset_id": {
"type": "string"
},
"dataset_version": {
"type": "integer"
},
"expected_version": {
"type": "integer"
}
},
"type": "object"
}
}
```
Store a dataset up to 4 MB after the user approves content\_consent=true and the column mapping. Preview first. A stored dataset becomes a test suite only when import\_dataset\_cases is called.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"form": {
"additionalProperties": false,
"properties": {
"content_consent": {
"type": "boolean"
},
"mapping": {
"additionalProperties": false,
"properties": {
"actual": {
"type": "string"
},
"expected": {
"type": "string"
},
"history": {
"type": "string"
},
"input": {
"type": "string"
},
"notes": {
"type": "string"
},
"resolved_prompt": {
"type": "string"
},
"template": {
"type": "string"
},
"variables": {
"type": "string"
}
},
"type": "object"
},
"mode": {
"enum": [
"classifier",
"recorded_outputs",
"replay"
],
"type": "string"
},
"name": {
"type": "string"
},
"sheet": {
"type": "string"
}
},
"type": "object"
},
"file_field": "file",
"max_file_bytes": 4194304
}
```
List datasets. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Get dataset. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"datasetID": {
"minLength": 1,
"type": "string"
},
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"version": {
"type": "string"
}
},
"type": "object"
}
}
```
Create a new dataset version with a corrected column mapping. Supply expected\_version from get\_dataset. Old reports retain their original mapping.
```json theme={null}
{
"path_parameters": {
"datasetID": {
"minLength": 1,
"type": "string"
},
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"expected_version": {
"type": "integer"
},
"mapping": {
"additionalProperties": false,
"properties": {
"actual": {
"type": "string"
},
"expected": {
"type": "string"
},
"history": {
"type": "string"
},
"input": {
"type": "string"
},
"notes": {
"type": "string"
},
"resolved_prompt": {
"type": "string"
},
"template": {
"type": "string"
},
"variables": {
"type": "string"
}
},
"type": "object"
},
"mode": {
"type": "string"
}
},
"required": [
"expected_version",
"mapping",
"mode"
],
"type": "object"
}
}
```
Compare recorded outputs with expected values for an explicit dataset version. No model or live app is called. Live replay is not supported by this operation.
```json theme={null}
{
"path_parameters": {
"datasetID": {
"minLength": 1,
"type": "string"
},
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"version": {
"type": "integer"
}
},
"type": "object"
}
}
```
List dataset reports. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"datasetID": {
"minLength": 1,
"type": "string"
},
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Delete dataset. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"datasetID": {
"minLength": 1,
"type": "string"
},
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Get system context. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"version": {
"type": "string"
}
},
"type": "object"
}
}
```
List context sources. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {
"limit": {
"type": "string"
},
"offset": {
"type": "string"
},
"q": {
"type": "string"
}
},
"type": "object"
}
}
```
Create or update versioned system understanding, feedback, a golden case or criterion. Read the current context version first and provide expected\_version. Preserve user-supplied expected values; do not infer business policy from observed output.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"expected_version": {
"type": "integer"
},
"source": {
"additionalProperties": false,
"properties": {
"case_id": {
"type": "string"
},
"category": {
"type": "string"
},
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"scope": {
"type": "string"
},
"text": {
"type": "string"
},
"title": {
"type": "string"
}
},
"required": [
"id",
"kind",
"category",
"scope",
"text"
],
"type": "object"
}
},
"required": [
"expected_version",
"source"
],
"type": "object"
}
}
```
Upload a text, Markdown, JSON or JSONL document up to 60 KB to system understanding. Provide category and expected\_version from get\_system\_context. Export other document formats to text first.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"form": {
"additionalProperties": false,
"properties": {
"category": {
"type": "string"
},
"expected_version": {
"type": "integer"
}
},
"type": "object"
},
"file_field": "file",
"max_file_bytes": 60000
}
```
Seed system context. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Queue a summary of the system's saved understanding. Poll get\_system\_context for completion. Does not start a Bench evaluation.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Delete context source. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
},
"sourceID": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
List context connections. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Connect a supported evidence provider using explicit content consent and selected project configuration. Store provider credentials only in secrets. Synchronization imports redacted evidence; it does not make that evidence business policy.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"config": {
"additionalProperties": false,
"properties": {
"base_url": {
"type": "string"
},
"dataset_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"environment": {
"type": "string"
},
"project": {
"type": "string"
},
"prompt_names": {
"items": {
"type": "string"
},
"type": "array"
},
"provider": {
"enum": [
"langfuse",
"langsmith"
],
"type": "string"
},
"since": {
"type": "string"
}
},
"required": [
"provider",
"base_url",
"project",
"since"
],
"type": "object"
},
"content_consent": {
"type": "boolean"
},
"credentials": {
"additionalProperties": false,
"properties": {
"api_key": {
"type": "string"
},
"public_key": {
"type": "string"
},
"secret_key": {
"type": "string"
}
},
"type": "object"
},
"retention_days": {
"type": "integer"
}
},
"required": [
"config",
"credentials",
"content_consent"
],
"type": "object"
}
}
```
Sync context connection. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"connectionID": {
"minLength": 1,
"type": "string"
},
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Delete context connection. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"connectionID": {
"minLength": 1,
"type": "string"
},
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Update system. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"description": {
"type": "string"
},
"name": {
"type": "string"
}
},
"type": "object"
}
}
```
Resolve finding. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"key": {
"type": "string"
},
"status": {
"type": "string"
}
},
"type": "object"
}
}
```
Delete system. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Attach a scanned prompt, tool, model configuration or harness to a system. Use the exact repo\_full\_name, branch and call\_site\_id returned by a scan. This does not edit source code.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"branch": {
"type": "string"
},
"call_site_id": {
"type": "string"
},
"repo_full_name": {
"type": "string"
},
"role": {
"type": "string"
}
},
"type": "object"
}
}
```
Remove system component. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"componentID": {
"minLength": 1,
"type": "string"
},
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Set continuous evaluation. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"path_parameters": {
"componentID": {
"minLength": 1,
"type": "string"
},
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"enabled": {
"type": "boolean"
}
},
"type": "object"
}
}
```
Add system connection. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"edge_type": {
"type": "string"
},
"from_component_id": {
"type": "integer"
},
"to_component_id": {
"type": "integer"
}
},
"type": "object"
}
}
```
Remove system connection. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"edgeID": {
"minLength": 1,
"type": "string"
},
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Create system run. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
List system runs. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Get system run. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
},
"runID": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Link system run. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
},
"runID": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"ai_system_component_id": {
"type": "integer"
},
"evaluation_run_id": {
"type": "integer"
}
},
"type": "object"
}
}
```
Complete system run. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
},
"runID": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"outcome": {
"type": "string"
},
"outcome_reason": {
"type": "string"
},
"status": {
"type": "string"
}
},
"type": "object"
}
}
```
Get production health. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Get production feedback. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Configure production checks and optional automatic investigations for one environment. Enabling automatic evaluations can spend the account allowance; confirm with the user first. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"auto_bench": {
"type": "boolean"
},
"enabled": {
"type": "boolean"
},
"environment": {
"type": "string"
}
},
"required": [
"enabled",
"environment"
],
"type": "object"
}
}
```
Production capabilities. Does not start an evaluation.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Sdk status. Does not start an evaluation.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"type": "string"
},
"environment": {
"type": "string"
},
"repo_full_name": {
"type": "string"
}
},
"type": "object"
}
}
```
Get trace privacy. Does not start an evaluation. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"type": "string"
},
"repo_full_name": {
"type": "string"
}
},
"type": "object"
}
}
```
Set trace privacy. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"type": "string"
},
"repo_full_name": {
"type": "string"
}
},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"enabled": {
"type": "boolean"
},
"fields": {
"items": {
"type": "string"
},
"type": "array"
}
},
"required": [
"enabled",
"fields"
],
"type": "object"
}
}
```
Get production check. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"jobID": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Retry production check. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"jobID": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Upload explicit SDK trace data. Prefer SDK instrumentation for automatic bounds and redaction. Capture content only when the application owner has opted in.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"branch": {
"type": "string"
},
"capture_content": {
"type": "boolean"
},
"repo_full_name": {
"type": "string"
},
"system_name": {
"type": "string"
},
"traces": {
"items": {
"additionalProperties": false,
"properties": {
"source": {
"type": "string"
},
"spans": {
"items": {},
"type": "array"
},
"trace_id": {
"type": "string"
}
},
"type": "object"
},
"type": "array"
}
},
"type": "object"
}
}
```
List traces. Does not start an evaluation.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {
"branch": {
"type": "string"
},
"environment": {
"type": "string"
},
"limit": {
"type": "string"
},
"repo_full_name": {
"type": "string"
}
},
"type": "object"
}
}
```
Get trace. Does not start an evaluation.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Delete trace. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Link span. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
},
"spanID": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"ai_system_component_id": {
"type": "integer"
},
"ai_system_id": {
"type": "integer"
}
},
"type": "object"
}
}
```
Unlink span. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
},
"linkID": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Evaluate span. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
},
"spanID": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"component_id": {
"type": "integer"
},
"evidence_scope": {
"type": "string"
},
"no": {
"type": "string"
},
"provider": {
"type": "string"
},
"question": {
"type": "string"
},
"share_with_typesafe": {
"type": "boolean"
},
"yes": {
"type": "string"
}
},
"type": "object"
}
}
```
Get workspace. Does not start an evaluation.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Create workspace. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"name": {
"type": "string"
}
},
"type": "object"
}
}
```
Rename workspace. Uses the same validation and plan limits as Bench.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"name": {
"type": "string"
}
},
"type": "object"
}
}
```
Invite workspace member. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"email": {
"type": "string"
},
"role": {
"type": "string"
}
},
"type": "object"
}
}
```
Accept workspace invite. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"token": {
"type": "string"
}
},
"type": "object"
}
}
```
Set workspace member role. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"path_parameters": {
"userID": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"role": {
"type": "string"
}
},
"type": "object"
}
}
```
Remove workspace member. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"path_parameters": {
"userID": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Create a scoped automation credential. Returns plaintext only once. Store it in the application's secret environment, never in source, logs or chat replies. OAuth account authorization is required; API keys cannot mint other keys. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"max_evaluations_per_month": {
"type": "integer"
},
"name": {
"type": "string"
},
"org_read": {
"type": "boolean"
},
"repo_allowlist": {
"items": {
"type": "string"
},
"type": "array"
}
},
"type": "object"
}
}
```
Retrieve the account's reusable SDK setup credential. Never print or commit the returned secret. replace\_revoked=true explicitly replaces a revoked setup key. OAuth account authorization is required. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"replace_revoked": {
"type": "boolean"
}
},
"type": "object"
}
}
```
List api keys. Does not start an evaluation. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Revoke api key. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"path_parameters": {
"id": {
"minLength": 1,
"type": "string"
}
},
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
List plans. Does not start an evaluation.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Get payment method. Does not start an evaluation.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
List invoices. Does not start an evaluation.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Create a Stripe checkout link for a selected plan price. Obtain the user's approval before starting payment. Return the URL for payment and coupon entry; do not handle card details in the agent. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"price_id": {
"type": "string"
},
"return_path": {
"type": "string"
}
},
"type": "object"
}
}
```
Change the subscription to the chosen configured price. This can bill immediately. Obtain explicit approval for the price and interval before calling; use get\_billing\_portal for coupon-aware upgrades. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
},
"body": {
"additionalProperties": false,
"properties": {
"price_id": {
"type": "string"
}
},
"type": "object"
}
}
```
Create a Stripe billing or upgrade link. An optional target price\_id opens immediate upgrade confirmation with promotion-code entry. Return the URL to the user to review payment. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {
"price_id": {
"type": "string"
}
},
"type": "object"
}
}
```
Cancel renewal of the subscription. This changes billing; require explicit user approval. Organization billing requires an administrator. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```
Reactivate subscription renewal. This changes billing; require explicit user approval. Organization billing requires an administrator. Requires Bench OAuth account authorization; a scoped API key is insufficient.
```json theme={null}
{
"query": {
"additionalProperties": false,
"properties": {},
"type": "object"
}
}
```