Headless operation reference

Use the API for your environment and the current SDK source build for platform clients. Account authority means Bench OAuth or a signed-in session; an API key retains its narrower permissions.

Use these operation IDs with SDK call. MCP uses bench_ names; existing MCP tools retain their own input schemas. Public sign-in operations are handled by the MCP connector, not exposed as tools. Download the complete machine-readable catalog. The API serves it at GET /api/headless/operations.

Operation	Method and path	Credential
`get_setup_status`	`GET /api/getting-started`	account
`list_operations`	`GET /api/headless/operations`	public
`capabilities`	`GET /api/headless/capabilities`	credential
`evaluation_allowance`	`GET /api/evaluation-allowance`	credential
`get_processing_notice`	`GET /api/legal/processing-notice`	public
`acknowledge_processing`	`POST /api/legal/processing-acknowledgements`	account
`get_login_url`	`GET /api/auth/login-url`	public
`complete_login`	`POST /api/auth/callback`	public
`verify_email`	`POST /api/auth/verify-email`	public
`login_with_password`	`POST /api/auth/password`	public
`logout`	`POST /api/auth/logout`	public
`exchange_oauth_token`	`POST /api/auth/mcp-token`	public
`whoami`	`GET /api/auth/me`	credential
`complete_onboarding`	`POST /api/auth/onboarding/complete`	account
`update_profile`	`PATCH /api/auth/me`	credential
`set_model_providers`	`PATCH /api/auth/me/model-providers`	credential
`receive_github_callback`	`POST /api/github/headless-callback`	public
`connect_github`	`GET /api/github/install-url`	account
`complete_github_callback`	`POST /api/github/callback`	account
`finish_github_connection`	`POST /api/github/connection`	account
`connection_status`	`GET /api/github/status`	credential
`activate_installation`	`POST /api/github/installations/{installationID}/activate`	account
`list_repos`	`GET /api/github/repos`	credential
`list_branches`	`GET /api/github/repos/{owner}/{repo}/branches`	credential
`upload_prompts`	`POST /api/prompt-scans`	credential
`scan_repo`	`POST /api/repos/{owner}/{repo}/scan`	credential
`get_scan`	`GET /api/repos/{owner}/{repo}/scan/latest`	credential
`get_repository_refresh`	`GET /api/repos/{owner}/{repo}/scan/refresh`	credential
`set_repository_refresh`	`PATCH /api/repos/{owner}/{repo}/scan/refresh`	credential
`set_prompt_models`	`POST /api/repos/{owner}/{repo}/scan/call-sites/model`	credential
`open_prompt_pr`	`POST /api/repos/{owner}/{repo}/prompt-pr`	credential
`generate_business_context`	`POST /api/repos/{owner}/{repo}/business-context`	credential
`get_business_context`	`GET /api/repos/{owner}/{repo}/business-context/latest`	credential
`generate_eval_benchmark`	`POST /api/repos/{owner}/{repo}/eval-benchmark`	credential
`get_eval_benchmark`	`GET /api/repos/{owner}/{repo}/eval-benchmark/latest`	credential
`review_eval_benchmark`	`POST /api/repos/{owner}/{repo}/eval-benchmark/review`	credential
`run_baseline`	`POST /api/repos/{owner}/{repo}/baseline`	credential
`get_baseline`	`GET /api/repos/{owner}/{repo}/baseline/latest`	credential
`run_search_optimize`	`POST /api/repos/{owner}/{repo}/search-optimize`	credential
`get_search_optimize`	`GET /api/repos/{owner}/{repo}/search-optimize/latest`	credential
`get_recommendation`	`GET /api/repos/{owner}/{repo}/recommend/latest`	credential
`list_golden_cases`	`GET /api/repos/{owner}/{repo}/golden-cases`	credential
`label_golden_case`	`PATCH /api/golden-cases/{id}`	credential
`fetch_website_text`	`POST /api/website-text`	credential
`get_link_metadata`	`POST /api/link-metadata`	credential
`start_evaluation`	`POST /api/evaluation-runs`	credential
`start_system_evaluation`	`POST /api/evaluation-runs/system`	credential
`list_evaluations`	`GET /api/evaluation-runs`	credential
`rerun_evaluation`	`POST /api/evaluation-runs/{id}/rerun`	credential
`get_evaluation`	`GET /api/evaluation-runs/{id}`	credential
`get_evaluation_artifacts`	`GET /api/evaluation-runs/{id}/artifacts`	credential
`get_fix_brief`	`GET /api/evaluation-runs/{id}/fix-brief`	credential
`cancel_evaluation`	`POST /api/evaluation-runs/{id}/cancel`	credential
`cancel_bench`	`POST /api/evaluation-runs/{id}/cancel-bench`	credential
`submit_run_review`	`POST /api/evaluation-runs/{id}/review`	credential
`archive_evaluation`	`POST /api/evaluation-runs/{id}/archive`	credential
`unarchive_evaluation`	`POST /api/evaluation-runs/{id}/unarchive`	credential
`create_system`	`POST /api/ai-systems`	account
`list_systems`	`GET /api/ai-systems`	credential
`get_system`	`GET /api/ai-systems/{id}`	credential
`list_runtime_evaluations`	`GET /api/ai-systems/{id}/runtime-evaluations`	credential
`publish_runtime_evaluation`	`POST /api/ai-systems/{id}/runtime-evaluations`	credential
`get_runtime_settings`	`GET /api/ai-systems/{id}/runtime-settings`	credential
`set_runtime_settings`	`PUT /api/ai-systems/{id}/runtime-settings`	account
`publish_runtime_fix`	`POST /api/ai-systems/{id}/runtime-jobs/{jobID}/publish`	account
`retry_runtime_job`	`POST /api/ai-systems/{id}/runtime-jobs/{jobID}/retry`	account
`preview_dataset`	`POST /api/ai-systems/{id}/datasets/preview`	credential
`list_test_library`	`GET /api/ai-systems/{id}/test-library`	credential
`import_dataset_cases`	`POST /api/ai-systems/{id}/test-library/import`	credential
`upload_dataset`	`POST /api/ai-systems/{id}/datasets`	credential
`list_datasets`	`GET /api/ai-systems/{id}/datasets`	credential
`get_dataset`	`GET /api/ai-systems/{id}/datasets/{datasetID}`	credential
`remap_dataset`	`PUT /api/ai-systems/{id}/datasets/{datasetID}/mapping`	credential
`compare_dataset`	`POST /api/ai-systems/{id}/datasets/{datasetID}/compare`	credential
`list_dataset_reports`	`GET /api/ai-systems/{id}/datasets/{datasetID}/reports`	credential
`delete_dataset`	`DELETE /api/ai-systems/{id}/datasets/{datasetID}`	credential
`get_system_context`	`GET /api/ai-systems/{id}/context`	credential
`list_context_sources`	`GET /api/ai-systems/{id}/context/sources`	credential
`put_context_source`	`PUT /api/ai-systems/{id}/context/sources`	credential
`upload_context_document`	`POST /api/ai-systems/{id}/context/upload`	credential
`seed_system_context`	`POST /api/ai-systems/{id}/context/seed`	credential
`summarize_system_context`	`POST /api/ai-systems/{id}/context/summarize`	credential
`delete_context_source`	`DELETE /api/ai-systems/{id}/context/sources/{sourceID}`	credential
`list_context_connections`	`GET /api/ai-systems/{id}/context/connections`	credential
`create_context_connection`	`POST /api/ai-systems/{id}/context/connections`	credential
`sync_context_connection`	`POST /api/ai-systems/{id}/context/connections/{connectionID}/sync`	credential
`delete_context_connection`	`DELETE /api/ai-systems/{id}/context/connections/{connectionID}`	credential
`update_system`	`PATCH /api/ai-systems/{id}`	credential
`resolve_finding`	`PUT /api/ai-systems/{id}/finding-resolution`	credential
`delete_system`	`DELETE /api/ai-systems/{id}`	credential
`add_system_component`	`POST /api/ai-systems/{id}/components`	credential
`remove_system_component`	`DELETE /api/ai-systems/{id}/components/{componentID}`	credential
`set_continuous_evaluation`	`PATCH /api/ai-systems/{id}/components/{componentID}/continuous-eval`	account
`add_system_connection`	`POST /api/ai-systems/{id}/edges`	credential
`remove_system_connection`	`DELETE /api/ai-systems/{id}/edges/{edgeID}`	credential
`create_system_run`	`POST /api/ai-systems/{id}/runs`	credential
`list_system_runs`	`GET /api/ai-systems/{id}/runs`	credential
`get_system_run`	`GET /api/ai-systems/{id}/runs/{runID}`	credential
`link_system_run`	`POST /api/ai-systems/{id}/runs/{runID}/components`	credential
`complete_system_run`	`POST /api/ai-systems/{id}/runs/{runID}/complete`	credential
`get_production_health`	`GET /api/ai-systems/{id}/production-health`	credential
`get_production_feedback`	`GET /api/ai-systems/{id}/production-feedback`	credential
`set_production_feedback`	`PUT /api/ai-systems/{id}/production-feedback`	account
`production_capabilities`	`GET /api/traces/capabilities`	credential
`sdk_status`	`GET /api/traces/sdk-status`	credential
`get_trace_privacy`	`GET /api/traces/privacy`	account
`set_trace_privacy`	`PUT /api/traces/privacy`	account
`get_production_check`	`GET /api/traces/checks/{jobID}`	credential
`retry_production_check`	`POST /api/traces/checks/{jobID}/retry`	credential
`ingest_traces`	`POST /api/traces`	credential
`list_traces`	`GET /api/traces`	credential
`get_trace`	`GET /api/traces/{id}`	credential
`delete_trace`	`DELETE /api/traces/{id}`	credential
`link_span`	`POST /api/traces/{id}/spans/{spanID}/link`	credential
`unlink_span`	`DELETE /api/traces/{id}/links/{linkID}`	credential
`evaluate_span`	`POST /api/traces/{id}/spans/{spanID}/evaluate`	credential
`get_workspace`	`GET /api/org`	credential
`create_workspace`	`POST /api/org`	credential
`rename_workspace`	`PATCH /api/org`	credential
`invite_workspace_member`	`POST /api/org/invites`	account
`accept_workspace_invite`	`POST /api/org/invites/accept`	account
`set_workspace_member_role`	`PATCH /api/org/members/{userID}`	account
`remove_workspace_member`	`DELETE /api/org/members/{userID}`	account
`create_api_key`	`POST /api/account/api-keys`	account
`get_setup_key`	`POST /api/account/api-keys/setup`	account
`list_api_keys`	`GET /api/account/api-keys`	account
`revoke_api_key`	`DELETE /api/account/api-keys/{id}`	account
`list_plans`	`GET /api/billing/plans`	public
`get_payment_method`	`GET /api/billing/payment-method`	credential
`list_invoices`	`GET /api/billing/invoices`	credential
`create_checkout`	`POST /api/billing/checkout-session`	account
`change_plan`	`POST /api/billing/change-plan`	account
`get_billing_portal`	`GET /api/billing/portal-session`	account
`cancel_subscription`	`POST /api/billing/cancel`	account
`reactivate_subscription`	`POST /api/billing/reactivate`	account

Inputs

Each contract lists path identifiers, query parameters, JSON body or multipart form fields. Required values are validated by the API; inspect the returned errors when a field depends on account state.

get_setup_status

Get setup status. Does not start an evaluation. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "query": {
    "additionalProperties": false,
    "properties": {
      "environment": {
        "type": "string"
      },
      "repo_full_name": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

list_operations

List operations. Does not start an evaluation.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

capabilities

Read credential authority, plan, model-selection entitlement, repository scope and pricing and upgrade links, and links to the current operation catalog and evaluation allowance. Does not grant permissions. Does not start an evaluation.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

evaluation_allowance

Evaluation allowance. Does not start an evaluation.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

get_processing_notice

Get processing notice. Does not start an evaluation.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

acknowledge_processing

Record the user’s explicit authorization for the current processing notice and selected source. Read get_processing_notice and show the notice first; never accept it on the user’s behalf without permission. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "accepted": {
        "type": "boolean"
      },
      "source": {
        "type": "string"
      },
      "version": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

Get login url. Does not start an evaluation.

{
  "query": {
    "additionalProperties": false,
    "properties": {
      "challenge": {
        "type": "string"
      },
      "provider": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

Complete login. Uses the same validation and plan limits as Bench.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "code": {
        "type": "string"
      },
      "newsletter_consent": {
        "type": "boolean"
      },
      "state": {
        "type": "string"
      },
      "verifier": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

verify_email

Verify email. Uses the same validation and plan limits as Bench.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "code": {
        "type": "string"
      },
      "newsletter_consent": {
        "type": "boolean"
      },
      "pending_authentication_token": {
        "type": "string"
      },
      "provider": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "email": {
        "type": "string"
      },
      "newsletter_consent": {
        "type": "boolean"
      },
      "password": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

logout

Logout. Uses the same validation and plan limits as Bench.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

exchange_oauth_token

Exchange oauth token. Uses the same validation and plan limits as Bench.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "token": {
        "type": "string"
      }
    },
    "required": [
      "token"
    ],
    "type": "object"
  }
}

whoami

Whoami. Does not start an evaluation.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

complete_onboarding

Record setup completion for the current account. Use onboarding_generation from whoami; a stale generation is rejected. No browser wizard is required. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "generation": {
        "type": "integer"
      }
    },
    "type": "object"
  }
}

update_profile

Update profile. Uses the same validation and plan limits as Bench.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "name": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

set_model_providers

Choose allowed model providers for future Bench evaluations. Requires an active Growth or Enterprise plan. Free and Builder use Bench defaults and cannot change this preference.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "providers": {
        "items": {
          "type": "string"
        },
        "type": "array"
      }
    },
    "type": "object"
  }
}

receive_github_callback

Receive github callback. Uses the same validation and plan limits as Bench.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "installation_id": {
        "type": "integer"
      },
      "state": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

connect_github

Start GitHub repository authorization. Set headless=true to return to the coding agent after the user approves access on GitHub. Keep the returned state and poll finish_github_connection. GitHub installation approval is required; no Bench onboarding is needed.

{
  "query": {
    "additionalProperties": false,
    "properties": {
      "headless": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

complete_github_callback

Complete a manual GitHub callback using the original account-bound state and installation_id. Prefer connect_github with headless=true and finish_github_connection. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "installation_id": {
        "type": "integer"
      },
      "state": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

finish_github_connection

Finish the connection started with connect_github(headless=true). Pass its state. A pending response means wait the returned poll_interval before calling again. On success, list_repos and choose a branch. This binds the installation to the signed-in user. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "state": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

connection_status

Connection status. Does not start an evaluation.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

activate_installation

Activate installation. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "path_parameters": {
    "installationID": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

list_repos

List repos. Does not start an evaluation.

{
  "query": {
    "additionalProperties": false,
    "properties": {
      "all_accounts": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

list_branches

List branches. Does not start an evaluation.

{
  "path_parameters": {
    "owner": {
      "minLength": 1,
      "type": "string"
    },
    "repo": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

upload_prompts

Upload prompts. Uses the same validation and plan limits as Bench.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "form": {
    "additionalProperties": false,
    "properties": {
      "model": {
        "type": "string"
      },
      "pasted_text": {
        "type": "string"
      }
    },
    "type": "object"
  },
  "file_field": "files",
  "max_file_bytes": 2097152
}

scan_repo

Scan repo. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "owner": {
      "minLength": 1,
      "type": "string"
    },
    "repo": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "description": "Repository branch, required for branch-scoped operations.",
        "type": "string"
      }
    },
    "type": "object"
  }
}

get_scan

Get scan. Does not start an evaluation.

{
  "path_parameters": {
    "owner": {
      "minLength": 1,
      "type": "string"
    },
    "repo": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "description": "Repository branch, required for branch-scoped operations.",
        "type": "string"
      }
    },
    "type": "object"
  }
}

get_repository_refresh

Get repository refresh. Does not start an evaluation.

{
  "path_parameters": {
    "owner": {
      "minLength": 1,
      "type": "string"
    },
    "repo": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "description": "Repository branch, required for branch-scoped operations.",
        "type": "string"
      }
    },
    "type": "object"
  }
}

set_repository_refresh

Set repository refresh. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "owner": {
      "minLength": 1,
      "type": "string"
    },
    "repo": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "description": "Repository branch, required for branch-scoped operations.",
        "type": "string"
      }
    },
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "enabled": {
        "type": "boolean"
      }
    },
    "type": "object"
  }
}

set_prompt_models

Record the model used by uploaded prompt call sites. This changes source metadata, not the allowed-provider setting. Requires model-selection entitlement.

{
  "path_parameters": {
    "owner": {
      "minLength": 1,
      "type": "string"
    },
    "repo": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "description": "Repository branch, required for branch-scoped operations.",
        "type": "string"
      }
    },
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "call_site_ids": {
        "items": {
          "type": "string"
        },
        "type": "array"
      },
      "model": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

open_prompt_pr

Open prompt pr. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "owner": {
      "minLength": 1,
      "type": "string"
    },
    "repo": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "description": "Repository branch, required for branch-scoped operations.",
        "type": "string"
      }
    },
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "base_branch": {
        "type": "string"
      },
      "body": {
        "type": "string"
      },
      "branch": {
        "type": "string"
      },
      "changes": {
        "items": {
          "additionalProperties": false,
          "properties": {
            "content": {
              "type": "string"
            },
            "path": {
              "type": "string"
            }
          },
          "type": "object"
        },
        "type": "array"
      },
      "title": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

generate_business_context

Generate business context. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "owner": {
      "minLength": 1,
      "type": "string"
    },
    "repo": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "description": "Repository branch, required for branch-scoped operations.",
        "type": "string"
      }
    },
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "business_doc_text": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

get_business_context

Get business context. Does not start an evaluation.

{
  "path_parameters": {
    "owner": {
      "minLength": 1,
      "type": "string"
    },
    "repo": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "description": "Repository branch, required for branch-scoped operations.",
        "type": "string"
      }
    },
    "type": "object"
  }
}

generate_eval_benchmark

Generate eval benchmark. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "owner": {
      "minLength": 1,
      "type": "string"
    },
    "repo": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "description": "Repository branch, required for branch-scoped operations.",
        "type": "string"
      },
      "call_site_id": {
        "type": "string"
      }
    },
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "previous_routing_json": {
        "type": "string"
      },
      "reference_today": {
        "type": "string"
      },
      "suite_name": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

get_eval_benchmark

Get eval benchmark. Does not start an evaluation.

{
  "path_parameters": {
    "owner": {
      "minLength": 1,
      "type": "string"
    },
    "repo": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "description": "Repository branch, required for branch-scoped operations.",
        "type": "string"
      },
      "call_site_id": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

review_eval_benchmark

Review eval benchmark. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "owner": {
      "minLength": 1,
      "type": "string"
    },
    "repo": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "description": "Repository branch, required for branch-scoped operations.",
        "type": "string"
      }
    },
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "type": "string"
      },
      "call_site_id": {
        "type": "string"
      },
      "test_cases": {}
    },
    "type": "object"
  }
}

run_baseline

Run baseline scoring for an existing reviewed benchmark. This can consume an evaluation credit; confirm spending first. Streamed events preserve progress and terminal errors. A model override requires Growth or Enterprise.

{
  "path_parameters": {
    "owner": {
      "minLength": 1,
      "type": "string"
    },
    "repo": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "description": "Repository branch, required for branch-scoped operations.",
        "type": "string"
      },
      "call_site_id": {
        "type": "string"
      }
    },
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "max_concurrency": {
        "type": "integer"
      },
      "model": {
        "type": "string"
      },
      "suite_name": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

get_baseline

Get baseline. Does not start an evaluation.

{
  "path_parameters": {
    "owner": {
      "minLength": 1,
      "type": "string"
    },
    "repo": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "description": "Repository branch, required for branch-scoped operations.",
        "type": "string"
      },
      "call_site_id": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

run_search_optimize

Evaluate prompt improvements after a baseline. Confirm evaluation spending first. Candidate lists are server controlled on all plans; use set_model_providers for eligible provider preferences.

{
  "path_parameters": {
    "owner": {
      "minLength": 1,
      "type": "string"
    },
    "repo": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "description": "Repository branch, required for branch-scoped operations.",
        "type": "string"
      },
      "call_site_id": {
        "type": "string"
      }
    },
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "candidate_models": {
        "items": {
          "additionalProperties": false,
          "properties": {
            "model_id": {
              "type": "string"
            },
            "provider": {
              "type": "string"
            },
            "tier": {
              "type": "string"
            }
          },
          "type": "object"
        },
        "type": "array"
      },
      "max_case_concurrency": {
        "type": "integer"
      },
      "max_combo_concurrency": {
        "type": "integer"
      },
      "max_rounds": {
        "type": "integer"
      },
      "min_quality_score": {
        "type": "number"
      },
      "suite_name": {
        "type": "string"
      },
      "variant_count": {
        "type": "integer"
      }
    },
    "type": "object"
  }
}

get_search_optimize

Get search optimize. Does not start an evaluation.

{
  "path_parameters": {
    "owner": {
      "minLength": 1,
      "type": "string"
    },
    "repo": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "description": "Repository branch, required for branch-scoped operations.",
        "type": "string"
      },
      "call_site_id": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

get_recommendation

Get recommendation. Does not start an evaluation.

{
  "path_parameters": {
    "owner": {
      "minLength": 1,
      "type": "string"
    },
    "repo": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "description": "Repository branch, required for branch-scoped operations.",
        "type": "string"
      },
      "call_site_id": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

list_golden_cases

List golden cases. Does not start an evaluation.

{
  "path_parameters": {
    "owner": {
      "minLength": 1,
      "type": "string"
    },
    "repo": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "description": "Repository branch, required for branch-scoped operations.",
        "type": "string"
      },
      "call_site_id": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

label_golden_case

Label golden case. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "label": {
        "type": "string"
      },
      "note": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

fetch_website_text

Fetch website text. Uses the same validation and plan limits as Bench.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "url": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

get_link_metadata

Get link metadata. Uses the same validation and plan limits as Bench.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "url": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

start_evaluation

Start evaluation. Uses the same validation and plan limits as Bench.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "ai_system_id": {
        "type": "integer"
      },
      "branch": {
        "type": "string"
      },
      "context_doc": {
        "type": "string"
      },
      "generate_context": {
        "type": "boolean"
      },
      "max_rounds": {
        "type": "integer"
      },
      "min_quality_score": {
        "type": "number"
      },
      "prompts": {
        "items": {
          "additionalProperties": false,
          "properties": {
            "call_site_id": {
              "type": "string"
            },
            "workflow_name": {
              "type": "string"
            }
          },
          "type": "object"
        },
        "type": "array"
      },
      "repo_full_name": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

start_system_evaluation

Start system evaluation. Uses the same validation and plan limits as Bench.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "ai_system_id": {
        "type": "integer"
      }
    },
    "type": "object"
  }
}

list_evaluations

List evaluations. Does not start an evaluation.

{
  "query": {
    "additionalProperties": false,
    "properties": {
      "include_active": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

rerun_evaluation

Rerun evaluation. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "context_doc": {
        "type": "string"
      },
      "generate_context": {
        "type": "boolean"
      },
      "single_prompt": {
        "type": "boolean"
      }
    },
    "type": "object"
  }
}

get_evaluation

Get evaluation. Does not start an evaluation.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

get_evaluation_artifacts

Get evaluation artifacts. Does not start an evaluation.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

get_fix_brief

Get fix brief. Does not start an evaluation.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

cancel_evaluation

Cancel evaluation. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

cancel_bench

Cancel bench. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

submit_run_review

Submit run review. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "test_cases": {}
    },
    "type": "object"
  }
}

archive_evaluation

Archive evaluation. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

unarchive_evaluation

Unarchive evaluation. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

create_system

Create an AI system with a name and optional description. Add prompt, tool or harness components after scanning a repository or uploading prompts. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "description": {
        "type": "string"
      },
      "name": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

list_systems

List systems. Does not start an evaluation.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

get_system

Get system. Does not start an evaluation.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

list_runtime_evaluations

List runtime evaluations. Does not start an evaluation.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "environment": {
        "type": "string"
      },
      "offset": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

publish_runtime_evaluation

Publish an SDK real app test report (maximum 500 KB). Execute the actual application with the SDK evaluateSystem/evaluate_system API first. Uploaded results remain client-reported evidence, not hosted verification.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": true,
    "properties": {},
    "type": "object"
  }
}

get_runtime_settings

Read hosted runtime availability, settings and jobs. For local real app tests, use the SDK even when hosted execution is unavailable. Does not start an evaluation.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

set_runtime_settings

Configure hosted real app execution for a connected system: repository, branch, enabled, daily_limit and config. Check get_runtime_settings for environment availability. Enabling it can execute application code and consume evaluation allowance; obtain user approval first. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "type": "string"
      },
      "config": {
        "additionalProperties": true,
        "properties": {},
        "type": "object"
      },
      "daily_limit": {
        "type": "integer"
      },
      "enabled": {
        "type": "boolean"
      },
      "repository": {
        "type": "string"
      }
    },
    "required": [
      "repository",
      "branch",
      "enabled",
      "daily_limit",
      "config"
    ],
    "type": "object"
  }
}

publish_runtime_fix

Publish runtime fix. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    },
    "jobID": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

retry_runtime_job

Retry runtime job. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    },
    "jobID": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

preview_dataset

Preview CSV, XLSX, JSON or JSONL and suggested column mapping without storing it. Upload the file bytes, optional sheet, mode and mapping. Inspect mapping_error before import. No model runs.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "form": {
    "additionalProperties": false,
    "properties": {
      "content_consent": {
        "type": "boolean"
      },
      "mapping": {
        "additionalProperties": false,
        "properties": {
          "actual": {
            "type": "string"
          },
          "expected": {
            "type": "string"
          },
          "history": {
            "type": "string"
          },
          "input": {
            "type": "string"
          },
          "notes": {
            "type": "string"
          },
          "resolved_prompt": {
            "type": "string"
          },
          "template": {
            "type": "string"
          },
          "variables": {
            "type": "string"
          }
        },
        "type": "object"
      },
      "mode": {
        "enum": [
          "classifier",
          "recorded_outputs",
          "replay"
        ],
        "type": "string"
      },
      "name": {
        "type": "string"
      },
      "sheet": {
        "type": "string"
      }
    },
    "type": "object"
  },
  "file_field": "file",
  "max_file_bytes": 4194304
}

list_test_library

List test library. Does not start an evaluation.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "kind": {
        "type": "string"
      },
      "offset": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

import_dataset_cases

Import dataset cases. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "component_id": {
        "type": "integer"
      },
      "dataset_id": {
        "type": "string"
      },
      "dataset_version": {
        "type": "integer"
      },
      "expected_version": {
        "type": "integer"
      }
    },
    "type": "object"
  }
}

upload_dataset

Store a dataset up to 4 MB after the user approves content_consent=true and the column mapping. Preview first. A stored dataset becomes a test suite only when import_dataset_cases is called.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "form": {
    "additionalProperties": false,
    "properties": {
      "content_consent": {
        "type": "boolean"
      },
      "mapping": {
        "additionalProperties": false,
        "properties": {
          "actual": {
            "type": "string"
          },
          "expected": {
            "type": "string"
          },
          "history": {
            "type": "string"
          },
          "input": {
            "type": "string"
          },
          "notes": {
            "type": "string"
          },
          "resolved_prompt": {
            "type": "string"
          },
          "template": {
            "type": "string"
          },
          "variables": {
            "type": "string"
          }
        },
        "type": "object"
      },
      "mode": {
        "enum": [
          "classifier",
          "recorded_outputs",
          "replay"
        ],
        "type": "string"
      },
      "name": {
        "type": "string"
      },
      "sheet": {
        "type": "string"
      }
    },
    "type": "object"
  },
  "file_field": "file",
  "max_file_bytes": 4194304
}

list_datasets

List datasets. Does not start an evaluation.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

get_dataset

Get dataset. Does not start an evaluation.

{
  "path_parameters": {
    "datasetID": {
      "minLength": 1,
      "type": "string"
    },
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "version": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

remap_dataset

Create a new dataset version with a corrected column mapping. Supply expected_version from get_dataset. Old reports retain their original mapping.

{
  "path_parameters": {
    "datasetID": {
      "minLength": 1,
      "type": "string"
    },
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "expected_version": {
        "type": "integer"
      },
      "mapping": {
        "additionalProperties": false,
        "properties": {
          "actual": {
            "type": "string"
          },
          "expected": {
            "type": "string"
          },
          "history": {
            "type": "string"
          },
          "input": {
            "type": "string"
          },
          "notes": {
            "type": "string"
          },
          "resolved_prompt": {
            "type": "string"
          },
          "template": {
            "type": "string"
          },
          "variables": {
            "type": "string"
          }
        },
        "type": "object"
      },
      "mode": {
        "type": "string"
      }
    },
    "required": [
      "expected_version",
      "mapping",
      "mode"
    ],
    "type": "object"
  }
}

compare_dataset

Compare recorded outputs with expected values for an explicit dataset version. No model or live app is called. Live replay is not supported by this operation.

{
  "path_parameters": {
    "datasetID": {
      "minLength": 1,
      "type": "string"
    },
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "version": {
        "type": "integer"
      }
    },
    "type": "object"
  }
}

list_dataset_reports

List dataset reports. Does not start an evaluation.

{
  "path_parameters": {
    "datasetID": {
      "minLength": 1,
      "type": "string"
    },
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

delete_dataset

Delete dataset. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "datasetID": {
      "minLength": 1,
      "type": "string"
    },
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

get_system_context

Get system context. Does not start an evaluation.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "version": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

list_context_sources

List context sources. Does not start an evaluation.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {
      "limit": {
        "type": "string"
      },
      "offset": {
        "type": "string"
      },
      "q": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

put_context_source

Create or update versioned system understanding, feedback, a golden case or criterion. Read the current context version first and provide expected_version. Preserve user-supplied expected values; do not infer business policy from observed output.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "expected_version": {
        "type": "integer"
      },
      "source": {
        "additionalProperties": false,
        "properties": {
          "case_id": {
            "type": "string"
          },
          "category": {
            "type": "string"
          },
          "id": {
            "type": "string"
          },
          "kind": {
            "type": "string"
          },
          "scope": {
            "type": "string"
          },
          "text": {
            "type": "string"
          },
          "title": {
            "type": "string"
          }
        },
        "required": [
          "id",
          "kind",
          "category",
          "scope",
          "text"
        ],
        "type": "object"
      }
    },
    "required": [
      "expected_version",
      "source"
    ],
    "type": "object"
  }
}

upload_context_document

Upload a text, Markdown, JSON or JSONL document up to 60 KB to system understanding. Provide category and expected_version from get_system_context. Export other document formats to text first.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "form": {
    "additionalProperties": false,
    "properties": {
      "category": {
        "type": "string"
      },
      "expected_version": {
        "type": "integer"
      }
    },
    "type": "object"
  },
  "file_field": "file",
  "max_file_bytes": 60000
}

seed_system_context

Seed system context. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

summarize_system_context

Queue a summary of the system’s saved understanding. Poll get_system_context for completion. Does not start a Bench evaluation.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

delete_context_source

Delete context source. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    },
    "sourceID": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

list_context_connections

List context connections. Does not start an evaluation.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

create_context_connection

Connect a supported evidence provider using explicit content consent and selected project configuration. Store provider credentials only in secrets. Synchronization imports redacted evidence; it does not make that evidence business policy.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "config": {
        "additionalProperties": false,
        "properties": {
          "base_url": {
            "type": "string"
          },
          "dataset_ids": {
            "items": {
              "type": "string"
            },
            "type": "array"
          },
          "environment": {
            "type": "string"
          },
          "project": {
            "type": "string"
          },
          "prompt_names": {
            "items": {
              "type": "string"
            },
            "type": "array"
          },
          "provider": {
            "enum": [
              "langfuse",
              "langsmith"
            ],
            "type": "string"
          },
          "since": {
            "type": "string"
          }
        },
        "required": [
          "provider",
          "base_url",
          "project",
          "since"
        ],
        "type": "object"
      },
      "content_consent": {
        "type": "boolean"
      },
      "credentials": {
        "additionalProperties": false,
        "properties": {
          "api_key": {
            "type": "string"
          },
          "public_key": {
            "type": "string"
          },
          "secret_key": {
            "type": "string"
          }
        },
        "type": "object"
      },
      "retention_days": {
        "type": "integer"
      }
    },
    "required": [
      "config",
      "credentials",
      "content_consent"
    ],
    "type": "object"
  }
}

sync_context_connection

Sync context connection. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "connectionID": {
      "minLength": 1,
      "type": "string"
    },
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

delete_context_connection

Delete context connection. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "connectionID": {
      "minLength": 1,
      "type": "string"
    },
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

update_system

Update system. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "description": {
        "type": "string"
      },
      "name": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

resolve_finding

Resolve finding. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "key": {
        "type": "string"
      },
      "status": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

delete_system

Delete system. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

add_system_component

Attach a scanned prompt, tool, model configuration or harness to a system. Use the exact repo_full_name, branch and call_site_id returned by a scan. This does not edit source code.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "type": "string"
      },
      "call_site_id": {
        "type": "string"
      },
      "repo_full_name": {
        "type": "string"
      },
      "role": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

remove_system_component

Remove system component. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "componentID": {
      "minLength": 1,
      "type": "string"
    },
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

set_continuous_evaluation

Set continuous evaluation. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "path_parameters": {
    "componentID": {
      "minLength": 1,
      "type": "string"
    },
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "enabled": {
        "type": "boolean"
      }
    },
    "type": "object"
  }
}

add_system_connection

Add system connection. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "edge_type": {
        "type": "string"
      },
      "from_component_id": {
        "type": "integer"
      },
      "to_component_id": {
        "type": "integer"
      }
    },
    "type": "object"
  }
}

remove_system_connection

Remove system connection. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "edgeID": {
      "minLength": 1,
      "type": "string"
    },
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

create_system_run

Create system run. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

list_system_runs

List system runs. Does not start an evaluation.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

get_system_run

Get system run. Does not start an evaluation.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    },
    "runID": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

link_system_run

Link system run. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    },
    "runID": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "ai_system_component_id": {
        "type": "integer"
      },
      "evaluation_run_id": {
        "type": "integer"
      }
    },
    "type": "object"
  }
}

complete_system_run

Complete system run. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    },
    "runID": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "outcome": {
        "type": "string"
      },
      "outcome_reason": {
        "type": "string"
      },
      "status": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

get_production_health

Get production health. Does not start an evaluation.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

get_production_feedback

Get production feedback. Does not start an evaluation.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

set_production_feedback

Configure production checks and optional automatic investigations for one environment. Enabling automatic evaluations can spend the account allowance; confirm with the user first. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "auto_bench": {
        "type": "boolean"
      },
      "enabled": {
        "type": "boolean"
      },
      "environment": {
        "type": "string"
      }
    },
    "required": [
      "enabled",
      "environment"
    ],
    "type": "object"
  }
}

production_capabilities

Production capabilities. Does not start an evaluation.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

sdk_status

Sdk status. Does not start an evaluation.

{
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "type": "string"
      },
      "environment": {
        "type": "string"
      },
      "repo_full_name": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

get_trace_privacy

Get trace privacy. Does not start an evaluation. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "type": "string"
      },
      "repo_full_name": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

set_trace_privacy

Set trace privacy. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "type": "string"
      },
      "repo_full_name": {
        "type": "string"
      }
    },
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "enabled": {
        "type": "boolean"
      },
      "fields": {
        "items": {
          "type": "string"
        },
        "type": "array"
      }
    },
    "required": [
      "enabled",
      "fields"
    ],
    "type": "object"
  }
}

get_production_check

Get production check. Does not start an evaluation.

{
  "path_parameters": {
    "jobID": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

retry_production_check

Retry production check. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "jobID": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

ingest_traces

Upload explicit SDK trace data. Prefer SDK instrumentation for automatic bounds and redaction. Capture content only when the application owner has opted in.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "type": "string"
      },
      "capture_content": {
        "type": "boolean"
      },
      "repo_full_name": {
        "type": "string"
      },
      "system_name": {
        "type": "string"
      },
      "traces": {
        "items": {
          "additionalProperties": false,
          "properties": {
            "source": {
              "type": "string"
            },
            "spans": {
              "items": {},
              "type": "array"
            },
            "trace_id": {
              "type": "string"
            }
          },
          "type": "object"
        },
        "type": "array"
      }
    },
    "type": "object"
  }
}

list_traces

List traces. Does not start an evaluation.

{
  "query": {
    "additionalProperties": false,
    "properties": {
      "branch": {
        "type": "string"
      },
      "environment": {
        "type": "string"
      },
      "limit": {
        "type": "string"
      },
      "repo_full_name": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

get_trace

Get trace. Does not start an evaluation.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

delete_trace

Delete trace. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

link_span

Link span. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    },
    "spanID": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "ai_system_component_id": {
        "type": "integer"
      },
      "ai_system_id": {
        "type": "integer"
      }
    },
    "type": "object"
  }
}

unlink_span

Unlink span. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    },
    "linkID": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

evaluate_span

Evaluate span. Uses the same validation and plan limits as Bench.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    },
    "spanID": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "component_id": {
        "type": "integer"
      },
      "evidence_scope": {
        "type": "string"
      },
      "no": {
        "type": "string"
      },
      "provider": {
        "type": "string"
      },
      "question": {
        "type": "string"
      },
      "share_with_typesafe": {
        "type": "boolean"
      },
      "yes": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

get_workspace

Get workspace. Does not start an evaluation.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

create_workspace

Create workspace. Uses the same validation and plan limits as Bench.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "name": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

rename_workspace

Rename workspace. Uses the same validation and plan limits as Bench.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "name": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

invite_workspace_member

Invite workspace member. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "email": {
        "type": "string"
      },
      "role": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

accept_workspace_invite

Accept workspace invite. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "token": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

set_workspace_member_role

Set workspace member role. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "path_parameters": {
    "userID": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "role": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

remove_workspace_member

Remove workspace member. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "path_parameters": {
    "userID": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

create_api_key

Create a scoped automation credential. Returns plaintext only once. Store it in the application’s secret environment, never in source, logs or chat replies. OAuth account authorization is required; API keys cannot mint other keys. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "max_evaluations_per_month": {
        "type": "integer"
      },
      "name": {
        "type": "string"
      },
      "org_read": {
        "type": "boolean"
      },
      "repo_allowlist": {
        "items": {
          "type": "string"
        },
        "type": "array"
      }
    },
    "type": "object"
  }
}

get_setup_key

Retrieve the account’s reusable SDK setup credential. Never print or commit the returned secret. replace_revoked=true explicitly replaces a revoked setup key. OAuth account authorization is required. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "replace_revoked": {
        "type": "boolean"
      }
    },
    "type": "object"
  }
}

list_api_keys

List api keys. Does not start an evaluation. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

revoke_api_key

Revoke api key. Uses the same validation and plan limits as Bench. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "path_parameters": {
    "id": {
      "minLength": 1,
      "type": "string"
    }
  },
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

list_plans

List plans. Does not start an evaluation.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

get_payment_method

Get payment method. Does not start an evaluation.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

list_invoices

List invoices. Does not start an evaluation.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

create_checkout

Create a Stripe checkout link for a selected plan price. Obtain the user’s approval before starting payment. Return the URL for payment and coupon entry; do not handle card details in the agent. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "price_id": {
        "type": "string"
      },
      "return_path": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

change_plan

Change the subscription to the chosen configured price. This can bill immediately. Obtain explicit approval for the price and interval before calling; use get_billing_portal for coupon-aware upgrades. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  },
  "body": {
    "additionalProperties": false,
    "properties": {
      "price_id": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

get_billing_portal

Create a Stripe billing or upgrade link. An optional target price_id opens immediate upgrade confirmation with promotion-code entry. Return the URL to the user to review payment. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "query": {
    "additionalProperties": false,
    "properties": {
      "price_id": {
        "type": "string"
      }
    },
    "type": "object"
  }
}

cancel_subscription

Cancel renewal of the subscription. This changes billing; require explicit user approval. Organization billing requires an administrator. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

reactivate_subscription

Reactivate subscription renewal. This changes billing; require explicit user approval. Organization billing requires an administrator. Requires Bench OAuth account authorization; a scoped API key is insufficient.

{
  "query": {
    "additionalProperties": false,
    "properties": {},
    "type": "object"
  }
}

Start here

Evaluate and improve

MCP

Bench SDK

Data controls

Reference

Headless operation reference

Inputs

​Inputs

Inputs