Code fixes

Local preview. Bench exports repair evidence and includes a local validation runner. Hosted coding-agent execution, automatic PR publication and deployment are not enabled.

The workflow

Failed evaluation -> fix brief -> coding agent -> candidate patch
                                                   |
                          regression + incident + holdout checks
                                                   |
                                      validated for review

An agent writes the proposed change. Bench’s validator determines whether it passes the configured checks. A validated patch is not a deployed fix.

1. Export the evidence

Open an evaluation in History, choose Fixes, then select Fix brief, or call bench_get_fix_brief with the completed run ID through MCP. This is a read-only operation with no evaluation charge. The brief includes that run’s rubric, cases, baseline and candidate evidence. Deleted or incomplete evidence cannot be exported as a repair task. Treat a brief as sensitive source data. Share it only with an approved coding agent. Saved optimization cases are not independent holdout cases. For a quicker handoff, choose Prepare fix prompt and Copy prompt. Paste it into Codex, Claude Code or another trusted coding agent. Open in Cursor uses Cursor’s local application link. It does not route your source evidence through a web redirect. If the prompt exceeds the link limit, Bench copies the complete prompt and opens a short instruction asking you to paste it. Review it before running the agent. Cursor documents this handoff.

2. Pin the source and checks

Confirm a clean checkout matches the evaluated code, then record its full commit SHA. New runs can include the commit recorded in their source snapshot. Older runs still need an explicit source revision; Bench does not invent one. Choose a small set of editable application files. The local runner rejects protected tests, policies, dependency manifests, hidden files and credential files. An operator supplies a validation plan outside the source checkout:

{
  "schema_version": 1,
  "source_revision": "REPLACE_WITH_FULL_40_CHARACTER_COMMIT_SHA",
  "allowed_paths": ["src/refunds.py"],
  "checks": [
    {
      "name": "existing_behavior",
      "kind": "regression",
      "argv": ["python3", "/trusted/checks/regressions.py"],
      "timeout_seconds": 60
    },
    {
      "name": "reported_failure",
      "kind": "incident",
      "argv": ["python3", "/trusted/checks/refund_failure.py"],
      "timeout_seconds": 60
    },
    {
      "name": "unseen_cases",
      "kind": "holdout",
      "argv": ["python3", "/trusted/checks/holdout.py"],
      "timeout_seconds": 60
    }
  ]
}

Commands execute with the snapshot as their working directory. Use an absolute interpreter path when needed. The runner does not install dependencies; use a pinned, pre-provisioned environment. Executable repository file modes are not preserved, so invoke scripts through their interpreter. Checks must return 0 for passing assertions, 1 for a reproduced behavior failure, and another exit code for setup or infrastructure errors. Do not hide provider failures as a failed quality assertion. Baseline regression checks must pass; each incident check must fail on baseline and pass on candidate. Candidate holdout checks must pass. Keep the holdout separate from the brief and authoring workspace. The local runner does not provide OS isolation or stop a malicious program reading other files.

3. Generate the candidate

Give the brief, source files and their SHA-256 hashes to Claude Code, Codex, Cursor, or an approved agent adapter. Ask for a minimal repair, not changed tests or weaker criteria. The proposal contract is:

{
  "changes": [{
    "path": "src/refunds.py",
    "before_sha256": "SHA256_OF_THE_ORIGINAL_FILE_BYTES",
    "content": "Complete replacement source text, not a partial snippet.\n"
  }]
}

Only existing, allowlisted UTF-8 files are supported. File additions, deletions, symlinks, submodules, dependency upgrades and infrastructure edits need a separate workflow. The author has no authority to declare validation successful.

4. Run validation

From the local bench-pipeline checkout:

.venv/bin/python -m pipeline.code_fixes \
  --repo /absolute/path/to/clean/source \
  --plan /absolute/path/to/validation-plan.json \
  --brief /absolute/path/to/bench-fix-42.json \
  --proposal /absolute/path/to/proposal.json \
  --output /absolute/path/to/new-review-directory \
  --allow-local-execution

For automatic generation and validation in one invocation, replace --proposal with an operator-owned adapter command:

--agent-command '["/trusted/agent-adapter","--workspace","{workspace}","--brief","{brief}","--output","{proposal}"]'

The runner supplies a separate authoring snapshot, brief and output path. The adapter must write the proposal contract and exit. Bench reconstructs a fresh candidate from the pinned source, so it ignores direct agent edits outside the proposal. This preview includes the command contract, not a bundled Codex/Claude API adapter. Configure an approved provider proxy in the adapter; ambient credentials are not inherited.

--allow-local-execution authorizes real commands on your machine. Use it only with operator-owned, trusted repositories and commands. It is not a sandbox. Never expose this runner directly to uploaded customer code or execute it inside the API process.

5. Review the outcome

The new output directory contains candidate.patch and report.json. Reports record the source revision, plan and brief hashes, file hashes, check exits, timeouts and output hashes. Raw command output is not retained. Output is bounded to 1 MB per command, checks to 300 seconds each, and an authoring command to 300 seconds. validated_for_review requires every configured gate. Rejected candidates remain rejected even if the agent says they passed. The original checkout stays unchanged. The emitted patch is applied to a fresh snapshot and checked against the validated candidate, including exact file bytes and line endings. No branch is pushed and no PR is opened by this runner. Review a patch against its recorded base before applying it. Changes after validation require validation again.

Hosted rollout

Use a separate isolated worker, such as an E2B microVM, before enabling repairs for untrusted repositories. Separate generation and validation sandboxes, restrict egress, keep provider/GitHub credentials behind an authenticated proxy, pin dependencies, enforce tenant and spend budgets, and destroy sandboxes on every exit path. The existing bench_open_prompt_pr tool writes supplied files to GitHub. It does not enforce this validator’s receipt or revalidate code, so it is not the production auto-fix publisher yet. A hosted publisher must verify the receipt and current base again, then require permission before opening a PR. Never automatically merge or claim a production issue is resolved before rollout measurements confirm it. See validation and rollout for the release gates.

Start here

Evaluate and improve

MCP

SDK

Development

The workflow

1. Export the evidence

2. Pin the source and checks

3. Generate the candidate

4. Run validation

5. Review the outcome

Hosted rollout

​The workflow

​1. Export the evidence

​2. Pin the source and checks

​3. Generate the candidate

​4. Run validation

​5. Review the outcome

​Hosted rollout

The workflow

1. Export the evidence

2. Pin the source and checks

3. Generate the candidate

4. Run validation

5. Review the outcome

Hosted rollout