Skip to main content
Open Test library in an AI system. Cases and Criteria are separate tabs. Cases starts with All cases. Browse ten items per page or filter by source. A case is a scenario; a criterion is a rule used to judge it. A rubric is the collection of those criteria. Bench still generates criteria and scenarios automatically.

​
Add a case

Choose Add case, enter the input and expected result, then save. If the system has several prompts, choose the target. Cases are prompt-specific so an expectation for one agent does not become a policy for every agent.

​
Edit a case

Choose Edit to open the case beside the table. Update Input and Expected result. Open More details for notes and any saved conversation, variables or context. Structured values use individual fields rather than a read-only JSON dump. Save for future benches creates a new context revision. Editing a Bench-retained case replaces its future-use copy with your correction. Bench does not run both versions, change previous scores, or allow a later automatic import to overwrite your correction. Use Pause to leave a case in the library without including it in future benches. Use Remove for a custom or corrected case you no longer want stored. Removing evidence can also erase the saved details of evaluations that used it; their scores remain unchanged.

​
Import a dataset

1

Upload

Choose Upload cases. CSV, XLSX, JSON and JSONL are supported, up to 4 MB and 5,000 rows. Select a worksheet for a multi-sheet workbook.
2

Map and preview

Map the input and expected result columns. You can also map history, actual outputs, variables, resolved prompts and notes. Extra columns and repeated inputs are preserved after redaction.
3

Save

Save the mapped dataset. Comparing recorded answers is an exact, case-sensitive comparison, not a semantic LLM judgment.
4

Include in a bench

Choose Use a dataset, select the saved dataset and target prompt, then Add golden cases. This copies that dataset version into the library. Uploading alone does not start an evaluation.
Missing inputs or expected labels must be corrected before promotion. Bench does not invent runtime variables or silently shorten conversations. Supported prompt replay uses text user/assistant history and simple named substitutions. Tool, multimodal or unsupported template histories need a runtime adapter and can remain unscored.

​
Customize a criterion

Choose Criteria, then Add criterion. State the rule, what passes and what fails. Bench carries it into rubric routing and test generation for that prompt. If a criterion cannot be observed by the available evaluator, it must remain unscored rather than receive a guessed result. Edit or pause a custom criterion to change future benches. Existing runs keep their pinned revision. Generated criteria show Customize to create a user rule; they are not editable historical records. Their saved rubric also remains in History. If no benchmark has generated a rubric yet, there are no generated criteria to show.

​
Regressions and capabilities

Retained cases from this exact system can seed a later bench, subject to the case budget below. A candidate that breaks a previously passing check in the selected suite is not recommended just because its average improves. Primary and secondary criteria both count. Changed context or expected behavior establishes a new version. For actual application-level capability, tool and orchestration tests, use application-runtime evaluation. The prompt benchmark does not execute your application tools. History alone does not guarantee coverage of every past failure; inspect the cases actually included.

​
Limits and inclusion

Your plan limits the number of cases executed per evaluation. Explicit cases are considered first; the saved suite records included and excluded supplied cases. Review that manifest when a large dataset exceeds the case budget. A source remaining in the library does not mean every row ran in every bench. Review the expected behavior before relying on a Bench case. An automatically retained case is a regression candidate, not independent human validation.