Skip to main content
Real app testing runs your application and its tools to check the outcome. Simulation testing tests prompts and models using simulated tool replies.

​
Real app testing

Use this when you need to know whether the app actually did the right thing. For a Weather Agent, check that its weather tool ran, returned a result and that the answer used it correctly. For a refund assistant, check the recorded refund, not only an answer saying “refunded.”
  1. Open Real app testing and choose Set up SDK.
  2. Connect the function that handles a request. Record its model and tool calls.
  3. Add example requests and expected results. Run in a test environment with test services or isolated fixtures for external actions.
  4. Publish the report to Bench and review it under Real app testing.
The tab opens with an attention strip (failures, proposed fixes, quality, cost) and Quality and Cost columns. Tests and cases open in a right-hand drawer with Bench outcome, Output and Case tabs, a timeline naming the tools and models per span, and a Copy fix prompt action for your coding agent. Costs are estimated from tokens when the SDK did not report bench.cost.usd. The real app testing setup guide includes the code and report-publishing steps. Installing the SDK or connecting GitHub alone does not run these tests. SDK reports cover the behavior you instrument and are labeled SDK test report.

​
Simulation testing

Use this to compare prompts and models before connecting the full app. Bench runs model conversations and supplies simulated replies for declared tools. For a Weather Agent, it can test how the model responds to an example weather tool result. It does not check whether your weather service actually works.
  1. Register the prompts from your coding agent, connect a repository or upload a complete prompt. If your app fills in parts of the prompt at runtime, register the template with its variables unfilled.
  2. Record the purpose and rules in Understanding (a bench needs them) and example inputs and expected answers in Test library.
  3. Choose Start benching, or the Bench button on a quick-scan finding. Review the cases, failed checks and proposed changes in Results. Runs using simulated tools are labeled Simulation mode. Failed production checks start this kind of bench automatically with the failing interactions as cases.
See test cases and evaluation results. Simulation results do not verify real tool execution or changes to a database.

​
How a scan fits in

A quick scan reads the components drawn from your traces and the registered prompts, and identifies things to investigate: prompts whose quality is not measured yet, tools and hand-offs not tested yet, and cheaper models to bench. It does not run either kind of test. Open a finding’s right sidebar to see its evidence and next step; every finding has a Bench button. An unmeasured prompt or an untested tool is a coverage gap until tested.
The SDK also has scripted conversation helpers such as simulateSystem. These can drive your real app with isolated test dependencies across several turns. Their evidence depends on the app and tools you connect; they are different from the simulated tool replies in a prompt bench. See scripted app conversations.