Skip to main content
Real app testing runs your application and its tools to check the outcome. Simulation testing tests prompts and models using simulated tool replies.

​
Real app testing

Use this when you need to know whether the app actually did the right thing. For a Weather Agent, check that its weather tool ran, returned a result and that the answer used it correctly. For a refund assistant, check the recorded refund, not only an answer saying “refunded.”
  1. Open Real app testing and choose Set up SDK.
  2. Connect the function that handles a request. Record its model and tool calls.
  3. Add example requests and expected results. Run in a test environment with test services or isolated fixtures for external actions.
  4. Publish the report to Bench and review it under Real app testing.
The real app testing setup guide includes the code and report-publishing steps. Installing the SDK or connecting GitHub alone does not run these tests. SDK reports cover the behavior you instrument and are labeled SDK test report.

​
Simulation testing

Use this to compare prompts and models before connecting the full app. Bench runs model conversations and supplies simulated replies for declared tools. For a Weather Agent, it can test how the model responds to an example weather tool result. It does not check whether your weather service actually works.
  1. Connect a repository or upload a complete prompt. If your app fills in parts of the prompt at runtime, provide a representative full prompt first.
  2. Add rules in Understanding and example inputs and expected answers in Test library.
  3. Choose Start benching. Review the cases, failed checks and proposed changes in Results. Runs using simulated tools are labeled Simulation mode.
See test cases and evaluation results. Simulation results do not verify real tool execution or changes to a database.

​
How a scan fits in

A quick scan reads code and identifies things to investigate. It does not run either kind of test. Open a finding’s right sidebar to see its source evidence and next step. A missing prompt or an untested tool is a coverage gap until tested.
The SDK also has scripted conversation helpers such as simulateSystem. These can drive your real app with isolated test dependencies across several turns. Their evidence depends on the app and tools you connect; they are different from the simulated tool replies in a prompt bench. See scripted app conversations.