Recognition is not execution
Bench’s static recognition catalog covers 18 framework and SDK families. It looks for dependencies, imports, supported declarations and model calls without executing repository code. An installed package is evidence of a dependency, not proof that an agent exists or is running.
The Python extractor also recognizes selected Cohere, Groq and LiteLLM call
patterns. These are additional call-site patterns, not three more complete
agent-runtime integrations.
Language coverage
The deepest extraction is in Python and JavaScript/TypeScript, including JSX/TSX. It can resolve selected imports, model clients, prompts, tools and functions. Java, Kotlin, Go, C#, Rust, Ruby, PHP and Swift have narrower extraction of explicit prompt constructs. These entries need review; they are not full parsers for those languages’ agent runtimes. Prompt/configuration files offer another entry point.Workflows, agents and tools
Known same-file relationships can group declared agents into a multi-agent candidate. Ordinary workflow nodes are not automatically separate agents. Two agents are not grouped merely because they share a repository. Tool discovery is based on declarations and references, not a fixed catalog of named SaaS tools. For supported patterns Bench retains available tool names, descriptions, parameters and source locations. A custom weather, search or order tool can therefore be visible without a provider-specific connector. There is no claim that every discovered tool can be executed by Bench. The current prompt runner can expose a best-effort tool declaration to a model and inspect its response or requested tool call. It does not run that tool’s code, call a production database, or prove a downstream action succeeded. Some parameter types are simplified, so complete schema fidelity is not guaranteed.What can be benched today?
The main execution unit is an extracted prompt and configured model, with supported case inputs and tool declarations. Eligibility depends on a resolvable prompt, supported model route, credentials, replayable inputs and available allowance. Recognizing a framework does not guarantee all its models, custom transports or runtime behavior can be evaluated. Bench can test observable response behavior, format, supported tool-call decisions and criteria grounded in supplied evidence. It can compare prompt/model candidates on the same test suite and report measured score and estimated model-cost changes. Whole-application tool execution, orchestration retries, permissions, side effects, real-time voice behavior and arbitrary multi-agent handoffs require additional runtime adapters and controlled execution. Static topology is not proof of those behaviors. E2B validation fixtures do not change that support boundary.Known scan limits
- Dynamic prompt values, runtime-only tools and remote prompt stores may need SDK evidence or explicit inputs.
- Cross-file agent relationships, function-local factories, subclasses and chained workflow construction can be incomplete.
- Notebooks and fenced documentation examples are not equivalent to application source files.
- Recognition traversal is bounded to 5,000 visited files, 2 MB per file and 20 MB total. Tests are excluded by default; hidden paths and symlinks are excluded.
- Scanned structure is a proposal with source evidence, not a runtime certification.