Google proposes evaluating coding agents through observable behaviour

The Google Developers Blog has described an evaluation strategy for coding agents based on observable behaviour rather than only on the final result of a task. The proposal addresses a practical problem: an end-to-end benchmark may show that an agent has fallen by a few points, but it rarely explains what changed or which behaviour needs correction.

The authors recommend using behavioural evaluations as an intermediate layer. Instead of asking only whether an agent completed a multi-file refactor, a test can check whether it asked for clarification when given an ambiguous instruction, ran the local validator after modifying a build file, or included canonical links when generating documentation. These are small, fast checks tied to concrete system actions.

The comparison with integration testing is useful. A behavioural evaluation acts as a contract for the agent harness: it observes tool calls, file changes, and intermediate decisions. A prompt, tool-schema, or model change can therefore be run through a local, deterministic suite before reaching an expensive benchmark or shared environment. The result does not prove that an agent can solve every task, but it helps detect specific regressions and locate their cause.

The order of adoption also matters. Google advises starting with developer instinct and real use of the system, a practice it calls dogfooding. A team can observe how the agent works on its own repository, automates routine tasks, and reaches its current limits. Once there are behaviours worth protecting, they can be turned into an evaluation suite. Building complex evaluation infrastructure before understanding those behaviours would produce metrics that are difficult to interpret.

This separation between broad capability and operational discipline fits established software engineering practice. An agent may produce a solution that passes tests while still ignoring a security instruction, inventing a command-line option, or changing an area that was supposed to remain untouched. Intermediate assertions can capture those failures even when the output appears correct in one particular case.

The lesson is not to replace broad benchmarks. End-to-end tests remain necessary for checking whether the system can complete whole jobs. The proposal is to combine them with cheaper controls that guide the agent’s evolution. That design creates a proportional verification chain: detailed observations for iteration, full benchmarks for capability confirmation, and human review for risks no metric fully covers.

For teams integrating agents into development workflows, the main benefit is maintainability of the AI system itself. Prompts, tools, and models change; the behaviours the team considers mandatory should be expressed as repeatable tests.