A survey separates real TDD from merely passing tests with LLMs

A review published on arXiv analyzes 87 studies involving tests, language models, and software-engineering agents. Its main contribution is not to claim that AI has replaced TDD, but to separate practices that are often grouped under the same label. For teams evaluating coding assistants, that distinction makes both benchmark results and continuous-integration evidence easier to interpret.

The study distinguishes the Red-Green-Refactor cycle from several neighboring techniques. In some systems, tests condition generation before an implementation exists. In others, they guide repair after a failure, select among candidate solutions, locate a change, or provide traces for an analysis task. All of these practices use execution or assertions, but they do not require the same ordering, source of expectations, or permissions for the agent.

The review proposes asking which decision a test actually changes. It may define expected behavior, modify an implementation, reject a candidate, stop an iteration, or provide evidence for comparing programs. The framework also separates four properties that are often conflated: test availability, test validity, effective use of feedback, and evaluation independence. The existence of a suite does not prove that it covers the important risk, and an agent running it does not prove that it interpreted the result correctly.

The most useful practical warning is that passing tests alone does not prove behavioral equivalence. A test generated by the same model that wrote the code may share its mistakes or assumptions. A compiler may accept an implementation that violates a functional property. A benchmark with public tests may measure the ability to fit a known output rather than robustness against cases the agent has never seen.

This does not make tests less valuable. It makes their role more specific. A team can begin by documenting who proposes each input, who determines the expected output, and what validates that expectation. For agent-generated changes, it is reasonable to combine visible tests with hidden or independent cases, check behavioral regressions, and measure the long-term cost of maintaining the test assets. Teams should also evaluate the procedure, not just the result: if an agent was expected to reproduce a failure before fixing it, the workflow should record whether that failure was actually observed.

The review extends its map to translation, refactoring, repair, code search, fault localization, and formal validation. Its editorial message is restrained: a test is an intervention in an engineering decision, not merely a coverage counter. That perspective helps teams design better experiments, avoid inflated claims about agents, and build quality gates that provide evidence about behavior, security, and maintainability.