Skip to content

Diff-Coverage Gating for Agent-Authored Pull Requests

Gate an agent's pull request on the test coverage of its changed lines instead of trusting that the agent tested its own diff.

Compute coverage over only the lines a pull request adds or edits, then block the merge when that number falls below a threshold. This makes the agent's own diff the coverage obligation, rather than the whole-repository percentage that a large existing suite already inflates. The gate exists because agents do not reliably test what they change: across 4,882 agent-authored PRs in the AIDev dataset, agents included a test change in only 49.6% of PRs that touched code under test, and existing tests executed 61.5% of changed lines in Java but just 27.0% in Python (arXiv:2607.18057).

The empirical case

The study covered five agents — Codex, Copilot, Cursor, Claude Code, and Devin — across 532 Java and 4,350 Python PRs (arXiv:2607.18057). Three findings drive the gate:

  • Half of code-touching PRs ship no test change at all (49.6% include one).
  • 64.8% of Python PRs have no changed line executed by any existing test.
  • Error paths are the worst blind spot: try/catch blocks were missed at 86.0% in Java and 81.0% in Python.

Among Java PRs without coverage gains, excluding those already at full coverage, agents delete 82 tests and add 31. In Python PRs without gains, 74.8% add tests that miss the changed lines (RQ2b).

How to apply it

  • Measure coverage restricted to the diff. Tools such as diff_cover compute the covered fraction of added and edited lines from a standard coverage report and a git diff.
  • Set a changed-line threshold on the PR check, not a whole-repo one. Codacy and CodeScene both gate on diff coverage with a default near 70%; Azure DevOps reports the same metric on pull requests.
  • Weight error-handling paths. Line coverage alone under-counts the try/catch branches agents miss most, so pair the gate with branch coverage or a targeted review of new exception handling.
  • Require a reason for each deleted test. Review whether the behavior still needs protection before accepting its removal.

Review existing tests alongside coverage

Milanese et al. compare 114,134 merged Java PRs from 391 repositories, spanning January 2025 to April 2026. Their within-repository models control for change size. Agent-supported PRs have higher odds of adding tests (1.50 to 1.58), but lower odds of modifying (0.70 to 0.76) or deleting them (0.51 to 0.77). These are associations in reviewed human-agent contributions (study, RQ2 and limitations).

This differs from the selected Java subset above. That result counts tests in agent PRs without coverage gains; this comparison measures whether PRs perform each task relative to human-authored PRs.

The study does not establish faster suite growth or obsolete-test accumulation. Those remain longitudinal research questions. Coverage analysis covers only 95 buildable repositories. Among PRs that change tests, it detects no production-change coverage difference. That does not establish equivalence, and it says nothing about PRs that change production code without tests. Smell differences remain uncertain (discussion and limitations).

Use the authors' maintenance recommendations alongside the gate (implications, section 6.4):

  • Ask the agent to inspect existing tests linked to changed production methods before adding tests.
  • Request a report of which existing tests it updates, removes, or retains.
  • Check whether modified tests still contain assertions.
  • For test-only PRs, inspect which parts of the tested classes the added tests exercise.

Treat these checks as review guidance. The observational study does not evaluate their effect as merge gates.

When this backfires

The gate measures execution, not correctness, and a threshold is a target an agent will optimize against:

  • Assertion-free gaming. An agent can clear the threshold with tests that execute the changed lines but assert nothing, so the gate certifies untested behavior as tested (Code4IT). Pair the gate with mutation testing, which fails a suite that would not notice a regression.
  • Trivial diffs blocked. Pure refactors, generated code, and config-only changes have no meaningful branches to cover, so a uniform threshold blocks safe PRs and pushes teams toward blanket exemptions that erode the gate.
  • Line coverage misses the branches that matter. A high line-coverage number can still leave the rare error paths agents miss most uncovered, so a line-only gate under-weights exactly the risk it was added for.

Key Takeaways

  • Scope the coverage metric to the PR's changed lines; a whole-repo percentage hides an untested diff behind a large existing suite.
  • The gate is justified empirically: agents test under half of code-touching PRs, and existing tests cover 27.0% of changed Python lines (arXiv:2607.18057).
  • Review the reason for test deletions and request maintenance of existing tests linked to changed code.
  • Coverage proves execution, not assertion; pair the gate with mutation testing or branch coverage so it resists assertion-free gaming.
  • Exempt trivial diffs deliberately rather than lowering the threshold for everyone.