Skip to content

Structure-Aware Diff Labeling with Two-Stage LLM Pipelines

A two-stage LLM pipeline labels diff hunks against a 12-type taxonomy, then refines cross-hunk relationships — a polyglot supplement to static analysis, not a replacement.

Structure-aware diff labeling assigns a change type to each hunk in a patch, then resolves cross-hunk relationships such as which hunk declares a renamed symbol and which hunks consume it. A two-stage LLM pipeline does this without per-language static analysis tooling, trading determinism and cost for language coverage and label customization (arxiv:2605.26100).

The approach is qualified: the paper's own evaluation shows performance varies across label types and recommends a hybrid with static analysis for the categories that matter most.

When this applies

Use a two-stage LLM labeler when:

  • The codebase spans multiple languages and per-language refactor detectors (RefactoringMiner for Java, ts-morph for TypeScript, libcst for Python) would require maintaining separate pipelines.
  • The label taxonomy needs to evolve over time. Few-shot prompting lets you add categories without retraining or rewriting AST rules (arxiv:2605.26100).
  • The downstream consumer (reviewer prioritization, PR routing, automated comment suppression) tolerates non-deterministic output and ~80% precision/recall rather than requiring 100% reproducible labels.

Skip it when a single-language project has mature static-analysis tooling, when CI demands deterministic outputs for audit purposes, or when per-PR token cost dominates the budget.

The two stages

Stage 1 is the Labeler. It runs per-hunk classification using few-shot prompting against a fixed label set. The paper uses 12 types: documentation, testing, output handling, retype, code move, style change, logging, rename, error handling, logic change, internal interface change, external interface change (arxiv:2605.26100). Each hunk gets a label set with 5 lines of local context. Three modes — per-hunk, per-file, per-patch — trade context length for token cost.

Stage 2 is the Refiner. It runs whole-patch inference to capture cross-hunk relationships. A rename labeled in isolation does not record which hunk holds the declaration and which hold the consequences. So the Refiner assigns a parent field (parent=0 for declaration, parent=N for usage) and extracts attributes like old/new names and types. It also corrects misclassifications visible only in whole-patch view (arxiv:2605.26100).

The single-shot Refiner pass is the central idea. Without it, a hunk-only labeler can name change types but cannot link them into the structural patterns that shape review prioritization. Knowing four hunks are renames is less useful than knowing they all consume one declaration.

Why it works

Decomposing classification into per-hunk labeling plus a whole-patch refinement makes each LLM call work within its strengths: short-context few-shot inference for the labeling step and large-context relational reasoning for the refinement step. Few-shot prompting transfers across programming languages without per-language training, so one prompt handles Java and Python diffs with comparable accuracy in the paper's benchmark (arxiv:2605.26100). The Refiner's whole-patch view supplies the global context that per-hunk inference structurally cannot represent. Declaration-to-usage parenting and move source-target pairing emerge from seeing the entire patch in one pass.

When this backfires

  • Mature single-language tooling exists. RefactoringMiner-class static analyzers produce deterministic, reproducible labels for Java refactors at near-zero marginal cost. Trading that for non-deterministic LLM output is a net loss when language coverage is not a requirement.
  • Cost-sensitive PR-level CI. The paper's best-performing model (Gemini-3-Pro-Preview) consumed up to 7.5× more output tokens than the second-best model (Claude Sonnet 4.5) (arxiv:2605.26100). Running this on every PR at scale is expensive, and cheaper models lose meaningful accuracy.
  • Under-represented label types matter most. Performance varies substantially across label types in the paper's evaluation, with external interface, error handling, and log labels suffering more than rename or logic-change labels (arxiv:2605.26100). A workflow that prioritizes one of those categories is precisely the case where a hybrid with static analysis is required.
  • Deterministic audit trails. Compliance contexts that require reproducible classification cannot tolerate LLM non-determinism. Static analysis — a RefactoringMiner-class tool — stays the only viable option.
  • Very large patches. The Refiner runs whole-patch inference, so long diffs blow out the context window and force chunking, which breaks the cross-hunk structural relationship signal the Refiner exists to capture.
  • Headline metrics hide category-level failures. LLM-as-classifier prompts optimized for catching true positives misclassify more false positives, and chain-of-thought reasoning amplifies misjudgement on certain inputs (arxiv:2601.18844, arxiv:2508.12358). The 84/81 paper result is one calibration point, not a Pareto frontier.

The paper's own recommendation: "a hybrid strategy can be adopted — using LLM-based labeling for most types while relying on static analysis for a small set of critical label types" (arxiv:2605.26100).

Example

The paper's benchmark uses patches drawn from SWE-bench Multilingual and SWE-PolyBench supplemented with fabricated patches for label-type coverage. It totals 95 hunks across 13 PRs, mostly Java with Python included to show language-agnostic behavior (arxiv:2605.26100).

A representative pipeline shape, per the paper:

Patch (Java + Python files, 12 hunks)
  -> Labeler (per-hunk, 5 lines local context, few-shot)
     hunk[1]: [Rename]
     hunk[2]: [Rename]      <- usage consequence
     hunk[3]: [Logic Change]
     ...
  -> Refiner (whole-patch, single inference)
     hunk[1]: Rename(parent=0, old="getUserId", new="getAccountId")
     hunk[2]: Rename(parent=1)
     hunk[3]: Logic Change

Best result on this benchmark: 84% recall, 81% precision with Gemini-3-Pro-Preview. Relative model rankings shift by label type, so the headline number is not the only signal to plan against (arxiv:2605.26100).

Key Takeaways

  • Skipping the Refiner stage leaves per-hunk types with no cross-hunk relationships. Run it whenever a downstream consumer needs to know which hunks are linked, not just what each one is.
  • Polyglot coverage and label customizability are the legitimate wins; determinism, cost, and per-category precision are the legitimate trade-offs.
  • Default to a hybrid rather than pure LLM labeling unless every label type in your taxonomy has been checked at an accuracy your workflow can tolerate — the paper's own recommendation, not a hedge.
  • Weigh cost against accuracy per model, not just headline recall. The paper's top scorer (Gemini-3-Pro-Preview) used up to 7.5× the output tokens of the runner-up (Claude Sonnet 4.5).
  • Spot-check external interface, error handling, and log labels by hand. Those are the categories furthest from the paper's headline accuracy numbers.
Emerging — single recent source Last reviewed: 2026-06-13