Skip to content

Human-Equivalent Hours for Autonomous Coding Agent Productivity

Estimate the human engineering hours an autonomous agent's output would have taken — credible only on PR-gated sessions with a paired downstream signal.

When the metric is credible

Human-equivalent hours estimates counterfactual human time, not measured output. It is credible under specific conditions and misleading outside them. Apply it only when:

  • Sessions terminate in a merged PR or pass an independent quality classifier. Cognition's calibration includes a PR-merged session only if any of its PRs merge; non-PR sessions go through a classifier that drops 1–20% as unproductive (Cognition, 2026-06-04).
  • The aggregate covers enough sessions to escape the noise floor. Their held-out r_log = 0.74 places ~50% of estimates within a factor of 2; below ~20 sessions, month-over-month deltas sit inside that band (Cognition, 2026-06-04).
  • A second, observed signal trends alongside it: PR review time, defect rate, or merge-to-revert ratio. Without one, the metric is unanchored — see when this backfires.

Outside these conditions, prefer cost per merged PR alongside review-time-to-merge — both are observed, not estimated.

The definition

Cognition asks "how long would a human engineer have taken to produce the same output?" Hours already denominate salaries and contractor rates, so the result is directly comparable to existing finance and headcount instruments (Cognition, 2026-06-04).

The estimator rests on four design principles:

  1. Reason about the human's path, not the agent's — discount retries, setup, and non-core artifacts a human would not produce.
  2. Credit only unspecified work — measure the agent's contribution against the user's initial problem statement, not the full diff.
  3. Account for codebase familiarity — infer a human's exploration time in an unfamiliar codebase.
  4. Assume relevant expertise — the reference engineer already has the skills; do not credit skill-substitution.

The uncalibrated model is corrected via log-space linear regression:

h = 2.28 × m^0.923

where m is the uncalibrated estimate and h the corrected human-hours figure (a simplified 2.08× constant performs comparably). Calibration: 258 sessions, 126 users; held-out r_log = 0.74 on 233 sessions; F(1,231) = 279.9, p < 10⁻⁵ (Cognition, 2026-06-04).

Why code volume is not the metric

Regressing lines changed against human-time estimates produces R²_log = 0.27, so code volume captures roughly a quarter of the variance in productive output (Cognition, 2026-06-04). That is the empirical case against task-completion-rate and PR-count metrics: they correlate weakly with the value the team pays for. Under bottleneck migration the cheap part — generation — is exactly what those metrics count.

Why it works

Engineering value is already denominated in human time — salaries, contractor rates, and estimates all use hours. Converting agent output back into hours makes ROI directly comparable to the instruments finance and headcount planning already run (Cognition, 2026-06-04). The mechanism is denominator alignment, not ground-truth measurement: it speaks the language of the decisions it informs (renew the seat, raise the cap, hire instead).

Google frames the same measurement challenge for Jules, its agentic coding tool: as agents shift from reactive assistants to proactive contributors, tracking their impact requires rethinking the unit of value (Google Developers Blog — Measuring what matters with Jules). This is a second vendor perspective that the denominator-alignment approach scales across tool boundaries.

The denominator is urgent now. Agentic workloads carry 58.9% of token volume on Vercel's AI Gateway, up from 31.6% six months earlier. Tool-using requests are ~2.6× more token-heavy than the rest (Vercel AI Gateway production index, 2026-05-12). Uber capped employees at $1,500/month per agentic coding tool after burning the annual AI budget in four months (TechCrunch, 2026-06-02). Token spend has a denominator; agent output, until now, did not.

When this backfires

The metric estimates counterfactual human time. Every failure mode below traces back to that one property.

  • High-context maintenance on familiar codebases. A randomized controlled trial of experienced open-source developers measured a 19% slowdown with AI tools while developers still reported a 20% speedup — a 39-point perception gap (METR, 2025-07-10). Cognition's model is calibrated against user reports and its corrected estimates still sit 1.4× below those reports (Cognition, 2026-06-04) — consistent with self-report inflation, not independent of it. Pair with an observed downstream signal; the productivity-experience paradox is the warning that perception and reality diverge here.
  • Downstream cost can absorb the gain. AI-assisted teams complete 21% more tasks and merge 98% more PRs while PR review time rises 91% — the bottleneck migrates (Osmani, 2025). An hours-saved figure that ignores review time spent is half a ledger.
  • Task selection bias inflates apparent value. Agents get the tasks they are best at; the reference human is then estimated for tasks pre-selected to favor the agent. Compare baselines on stratified task mixes, not aggregate counts.
  • Small-team noise floor. At r_log = 0.74, ~50% of per-session estimates fall within a factor of 2. A 10-session month sits inside that band; reading a 30% month-over-month change as signal is reading noise.
  • Greenfield work has no stable reference. "How long would a human have taken?" assumes a stable counterfactual. For novel problems with no comparable human baseline, the denominator is fabricated and the hours figure is no more grounded than an opinion.

Example

A platform team runs Devin and Claude Code across two months, defending (or canceling) a $1,500/seat agentic-coding budget.

Before — counting completions:

Month 1: 47 PRs merged, 12,400 lines changed, $9,200 spend
Month 2: 51 PRs merged, 11,800 lines changed, $11,400 spend

Lines changed and PR counts both rise; spend rises faster; the conversation stalls on whether 47 PRs are "worth" $9,200.

After — denominating in human-equivalent hours, with downstream signals:

Month 1: 47 PRs merged → 184 estimated human-hours
         PR review time: 38h spent; defect-rate flat
         Implied rate: $9,200 / 184h = $50/h
Month 2: 51 PRs merged → 201 estimated human-hours
         PR review time: 61h spent; defect-rate flat
         Implied rate: $11,400 / 201h = $57/h

The estimate is calibrated on PR-merged sessions only (Cognition's gate). The implied $/h is now directly comparable to the team's loaded hourly rate. The 61h of review time is the observed signal that anchors the estimate. If review time were rising faster than agent hours, the gain is being absorbed downstream and the metric must say so.

Key Takeaways

  • The metric: how long would a human engineer have taken to produce the same output — chosen because engineering value is already denominated in hours (Cognition, 2026-06-04).
  • Cognition's calibration: h = 2.28 × m^0.923 (or a simplified 2.08× constant) fit on 258 sessions across 126 users; r_log = 0.74 on held-out data; ~50% within a factor of 2.
  • Code volume is empirically rejected as the metric — lines-changed regression returns R²_log = 0.27 against human estimates.
  • Anti-gaming: include sessions only if a PR merges; for non-PR sessions, run an unproductive-session classifier (drops 1–20%).
  • The signal inherits self-report inflation. Pair with an observed downstream signal (PR review time, defect rate) — METR's RCT measured a 19% slowdown while developers perceived a 20% speedup (METR, 2025-07-10).
  • Apply only on PR-gated, multi-month aggregates with a paired downstream signal; small samples and greenfield work sit inside the noise floor.
Adopted — practitioner-sourced, corroborated Last reviewed: 2026-06-30