AI adoption among developers surged from 76% to 84% between surveys, with major tech companies generating significant portions of their codebases through AI tools. However, traditional engineering metrics—velocity, lines shipped, and pull requests merged—haven’t evolved accordingly. These indicators now reveal only half the story, masking quality risks that lack established names.

The Phantom KPI: Code Knowledge

A widely discussed but unmeasured metric tracks “the percentage of the codebase a team understands.” Rather than existing as a standardized KPI, the underlying phenomenon is real: codebase familiarity erosion.

When developers review and approve generated code rather than authoring it, they gain speed but lose comprehension. Over months, even experienced engineers become less expert in their own systems. Indirect measurements include:

Review depth
Incident diagnosis capability without AI assistance
Number of team members who can explain critical modules from memory

A critical concern emerges when single individuals or agents become the sole experts in system components—the modern “bus factor” problem.

Perception vs. Reality: The Data Gap

METR’s July 2025 randomized controlled trial examined 16 experienced developers completing 246 tasks on mature repositories:

Time estimates before tasks: “24% faster” with AI
Perceived speed after tasks: “20% faster”
Actual measurement: “19% slower”

A February 2026 follow-up found control groups impossible to establish—developers refused working without AI—suggesting tools improved, though measurement uncertainty remains high.

The lesson: Never trust productivity intuition; always measure objectively.

The KPIs That Actually Emerged

Key distinctions commonly confused:

Adoption vs. Code Share: Adoption measures how many developers use AI; code share reveals how much code originates from it. High adoption doesn’t indicate high AI contribution.
Velocity vs. Delivered Throughput: More merged pull requests don’t necessarily translate to faster production deployments if review times extend dramatically.

The Vanity Metric Trap

“The percentage of AI-generated code, on its own, means nothing.” This metric rises easily and looks impressive in presentations but drives no actionable decisions. 30% AI-generated code is positive with stable churn; it becomes problematic when accompanied by rising churn, duplication, and defect density.

The golden rule: Pair every speed metric with a corresponding quality metric—never use one independently.

Measuring Without Self-Deception

Three implementation filters:

Pair metrics: Link cycle time with change failure rate; align code share with churn rates.
Tie to outcomes: Eliminate KPIs from dashboards if improvements don’t drive decisions or deliver value (applying Goodhart’s Law principle).
Protect knowledge: Treat codebase familiarity as a strategic asset. Rotate code reviewers, maintain documentation, ensure multiple humans can explain critical modules without agent assistance.

Takeaway

AI didn’t fundamentally change measurement approaches—it made traditional metrics dangerous by emphasizing speed exclusively. Mature teams in 2026 are measured not by shipping velocity but by understanding retained about shipped systems. Uncomprehended code represents invisible technical debt.

References

METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (July 2025). arXiv 2507.09089
METR, We are Changing our Developer Productivity Experiment Design (February 24, 2026)

The new KPIs of AI-era engineering

The Blind Spot of the AI Era