Kelright
← All writing

How do you measure what AI coding agents cost per change?

3 min read

›Contents

The short answer: divide what agent work cost in a window by the pull requests that merged in it. That is AI spend per merged pull request: what agent work costs for each change that ships. Read it beside cycle time and rework for agent-assisted work, because a low number is only good if the work does not come back.

Why per merged pull request#

There are easier numbers. Each tells you something, and each can mislead:

  • Spend per seat tells you what you are paying for access. It says nothing about what the access produced.
  • Tokens, or cost, per line written rewards the wrong thing. Lines are cheap to generate and expensive to review, and a larger change is not a better one.
  • Total spend rises with adoption. It cannot tell a team that is getting value from a team that is simply using more.

A merged pull request is a unit of work that passed review and landed. Pricing agent work per merged change counts only spend that lands: work that reached done, not work that was started, abandoned or thrown away.

How the Benchmark computes it#

The Kelright Benchmark computes agent cost per merged pull request as the agent cost in a window over the pull requests merged in it, with lower being better. Three companions sit beside it:

  • Agent cost per delivered issue: agent cost in the window over the issues completed in it.
  • Agent tokens per thousand lines changed: input plus output tokens in the window per thousand lines merged in it.
  • Agent cost per active engineer day: agent cost over the person-days of agent use. It is read as neutral, because more use is not better or worse in itself.

The cost comes from where it already lives. Agent tools report their own usage and spend to your Datadog through their integrations; the Benchmark reads that back. Nothing an agent emits passes through a Kelright server.

Read it with the work, not alone#

Spend per merged pull request falling can mean two opposite things. Agents may be getting cheaper per change. Or agents may be doing smaller, easier work, or work that comes back for fixing. So read it beside:

  • Pull request cycle time, agent-assisted and not. If agent-assisted changes are merging faster at similar cost, agents are helping. If they merge slowly because they wait for review, the cost is in people's time.
  • Rework rate, agent-assisted and not. The rework rate is the share of pull requests merged in a window that were since reopened or reverted. If agent-assisted rework is higher, a low cost per change is borrowed against later fixes.
  • Pull request size. Median lines changed. A drop in cost per change that comes with a drop in change size is not a drop in cost.
  • Tests on agent-written pull requests. The share that changed a test file. Cheap changes that arrive without tests are cheap for a reason.

Pitfalls#

  • Windows. A short window with few merged pull requests makes the ratio jump. Read the 30-day and 90-day windows together.
  • Attribution. If you cannot say which pull requests an agent did, you cannot split the comparison. Ask how credit is given and whether the method is versioned. More on that.
  • A moving baseline. Spend per change means something only against a before that does not move. Why the baseline is frozen once.

Set it inside a budget#

Measuring cost is the second half. The first is deciding what a change may cost. In the Guild, the milestone that adopts the agentic workflow (M5) is accepted when the share of pull requests agents helped write, review time and AI spend per merged pull request sit inside targets agreed with you, and agents work inside a spend budget. The milestones, one by one.

Where to start#

Read ninety days back, freeze the baseline, and watch this number beside cycle time and rework every week. The full method. If you want it read from your own systems and landed in your own Datadog, the Benchmark does that.

Get the next essay

New writing, straight to your inbox. No spam, unsubscribe anytime.