How do you measure whether AI coding agents are actually working?
6 min read
›Contents
The short answer: measure delivery before and after, not activity. Freeze a baseline of the four DORA measures and pull request cycle time before agents change anything. Then read the measures that only exist once agents write code (how much of the work agents do, what each merged change costs, whether agent work comes back more often) against that baseline, under rules that do not change. Seats, suggestions accepted and lines generated tell you whether people are using the tools. They do not tell you whether the team got faster or whether the work is right.
Why usage is not the answer#
Most of the first numbers a team collects about coding agents are usage numbers: how many engineers have a seat, how often they accept a suggestion, how much code was generated. They are easy to get, and they look like progress.
They can also move while delivery does not. A team can generate more code and ship no faster if review is where work waits. It can merge more pull requests and see more of them reverted. Usage is the input; the question is what came out.
That is why the measures worth reading are the ones on the delivery side of the line: how often you ship, how long a change takes to reach production, how often a deployment fails, how quickly you recover, and where a pull request waits between being opened and being merged.
Start with a before that cannot move#
An improvement is only a claim if the starting point cannot be quietly recomputed. Three habits keep it honest:
- Read history back. When you connect your systems, read the ninety days before, so a before exists on the first day and not a quarter later.
- Freeze it once. Fix the baseline when every system has been read, and let nobody recompute it afterwards.
- Name the rules. Compute every value under a named version of the rules, and read an after under the same version as the before it is compared with. If the rules change, measure the original window again under the new ones and keep the first reading beside it.
This is how the Kelright Benchmark works, and it is the question to put to any tool that reports a trend: what exactly is the before, and can it move?
Read the four delivery measures#
The four DORA measures describe delivery whoever writes the code. DORA, the DevOps Research and Assessment program, publishes its own guide to them. Its current guidance names five, adding deployment rework rate and redefining recovery time as failed deployment recovery time; the four below are the earlier set, which is what the Benchmark's Delivery family reads.
- Deployment frequency is how often you deploy to production. Computed as successful production deployments per day over a window, higher is better.
- Lead time for changes is how long a change takes to reach production. Computed as the median time from a commit being authored to its first successful production deployment, lower is better.
- Change failure rate is the share of deployments that fail. Computed as the share of successful production deployments that were rolled back or linked to an incident, lower is better.
- Time to restore is how long recovery takes. Computed as the median time from an incident being opened to its being resolved, lower is better.
Add pull request cycle time beside them, and split it into time to first review, review time and merge time. When agents write more of the code, the wait can move to review, and the split shows you where.
Add the measures that only exist once agents write code#
The delivery measures cannot say how much of the work agents do, what it costs, or whether it holds. Three more families can:
- Adoption. The share of merged pull requests credited to an agent, the share of commits, and the share of issues delegated to an agent.
- Economics. AI spend per merged pull request, spend per delivered issue, and cycle time and rework for agent-assisted pull requests against the rest.
- Quality. The CI pass rate, the share of agent-written pull requests that changed a test file, and the rework rate and revert rate.
The comparison that matters is within your own team: agent-assisted work against other work, under the same rules, over the same window. If agent-assisted pull requests merge faster and come back no more often, agents are helping. If they merge faster and come back more often, you have moved the cost into rework.
Be exact about what counts as agent work#
Every one of those numbers depends on deciding which work an agent did. The credit can come from a co-author trailer on a commit, a declared agent account, a label on a pull request, or a join to the agent tool's own telemetry. A tool that does not say how it credits work gives you figures you cannot check, and a change in its method will change your trend without anything changing on your team.
So ask two things of any measurement: how is the work attributed, and is the method versioned like every other rule?
Read for the right things, not only the fast ones#
Speed is half of working well. The other half is working on the right things, and that shows in a handful of places:
- Planned work, finished. The share of the work planned into a cycle that was done by its end. A team that ships quickly and never finishes what it planned is busy, not well directed.
- Work that stays done. Issues reopened after they were called finished, and changes reverted after they merged.
- Where work waits. How long an issue spends in each state, and how old the work in progress is.
- Spend that lands. What agent work costs for each piece of work that reaches done, not for each line written.
What the right thing is, your company decides. That is the part a measurement tool cannot see, and the reason Kelright pairs the Benchmark with a record of what the company decided.
What a good reading looks like#
You are looking for a pattern, not a number. Lead time and cycle time falling, the agent share rising, spend per merged pull request steady or falling, and rework and reverts not rising: that is agents helping. The same rise in agent share with rework climbing, or review time growing faster than cycle time falls, is output outrunning the checks, and the answer is to tighten the gates before handing agents more.
How to start#
- Pick the systems that hold the truth: your forge for pull requests and deployments, your tracker for the work, and your telemetry vendor for what the agents cost.
- Freeze a baseline before you change anything else.
- Read the same measures every week under the same rules.
If you want to know where your team stands first, the Agentic Readiness Check takes about six minutes and needs no account. If you want the reading done for you, the Benchmark reads these measures from your own systems and lands them in your own Datadog.