AI Is Getting Cheaper. But Are We Measuring What It Actually Delivers?

A four-layer framework for measuring the true economics of AI — from token consumption to business value.

We are getting remarkably good at measuring what AI costs. Tokens, API spending, latency, infrastructure. We compare models by price, optimize prompts to consume fewer tokens, and celebrate when a task that previously cost a dollar now costs a few cents.

Those improvements matter. But I think we’re focusing too much on the wrong end of the equation.

Knowing what AI costs tells us very little about what AI actually delivers.

A workflow that costs $0.10 but fails nine times in ten is more expensive than one that costs $0.50 and reliably finishes. An AI coding agent that writes 5,000 lines may deliver less value than one that removes 200. And a perfect automation of a problem nobody has is still a poor investment.

As AI moves from answering questions to performing real work, the economics become more interesting — and considerably harder to measure.

The industry is beginning to recognize this. The FinOps Foundation is connecting token consumption to economic outcomes, while McKinsey and Google Cloud have published frameworks for measuring the business impact of AI. Yet there’s still a practical question for those of us building and operating these systems: how do we connect what an AI consumes to what it actually achieves?

I find it useful to separate four questions that are often mixed together.

The four layers of AI economics

Layer	Dimension	The question
L1	Consumption	What did the AI cost?
L2	Output	What did it produce for that cost?
L3	Delivery	Did it successfully complete the work?
L4	Value	Was the completed work worth it?

These are not four competing metrics. They are four different perspectives on the same system. And that’s important, because improving one layer doesn’t necessarily improve the next.

L1 Consumption: What did it cost?

This is what most AI platforms already track: tokens consumed, inference costs, execution time, tooling, infrastructure, and sometimes human supervision.

Say an AI coding agent consumes two million tokens, takes 15 minutes, and costs $4 in API charges. Was that efficient? We can’t tell.

Perhaps it implemented a substantial feature, wrote tests, and prepared everything for release. Or perhaps it changed three lines of configuration after several unsuccessful attempts. Same spending. Very different economics.

Even the $4 figure may be incomplete. The total cost of running AI includes more than inference: infrastructure, tools, retries, human supervision, integration, and maintenance can all contribute.

L1 is essential for operational control. Without it, we can’t manage budgets or understand what resources our systems consume. But reducing consumption isn’t the same as improving efficiency.

Consumption tells us how much we spent, not whether the spending was productive.

L2 Output: What did we get for the money?

The next step is to normalize cost against a measurable unit of output. Instead of asking only what a task cost, we ask: how much did it cost per unit of work produced?

For AI coding agents, one possible metric is cost per 1,000 real changed lines of code (Cost/KLOC). Consider two hypothetical tasks:

Metric	Task A	Task B
AI cost	$20	$20
Real changed code lines	1,000	100
Cost per 1,000 lines	$20	$200

Fig. 1 — Same spend, tenfold difference in cost per unit of changed code.

The spending is identical, but the cost per unit of changed code differs by a factor of ten. That difference is interesting. It doesn’t mean Task A is better. A 100-line security fix may require far more reasoning and deliver far more value than 1,000 lines of routine implementation. But the normalization adds information we didn’t have before.

And defining the denominator carefully matters. A Git diff can contain documentation, comments, blank lines, dependency lockfiles, generated artifacts, and other changes that aren’t actual source code. Counting all of those as productive engineering output would distort the measurement.

A more meaningful code-change metric can count actual added and removed source-code lines, distinguish product code from test code, and exclude non-code artifacts.

Why count additions and removals rather than net additions? Because removing unnecessary code is genuine engineering work. A metric that treats a substantial refactor as zero output would be misleading.

But there’s an important limitation here.

More code is not necessarily better code. And a lower cost per line is not necessarily greater productivity.

If two agents solve the same problem, and one needs 2,000 lines while the other needs 200, the smaller implementation might be considerably easier to maintain.

Cost/KLOC is therefore best treated as a diagnostic: useful for comparing similar tasks, identifying unusual spending patterns, and understanding code-change intensity. Not as a target to optimize independently.

This is consistent with the broader findings of DORA’s 2025 research into AI-assisted software development, which describes AI as an amplifier of an organization’s existing strengths and weaknesses. More activity alone is not proof of better software delivery.

The same principle applies outside engineering. We might measure cost per processed document, generated report, extracted record, or customer-support response. But producing something is only the first step.

Output tells us how economically AI generates work, not whether that work is correct or useful.

L3 Delivery: Did it successfully complete the work?

This is where the economics become particularly interesting. Imagine two AI agents handling comparable tasks. Agent A costs $0.20 per attempt and succeeds 40% of the time. Agent B costs $0.40 per attempt and succeeds 95% of the time.

On price per attempt, Agent A is half the price. But assume failures are retried until success, with independent attempts, constant costs, and unchanged success probabilities. The expected cost per successful task becomes:

	Agent A	Agent B
Cost per attempt	$0.20	$0.40
Success probability	40%	95%
Expected cost per success	$0.50	$0.42

Fig. 2 — The pricier agent is cheaper per successful outcome.

The more expensive agent is actually cheaper at delivering successful outcomes. That’s the difference between output economics and delivery economics. At L3, we stop measuring what the AI generated and start measuring what it successfully accomplished.

Useful metrics include cost per accepted task, first-attempt success rate, retry count, verification pass rate, rework, human intervention, and end-to-end delivery time.

And failed attempts must remain in the calculation. If an agent spends $15 on unsuccessful attempts before producing a successful $5 result, the cost of delivering that result was $20. Not $5.

We should also distinguish AI execution time from human waiting time. A task that waits over a weekend for human approval may have a long end-to-end duration without requiring much additional AI execution. Both measurements matter. They simply answer different questions.

This distinction is increasingly important as agents become more autonomous. Anthropic’s January 2026 guidance on agent evaluations makes a similar point: reliable agent assessment requires explicit success criteria and verification of outcomes, not just inspection of generated responses.

And there is empirical evidence that perceived AI efficiency can differ from actual results. In a 2025 randomized controlled study, METR examined experienced open-source developers working with early-2025 AI tools. Developers expected AI to make them faster and believed afterward that it had done so. Yet the study found that allowing those AI tools increased task completion time by 19% in the tested setting.

That doesn’t mean AI generally slows developers down. The study involved specific tools, developers, and tasks at a particular point in time. But it demonstrates something important.

Perceived productivity is not the same as measured productivity.

If we want to know whether AI makes work cheaper or faster, we need to measure the completed work — including the mistakes, retries, verification, and corrections along the way.

Delivery tells us what accepted outcomes actually cost, not just what generated output costs.

L4 Value: Was it worth doing?

Now imagine an AI agent that implements a feature perfectly. The code is clean, tests pass, verification succeeds, and the entire implementation costs $10. Excellent engineering economics.

But what if nobody uses the feature? What if it saves no time, reduces no costs, solves no meaningful problem, and creates no value? The delivery was efficient. The investment may still have been pointless.

This is where L4 begins.

L4 measures the effect of completed AI work outside the AI workflow itself.

That might mean human hours saved, operational costs reduced, revenue generated, incidents prevented, risks reduced, or new capabilities enabled.

But there’s an important step between delivery and value that is easy to overlook: adoption. A correctly implemented solution doesn’t create value simply because it exists. People or systems generally have to use it, and that usage must produce a meaningful improvement.

This is why McKinsey’s 2026 AI measurement framework explicitly includes adoption among its measurement dimensions, alongside technical performance, operational outcomes, and financial impact.

Consider an automation that costs $100 to build and saves ten hours of manual work every month. At an estimated economic value of $40 per hour, that suggests $400 in monthly benefits, or $4,800 annually.

But does it really? Saving ten hours doesn’t automatically save $400. Those hours might create additional capacity without reducing payroll expenses. Employees might spend the time on more valuable work. Or the claimed savings might never materialize because nobody actually uses the automation. And the system itself may require ongoing infrastructure, supervision, and maintenance.

We need to distinguish three things:

• Expected value. What we believed the feature would achieve.
• Observed value. What actually changed after deployment.
• Attributable value. How much of that change can reasonably be credited to the feature.

Only then does financial ROI become meaningful. The formula itself is simple:

ROI = (Realized benefits − Total costs) / Total costs × 100%

Measuring and attributing the benefits is the difficult part. It often requires observing real-world outcomes over weeks or months, comparing them against a meaningful baseline, and accounting for unrelated changes.

Google Cloud’s 2025 guidance on AI value measurement recommends starting with clearly defined business outcomes and including the total cost of ownership before calculating ROI. That is a useful discipline: decide what success means before deploying the AI, rather than trying to invent a benefit afterward.

And not everything valuable can be expressed credibly in dollars. A security improvement, a more reliable system, or a better user experience may create substantial long-term value without an immediately measurable financial return. We should track those outcomes rather than invent monetary precision.

Value tells us whether the work mattered, not merely whether it was completed efficiently.

The layers can pull in different directions

This is why I think separating the four layers matters.

A more capable model might cost more per token but require fewer retries, reducing the cost of successful delivery. Stricter verification might increase task costs while reducing defects and operational risk. A smaller code change might look less efficient per KLOC but produce a better, more maintainable solution. And an expensive automation might generate enough long-term value to justify its cost many times over.

Optimizing one layer in isolation can therefore be misleading.

The cheapest AI isn’t necessarily the most efficient. The most productive AI isn’t necessarily the most reliable. And the most reliable AI isn’t necessarily working on the most valuable problems.

These distinctions are not entirely new. FinOps, software engineering, and business-performance measurement have long dealt with questions about resource consumption, output, outcomes, and value. What changes with autonomous AI is the speed, scale, and variability at which those questions arise.

When an agent can attempt hundreds of tasks, execute tools, recover from failures, and deliver artifacts without constant supervision, understanding the economics of the entire workflow becomes increasingly important.

How should we actually measure this?

I would start with four practical steps.

01 Record the full execution cost.

Track model spending, tools, infrastructure, execution time, and relevant human effort. Include failed attempts. Distinguish direct AI costs from the full cost of ownership.

02 Define a meaningful unit of output.

Choose a denominator appropriate to the task and explain what counts. For coding agents, changed source-code lines may be one useful diagnostic. For document processing, correctly extracted records might be more appropriate. Be explicit about limitations. Never assume that maximizing output volume is the goal.

03 Define successful delivery before execution.

What does success actually look like? A generated answer isn’t necessarily a resolved support case. A pull request isn’t necessarily a working feature. A completed workflow isn’t necessarily a correct workflow. Establish acceptance criteria and measure the cost of meeting them, including retries, corrections, and verification.

04 Measure real-world impact after delivery.

Define the expected benefit before starting. Measure adoption and actual outcomes afterward. Compare against a meaningful baseline, and distinguish projected savings from observed and attributable benefits.

Not every workflow needs a complex financial model. Sometimes a measurable reduction in error rates or manual handling time is a useful starting point. The important thing is to connect AI activity to consequences.

A dashboard that tells a story

Imagine opening an AI dashboard and seeing:

AI spending this month: $1,200

Is that good or bad? There’s no way to tell. Now imagine seeing four different perspectives on that same spending:

Layer	What the dashboard shows
L1 Consumption	$1,200 spent
L2 Output	400 work items generated
L3 Delivery	320 accepted, at $3.75 per accepted item
L4 Value	Approximately 150 hours of monthly manual work avoided; economic benefit still being assessed

Illustrative, hypothetical figures.

These are hypothetical figures, but they illustrate the difference. The discussion is no longer about whether $1,200 is too much. It’s about whether the outcomes justify the spending, whether delivery efficiency is improving, whether people are using the results, and how much value is actually being created.

That’s a far more useful conversation.

The next phase of AI optimization

AI models will continue to become faster, more capable, and cheaper to operate. But cheaper inference doesn’t guarantee cheaper outcomes. And cheaper outcomes don’t guarantee greater value.

We need to move from measuring token consumption to understanding output, from output to successful delivery, and from delivery to real-world impact. Four simple questions provide a useful way to think about this:

• L1 What did we spend?
• L2 What did we produce?
• L3 What did we successfully deliver?
• L4 What value did it create?

We shouldn’t stop measuring tokens, latency, or API spending. Those are essential operational metrics. But we should stop treating them as the final measure of AI efficiency.

Nobody invests in AI just to consume fewer tokens. We invest in AI to accomplish something.

The future of AI economics isn’t just making intelligence cheaper. It’s understanding what that intelligence delivers — and whether the result is worth the cost.

References and further reading

1. McKinsey (2026) — From promise to impact: How companies can measure—and realize—the full value of AI. A layered framework connecting AI adoption, operational performance, and business impact.
2. FinOps Foundation (2026) — Token Economics: The Atomic Unit of AI Value. Connects AI resource consumption and unit economics to organizational value.
3. Anthropic (2026) — Demystifying Evals for AI Agents. Practical approaches to evaluating whether AI agents successfully complete tasks.
4. METR (2025) — Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. A randomized controlled study of AI-assisted software development and measured task completion time.
5. Google Cloud (2025) — Measure the Value and Impact of Your AI. A three-part methodology for defining business outcomes, costs, and AI ROI.
6. DORA (2025) — State of AI-assisted Software Development. Research into how AI interacts with software delivery performance and organizational capabilities.