The hidden economics of AI coding

An AI coding video caught me with a line I could not stop thinking about.

The presenter opened with a deliberately simple observation: in one observed workflow, it had taken roughly 8,000 tokens to say “hello” to an AI coding agent.

That number is easy to misread. It doesn’t mean a greeting inherently costs 8,000 tokens, and it isn’t a universal rule about coding agents. This particular agent had simply done a lot of work before the visible work began: it had gathered repository context, carried previous state forward, and processed tool output to get itself ready to operate inside a software project. By the time the interaction looked simple, the system had already spent thousands of tokens figuring out where it was.

What caught my attention wasn’t the price. It was how much the agent needed to know before a small interaction could make sense.

A few days later, I came across a very different story. Coinbase described reducing AI spend while token usage kept climbing. Rather than asking developers to use AI less, the engineering team worked on the infrastructure around their AI systems: routing, caching, cheaper defaults, spend services, logging, and cost controls.

Those two stories stuck with me. One was an agent burning thousands of tokens before producing anything useful; the other was an organization cutting costs without cutting usage. They looked unrelated, but they pointed at the same idea: AI coding costs are driven more by understanding than by code generation. The more useful question isn’t how many tokens a model consumes, but why it needed them in the first place. That’s where this guide starts.

Following the tokens in an AI coding workflow

So the agent has burned thousands of tokens before writing a meaningful line of code, which feels excessive. Then look at the task that follows.

The prompt is simple:

“Add support for exporting reports to PDF.”

If a junior developer joined your team this morning, you wouldn’t expect them to start typing right away. They’d browse the repository, read the existing implementation, look for similar features, and pick up the project’s conventions before touching anything. A coding agent does much the same: it explores the repository, works out which parts of the codebase are relevant, and builds up a picture of how the application fits together before it writes code.

Exactly how this happens depends on the agent and its tooling. Different systems explore, manage context, and revisit earlier work in different ways, so treat the workflow here as representative rather than universal.

Even then the work is rarely finished. The agent runs the test suite and some tests fail. It reads the failures, goes back to the code, and tries again. A failure might reveal an unexpected dependency, so it opens another file; a build log might expose a bad assumption and send it down a different path. Most coding sessions follow this same predictable loop.

What matters in Figure 1 isn’t the sequence of steps but what grows while the loop runs. Generated code accumulates slowly. The information the agent has to hold in mind grows much faster. By the time the feature is done, the agent may have opened dozens of files, waded through pages of compiler output, and read several rounds of test failures to produce one modest pull request.

That changed how I think about AI coding costs. Instead of asking “How much code did the model generate?” I started asking:

How much information did the model need to understand before it could generate that code?

Similar-sounding questions, very different ways of thinking about cost.

Why AI coding agents consume more context

Most of us met LLMs through chat interfaces. You asked a question, got an answer, maybe refined it with a follow-up. Each exchange was fairly self-contained.

Coding agents ask something else of these models. They aren’t answering isolated questions; they’re expected to finish a job: explore a repository, edit files, run the tests, chase down failures, and walk back earlier decisions before the work is done. That changes the economics of every interaction.

Think about the last time you chased a failing test through a large codebase. You probably didn’t solve it on the first try. You read the logs, ruled out a few possibilities, traced the problem through several files, and slowly built up a sense of what was going on. Each step narrowed the search.

Agents work much the same way, and the more we ask of them the more context they accumulate over a session. Tools handle that context in different ways (summarizing, truncating, reopening files, starting fresh), but the need underneath doesn’t change. To make the next useful decision, the model needs enough information about the system in front of it. The output may still be a single pull request, even though the work behind it has changed completely.

Figure 2 shows the flip. The visible output is the code change; the hidden work is all the reading, re-reading, and interpretation that has to happen first.

None of this is an argument against bigger context windows or more capable agents. Those capabilities are letting AI tackle problems that were impractical not long ago. The challenge isn’t to avoid context but to supply the right context at the right time, as efficiently as possible. Understanding costs money, and the job is to engineer it well rather than to minimize it away.

Context grows with every iteration

The hidden economics become easier to see when a session lasts longer than a single prompt, as Figure 3 illustrates.

In a quick exchange the context is small enough to ignore. In a real task the agent has to track what it has already inspected, which assumptions failed, what the test runner said, and what still needs checking. Some of that is useful and some of it is noise, and the model has to carry the whole working set either way.

This is why a small final patch can still hide a lot of AI work. The patch survives into the repository; the investigation around it doesn’t. The files the agent opened, the logs it read, the failed tests it puzzled through, and the attempts it threw away are mostly gone by the time the pull request is up for review.

That’s worth dwelling on. When the cost lands later, it’s tempting to pin it on the thing you can see, the code itself. But the code is usually just the last step in a longer process of building up enough context to act.

Tokens are one part of AI cost. Latency, retries, tool calls, cache behavior, and model choice all feed into the economics of a workflow too. Tokens are handy because they make part of the work measurable, which isn’t the same as measuring all of it.

How engineering teams reduce AI coding costs

Looking across public engineering write-ups, I expected to find advice about writing shorter prompts and trimming token counts. That mostly isn’t what teams were talking about.

The recurring moves were structural: caching prompts so repeated context isn’t reprocessed, routing requests to different models by task complexity, and building AI gateways to centralize policy, budgets, logging, cost controls, and observability. Caching cuts repeated work, routing improves cost efficiency, and gateways simplify operations. Three different problems, and at first they looked unrelated. What they share only became obvious when I stepped back: none of them changes what a model can do. They change how efficiently the system around the model uses it.

Several major LLM platforms now support prompt caching to cut repeated processing, latency, and cost. Anthropic’s prompt caching documentation is one of the clearest public explanations of how it works, and Amazon Bedrock describes prompt caching for supported models in similar terms.

Caching helps most when big chunks of the context stay the same between requests, and helps less when the context churns on every call. That caveat keeps the idea honest: caching isn’t magic, just an engineering tool that pays off when the workload has reusable structure. If you’re a solo developer or a small team, I’d reach for caching before anything else on this list. It’s the change that moves the bill most, and routing and gateways don’t earn their added complexity until you’re running past a single provider. Cutting repeated context almost always beats cutting generated output.

Model routing tackles a different problem: not every request needs the most capable or most expensive model. OpenRouter builds this in through its Auto Router and its broader unified model API, which can direct requests across models and providers. Routing has its own trade-offs, of course, around quality, latency, fallback behavior, and how much control developers give up. The aim is simple: send each task to a model that fits it.

AI gateways carry the same thinking past the individual request. The LiteLLM documentation describes a self-hosted gateway for managing LLM access across an organization, covering authentication, budgets, logging, caching, guardrails, and provider routing.

Taken together, these examples pushed me toward a different read: AI cost work is becoming less about individual prompts and more about engineering the systems around them.

The mental model behind AI coding costs

When I say “understanding” in this guide, I don’t mean consciousness or reasoning in any human sense. I mean the information the model needs to produce a useful next step, and that distinction matters. An agent isn’t just turning a prompt into code; it’s moving through a software system, gathering enough information to decide what to do next. The generated code is the visible end of the chain.

Key takeaway

AI coding costs are driven less by the code a model produces than by the information it must repeatedly understand before producing that code.

Figure 5 captures the mental model behind this guide:

Information -> Understanding -> Decision -> Generated code

If that’s right, a lot of familiar engineering questions turn into AI cost questions too:

• Is the repository easy to navigate?
• Does the agent have to rediscover the same thing every session?
• Are test failures clear enough to guide the next step?
• Do build logs bury the useful signal?
• Are we measuring where the work actually happens?

None of those are prompt-writing questions. They’re the same systems questions we’ve always asked. And it’s why the hidden economics of AI coding have much less to do with the line count of the final patch than most developers expect: the visible output is the part that lands in Git, but the cost lives in everything the model had to read, infer, retry, and carry forward to get there.

Reading an AI bill backwards

Working through these examples, I started reading AI bills a little differently. A monthly invoice tells you how much you spent, but not what happened.

Suppose two teams each spend $5,000 this month on the same model. Identical numbers. But one might be resending the same repository context on every call, retrying failed sessions, and routing everything to its priciest model, while the other caches common context, picks models by task, and actually measures where its time and tokens go. Same bill. Very different engineering behind it.

That’s the case for observability. It doesn’t lower costs on its own; it tells you where your optimization effort will actually pay off.

Figure 6 leads to a sharper set of questions:

• Which workflows keep reprocessing the same context?
• Which tasks burn the most input tokens per unit of useful output?
• Where are agents spending most of their time?
• Which workflows retry over and over before they succeed?
• Where would caching or routing pay off most?

They don’t replace ordinary cost reporting; they make it actionable. Once you understand how the system behaves, you can improve the system instead of just flinching at the monthly invoice. That’s an old engineering problem, and we’ve been solving that kind of thing for decades.

Of course, you have to measure a system before you can improve it. If you’re wondering how to pull usage and cost together across several AI providers, and why that’s harder than it looks, our guide on how to track AI costs across multiple providers walks through the approaches and their trade-offs. This guide is about why AI coding costs behave the way they do; that one is about how to measure them in production.

Looking at AI costs differently

When I first watched an agent burn thousands of tokens before writing anything meaningful, I figured I’d hit an interesting anomaly. Looking back, I’d just noticed something that’s getting easier to see: modern AI systems spend much of their time building an understanding of the problem before they answer it.

For a coding agent, that understanding comes from reading source files, examining compiler output, following execution traces, and working through test failures. None of it shows up in the final pull request, yet all of it shapes how good the result is.

That’s why the examples here feel connected. Caching cuts the cost of reusing context you’ve already paid for. Routing sends each task to a model that fits it. Gateways let a team apply those decisions consistently. Observability shows where the work, and the cost, actually happen. Different problems, one shared aim: helping models build and reuse what they need instead of reprocessing it over and over.

The goal isn’t to use fewer tokens. It’s lower cost per unit of engineering value. Once you start thinking about AI systems in terms of understanding rather than raw generation, caching, routing, gateways, and observability stop looking like separate tricks. They’re all ways of cutting wasted work without slowing developers down.

The bill tells you the total. The engineering starts when you work out how your system produced it.