The Hidden Cost of AI Tokens

On paper, token pricing is the most honest billing model in software. There is a published price per million input tokens and per million output tokens. No mystery, no sales call. In practice, it is one of the slipperiest costs a business can take on, because almost nothing about your actual usage is visible to you at the moment you incur it.

A unit you cannot feel

A token is roughly three-quarters of a word, except when it is a punctuation mark, or part of a word, or a single character in a language the tokeniser handles badly. Nobody thinks in tokens. You think in questions and answers and documents. The billing unit and the human unit have no intuitive conversion, which means you cannot glance at a task and estimate its cost the way you can glance at a taxi meter and estimate a fare.

This is not an accident of physics. It is a genuinely reasonable way to price the underlying compute. But it does mean the customer is billed in a currency they cannot mentally convert, and that gap is where the surprises live.

The demo that scales into a shock

The place token costs bite hardest is the gap between a prototype and production. A demo makes a handful of calls and costs pennies, which is precisely why it is so easy to green-light. Then the thing ships, real users arrive, and each of them is having long conversations, uploading documents, triggering retrieval, and prompting the expensive reasoning models — and the bill scales not with your revenue but with your usage, which are not the same curve. Plenty of AI features are technically delightful and quietly unprofitable, costing more to serve than the customer is paying, a fact that only becomes visible once the invoice arrives with a comma in it.

This is the structural trap of building on someone else's metered model: your marginal cost is real, it is variable, and it is set by a supplier who can revise it. A traditional software feature, once built, costs almost nothing to serve to one more user. An AI feature costs money every single time it runs, forever. That changes the economics of “just add AI to it” from a free enhancement into an ongoing liability — one that a great many products bolted on during the enthusiasm of the last two years without ever modelling what it would cost at the scale they were hoping for.

The costs you did not authorise

The headline price is for the tokens you send and receive. Your real bill includes several things that are easy to forget you are paying for:

• The system prompt. Every request silently re-sends the hidden instructions that shape the model's behaviour. You pay for those tokens on every single call, forever.
• The conversation history. Chat interfaces feel like the model “remembers” the discussion. It does not. The entire history is re-sent with each new message, so a long conversation gets more expensive with every turn — you are paying to remind it what it just said.
• Retrieved context. The moment you bolt on document search, every relevant chunk gets stuffed into the prompt. Useful, and quietly the largest line on many bills.
• Reasoning tokens. The newer “thinking” models generate long internal working before answering. You are billed for that working, even though you frequently never see it.

None of this is hidden in the sense of being concealed. It is hidden in the sense that a first-time user has no way of anticipating it, and the pricing page does not volunteer it.

Output is the expensive end, and output is the least predictable

Output tokens typically cost several times more than input tokens, which is rational — generation is the hard part. But output length is the thing you control least. Ask a concise question and the model may still return six paragraphs, three of which are throat-clearing. A verbose model is not just annoying; it is a variable multiplier on your most expensive line item. “Be concise” in your prompt is, among other things, a cost-control instruction.

The pricing changes, and so does the model beneath the name

Two more things move under your feet. First, list prices are revised — sometimes down, which is lovely, sometimes with a new premium tier that quietly becomes the one you actually need. Second, the model behind a given name can be updated, and a “more efficient” successor that is cheaper per token can still cost you more per task if it happens to be chattier. Per-token cheaper is not per-job cheaper, and only the second one shows up on your invoice.

How to stop being surprised

The unit is not just opaque; it is misaligned

There is a deeper problem than mere opacity. The token is misaligned with the thing you actually value. You care about a task being done — a question answered, a document drafted, a bug found. The meter counts tokens, and tokens correlate only loosely with task value. A short, precise, genuinely useful answer might cost a fraction of a long, waffling, useless one. You can pay more for a worse outcome simply because the model was chattier, and nothing in the billing distinguishes the two. The meter rewards volume; you wanted usefulness; these are not the same quantity, and only one of them appears on the invoice.

This misalignment quietly shapes the products built on top. A vendor billed per token has no incentive to make the model concise, and a subtle one to let it ramble. A “thinking” model that generates pages of internal reasoning bills you for every line of it, whether or not the extra reasoning improved the answer. You are paying for effort, not results, in a domain where effort and results have come uncoupled — which is precisely the arrangement you would design if you wanted revenue to grow faster than value delivered.

Where the surprise bills actually come from

In practice, the token bills that shock people almost never come from the obvious place — a human sending a few messages. They come from automation. A background job that summarises every incoming email. A feature that re-embeds an entire document library on each update. A retry loop that quietly re-runs a failed expensive call. An agent that calls itself in a loop, each step re-sending the full accumulated context. These do not feel like usage because no person is sitting there typing, yet they generate tokens by the million, invisibly, until the invoice arrives with a figure nobody can immediately explain.

That is the real hazard of a meter only the vendor can read: the costs that scale are the ones no human is watching in real time. A person notices a slow, expensive session and stops. A cron job does not get bored, does not get tired, and does not check the balance. The organisations that get burned are rarely the ones with heavy human users; they are the ones that wired a token-metered model into an automated pipeline and did not instrument it, because the pipeline felt free the way electricity feels free right up until the bill.

The abstraction that hides the cost from the people spending it

The token model has one more quiet hazard: it separates the people who incur the cost from the people who see it. A developer writes a prompt that stuffs a whole document into context on every call; the cost of that decision lands weeks later on a finance dashboard the developer never looks at. A product manager approves a feature that re-summarises everything hourly; the bill arrives in an account owned by someone else. The person making the token-generating decision and the person feeling the token-generating pain are frequently different people, and the gap between them is where waste accumulates unchecked.

Good utilities close that gap by putting the meter where the decision is made — a thermostat in the room, a data counter on the phone. Token costs, by default, do the opposite: they pool the consequences far downstream of the choices, so no single decision ever feels expensive in the moment it is made. The fix is organisational as much as technical: make the cost visible to the people who generate it, in something close to real time, so that “just add more context” and “run it on everything” carry a number the decision-maker can actually see. A meter only the vendor can read is bad enough; a meter that even your own team cannot feel is how a genuinely cheap technology quietly becomes an expensive one.

The fix is boring and effective: instrument before you scale. Log token counts per request from day one, set hard budget alerts at the account level, cap output length wherever you can, trim system prompts and conversation history aggressively, and cache anything you send repeatedly. The token model can be genuinely cheap — often dramatically cheaper than the flat subscriptions we have complained about elsewhere — but only for the customer who measures it. For everyone else, it is a meter running in a language they never learned to read.