Every major LLM API bills the same way: you pay per token in, and you pay per token out. That sounds simple, and mechanically it is. What surprises teams is where the tokens actually come from — conversation history you resend on every turn, tool transcripts that balloon in agent loops, a system prompt that gets reprocessed thousands of times a day — and how much of that spend two boring features, prompt caching and batch processing, can eliminate.
By the end of this post you will know what a token is and why counts vary by language and content; why output tokens cost several times more than input tokens; why a naive chat loop makes cost grow roughly with the square of conversation length; how prompt caching works (prefix matching, read vs. write pricing, and when it pays for itself); and when to route work through a batch API for a flat 50% discount. We will close with a worked example: the same hypothetical support-bot workload priced naively and priced with caching plus batching, arithmetic shown.
Prices in this post are current as of August 2026 and will drift — treat the official pages from Anthropic, OpenAI, and Google as the source of truth.
What a token actually is
Models do not read words; they read tokens — subword chunks produced by a tokenizer. As a rough rule for English prose, a token is three to four characters, so a thousand tokens is on the order of 700–800 words. But that ratio is an average, not a law. Languages that the tokenizer was not optimized for can take noticeably more tokens for the same meaning. Code, deeply nested JSON, long identifiers, and anything base64-encoded are all token-dense relative to their visual length. The same text can also tokenize to different counts on different vendors — and even across model generations from the same vendor, since tokenizers get replaced.
The practical consequence: never estimate spend by eyeballing character counts. Every major provider exposes a token-counting endpoint or library; run your real prompts through it, on the exact model you plan to use, before you extrapolate a monthly bill.
Why output tokens cost more than input
Across vendors, output tokens are typically priced around five to eight times the input rate. That is not arbitrary. When a model processes your prompt (the prefill phase), it can chew through all the input tokens in parallel in one pass — cheap per token. Generating the response (the decode phase) is sequential: each new token requires a full forward pass that attends to everything before it, one token at a time. Sequential generation ties up accelerators far longer per token, and the price reflects that.
For a concrete anchor, here is the Claude lineup as of August 2026, per million tokens:
| Model | Input | Output | Context window |
|---|---|---|---|
| Claude Fable 5 | $10 | $50 | 1M tokens |
| Claude Opus 4.8 | $5 | $25 | 1M tokens |
| Claude Sonnet 5 | $3 ($2 intro through Aug 31, 2026) | $15 ($10 intro) | 1M tokens |
| Claude Haiku 4.5 | $1 | $5 | 200K tokens |
Note the 5× input-to-output ratio all the way down the lineup. The implication cuts both ways: trimming a verbose reply saves five times as much per token as trimming the prompt, so a sane max_tokens cap and an “answer directly, skip the preamble” instruction are among the cheapest optimizations available. But in history-heavy workloads, input volume dwarfs output volume — which brings us to the real cost driver.
The quadratic cost of a naive chat loop
LLM APIs are stateless. The model does not remember your last request; your application resends the entire conversation — system prompt, every prior user message, every prior reply — on every turn. So the input for turn k includes all k−1 earlier exchanges, and total input over an n-turn conversation grows roughly with n². Double the length of your average conversation and you roughly quadruple its input cost, not double it.
Agent workloads make this worse, because every tool call and its result get appended to the transcript and resent on each subsequent model call — a single “turn” of agent work can add thousands of tokens of tool output to everything that follows. (If that loop is unfamiliar, see the no-hype explainer on what AI agents actually are.) This quadratic growth is exactly the shape prompt caching was built to flatten.
Context windows: a ceiling, not a budget
A 1M-token context window means the model can accept a million tokens per request — not that doing so is free or even sensible. You pay for every token in the window on every call: a single full-context request on a $10-per-million-token model costs $10 in input alone, before any output. Long context is best treated as headroom for the cases that genuinely need it, while retrieval, summarization, and history-trimming keep the typical request lean.
The same logic applies to model choice. The spread between the cheapest and most expensive tier in a lineup is often 10× on both input and output, and plenty of production traffic — classification, extraction, routing, short answers — runs fine on the small tier. Which tier fits which job is its own topic; see how to choose between Claude, GPT, and Gemini in 2026 for that decision.
Prompt caching: read cheap, write once
Prompt caching lets the provider reuse the computation for a prompt prefix it has recently seen, and charge you a fraction of the input price for the reused part. The mental model that makes everything else make sense: caching is a strict prefix match on the rendered prompt. The cache key is the exact byte sequence from the start of the request; any change anywhere in that prefix — a timestamp interpolated into the system prompt, a reordered tool definition, a per-user ID near the top — invalidates everything after it.
On the Claude API, cached prefix reads cost roughly 0.1× the base input price, and writes carry a premium: 1.25× base for the default 5-minute cache lifetime, or 2× for a 1-hour lifetime. That makes the break-even arithmetic easy. With the 5-minute cache, a write plus one read costs 1.35× where two uncached passes cost 2× — caching pays for itself on the second request. The 1-hour variant needs at least three requests to win, in exchange for surviving gaps in traffic. Two more mechanics worth knowing: prompts below a model-specific minimum (on the order of a few thousand tokens) silently do not cache at all, and the usage block on each response reports cached-read and cache-write token counts. Full details are in Anthropic’s prompt caching docs.
The design rule follows directly from the prefix model: stable content first, volatile content last. Freeze the system prompt and tool definitions at the front of the request; keep timestamps, request IDs, and the user’s current question at the end, after the cache boundary. The most common caching failure in the wild is a system prompt that embeds the current date — a one-line template choice that silently zeroes the hit rate.
The other vendors run the same idea with different billing. OpenAI’s caching is automatic on prompts longer than 1,024 tokens; on models before the GPT-5.6 family there was no separate write charge, but GPT-5.6 and later bill cache writes at 1.25× the input rate — matching Anthropic’s 5-minute write premium — with cached reads at a 90% discount on recent models (see OpenAI’s prompt caching guide and current pricing page). Google’s Gemini API bills cached tokens at roughly 10% of the input rate but adds a storage fee — $1.00 per million tokens per hour on Flash-tier models and $4.50 on Pro-tier models as of August 2026 — for explicitly cached content, so idle caches cost money there in a way they do not elsewhere. Same concept, three different cost curves; check the billing details before porting a caching strategy across vendors.
Batch processing: half price for patience
If a job does not need an answer in seconds, it should probably not pay real-time prices. Batch APIs accept a file of requests, process them asynchronously within a window (typically 24 hours, often much faster), and charge 50% off both input and output tokens. Anthropic, OpenAI, and Google all offer this at the same flat discount as of August 2026.
The trick is noticing how much of a “real-time” product is secretly batchable: nightly evals, backfills over historical data, ticket tagging and summarization, content moderation sweeps, report generation, embedding-adjacent enrichment. A common pattern is a latency split — the user-facing turn runs synchronously, while everything derived from it (summaries, analytics, follow-up classification) drains through the batch queue overnight at half price.
Worked example: pricing a support bot two ways
Let us price a hypothetical workload honestly, using the Claude Haiku 4.5 rates above ($1 input / $5 output per million tokens). The numbers are illustrative — the point is the shape of the arithmetic, which transfers to any vendor.
- 100,000 support conversations per month, 6 turns each
- A stable 8,000-token prefix (system prompt, tool definitions, policy snippets) on every call
- Average user message: 100 tokens; average reply: 300 output tokens
- Plus a nightly job: summarize and tag each of the 100,000 transcripts (2,500 tokens in, 150 out)
The naive bill
Each turn resends the prefix plus the growing history, so the six calls in one conversation carry inputs of 8,100 / 8,500 / 8,900 / 9,300 / 9,700 / 10,100 tokens — 54,600 input tokens total, plus 1,800 output tokens. Per conversation that is $0.0546 of input and $0.0090 of output, about $0.064. Across 100,000 conversations: $6,360 per month. Running the summaries in real time adds 250M input tokens ($250) and 15M output tokens ($75) for $325 more. Naive total: roughly $6,685 per month.
The cached-and-batched bill
Now structure the prompt for caching (stable prefix first, cache boundary advanced each turn) and assume the 5-minute cache holds between turns. The shared 8,000-token prefix stays warm across the whole fleet — at this volume a conversation starts every few seconds — so even turn one reads it at 0.1×. Each turn then writes only the new suffix: 100 tokens on turn one, 400 tokens (prior reply plus new message) on each later turn. Per conversation: 52,500 cached-read tokens at $0.10 per million ($0.00525), 2,100 cache-write tokens at $1.25 per million ($0.00263), and the same $0.0090 of output — about $0.017, or $1,688 per month. Note what happened: output, untouched by caching, is now more than half the bill.
Moving the nightly summaries to the Batch API halves them to $162.50. Optimized total: about $1,850 per month, versus $6,685 naive — a 72% reduction with zero product changes, just prompt structure and scheduling. Real workloads are messier (turns that outlive the cache TTL pay re-writes, tool calls add tokens, retries happen), so treat this as directionally right rather than precise.
Bottom line: a cost-control checklist
To run these numbers on your own workload, the free LLM cost calculator on this site models cached share and batch discounts across current Claude, GPT, and Gemini prices.
- Measure before estimating — count tokens with the provider’s own tooling, on the exact model, on real prompts.
- Structure every prompt cache-first: frozen system prompt and tools up front, timestamps and per-request content last.
- Verify cache hits in the usage fields of real responses; a silent invalidator can zero your hit rate without any error.
- Cap and shape output — it is typically priced at 5–8× input across current lineups, and caching never discounts it.
- Trim history deliberately (summarize, truncate, retrieve) instead of riding the context window; a big window is a ceiling, not a budget.
- Right-size the model tier per route — much of production traffic does not need the flagship.
- Route everything non-interactive through the batch API.
- Re-check vendor pricing pages quarterly — lineups and discounts change fast.
Per-token pricing rewards teams who understand where their tokens come from. Most bills are dominated by re-sent history and reprocessed prefixes — the two things caching and batching directly attack. Get the prefix structure right, batch the patient work, watch the usage numbers, and the same product often runs at a quarter of its naive cost.