What Is a Token? How LLMs Actually Read Text

Before a language model does anything intelligent with your prompt, something entirely mechanical happens: the text is chopped into tokens. A token is a chunk of text — sometimes a whole word, often a fragment of one — drawn from a fixed vocabulary the model learned before training. The model never sees letters, words, or sentences. It sees a sequence of integer IDs, one per token, and its only job is to predict the next ID.

That implementation detail leaks into everything about working with LLMs. Pricing is per token. Context windows are measured in tokens. Rate limits are tokens per minute, and speed is tokens per second. The reason a model can write a decent sonnet but miscount the letters in strawberry is tokenization. If you build on these APIs, the token is the unit your bill, your latency, and several of your bugs are denominated in — so it is worth understanding properly.

Every token count in this post was measured by running text through OpenAI’s open-source tiktoken library (the o200k_base encoding used by recent GPT models, vocabulary of roughly 200,000 tokens). Other vendors use different tokenizers, so the exact numbers differ — but the patterns hold everywhere.

From text to tokens: how BPE works

Modern tokenizers descend from byte-pair encoding (BPE), a compression-inspired algorithm adapted for neural machine translation in 2016. The training procedure is simple: start with individual bytes, scan a large corpus, and repeatedly merge the most frequent adjacent pair into a new vocabulary entry. Run that a couple hundred thousand times and you get a vocabulary where common character sequences — which mostly means common words — are single tokens, while everything rare gets assembled from pieces.

The result is a frequency-sorted view of language. Here is how a few English words tokenize:

Two details surprise people. First, tokens usually absorb the leading space: the and _the (with a space) are different tokens, which is why token counts do not line up neatly with word counts. Second, context changes the split — strawberry at the start of a string is three tokens, but with a leading space it collapses to a single token, because the corpus mostly contained it mid-sentence. Tokenization is deterministic, but it is not intuitive.

The four-characters-per-token rule of thumb

For ordinary English prose, the standard approximations are about four characters per token, or about three-quarters of a word per token — so 1,000 tokens is on the order of 750 English words. Both numbers are rough averages, and both vary by tokenizer and by text. My own measurements land close: a 44-character, 9-word sentence came out at 10 tokens (4.4 characters per token), and a 199-character paragraph of plain technical prose came out at 38 tokens (5.2 characters per token).

The | quick | brown | fox | jumps | over | the | lazy | dog | .

44 characters, 9 words -> 10 tokens (o200k_base)

Use the rule of thumb for napkin math and nothing else. It drifts with vocabulary size, with how much whitespace and punctuation your text has, and — as the next section shows — it falls apart entirely for code and for most languages other than English. When a number actually matters, count; do not estimate.

Why code and non-English text cost more

Tokenizer vocabularies are trained on corpora that skew heavily toward English prose. Anything that looks different from English prose fragments into more, smaller pieces — which means more tokens per unit of meaning, and therefore more money per unit of meaning. Measured on the same tokenizer:

Code is token-dense because brackets, operators, indentation, and identifier fragments each burn tokens: a trivial 38-character function costs 14 of them, and JSON keys plus quoting overhead push structured data to roughly three characters per token. This is why agentic coding workloads — which shuttle diffs, file contents, and tool output through the context on every turn — consume tokens far faster than a chat about the same subject would.

For natural languages the chars-per-token column is misleading, because characters carry different amounts of meaning. The better comparison is tokens per word: the English sentence above costs about 1.1 tokens per word, while the Hindi sentence costs about 1.6 — a premium of roughly fifty percent for saying the same kind of thing. And this tokenizer is a good one for Hindi; its predecessor (cl100k_base, vocabulary of roughly 100,000) needed 51 tokens for that same sentence, nearly three times as many. Research has quantified how bad the spread gets: Petrov et al. (2023) found the same text can tokenize to up to 15 times as many tokens across languages, a disparity that persisted across all 17 tokenizers they evaluated. Since price, latency, and effective context are all proportional to token count, speakers of underrepresented languages pay more for less on the same API — a genuine fairness problem baked in before the model runs a single layer.

Everything is priced and limited in tokens

Once you see the token as the atomic unit, LLM API economics becomes one sentence: you pay a rate per million tokens in, a higher rate per million tokens out, and every limit you hit is a token budget. As of August 2026, Claude Sonnet 5 lists at $3 per million input tokens and $15 per million output tokens (with an introductory $2/$10 through the end of the month), and the flagship tiers run up to $10/$50 — see Anthropic’s pricing page for current numbers. I cover the full mechanics — why output costs more, prompt caching, batch discounts — in LLM API Pricing, Explained, and you can model a real workload across Claude, GPT, and Gemini with the LLM cost calculator.

• Context windows are token budgets, not character budgets. A 200K-token window holds roughly 150,000 words of English — but meaningfully less code, JSON, or non-English text, for the reasons above. Frontier models advertise 200K to 1M tokens as of August 2026.
• Rate limits are usually expressed as tokens per minute (input and output separately), alongside requests per minute. A single huge prompt can exhaust a token budget that thousands of small requests would never touch.
• Speed is quoted in output tokens per second. Reading a prompt is fast because input tokens are processed in parallel; generation is sequential, one token at a time, which is why long answers dominate latency.

This is also why token counts quietly shape architecture decisions. Retrieval pipelines chunk documents by token count so retrieved passages fit the budget — a detail that matters as much as embedding quality, as I discuss in the RAG explainer — and model selection is partly a question of how many tokens of context your workload genuinely needs, which is a core part of choosing an LLM at all.

Why models fail at counting letters

Ask a model how many times the letter r appears in strawberry and you are asking it to reason about characters it has never seen. The model receives st · raw · berry — three opaque IDs. Any knowledge that the middle piece contains an r has to come from statistical association, not from looking, because there is nothing to look at.

The same blindness explains a whole family of failures: reversing a string, counting letters or syllables, alphabetizing precisely, acrostics, character-level ciphers, and off-by-one edits inside long identifiers. Newer models do better, mostly by learning to spell words out token by token before answering — effectively simulating character access — or by writing a few lines of code when a tool is available. But the weakness is structural. If your application needs character-exact manipulation, do it in code and let the model decide what to do, not perform the string surgery itself.

Counting tokens in practice

The first rule: use the vendor’s own counter, on the exact model you will call. Tokenizers are not interchangeable — tiktoken counts GPT tokens and will materially miscount for Claude or Gemini. Anthropic exposes a free token counting endpoint that accepts the same payload as a real request, including system prompts, tools, and images; OpenAI ships tiktoken for local counting; Google provides token counting in the Gemini API. For quick visual intuition, OpenAI’s interactive tokenizer lets you paste text and watch it split.

import anthropic

client = anthropic.Anthropic()
count = client.messages.count_tokens(
    model="claude-sonnet-5",
    messages=[{"role": "user", "content": "Hello, tokens"}],
)
print(count.input_tokens)

The second rule: re-baseline whenever you change model generations, even within one vendor. Tokenizers get replaced, and when they do, the same text can cost meaningfully more. This is not hypothetical: Anthropic documents that Claude models from Opus 4.7 onward use a newer tokenizer that produces roughly 30 percent more tokens for the same text than earlier models, and billing reflects the new counts. A migration that looks price-neutral on the per-token rate card can still move your bill. Count representative prompts on the new model before extrapolating.

Third: trust measured usage over estimates. Every API response includes a usage object with actual input and output token counts — log it, aggregate it, and alert on it. Estimation is for planning; production cost tracking should come from what the meter actually read. And leave headroom in context-window math, since system prompts, tool schemas, and formatting overhead all consume tokens your mental model of “the conversation” tends to forget.

The short version

• Models read subword tokens, not words. Common words are one token; rare words split into pieces via BPE-style merging.
• For English prose, roughly four characters or three-quarters of a word per token — an approximation that varies by tokenizer, and breaks down for code and non-English text.
• Code and most non-English languages cost more tokens per unit of meaning — up to 15 times more across languages — which means higher cost, higher latency, and less usable context.
• Pricing, context windows, rate limits, and speed are all denominated in tokens. Understand tokens and you understand the whole bill.
• Character-level tasks fail because the model never sees characters. Do string surgery in code.
• Count with the vendor’s tokenizer for the exact model, re-baseline on new model generations, and track real usage from API responses rather than estimates.