Tokens Explained: What AI Actually Counts
A clear explanation of what tokens are, how text gets tokenized, and why token count, not word or character count, determines cost and limits.
What a token actually is
A token is a chunk of text, sometimes a whole word, sometimes part of one, sometimes a single punctuation mark or a chunk of whitespace, that a model's tokenizer splits input into before processing it. Common English words are often a single token; rare words, unusual formatting, and most non-English text tend to split into several.
This is why token count does not map cleanly onto word count or character count. A rough rule of thumb for English prose is around four characters per token, but that estimate breaks down quickly for code, which tends to tokenize less efficiently due to symbols, indentation, and identifiers that are not common English words.
Why this matters beyond trivia
Context window limits are measured in tokens, not characters, so the same amount of visible text can consume very different amounts of a model's available context depending on what language, formatting, or code it contains. A block of minified JSON or a deeply nested code structure can burn through tokens surprisingly fast relative to its character count.
Pricing for most API-based AI tools is also metered in tokens, both for what you send in and what the model generates back. A prompt padded with unnecessary boilerplate or a response that rambles both cost real money at scale, which is a very concrete reason to keep prompts and outputs tight.
- Roughly four characters per token for English prose, less reliable for code
- Context window limits and API pricing are both measured in tokens
- Code and dense symbols often tokenize less efficiently than plain prose
- Non-English text frequently uses more tokens per word than English
Practical implications for developers
When working with agents on code, verbose comments, repeated boilerplate, and unnecessarily large pasted context all consume tokens that could otherwise go toward the actual task. Trimming irrelevant material is not just tidiness; it is a real budget decision.
If you are building anything that calls a model programmatically, count tokens explicitly with the provider's tokenizer rather than estimating from character count, especially for code-heavy or non-English input where the four-characters-per-token rule of thumb is least reliable.
Key takeaways
- Apply one concrete change from this post before collecting more reading.
- Prefer browser-side tools when the work involves secrets, tokens, or PII.
- Document the why next to the how so the next reviewer inherits context.
FAQ
- Who is this guide on ai for?
- Working developers who need a practical take on tokens explained: what ai actually counts — not a marketing overview. Skim the sections, apply one tip, then come back when you hit an edge case.
- Do I need an account to use the related tools?
- No. code.live tools run in your browser with no signup. Nothing you paste is uploaded to a server for the client-side utilities linked from this post.
- How often is this article updated?
- This post was published November 9, 2026. Fundamentals stay stable; check linked tool pages and official docs when version-specific behavior matters.