--- title: "Token: definition and example – Carl-Uwe Würfel" description: "Tokens in AI explained: how language models split text, why tokens drive cost and capacity, and how companies measure and manage token usage in practice." url: https://carl-wuerfel.de/en/glossary/token author: Carl-Uwe Würfel --- 1. [Home](https://carl-wuerfel.de/en) 2. [Knowledge](https://carl-wuerfel.de/en/knowledge) 3. [Glossary](https://carl-wuerfel.de/en/glossary) 4. Token # Token A token is the unit a language model splits text into, such as a word, part of a word or a punctuation mark; providers measure both cost and capacity in tokens. ## On this page 1. How a model splits text 2. How many words make a token 3. Everything that counts as a token in a call 4. How the price of a call adds up 5. What good cost control looks like 6. Common misconceptions 7. Where the term stops working 8. Frequently asked questions 9. Sources The biggest item is often not the visible question but the material sent along with it. _Figure: A bill, using the example of an insurance broker that has a language model summarize incoming claims in one paragraph each. On the input side, the instruction with the claim text, the photos and the multi-page assessment count as tokens; on the output side, the summary. Photos and assessment are highlighted. Result: almost the entire bill comes from the input. Below, the question whether questions come up: if yes, the photos are analyzed too; if no, only the text goes to the model first._ **Example** An insurance broker has a language model summarize incoming claims in one paragraph each. Reviewing the calls shows that almost the entire bill comes from the input, because every claim arrives with photos and a multi-page assessment. Since then, the company sends only the text first and has the photos analyzed only when questions come up; before switching, it checked summary quality on a sample of older cases. Before a language model can process text, it breaks input and output into tokens. The component that does this is called a tokenizer: it maps each token to a number from a fixed vocabulary, and the model only works with these numbers. It generates its answer one token at a time. Common words are often a single token, while rare words, long compounds and numbers break into several. How exactly the text is split depends on the model. Tokens are the unit providers use for almost everything about a language model. Prices are quoted per million tokens, separately for input and output. The [context window](https://carl-wuerfel.de/en/glossary/context-window), meaning how much a model can take into account in a single call, is measured in tokens. Usage limits and response speed depend on token counts too. For executives, the token is therefore the unit of AI cost. To understand what an AI deployment costs, you need to know everything that counts as a token in a call. The biggest item is often not the visible question but the material sent along with it. ## How a model splits text. Most current language models use subword methods. A well-known one is byte-pair encoding, which Rico Sennrich and colleagues introduced in 2016 to split words for machine translation. The idea is that character sequences that appear often in the training text are merged into a single token, while rare sequences stay split into smaller pieces. This leads to quirks you notice in everyday use. A leading space is often part of the token. Uppercase and lowercase can produce different tokens. Numbers are often split into groups of digits, which helps explain why language models are shaky with long numbers unless they have a calculator tool. Each model family has its own tokenizer, and providers change it from time to time. Anthropic, for example, says its models from the Claude 4.7 generation onward use a newer tokenizer that produces roughly 30 percent more tokens for the same text than the previous one (pricing documentation, as of November 2026). The same text therefore yields different token counts on different models. Comparing prices per token only means something together with the token count for your own task. ## How many words make a token. There is no fixed conversion. For English text, OpenAI gives a rule of thumb: one token is about four characters, or roughly three quarters of a word. Other languages often need more tokens for the same content. A study by Aleksandar Petrov and colleagues, presented at NeurIPS 2023, measured this with translated texts: depending on the language and the tokenizer, the same content took up to fifteen times as many tokens. German is not at the extreme end, but long compound words such as Gewährleistungsfrist (warranty period) usually break into several tokens. A German text is therefore typically longer in tokens than its English translation. The type of content matters as well. Code, tables, structured data such as JSON, columns of numbers and special characters produce different ratios than running prose. Only measurement is reliable. Providers offer tools or endpoints that count a text before you send it, and every API response reports how many tokens the call actually used. ## Everything that counts as a token in a call. You pay for the entire input, not just the question a person types. The input includes the application's fixed instructions (the [system prompt](https://carl-wuerfel.de/en/glossary/system-prompt)), the conversation so far, attached documents, passages added by a search step, the descriptions of connected tools and the results those tools return. Images and PDF pages are converted into tokens as well. The output is the answer. For models that work through intermediate steps before answering, those thinking tokens come on top. Anthropic bills them as output tokens even when they are not shown in full, and other providers handle it similarly. A short answer can therefore use a lot of output tokens if the model thought for a long time. Two effects stand out in practice. First, the system prompt is sent again with every single call. In an application that makes many calls a day, it acts like a fixed fee per call. Second, a chat application resends the entire conversation history with every new message. Each additional turn costs more than the last, even when the new question is short. To understand what an application costs, start by breaking down how many tokens each part of the input accounts for. ## How the price of a call adds up. The price of a call is input tokens times the input price plus output tokens times the output price. Prices are usually quoted per million tokens and vary widely by model. Output tokens usually cost more than input tokens. At Anthropic, output currently costs five times as much as input, and the input price of the cheapest and the most expensive current model differs by a factor of about one hundred (price list, as of November 2026). Model choice is often the biggest lever on cost. Discounts change the bill considerably. With prompt caching, the provider stores a recurring beginning of the input, such as the system prompt or a fixed manual. When it is read again, it costs only a fraction of the normal input price; at Anthropic, a tenth or less. With batch processing, tasks that are not time-sensitive are collected and processed together; Anthropic offers 50 percent off input and output for this. The two discounts can be combined. Prices change often, frequently with each new model generation. It is more robust to know the usage per task and plug in the current price. ## What good cost control looks like. Good cost control starts with visibility. Every call is logged with the model, input tokens, output tokens and cost, and assigned to an application and a task. Only then can you say which application uses a lot and why. The usual measures build on that foundation. Each task gets the cheapest model that handles it reliably in tests. The input contains only what the answer needs, for example selected passages instead of entire documents. Long conversation histories are summarized. Recurring beginnings are cached, and tasks without time pressure are batched. A cap on output length prevents outliers. When an answer hits that cap, it stops mid-way; the application has to detect and handle that instead of passing on a truncated answer. The order matters. Every cost-saving measure is tested against a fixed list of real tasks before it goes live. Cutting inputs or switching to a smaller model without measuring quality saves money in the wrong place. When talking to vendors, it helps to ask whether usage and cost per call are visible and who gets alerted when there is an outlier. ## Common misconceptions. "A token is a word." That is only roughly true and only for common words. A word can be one token or several, and the model decides the split, not grammar. "A short answer means a cheap call." In many applications, most of the cost is on the input side: system prompt, history, attachments. A long attachment adds to the context that has to be processed, even when you only want a short answer. "The price per million tokens shows which model is cheaper." Different tokenizers produce different token counts for the same text, and a model that thinks at length uses more output. Only the cost per completed task is comparable. "Tokens don't matter on a subscription." Subscriptions to AI assistants have usage limits too, and long conversations with large attachments hit them faster. "Fewer tokens are always better." A lean context saves money. But if information the answer needs is missing, the answer gets worse, and fixing it costs more than the tokens saved. ## Where the term stops working Tokens measure the amount of text and other input, not how hard a task is or how good the answer is. A fixed conversion from words to tokens does not fit every language and every type of content. The token is distinct from the context window, which sets the limit in tokens, and from [embeddings](https://carl-wuerfel.de/en/glossary/embeddings), which represent the meaning of a text as a string of numbers. API tokens used as access keys and crypto tokens share only the name. ## Frequently asked questions What is a token in AI? A token is the unit a language model splits text into before processing it. It can be a whole word, part of a word, a punctuation mark or a space attached to the start of a word. Providers use tokens to measure the cost of a call and the size of the context window. How many words are in a token? There is no fixed number, because each model splits text differently. For English text, OpenAI's rule of thumb is about three quarters of a word per token; languages with long compound words, such as German, usually need more tokens. Only counting with the provider's own tool is reliable. How much do AI tokens cost? That depends on the model and the provider and changes often. Billing is per million tokens, separately for input and output, and output usually costs more. Small and large models from the same provider are often one to two orders of magnitude apart. Why does a short answer sometimes cost a lot? Because you pay for the input too, and it often makes up most of the cost. The system prompt, conversation history, attachments and passages added by a search count in every call. Models that think before answering also add output tokens that do not appear in the answer. Do images and PDF files count as tokens? Yes. Images and PDF pages are converted into tokens and take up space in the context window. How many tokens an image uses depends on its size and the model; providers list the details in their documentation. What are thinking tokens or reasoning tokens? Thinking tokens are intermediate steps some models generate before the actual answer to break a task down. They are billed as output tokens even when they are not shown in full. They often improve results on multi-step tasks and make calls more expensive and slower. How can you reduce token usage? The most effective levers are choosing a suitable smaller model and sending only the input the task needs. Summarizing long histories, caching recurring beginnings (prompt caching) and batch processing for tasks without time pressure help as well. Test every measure against real tasks so quality does not suffer. What is the difference between a token and a context window? The token is the unit of measurement, the context window is the limit. The context window states how many tokens a model can take into account in a single call, including instructions, history, documents and, depending on the model, the answer. ## Further reading * [**AI cost management: know the cost per call or fly blind** Note](https://carl-wuerfel.de/en/notes/controlling-ai-api-costs) * [**ChatGPT or Claude: judge the model by the task** Note](https://carl-wuerfel.de/en/notes/chatgpt-vs-claude-in-production) * [**Measuring AI ROI: time saved isn 't money saved**Note](https://carl-wuerfel.de/en/notes/ai-roi-in-an-agency) * [**AI glossary**](https://carl-wuerfel.de/en/glossary) * [**LLM: Large Language Model**](https://carl-wuerfel.de/en/glossary/llm) * [**Context Window**](https://carl-wuerfel.de/en/glossary/context-window) * [**LLM gateway** Term](https://carl-wuerfel.de/en/glossary/llm-gateway) ## Sources 1. [Sennrich et al.: Neural Machine Translation of Rare Words with Subword Units, 2016](https://aclanthology.org/P16-1162/) 2. [Anthropic: Pricing (documentation)](https://platform.claude.com/docs/en/about-claude/pricing) 3. [OpenAI: Understanding and counting tokens (Help Center)](https://help.openai.com/en/articles/4936856-understanding-and-counting-tokens) 4. [Petrov et al.: Language Model Tokenizers Introduce Unfairness Between Languages, 2023](https://arxiv.org/abs/2305.15425) 5. [Anthropic: Thinking (documentation)](https://platform.claude.com/docs/en/build-with-claude/thinking) 6. [Anthropic: Prompt caching (documentation)](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) 7. [Anthropic: Batch processing (documentation)](https://platform.claude.com/docs/en/build-with-claude/batch-processing)