---
title: "AI inference: definition and example – Carl-Uwe Würfel"
description: "AI inference explained: what happens when a trained model runs, why output tokens cost more, and how batching, quantization, and caching cut the bill."
url: https://carl-wuerfel.de/en/glossary/inferenz
author: Carl-Uwe Würfel
---

1. [Home](https://carl-wuerfel.de/en)
  2. [Knowledge](https://carl-wuerfel.de/en/knowledge)
  3. [Glossary](https://carl-wuerfel.de/en/glossary)
  4. AI inference

# AI inference

AI inference is running a fully trained AI model on new input, such as a language model writing an answer; the model's weights do not change.

## On this page

  1. The model learns in training and works in inference
  2. A language model reads the input at once and writes the answer token by token
  3. Memory slows answers down more often than compute does
  4. Billing is per token, and output costs more
  5. Batching, quantization, and caching lower the cost per request
  6. What good implementation looks like
  7. Where the term stops working
  8. Frequently asked questions
  9. Sources

In inference, you can trade waiting time for cost per token.

_Figure: A balance based on an accounting firm with two tasks for language models. On the left, the client chat: answers are needed right away and every second of waiting counts, so it runs on a smaller, faster model. On the right, sorting receipts overnight: nobody needs the results before morning, so the job runs as a batch, which the provider charges less for. In the middle sits the trade of waiting time for cost per token; below, a note that fixed instructions come first so the provider can read them from the cache._

An AI model goes through two phases. In training, it learns from data and adjusts its weights, often billions of numbers that hold what it can do. In inference, it applies those weights to new input: a language model answers a question, a vision model spots a defect in a photo.

The word comes from logic, where it means drawing a conclusion; in statistics, it means reasoning from a sample to a population. In AI, it means the computation itself.

For businesses, inference is the part of AI that costs money every day, billed per [token](https://carl-wuerfel.de/en/glossary/token) or paid for with in-house hardware and power. Knowing how an answer is produced shows why long documents and long answers get expensive, and where waiting time can be traded for cost.

Example

An accounting firm has a language model sort incoming receipts overnight as a batch, since nobody needs the results before morning and the provider charges less for batches. Its client chat runs on a smaller, faster model, because every second of waiting counts there. In both cases, fixed instructions come first so the provider can read them from the cache.

Chapter 1 of 7

## The model learns in training and works in inference.

Training and inference differ in what happens to the weights. In training, they are learned from data; in [fine-tuning](https://carl-wuerfel.de/en/glossary/fine-tuning), they are adjusted with a company's own examples. In inference, they stay fixed, and the model only computes with them. Improvements arrive only with a new model version.

The EU [AI Act](https://carl-wuerfel.de/en/ai-act) uses the same term. Article 3(1) defines an AI system as one that infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions. Recital 12 calls the capability to infer a key characteristic of AI systems and applies it to two things: producing outputs, and deriving models or algorithms from data.

A quote that lists "inference" charges for using a model, not for building one.

Chapter 2 of 7

## A language model reads the input at once and writes the answer token by token.

Researchers at Google split language model inference into two phases. In the first, called prefill, the model processes all input tokens in a single pass, since they are all known upfront. Then it generates the answer, a phase called decode: each step runs the full model once and produces exactly one new token, which depends on the input and everything written so far.

To avoid recomputing earlier tokens, the model stores intermediate results in memory, the KV cache. The system picks each next token based on probabilities. The same question can therefore yield different wording.

This explains the waiting times: processing the input grows with its length, and writing the answer grows with the number of output tokens.

Chapter 3 of 7

## Memory slows answers down more often than compute does.

Inference runs on GPUs or specialized accelerators such as Google's TPUs. At every step, weights and KV cache have to be loaded from memory into the compute units, so short waiting times depend above all on memory bandwidth.

Woosuk Kwon and colleagues show how tight this gets: on an NVIDIA A100 with 40 GB of memory serving a 13 billion parameter model, the weights take up about 65 percent and the KV cache of active requests close to 30 percent. For the OPT-13B model, a single token needs 800 KB of KV cache, and a request of 2,048 tokens up to 1.6 GB. Systems common at the time used only 20.4 to 38.2 percent of the cache memory they reserved, by their measurement.

A request with a long document therefore ties up memory that other requests cannot use.

Chapter 4 of 7

## Billing is per token, and output costs more.

Cloud providers bill inference per token, with separate rates for input and output. On Anthropic's price list, an output token costs five times as much as an input token for every Claude model listed (as of November 9, 2026). OpenAI also charges more for output. That fits the mechanics: input is processed in one pass, every output token needs its own.

Pricing tiers add to this. Anthropic's Batch API gives a 50 percent discount on input and output for asynchronous processing. A fast mode delivers output much faster at a premium price. A fixed prompt prefix read from the cache costs at most 10 percent of the regular input price at Anthropic.

The Google researchers separate two goals: chatbots need short waiting times, while background jobs need low cost per token. In inference, you can trade waiting time for cost per token.

Chapter 5 of 7

## Batching, quantization, and caching lower the cost per request.

Batching groups several requests into one computation and raises throughput, meaning tokens per second. Memory limits how many requests fit in a batch. The vLLM system therefore manages the KV cache in pages, much like an operating system manages main memory (PagedAttention). Kwon and colleagues report two to four times the throughput at the same latency compared with FasterTransformer and Orca.

Quantization stores the weights at lower precision. In 2022, Tim Dettmers and colleagues showed with [LLM](https://carl-wuerfel.de/en/glossary/llm).int8() that this can halve the memory needed for inference with no performance loss in models of up to 175 billion parameters. Stronger quantization can reduce quality and needs testing first.

Prompt caching keeps the KV cache of a fixed prompt prefix, such as long instructions, so follow-up requests do not have to recompute it.

Chapter 6 of 7

## What good implementation looks like.

It starts with measuring, for each application, how many input and output tokens a request uses and how long the answer takes.

Each task gets the right kind of inference. A customer chat needs fast answers; overnight analysis of thousands of documents runs as a batch. Fixed instructions go at the start of the prompt so the provider can read them from the cache.

The company decides deliberately where inference runs. With a cloud API, input leaves the company, and the contract governs where it is processed and whether it is stored. On in-house hardware, data stays put, but procurement, operations, and utilization fall to the company.

When all calls go through a central layer such as an [LLM gateway](https://carl-wuerfel.de/en/glossary/llm-gateway), usage and cost per application show up in one place, and the model can be swapped.

Chapter 7 of 7

## Where the term stops working

Inference covers only a trained model producing an output. Data preparation, training, and fine-tuning are not part of it, nor is the application logic around it, such as document search in [RAG](https://carl-wuerfel.de/en/glossary/rag). AI inference is also distinct from the same term in logic and statistics.

## Frequently asked questions

What is AI inference?

AI inference is running a fully trained AI model on new input, for example when a language model answers a question. The weights stay unchanged. Every request to a model is an inference.

What is the difference between training and inference?

In training, a model learns from data and adjusts its weights; in inference, it only applies them. Training happens once or at intervals, inference with every request, so inference drives ongoing operating cost.

Does an AI model learn during inference?

No, the weights do not change during inference. What the model considers in a conversation sits in the [context window](https://carl-wuerfel.de/en/glossary/context-window) of each request. Whether a provider later uses inputs for training is set by its contract terms.

How is inference priced?

Cloud providers charge per token, with separate rates for input and output. At Anthropic, an output token costs five times as much as an input token, while batch processing and cache hits cost less (as of November 9, 2026). In-house hardware means paying for equipment, power, and operations instead.

What does latency mean for AI models?

Latency is the time until an answer arrives. For language models, it combines processing the input and generating the output, where each output token takes its own step. Long answers and large models increase it.

What hardware does AI inference need?

Large language models usually run on GPUs or specialized accelerators. The bottleneck is often memory for weights and KV cache, and the bandwidth to read it, more than compute. Small or quantized models also run on a single server.

What does inference mean in statistics?

In statistics, inference means reasoning from a sample to a whole population, such as from a survey to all customers. In AI, it means the computation a trained model performs to produce an output.

## Further reading

  * [**AI glossary**](https://carl-wuerfel.de/en/glossary)
  * [**Transformation: AI calls through one layer**](https://carl-wuerfel.de/en/transformation)
  * [**Token**](https://carl-wuerfel.de/en/glossary/token)
  * [**LLM gateway**](https://carl-wuerfel.de/en/glossary/llm-gateway)
  * [**Local and on-premise AI**](https://carl-wuerfel.de/en/glossary/lokale-ki)
  * [ Term**Prompt caching**](https://carl-wuerfel.de/en/glossary/prompt-caching)
  * [ Term**Reasoning model**](https://carl-wuerfel.de/en/glossary/reasoning-modell)

## Sources

  1. [European Union: Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 3(1) and Recital 12, 2024](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689)
  2. [Duden online: Inferenz, meaning and origin, as of November 2026 (German)](https://www.duden.de/rechtschreibung/Inferenz)
  3. [Pope et al. (Google): Efficiently Scaling Transformer Inference, arXiv:2211.05102, 2022](https://arxiv.org/abs/2211.05102)
  4. [Kwon et al.: Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv:2309.06180, SOSP 2023](https://arxiv.org/abs/2309.06180)
  5. [Dettmers et al.: LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, arXiv:2208.07339, NeurIPS 2022](https://arxiv.org/abs/2208.07339)
  6. [Anthropic: Pricing, Claude Platform documentation, as of November 9, 2026](https://platform.claude.com/docs/en/about-claude/pricing)
  7. [OpenAI: API Pricing, developer documentation, as of November 9, 2026](https://developers.openai.com/api/docs/pricing)
