Small language model (SLM)

A small language model works on the same principle as a large one. It is a neural network built on the Transformer architecture that splits text into tokens and predicts the next one. The difference is size. Large models run in data centers on many GPUs; an SLM fits on a single server, a workstation, a laptop or a phone. There is no fixed threshold. NVIDIA researchers described an SLM in 2025 as a language model that fits on a common consumer device and responds fast enough to serve one user's requests. As a rule of thumb for that time, they named fewer than 10 billion parameters.

The category moves with every model generation. What counts as small today would have been a large model a few years ago. Small variants come from Microsoft (Phi), Google (Gemma), IBM (Granite), Alibaba (Qwen), Mistral AI and Meta (Llama), among others, many with downloadable weights (as of November 2026).

The term matters to executives because it reopens questions of cost, data control and vendor dependence. Not every task needs the biggest model. Recurring, well-defined tasks such as sorting requests or reading data from invoices can often run on a small model faster, cheaper and on infrastructure you control. Only a test on real examples shows whether that holds.

How a language model gets small.

Small models come out of four techniques that vendors often combine. The first is training on carefully selected data. In 2024, Microsoft introduced Phi-3-mini, a model with 3.8 billion parameters small enough to run on a phone. Its technical report credits the results mainly to heavily filtered web data and synthetic training data.

The second is knowledge distillation, described by Geoffrey Hinton, Oriol Vinyals and Jeff Dean in 2015. A large model acts as the teacher, and the small model learns to reproduce its answers. The third is pruning: parts of the network that contribute little are removed. In 2024, NVIDIA showed how pruning and distillation turn a 15-billion-parameter model into 8- and 4-billion-parameter versions using less than 3 percent of the original training data.

The fourth is quantization, which stores the weights with fewer bits. The GPTQ method showed in 2022 that even very large models can be cut down to 3 or 4 bits per weight with little loss in quality. Memory needs shrink accordingly.

SLM and LLM compared.

SLMs and LLMs share the same technology and differ in breadth, hardware and cost structure. A large language model brings broad world knowledge and sustains long, open-ended reasoning. A small model has less room for knowledge and does best on narrow tasks: it responds faster, uses less power and runs on hardware a company controls.

Billing differs too. Most companies use a large model through a vendor's API and pay per token. A small model run in-house carries fixed costs for hardware and operations, no matter how many calls it handles.

Parameter count alone is a blurry marker. In mixture-of-experts models, only part of the network computes each token. According to its model card, OpenAI's gpt-oss-20b, released in 2025, has 21 billion parameters, 3.6 billion of them active per token, and runs within 16 GB of memory thanks to quantization. Where a model runs and what a completed task costs tell you more than its size.

What small models are good for.

Small models suit tasks that come in high volume and look alike each time. Typical examples are sorting requests, tickets or documents into fixed categories, pulling fields such as invoice number, amount or delivery date, summarizing short texts and removing personal data before a text goes on to a large model. On devices without a reliable connection, such as in vehicles or machines, they are often the only option.

Small models play a growing role in AI agents. The 2025 NVIDIA paper argues that many steps inside such systems are repetitive, narrowly scoped and non-conversational. Small models are good enough for those steps, the authors say, and more economical. They propose mixed systems: small models by default, large ones only where a task needs them. That is a reasoned position, not a measured industry standard.

Limits show up in open research, multi-step planning, rare specialist knowledge and tasks that demand broad general knowledge. There, a large model remains the better choice, and a small one is more likely to make up details.

Adapt, don't train from scratch.

Companies don't train their own SLM from scratch. They pick an existing model and adapt it, in order of effort. First come clear instructions and a few examples in the prompt. If the model lacks knowledge, for instance from internal manuals, retrieval-augmented generation (RAG) supplies the relevant passages before each answer.

If that isn't enough, fine-tuning on your own examples comes next. It works especially well for small models because they can be specialized on a single task with modest effort. A common method is LoRA, published in 2021: the original weights stay frozen and only small add-on matrices are trained. For a very large model, the authors report a 10,000-fold reduction in trainable parameters. For small models, a single GPU is often enough.

The quality of the examples decides the outcome. A small model absorbs the errors and gaps in its training data. Before training, fix a test set of real, labeled cases and measure every version against it.

What running it in-house costs, and what drives the cost.

Many small models carry no license fee. Running them still costs money: GPUs bought or rented, power, setup and integration, updates and security maintenance, quality monitoring and staff time. These costs accrue whether the model is busy or idle.

An API for a large model, by contrast, costs only when used, billed per token. Whether running a model yourself pays off depends mainly on volume and how steady the load is. With few or highly variable calls, the API is usually cheaper. With high, steady volume on one repetitive task, the balance can flip. Reasons beyond price count as well: data that should stay in-house, response times in milliseconds or operation without a network.

Many vendors also offer small models through their APIs. The price per token falls, and operations stay with the vendor. The only sound basis for a decision is cost per completed task across all options, calculated with your real volume.

How to recognize a good implementation.

A good implementation starts with a clearly described task and a test set of real cases. Small and large models are measured on the same cases, and the result decides, not a public leaderboard. Leaderboards measure general skills and go stale fast.

The license has been checked before use. Downloadable weights do not automatically mean commercial use, modification or redistribution is allowed. The architecture separates application and model: an intermediate layer routes requests to the right model, sends hard cases to a larger one and logs the cost of every call. That way a model can be swapped without rebuilding the applications.

Data protection and law are part of the design. Running locally keeps data in-house, but GDPR obligations remain: legal basis, purpose limitation, deletion and access rights. The EU AI Act's obligations for companies that deploy AI apply regardless of model size.

Warning signs include proposals with no testing on your own cases, the promise that local automatically means compliant, and systems where nobody rechecks quality after a model change.

Where the term stops working

Small language model describes the size and deployment of a language model, not its quality or its license. The line between SLM and LLM is fluid and moves with each hardware generation. An SLM can itself be a foundation model, broadly pretrained; a classic single-task model, such as a spam filter that generates no text, is not an SLM. Small also does not mean local: many small models run in the cloud with vendors.