Local and on-premise AI
Local AI means AI models that run on hardware the company itself controls, such as servers in its own data center or an end device, rather than at a provider in the cloud.
Memory decides which model can run in-house.
With local AI, inference, the computation behind each answer, runs on machines the company operates itself. That takes a model with downloadable weights, a runtime, and hardware with enough memory. On-premise means the company’s own data center; edge means a device at the point of use, such as a factory machine.
Two things made this practical: open-weight models, whose weights anyone can download, and quantization, which stores those weights at lower precision. According to its model card, gpt-oss-20b, released by OpenAI in 2025 under the Apache 2.0 license, runs within 16 GB of memory. The larger gpt-oss-120b fits on a single 80 GB GPU.
Local describes where the computation happens. That decides whether data leaves the building and who controls the hardware. It doesn’t make a system secure, compliant, or cheap by itself, and responsibility for operations, updates, and access control moves in-house.
Chapter 1 of 7
How a model runs in-house.
It starts with a model whose weights can be downloaded, usually from Hugging Face. Weights are the numbers the model learned in training. A runtime loads them into memory and computes the answers.
Common runtimes are llama.cpp, Ollama, and vLLM. llama.cpp is MIT-licensed and runs on CPUs, Apple silicon, and NVIDIA GPUs, among others. Ollama targets single machines, vLLM servers handling many requests at once. llama.cpp includes llama-server, an OpenAI-compatible API, so applications that call a cloud model can often switch to the local one by changing an address.
Company knowledge usually comes in through retrieval-augmented generation (RAG): a separate system finds relevant passages in approved documents and passes them to the model with the question. That search can run in-house too.
Chapter 2 of 7
Memory decides which model can run in-house.
The weights must fit in GPU memory, or in shared memory on Apple machines. As a rough rule, at 16-bit precision each parameter takes two bytes, so a model with 8 billion parameters needs about 16 GB for the weights alone. Quantized to 4 bits, it’s half a byte per parameter, about 4 GB.
On top of that comes the working memory of each active request, the KV cache. It grows with the length of the text being processed. The vLLM team noted in 2023 that this memory is large per request and changes size during generation. Many concurrent users and long documents therefore need more memory than the model itself.
If everything doesn’t fit in GPU memory, llama.cpp runs part of the model on the CPU. That works, but answers arrive more slowly. Quantization can also lower quality; only a test with real cases shows whether that matters.
Chapter 3 of 7
Local, private cloud, and EU cloud differ in who has control.
The cloud definition of NIST, the US standards agency, distinguishes private and public clouds, among others. A private cloud serves one organization exclusively; the organization or a third party can run it, on or off premises. A public cloud sits on the provider’s premises.
Applied to AI, control comes in degrees. With local AI in the narrow sense, the company runs hardware and model. With rented dedicated servers elsewhere, data leaves the building, but model and software stay under the company’s control. With a cloud model in an EU data center, the provider runs the model, and the location says little about which law the provider is subject to. That question belongs to sovereign AI.
Many companies work hybrid, choosing local or cloud per task based on data confidentiality and quality needs.
Chapter 4 of 7
Local doesn’t mean secure.
According to its documentation, Ollama binds to 127.0.0.1 on port 11434 by default, so it only accepts requests from the same machine. Anyone who opens it to the network via OLLAMA_HOST has to add authentication and permissions. An open model server on the network is local and still a risk. The same documentation covers cloud models and how to turn those features off, so a tool considered local may send requests outside depending on its settings.
Data protection law applies regardless of location. Article 32 GDPR requires appropriate technical and organizational security measures. What falls away is the data processing agreement with a model provider under Article 28, as long as no service provider handles the data.
The EU AI Act ties its obligations to roles such as provider and deployer (Article 3) and to the risk of the use case, not to the hardware’s location.
Chapter 5 of 7
What running it yourself costs, and what drives the cost.
Local AI isn’t free; it shifts costs. Instead of per-token fees, the company pays for GPUs or rentals, power, setup, updates, monitoring, and staff time. These costs are largely fixed, even when the model sits idle.
High, steady volume and data that shouldn’t leave the building favor in-house operation. With few or very uneven calls, a cloud model billed by usage is usually cheaper.
One advantage is hard to price: control over the model version. Cloud providers retire older models. Anthropic, for example, gives at least 60 days’ notice; after retirement, requests to the old model fail. A local model stays unchanged as long as the company wants. Testing and rolling out new versions is then the company’s job.
Chapter 6 of 7
What good implementation looks like.
It starts with a specific task and a test set of real cases, on which local and cloud models are compared before a decision. The model license has been checked, because downloadable weights don’t automatically permit every commercial use.
The model server requires authentication, every application has its own credentials, and calls are logged. Applications reach the models through a shared layer such as an LLM gateway, so each task can switch between local and cloud without rebuilding applications.
One person owns operations. Model files come from verified sources, runtime and model updates follow a plan like any other software, and hardware utilization is monitored.
Chapter 7 of 7
Where the term stops working
Local AI describes where computation happens, not quality, security, or legal compliance. An open-weight model is a prerequisite but can also run at a hosting provider; a small language model describes a model’s size, not its location. Sovereign AI asks whether a company can make its own decisions about models, data, and providers. Running models locally is one way to get there, not the only one.
Frequently asked questions
What is local AI?
Local AI is an AI model that runs on hardware the company controls, such as its own server or a workstation. Requests and data don’t go to a model provider in the cloud.
What hardware do you need for local AI?
Above all, local AI needs enough fast memory for the model, usually the memory on a graphics card. Small quantized models run on a well-equipped workstation; large models with many users need servers with powerful GPUs.
How much VRAM does a local language model need?
A rough rule is parameter count times memory per weight, plus headroom for the working memory of active requests. A model with 8 billion parameters needs about 16 GB for its weights at 16 bits, and roughly 4 GB quantized to 4 bits.
What is the difference between local AI and cloud AI?
The difference is who controls the hardware the model runs on. With local AI it’s the company itself; with cloud AI it’s the provider, and every request travels over the internet. Local AI comes with fixed costs, cloud AI with usage fees.
Is local AI GDPR-compliant?
Not automatically. Running a model locally can keep personal data away from a model provider, but it doesn’t replace a legal basis, a deletion policy, access rights, or the security measures required by Article 32 GDPR. Specific cases are for the data protection officer to assess.
What software can you use to run AI locally?
Common choices are Ollama for single machines, llama.cpp as an open-source runtime with its own server, and vLLM for servers handling many concurrent requests. All three load open-weight models and expose them through an API.
Sources
- OpenAI: gpt-oss-20b model card on Hugging Face, 2025
- ggml-org: llama.cpp, project documentation on GitHub, as of November 2026
- Ollama: FAQ, documentation, as of November 2026
- Woosuk Kwon et al.: Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023
- NIST: Peter Mell, Timothy Grance, The NIST Definition of Cloud Computing, SP 800-145, September 2011
- Anthropic: Model deprecations, documentation, as of November 2026
- European Union: Regulation (EU) 2016/679 (General Data Protection Regulation), Articles 28 and 32, 2016
- European Union: Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 3, 2024