--- title: "LLMOps and MLOps: definition and example – Carl-Uwe Würfel" description: "LLMOps explained: how companies run LLM applications with versioning, evals, monitoring, cost control, and fallbacks, and how LLMOps differs from MLOps." url: https://carl-wuerfel.de/en/glossary/llmops author: Carl-Uwe Würfel --- 1. [Home](https://carl-wuerfel.de/en) 2. [Knowledge](https://carl-wuerfel.de/en/knowledge) 3. [Glossary](https://carl-wuerfel.de/en/glossary) 4. LLMOps and MLOps # LLMOps and MLOps LLMOps is the disciplined operation of applications built on large language models: versioned prompts and models, testing, monitoring, cost control, logging, and fallback paths. ## On this page 1. From DevOps to MLOps to LLMOps 2. More than code gets versioned 3. Evals and monitoring measure quality 4. Costs accrue with every call 5. Plan for fallbacks and model changes 6. Common misconceptions 7. Where the term stops working 8. Frequently asked questions 9. Sources LLMOps makes an application measurable and controllable. _Figure: A branching diagram using the example of an online retailer that uses a language model to sort incoming customer emails by request type and to draft replies that staff review and send. What happens in three cases is defined in advance. Before any change, a fixed set of anonymized emails with known correct categories runs through the system; the prompt, model version and category list are versioned. When the vendor announces the model 's retirement, the successor is tested on the same emails. If the API goes down, emails land unsorted in the regular inbox. Below: a dashboard shows error rate, latency and daily cost._ **Example** An online retailer uses a language model to sort incoming customer emails by request type and to draft replies that staff review and send. The prompt, model version, and category list are versioned, and before any change a fixed set of anonymized emails with known correct categories runs through the system. A dashboard shows error rate, latency, and daily cost; when the vendor announces the model’s retirement, the successor is tested on the same emails, and if the API goes down, emails land unsorted in the regular inbox. LLMOps stands for [large language model](https://carl-wuerfel.de/en/glossary/llm) operations. It covers the processes, tools, and responsibilities needed to put an LLM application into daily use and keep it reliable, such as a customer service [chatbot](https://carl-wuerfel.de/en/glossary/chatbot) or triage of incoming requests. Microsoft describes the field under the name GenAIOps as the practice of managing large language models in production. Databricks lists evaluation, prompt and configuration management, cost control, and monitoring among its core areas. No formal standard defines the term. The term belongs to a lineage. DevOps joins software development and operations through automated delivery and monitoring. MLOps applies that approach to [machine learning](https://carl-wuerfel.de/en/glossary/machine-learning), which adds data, training, and model evaluation. LLMOps focuses on applications whose core is usually a third-party language model, accessed through an API, that the vendor can change or retire. For executives, LLMOps answers the question of what happens after a successful pilot. Without it, nobody knows whether an application still performs after a model change, who steps in when it fails, or what it costs each month. Those three questions decide whether an AI application stays in use. ## From DevOps to MLOps to LLMOps. Google describes MLOps as an ML engineering culture and practice that unifies the development and the operation of machine learning systems. Google names continuous training as new compared with DevOps: models are automatically retrained on new data and redeployed, because the world they describe keeps changing. An early foundation of the field is a paper by Google researchers led by D. Sculley, presented at NeurIPS 2015. It frames machine learning as a source of hidden technical debt: the model code is only a small part of a real-world system, and maintenance costs pile up in data dependencies, configuration, feedback loops, and changes in the environment. In LLMOps, most companies do not train their own language model; they call a finished model from a vendor. Its weights cannot be changed. The work centers on prompts, context from company data, evaluation of open-ended answers, and cost per call. Only when a company retrains a model on its own data, through [fine-tuning](https://carl-wuerfel.de/en/glossary/fine-tuning), do classic MLOps tasks such as managing training data and model versions come back. ## More than code gets versioned. The behavior of an LLM application depends on several parts that can each change on their own: the [system prompt](https://carl-wuerfel.de/en/glossary/system-prompt) and request templates, the model with its exact version identifier, settings such as maximum response length, the connected knowledge sources, and the tools an agent is allowed to call. LLMOps treats each of these parts like source code. Every change gets a version, and every answer in the log can be traced back to the combination that produced it. A 2023 study by Lingjiao Chen, Matei Zaharia, and James Zou showed why this matters. They compared the March and June 2023 versions of GPT-3.5 and GPT-4 across several tasks. Behavior differed substantially, and on some tasks the newer version performed much worse. The same product name can hide different behavior. Two rules follow. Where the vendor offers fixed version identifiers, the application uses them instead of an alias that automatically points to the latest release. And every change goes through the same sequence: change, test against fixed cases, approve, then watch in production. ## Evals and monitoring measure quality. Language models have no single correct output to check off, so quality is measured against criteria and reference cases. Before every change, [evals](https://carl-wuerfel.de/en/glossary/evals) run: fixed typical and difficult cases with known good answers. Scoring comes from rules, from human reviewers on samples, or from a second model that grades answers against defined criteria and is itself regularly checked against human judgment. In production, ongoing monitoring adds error rates, latency, abandoned sessions, user feedback, the share of cases handed to people, and spot checks for fabricated statements. Quality can drop without any model change when requests shift, say after a product launch or a change in the law. The AI Risk Management Framework published by the US National Institute of Standards and Technology (NIST) in January 2023 offers a frame for this. It organizes risk management into four functions: Govern, Map, Measure, and Manage. LLMOps mainly supplies the ongoing measurement and intervention in production. The log of inputs, outputs, versions, latency, and cost is the foundation. Because it often contains personal data, it needs access controls and retention limits. ## Costs accrue with every call. In MLOps, the big costs sit in training and infrastructure. In LLMOps, a company usually pays per call, billed by [tokens](https://carl-wuerfel.de/en/glossary/token), the units of text the model reads and writes. Cost depends more on design than on list price. Expensive patterns include long contexts, where every request resends the full conversation history and many document passages; a large model for simple tasks; automatic retries after errors; and agents that call the model many times for a single task. Evals and model-based checks are paid calls too. Disciplined operations make these costs visible before the invoice arrives. Every call is booked with its application, model, and cost. Budgets and caps per application keep a faulty agent from getting stuck in a loop. Smaller models handle simple steps such as classifying a request; larger ones only the steps that need them. Many vendors also offer caching for the unchanging parts of a request, which lowers recurring costs. Per-call booking shows which measure pays off. ## Plan for fallbacks and model changes. A vendor’s language model is a dependency with an expiration date. Anthropic’s documentation commits to notifying customers with active deployments at least 60 days before retiring publicly released models, and requests to retired models fail. OpenAI states minimum notice of six months for generally available models and three months for specialized variants, unless safety or compliance concerns require a faster timeline (as of November 2026). Outages, rate limits, and price changes come on top. Each application has a defined response to an outage: a second model tested with the same evals, a fixed default answer, a queue, or a handoff to people. A model change is a planned event with a test run, a comparison of results, and an approval. A central layer between applications and models, often called a gateway, helps. When all calls run through it, models can be swapped without rebuilding each application, and logging, cost tracking, and permissions sit in one place. In its GenAIOps maturity model, Microsoft explicitly lists advanced version control and rollback capabilities as a mark of mature teams. ## Common misconceptions. First misconception: LLMOps is a tool you buy. It is a way of working with defined processes and responsibilities. Platforms for logging, evals, or prompt management cover parts of it; people decide what gets tested, approved, and monitored. Second misconception: good prompts make testing unnecessary. Prompts, models, and incoming requests all change. Without ongoing checks, a decline in quality only surfaces when customers or employees complain. Third misconception: LLMOps is for large enterprises. A single customer-facing application already needs versioning, tests, monitoring, cost control, and a fallback path. The scope grows with the number of applications; the principle stays the same. Fourth misconception: LLMOps is the same as MLOps. A company that trains no model of its own needs no training pipelines, but it does need prompt versions, answer evaluation, and per-call cost tracking. Fifth misconception: LLMOps and AIOps are related. AIOps uses AI to run IT operations, for example to spot incidents early in server logs. The names are similar; the jobs are different. ## Where the term stops working LLMOps makes an application measurable and controllable; it does not decide whether using it makes sense, is legally permissible, or is factually correct. It differs from MLOps, which governs training and deployment of a company’s own models, from DevOps as the general foundation of software operations, and from AIOps, the use of AI to monitor IT systems. [AI governance](https://carl-wuerfel.de/en/ai-governance) sets the rules and responsibilities, and LLMOps enforces them technically in operation. ## Frequently asked questions What is LLMOps? LLMOps is the disciplined operation of applications built on large language models. It covers versioned prompts and models, testing before every change, ongoing monitoring, per-call cost control, logging, and a plan for outages. The term derives from DevOps and MLOps. What is the difference between LLMOps and MLOps? MLOps governs training, deployment, and retraining of a company’s own machine learning models. LLMOps governs the operation of applications built around language models that are usually third-party and accessed through an API. Instead of training data, the focus is on prompts, context, answer evaluation, and cost per call. What is MLOps? MLOps is the practice of building and running machine learning systems so that data, training, testing, and deployment are automated and traceable. Google describes it as an engineering culture that unifies development and operations. A distinctive feature is continuous retraining on new data. What does GenAIOps mean? GenAIOps stands for generative AI operations and is used largely interchangeably with LLMOps. Microsoft uses the term for operating large language models in production and explicitly names LLMOps as an alternative label. Microsoft has also published a four-level maturity model, from first experiments to fully managed operations. Does a mid-sized company need LLMOps? Yes, as soon as an LLM application runs in daily operations, especially in customer contact. The scope can be small: pinned model versions, a set of test cases, a log with cost per call, and a defined fallback path. Occasional use of a chat window does not require it. Which tools are part of LLMOps? Typical components are a central gateway for all model calls, logging and tracing, prompt version management, eval tooling, and a cost dashboard. No tool replaces deciding what gets tested and approved. What does good LLMOps look like? Every call can be traced to its application, version, and cost, prompts and models are versioned, and the same test cases run before every change. Models can be swapped without rebuilding applications, and there is a tested fallback path for outages. What is the difference between LLMOps and AIOps? LLMOps runs applications that use language models. AIOps uses AI to monitor IT operations, for example to detect incidents early in logs and metrics. The names are similar; the jobs are not. ## Further reading * [**AI glossary**](https://carl-wuerfel.de/en/glossary) * [**The shared AI infrastructure**](https://carl-wuerfel.de/en/transformation) * [**AI governance: permissions, access, logs**](https://carl-wuerfel.de/en/ai-governance) * [**Evals (LLM evaluation)**](https://carl-wuerfel.de/en/glossary/evals) * [**AI guardrails**](https://carl-wuerfel.de/en/glossary/guardrails) * [**Token** Term](https://carl-wuerfel.de/en/glossary/token) * [**Fine-Tuning** Term](https://carl-wuerfel.de/en/glossary/fine-tuning) * [**AI orchestration** Term](https://carl-wuerfel.de/en/glossary/ki-orchestrierung) ## Sources 1. [Google Cloud: MLOps: Continuous delivery and automation pipelines in machine learning, documentation, as of November 2026](https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning) 2. [Sculley et al.: Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015](https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html) 3. [Microsoft: Advance your maturity level for GenAIOps, Azure Machine Learning documentation, as of November 2026](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/concept-llmops-maturity) 4. [Databricks: What is LLMOps?, as of November 2026](https://www.databricks.com/blog/what-is-llmops) 5. [Chen, Zaharia, Zou: How Is ChatGPT’s Behavior Changing over Time?, 2023](https://arxiv.org/abs/2307.09009) 6. [NIST: Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, January 2023](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf) 7. [Anthropic: Model deprecations, documentation, as of November 2026](https://platform.claude.com/docs/en/about-claude/model-deprecations) 8. [OpenAI: Deprecations, API documentation, as of November 2026](https://developers.openai.com/api/docs/deprecations)