Most organizations that run machine learning in production built their MLOps practices around a familiar loop: collect labeled data, train a model, validate it against held-out metrics, deploy it behind an API, and monitor for data drift until it is time to retrain. That loop still matters for forecasting, fraud detection, recommendation and the many other predictive models that quietly run the business.
Generative AI breaks several of its assumptions. Teams rarely train the core model; they consume it from a provider or an open-weights release. Behavior changes when someone edits a prompt, swaps a retrieval index or when the provider ships a new model version. Outputs are open-ended text or actions rather than a score, cost scales with tokens rather than training runs, and the biggest risks are hallucination, prompt injection and unsafe actions rather than slow statistical drift. LLMOps is the discipline that adapts MLOps to these realities. This article sets out a practical operating model: what to version, which platform components you need, how to observe and control cost, and how to respond when things go wrong.
What Changes From MLOps to LLMOps
The differences are not cosmetic. They change what teams build and who owns what.
- The unit of change is configuration, not weights. In an LLM application, most releases change prompts, tool definitions, retrieval settings or the model version selected — not a model you trained. Each of these can alter behavior as much as retraining would.
- Quality is multidimensional. Accuracy, precision and AUC are replaced by faithfulness to sources, task completion, tone, safety and format compliance, often scored by a mix of rules, human review and LLM-based judges.
- Dependencies move under you. Hosted models are updated or deprecated on the provider’s schedule. A prompt tuned for one version can regress on the next without any change on your side.
- Cost is per request. Spend tracks tokens, context length, retries and agent steps, so a design decision such as retrieving twenty passages instead of five shows up directly on the invoice.
- The system is a pipeline. Retrieval, tools, guardrails and multiple model calls interact. Many production failures start in retrieval or tool layers rather than in the model itself, as MLflow’s LLMOps guidance notes.
- Security is part of operations. Prompt injection, data leakage through outputs and tool misuse are operational risks that must be monitored like availability and latency.
The Artifacts You Must Version
A useful test for LLMOps maturity is whether you can answer, for any response your system produced last Tuesday, exactly which configuration produced it. That requires versioning a broader set of artifacts than classic MLOps:
- Prompts and system instructions, stored in a registry with version history, owners, staging labels and fast rollback — treated as configuration, not strings buried in code.
- Model selection, including provider, model identifier, version or snapshot, and generation parameters such as temperature and maximum tokens.
- Tool and function definitions, including descriptions and schemas, since models choose tools based on those descriptions.
- Retrieval configuration: embedding model, chunking strategy, index version, filters and the number of passages retrieved.
- Guardrail policies: input and output filters, blocked topics, PII handling rules and thresholds.
- Evaluation datasets and judge prompts, so that a score from last quarter is comparable with a score today.
- Fine-tuned adapters or models, with lineage to the data used to produce them.
Bundle these into an application version that is deployed, evaluated and rolled back as a unit. Many teams do this with a configuration file in source control plus a prompt registry, with the release pipeline recording the full bundle hash on every trace.
Platform Components of an LLMOps Stack
You do not need a large platform team to run LLM applications well, but you do need a handful of shared components so each product team is not reinventing them.
Model gateway
A single entry point for all model calls that handles provider credentials, routing across models based on cost, latency and task type, fallbacks when a provider is degraded, rate limits, budget enforcement and logging. It is also where you enforce which models are approved for which data classifications.
Prompt and configuration registry
Versioned storage for prompts and application bundles with environments such as development, staging and production, and with approvals for production promotion.
Evaluation service
Automated test suites that run on every change: regression sets of real and synthetic cases, adversarial cases for injection and jailbreaks, and judges for faithfulness, task success and format. Evaluation also runs on a sample of production traffic to catch issues that tests missed.
Tracing and observability
End-to-end traces that capture each step: the incoming request, retrieved documents, model calls with token counts, tool invocations, guardrail decisions and the final response. The OpenTelemetry generative AI semantic conventions — still in development status in 2026, with a dedicated repository created in June — define attributes such as the operation name, provider, requested and response model, and input and output token counts, plus span types for agent invocations and tool execution. Adopting them keeps your telemetry portable across observability tools; just expect attribute names to change before they stabilize, and note that prompt and response content capture is opt-in by design for privacy reasons.
Feedback and data loop
Mechanisms to capture user ratings, corrections and escalations, route them to reviewers, and turn them into new evaluation cases and, where appropriate, fine-tuning data.
In LLMOps, a prompt edit is a production deployment. If it is not versioned, evaluated and traceable to every response it produced, you are not operating the system — you are hoping.
The Release Pipeline
A dependable release process for an LLM application looks like this:
- Change proposed in a branch: a prompt edit, a new model version, a retrieval tweak.
- Offline evaluation runs the regression suite and compares results with the current production bundle on quality, safety, latency and cost per request.
- Review by a product owner for user-facing behavior changes, and by security for new tools or data sources.
- Shadow or canary deployment sends a slice of real traffic to the new bundle while scoring outputs automatically.
- Progressive rollout with automated rollback triggers for quality, cost or error-rate thresholds.
- Post-release monitoring with a short window of heightened human review for high-impact changes.
Apply the same pipeline when the change comes from outside. When a provider announces a new model version or a deprecation date, treat it as a release: run the full evaluation, adjust prompts if needed, and migrate deliberately rather than discovering the change in production.
Monitoring: What to Watch in Production
LLM monitoring spans four families of signals:
- Quality: sampled judge scores for faithfulness and task success, user feedback rates, human override and escalation rates, and the share of responses that fail format validation.
- Safety and security: guardrail trigger rates, detected prompt-injection attempts, PII appearing in outputs, and blocked tool actions.
- Performance: time to first token, end-to-end latency, error and timeout rates by provider, and retrieval latency.
- Cost: tokens and spend per request, per feature, per customer and per team, plus the number of agent steps and retries per task.
Alert on changes rather than absolute values where possible. A sudden rise in average input tokens often signals a retrieval bug; a jump in escalation rate after a provider update signals a regression; a spike in guardrail triggers may indicate an attack. Segment every metric by application version so you can tie shifts to specific releases.
Cost Engineering
Inference cost is the line item that most often surprises leadership after launch. Levers that reduce it without hurting quality include:
- Routing simple requests to smaller or specialized models and reserving frontier models for hard cases.
- Trimming context: retrieve fewer, better passages and summarize long histories rather than resending them.
- Caching responses for repeated queries and using provider prompt caching for long shared prefixes.
- Batching non-interactive workloads such as document processing through lower-cost batch interfaces.
- Capping agent steps and retries per task, with escalation instead of endless loops.
- Attributing spend to products and teams so owners see and manage the cost of their own design choices.
Incident Response for LLM Systems
LLM incidents look different from classic outages. The service is up, but it is producing wrong, harmful or embarrassing answers, leaking information, or taking unintended actions. Prepare runbooks for at least these scenarios:
- Quality regression after a prompt, retrieval or provider change: roll back the bundle, then investigate with traces.
- Harmful or non-compliant output reaching users: tighten guardrails, notify stakeholders, preserve evidence, and assess regulatory reporting obligations.
- Prompt injection or data exposure: disable affected tools or data sources, rotate credentials if needed, and review logs for scope.
- Provider outage or deprecation: fail over through the gateway to a tested alternative model.
- Runaway cost: enforce budget caps automatically and identify the release or traffic pattern responsible.
Feature flags and kill switches for individual capabilities — a specific tool, a data source, autonomous actions — let you contain an incident without taking the whole application offline.
Team Structure and Ownership
LLMOps blurs boundaries between data science, software engineering, platform and risk teams. A model that works well in practice:
- Product teams own their applications end to end, including prompts, evaluation sets, quality targets and cost budgets.
- A small AI platform team owns the gateway, registries, evaluation tooling, tracing standards and approved-model catalog.
- Security and risk define guardrail policies, review new tools and data sources, and receive security-relevant telemetry.
- Domain experts help build gold datasets and review samples; their time should be planned, not borrowed.
Worked Example: Maturing an Internal Knowledge Assistant
A company launched an internal assistant that answers employee questions from policies, product documentation and engineering wikis. It was popular, but problems accumulated: answers occasionally cited outdated documents, nobody could explain why quality dropped one week, and monthly spend tripled without a clear reason.
The team introduced LLMOps practices over a quarter. Prompts moved from code into a registry, and every response trace now records the bundle version, retrieved document IDs and token counts using OpenTelemetry-style attributes. A regression set of several hundred real questions with expert-approved answers runs on every change. Tracing revealed that the cost spike came from a retrieval change that doubled the number of passages per request; the quality drop coincided with a provider model update that changed how the model followed citation instructions. The fix for both was a controlled release: fewer, re-ranked passages and an adjusted prompt, validated offline and rolled out via canary. The assistant now has quality, cost and latency targets on a dashboard reviewed weekly, and provider model updates go through the same pipeline as internal changes.
Turning Operations Data Into Governance Evidence
A well-run LLMOps platform does double duty: the same artifacts that keep the system reliable also answer the questions auditors, customers and regulators ask. Version histories show who changed what and when. Evaluation reports demonstrate testing before release. Traces and monitoring dashboards document how the system behaves in production, and incident records show that problems were detected and handled. Frameworks such as ISO/IEC 42001 and, for systems in scope, the EU AI Act expect exactly this kind of lifecycle evidence — technical documentation, record-keeping, accuracy and robustness testing, and post-market monitoring.
Design for this from the start. Retain traces and evaluation results for a defined period that matches your regulatory and contractual obligations, link each production bundle to its evaluation report, and make sure logs that contain prompts or outputs follow the same access controls and data-retention rules as the underlying data. Teams that do this find that governance stops being a separate documentation exercise and becomes a report generated from the platform they already run.
A Maturity Roadmap
- Level 1 – Ad hoc: prompts in code, manual testing, provider dashboards for cost.
- Level 2 – Managed: prompt registry, model gateway, basic tracing and a regression evaluation set.
- Level 3 – Automated: evaluation gates in CI/CD, canary releases, production sampling with judges, cost attribution and incident runbooks.
- Level 4 – Optimized: automated routing across models, continuous feedback loops into evaluation and fine-tuning, and governance evidence generated directly from the platform.
Conclusion
LLMOps is not a rebranding of MLOps; it is MLOps extended to systems whose behavior is defined by prompts, retrieval, tools and third-party models as much as by training. The organizations that run generative AI reliably treat every one of those elements as versioned configuration, route all model traffic through a gateway, trace every step with open standards, evaluate on every change, and manage cost and incidents as first-class operational concerns. Start with a registry, a gateway and tracing, add evaluation gates, and grow from there — the investment pays back the first time a provider update or a prompt tweak goes wrong and you can see, explain and roll it back in minutes.
How Hanoi Technology Can Help
Hanoi Technology builds LLMOps foundations — model gateways, prompt registries, evaluation pipelines and OpenTelemetry tracing — and can operate them as a managed service.