For three years, enterprise teams framed LLM customization as a binary: retrieve documents at query time (retrieval-augmented generation, or RAG) or train the model on your data (fine-tuning). In 2026 that framing is out of date. Context windows that hold hundreds of thousands of tokens, cheap prompt caching, agents that decide for themselves when to search, and a sharp shift in how vendors offer fine-tuning have turned a two-way choice into a four-way one. Teams that still argue “RAG or fine-tune?” in architecture reviews often pick the wrong tool, or pay for two when one would do.
This guide explains what changed, offers a decision framework built around the question each technique actually answers, and walks through a worked example. It is written for product owners, architects and AI leads who must commit budget to one approach and defend that choice six months later.
What Changed Since the Original Debate
Four developments reshaped the trade-offs.
1. Long context became practical, but not free of failure
Frontier models now accept very long inputs, and prompt caching makes re-sending the same large prefix far cheaper and faster. Anthropic’s engineering team has gone as far as recommending that if a knowledge base is under roughly 200,000 tokens (about 500 pages), teams can simply place the whole thing in the prompt, and it reports that caching can cut latency by more than half and cost by up to 90% in that pattern. For small, stable corpora, “no retrieval at all” is now a legitimate architecture.
The catch is quality. Chroma’s July 2025 Context Rot study tested 18 models and found that performance degrades as input length grows, even on simple tasks, and that distracting but similar passages hurt more as context lengthens. In one experiment models answered far better from a focused ~300-token input than from a full ~113,000-token conversation history. A large window is capacity, not comprehension.
2. Retrieval got much better
Naive RAG, which splits documents into chunks, embeds them and returns the nearest few, has been superseded by hybrid search (vector plus keyword), reranking and chunk enrichment. Anthropic’s contextual retrieval technique prepends a short, document-aware explanation to each chunk before indexing. In its published benchmark, contextual embeddings cut the retrieval failure rate by 35%, adding contextual BM25 keyword search raised that to 49%, and adding a reranker brought the total reduction to 67%. Most “RAG doesn’t work for us” complaints we hear trace back to retrieval quality rather than to the model.
3. Retrieval became agentic
Instead of a fixed pipeline that always retrieves top-k chunks, agentic systems give the model search tools, often exposed through the Model Context Protocol, and let it decide whether to search, what to search for, and whether the results are sufficient. That turns retrieval into a multi-step reasoning task: decompose the question, query several sources, read, refine, and cite. It improves hard, multi-hop questions, at the cost of more tokens, more latency and more ways to fail.
4. Fine-tuning moved, and in one major case is going away
The fine-tuning landscape changed in two directions at once. On one hand, methods matured: beyond supervised fine-tuning (SFT) on example pairs, platforms offer direct preference optimization (DPO) on preferred-versus-rejected answers and reinforcement fine-tuning (RFT), where a grader scores model outputs and training optimizes for that score. Microsoft Foundry’s RFT documentation lists string-check, text-similarity, model-based, Python and multi-graders, and warns teams to test graders against correct, incorrect and malformed answers before training.
On the other hand, OpenAI announced in 2026 that it is winding down its self-serve fine-tuning platform. Its documentation states the platform is no longer open to new users; according to the notice shared on OpenAI’s developer forum, existing customers can create training jobs until January 6, 2027, and fine-tuned models remain available for inference only until their base models are deprecated. Whatever the strategic reasons, the lesson for enterprises is clear: a fine-tuned model hosted by a single provider is a dependency that can be withdrawn. Your training data, graders and evaluation sets are the durable asset; the weights may not be.
Reframe the Question: Knowledge, Behavior or Reasoning?
The cleanest way to choose is to name the gap between the base model and what you need. Every customization technique fixes a different kind of gap.
- Knowledge gap: the model does not know your facts, such as policies, product specs, contracts or tickets. Facts change and must be cited. This is a context problem, solved by putting the right information in front of the model: full context, RAG or agentic retrieval.
- Behavior gap: the model knows enough but does not respond the way you need, such as an exact output schema, a house tone, a classification taxonomy, or a refusal policy. Try prompt and context engineering first; if behavior is still inconsistent at volume, SFT or DPO is the right tool.
- Reasoning gap: the model must perform a specialized judgment you can score, such as coding a medical record, checking a contract clause against a playbook or choosing a next-best action. If good answers can be graded automatically, RFT can teach a model to reason toward them.
- Economics gap: a large model already does the task well, but cost or latency is too high at scale. Distilling a large model’s outputs into a smaller, fine-tuned model is a common answer.
Retrieval changes what the model knows at the moment of answering; fine-tuning changes how it behaves every time. Most enterprise failures come from using one to fix the other.
A practical corollary: fine-tuning is a poor way to inject facts. Facts learned in training cannot be cited, cannot be updated without retraining, cannot be permission-filtered per user, and tend to be recalled unreliably. If your documents change weekly or users have different access rights, the knowledge must live in context.
The Four Options Compared
Option A: Prompt and context engineering
Clear instructions, a few well-chosen examples, structured output schemas and careful ordering of information. It is cheap, fast to iterate and fully portable across models. OpenAI’s own model optimization guidance puts evals and prompting before any fine-tuning in its improvement loop. Use it as the baseline for every project.
Option B: Full context with caching
Load the entire relevant corpus into the prompt and cache it. Best for small, stable, high-value bodies of knowledge, such as a single product manual, a rulebook or one long contract. Strengths: no indexing pipeline, no retrieval misses. Weaknesses: context rot on long inputs, no per-user permission filtering inside the blob, cost that scales with corpus size on every uncached call, and a hard ceiling as the corpus grows.
Option C: RAG, classic or agentic
Index content and retrieve what is relevant per question. Best for large, changing, permissioned corpora where answers must cite sources. Modern RAG means hybrid retrieval, reranking, metadata filters (for date, product, region and access rights), and grounding prompts that require citations. Agentic RAG adds iterative search for complex questions. Weaknesses: an indexing pipeline to operate, chunking and freshness to manage, and retrieval quality that must be measured continuously.
Option D: Fine-tuning (SFT, DPO, RFT, distillation)
Change the model’s weights, through a managed platform or by training lightweight adapters such as LoRA on open-weight models. Best for consistent formats at very high volume, narrow classification and extraction, specialized graded reasoning, and moving work to a smaller model for cost, latency or on-premises deployment. Weaknesses: you need hundreds to thousands of high-quality examples or a reliable grader, retraining when base models change, hosting and lifecycle work, and the portability risk described above.
A Decision Framework You Can Use in an Architecture Review
Answer these questions in order. Stop at the first option that meets your quality bar on a representative evaluation set.
- Have you built an eval set? If not, stop. Without 50 to 200 representative questions with expected answers or grading rules, you cannot compare options. Every option below is judged against this set.
- Does prompt engineering alone reach the bar? Try a strong current model with good instructions, examples and a schema. Many “we need fine-tuning” projects end here.
- Is the gap knowledge? If the relevant corpus is small (comfortably within the window), stable and shared by all users, try full context with caching. If it is large, changing or permissioned, build RAG with hybrid search and reranking.
- Are questions multi-step or spread across systems? Add agentic retrieval with a bounded number of tool calls, and measure the extra cost and latency against the accuracy gain.
- Is behavior still inconsistent at volume? Consider SFT or DPO on curated examples, while keeping knowledge in context.
- Is the task a gradable judgment the model gets wrong? Consider RFT, but only after proving your grader cannot be gamed by superficial answers.
- Is the solution right but too expensive or slow? Distill into a smaller model, ideally open-weight so you control the weights and hosting.
Alongside the sequence, score each candidate on criteria that matter to the business, not just accuracy:
- Freshness: how quickly must new information appear in answers? Hours favor retrieval; quarterly favors anything.
- Traceability: must answers cite a source a reviewer can check? Only context-based approaches provide this natively.
- Access control: do different users see different documents? Retrieval with permission filters is the only robust pattern.
- Unit economics: cost per resolved request, including retries and agent loops, not cost per token.
- Latency budget: agentic retrieval may add several seconds; a small fine-tuned model may respond in a fraction of that.
- Portability: can you move to another model or provider in a quarter if prices, terms or availability change?
- Operating burden: who maintains the index, the training pipeline and the evals after launch?
Hybrid Patterns That Work in Production
The strongest systems in 2026 usually combine techniques, each fixing its own gap.
- RAG plus a small fine-tuned formatter: retrieval provides facts with citations; a fine-tuned small model turns them into a strict output schema at low cost.
- Fine-tuned retriever components: instead of fine-tuning the generator, fine-tune the embedding model or reranker on your domain’s query-document pairs. This often lifts retrieval quality more cheaply than any change to the LLM.
- Cached core plus retrieval tail: keep a stable core (policies, glossary, product taxonomy) cached in the prompt and retrieve the long tail of documents per query.
- Router across tiers: a lightweight classifier sends simple, high-volume requests to a small tuned model and complex ones to a frontier model with agentic retrieval.
Worked Example: A Claims Operations Assistant
Consider a generic mid-sized insurer that wants an assistant to help claims handlers answer coverage questions and draft decision letters. The corpus includes about 4,000 policy wordings across products and regions, a 60-page claims handling manual, regulator guidance, and the claim file itself.
The team’s first instinct is to fine-tune a model “on all our policies.” Applying the framework changes the plan:
- Coverage answers are a knowledge gap, and policy wordings change at every renewal, differ by region and must be quoted exactly. That points to RAG with metadata filters for product, version and jurisdiction, hybrid search so exact clause numbers match, and a reranker.
- The 60-page manual is small, stable and relevant to almost every request, so it is cached in full as part of the system prompt rather than chunked.
- Each claim file is retrieved through a tool call scoped to the handler’s permissions, so the model only sees claims the user is allowed to see.
- Decision letters must follow a strict template and tone, and the prompt-only baseline gets the structure right most of the time but not consistently. This is a behavior gap; after a few months of production use, the team curates approved letters into an SFT dataset for a smaller open-weight model with a LoRA adapter, which it hosts in its own cloud tenancy.
The eval set, built before any of this, includes 150 real (anonymized) handler questions with expected clauses and conclusions, plus 40 letters scored against a rubric. Each design change is accepted only if it improves those scores without raising cost per resolved case beyond the agreed budget. Because the letter model was trained on the team’s own dataset with open weights, a future model change means retraining an adapter, not losing a capability.
Pitfalls to Avoid
- Fine-tuning to fix bad retrieval. If the right passage never reaches the model, no amount of training helps. Measure retrieval recall separately from answer quality.
- Treating the context window as a database. Stuffing everything in degrades accuracy and drives cost. Curate what goes in and where it is placed.
- Skipping permission design. A shared index without access filters becomes a data leakage channel the first time someone asks the right question.
- Ignoring freshness. Stale indexes produce confident, outdated answers. Monitor index lag like any other pipeline SLA.
- Letting agents loop. Agentic retrieval without step limits and cost budgets can multiply token spend on hard questions.
- Betting on one provider’s fine-tuning service. Keep datasets, graders and evals in your own repository and be able to retrain elsewhere.
- Comparing options on demos. Ten hand-picked questions will favor whichever option was built last. Use the eval set.
Outlook: Where the Choice Is Heading
Several trends will keep shifting the balance over the next 12 to 24 months. Context windows and caching will keep improving, so the threshold below which “just put it in the prompt” works will rise, but context rot means curation remains a skill. Retrieval will become more agentic and more standardized through protocols such as MCP, making enterprise systems searchable tools rather than one-off pipelines. Fine-tuning will concentrate in two places: grader-driven reinforcement methods for specialized reasoning, and adapters on open-weight and small models where enterprises want control of cost, latency and data residency. Managed platforms will continue to change their offers, which makes portable assets, meaning datasets, rubrics and evals, more valuable than any single trained model.
Meanwhile, the evaluation set becomes the center of gravity. It is what lets you swap models, re-run decisions as prices fall, and prove to stakeholders that a change made things better rather than different.
Conclusion
The useful question is no longer “RAG or fine-tuning?” but “what kind of gap are we closing, and what is the cheapest, most portable way to close it?” Put knowledge in context, through full context for small stable corpora and retrieval for large, changing or permissioned ones. Use fine-tuning for behavior, graded reasoning and economics, and keep its inputs portable. Build the eval set first, and let it decide. Teams that work this way spend less, ship sooner and can change models without starting over.
How Hanoi Technology Can Help
Hanoi Technology helps teams build eval sets, retrieval pipelines and fine-tuned models that fit the gap they actually need to close, through projects, dedicated teams or managed services.