Every team building on an LLM hits the same wall within weeks: the model does not know your product, your internal docs, your acronyms, or anything that happened after its training cutoff. The standard answer to that problem is a pile of three terms — prompt engineering, retrieval-augmented generation, fine-tuning — usually presented as if they were competing options at the same tier.
They are not the same tier. They change different things, cost wildly different amounts, and fail in different ways. Prompting changes what you tell the model. RAG changes what the model can see at query time. Fine-tuning changes how the model behaves. Most of the confusion dissolves once you stop asking which technique is best and start asking what, specifically, is missing from your model’s output.
This is a decision guide: what each lever actually does, the misconception that sends teams to fine-tuning first and burns months, the order a cost-conscious engineer should try things in, and the concrete scenarios where each lever is the right answer.
What each lever actually changes
Prompt engineering: instructions plus examples, zero infrastructure
Prompting covers everything you can do without building anything: a system prompt that sets role, rules, and tone; explicit output constraints; and in-context examples — the few-shot behavior described in the GPT-3 paper back in 2020 and still the highest-leverage trick in the book. Three good examples of the exact output you want routinely beat a page of abstract instructions. It is the only lever with genuinely zero infrastructure: you change a string, redeploy, and the iteration loop is measured in minutes. Its ceiling is real, though — the model can only work with what it already knows, and very long instruction lists degrade: models start dropping rules the way people skim contracts.
RAG: fresh and private knowledge at query time
Retrieval-augmented generation fetches relevant documents at query time and pastes them into the prompt before the model answers. It changes what the model can see: information that is private, larger than the context window, or newer than the training cutoff. The model’s weights never change. The knowledge lives in your document store, which means updating it is re-indexing a file, not retraining anything. I cover the pipeline mechanics — embeddings, chunking, vector search — in the RAG explainer; the short version for this post is that RAG’s failure modes concentrate in retrieval, not generation. When a RAG system gives a bad answer, it is usually because the right passage never made it into the prompt.
Fine-tuning: behavior baked into the weights
Fine-tuning continues training a model on your own input–output examples, shifting its default behavior: output format, tone, terseness, adherence to a schema, judgment on domain-specific classifications. In practice this is usually parameter-efficient tuning such as LoRA, which trains small adapter matrices instead of the full network. What you get is consistency without paying for it in prompt tokens on every request: the behavior is the model’s new default rather than an instruction it might ignore. What you do not get — and this deserves its own section — is reliable new knowledge.
The expensive misconception: fine-tuning does not add knowledge
The most common failure pattern looks like this: a team wants a model that “knows our documentation,” so they fine-tune on the docs. The result disappoints everyone. Fine-tuning is good at shaping how a model answers and bad at storing what it should know, because a few thousand training examples are nudging weights shaped by trillions of tokens of pretraining. The tuned model tends to pick up the vocabulary and cadence of your docs while still misremembering the specifics — which is worse than the base model, because now it sounds authoritative in your domain while being wrong.
Facts also change. Every price update, policy revision, or renamed feature means curating new data and running another training job. With retrieval, the same change is a one-file re-index that takes effect on the next query. The rule of thumb worth memorizing: knowledge belongs in retrieval, behavior belongs in weights. If the thing you want the model to absorb has a version number or a last-modified date, it should not be trained in.
The cost ladder, and why the order is prompt, then RAG, then fine-tune
The three levers form a ladder where each rung costs roughly an order of magnitude more effort than the one below it.
- Prompting costs minutes of iteration and some extra input tokens. There is no system to operate and no new failure surface.
- RAG is a real system: an ingestion pipeline, chunking decisions, an embedding model, a vector store, and — the part everyone underestimates — ongoing evaluation of retrieval quality. Expect days to weeks before it is trustworthy, and a permanent operational surface afterward.
- Fine-tuning is dominated not by GPU cost but by dataset curation: hundreds to thousands of high-quality examples, an evaluation harness to prove the tune helped, and the recurring tax of redoing the work when your base model is deprecated. A tuned model is pinned to a base that will be superseded, so you also give up the free ride of vendor model upgrades.
This is why the order matters. You exhaust the cheaper lever first not because it always wins, but because when it does you have saved weeks, and when it does not, its measured failures tell you exactly what the next lever has to fix. Teams that start with fine-tuning skip that diagnostic step and routinely end up solving a problem a better system prompt would have solved.
Decision scenarios: match the symptom to the lever
| Symptom | Right lever | Why |
|---|---|---|
| Support bot must answer from your product docs | RAG | Knowledge is private and changes often |
| Exact output schema, millions of calls | Fine-tune a small model | Consistency without a long instruction block per call |
| Brand tone and voice | System prompt first, fine-tune at scale | Style is cheap to instruct, cheap to iterate |
| Answers must reflect this morning’s data | RAG — never fine-tune | Weights are frozen at training time |
| Domain-specific classifier | Few-shot first, then fine-tune | Tune when examples outgrow the prompt |
Two of these deserve expansion. The format-at-scale case is the strongest genuine argument for fine-tuning: a small, cheap model tuned to one narrow task can replace a flagship model carrying a 2,000-token instruction block, and at millions of calls the savings are dramatic — though check your provider’s structured-output mode first, since schema enforcement at the API level has made a chunk of format-tuning unnecessary. Which tier of model to tune down to is the same capability question I walk through in the model-selection guide. The fresh-data case, meanwhile, is absolute: no amount of tuning puts this morning’s numbers into weights trained last month.
The fourth lever: long context plus prompt caching
The 2026 twist to this decision is that the classic tradeoff between prompting and RAG has a middle option. Flagship models from all three major vendors now take around a million tokens of context, and prompt caching makes it affordable to resend a large, fixed corpus on every request — on Claude, as of August 2026, cache reads cost about a tenth of the base input price (Anthropic’s caching docs have the exact multipliers). If your entire knowledge base is a few hundred pages, you can skip retrieval and put all of it in the prompt. That eliminates RAG’s dominant failure mode — the right passage not being retrieved — because the model sees everything.
This is not free at scale, and the arithmetic deserves five minutes of your time before you commit: I walk through how caching pricing works in the LLM pricing explainer, and you can put your own traffic numbers into the cost calculator on this site. The rule of thumb: if the corpus fits comfortably in context and query volume is moderate, try context-stuffing with caching before you build a retrieval pipeline. Build RAG when the corpus outgrows the window or the per-request token cost stops penciling out.
Combining the levers
These are not mutually exclusive, and mature systems usually compose them: retrieval supplies the facts, the system prompt supplies the voice and rules, and a fine-tuned small model handles a narrow high-volume step — all sitting on cached prompts. The combinations follow the same division of labor as the individual levers. RAG for content plus a prompt for style is the standard support-bot shape; fine-tune for format plus RAG for facts is the standard extraction-pipeline shape.
One caution when you combine retrieval with anything: a RAG pipeline pastes documents you may not fully control into the model’s prompt, and documents can carry adversarial instructions. That attack surface is the subject of the prompt injection post, and it gets more serious the moment your app has tools that act on the model’s output.
Evaluate before you reach for the next lever
The gate between rungs of the ladder is measurement. Before concluding that prompting is not enough, you need an evaluation set: fifty to two hundred real queries with graded expected outputs. It does not need to be fancy — golden answers plus spot checks, or a model grading against a rubric, is plenty. Without it, you cannot say whether RAG actually improved anything, and fine-tuning without an eval set is doubly absurd because the labeled examples you would need for evaluation are the same artifact as training data — if you cannot produce the first, you were never ready for the second.
If you cannot measure the failure, you cannot justify the next lever. In my experience the humbling, recurring discovery once the eval exists is that the prompt was fixable all along — the original instructions were ambiguous, the examples contradicted the rules, or the task needed to be split in two.
The short version
- Prompting changes what the model is told. RAG changes what it sees. Fine-tuning changes how it behaves.
- Fine-tuning teaches behavior, not knowledge. Anything with a last-modified date belongs in retrieval, never in weights.
- Try them in cost order: prompt, then long context with caching, then RAG, then fine-tuning — and let each lever’s measured failures justify the next.
- Combine freely in production: retrieval for facts, prompt for voice, a tuned small model for format at scale.
- No eval set, no next lever.