RAG vs Fine-Tuning — When Each Actually Wins

Every argument I've watched about RAG versus fine-tuning could have ended in ninety seconds with one question: is the thing your model gets wrong a knowledge problem or a behaviour problem?

Knowledge means it doesn't know something. Your pricing tiers, a customer's contract terms, last Tuesday's changelog. Behaviour means it knows perfectly well and responds badly: wrong tone, wrong shape, ignores half your formatting rules, writes four paragraphs when you asked for one line.

Knowledge is retrieval. Behaviour is fine-tuning. That resolves most real cases. The rest of this is the remainder.

Where retrieval genuinely wins

Facts that change. Anything with a version number, a date, or an owner. A retrieval index updates the moment you re-embed a document. A fine-tuned model updates when you rebuild a dataset and pay for another training run. If your knowledge moves weekly, that comparison isn't close.

Citation. RAG can show its work: here are the three chunks I used, here are the source documents. For anything a support agent, a lawyer, or a compliance reviewer touches, that's load-bearing. A model that absorbed a policy document into its weights can't tell you which sentence it came from, so when it's wrong you have no thread to pull.

Permissions. Retrieval respects access control naturally. Filter by tenant, role, or document ACL before the model sees anything. Weights have no permissions. Once a fact is in the weights it's there for every user who can reach the endpoint, which is an awkward conversation to have with an enterprise customer.

Deletion. A customer asks you to remove their data. From an index that's a delete statement. From a fine-tuned model it's a retraining job.

Where fine-tuning genuinely wins

Format conformance. If you need output that always matches a schema, always uses your field names, always emits the same five section headers, fine-tuning gets you closer than prompting does. Prompted models comply most of the time. Most of the time is a bad property for a parser.

Tone and voice. Teaching a model to sound like your product's writing, or to always answer in two sentences, or to never apologise, is exactly what fine-tuning is for. It's a style transfer problem and style is what weights are good at.

Prompt length and latency. This is the underrated one. A 2,000-token system prompt full of rules gets sent on every single call, and you pay for it every single call. Fine-tune those rules into the model and the prompt collapses to a couple hundred tokens. Lower cost per call, lower time-to-first-token, and on a high-volume endpoint that's real money.

Narrow classification. For routing, tagging, or intent detection, a small fine-tuned model will usually beat a large prompted one on both cost and latency, and often on accuracy too.

The thing people get wrong

Fine-tuning does not reliably teach new facts. This is the misconception that costs the most.

Training on a few hundred question-answer pairs about your product doesn't install those answers as retrievable facts. It teaches the model the shape of answers about your product. Ask a question near your training data and it may get it right. Ask one slightly outside and it will confidently produce something in exactly the right format that is completely wrong.

That's worse than an obvious failure. A model that says "I don't know" is a support ticket. A model that invents a plan limit in your house voice is a refund.

What each actually costs

RAG's costs are ongoing and operational. Embedding is cheap per document but you re-embed on every content change, and you re-embed the entire corpus if you ever switch embedding models. You pay for vector storage. You add 50–300ms of retrieval latency to every request. And chunking strategy, hybrid search, and reranking turn out to be most of the work — the model is rarely the weak link, the retrieval is. I've written separately about picking a vector database without turning it into a six-week evaluation.

Fine-tuning's cost is front-loaded and human. The training run itself is often the cheapest part; a small model on a few thousand examples is tens of dollars. The expensive part is building the dataset. Several hundred to a few thousand examples, each one written or reviewed by somebody who knows what a correct answer looks like. Then the base model gets deprecated, or a better one ships, and you do it again.

Evaluation is the line item nobody budgets

Both approaches need a way to tell whether a change made things better. Nearly every team I've seen skips this and ships on vibes.

Build a set of 50 real inputs with known-good outputs before you change anything. Run it before and after. Without it you can't tell whether your new chunking strategy helped or you just got a good demo. It's the only thing that makes either approach improvable rather than a slot machine.

The one I got wrong

An internal support assistant kept handing out stale pricing and wrong plan limits. I decided it needed fine-tuning, and spent the better part of two weeks pulling roughly 600 question-answer pairs out of old tickets and cleaning them up.

The result answered in a perfect house voice and still got the plan limits wrong, now with more confidence than before. The actual fix took two days: index the pricing docs and the changelog properly, retrieve on every question, cite the source. The fine-tune was solving a tone problem I never had.

I'd built the expensive thing because it felt like the serious engineering choice. It wasn't. It was the wrong diagnosis executed well.

The order I'd actually work in

Prompt engineering first, and take it seriously — most "we need to fine-tune" conversations end when someone finally writes a good prompt with real examples in it. Retrieval second, and spend your effort on chunking and hybrid search rather than on which vector store. Build the golden set. Only then consider fine-tuning, and only when you can name the number it's supposed to move.

Fine-tuning is mostly an optimisation, not a capability. Every team I've seen get value from it had already shipped a working prompted system, knew exactly what it cost per call, and was fine-tuning to make that system cheaper, faster, or more consistent. Nobody fine-tuned their way to a product that didn't work yet. If you're reaching for it before you've shipped anything, you're solving the wrong problem in the most expensive available way. Start with a working prompted chatbot and let the bills tell you what to fix.