RAG or Fine-Tuning? A Decision Framework

"Should we fine-tune it on our data?" is usually the first question asked once an organisation decides to build something with a language model. It is also usually the wrong first question, because it presumes a diagnosis. Fine-tuning and retrieval-augmented generation address different deficiencies, and choosing between them is mostly a matter of naming which deficiency you actually have.

The distinction that matters

A useful way to separate them: retrieval changes what the model knows; fine-tuning changes how the model behaves.

If your system gives wrong answers because the model has never seen your product catalogue, your policies or last quarter's pricing, that is a knowledge problem and retrieval is the direct fix. If it gives answers that are factually fine but in the wrong format, register or structure, ignoring your schema, refusing tasks it should accept, producing prose where you need JSON. That is a behaviour problem, and retrieval will not touch it.

Fine-tuning on a corpus of documents to make a model "know" them is the most common and most expensive mistake in this area. It is an unreliable way to store facts and a very reliable way to spend money.

Where retrieval wins

• Content changes. Anything that updates on a cadence faster than you want to retrain. Prices, policies, inventory, documentation, regulations. Updating a retrieval index is a write; updating a fine-tune is a training run and a revalidation.
• You need to show your working. Retrieval returns the passages the answer drew on, so a reviewer can audit the path from source to response. A fine-tuned model cannot tell you which of its training examples produced an answer. In regulated settings this alone decides it.
• Access control varies by user. Retrieval can filter the candidate set per requester. A fine-tuned model has absorbed everything it saw in training and cannot selectively forget for one user.
• The corpus is large. Millions of documents are cheap to index and expensive to train on.

Where fine-tuning wins

• Output structure must be reliable. If you need a specific schema on every call and prompting gets you to 95% conformance, fine-tuning closes the remaining gap far more reliably than a longer prompt.
• Domain register and vocabulary. Clinical, legal and industrial language have conventions a general model approximates. Fine-tuning teaches the idiom.
• Latency and cost at volume. A fine-tuned smaller model can replace a larger model plus a long prompt. At high request volumes this can dominate the economics.
• Tasks that resist description. Some judgements are easier to demonstrate with a thousand examples than to specify in a prompt.

A decision sequence

In practice we work through this in order, and most projects stop at step two or three:

1. Establish an evaluation set first. Fifty to two hundred real cases with agreed correct answers. Without this you cannot tell whether any subsequent change helped, and every later decision becomes taste.
2. Try careful prompting with good context. A surprising share of projects end here. It costs days.
3. Add retrieval. If failures are knowledge failures, the model doesn't know a fact, or is inventing one. This is the fix. Most enterprise use cases are here.
4. Fix the retrieval before blaming the model. Poor chunking, weak embeddings, no reranking and a naive top-k will produce bad answers from a perfectly good model. This is where most "RAG doesn't work for us" conclusions actually come from.
5. Fine-tune for behaviour that survives all of the above. Structure, format, register, refusal behaviour.

Why step four deserves its own paragraph

Retrieval quality is where most of the engineering actually lives, and it is consistently underestimated. The failure modes are specific and fixable:

• Chunking that severs context. A clause split from its definition, a table separated from its header, a procedure step without its precondition. Chunk on document structure, not on a fixed token count.
• Pure vector search on keyword-shaped queries. Part numbers, policy codes and names are exact-match problems. Hybrid retrieval, lexical plus semantic, outperforms either alone on most enterprise corpora.
• No reranking. Retrieving twenty candidates and reranking to five is usually a larger quality gain than any change to the generating model.
• No evaluation of retrieval separately from generation. If you only measure the final answer you cannot tell whether the model reasoned badly or was handed the wrong passages.

They are not exclusive

The mature pattern in production is usually both: retrieval supplies current, access-controlled, citable knowledge, and a light fine-tune enforces output structure and domain register. They are addressing different layers, and using one does not exclude the other.

What we would push back on is starting with the fine-tune. It is the option with the longest feedback loop, the highest cost to revise, the weakest audit story and the greatest chance of being the wrong tool for the failure you actually have.

The short version