RAG vs. fine-tuning: how to give an LLM your company's knowledge
Large language models are trained on public text. They know nothing about your contracts, your product catalog, your support history, or the policy that changed last Tuesday. Every enterprise AI initiative eventually meets the same question: how do we make the model answer from our knowledge? There are two main answers - retrieval-augmented generation (RAG) and fine-tuning - and choosing wrong is expensive.
What RAG does
RAG keeps the model unchanged and changes what it sees. When a user asks a question, the system first searches your content - documents, tickets, database rows, wiki pages - for the most relevant passages, then places those passages in the prompt and asks the model to answer using them. The model becomes a reader and reasoner rather than a memory.
That architecture has consequences that matter in an enterprise:
- Freshness. Update a document and the next answer reflects it. No retraining.
- Citations. Because the answer was built from specific passages, you can show them. Users trust what they can verify, and auditors require it.
- Access control. Retrieval can be filtered by the user's permissions before the model ever sees a passage. A model that has been fine-tuned on confidential data cannot un-know it for a user who should not see it.
- Measurability. Retrieval quality and answer quality can be evaluated separately, which makes failures diagnosable.
What fine-tuning does
Fine-tuning continues training the model on your examples so its weights change. It is excellent at teaching behavior: a house style, a rigid output format, a specialized vocabulary, a classification task with thousands of labeled examples. It is poor at teaching facts. Models fine-tuned on documents still hallucinate details, cannot cite sources, and go stale the day after training.
Choosing between them
| Need | Best fit | Why |
|---|---|---|
| Answer questions over changing documents | RAG | Freshness, citations, permissions |
| Enforce a strict output schema or tone | Fine-tuning (or strong prompting) | Behavior, not knowledge |
| High-volume classification with labeled history | Fine-tuning a small model | Lower cost per call, higher consistency |
| Domain jargon the base model misreads | Both | Fine-tune for vocabulary, retrieve for facts |
| Data that must never leave your network | RAG with a private model | Retrieval and inference stay in your VPC |
Where RAG projects go wrong
RAG is simple to demo and hard to do well. The failures we see most often are not in the model but in the retrieval layer: documents chunked without regard to structure, tables and scanned PDFs silently dropped, embeddings that cannot distinguish "termination for cause" from "termination for convenience", and no evaluation set to catch any of it. A production RAG system needs document parsing that preserves structure, hybrid search (semantic plus keyword), reranking, permission filtering, and a gold set of a few hundred questions with known answers that runs before every release.
The SubNeta approach
We start with retrieval, because most enterprise value lives in knowledge that changes. We build the evaluation set with your subject-matter experts in the first two weeks, so accuracy is a number rather than an impression. Fine-tuning enters when the metrics show a behavior problem that prompting cannot fix - and by then, the retrieval pipeline that feeds it already exists.