En
Fr
De
It
Es
Uk
RAG or fine-tuning: which one do you need?
Use RAG when the model has to answer from your own documents or data, especially if that knowledge changes or different users may see different parts of it. Use fine-tuning when you need a consistent style, format or narrow behaviour that prompting cannot get reliably, and you have hundreds to thousands of good examples. Most business chatbots and assistants need RAG and no fine-tuning at all. Before either, try a well-written prompt with a few examples: it solves a large share of the cases where teams assume they need fine-tuning.
The two are often presented as alternatives. In practice they solve different problems. RAG, short for retrieval-augmented generation, changes what the model sees for one answer. Fine-tuning changes how the model behaves on every answer. Our RAG implementation guide explains how a retrieval system is built; this guide is about the decision before that.
How do RAG and fine-tuning work, in one paragraph each?
RAG. Your documents are split into passages and indexed. When a user asks a question, the system finds the most relevant passages, adds them to the prompt and asks the model to answer from them, usually with citations. Update a document and the next answer uses the new version.
Fine-tuning. You prepare a set of example inputs with the ideal outputs and train the model on them. Hosted providers offer fine-tuning on some of their models; with open models you train an adapter such as LoRA yourself and host the result. The trained model then responds in the learned way without being shown the examples each time.
Which one fits your case? A five-question test
Answer these five questions about the feature you are planning:
- Does the answer depend on facts from your documents or databases? If yes, you need retrieval of some kind.
- Do those facts change more often than every few months? If yes, fine-tuning them into a model will go stale. RAG.
- Do different users have the right to see different documents? If yes, RAG with permission filters. A fine-tuned model cannot forget selectively.
- Is the problem how the model writes or decides, with the facts already in the prompt? If yes, try prompting with examples first, then fine-tuning.
- Do you have at least several hundred checked examples of the ideal output? If no, fine-tuning is not an option yet, whatever the problem is.
If you answered yes to 1, 2 or 3, start with RAG. If only question 4 is yes and question 5 is also yes, fine-tuning is a candidate. If question 4 is yes and you have no examples, the work starts with collecting them.
Ten business use cases and what they need
| Use case | Typical answer | Why |
|---|---|---|
| Support bot over help articles and policies | RAG | Content changes, answers need citations |
| Internal knowledge assistant for staff | RAG with permissions | Access differs by department |
| Sales assistant answering from product specs and price lists | RAG | Specs and prices change |
| Contract or document review against your rules | Prompt plus retrieval of the rule set | Rules change, reviewers need the source |
| Ticket or email classification into your categories | Prompt first, fine-tuning at high volume | A small tuned model can be cheaper per item |
| Extraction into a fixed JSON format | Prompt with schema, fine-tuning rarely | Structured output features usually suffice |
| Writing in a strict brand voice across many texts | Prompt with examples, then fine-tuning | Style is behaviour, not knowledge |
| A persona that writes like one specific person | Fine-tuning plus RAG memory | Voice from training, facts from retrieval |
| Domain language the base model handles poorly | Fine-tuning, often of an open model | Vocabulary and patterns, measured on a test set |
| Answers from live data such as orders or stock | Tool calls to your API | The data is in a system, not in documents |
The last row matters because it is often missed. When the answer lives in a database, neither RAG over documents nor fine-tuning is the right tool. The model should call your API, as an AI agent does.
What do RAG and fine-tuning cost?
Build cost depends mostly on your data and integrations, and running cost on volume and hosting. The table compares typical profiles for one business feature.
| RAG | Fine-tuning a hosted model | Fine-tuning an open model | |
|---|---|---|---|
| First version, with us | From $10,000 | Usually part of a larger project | Usually part of a complex AI product |
| Data work | Parsing, chunking, permissions, test questions | Hundreds to thousands of checked examples | Same, plus training setup |
| Training cost | None | Per training token, usually modest for a narrow task | GPU time, often tens to a few hundred dollars per run for a 7B to 14B model with LoRA |
| Running cost | Model API tokens plus a vector index, often in existing PostgreSQL | Per-token price of the tuned model, often higher than the base model | GPU hosting, roughly $700 to $3,000 a month per GPU running around the clock |
| Updating knowledge | Re-index a document | Retrain | Retrain |
| Time to first version | 3 to 6 weeks | 4 to 8 weeks including data | 6 to 12 weeks including hosting |
With Lytvynov Production a RAG assistant or AI chatbot starts from $10,000, and complex AI products that combine fine-tuning, RAG memory or custom models typically cost $60,000 to $150,000. A typical US or UK agency quotes roughly 2.5 to 4 times more for the same scope. For the token side of the running bill, our guide to OpenAI and Claude API cost prices three workloads line by line.
What data does each approach need?
For RAG you need the documents themselves in a form a parser can read, the permission rules that say who may see what, and 50 to 200 real questions with the correct answer and source. That question set is the most valuable asset of the project: it shows whether retrieval finds the right passage and whether the answer stays faithful to it.
For fine-tuning you need examples of the exact input and the exact ideal output, checked by someone who knows the domain. Common sources are past tickets with the final resolution, documents your team wrote by hand, or classifications staff made over the last year. Budget time for cleaning: removing duplicates, fixing wrong labels and dropping examples that contain personal data you should not train on.
When is a prompt enough?
More often than teams expect. Current models follow detailed instructions, a few worked examples and a required output schema well. Before fine-tuning, we try three things on the evaluation set:
- A clear instruction with your rules written out, including what to do when unsure.
- Five to twenty examples of ideal outputs placed in the prompt and cached, so they cost little on repeat calls.
- Structured output with a JSON schema for anything a machine will read.
If these reach the accuracy target, fine-tuning adds cost without benefit. If a gap remains and it is about consistency or style, fine-tuning is worth testing. If the gap is about missing facts, the fix is retrieval.
When does it make sense to combine both?
When a product needs both a voice and facts. A clear example from our work is an AI companion product built for a US startup: LoRA adapters trained on one person's messages give the replies that person's way of writing, and RAG memory with a fact graph supplies what they actually said and knew. The chat uses whichever layer is ready, so a conversation is possible before training finishes. Fine-tuning carries how the person writes; retrieval carries what they said.
Combining both also makes sense at high volume. A small fine-tuned model can classify or route requests at a fraction of the cost of a large model, while a RAG pipeline answers the requests that need knowledge.
Common mistakes
- Fine-tuning to add knowledge. The model learns the style of your documents and still invents details. Retrieval fixes this; more training does not.
- RAG without an evaluation set. Nobody can tell whether a change to chunking or ranking made answers better or worse.
- Ignoring permissions. An assistant that returns a passage from an HR document to the wrong person is a security incident.
- Fine-tuning on unchecked data. The model reproduces the mistakes in your past tickets as faithfully as the good answers.
- Choosing by trend. The right approach is the one that passes your test set at an acceptable cost per answer.
How we decide with clients
We are a software development company founded in 2020 that builds AI features, SaaS products and internal systems for companies in the US, UK and EU. Our own AI resume builder generates text for 50,000 monthly active users, and we have built products that combine LoRA fine-tuning with RAG memory.
The decision starts with your data and a set of real questions or tasks. We test prompting, retrieval and, where justified, fine-tuning on that set and recommend the cheapest option that meets the accuracy target. You get a fixed quote after a short scoping call. See our RAG development service for how we build retrieval systems that stay accurate in production.
