LLM Fine-Tuning vs RAG: Decision Guide Interview Prep

RAG vs KB-Routing vs Fine-tuning vs Hybrid — Eight Decision Axes

Sample interview questions

1. A product team is building a customer-support agent. Refund-policy questions account for 60% of traffic; the rest are open-ended. Which architecture is the production default for mixed-intent traffic?
   • A. Pure RAG over the entire knowledge base — consistent retrieval for all query types.
   • B. Pure KB-Routing with a template for every possible query — maximum predictability.
   • C. Hybrid: route predictable refund-policy queries to templates (KB-Routing); fall through to RAG for the long-tail open-ended queries. ✓
   • D. Fine-tune the model on past support transcripts so it learns both response styles.

   Option C is correct. The production-default hybrid pattern routes predictable intents to predetermined templates (KB-Routing) and falls through to RAG for everything else. KB-Routing on the 60% deterministic majority reduces cost, latency, and hallucination risk compared with synthesizing the same canned answer through RAG each time. RAG handles the unpredictable 20-40% long tail, where synthesis capability is actually needed. Option A is wrong. Running every query through RAG burns LLM-generation cost and context budget on queries whose answer is already known and unchanging — this is the over-RAG failure: reaching for retrieval-and-synthesis when a template lookup would do. Option B is wrong. Template count is finite and maintainable up to roughly 30-100 intents; beyond that, the maintenance burden collapses. The under-RAG failure is the mirror image — trying to template the long tail fails when novel queries inevitably arrive. Option D is wrong. Fine-tuning changes model weights to improve synthesis or reasoning patterns, not to encode answers to specific policy questions. Training on past transcripts does not help the model stay current as the refund policy changes over time. Production reality: most customer-support agents, banking assistants, and regulated-domain bots use this hybrid pattern. The router's confidence threshold is the single most important tuning knob — tune it against a held-out intent set to balance false-routing rate against template-coverage.

2. What is the key behavioral difference between RAG and KB-Routing that makes them operationally distinct — even though both retrieve from a knowledge base and end with an LLM call?
   • A. RAG uses dense embeddings; KB-Routing uses BM25 sparse retrieval.
   • B. In RAG the LLM synthesizes new content from retrieved passages; in KB-Routing the LLM follows a predetermined template, filling variable slots but not composing freely. ✓
   • C. RAG requires a vector database; KB-Routing works with any SQL database.
   • D. KB-Routing is an older term for RAG — they are the same pattern with different names.

   Option B is correct. This is the central operational distinction: RAG retrieves content chunks and asks the LLM to synthesize a new answer from them — every answer is composed at runtime, high output variability, high flexibility, and hallucination risk from generation. KB-Routing retrieves a predetermined template (a response shape with variable slots) and asks the LLM to follow it — the output shape is controlled, deterministic for the same intent, low hallucination risk, and fast. Same retrieval substrate; completely different post-retrieval behavior. Option A is wrong. Both patterns can use dense embeddings or BM25 in their retrieval layer. The choice of retrieval algorithm is orthogonal to whether the post-retrieval step is synthesis or template-following. Option C is wrong. Both patterns can use any backend (vector databases, SQL, YAML registries). Database type does not determine which pattern applies. Option D is wrong. Treating them as synonyms is a costly conflation. A system designed for RAG will be blindsided when the LLM produces fluent-but-unsupported answers; a system designed for KB-Routing will fail on novel queries the template library never anticipated. Production reality: KB-Routing produces structurally similar answers for queries with the same intent — two identical-intent queries always get the same template-shaped response. RAG may produce slightly different wording each time. That predictability difference is what makes KB-Routing appropriate for regulated or compliance-sensitive outputs.

3. RAGAS faithfulness measures whether a RAG answer is supported by retrieved passages. What critical quality dimension does faithfulness NOT cover?
   • A. Faithfulness measures real-world factual correctness. It misses grammatical quality.
   • B. Faithfulness measures whether the answer stays within the bounds of the retrieved context. It does not measure whether the retrieved passages themselves are correct, current, or complete. ✓
   • C. Faithfulness and context precision are the same RAGAS metric — the distinction is only terminology.
   • D. Faithfulness measures citation formatting. It misses semantic relevance to the query.

   Option B is correct. RAGAS faithfulness is a retrieval-conditioned consistency metric: it checks whether each factual claim in the generated answer can be attributed to a retrieved passage. If the retrieved passages are outdated, incorrect, or incomplete, a high faithfulness score tells you the generator did not hallucinate beyond them — not that the answer is actually correct. A system running RAGAS faithfulness on a stale or wrong knowledge base will score highly and still mislead users. Option A is wrong. Faithfulness is not a real-world factual correctness measure. A grammatically perfect answer that introduces facts not in the retrieved passages will score low on faithfulness; a factually wrong answer that faithfully reproduces its (wrong) retrieved context will score high. Option C is wrong. RAGAS separates faithfulness, answer relevance, context precision, and context recall as distinct metrics. Context precision measures whether retrieved passages were actually relevant to the query — a different question from whether the generator stayed within them. Option D is wrong. Citation formatting is not what faithfulness measures. Faithfulness checks semantic grounding (each claim traceable to a retrieved passage), not formatting. Production reality: faithfulness should be paired with a knowledge-base freshness audit. High faithfulness on a stale KB is a false confidence signal. Run faithfulness as both a CI gate and an online sampled metric, and track it per model release — a model upgrade can quietly shift faithfulness because the new model leans harder on its parametric priors.

4. When is adding a cross-encoder reranker to a RAG pipeline most justified?
   • A. Always — rerankers improve results in every RAG pipeline regardless of current precision.
   • B. When retrieval recall is high but precision is low: the initial retrieval returns many candidates, but many are only loosely relevant, and the LLM needs a cleaner top-k to reduce prompt noise. ✓
   • C. Only when using pgvector — other vector databases handle reranking automatically.
   • D. When the knowledge base exceeds 100,000 documents.

   Option B is correct. A cross-encoder reranker (Cohere Rerank or a fine-tuned cross-encoder) re-scores each candidate by passing the query and document through the model together and emitting a single relevance score — no separate embedding (ColBERT, by contrast, is a late-interaction model, not a cross-encoder). This is more accurate than the bi-encoder similarity scores used in first-stage retrieval, but with higher latency (typically 100-300ms for a cloud API reranker). The cost is worth paying when first-stage retrieval has high recall (right documents in the candidate set) but low precision (many irrelevant documents mixed in). The reranker surgically re-orders, letting the LLM see a cleaner top-k. Option A is wrong. When first-stage precision is already high (the top-5 retrieved documents are already the right ones), a reranker adds latency with no quality benefit. Reranking cannot create relevance where it does not exist in the candidate set. Option C is wrong. pgvector does not have any special relationship with reranking. All vector databases serve candidate sets that can be fed to a reranker; the choice of vector store does not determine whether reranking is needed. Option D is wrong. Knowledge base size affects retrieval cost and index strategy, not whether reranking is valuable. A small KB with poor first-stage precision may need a reranker far more than a large KB with well-calibrated embeddings. Production reality: measure precision@k before committing to a reranker. If precision@5 is already 0.8+, the added latency is hard to justify. If precision@5 is 0.4-0.6 and recall@20 is 0.9+, a reranker is a high-value addition. Chunking and reranker tuning are the two most-overlooked RAG levers — teams reach for fine-tuning while leaving both untouched.

5. A RAG system retrieves the correct documents for a query but the final answer still contains claims not present in any retrieved passage. What is the failure?
   • A. The embedding model is not capturing query semantics correctly.
   • B. The generator LLM is not sufficiently constrained to ground its answer in retrieved context — the system prompt instruction to stay within provided passages is absent, weak, or overridden by the model's parametric knowledge. ✓
   • C. The vector database index is corrupted and returning wrong results.
   • D. The context window is too small to fit all retrieved passages.

   Option B is correct. When retrieval is verified correct (right documents in context) but the model still produces claims beyond them, the failure is in the generation constraint: the model is drawing on its parametric knowledge (training-data weights) to supplement or embellish the retrieved information rather than staying within it. This happens when the grounding instruction ("answer only based on the provided context") is absent, too weak, or overridden by the model's training distribution. The fix is a stronger grounding instruction, combined with faithfulness scoring (RAGAS or HHEM-2.1) that penalizes claims not attributable to retrieved passages. Option A is wrong. The premise states retrieval is correct — the right documents are being retrieved. The embedding model is not the failure point here. Option C is wrong. A corrupted index would cause wrong or missing documents, not hallucination beyond correct retrieved context. Option D is wrong. A context window too small to fit all passages is a truncation problem — some documents are cut off — which is a different failure mode from generation beyond correct context. Production reality: "stay within the context" grounding instructions need to be explicit, tested, and measured. Run RAGAS faithfulness on a held-out set where you have verified the retrieved context is correct. If faithfulness is low despite correct retrieval, strengthening the grounding instruction is the highest-leverage fix before reaching for fine-tuning.

6. What is the standard diagnostic for measuring whether a RAG pipeline has reached its performance ceiling before committing to fine-tuning?
   • A. If the RAG system achieves 80% accuracy on a test set, it has hit its ceiling.
   • B. Oracle retrieval ablation: inject known-correct documents directly into the prompt, bypassing retrieval entirely, to isolate whether failures are in retrieval or generation. ✓
   • C. Run 10 manual test queries. If 3 or more fail, fine-tuning is needed.
   • D. Compare RAG performance against a fine-tuned model baseline on a 100-example set.

   Option B is correct. The oracle retrieval ablation bypasses the retrieval component entirely by injecting the known-correct document(s) directly into the LLM prompt. This isolates generation from retrieval. If oracle-context performance is significantly higher than end-to-end RAG performance, retrieval is the bottleneck — optimize the embedding model, reranker, or chunking strategy before touching weights. If oracle-context performance is close to end-to-end performance, retrieval is working but the generator is the bottleneck — at that point, fine-tuning on domain-specific synthesis patterns may genuinely help. Option A is wrong. An 80% accuracy number without an oracle ablation does not reveal whether the remaining 20% failures are in retrieval or generation. You cannot diagnose the bottleneck from end-to-end accuracy alone. Option C is wrong. 10 manual test queries is too small for statistical reliability, introduces selection bias, and provides no decomposition into retrieval vs generation failures. Option D is wrong. A comparison against a fine-tuned baseline tells you which system currently performs better, but not whether RAG has headroom before it plateaus. Teams that skip this step and jump to fine-tuning often discover they fine-tuned around a retrieval problem that a better embedding model or reranker would have fixed at far lower cost. Production reality: the oracle ablation is the highest-leverage diagnostic step in RAG engineering. Run it before any significant investment in fine-tuning, reranker evaluation, or retrieval-algorithm changes.

7. A KB-Routing system is giving customers confidently wrong answers about refund eligibility. Given the KB-Routing failure-mode taxonomy, what is the most likely root cause?
   • A. The LLM is hallucinating beyond the retrieved template — this is the standard RAG hallucination failure.
   • B. The router is mis-classifying query intent OR the matched template is stale — the two primary KB-Routing failure modes. ✓
   • C. The vector database embedding index is out of date.
   • D. The system prompt is too long, causing the model to forget the template.

   Option B is correct. RAG and KB-Routing fail differently. In KB-Routing, the two primary failure modes are: (1) routing precision failure — the intent classifier selects the wrong template for the query; and (2) stale templates — the template was written when the policy was different and was never updated. A confidently wrong answer about refund eligibility, structurally well-formed, is the KB-Routing signature failure: a plausible template response to the wrong question, or the right template with outdated data. This is harder to detect than RAG hallucination precisely because the response looks well-formed. Option A is wrong. RAG hallucination (generating claims beyond retrieved context) is a different failure mode. KB-Routing does not ask the LLM to synthesize beyond the retrieved template — it follows it. The failure here is in the template path, not in free-form generation. Option C is wrong. KB-Routing retrieves templates, not content chunks for synthesis. Outdated embedding indices would affect which template is matched (routing precision), not the template content itself. Stale templates are the more specific and likely cause. Option D is wrong. Context-length issues cause truncation or repetition, not structurally coherent but factually wrong answers. Production reality: the fix for stale templates is to link each template to a source-of-truth document with a last_reviewed date, and have CI fail when the template is older than its source. Version templates in git, review changes via PR, and run CI checks on length and completeness.

8. When should a team escalate from single-pass RAG to agentic RAG (Self-RAG, CRAG, or LangGraph agentic loops)?
   • A. Immediately — agentic RAG always outperforms single-pass RAG.
   • B. When the knowledge base exceeds 1 million documents.
   • C. When single-pass quality has plateaued on measured failures, multi-hop questions are common, or retrieval quality varies materially by query and requires self-evaluation. ✓
   • D. Agentic RAG is only useful for code-generation tasks, not knowledge Q&A.

   Option C is correct. There is a three-step escalation ladder: (1) single-pass RAG — cheapest, sufficient for most queries; (2) agentic RAG — when single-pass quality plateaus, multi-hop questions are common, or retrieval quality varies materially by query; (3) GraphRAG/hybrid — when cross-document relationship traversal is a common query pattern. Agentic RAG (Self-RAG, CRAG, LangGraph agentic retrieval) adds latency and cost by allowing the LLM to decide when to retrieve, what to query, and how many rounds to run. Start at (1) and escalate only on measured failures, not anticipated ones. Option A is wrong. Agentic RAG adds latency and cost; on single-hop factual queries where single-pass retrieval is already high quality, the agentic loop produces no improvement and degrades user experience. Option B is wrong. Knowledge base size is not the escalation criterion. Large but single-hop knowledge bases work well with single-pass RAG. Small knowledge bases with multi-hop relationship questions may need agentic RAG. Option D is wrong. Agentic RAG is used across knowledge Q&A, research summarization, engineering on-call assistants, and other domains. Code generation is not a distinguishing criterion. Production reality: measure single-pass quality on your actual query distribution first. Run the oracle retrieval ablation. Only if oracle-context performance is high but end-to-end performance plateaus does agentic multi-pass retrieval become worth the latency and cost.

Frequently asked questions

What is the fundamental question that determines whether to use RAG or KB-Routing?

One question separates them: do you want the LLM to compose freely, or to follow a script? If the answer is open-ended and every response should be different, use RAG — the LLM synthesizes new content from retrieved passages. If the answer is one of a finite, known set of policies, FAQ entries, or workflows, use KB-Routing — the LLM follows a predetermined template, filling variable slots without free-form generation. Most production systems use both: KB-Routing for the predictable majority (fast, cheap, deterministic), RAG for the open-ended long tail (flexible, higher cost, higher hallucination risk). The hybrid pattern routes by intent confidence, falling through to RAG when no template matches above the confidence threshold.

When does RAG beat fine-tuning for adding domain knowledge?

RAG wins when the knowledge changes frequently (new documents are immediately available without retraining), when verifiability is required (each claim can be traced back to a retrieved passage), or when data volume makes training expensive or leads to catastrophic forgetting. Fine-tuning wins when the goal is behavioral change — adapting output style, response format, reasoning patterns, or domain terminology — rather than knowledge injection. The practical rule: start with RAG and measure retrieval recall and answer faithfulness. Fine-tune only if RAG plateaus on generation quality after retrieval is already working well.

What are the primary failure modes of KB-Routing, and how do they differ from RAG failures?

KB-Routing fails in two ways: (1) routing precision failure — the intent classifier maps the query to the wrong template, producing a confident and well-formed but irrelevant answer; and (2) stale templates — a template written months ago no longer reflects the current policy, and queries get the old answer silently. These are harder to detect than RAG hallucination because KB-Routing responses look structurally clean even when wrong. RAG fails via hallucination (claims beyond retrieved context) and retrieval miss (wrong or missing documents). The fix for KB-Routing is to version templates like code, link each to a source-of-truth with a last-reviewed date, and run CI checks when templates diverge from their source.

What is the router confidence threshold, and why is it the load-bearing knob in a hybrid pipeline?

The confidence threshold on the intent classifier is the value below which a query falls through from KB-Routing to the RAG fallback path. Too high a threshold and edge-case queries get wrong template responses; too low and the template path is bypassed for queries it would handle well. Tune this threshold against a held-out validation set, targeting router precision around 95% — measuring false-routing rate on misclassified queries. Open-source routers such as semantic-router provide calibrated per-route thresholds out of the box. Always define a RAG fallback; a router without a fallback produces silent failures on out-of-distribution queries.

How do you measure whether a RAG pipeline has hit its performance ceiling before committing to fine-tuning?

Use oracle retrieval ablation: inject the known-correct document(s) directly into the LLM prompt, bypassing retrieval entirely, and measure answer quality. If oracle-context performance is significantly higher than end-to-end RAG performance, retrieval is the bottleneck — optimize the embedding model, chunking strategy, or add a reranker. If oracle-context performance is similar to end-to-end performance, retrieval is working but the generator is the bottleneck, and fine-tuning on domain-specific synthesis patterns may help. Teams that skip this diagnostic and jump to fine-tuning frequently discover they fine-tuned around a retrieval problem that better embeddings or a reranker would have fixed at far lower cost.

What is domain-stratified RAG, and when should you use it?

Domain-stratified RAG routes every query through RAG, but a domain classifier first fans the query out to a domain-specific RAG pipeline — legal queries to a legal-only retriever, medical queries to a medical-only retriever, and so on. Each domain pipeline has its own chunk corpus, embedding model, reranker, and faithfulness thresholds. This reduces the retrieval search space by 5-50x on large, stratified knowledge bases, makes per-domain tuning feasible, and prevents cross-domain retrieval noise. Use it when the knowledge base is large and stratified and per-domain precision matters more than the overhead of maintaining separate pipelines. For homogeneous knowledge bases, a single pipeline is simpler and sufficient.

What is agentic RAG, and when does single-pass RAG justify escalating to it?

Agentic RAG gives the LLM control over the retrieval loop — deciding when to retrieve, what to query, and when it has enough context. Variants include Self-RAG (the model emits special tokens to request retrieval and critiques retrieved content), CRAG (retrieves, evaluates quality, falls back to web search if KB is insufficient), and LangGraph agentic loops (the retriever is a tool the agent calls). Escalate from single-pass only when measured failures are in multi-hop questions requiring multiple retrieval rounds, when retrieval quality varies materially by query and self-evaluation helps, or when single-pass quality has plateaued after thorough optimization. Start at single-pass and escalate on evidence, not anticipation.

Why is it a mistake to chunk templates into the RAG corpus and let retrieval pick which template to use?

Template selection requires intent classification — mapping a query to the closest known behavior in a finite set. Embedding models are trained for semantic similarity across general text, not for disambiguating between behavioral templates that may look similar on surface text but differ in what workflow they trigger. Stuffing templates into a chunk corpus "and letting RAG figure out which template applies" produces poor routing precision because the embedding space was not built for that purpose. Use a dedicated intent classifier (rule-based, embedding-based semantic router, classifier model, or small LLM classifier) for routing, and a separate RAG pipeline for content retrieval. Conflating them degrades both.

What does the hybrid route-then-RAG pattern look like in production, and what OSS tools implement it?

The hybrid pattern routes by intent confidence at the front of the pipeline. A lightweight classifier (semantic-router, LangChain RouterChain, or LlamaIndex Router Query Engine) maps each query to a known intent above a confidence threshold and sends it to the matching template path; queries below the threshold fall through to a full RAG pipeline backed by LlamaIndex, LangChain, or Haystack. The control flow is small — a route_then_template() step returns a template response or "FALLBACK", and on FALLBACK you call rag_answer(). Most production customer-support agents, banking assistants, and engineering on-call bots run this architecture: templates for the 70-80% predictable volume, RAG for the 20-30% long tail.

If a model has a 128K or 1M context window, do you still need RAG, or can you paste the whole corpus in?

Long context and retrieval solve overlapping problems, and the right answer depends on corpus size, task type, and the model's actual long-context ability. For a corpus that fits in the window, pasting it in (sometimes called "cache-augmented generation") removes the retrieval failure mode entirely and can match or beat RAG — the 2025 LaRA benchmark found no universal winner across 32K and 128K contexts and four QA task types. But long context does not scale: every query re-pays the token cost of the full corpus, latency climbs with input length, and models degrade on facts buried mid-context (the "lost in the middle" effect). RAG stays cheaper per query, scales past any window, and is auditable via citations. Practical rule: if the relevant material is small and changes rarely, long-context prompting is the simplest baseline; once the corpus is large or per-query cost matters, retrieval wins.

Should I try prompt engineering before RAG or fine-tuning?

Yes — it is the cheapest rung and frequently sufficient. Many failures attributed to "the model doesn't know our domain" are really weak instructions: no explicit grounding directive, no few-shot examples of the desired format, or a system prompt that lets the model fall back on its training priors. Fixing the prompt costs an afternoon and zero infrastructure. The escalation order is prompt engineering → RAG or KB-Routing (when the model needs facts it was never trained on, or facts that change) → fine-tuning (when the bottleneck is durable behavior — tone, schema compliance, reasoning style — that no amount of context injection fixes). Skipping the first rung is how teams end up fine-tuning around a problem a better prompt would have solved.

How do you keep RAG answers verifiable with citations, and why does it matter?

Require the LLM to emit explicit citation markers (e.g., [chunk 2] or [doc-id]) in a structured format, then have downstream code verify that every claim carries one. Without enforced citations, RAG answers are unverifiable — you cannot detect when the model is blending retrieved facts with parametric knowledge. An adversarial faithfulness critic that checks whether cited chunks actually support each claim is the next layer; RAGAS faithfulness and Vectara's HHEM open hallucination-evaluation model automate it in CI and online production sampling. Treat citation-presence as a structural quality gate, not a dashboard metric — a RAG answer without citations should fail the gate regardless of how fluent it reads.

Related topics