A shipped RAG chatbot that makes things up is usually retrieving the wrong text, not inventing it. The model read what it was handed and summarised it faithfully. Fix the retrieval path first: document hygiene, chunking, search, ranking, permissions. Prompt rewrites and model swaps come last, after you've proved the right passage reached the context window.
This isn't the post about choosing between RAG, fine-tuning and prompt engineering. We wrote that one already, and it's the right read if the architecture is still open. Here the decision is behind you. The bot is live, and somebody in sales has screenshotted a wrong answer into a group chat.
in the fix checklist, ordered as a diagnosis rather than a menu
candidates retrieved and re-ranked versus chunks sent to the model
golden-set questions enough to gate every index rebuild
Why does a chatbot that passed its demo hallucinate in production?
Two failure modes look identical to the user. Retrieval failure means the correct passage never entered the context window. Generation failure means the passage was there and the model ignored it, misread it, or stretched it past what it said. One test separates them in about ten minutes.
Take the question that produced the bad answer, find the paragraph in your sources that genuinely answers it, and paste that paragraph into the prompt with retrieval bypassed. Ask again. A correct answer means your search is broken. A still-wrong answer is a generation problem.
The first outcome is far more common. Instruction-tuned models rarely contradict text sitting in front of them, but they can't know the text is last year's refund policy. The bot didn't hallucinate. It summarised the wrong chunk. The ratio in your system is a measurement, not a statistic to look up: run the test across twenty bad answers and count which side they land on. Either way, you can't run it without a retrieval log, so build one first.
What silently poisons a RAG index?
Index poisoning is usually mundane. Superseded documents sit beside current ones, near-duplicates compete for the same query, scanned PDFs with no text layer ingest as empty pages, and tables get flattened into a run-on string of numbers with no headers attached. The retriever can't tell stale from current.
The version problem burns people most. Someone dropped pricing-policy-v3.pdf and pricing-policy-v7.pdf into the same folder three years apart. Both are about pricing, both score well, and your bot now quotes a discount you retired in 2023.
Scanned files are the quiet failure. A 40-page contract that's actually 40 images ingests cleanly, yields almost no text, and contributes nothing until someone asks about clause 14. Run OCR at ingest and assert on the result: under a few hundred characters per page, fail loudly rather than writing an empty chunk. Tables need the same discipline, converted to Markdown with the header row attached to every row.
Is your chunking splitting answers in half?
Chunking decides what one retrieved unit contains. Chunks that are too small drop the qualifying sentence that changes the meaning. Chunks that are too large dilute the embedding and bury the answer inside unrelated text. Overlap exists so a sentence spanning a boundary survives in both neighbours. Splitting inside a table or a clause corrupts both halves.
Fixed-size character splitting is the tutorial default and the wrong default for real documents. A 500-character window doesn't know it just cut a procedure mid-step, or separated "customers on annual plans" from the exception that follows. Split on structure first: headings, sections, list boundaries, table edges.
Then prefix every chunk with its document title and heading path, so "This does not apply to enterprise accounts" carries the fact that it came from Refund Policy, Section 3.
How do you fix the retrieval layer itself?
Four dials control retrieval quality: the embedding model, whether keyword search runs alongside semantic search, whether a cross-encoder re-ranks candidates before they reach the model, and how many chunks you pass forward. Pure vector search reliably misses exact identifiers such as SKUs, error codes and policy numbers, because embeddings blur rare tokens.
Hybrid search is the highest-value change most teams haven't made. Run BM25 and vector search together, fuse the lists, and a user asking about error E-4412 gets the page containing that literal string. Then re-rank: a cross-encoder scoring your top 30 candidates reorders them in ways cosine similarity can't. Lift from re-ranking is dataset-specific, so measure it on your own evaluation set rather than borrowing a figure from someone else's.
Use this table to get from what users report to what you actually change.
| Failure symptom | Likely cause | Fix |
|---|---|---|
| Confidently wrong answer, reads plausibly | Wrong chunk retrieved, summarised faithfully | Log chunks; check where the correct one ranked |
| Misses exact codes, SKUs, policy numbers | Semantic search blurs rare tokens | Add BM25 keyword search, fuse both lists |
| Right document, wrong detail quoted | Chunk too large; answer buried | Structure-aware chunks plus re-ranking |
| Correct chunk ranks 8th, never used | top_k too low, or ordering poor | Retrieve wide, re-rank, pass top 3 to 5 |
| Answers reflect a policy you retired | Superseded versions still indexed | Version + effective-date metadata, filtered at query time |
| Quality collapses after a re-index | Silent ingest failure, empty OCR pages | Assert chunk and character counts per document |
| Follow-up questions answered badly | Pronouns searched literally, no context | Rewrite follow-ups as standalone queries |
What should the prompt contract actually say?
The prompt contract governs what the model may do with retrieved text: answer only from the supplied passages, attribute each claim to the passage it came from, and decline when the passages don't contain the answer. Grounding instructions won't rescue bad retrieval. They stop good retrieval from being embellished, which is a smaller but real category of bug.
Be specific about the failure path, because that's the instruction models drop first. "If the context does not contain the answer, say you don't have that information and name what you searched" beats "be helpful and accurate." Ban merging passages from different documents into one claim.
If the model still ignores clean context after all that, it's the weak link and swapping it is justified. Instruction-following and long-context recall differ enormously between families, and our guide to picking an LLM in 2026 covers testing both.
Why do citations change trust and debugging at once?
Citations show which passage produced each claim, with a link back to the source document and section. For users, that turns a black box into something checkable. For your team, it turns vague bug reports into precise ones: instead of "the bot was wrong about refunds," you get "the bot cited Refund Policy v3," which names the fix.
Build citations into generation, not after it. A pass that guesses which document supported an answer guesses wrong sometimes, and a wrong citation is worse than none because it borrows credibility it hasn't earned.
Can your bot return documents the asker should never see?
Yes, if permissions live in the application layer instead of inside retrieval. Your index doesn't know who's asking. Every chunk needs the source document's access-control metadata attached at ingest, and every query needs a pre-filter on the asker's identity, so restricted chunks are never candidates in the first place.
Filtering after retrieval is the pattern to kill. Once a restricted chunk enters the context window the model has read it, and dropping it from the citation list won't stop the content surfacing in a paraphrase. Salary bands and unsigned contracts leak this way. Not as quotes. As summaries.
Two details get missed. Permission changes must trigger a re-index, because the ACL you fixed in your identity provider is still the old one inside the vector store. And if retrieval runs as a single privileged service account, you've built a document leak with a chat interface on it.
How do you stop fixing one bug and creating two?
Build a golden set: real user questions, the answer each should produce, and the document or chunk that should have been retrieved. Score retrieval and generation separately, since a change that improves one often damages the other. Run the full set before and after every index rebuild, embedding change, chunking change or prompt edit.
Fifty to a hundred questions is enough to start, and they should come from your logs, not your imagination. Add the category everyone forgets: questions your documents genuinely cannot answer. If those don't produce a decline, your refusal path doesn't work.
Then wire it into CI as a gate, so a rebuild triggers the eval and the eval blocks the deploy when retrieval accuracy drops. Teams skip this because the bot "mostly works," and mostly working is where silent regressions hide.
When is "I don't know" the right answer?
Whenever the retrieved context doesn't support one. A refusal is correct behaviour, not a product failure. Set a relevance threshold on the re-ranked top score, and below it return a scoped decline that names what was searched, then offers a route forward: a human handoff, a support link, or the closest document you did find.
Vague declines are their own bug. "Sorry, I can't help with that" trains users to stop asking. "I searched the 2026 refund and billing policies and found nothing on partial-month credits. Want me to open a ticket?" tells your team what's missing from the index. We covered this in designing AI error states that don't lose the user, and refusal is the error state that matters most in RAG.
One position worth holding: drop answer rate as a success metric. It rewards a bot for answering everything, which is the behaviour you're removing.
What's on the 10-point RAG hallucination fix checklist?
This is a diagnostic order, not a menu. Steps 1 and 2 establish whether the problem is retrieval or generation. Steps 3 to 7 fix retrieval, cheapest first. Steps 8 to 10 make the fix survive the next rebuild. Work top to bottom.
- Reproduce with the retrieval log open. Capture the chunks that reached the model before changing any code.
- Paste the correct source in by hand. Answer becomes right, it's retrieval. Answer stays wrong, it's generation.
- Purge superseded and duplicate documents, and attach version plus effective date to every chunk as filterable metadata.
- Assert on ingestion. Per-document chunk and character counts, so a text-layerless PDF fails loudly.
- Re-chunk on structure. Keep tables and numbered clauses whole; prefix each chunk with its title and heading path.
- Turn on hybrid search so identifiers, error codes and part numbers survive the trip through the embedding.
- Retrieve wide, re-rank, send few. Thirty candidates in, three to five chunks into the prompt.
- Write the grounding contract explicitly: answer only from context, quote the span, decline below the threshold.
- Enforce permissions as a retrieval pre-filter, and re-index when access rights change.
- Freeze a golden question set, unanswerable questions included, and gate every rebuild on it.
If that's work your team has no bandwidth for, our generative AI engineering team runs retrieval audits on a roughly two-week delivery cycle, with a 7-day free trial so you see the diagnosis first. Book a 30-minute call and bring three wrong answers with you.
Frequently Asked Questions
Will a bigger context window fix hallucinations?
No. A larger window lets you send more chunks, but irrelevant chunks still get read and summarised. Retrieval precision decides what the model sees; a bigger window just adds cost and latency on top.
Should we swap the embedding model first?
Usually the wrong first move, since new embeddings force a full re-index and invalidate every threshold you tuned. Work in cost order: document hygiene, chunking, hybrid search, re-ranking, then embeddings.
Does fine-tuning stop a RAG bot from hallucinating?
Not the way people hope. Fine-tuning shifts tone, format and instruction adherence. It doesn't change which facts land in the context window, so a fine-tuned model handed a stale policy chunk still answers from the stale policy.
Can hallucinations be measured automatically?
Partly. Score faithfulness by asking a second model whether each claim is supported by its cited chunk, and check quoted spans appear verbatim in the retrieved text. Both need your golden set as ground truth.
Our bot cites a source but the answer doesn't match it. What's happening?
That's post-hoc attribution: the citation got picked after the answer was written instead of during generation. Move attribution into the generation step, require a quoted span, and verify it before display.
Have a project in mind? Let's scope it together.
You get a named team, written estimates, full code and IP ownership, and 48-hour response times. CMMI Level 5 certified. 700+ projects delivered across the UK, US, UAE, and Australia.
