RAG: it searches first, then answers, and retrieval is where it breaks

I build Settl, an expense-splitting app, solo with Claude, and I once watched it answer a question about my own docs with a real link and a number that had been wrong for a month. It was not inventing anything. The search handed it an old page. Here is where to look instead.

Why it can answer docs it never read

The weights froze at the end of training. Your docs are not in them. A search runs first, and its results are pasted into the prompt as ordinary text, above your question.

you:        what is our refund window
retriever:  search("refund window") -> 3 chunks, scores 0.71 / 0.66 / 0.58
prompt:     [chunk 1][chunk 2][chunk 3] + your question + "answer only from the text above"
model:      "30 days" (source: /docs/refunds)

It is reading, not remembering. Everything it says about your product came from text something else chose, and nobody inspects that step.

The five stages, and the one that actually fails

Ingest. Documents are crawled and cut into chunks. A page behind auth, in a PDF, or in a repo the crawler never walked does not exist downstream.
Index. Chunks are embedded and stored as vectors. Why their wording decides your hit rate: settl.fyi/social/it-found-the-file-you-didnt-name.
Retrieve. The question is embedded, the top k chunks come back ranked.
Assemble. Chunks are concatenated into the prompt, in some order, up to some budget.
Generate. The model answers from what it was handed.

Nearly every bad answer starts at stage 1 or 3. Nearly every fix is aimed at stage 5. Rewriting the system prompt cannot conjure a chunk that was never retrieved.

Copy these prompts

Run these in order in your retrieval repo.

1. Replay the retrieval without the model

Write a script that runs only the retrieval half of our pipeline: it takes a question, runs
the exact search the app runs, and prints per result the rank, score, source path, and first
200 characters of the chunk. No model call, no answer. Run it on three real questions of mine.

2. Prove whether the answer was ever in the context

For each question, check whether the correct current answer appears verbatim in a retrieved
chunk. One verdict each: RETRIEVED, MISSING if no chunk contains it, STALE if a chunk holds an
older value contradicting the document. Show the chunk behind each verdict. No fixes yet.

3. Fix the source, not the wording

For everything MISSING or STALE, open the source document and say which is true: the fact is
written nowhere, it is phrased with words nobody searches for, it is split across two chunks,
or an old copy lives elsewhere. One-line fix each, plus the documents to delete.

Why a real citation can carry a stale answer

The citation is attached to the chunk, not to the truth. A chunk accurate in March gives a wrong answer with a working link, which is worse than an obvious hallucination: it survives a spot check. Two causes, two fixes:

The index was never rebuilt. The document changed, the vector did not.
The old chunk was never deleted. The one that bites. A reindex that only adds chunks leaves the March copy competing for the same query, sometimes winning. Delete by document id before the insert.

Cheap guard: carry a last-reviewed date into the chunk text and surface it in the answer. A reply stamped "last reviewed 2026-03-02" gets caught in a second.

Write docs that survive being chopped up

One fact, one place. A number on three pages guarantees one copy goes stale.
Answer directly under its heading, not four paragraphs down after two caveats.
No orphan pronouns. "It is 30 days" is useless in a chunk split from the sentence naming its subject.
Write the aliases in: refund window, cancellation period, money back.

When retrieval is the wrong tool

A small, stable corpus. Twelve pages fit in a prompt whole.
Aggregation questions. "How many docs mention X" is a query, not a top-k lookup. The retriever returns five chunks and the model counts five.
Anything that must be current to the second. Balances, prices, inventory. Call the source system as a tool: settl.fyi/social/the-model-never-touched-your-file.
Rules that must always apply. Those go in the system prompt, never in a search that might miss them.

Where you keep it, cap what enters the prompt: similarity floor, dedup on write. My values: settl.fyi/social/ai-memory-diet.

Run this on your codebase

Paste this into a session in your repo:

Audit the retrieval path in this repo.
1. Find every place a search result, chunk, or row is inserted into a prompt. Give the file,
   the function, and how many items.
2. Is that text logged before the model sees it? If not, the pipeline cannot be debugged,
   and this is finding number one.
3. Does re-indexing delete the old chunks for a changed document, or only insert new ones?
   Show me the delete path, or state there is not one.
4. Is a similarity floor applied before results enter a prompt, and what happens when nothing
   relevant is found?
5. Quote the wording telling the model to answer only from the provided text and to say when
   the answer is not there. If there is none, say so.
Report all five as a checklist and change nothing. Then write the logging from step 2: query,
chunk id, source path, score, characters pasted.

Prompt, examples, retrieval, or fine-tuning

Use the cheapest layer that owns the problem:

Problem	Start here	Why
The instruction is unclear	Prompt	State the rule directly.
The format or judgment is hard to describe	Few-shot examples	Show approved input-output pairs.
Facts are private or change	Retrieval or a tool	Read the current source at request time.
A stable behaviour repeats at high volume after prompts and examples plateau	Fine-tuning	Train a measured behaviour, not a changing fact set.

Fine-tuning does not turn your current help centre into a live database. The trained weights are a snapshot. Change a price or policy and you would need new training, with no clean citation path. Retrieval lets the next request read the changed source.

Use fine-tuning for stable behaviour such as a classification boundary, house format, or repeated transformation only after a fixed evaluation set proves the base model plus prompt/examples is not good enough. Keep a holdout set and compare quality, latency, and cost.

One sharp test: change a source fact. If the answer must update without training, the fact belongs in retrieval or a tool.