Your RAG Did Not Notice the Document Changed

If you run RAG in production, this bug is already in your system. Your documents changed last week. Your index did not. Somewhere, a user is reading an answer from a policy that no longer exists, and nothing in your logs looks wrong.

AI School 369 builds in public: drafts are prepared with AI assistance, and every project, number, and line of output in this story was run and verified by our team before publishing. This one started as a question we kept hearing from engineers: why does the chatbot sound confident and still get the returns policy wrong?

The index is a photograph, not a window

When you build a RAG index, you take a photograph of your documents at one moment. Then the document changes. The photograph does not.

The chatbot keeps answering from the old photograph, in the same tone, with the same formatting. Nothing crashes. No error fires. The answer is simply wrong, and it stays wrong until a human notices, if a human notices.

We wanted to watch this happen, not just describe it. So we built the smallest system that shows the failure and the fix: three tiny store documents (a returns policy, a shipping guide, a warranty), a simple index, and a script that edits one document, deletes another, and asks questions at every step.

Three habits that keep an index honest

1. Version stamps. Every chunk carries the version of the document it came from. When a document changes, you know exactly which chunks are stale, because they are the ones whose version is behind.

2. Reindex on change, and only on change. A file hash tells you what actually changed. One edited paragraph costs you one document of re-embedding, not the whole corpus.

3. Deletion propagation. When a document is deleted, its chunks leave the index in the same run. A deleted policy should never answer a question again. This is the habit teams forget most often, and it is the one with legal consequences when the deleted document is a policy.

Over all three sits a safety net: canary questions, a small set of questions with known answers, run after every refresh. If a refresh breaks an answer, the canaries fail loudly before your users do.

The refresh loop, in real code

The manifest is the brain. For each document it remembers the file hash (what the content is) and a version (how many times the content changed since we first saw it). Refresh is two passes. Deletions first, because a vanished file should take its chunks with it:

# Pass 1: deletions. A file that vanished takes its chunks with it.
for name in list(self.manifest):
    if name not in on_disk:
        kept = [c for c in self.chunks if c["doc"] != name]
        dropped = len(self.chunks) - len(kept)
        self.chunks = kept
        del self.manifest[name]
        log.append(f"  DELETED {name}: removed {dropped} chunks")

Then new and changed files. The hash decides, not the clock:

# Pass 2: new and changed files. The hash decides, not the clock.
for name, path in on_disk.items():
    raw = path.read_bytes()
    digest = hashlib.sha256(raw).hexdigest()
    known = self.manifest.get(name)
    if known and known["hash"] == digest:
        log.append(f"  SAME    {name}: unchanged, skipped")
        continue
    version = 1 if known is None else known["version"] + 1
    # Drop the old chunks for this document, then embed fresh ones.
    self.chunks = [c for c in self.chunks if c["doc"] != name]
    ...

Here is what a single chunk looks like inside the index:

{"doc": "returns-policy.txt", "version": 2,
 "text": "You may return any unused item within 60 days of delivery.",
 "vector": embed(text, vocab)}

Four fields, and the first two do the freshness work. The doc name ties the chunk to its source, so a delete knows exactly what to remove. The version is the chunk's generation: when the returns question gets answered, we can ask which generation answered it. That one question, "which version of the document produced this answer," is the difference between a five-minute diagnosis and an afternoon of guessing. Most production indexes cannot answer it, because the chunk was divorced from its source the moment it was embedded. Keep the marriage certificate. It costs four fields.

Two details worth stealing. Unchanged files cost nothing: no re-embedding, no API spend, just one hash comparison. And the version counter is what makes staleness visible later, when you are debugging at 2 AM and need to know which generation of a document an answer came from.

The canary runner is deliberately boring. Boring is what you want standing guard:

def run_canaries(index, canaries, phase):
    """Ask every canary question and check the answer against
    the truth for this phase ('before' or 'after' the change).

    An expected value of None means the honest answer is
    'no answer': the document that held the fact is gone, so
    the system must not invent one."""
    failures = 0
    for canary in canaries:
        expected = canary[phase]
        got = index.answer(canary["question"])
        ...
    return failures

Note the None case. For the shipping questions, the correct answer after deletion is no answer at all. A system that keeps answering from a deleted document is not being helpful. It is hallucinating with a citation.

What actually happens when you run it

Real output, not a mock-up:

• The fresh index answers all 20 canary questions correctly. 20/20.
• The script edits the returns policy: 30 days becomes 60 days. It deletes the shipping guide. Nobody refreshes.
• Asked the returns question, the old index answers "within 30 days of delivery." That is the bug, live, in one line.
• Refresh runs. The log touches only what changed: CHANGED returns-policy.txt (version 2). DELETED shipping-guide.txt, 5 chunks removed. SAME warranty.txt, skipped.
• The refreshed index answers "within 60 days of delivery." The shipping question returns no answer, because that document no longer exists. Canaries: 20/20 again.

The whole demonstration takes about two minutes, most of it reading.

Get the code

The project is public, MIT licensed, and runs offline with no API key:

git clone https://github.com/urbanschool369-oss/ai-school-369-projects
cd ai-school-369-projects/projects/rag-freshness
pip install -r requirements.txt
python main.py

Try breaking it. Edit a document, ask its question before refreshing, and watch the stale answer come back. That is what your users see when nobody owns freshness.

Why smart teams still ship without it

Freshness rarely fails because engineers do not know. It fails because nobody owns it. The indexing job was written during the launch sprint, it ran once, it worked, and it quietly became permanent. The document store lives in another team's system, so edits arrive silently. Deletes are rarer than edits, so deletion handling gets cut from the sprint entirely. And there is no alarm for "the answer is stale," because stale answers do not throw exceptions. They throw users.

That is why the canaries matter more than the refresh logic. A refresh you remember to run is a habit. A canary suite that fails the build when an answer goes stale is a guarantee. Habits decay under deadline pressure. Guarantees page somebody.

Taking the pattern to a production stack

The demo scans a folder. Production systems swap the trigger, not the logic:

• Change detection: document events (webhooks from your CMS, SharePoint, Drive, or ticketing system) replace the folder scan. The hash comparison stays; it is your proof that an event corresponds to a real content change and not a metadata touch.
• The manifest: in a real vector store, the manifest becomes metadata on every chunk: source document, version, hash, indexed timestamp. If your store cannot filter deletes by document ID, that is a selection criterion for your next store, not a detail.
• Deletion: propagate as a tombstone first (mark chunks dead so they stop answering immediately), then purge physically in a background pass. The user-visible guarantee is the tombstone.
• Canaries: run them on a schedule and after every refresh, sampled per document family. When a canary fails, the alert should name the document and the expected answer. "Index unhealthy" is not an alert; it is a shrug.

Designing canaries that stay honest

Three rules from running this pattern. Pick factoid questions whose answers live in exactly one document, so a failure names its culprit. Include negative canaries: questions whose correct answer is "no answer," covering documents you have deleted on purpose. And retire canaries when their documents retire, or the suite fills with questions about a system that no longer exists and the team learns to ignore it. A guard the team ignores is worse than no guard, because it provides the feeling of safety without the fact of it.

What this demo does not solve (honest limits)

Three simplifications, stated plainly. The embedder is a tiny bag-of-words model so the demo runs offline; swap in a real embedding model and the freshness logic around it stays exactly the same, but retrieval quality becomes a separate problem this demo does not measure. The index is in memory; a production system needs the same manifest discipline against a real vector store, with refresh triggered by document events rather than a folder scan. And 20 canaries guard three documents; a real corpus needs canaries sampled per document family, refreshed as documents evolve, or the guard itself goes stale.

None of that changes the core point. Freshness is not a feature you add later. It is the difference between an index and a photograph.

The one-line version

An index you never refresh is a photograph of a system that no longer exists. Stamp versions, reindex on change, propagate deletions, and let canaries stand guard.

Next in this series: the anatomy of an agent tool loop, and where it spins forever. The build there is a loop with a step budget, a retry budget, and a circuit breaker, and yes, we break it on purpose.