RAG Data Freshness: Keeping Indexes Current as Documents Change

RAG data freshness usually fails without an obvious error. A source policy changes on Monday, but the assistant keeps retrieving Friday's chunks because the sync missed a deletion, wrote only half the replacement, or advanced its cursor too early. The answer cites a real document and looks current even though the text is obsolete. Rebuilding the entire index after every edit reduces some drift, but it repeats embedding work and does not define what happens when a rebuild stops halfway.

Use a versioned synchronization contract instead of trying to fix stale data in the retrieval prompt. The pipeline has to carry additions, edits, and deletions from the source into the searchable index. It also needs to recover incomplete runs, measure source-to-search lag, and reject stale chunks after a successful sync.

Define what fresh means before building the pipeline

"Updated regularly" cannot be tested. A freshness objective needs a source event, the expected retrieval state, and an allowed delay. A team might require a committed source edit to become the only retrievable version within fifteen minutes, while a deletion must stop retrieval within five minutes. These are example targets. Set the actual times from the risk and update rate of each collection.

A usable contract should answer six questions:

1. What counts as a source version: an update timestamp, revision number, change token, or event sequence?
2. Which identity remains stable when a document title, folder, URL, or body changes?
3. When does a synchronization checkpoint become durable?
4. How are deletions represented before the source object disappears?
5. What should retrieval do while one document is between versions?
6. Which measurements and tests prove that the index matches the source?

Azure AI Search documents that supported indexers can detect changed source content on later runs and can run on a recurring schedule. It also notes that workloads needing more frequent updates may need a push model that updates the source and search index together. The Azure indexer overview is useful here because it separates scheduled polling from application-controlled synchronization.

Treat freshness separately from answer quality. A relevance evaluation may show that the best available chunk was retrieved. It does not prove that the chunk represents the current source revision. Track both.

Give documents and chunks stable identities

A title or URL is a poor primary key. People rename files, move folders, and change slugs. Use the source system's immutable document identifier when it has one. Derive chunk identities from that document ID, the source version, and a deterministic chunk position or content digest.

A compact synchronization record can look like this:

{
  "source_document_id": "policy-482",
  "source_version": "rev-41",
  "source_updated_at": "2026-08-10T09:30:00Z",
  "content_hash": "sha256-of-normalized-content",
  "permission_hash": "sha256-of-normalized-access-metadata",
  "sync_state": "active",
  "chunk_ids": ["policy-482:rev-41:0", "policy-482:rev-41:1"],
  "indexed_at": "2026-08-10T09:32:18Z"
}

These values are examples. Keep the properties even if your field names differ:

• source_document_id groups every version and derivative of one source object.
• source_version distinguishes the currently intended revision from old chunks.
• content_hash lets the pipeline skip embedding when only metadata changed.
• permission_hash lets authorization metadata change without pretending the body changed.
• sync_state represents active, pending, failed, and deleted states explicitly.
• chunk_ids make replacement and cleanup bounded instead of requiring a broad metadata scan.

A current Mozilla SUMO issue lists similar RAG indexing pipeline requirements. It calls for skipping new embedding calls when content is unchanged, preserving vectors for metadata-only updates, removing deleted content from active indexes, and repairing missing or partially written chunk sets. This is a practitioner requirement rather than a guarantee about every RAG system, but the failure cases are concrete.

Process additions and edits idempotently

Make synchronization idempotent: replaying the same source version must produce the same index state without duplicate chunks. Workers crash. Queues can redeliver events. Operators also rerun failed batches, so replay is normal rather than exceptional.

Use this sequence for an addition or edit:

1. Read the source document and its authoritative version token.
2. Normalize the body and access metadata, then compute separate hashes.
3. Compare those values with the last committed synchronization record.
4. If neither hash changed, record the observation and skip indexing.
5. If only access metadata changed, update metadata on the current chunks without embedding the body again.
6. If content changed, create the full new chunk set under the new source version.
7. Verify that every expected new chunk is readable from the index.
8. Switch the document's active version pointer to the new version.
9. Remove the old chunks after the switch succeeds.
10. Commit the per-document synchronization record.

Pinecone exposes separate operations to update record values or metadata and to delete obsolete records. Other vector stores use different APIs, but the pipeline still needs explicit update and deletion behavior. Do not assume an upsert of new chunks automatically removes chunks that no longer exist in the latest version.

The active version pointer can live in the vector metadata, a relational control table, or an alias mechanism, depending on the store. Retrieval must filter for the active version. This prevents a failed replacement from exposing a mixture of revisions. Only flip the pointer after the complete new set passes a read-back check.

Here is storage-neutral pseudocode:

def sync_document(event):
    source = load_source(event.document_id)
    previous = control_store.get(source.id)
    candidate = build_manifest(source)

    if previous and candidate.matches(previous):
        return mark_observed(previous, event.sequence)

    if previous and candidate.content_hash == previous.content_hash:
        vector_store.update_metadata(previous.chunk_ids, candidate.metadata)
        return commit_manifest(candidate, event.sequence)

    chunks = chunk_and_embed(source, version=candidate.version)
    vector_store.upsert(chunks, active=False)
    assert_complete_and_readable(chunks)

    control_store.activate(source.id, candidate.version)
    vector_store.delete(previous.chunk_ids if previous else [])
    return commit_manifest(candidate, event.sequence)

The activation and deletion steps may not be one database transaction. If they are not, make each operation replayable and let a reconciliation job finish the intended state.

Treat deletion as a first-class source event

Polling handles changed objects more easily than missing ones. A deleted object cannot report a current timestamp. Microsoft's guidance for changed and deleted blobs says that change detection can use source timestamps, while deletion detection needs a separate strategy. Its soft-delete sequence removes search documents before the source is physically deleted, which prevents orphaned index entries.

Use a tombstone event or soft-delete marker that carries the stable source ID and version. The deletion worker should:

1. Mark the document unavailable for retrieval.
2. Delete every known chunk and derived summary for that source ID.
3. Invalidate answer caches that cite the document.
4. Confirm that a direct filtered search returns no active record.
5. Commit the tombstone and its processed sequence.
6. Allow physical source cleanup only after the required downstream consumers acknowledge it.

If the source provides no deletion feed, run periodic reconciliation. Compare the set of active source IDs with the control table, partitioning the scan so it remains bounded. Missing IDs become deletion candidates, but apply a grace period or second confirmation if the source listing can be temporarily incomplete.

Do not reuse a deleted document ID for unrelated content. If the source system does that, combine the source ID with a generation identifier controlled by your pipeline.

Advance checkpoints only after durable index state

A global cursor records which source events the pipeline believes it has processed. Advancing it before every index mutation is durable lets a crash skip unfinished work permanently. Holding it back after a successful write causes a replay, which should be harmless because each document operation is idempotent.

Prefer per-partition checkpoints plus per-document manifests. For each event, record three states:

• observed: the event entered the pipeline.
• applied: expected chunks and metadata are readable in the index.
• committed: the durable checkpoint includes the event sequence.

A recovery worker scans records stuck between these states. If chunks are missing, it replays the source version. If both old and new versions are active, it keeps the intended version and deactivates the other. If the source version changed again during recovery, it abandons the obsolete candidate and processes the latest event.

Never mark a batch successful because most documents completed. Store failures by document and retry them within a bounded policy. A poison document that cannot be parsed should become visible in an operations queue rather than blocking every later source change or disappearing from the run summary.

Measure freshness from source to retrieval

A green scheduler result does not prove freshness. A filter may exclude part of the source, the deletion worker may be disabled, or accepted writes may not yet be visible to reads.

Record these measurements:

• Source-to-observation lag: source update time to event or poll discovery.
• Observation-to-index lag: discovery to committed searchable version.
• End-to-end freshness lag: source update to successful retrieval of the new version.
• Deletion lag: source tombstone to zero retrievable active chunks.
• Repair backlog: manifests stuck in observed or applied state.
• Version conflicts: documents with zero or more than one active version.
• Orphan count: indexed source IDs absent from the active source inventory.
• No-op rate: events skipped because content and permission hashes matched.

Use a synthetic canary document in a non-sensitive test collection. Update a unique marker, wait for the declared objective, and query the real retrieval endpoint. Then delete the canary and prove that neither the new marker nor its citation is retrievable. This checks the route users depend on, not just the control table.

Test edits, deletes, replay, and partial failure

Use stale-state failures as the release gate. Test at least these scenarios:

• Edit one paragraph so chunk boundaries change. Only the new version remains active.
• Rename and move a source document. Its stable identity remains unchanged.
• Change metadata without changing text. Vectors remain reusable and metadata becomes current.
• Delete a document. Exact-title and semantic queries return no chunk or cached answer from it.
• Deliver the same event twice. The second run creates no duplicate records.
• Crash after writing half the new chunks. The old complete version stays active until repair succeeds.
• Crash after activation but before old-chunk deletion. Retrieval sees only the new active version and reconciliation removes the old set.
• Advance source content during a retry. The pipeline does not reactivate the superseded candidate.
• Pause the deletion consumer. The deletion-lag alert fires before the promised window expires.
• Corrupt a manifest. Reconciliation identifies the mismatch instead of trusting the last job status.

Run retrieval assertions with version markers that are harmless and unique. Check returned text, citations, caches, and trace metadata. A test that only counts vector records can miss an old answer cache or a query filter that still selects an inactive version.

Common mistakes that create stale RAG answers

Do not use updated_at as both identity and ordering. Clock skew and same-second updates can cause missed records. Prefer source revision tokens or a monotonic change sequence where available.

Do not treat upsert as replacement. Upsert changes the supplied record IDs, but it may leave old chunk IDs behind when a document becomes shorter.

Keep the source until downstream systems have observed its tombstone. If the object disappears first, the indexer may have no evidence that cleanup is required.

Commit a batch cursor only after every included document reaches durable index state. Otherwise, one document failure can hide behind a successful batch result.

Monitor retrieval freshness rather than scheduler status alone. Measure what users can retrieve and the age of that version.

Keep full rebuilds for disaster recovery and occasional reconciliation. They should not be the only update mechanism for a live internal knowledge base.

Put the freshness contract into production

Start with one collection that changes often enough to expose the problem but does not contain your highest-risk data. Define stable source IDs, source versions, manifests, and tombstones. Make update and delete operations idempotent, then add active-version filtering, checkpoint recovery, and a daily reconciliation pass. Run the failure matrix through the deployed retrieval endpoint.

The index should be able to identify the active source revision, recover an interrupted replacement, remove a deleted document, and prove each result through retrieval. Test one frequently edited document now: change a marker, interrupt the sync halfway, replay it, and then delete the source. Any chunk or citation that survives in stale form points to a specific pipeline defect.

References

• Azure AI Search indexer overview supports scheduled incremental indexing and the distinction between polling and push synchronization.
• Azure AI Search changed and deleted blobs supports the change-detection and soft-deletion sequence used to avoid orphaned search documents.
• Pinecone update records supports explicit vector and metadata update operations.
• Pinecone delete records supports explicit removal of obsolete records and namespaces.
• Mozilla SUMO issue 3247 supplies practitioner requirements for incremental embedding, metadata-only updates, deletion, and repair of partial chunk sets.

---