Embedding Model Migration for RAG Without Retrieval Downtime

An embedding model migration can break retrieval before any service reports an error. If new query vectors search old document vectors, the numbers may have compatible shapes while representing different spaces. If dimensions differ, writes or searches fail outright. Rebuilding the live index in place is not safer: it creates partial coverage, mixes versions, and removes the fastest rollback path. The practical fix is a blue-green migration. Build a separate versioned index, keep it synchronized, test it with real queries, switch reads through one controlled pointer, and retain the old index until the rollback window closes.

This guide gives AI, data, and search engineers a migration state machine, cutover rules, recovery paths, and verification checks for an internal-document RAG system.

Why an embedding upgrade is a data migration

An embedding is not a durable property of a document. It is the output of a specific model, model configuration, preprocessing pipeline, and input text. Change any of those inputs and the stored vector has a new interpretation. A model name alone is therefore not enough to identify an index.

Storage engines enforce part of this contract. Qdrant requires vectors within a collection to share dimensionality and metric requirements. That catches an obvious 768-to-384 dimension change, but equal dimensions do not prove compatibility. Two models can emit vectors of the same length in unrelated spaces. The database cannot detect that semantic mismatch.

The model configuration matters too. The OpenAI embeddings guide documents model selection and a dimensions parameter for supported models. A configuration change that shortens vectors belongs in the index identity even when the model name stays constant.

Treat the following tuple as the index version:

embedding_contract = {
  provider,
  model,
  model_revision,
  dimensions,
  distance_metric,
  text_normalization_version,
  chunking_version,
  metadata_schema_version,
  access_policy_version
}

Do not let an application infer this tuple from whichever model happens to be configured at startup. Store it with the collection and compare it before every indexing job and deployment. A mismatch should stop the job before it writes one vector.

Define the migration states before rebuilding

A migration needs explicit states because backfills, document updates, and serving traffic happen concurrently. Use a durable record rather than a deployment checklist in a ticket.

planned -> backfilling -> catching_up -> shadowing -> ready
ready -> active -> rollback_window -> retired
any pre-cutover state -> failed
active -> rolled_back

The record should include source and destination index IDs, both embedding contracts, a frozen source manifest timestamp, backfill cursor, high-water mark for live changes, evaluation-set version, cutover time, rollback deadline, and the operator who approved each transition.

State transitions need predicates. backfilling becomes catching_up only when every item in the frozen manifest has a destination record or a documented terminal error. catching_up becomes shadowing only when the destination has consumed all document changes through a known event offset. shadowing becomes ready only when coverage and quality gates pass. The database being green is not a quality gate.

Build a destination that cannot mix versions

Create a new physical collection or index. Never reuse the source collection name for the rebuild. Give the destination an immutable ID such as knowledge-emb-v4-20260903, then attach the full embedding contract as collection metadata and in the migration record.

Use a stable document identity across both indexes. Each vector record should carry at least:

• document ID and source revision
• chunk ID and chunking version
• content checksum
• embedding contract ID
• tenant and access-control identifiers
• source URI and citation coordinates
• indexed event offset or update timestamp

Reject a record when its embedding contract ID differs from the destination contract. This guard prevents a retry worker with stale configuration from contaminating the new collection.

Rebuild from source documents or a canonical parsed-document store, not from old vectors. Vectors cannot be translated reliably between arbitrary embedding spaces. If parsing and chunking are unchanged, replay the preserved chunks. If they also change, call that out as a combined migration because retrieval differences can no longer be attributed to the embedding model alone.

Backfill while preserving live document changes

A long backfill races with edits, deletions, permission changes, and new documents. Freezing all writes avoids the race but usually creates unacceptable downtime. Use a snapshot plus change-log approach instead.

First, record a high-water mark from the authoritative document event stream. Enumerate the source corpus as of that point and write the frozen manifest. Backfill those revisions into the destination with idempotent upserts. At the same time, continue indexing current changes into the active source index as usual.

After the snapshot backfill completes, replay events after the high-water mark into the destination. Keep consuming until lag reaches the agreed threshold, ideally zero. Then enable dual writes for new events or keep both consumers on the same durable stream. The event log must include deletes and permission changes. Copying only current documents leaves revoked content searchable in the destination.

Use deterministic record keys so replay is safe. A useful key is tenant_id/document_id/source_revision/chunk_id. Store the last applied event offset separately from per-record timestamps. Wall clocks cannot reliably order updates across workers.

If a worker fails, restart from its saved cursor. If one document repeatedly fails embedding, quarantine it with the source revision and error category. Do not silently omit it to make the completion percentage reach 100.

Compare coverage before comparing relevance

Quality tests are meaningless when the destination corpus is incomplete. Verify structural parity first.

Check document counts by tenant, source connector, content type, and access-policy class. Compare the set of current source revisions, not just total vector counts, because different chunking can legitimately change vector totals. Sample content checksums and citation coordinates. Confirm that deleted documents and revoked permissions are absent.

Set explicit tolerances. For example, require every high-risk policy document to be present, no unauthorized document to appear, no unexplained missing current revision, and all quarantined failures to have an owner. A global 99.9 percent coverage number can hide the only handbook a support workflow needs.

CodeScout issue 18 reports that a forced rebuild could not migrate an immutable SQLite vector table from 768 to 384 dimensions. The job initially appeared started, then failed on the original mismatch. Treat this as a practitioner report, not a universal database behavior. Final-state checks must inspect the destination contract and searchable contents rather than trusting that a rebuild command was accepted.

Shadow retrieval with representative queries

Run the same normalized query against both indexes without changing the user-visible answer. Record top results, scores, filters, latency, and citation coordinates. The comparison should happen before generation so a fluent answer cannot hide weaker retrieval.

Build the evaluation set from three sources: known-answer questions, recent production queries with sensitive text removed, and failure fixtures from support or incident history. Include exact-name lookup, paraphrases, acronym expansion, policy exceptions, date-sensitive questions, sparse entities, long questions, and access-control negatives.

Compare outcomes that matter:

• whether a supporting chunk appears within the retrieval budget
• rank of the first supporting chunk
• unsupported or conflicting chunks in the top results
• citation source and span correctness
• cross-tenant and revoked-document leakage
• p50 and p95 retrieval latency
• no-result rate by query class

Do not require identical nearest neighbors because a new model should change some rankings. Define acceptance around evidence needed for the downstream task. Review regressions individually for high-risk query classes, even when the aggregate score improves.

Run a bounded answer evaluation only after retrieval passes. Keep the generator, prompt, reranker, and context budget fixed so the embedding change remains the variable under test.

Cut over through one atomic pointer

Applications should resolve a logical retrieval name to one physical index. The pointer can be a database alias, a configuration record read per request, or a routing service. It must support an atomic switch and an equally simple reversal.

Qdrant documents collection aliases and atomic alias operations. Elasticsearch also documents alias routing and atomic multi-action updates. The product syntax differs, but the cutover rule is the same: no request should observe a half-updated routing state.

Before switching, require all of these conditions:

1. Destination contract matches the approved migration record.
2. Snapshot coverage and change-stream lag pass their gates.
3. Access-control negative tests return no forbidden records.
4. Shadow retrieval passes by query class, not only in aggregate.
5. Destination capacity and latency pass a load check.
6. The rollback pointer and operator command have been tested.
7. Indexing workers are writing compatible updates to both sides.

Switch a small internal cohort first if application routing permits it. Otherwise switch the logical alias once the destination is ready, then watch retrieval errors, no-result rate, latency, and sampled quality outcomes. Record the exact event offset and contracts at cutover.

Handle failure without improvising

If backfill fails, leave reads on the source and resume from the durable cursor. Recreate only the destination when its contents are suspect. Never delete the active index to make room for another attempt.

If the destination cannot catch up, stop progression and diagnose the event consumer. Common causes include omitted delete events, poison documents, stale credentials, and a writer still using the old embedding configuration.

If shadow quality regresses, classify failures before tuning. Missing documents indicate coverage. Wrong chunks may indicate model behavior, preprocessing, chunking, filters, or reranking. Change one layer at a time and version the evaluation result.

If production quality or latency breaks after cutover, switch the pointer back to the source. Keep document changes flowing to both indexes during the rollback window so either remains viable. If dual writing fails after cutover, prioritize the active destination, record the source lag, and repair it before claiming rollback readiness.

A rollback is complete only when reads resolve to the source, source indexing lag is within tolerance, and a smoke set returns expected evidence. Changing an alias back without checking source freshness can restore an outdated system.

Verify retirement instead of deleting on schedule

Retain the source through a defined rollback window based on document-change volume and how quickly quality labels arrive. During that period, reconcile both indexes and keep the source credentials, capacity, and runbook operational.

Retire only after the destination has remained stable, delayed evaluations have arrived, no rollback condition is open, and downstream caches reference the new contract. Remove the source from routing first. Disable its writers next. Take any required audit snapshot, then delete the old collection under a separate approved action.

Run one final negative test: query the logical name and confirm every returned record carries the destination embedding contract. Also assert that direct source-index access is unavailable to the application. This catches stale configuration that bypasses the alias.

Use this migration checklist

Start by writing the two embedding contracts and creating a distinct destination. Backfill from canonical content at a recorded high-water mark. Replay every update, deletion, and permission change. Prove corpus coverage before measuring relevance. Shadow real query classes, inspect security negatives, and hold all non-embedding variables steady. Test the atomic pointer in both directions. Cut over only when every gate has evidence, then preserve a synchronized rollback index until delayed quality checks pass.

The next action is concrete: inventory the model, dimensions, metric, preprocessing, chunking, metadata, and access-policy versions behind the current index. If any field is unknown, recover that contract before starting the rebuild. A migration without a known source contract cannot explain regressions or prove rollback safety.

References

1. Qdrant collections documentation supports collection dimensionality, metrics, and atomic collection aliases.
2. OpenAI embeddings guide supports embedding model and dimensions configuration.
3. Elasticsearch index aliases supports alias routing and atomic alias changes.
4. CodeScout issue 18 supplies a practitioner-reported vector-dimension migration failure and requested regression checks.

---