Evaluate an embedding-model migration before reindexing

Treat the embedding change as an index change

Sentence Transformers documents retrieval evaluation over queries, a corpus, and relevance labels. A migration must preserve that evaluation contract while changing the representation. Record model identifiers, preprocessing, vector dimensions, similarity function, and index settings for both versions. Do not assume vectors from unrelated embedding spaces are interchangeable because they have the same dimension. Keep the old searchable index intact until the new corpus representation and query path have both been checked.

Evidence: Sentence Transformers evaluation reference [1]

Build paired migration cases

Create an authored product-support set containing exact part numbers, paraphrased symptoms, multilingual descriptions, and a rare exclusion clause. For each question, preserve the source IDs and supporting spans independently of either embedding model. Run the old and candidate query encoders against their matching indexes. Include a deliberately mismatched query-encoder configuration as an integration check. Its purpose is to verify that the application detects configuration mistakes, not to estimate model quality from an invalid setup.

Separate representation and pipeline effects

First compare retrieval with the same parsed text, chunk boundaries, filters, and candidate budgets. If you also change parsing or chunking, keep a separate experiment so you can explain the source of any improvement. Report paired evidence gains and losses, not just the difference between two aggregate scores. Then test generated answers using the same generation settings. Record indexing failures, missing records, ingestion cost, and operational latency because a retrieval improvement does not establish that the migration is operationally acceptable.

Cut over using explicit acceptance evidence

Agree which query groups and critical records must retain evidence before switching traffic. Inspect every critical regression and verify corpus completeness with stable document IDs. Exercise rollback against the previous index rather than relying on a configuration value that has never been tested. Save the migration manifest with model and corpus hashes, evaluation cases, observed differences, and unresolved limitations. That record supports the next migration and prevents a later team from confusing a fresh index with a fully validated one.

Plan the observation period around changed queries

After a controlled cutover, inspect queries from the migration slices instead of monitoring only total request success. A rare product-code failure may never move an aggregate dashboard. Record the new index revision with every sampled query and retain a route back to the old index for the approved observation period. If a rollback happens, confirm that cached retrieved contexts and query encoders also return to their matching versions. Switching the index alias alone is insufficient when the surrounding application still uses assumptions from the candidate representation. Verify that query caching cannot accidentally reuse vectors created by the previous encoder after the index switch.

Common questions