Embeddings are a versioned interface

A coordinate system is part of the data

An embedding is often stored as an array of floating-point values with a document identifier beside it. That representation hides most of the contract. The encoder, tokenizer, input template, truncation policy, pooling method and normalization all influence the coordinates. Change one and the resulting vectors may inhabit a different effective space even when the dimension stays constant. I would store a representation version with every indexed item and query. A dimension check catches a mechanical incompatibility; it cannot establish that distances retain their previous meaning or that an old threshold still selects the same kind of evidence.

Reimers and Gurevych’s Sentence-BERT paper describes sentence representations that can be compared using similarity measures, avoiding repeated joint processing of every sentence pair for certain tasks. That is an architectural contribution, not a promise that every embedding model shares one universal semantic coordinate system. My interface argument follows from the distinction between a representation and the procedure that produced it. If the procedure changes, downstream consumers need a compatibility decision. Silently mixing representations resembles changing the units of a stored measurement while leaving its numeric column unchanged, except the transformation between spaces may not be simple or recoverable.

References: [1] Sentence-BERT — Reimers and Gurevych

A rotation exposes the migration failure

Consider an illustrative two-dimensional space. The old query is [1, 0], and an exactly matching document is also [1, 0], so their dot product is one. Suppose a new encoder applies a ninety-degree rotation to every representation. In the new space both become [0, 1], and their mutual similarity is still one. But a new query compared with the old document has dot product zero. The new encoder can preserve all within-version geometry while cross-version search becomes meaningless. Equal dimension and apparently sensible vector norms do not protect the mixed index from this failure.

The example is deliberately simpler than a real model migration. Real changes need not preserve geometry through a rotation at all, and a learned alignment would require its own validation. Its purpose is to isolate the compatibility issue without invoking model quality. An index can be internally healthy, the new encoder can perform well on its own evaluation, and the combination can still fail because the query and corpus were produced under different contracts. I would make incompatible versions impossible to combine by default, rather than relying on an operator to remember which deployment finished re-embedding first.

def dot(a, b):
    return sum(x*y for x, y in zip(a, b))
old_query, old_doc = [1, 0], [1, 0]
new_query, new_doc = [0, 1], [0, 1]
assert dot(old_query, old_doc) == 1
assert dot(new_query, new_doc) == 1
assert dot(new_query, old_doc) == 0

Version preprocessing and document identity together

The encoder weights are only one dependency. Adding a document title before its body changes the embedded text. Replacing a tokenizer can change which suffix is truncated. A normalization fix can alter distances without changing a single weight. I would define an embedding specification that fingerprints these choices and records the source document revision. The source identifier answers which document this vector describes; the representation identifier answers how it was encoded. Keeping them separate lets a document update invalidate one item while an encoder update creates a new version of the whole representation family.

Chunk boundaries belong in this specification as well. If one document becomes five chunks instead of three, the index now contains a different set of retrievable objects. Old chunk identifiers should not accidentally resolve to new text with different offsets. A stable document identity plus versioned chunk identities makes deletions and citations tractable. I would retain enough metadata to reconstruct the exact text that was embedded, subject to the source’s retention rules. A vector without a resolvable source revision can still return neighbors, but the application loses the ability to explain what the retrieved object actually meant at indexing time.

Measure retrieval quality on the intended questions

Muennighoff and colleagues’ MTEB paper evaluates embeddings across multiple task types and reports that no method dominates all of them in that study. I read that as a warning against treating a broad benchmark position as an application-specific guarantee. A documentation search system, a duplicate detector and a clustering tool use the representation differently. For a migration, I would preserve a query set with relevant document judgments and compare the old and new complete retrieval paths. Include ambiguous names, version-specific questions and cases where the right outcome is that no indexed document answers the request.

The index algorithm also affects observed quality. An approximate search configuration can miss neighbors even when the underlying representation is useful. Separate encoder quality from index recall by comparing a manageable exact-search subset against the approximate path. Otherwise a model upgrade can be blamed for an indexing configuration change, or a more expensive index can receive credit that belongs to a different encoder. The release decision should consider quality, latency, memory and ingestion cost together, but the experiment should retain enough decomposition to identify which component caused each difference and whether that difference is acceptable.

References: [2] MTEB — Muennighoff and colleagues

Migrate through two complete spaces

I would build a new index alongside the old one, with each receiving vectors from its corresponding encoder specification. During backfill, new source revisions need a defined path into both indexes or a replay log that closes the gap before cutover. A snapshot alone is insufficient if documents keep changing while it is embedded. The cutover condition should describe source coverage and revision consistency, not merely the number of vectors processed. A new index containing every old document can still omit changes and deletions that occurred during the migration window.

Shadow queries can compare ranked results before users depend on the new space. Keep the old query encoder available for rollback, because routing a new query representation back to an old index recreates the original compatibility failure. A migration manifest should identify both artifacts and their source watermark. I would also budget temporary storage explicitly: parallel indexes and dual ingestion consume resources before any improvement reaches users. If that cost is unacceptable, a segmented migration may be necessary, but queries must then select compatible partitions rather than compare arbitrary scores across versions as though they were calibrated on the same scale.

Do not carry thresholds across versions by habit

A similarity threshold can encode several policies at once: candidate admission, duplicate detection or a decision to abstain. After an embedding change, its numerical value has no automatic right to survive. Suppose an illustrative old system accepted matches above 0.8. A new representation might place equally useful pairs around 0.65 while preserving their ranking. Reusing 0.8 could destroy coverage without improving the meaning of accepted results. Conversely, a compressed score distribution could admit many irrelevant pairs. Re-estimate threshold behavior on labeled cases and report precision and coverage, rather than copying a configuration constant into the new deployment.

The strongest objection is operational burden: versioning every preprocessing detail can make a small search feature feel like a data platform. I would keep the implementation proportional, but not erase the identity. A compact manifest and two clearly named indexes may be enough. The expensive mistake is discovering an incompatible mixture after results have degraded, when it is unclear which items were encoded by which process. Recording the contract during ingestion is usually cheaper than reconstructing it from deployment timestamps, especially if a failed job resumed under a different model or configuration.

Make compatibility observable

The query path should expose its representation version, index version and source watermark in internal diagnostics. Reject unknown combinations before searching. Track incompatible requests, missing revisions and documents that could not be re-embedded, instead of silently treating those cases as ordinary low relevance. For a controlled test, deliberately send an old query vector to the new index and assert that the interface refuses it. That test checks the contract directly; a large end-to-end relevance suite may not reliably expose a mismatch that happens to return plausible neighbors for common queries.

I want an embedding service to promise more than an array length. It should identify the transformation, the comparison rule and the population of indexed objects to which the output belongs. This does not make semantic retrieval deterministic in the sense of guaranteed relevance. It makes its uncertainty easier to locate. When quality changes, the team can ask whether the representation changed, the corpus changed or the search approximation changed. A versioned interface turns an opaque similarity regression into a set of concrete hypotheses that can be tested and, when necessary, rolled back coherently.