Embedding Model Selection and Vector Quality Evaluation: Dimensions, Domains, and Methodology

Introduction: "Just Swap the Embedding Model and See" Is Not an Evaluation

In a RAG pipeline, the embedding model is the component most likely to get swapped out on gut feeling alone. When retrieval misses, the first reaction for far too many people is "the model isn't good enough" — so you reach for a bigger one, a newer one, or one that ranks higher on some leaderboard, and see whether it feels better. The fatal flaw in that approach is that there is no mechanism telling you whether the judgment "the new model is better" actually holds. A feeling rules out neither noise nor coincidence, and it certainly cannot tell you, at the next model iteration, whether you actually got better or got worse.

It gets worse, because the quality of an embedding model is never an attribute you can look at separately from the system. The same set of vectors, under different chunking schemes, different index parameters, different rerankers, and different query phrasings, can produce recall behavior that varies enormously. A higher average score on a leaderboard, transplanted onto your own corpus, may be neither faster nor more accurate — and may be worse on your actual query distribution.

So what this article is about is not "which model is best," but how to choose an embedding model scientifically and produce evidence that it really is better. There are three core claims. First, selection must be built on explicit trade-off dimensions, not on an aggregate leaderboard rank. Second, any general-purpose benchmark can only be used to eliminate obviously unqualified candidates — never to make the final decision. Third, your own business evaluation set is the only judge, and its quality ceiling determines the quality ceiling of the whole evaluation system.

The article unfolds in this order: first the selection dimensions (dimension, language, text form), then what general benchmarks can and cannot tell you, then task-level evaluation, evaluation set construction, reranker synergy, the boundaries of domain fine-tuning, index parameter masking effects, and finally a trap checklist plus a pipeline skeleton you can run as-is.

1. The First Selection Dimension: The Dimension / Storage / Latency Triangle

1.1 More Dimensions Is Not Automatically Better

Intuitively, 768 dimensions should lose to 1024, and 1024 should lose to 1536. That intuition held in the early days, because model capacity genuinely was tied to dimension back then. But since 2023, publicly available mainstream embedding models no longer automatically deliver better retrieval quality from "more dimensions at the same parameter count," and there are several structural reasons why.

First, the effective rank problem. The intrinsic dimension of the semantic manifold of a text corpus is far smaller than the nominal dimension of the model output. What the extra dimensions add is often just anisotropic noise, amplified along with everything else. In practice the common outcome is this: after adding dimensions, the distribution of cosine similarity becomes more concentrated, or more spread, depending on the model — but either way, the marginal discriminative power of the ranking does not improve linearly.

Second, diluted training signal density. With a fixed batch size and a fixed number of training steps, a higher-dimensional output space spreads the gradient signal thinner: each dimension receives less signal on average. That can leave a high-dimensional model worse than a well-trained medium-dimensional one on certain fine-grained tasks.

Third, cost is a hard constraint. Vector storage is approximately number of vectors × dimension × bytes per element. Going from 1024-dim fp32 to 256-dim fp16 cuts storage straight to about 1/8; and when a single query scans millions of candidates, the FLOPs of the inner-product computation scale linearly with dimension. Dimension is not purely a quality dimension — it is also a budget dimension.

So the correct question is not "how many dimensions is best," but: under the storage and latency budget I can accept, which model delivers the best ranking quality on my corpus. All three variables in that sentence are load-bearing.

1.2 Matryoshka: Turning Dimension into a Tunable Knob

If a model claims to be a variable-dimension embedding trained with Matryoshka, then dimension stops being a model-selection problem and becomes a runtime knob. Such models constrain the prefix-dimension subspace as if it were an independent embedding during training, so truncating to any dimension keeps retrieval behavior relatively stable.

The engineering value here is that the same batch of documents only needs to be encoded once. You can screen a large candidate pool cheaply at the lowest dimension, then recompute just a few candidates at high dimension for precise ranking — or even store vectors at different dimensions in separate segments of the index, querying the low dimension first and the high dimension on demand. This layered retrieval structure pays off markedly more on Matryoshka models than on traditional ones.

But note the boundary: truncation quality after Matryoshka training is not uniform, and the loss usually concentrates in the lowest dimension bucket. The prudent practice is not to treat every dimension as an equivalent option during training or evaluation. Instead, run the evaluation across 2~3 candidate levels (say 256 / 768 / full dimension) first, see where the quality curve inflects as dimension drops, and only then pick a production level.

DIMENSIONS = [256, 512, 768, 1024]   # 仅在模型支持可变维度时有意义


def sweep_dimension(encoder, corpus, queries, gold, k=10):
    """维度是一个实验因子，而不是模型的固有属性。"""
    rows = []
    for dim in DIMENSIONS:
        vecs = encoder.encode_corpus(corpus, truncate_dim=dim, normalize=True)
        row = evaluate(build_index(vecs, metric='cosine'), queries, gold, k)
        row.update(dim=dim, index_mb=len(vecs) * dim * 2 / 1024 / 1024)
        rows.append(row)
    return rows   # 关注质量曲线的拐点，而不是最高那一点的绝对值

1.3 Normalization and Distance Metric: Something You Must Fix Up Front

Before comparing any two models, you have to pin down the measurement convention. The industry mainstream is to train with normalized inner product (that is, cosine), or with normalized dot product plus a temperature (common in contrastive learning); the two produce similar but not identical rankings. On the index side, the metric actually used is decided by the vector database.

The classic way this goes wrong: model A recommends normalize + cosine, model B recommends raw L2, the index configuration then gets set following A's advice — and the difference you end up measuring actually comes from the metric convention rather than from model capability. The correct move is to treat "model + normalization on/off + distance metric + index type + index parameters + reranker" as a single system configuration under evaluation, rather than pulling the model out and comparing it alone.

There is also an easy-to-miss precision detail: normalized vectors stored in fp16 introduce relative error, and that error accumulates in the inner product at high dimension. When the true similarity differences between candidate documents are themselves on a very small scale, storage precision alone can flip the top-k ordering. So evaluation must fix and report encoding precision, storage precision, and compute precision together. Many cases of "the model does slightly worse in production than in offline evaluation" trace back to exactly this, not to the model itself.

The same logic applies on the serving side: if the index service restores fp16 back to fp32 in a way that differs from your local evaluation implementation, you get irreproducible differences. So the configuration fingerprint should include dtype alongside dimension and the normalization switch.

1.4 Model Scale Isn't Free, but It Isn't the Only Source of Quality Either

A larger base model usually brings better semantic understanding, and this shows up consistently on general benchmarks. But when you fold it into an engineering decision, you have to look at three ledgers at once.

The first is offline encoding cost. Full-corpus encoding is a one-time expense, but the corpus keeps growing, and index rebuilds may happen weekly or even daily. Double the base model's parameter count and encoding time roughly doubles with it.

The second is online serving cost. Query encoding and cross-encoder reranking both sit on the critical path. A large model's inference latency and memory footprint directly squeeze the budget you have left for efSearch — and as noted earlier, lowering efSearch is lowering recall.

The third is deployment complexity. Past a certain scale, self-hosting inference demands dedicated engineering investment in memory, concurrency, and quantized deployment; push further, and the diminishing latency returns are usually already obvious.

The practical criterion is not "bigger is better" but how much recall the remaining budget can still buy under a fixed latency budget. If doubling the model size buys less than a percentage point of recall while eating a third of the latency budget — driving efSearch down and making recall worse — then that upgrade is net negative. An evaluation system has to be able to catch this kind of non-monotonic relationship, which is exactly why it must report latency and recall together.

2. The Second Selection Dimension: Language, Domain, and Text Form

2.1 Mixed Chinese-English Corpora and Cross-Lingual Retrieval

A typical characteristic of a Chinese enterprise knowledge base is that the documents are mostly Chinese but peppered with English technical terminology, code identifiers, product names, and acronyms. At that point you have to distinguish two different goals.

The first is monolingual retrieval: Chinese queries retrieving Chinese documents. This demands that the model be robust enough on Chinese long-tail entities and specialist phrasing. The second is cross-lingual retrieval: a Chinese query retrieving English documents, or the reverse. This demands that vectors in different languages land in one comparable semantic space.

Many models that advertise multilingual support genuinely do well at cross-lingual retrieval, but whether that capability is usable in your business is a precondition, not a detail. If your corpus is 95% Chinese with English documents as decorative terminology, the training capacity spent on cross-lingual ability is waste. If your users search English technical docs in Chinese, that capability is a hard requirement.

There is another commonly overlooked point: the representation quality of Chinese-English mixing inside a single chunk is usually lower than that of pure Chinese or pure English. The reason is that when the language switches mid-passage, the model has to reconcile two languages' semantic projections inside one continuous vector. Your evaluation set must contain mixed-language samples — otherwise the first badcase you see after launch will be exactly that.

2.2 Terminology, OOV, and Tokenization Strategy

The hardest part of an enterprise knowledge base is usually not general semantics but private terminology: a product's internal codename, the several ways one person's name is written across systems, an internal acronym. For a general-purpose tokenizer these words will most likely be shredded or split wrong; if the model vocabulary doesn't cover them, you have to fall back on subword segmentation.

Subword segmentation preserves information, but it brings two side effects. First, once a long term is broken into multiple tokens, the quality of its vector representation depends on whether the split is semantically sensible. Second, certain frequent mis-splits make completely unrelated documents look spuriously similar because they share fragments — especially common in technical documentation, where everyone talks about APIs, tokens, and embeddings.

There is a direct implication for evaluation set design: you need a group of pure terminology queries — no syntactic cues at all, just a product name or a person name plus a vague intent. These queries are the touchstone for telling "strong general capability" apart from "actually adapted to your domain," because general benchmarks almost never contain such samples.

2.3 Long Documents and Code: Two Systematically Underestimated Forms

For the vast majority of embedding models, the effective input length sits at a few hundred tokens, and the training corpus consists mostly of sentences and short paragraphs. This produces a very typical failure mode: when a single chunk approaches or exceeds the model window, meaning gets diluted. The model doesn't "compress" a long document into a more complete summary; instead the later parts lose weight in the vector, so the similarity between a long chunk's vector and a short query ends up determined mostly by the leading content.

There are three responses, ordered by cost. First, control chunk length — don't expect long-document embedding models to save you. Second, do a second encoding pass with a long-context-optimized model (re-encode the long documents among your recalled candidates). Third, build hierarchical summaries of long documents first and embed those, which has a very favorable cost-benefit ratio when an LLM is available.

Code is a separate problem. General-purpose natural language models are uniformly weak at the semantics of code: what they learned is the "natural language description" of code, not its structure and behavior. This directly breaks requirements like "retrieve the relevant code for this error message." For code retrieval, either pick an embedding model specifically trained on code corpora, or at minimum confirm that the model's evaluation included real code retrieval tasks rather than only code summarization.

2.4 Domain Distribution Shift: Easier to Miss Than Language

The dimension that gets the most serious attention during selection is language, because it is enumerable, describable, and easy to discuss. But what actually determines the outcome is often domain distribution shift — the distance between your corpus and the model's training corpus in topic, register, and recency.

Shift shows up in several directions. The first is register shift: training corpora skew toward narrative text, while your documents may be mostly table dumps, log excerpts, config listings, ticket descriptions — texts with very different semantic density and syntactic structure, so the vectors are naturally harder to align. The second is topic concentration: if the corpus is highly concentrated on a handful of topics, the model's learned general relevance judgment — "which topics ought to match which topics" — may not be reliable on those narrow topics, especially when distinctions inside a topic can only be made through fine-grained terminology.

The third is recency. New concepts, new product forms, and new technology stacks from the last two or three years are thinly covered in many models' training data. If your evaluation set consists mostly of historical documents, it will systematically underestimate this problem. So the evaluation set should deliberately include some recently added content rather than sampling uniformly by time.

There is a cheap way to judge the degree of shift: manually scan the model's zero-shot top-10 results grouped by document source. If one document keeps showing up in the results of a large number of completely unrelated queries, that means those fragments are represented in a generic, low-discriminative way in vector space — usually a direct signal that the model isn't suited to this kind of text. Conversely, if results are highly concentrated on a small handful of documents, the model may carry a bias, and that is worth chasing down too.

3. What General Benchmarks Tell You, and What They Deliberately Don't

3.1 How MTEB Is Built, and What Its Average Hides

The value of a benchmark like MTEB is clear: it standardizes a dozen-odd retrieval, reranking, classification, and clustering tasks and gives you a coordinate system that is comparable across models. For someone doing a technical selection for the first time and needing a starting point, it can quickly eliminate obviously unqualified candidates — say, a model that is significantly behind on the multilingual retrieval subset. If your business is primarily Chinese, that kind of information has real decision value.

But its average score has four structural problems.

• The average hides variance. A model may rank near the top on average simply because it scores well on very large English tasks like "argument retrieval," while sitting in the middle on Chinese tasks like "financial regulation retrieval." The average squashes those two facts into one number.
• Task sizes are wildly unbalanced. Some subtasks have hundreds of thousands of queries; some have only a few thousand. Scores on the latter fluctuate enormously on their own, yet they still enter the average with equal weight.
• Corpus distribution differs from business distribution. Benchmark corpora are public, cleaned, and multilingual, while yours is noisy, non-standard, and thick with internal jargon. Different distributions can mean a different optimal model.
• Saturation and contamination. As more models train and distill on public data, benchmark scores shift upward overall and discriminative power drops; at the same time you cannot rule out that some models' training corpora contain the benchmark data itself.

3.2 The Leaderboard Keeps Changing, and Experience Does Not Update Itself

Beyond those four structural problems, there is a fifth: the subset composition, scoring conventions, and participant model list of an evaluation benchmark all keep iterating. Experience from last year easily stops holding on this year's version. When you come away with the impression that some model "was very strong back then," first check whether that version's subset composition matches your current business, and only then decide whether to admit it as a candidate.

Equally important is not treating leaderboard rank as a character judgment. A model topping one version of a benchmark means neither that it is best in Chinese technical scenarios nor that it will still be first two years from now. There is only one category of transferable judgment: under what conditions the technical approach behind a model wins, and under what conditions it fails. Only conclusions of that mechanistic kind belong in a selection document, and they are worth revisiting a year later.

Scores that were pulled from a leaderboard mid-flight also deserve attention. When maintainers discover data contamination or a scoring problem, they often delist a model outright rather than correct its score. When you see a model vanish from a leaderboard, your first reaction should not be "it must be bad," but rather to check whether it was removed for capability reasons or for methodology-compliance reasons — the lessons differ completely, and the latter may even suggest that this technical direction deserves a closer look.

3.3 Sub-Benchmarks Have Their Own Applicability Boundaries

Some people settle for the next best thing and look at multilingual versions of the benchmark, Chinese sub-benchmarks, code sub-benchmarks, long-document sub-benchmarks. That is far more reasonable than looking only at the overall leaderboard. But even for these, you still have to ask: is this sub-benchmark's corpus, task form, and difficulty distribution close to your scenario?

A more critical and counterintuitive observation is this: scores on dedicated Chinese benchmarks often correlate with actual Chinese retrieval experience far less strongly than people assume. The reason is that documents in these sub-benchmarks are usually cleaner, better structured, and shorter than real enterprise documents. The challenges of the real world — long-chunk dilution, terminology fragmentation, OCR noise from scanned documents, multiple document versions contaminating each other — essentially do not appear in the benchmark.

Another mechanism deserves attention: a reranker often eats part of an embedding model's leaderboard advantage. If your architecture has a reranker, then an embedding model that is 5 points weaker, paired with a strong reranker, may show only 1~2 points of difference at the end. Conversely, an embedding model that is 10 points stronger, if your reranker is already strong, may show its gain on extremely hard cases rather than on the average. This directly means that in a system with reranking, choosing the embedding model by leaderboard alone systematically overestimates what the top models buy you.

3.4 When You Can Trust the Leaderboard Alone

Not every situation requires building your own evaluation set. In three cases you can rely directly on public benchmarks plus a small amount of sampling validation: the corpus is public and general-purpose (say, English Wikipedia or public papers); the task form is the standard "short query retrieves a short passage"; and the team has no annotation capacity and, at the current stage, only needs a usable starting point.

Once you leave those three cases, building your own evaluation set stops being optional. The leaderboard cannot tell you the three things that matter most: whether your document chunking puts the target information inside the same chunk; what your real queries actually look like; and how your corpus noise affects a given model's robustness.

3.5 The Three Legitimate Uses of General Benchmarks

Dismissing leaderboards wholesale is laziness of another kind. They are genuinely useful in three places, and using these three well can substantially cut the cost of your own experiments.

1. Eliminating obviously unqualified candidates. Falling far behind on the multilingual sub-benchmark, collapsing on the Chinese subset, losing out on long-document tasks due to truncation — signals like these are enough to delete a model from your candidate list without measuring it yourself.
2. Establishing the first-round candidate set. The leaderboard gives a roughly reasonable prior, so your coarse screening need not start from zero; you can concentrate limited compute on the three to five genuinely promising candidates.
3. Serving as a pipeline self-check baseline. When you measure a suspiciously low number yourself, your first reaction should be to suspect the evaluation implementation — chunking, normalization, k, filtering logic, encoding prefixes — rather than the model. A common debugging order is exactly this: first reproduce a known-magnitude score on a public benchmark to confirm the pipeline, then switch to your own data.

Keeping those three points straight makes it less likely you slide into the opposite extreme — abandoning quantitative comparison because "leaderboards are untrustworthy" and retreating to pure subjective feel. That is the genuinely indefensible option: it is neither reproducible nor able to answer, six months later, whether we actually got better.

There is one more usage discipline: do not decide which sub-benchmark to use after seeing the results. If you look at the total score first and then pick whichever subset flatters you, that is selective reporting, which is entirely different from the three legitimate uses above. The choice of sub-benchmark should be written down before the experiment starts, based on your business distribution.

4. Task-Level Evaluation: Turning "Quality" into Measurable Intermediate Quantities

4.1 Metric Selection: The Difference Between Recall@k, MRR, and nDCG

The first step of task-level evaluation is to collapse the vague phrase "retrieval quality" into a concrete set of metric definitions. There is an easy-to-confuse but very important distinction here.

If you care about "is it in the candidate set at all," use Recall@k. If you care about "will the user see it at first glance," use MRR or Hit Rate@1. If you care about "how well are the top ten ordered relative to each other," use nDCG@k. The model that is optimal for one of these may be different from the model that is optimal for the others.

For a typical two-stage RAG setup (vector recall + cross-encoder reranking), the correct approach is to use metrics that match each stage's responsibility: at the recall stage, Recall@100 (are the candidates enough); at the rerank stage, nDCG@10 (are they ordered correctly); end to end, final answer correctness or human rating. Then when a production metric drops, you can immediately locate which stage degraded, instead of only knowing that "search got worse."

There is also a rule of thumb: the value of k should be determined by downstream capacity, not by habit. If your reranker only processes the top 20, then ranks 21 through 100 in Recall@100 are recalled for nothing. If you plan to use an LLM to do a second generation pass over the top 5, then Recall@5 is the metric that matters.

There is also a detail that tends to get defaulted away: different metrics make different assumptions about how many correct answers exist. Recall means something completely different when there is one relevant document versus when there are many — the former is close to a hit test, the latter becomes coverage. On the same evaluation set, a model may raise Recall by pulling back a few more weakly related documents while precision drops at the same time. So your report should give Recall and precision together, and state whether the annotation allows multiple correct answers. Reporting only one of them is hiding half the information.

from __future__ import annotations
import math
import random
from typing import Dict, Iterable, List, Sequence


def dcg(gains: Sequence[float]) -> float:
    return sum(g / math.log2(i + 2) for i, g in enumerate(gains))


def ndcg_at_k(ranked_rel: Sequence[int], k: int = 10) -> float:
    """ranked_rel: 该 query 下按检索顺序排列的每篇文档的相关性等级。"""
    gains = [float(r) for r in ranked_rel[:k]]
    ideal = sorted((float(r) for r in ranked_rel), reverse=True)[:k]
    idcg = dcg(ideal)
    return dcg(gains) / idcg if idcg > 0 else 0.0


def recall_at_k(retrieved: Iterable[str], relevant: Iterable[str], k: int = 10) -> float:
    rel = set(relevant)
    if not rel:
        return 1.0
    head = list(retrieved)[:k]
    return sum(1 for d in head if d in rel) / len(rel)


def hit_rate_at_k(retrieved: Iterable[str], relevant: Iterable[str], k: int = 1) -> float:
    rel = set(relevant)
    return 1.0 if any(d in rel for d in list(retrieved)[:k]) else 0.0


def bootstrap_ci(values: List[float], n_boot: int = 2000,
                 alpha: float = 0.05, seed: int = 7) -> tuple:
    """配对比较前先看区间：区间大幅重叠时，单点均值的差异没有意义。"""
    rnd = random.Random(seed)
    n = len(values)
    means = []
    for _ in range(n_boot):
        sample = [values[rnd.randrange(n)] for _ in range(n)]
        means.append(sum(sample) / n)
    means.sort()
    return means[int(alpha / 2 * n_boot)], means[int((1 - alpha / 2) * n_boot)]

4.2 The Directional Bias of Metrics

Recall-type metrics have an inherent bias: they are naturally favorable to queries whose relevant documents are many and scattered, and naturally unfavorable to queries with a single correct answer buried deep. This means that if broad-intent queries are over-represented in your evaluation set, you will select a model biased toward high recall and low precision — and the reverse holds as well.

So metrics must be reported together with the composition of the evaluation set. The safer approach is to report stratified by query type: navigational (clear intent, unique answer), informational (broad intent, needs several documents), exclusionary (the user actually wants a negative answer), and multi-turn contextual (the query depends on history). The share of each query type in real traffic differs enormously, and different models' strengths and weaknesses across types are frequently reversed.

4.3 Stage Attribution: Turning Metrics into a Contract

End-to-end metrics are the most honest, but also the noisiest and the hardest to attribute — because the randomness of the generation stage itself drowns out 2~3 point differences on the retrieval side. In practice you should therefore establish a metric contract:

Stage	Primary metric	Secondary metric	Typical failure signal
Vector recall	Recall@k	Latency P99, index size	Recall goes up but latency doubles
Filtering / rewriting	Candidate hit rate before vs after filtering	Fraction wrongly removed	A rule deleted the correct document
Cross-encoder reranking	nDCG@10	Top-1 change rate	Reranking costs more time than recall itself
Context assembly	Answer coverage	Redundancy rate	Evidence is truncated or full of noise
Generation	Answer correctness	Citation accuracy	Retrieval was right but the model answered wrong

The value of this table is that it cuts the vague question "is the retrieval algorithm good" into five sub-problems that can be measured and regressed separately. Only this way will you know, on the next iteration, which knob to turn.

This table has a second use that is often overlooked: it doubles as the directory for regression tests. Each row maps to a set of small tests that can run and assert independently, wired into CI. When someone changes the parsing stage months from now, the change gets caught by that row's assertions, without anyone having to wait for the end-to-end metric to drop and then do archaeology. Turning metrics into assertions is the key step that moves an evaluation system from a report into engineering infrastructure.

4.4 Significance: Don't Get Fooled by Differences After the Decimal Point

An evaluation set usually has only a few hundred queries. At that scale, the confidence interval around a metric mean is often very wide. An embedding model's Recall@10 being 1.5 points above another's may well be nothing but sampling noise. The correct approach is a paired comparison: use the same batch of queries, compute the per-query metric difference between the two models, then bootstrap-resample that difference sequence to judge whether the distribution of differences is stably greater than zero.

def compare_runs(run_a: Dict[str, float], run_b: Dict[str, float],
                 n_boot: int = 4000, alpha: float = 0.05) -> dict:
    """配对自助检验。run_* 是 {query_id: 指标值}，必须来自同一批 query。"""
    ids = sorted(set(run_a) & set(run_b))
    if not ids:
        raise ValueError('no overlapping queries')
    diffs = [run_a[q] - run_b[q] for q in ids]
    wins = sum(1 for d in diffs if d > 1e-9)
    losses = sum(1 for d in diffs if d < -1e-9)
    observed = sum(diffs) / len(diffs)
    rnd = random.Random(13)
    ge = 0
    for _ in range(n_boot):
        sample = [diffs[rnd.randrange(len(diffs))] for _ in diffs]
        if sum(sample) / len(sample) >= observed - 1e-12:
            ge += 1
    p = ge / n_boot
    return {
        'delta': round(observed, 4),
        'wins': wins,
        'losses': losses,
        'n': len(ids),
        'p_value': p,
        'significant': p < alpha,
    }

Note that the decision logic here is not "small p means switch," but "only a small p is worth discussing further." For an insignificant small difference, the correct conclusion is "these cannot be distinguished on this batch of data," not "pick the cheaper one." These two sentences are constantly conflated in practice, leading teams to make decisions over and over on nothing but noise.

4.5 Keeping the Evaluation Set Hard: Injecting Distractors

Evaluation sets decay naturally: as the system gets stronger, samples that once had high discriminative power gradually become "things every model gets right," metrics saturate, and no amount of running gets you signal again. Adding samples certainly helps, but the more economical approach is to inject distractors deliberately — because a distractor does not require re-annotating query correctness; you only have to judge whether the ordering is right.

There are three common approaches, ordered by cost.

• Paraphrase distractors: for each correct answer, construct a semantically equivalent distractor document with different wording. It specifically targets scenarios where "everything looks relevant but only one actually answers the question," and is the most effective way to separate a general model from a domain-adapted one.
• Cross-version distractors: put an older version of the same document into the candidate set. This corresponds to the most common class of badcase in real systems — a stale answer got retrieved even though it is semantically perfectly correct.
• Cross-document fragment distractors: cut a passage out of an entirely unrelated document that shares a large amount of vocabulary with the query. This targets the morphological mismatch problem mentioned earlier, which almost certainly exists in any technical documentation corpus.

These three classes of distractor share one property: the existence of the correct answer is certain, and the judging cost is extremely low. Building an evaluation set with distractors is often far cheaper than annotating another batch of queries, yet it can reopen a gap in metrics that have already saturated.

4.6 Sensitivity Analysis over Candidate Set Size k

Earlier I said k should be determined by downstream capacity, but that principle also has an operational implication: k is not just a parameter, it is a variable that deserves its own sensitivity analysis.

The full approach is to record three curves across a set of k values (typically 10 / 30 / 100 / 300): Recall@k reflects the recall ceiling, nDCG@k reflects the effect after precise ranking, and end-to-end answer quality reflects the real business payoff. The saturation points of the three curves usually do not coincide, and the genuinely important information is the region where they intersect.

For example: if Recall@100 is already near saturation and Recall@300 barely improves further, investment in the recall stage is no longer paying off. But if nDCG@30 differs clearly from nDCG@10, reranking still has room to work, it just needs a larger candidate set. These two readings lead to completely different optimization directions, and that structure is invisible if you only look at a single number under a single k.

Conversely, if some curve actually declines — say answer quality gets worse as k grows — there is almost always a concrete engineering reason behind it: longer context bringing more noise, reranking timing out and being truncated, or a token budget squeezing out the key evidence. This kind of non-monotonic behavior is the most valuable output of the k analysis, because it points directly at a problem that genuinely exists in production.

4.7 The Cost of End-to-End Evaluation, and How to Simplify It

End-to-end evaluation is the only method that can answer "are users actually satisfied," but it has two practical difficulties: the judging itself is noisy, and the randomness of the generation stage drowns out retrieval-side differences. The former is mitigated by using multiple annotators and taking agreement; the latter can only be diluted through repeated sampling.

One workable simplification is to downgrade end-to-end evaluation from "run it on every change" to "run it once per candidate release": use a large-scale evaluation set at the retrieval layer as the daily gate, and only invest human effort in end-to-end evaluation once a candidate release passes the significance test at the retrieval layer. This guarantees that substantial changes get verified, without letting end-to-end evaluation become the pipeline's bottleneck.

Another way to simplify is to replace free-form judgment with closed questions: in advance, write down "which points must a correct answer cover" for each query, and at evaluation time only check whether those points appear. The judging space shrinks from "is this answer good" to "yes or no on a few points," and noise drops significantly. The cost is that it cannot measure expression quality, but expression quality is usually not what retrieval selection cares about — if it is, then the conversation is no longer about the same problem.

5. Building a Small Evaluation Set That Actually Fits Your Business

5.1 Data Sources, Ordered by Confidence

The data sources for an evaluation set, from lowest cost to highest, roughly form four tiers with rising confidence:

1. Production log sampling. The most realistic, but it requires handling length skew — users tend to type long queries, which biases the evaluation set toward long-tail hard cases. It also requires desensitization, especially in internal-document scenarios.
2. Annotation. Have annotators judge, for a batch of sampled queries, "which documents can answer this question." This is the standard approach; the cost sits mainly in annotation guideline design and consistency checking.
3. Reverse-engineering from clicks and feedback. Treat queries with interaction as positives and those without as negatives. The cost is extremely low but the noise is high, because no interaction may mean the user already knew the answer, or that there was a UI problem.
4. LLM synthesis plus human verification. Good for covering scenarios that are rare in real logs but important to the business, such as pure terminology queries or cross-lingual queries. A human verification step is mandatory, otherwise synthetic data carries the generator model's systematic preferences and structurally biases the evaluation set toward one class of model.

The practical approach is usually a mix of sources, with a source label recorded for every sample, so that later you can report performance stratified by source and run sensitivity analysis when you suspect source bias.

5.2 The Lower Bound on Size: 300 Stratified Samples Beat 5000

Anxiety about evaluation set size is widespread, but a bigger evaluation set does not equal a better decision. A more practical goal is that each business stratum has enough samples to support a conclusion, not that the total is large.

One operable reference: split queries into 4~6 types (navigational, broad informational, terminology, cross-lingual, context-dependent, negative), at least 50 per type, 300~500 overall. At that scale, metric differences of 5~10 points are usually decidable; differences of 1~2 points usually are not, and you should honestly say "cannot be distinguished."

Rather than spending effort on sheer volume, spend it on raising annotation quality. One practical approach is to annotate, for each query, only the set of documents "sufficient to answer it," rather than all relevant documents — consistency on the latter is terrible, because different annotators draw the boundary of "relevant" in completely different places.

5.3 Stratified Sampling and Difficulty Slices

When deciding "how many per type," there is one piece of experience worth remembering: sample size should be allocated according to "how reliable do I need the conclusion for this type to be," not according to traffic share. The type with the largest traffic share is usually the easiest to add samples to, but its metrics are also the most stable and the differences the smallest. What actually decides the selection is those categories with few samples and high variance — and they are precisely the ones that need more samples to support a judgment.

There is a very useful operational trick: first run a baseline to get a difficulty label for every query, then back out the required sample size from "I need to distinguish a 5-point difference in this category," and check afterward whether it holds. For a category that falls short, either add samples, or explicitly write in the report "insufficient samples for this slice, conclusion unusable." Explicitly writing "unusable" is far more useful than reporting a number with huge jitter, because the latter always ends up being used to argue something.

If you can only sample 300 queries at random from logs, you will very likely draw a set that is too easy, where every model scores 95 and nothing can be distinguished. The better approach is to run a baseline first and then sample stratified by baseline difficulty:

Slice	Definition	Purpose
Easy	The baseline model already ranks it into the top 1~3	Prevents the sample set from being so easy that it saturates; these usually cannot decide the selection
Medium	The baseline model recalls it but cannot rank it into the top 5	The main battlefield of the reranker, and the main interval for distinguishing embedding models
Hard	The baseline model fails to recall it	Exposes lexical mismatch and domain drift; model differences are most visible here
Adversarial	Highly similar but incorrect distractor documents exist	Tests fine-grained discrimination, separating general models from domain-adapted ones

A second source of hard cases is hard negative mining: retrieve with the current model, collect the documents that rank high but are not relevant, and add them as distractors. This can significantly raise the discriminative power of the evaluation set. But watch out carefully for circular reasoning — hard negatives mined by model A, used to evaluate an improved version of model A, are naturally biased toward A. The correct approach is to mine negatives in rotation with several models from different sources, or at minimum to construct the distractor set with a model unrelated to the one under test (say BM25, or a model of a different architecture).

5.4 Preventing Leakage: The Evaluation Set Needs Its Own Lifecycle

The biggest risk to an evaluation set is not that it is small, but that it gets contaminated by the optimization process. Once a team starts repeatedly looking at one evaluation set's scores and tuning prompts, weights, and models accordingly, that set has become a validation set, no longer a test set. Scores keep climbing while production improves not at all.

Safeguards include at least three measures. First, maintain a fixed "pre-release holdout set," stipulating that it may be run only a limited number of times during model iteration (for example once per candidate release), and that per-item details are not published. Second, slice the evaluation set by time and use a newer time slice as the holdout. Third, if the evaluation set comes from production logs, split it by user or session rather than by randomly splitting individual queries — if near-duplicate questions from the same user appear in both the tuning set and the evaluation set, you will badly overestimate results.

import random
from collections import defaultdict
from typing import Dict, Iterable, Iterator, List, Tuple


def group_kfold(records: List[dict], folds: int = 5,
                group_key: str = 'session_id',
                seed: int = 0) -> Iterator[Tuple[List[dict], List[dict]]]:
    """按会话/用户分组切分，避免同一话题的近重复 query 跨越训练与评测两侧。"""
    groups = sorted({r[group_key] for r in records})
    random.Random(seed).shuffle(groups)
    buckets = defaultdict(list)
    for i, g in enumerate(groups):
        buckets[i % folds].append(g)
    for f in range(folds):
        test_groups = set(buckets[f])
        train = [r for r in records if r[group_key] not in test_groups]
        test = [r for r in records if r[group_key] in test_groups]
        yield train, test


def assert_no_overlap(train: Iterable[dict], test: Iterable[dict],
                      group_key: str = 'session_id') -> None:
    a = {r[group_key] for r in train}
    b = {r[group_key] for r in test}
    leak = a & b
    assert not leak, 'group leakage detected: %d' % len(leak)

5.5 The Three Things the Annotation Guidelines Must Nail Down

The ceiling of an evaluation set is annotation quality, and the ceiling of annotation quality is the guidelines. If the following three are not spelled out, annotation consistency will certainly collapse.

The first is binary judgment versus graded relevance. Binary judgment asks annotators "can this document answer this question," and the boundary is inherently fuzzy; three-level annotation (directly answers / partially answers / merely relevant) has much clearer boundaries, but costs more and constrains metric choice. In practice binary is more common, and what matters is providing enough judged examples, especially examples at the boundary.

The second is what to do when several documents are relevant. Requiring annotators to mark all relevant documents introduces enormous noise, because annotators differ wildly in judging "is there another answer." One workable compromise: allow annotating multiple relevant documents, but require them to be ranked by relevance, and require at least one best document to be explicitly marked. This preserves the meaning of recall while making nDCG computable.

The third is what to do with queries whose answers simply do not exist in the corpus. Dropping them outright costs the evaluation set the dimension of testing refusal to answer, which is exactly an important category of production experience. The recommendation is to maintain a separate "should refuse" set used to calibrate the similarity threshold — this set is usually far smaller than the main evaluation set, and a few dozen items can tune the threshold quite stably.

Once these three are written down, be sure to run one round of double independent annotation and compute the agreement rate. When agreement falls short, fix the guidelines first rather than expanding the sample size — under unclear guidelines, the more you annotate, the faster noise accumulates, and eventually the whole batch has to be redone.

If you plan to use a large model as an automatic judge, you need to watch two extra biases. One is position bias: a judging model naturally scores candidates placed earlier in the list higher, so you must shuffle the order or run position-swapped control experiments. The other is length bias: more verbose candidates tend to be judged as "more complete," even when they do not answer the question more accurately. The mitigation is to explicitly require in the prompt that only the presence of the information needed to answer the question be judged, and to calibrate the automatic judge's accuracy against a known human-annotated subset. If calibrated agreement is still insufficient, just use humans — that is not an efficiency problem, it is a credibility problem.

5.6 The Evaluation Set Needs Its Own Update Mechanism

An evaluation set is not a one-off project; it expires. The corpus grows, historical documents get deleted or revised, and the way users phrase queries changes. An evaluation set built a year ago no longer represents the real behavior of the current system.

A workable update cadence is: fix an immovable core set for cross-version comparison, and maintain a separate dynamic set that rotates quarterly to surface new problems. The core set is small and stable, dedicated to regression comparison, with samples rarely swapped out; the dynamic set is sampled from the latest logs, allowed to be replaced, and used to capture new failure modes.

You also have to monitor an easily overlooked signal: drift between the production query distribution and the evaluation set distribution. A simple approach is to periodically compute, over new logs, the length distribution of queries, their vocabulary distribution, and their similarity distribution against the evaluation set. If the drift is obvious, the evaluation set no longer represents production, and you should trigger a rebuild rather than keep comparing models on the old set.

5.7 Growing the Evaluation Set Through Badcases

The healthiest way for an evaluation set to grow is not periodic vague sample expansion, but feeding production badcases back into it one by one, in a structured way. The key to this loop is those three words: structured.

A badcase is worth keeping if you can answer three questions clearly: what was the user's query at the time, which documents did the system return, and what the correct answer should have been. The third is usually the hard part — if nobody can say what the right answer is, it is just one moment of confusion; if you can localize it, it becomes a high-value sample, because it naturally corresponds to a real failure of the system.

In practice you can give each accumulated sample three labels: a root cause label (chunking problem, recall failure, reranking error, generation hallucination, threshold too high), a form label (terminology, long document, cross-lingual, context-dependent), and a source label (production log, manually constructed, regression finding). With those three labels, the evaluation report can aggregate by root cause: which failure modes a model actually fixed, and which ones it broke. That carries far more information than looking at an average-score improvement, and it is much easier to convince people with.

There is one side effect to guard against: an evaluation set constructed from targeted badcases overfits rapidly. The team unconsciously optimizes around currently known failure modes and is completely unprepared for problems that have not yet surfaced. So this set should exist in parallel with a broader stable set, and the ratio should be controlled — somewhere around 30% to 40% for the badcase set, with the stable set covering the rest, is reasonable.

6. Rerankers and Embedding Models: Synergy, Not a Showdown

6.1 The Responsibility Boundary of the Two Stages

Embedding models and cross-encoder rerankers have a clear division of capability, and that division determines what you should look at during selection.

The embedding model is responsible for recall, that is, narrowing a large-scale candidate set down to a small, manageable set of possibly relevant documents. Its core capability metric is semantic coverage — whether the answer can be fished out without dropping anything. It is trained with two-tower contrastive learning and cannot see fine-grained interactions between query and document, so it is good at semantic-level matching and weak at precise fine-grained correspondence.

The reranker is responsible for precise ranking. It encodes the query and candidate documents together in full, so it can capture token-level correspondences and logical constraints. Its core capability metric is ranking accuracy. Its compute scales linearly with the number of candidates, so it can only run over the small set that survives recall.

This division yields an important selection corollary: if you plan to use a strong reranker, the embedding model only needs to "not miss on recall," not "rank very accurately". That means a mid-tier embedding model plus a strong reranker can easily beat a top-tier embedding model plus a weak reranker. Put the other way, a small advantage in the embedding model shows up nowhere if the reranking stage gets both candidates right. So the correct unit of evaluation is always the "embedding + reranking" combination.

6.2 The Right Way to Evaluate Them Jointly

When evaluating jointly, candidate set size k is a key parameter that must be explicitly fixed. Raising k from 100 to 500 usually improves rerank quality (because the correct answer gets pulled in), but the latency cost rises linearly. So:

• Fix the constant recall_k when comparing across embedding models (for example always take the top 100 into reranking), so that what you compare is the final ordering quality after reranking.
• Fix an end-to-end latency budget (say 300ms) and let each embedding model choose the k and efSearch it needs, so that what you compare is the final quality within budget.
• Run both perspectives. If the two perspectives disagree, the system sits at a sensitive spot on the trade-off frontier and needs finer slice analysis.

Another practical technique is to measure the marginal contribution curve of reranking: for a fixed embedding model, run reranking at top 20 / 50 / 100 / 200 and see when nDCG@10 saturates. That inflection point tells you how many effective candidates the reranking stage is actually processing, and directly determines the rerank batch size and timeout settings.

Finally, note what is special about the late interaction (multi-vector representation) family of approaches: it bakes fine-grained matching capability into the embedding stage and can theoretically replace reranking. But its index cost is dozens to hundreds of vectors per document, usually requiring compression. When evaluating such a scheme you must report retrieval quality and index size together, otherwise the conclusion cannot be compared against single-vector approaches.

6.3 An Evaluation Perspective on Sparse, Dense, and Hybrid

Before evaluating any fusion strategy, there is a more basic step that gets skipped: establish your own control baseline for the reranking stage. Many teams have never recorded what happens when you swap the reranker while holding the embedding model fixed, so they attribute the improvement from reranking and the improvement from the embedding model together, and end up with conclusions like "the embedding model matters a lot" that may be entirely wrong.

The measurement is not complicated, but it has one precondition: you need a ranking result produced without reranking to serve as fixed input. Hold that input fixed, swap in different rerankers, and the difference is the clean contribution of the reranking stage. This precondition is key because if every reranking experiment re-runs recall, the random fluctuation on the embedding side leaks into the comparison, and the conclusion becomes untrustworthy again.

Once attribution is separated, prioritizing usually goes in an order contrary to intuition: a medium-to-good embedding model with a strong reranker may be more stable than a top-tier embedding model with a weak one. Conversely, if your reranker is already near its capability ceiling and increasing k no longer helps, then the recall ceiling on the embedding side becomes the real bottleneck, and only then is switching embedding models a worthwhile investment.

Sparse retrieval is orthogonal to dense vectors on the capability axis: it relies on exact lexical matching, so it works naturally for proper nouns, model numbers, error codes, and rare acronyms — precisely the region where dense retrieval most often fails. Its weaknesses are equally clear: it is nearly powerless against paraphrases, cross-lingual queries, and colloquial phrasing.

From an evaluation standpoint this means the gains of the two cannot simply be added together. If your evaluation set contains almost no terminology queries, you will underestimate the value of sparse retrieval; if it is all terminology queries, you will underestimate dense retrieval. So hybrid evaluation must guarantee that both kinds of samples have sufficient weight, and must report each path's contribution separately.

A practical attribution method is to split the candidate set into three buckets — "sparse hits only," "dense hits only," and "both hit" — and report final ordering quality for each. If the "both hit" bucket has noticeably higher quality, the two paths are ranking the same batch of correct answers and the overlap is waste; if the "both hit" quality instead drops, the fusion strategy is pushing good candidates out. This split costs very little extra computation, yet it directly tells you how to tune the fusion parameters.

One more judgment directly relevant to selection: if evaluation surfaces "terminology queries are the main failure mode," the correct conclusion is often not to switch embedding models but to add a sparse retrieval leg. That costs far less than fine-tuning and carries no risk of general capability collapse.

6.4 When the Embedding Selection Step Should Not Be Touched

There is a common situation that makes the whole "pick an embedding model" exercise meaningless: reranking is already strong and the recall stage has plenty of headroom. The test is simple: set recall_k to 200 and check whether Recall@k is already near saturation (say >0.95), and check whether nDCG@10 differs little between recall_k of 50 and 200. If so, the embedding model is already the weakest link in the system, and the correct order of investment is to fix reranking and chunking first, and only then consider changing the embedding model.

Conversely, if the Recall@k curve is still climbing clearly as k grows, the bottleneck is in recall, and switching embedding models or tuning index parameters has a much higher marginal return.

7. The Return Boundary of Domain Fine-Tuning

7.1 What Fine-Tuning Actually Changes

The usual approach to domain fine-tuning is contrastive learning on your own real query-document pairs (pulling positive pairs together, pushing negative pairs apart), or adding a projection layer on top of an existing embedding model for lightweight adaptation. What it can actually change is quite specific:

• Representation alignment of terminology and phrasing. Teaching the model that "ticket suspended" and "order paused" may mean the same thing in your business.
• Discrimination among hard negatives. This is the largest single gain, especially when there is a large body of highly similar near-synonymous documents.
• Discriminativeness of the output space. Reducing cases where irrelevant documents score spuriously high because they share generic vocabulary.

What it cannot change is equally clear: how the tokenizer splits private terminology, the model's dilution problem on long documents, and the interface contract between the embedding and the generation layer above it. Those are architectural problems, and fine-tuning does not solve them.

While we are here, one technical detail that often gets skipped: the choice of loss function directly affects how the model performs on fine-grained discrimination. Triplet- or pairwise-ranking-based losses emphasize relative order, suit ranking-quality-oriented scenarios, and are more robust to sampling noise; pure contrastive loss has a higher ceiling but depends more heavily on negative sample quality. The former is stable, the latter is high — and that trade-off points at something routinely underestimated: how you mine negatives usually matters more than which loss you pick. Which is exactly the job an evaluation set should be doing.

7.2 Four Scenarios Where the Gains Vanish

The first is insufficient data. Contrastive learning is very sensitive to the number of negatives; a few thousand pairs rarely produces stable gains and overfits easily to that batch, showing up as beautiful evaluation numbers and unstable production behavior. Research suggests that the stability of gains improves roughly with the number of positives, but the threshold varies a lot across tasks; in practice "can it be reproduced reliably" should be the only criterion, not the score from a single run.

The second is insufficient document coverage. Fine-tuning on department A's queries often degrades the model on department B. This is the most easily missed form of degradation, because it is invisible in aggregate metrics and only shows up when you slice by business line.

The third is contamination of general capability. Training for a long time on a narrow domain makes the model collapse toward that domain, and it clearly drops points on cross-domain or cross-lingual retrieval. If your system contains multiple domains, or if long-tail new domains have to be added at any time, the long-term maintenance cost of that narrowing can exceed the current-period gain.

The fourth is amplified upstream change. A fine-tuned model is bound to a specific tokenization scheme and a specific maximum length. The day you change your chunking strategy, or swap to a base model with a shorter limit, the entire fine-tuning asset is wasted. By comparison, the combination of a general model + prompting + hard-negative reranking is far cheaper to replace.

7.3 Cheaper Alternative Paths

Before deciding on fine-tuning, there are several paths that cost an order of magnitude less and are worth exhausting first.

First, use an LLM for hard negative mining plus reranking, leaving the embedding model alone. The gain here often covers most of what fine-tuning would give, because discrimination capability is essentially provided by the reranking stage.

Second, use hybrid retrieval to make up for exact terminology matching. Vector retrieval is good at semantics but weak at exact lexical matching; combined with BM25 or a proper-noun dictionary, it markedly improves performance on terminology queries without sacrificing any general capability.

Third, query rewriting and expansion. Have an LLM rewrite a colloquial query into a search expression containing the key terminology. This directly raises the consistency between the query and document distributions, at a cost far below fine-tuning.

Fourth, representation enhancement on the document side. Give each chunk a layer of structured context summary or keywords, and encode them along with the text. This solves the problem of "the term appears in the document but gets diluted," sidestepping fine-tuning entirely.

Only when all four have been done, and evaluation data clearly points to "recall is the bottleneck, and failures concentrate on confusing near-synonymous documents," should fine-tuning enter the options.

7.4 A Minimal Viable Experimental Design for Fine-Tuning

If you do end up fine-tuning, the experimental design itself has to be stricter than you might expect, otherwise it is very easy to reach a conclusion that "looks effective."

First, balance the base and control arms. Run at least four arms: the original general model, the general model plus a projection layer, a full fine-tune on your data, and a control baseline trained on the same amount of data. Missing any one of them means you cannot tell whether the gain comes from domain knowledge or merely from "training one more round."

Second, evaluation must include a domain it has never seen. Hold out an entire business line from the fine-tuning data, exclude it completely from training, and use it only for evaluation. This set of numbers is the most direct evidence of whether capability collapse occurred, and it is the step most often skipped in fine-tuning projects — its result is often unflattering, but it is precisely what decides whether this fine-tune was a good trade.

Third, record the convergence curve over data scale. Evaluating at several data volume levels gives you a curve rather than a single point. A high single-point score may just be luck at that scale; only the curve tells you whether gains are still growing, whether current investment is already near saturation, and whether it is worth accumulating more data.

Fourth, count maintenance cost in the decision. What fine-tuning produces is not just a model file, but a whole set of dependent training data, base version, inference configuration, and regression tests. If the maintenance cost of those assets exceeds the two or three points of gain that fine-tuning delivers, then even a clear offline improvement is not worth it in engineering terms. An evaluation report should contain not only the metric table but also this cost judgment.

8. How Index Parameters Mask Model Quality

8.1 HNSW's M, efConstruction, and efSearch

This is the layer most easily overlooked during evaluation, and also the one with the most severe consequences. With the same embedding model, the gap in Recall between different index configurations can run into the tens of points. If you do not lock down the index configuration while comparing models, what you are most likely measuring is the configuration difference rather than the model difference.

The semantics of the three parameters need to be kept distinct:

• M is the out-degree of each node, and directly determines the index's graph quality and memory footprint. A common starting point is 16 or 32; raising it mainly improves the very-high-recall region (above 95%) and yields limited benefit in the low-to-mid recall range.
• efConstruction is the size of the candidate queue during graph construction. It affects graph structure quality, is an offline parameter, and is decided for life at build time. A common starting point is 200; raising it shows clearly diminishing returns, and index build time rises linearly.
• efSearch is the query-time candidate queue size. It is a runtime parameter that can be tuned dynamically within the latency budget, and it is the most direct handle on the recall-versus-latency trade-off.

The key judgment is this: if Recall@10 is currently around 80%, the bottleneck is usually the model rather than the index, and tuning efSearch will help; if you are already above 95%, the bottleneck is the model and the index is already squeezed dry. Conversely, if the index configuration is loose, you may see no improvement even after switching to a better model — because the ceiling is pinned down by efSearch.

8.2 Quantization and Approximation: The Source of the Recall Ceiling

The memory wall forces essentially every system to introduce compression. Common routes are scalar quantization (int8), product quantization (PQ/OPQ), binarization, and each vector database's own on-disk index format. Their mechanisms differ, but the cost is consistent: quantization error reduces vector discriminability, so a correct answer that ranked 3rd drops to 30th.

There are several points here that must be evaluated separately:

• Whether the pre-quantization distribution is suitable. Vector distributions differ greatly across embedding models — some dimensions are tightly concentrated, others have high variance. A PQ codebook is obtained by clustering on a training set; if a model's vector distribution is not well covered by the training distribution, quantization loss is markedly larger. That is why the same quantization configuration damages different models to different degrees.
• The role of residual correction. OPQ-style approaches rotate before quantizing, which can significantly reduce loss. This rotation usually has to be trained alongside your corpus, belongs to the configuration, and must be evaluated along with it.
• The scenario boundary of binarization. Pure binarization retains very little information and typically causes large recall loss; it is only marginally viable at extreme scale combined with strong reranking — and even then, the post-quantization Recall@k degradation must be reported separately.

8.3 Evaluation Must Lock Down and Report the Index Configuration

Before you start sweeping parameters, there is one easily missed precondition: how much damage a quantization configuration does depends on the vector distribution the model outputs, and that can differ a great deal between models. The same product quantization configuration may be nearly lossless on a model with concentrated distributions and clearly lossy on one with dispersed distributions. So a quantization scheme cannot simply be carried over across models; you must measure each candidate model's degradation separately.

from dataclasses import dataclass
from typing import Iterable, List, Sequence


@dataclass(frozen=True)
class IndexConfig:
    m: int = 16
    ef_construction: int = 200
    ef_search: int = 64
    metric: str = 'cosine'          # 与模型的归一化约定保持一致
    quantize: str = 'none'           # none | scalar | pq | binary
    normalize: bool = True


def sweep_index(corpus, queries, gold: dict,
                configs: Sequence[IndexConfig], k: int = 10) -> List[dict]:
    """固定语料与模型，只扫索引配置，定位“召回天花板”到底在哪里。"""
    rows = []
    for cfg in configs:
        index = build_index(corpus, cfg)
        hits, lat = [], []
        for q in queries:
            res = index.search(q.vector, top_k=k,
                               ef_search=cfg.ef_search)
            hit = 1.0 if (res.ids and gold[q.qid] in res.ids) else 0.0
            hits.append(hit)
            lat.append(res.latency_ms)
        rows.append({
            'm': cfg.m,
            'ef_construction': cfg.ef_construction,
            'ef_search': cfg.ef_search,
            'quantize': cfg.quantize,
            'recall_at_%d' % k: round(sum(hits) / len(hits), 4),
            'p50_ms': round(percentile(lat, 50), 2),
        })
    return rows

The right way to use this table is: first plot recall as a function of efSearch and find where the inflection point sits under the current model configuration. If the curve saturates at a very small efSearch, the index is not the bottleneck. If the curve is still rising steeply, the index configuration is suppressing the model's performance, and any model comparison conclusion at that point is untrustworthy.

Another frequently forgotten configuration is the similarity threshold and the empty-result policy. Many systems return nothing when similarity falls below a threshold; if that threshold is not surfaced during evaluation, the Recall you see has been silently pushed down. So the report must include the threshold distribution and the empty-result rate.

8.4 IVF-Style Indexes and Structured Filtering: Another Masking Effect

Beyond graph indexes, inverted-file-style indexes behave completely differently on recall: they first coarsely select a number of buckets, then compute exactly only within the selected buckets. So their key knob is the probe count, and at small data volumes its effect on recall is often more dramatic than that of a graph index — probing one more bucket multiplies the search scope, and multiplies the cost as well.

This produces a phenomenon that is easy to misjudge: the same model performs stably on a graph index and suddenly drops points on IVF. The correct conclusion at that point is that the index configuration does not match, not that the model is bad. The reverse holds as well — the gap between models can be amplified or shrunk by different index structures, so strictly speaking the question "which model is better" only has a definite answer once it is bound to an index configuration.

Structured filtering is another masking source. Real systems almost always layer on metadata conditions like permissions, department, and time range. When filtering is applied after retrieval, it silently removes some correct documents, which shows up as "recall dropped and nobody can say why." When filtering is pushed down into the index, the algorithm has to keep searching within the filtered candidates, and effective recall varies with how selective the filter is — the stricter the filter, the lower the effective recall.

So the evaluation configuration must explicitly include two things: the distribution and selectivity of the filter conditions, and whether filtering runs before, during, or after retrieval. A recall number measured only under "no filter" conditions carries almost no reference value in a real multi-tenant scenario.

8.5 The Recall / Latency Trade-off Must Be Fixed Within Evaluation

In most vector retrieval interfaces, the relationship between latency and recall is monotonically tunable: give it more latency, you get higher recall. So the form of the question "which model is better" becomes — at the same point on the latency curve, who sits in the better position.

This also means that reporting "recall X, latency Y" for a single configuration in isolation carries almost no information, because a different set of configurations produces an equally legitimate set of numbers. Only two formulations are truly comparable: recall comparison at equal latency, and latency comparison at equal recall. The former corresponds to the constraint "perceived speed must not degrade"; the latter to the constraint "quality must not be traded away."

Which one you pick depends on the business promise. If there is a hard response-time SLA in production, the first is closer to reality; if quality is the floor and latency is negotiable, the second fits better. Either way, you must state clearly in the report which formulation was used and at which latency point the comparison was made — otherwise two teams' numbers cannot be placed in the same conversation.

8.6 Index Construction Is Itself Non-Deterministic

An easily overlooked fact is that even with identical models, parameters, and data, two index builds can differ in query results. The sources include concurrency ordering and randomness during graph construction, implementation differences in the bucket assignment algorithm, and sampling randomness when training the quantization codebook.

This imposes two concrete requirements in practice. First, all comparisons within a single evaluation report must come from the same already-built index. If each candidate model builds its own index before comparison, the randomness of graph construction leaks into the results as noise. Second, if you rebuild the index, you must report the rebuild itself as an experimental variable: build the same configuration twice and look at how much the metrics move — that swing is the noise floor of your system at the lowest level.

An incidental piece of engineering advice: make index construction a reproducible artifact too. Fix the random seed, fix the parallelism, and record the build parameters together with an index version number. When you cannot achieve exact reproducibility, at least record the build time and the concurrency settings of each stage, so that the range of interpretation for latency numbers has a basis. In an evaluation system, half the work of reproducibility actually happens on the index side, not the model side.

Once you know the noise floor, only then does every "X is two points higher than Y" claim even qualify for discussion. That number is often larger than you would expect, especially for large-scale indexes built with parallel graph construction. It is also why "run it once and draw a conclusion" is especially dangerous in evaluation — not because of luck, but because you never measured the system's inherent range of fluctuation at all.

8.7 The Minimum Reporting Checklist for Index Configuration

Condensing the previous sections into a checklist: any evaluation report must contain at least the items below. Missing any one of them means some conclusion cannot be independently reviewed.

Category	Items that must be stated	What happens if you leave them out
Data	Corpus snapshot version, chunking rules and parameters, length truncation policy	Change the chunking once and all historical scores are invalidated
Encoding	Model name and revision, dimension, normalization, precision type, query prefix	You cannot tell whether the model changed or the configuration changed
Index	Index type, M, efConstruction, efSearch or nprobe, quantization method	Conclusions get masked by index parameters
Filtering	Filter fields, selectivity, which stage the filtering executes in	Numbers stop being comparable once permission filtering is added in production
Reranking	Reranker model, candidate count, timeout and truncation settings	Attribution is misplaced and you cannot tell which stage produced the gain
Evaluation	Evaluation set version, slice composition, annotation agreement, confidence intervals	You cannot judge significance from the numbers
Environment	Hardware, concurrency level, warm/cold cache state	Latency numbers are not reproducible

This table can also serve directly as the skeleton of a selection document. Its value lies not in the fields themselves but in the fact that it translates "I think this model is better" into a checklist you can tick off item by item — and any conclusion that cannot be ticked off does not yet belong in the decision.

9. Nine Traps in Evaluation

Trap One: Using the Same Data for Both Tuning and Evaluation

This is the most common and most serious trap. Its typical symptom is that scores rise steadily with every iteration while production has long since stalled or even regressed. The reason is simple — you optimized for performance on that batch of data, not for generalization.

The detection method is direct: if your team knows the correct answer for every evaluation sample, and when discussing why a particular sample was mis-ranked you keep coming back to the same handful of samples, then this data has in fact already become training data. The remedy is to physically separate the tuning set from the evaluation set, and to carry a tuning-round counter in the report; making repeated selections against a fixed evaluation set is itself a form of overfitting.

Trap Two: Judging with Exact Match Only

The rule "the keyword appears in the query and in the document, so call it correct" looks reasonable but systematically favors documents stuffed with keywords. A genuinely discriminative criterion should be: can this document answer this question, not "does it contain those words."

This is more acute in Chinese technical scenarios, because synonymous expressions abound ("won't start" vs "unable to launch," "throws an error" vs "raises an exception"), and exact matching will judge a large number of correct documents as irrelevant. The safest approach is to double-annotate a small batch of samples and compute annotation agreement, using that agreement rate to verify whether your judging rule is trustworthy. If the agreement rate is below some threshold, the judging rule itself needs revising first, rather than rushing to compare models.

Trap Three: Ignoring Long Documents and Code

The mechanism was covered earlier; here I want to emphasize how it shows up in evaluation: recall quality on long chunks is clearly lower than on short chunks, and the magnitude of the degradation varies a lot across models. A model selected on a short-passage evaluation set may degrade badly once deployed into a long-document scenario.

Code is even more special. A general embedding model's understanding of code is close to its natural language description; if your evaluation set contains no real code queries, this defect is completely invisible. The cheapest remedy is to report metrics bucketed by chunk length, and separately construct a set of "error message → code location" queries. These two sets of data will usually expose problems in the direction of the ranking.

Trap Four: Looking Only at Aggregate Metrics

Averages compress away the most important information, especially when sample sizes are extremely unbalanced across slices. Slice-based reporting was mentioned earlier; one addition: when two models have close averages, the correct basis for the decision is to look at the slice where the difference is largest. If A is consistently better than B on your primary scenario slice, then A is the one to pick — even if its average is slightly lower — because your traffic is not uniformly distributed.

Trap Five: Ignoring Length Skew in the Query Distribution

The length distribution of queries in production logs is usually different from the one in your evaluation set. This matters especially where very short queries dominate (say, someone typing two or three words into a search box), because it significantly changes the direction of optimization: if your evaluation set is heavy on long queries, the model you select will lean toward handling complex semantics rather than toward exact lexical matching.

The correct approach is to resample the evaluation set using the real production length distribution, or simply to report stratified by length. Neither method is hard, and both avoid a selection mistake with a considerable price tag.

Trap Six: Ignoring the Mismatch Between Retriever and Generator

This is a pitfall many people only fall into during end-to-end evaluation. Retrieval metrics improve, but answer quality does not change, or even gets worse. There are two common causes: retrieving more relevant documents lengthens the context and introduces noise; or the retrieved document is relevant but the key information sits in the middle, weakened by the "lost in the middle" effect of context.

So every embedding model change should report, alongside the retrieval metrics, an answer quality metric judged against a reference answer — human or automatic — even if it is just a rough 50-item sample. Shipping on retrieval metrics alone is the classic mistake of treating retrieval as the goal rather than as the means.

Trap Seven: Inconsistent Encoding Parameters

Document encoding and query encoding go through different code paths in many frameworks (different prefixes, different normalization, different maximum-length truncation). This class of problem is especially sneaky during evaluation, because it usually affects only production behavior and not offline batch evaluation — offline uses the correct configuration, while some service in production has cached the wrong one and has been using it all along.

The defense is to put the encoding configuration into a version-controlled config file, and to write a separate consistency self-check for the document side and the query side: assert that the normalization switch, prefix, and maximum length match before every release.

Trap Eight: Ignoring Document Freshness and Versioning

In a real system, the "correct answer" corresponding to a query in your evaluation set may have been a different document a few months ago. If the evaluation set does not record document versions, scores drift over time, and part of that drift has nothing to do with model changes. The remedy is to give every annotation record a document version or timestamp and check at evaluation time whether it is still valid; invalidated samples should be explicitly flagged rather than silently participating in the average.

Trap Nine: Treating a Single Run as a Conclusion

Embedding model encoding is deterministic, so "results differ across runs" is usually not a randomness problem but a configuration inconsistency problem. Even so, index construction order, concurrency level, and cache state can all bring noticeable fluctuation. The prudent approach is to record the complete configuration fingerprint on every evaluation (model version, dimension, normalization, index parameters, quantization method, dataset version), and to keep the reports from multiple runs. If two runs reach different conclusions, the difference lies within the noise magnitude, and you should not make any decision at that point.

A complementary recording discipline: write down the noise floor of that run in the report (derived from the index rebuild experiment and the bootstrap resampling), and label every difference smaller than it as "indistinguishable." This rule looks like mere formatting, but its actual effect is strong — it downgrades "this model is a bit better" from a subjective judgment to a proposition with an explicit threshold, and along the way it eliminates a great deal of "it differs by 1.5 points, so let's switch models" back-and-forth.

def config_fingerprint(**fields) -> str:
    import hashlib
    payload = '|'.join('%s=%s' % kv for kv in sorted(fields.items()))
    return hashlib.sha256(payload.encode('utf-8')).hexdigest()[:12]


fp = config_fingerprint(model=name, revision=rev, dim=dim, max_len=max_len,
                       normalize=True, metric=metric, m=m, ef_c=efc,
                       ef_s=efs, quantize='none', dataset=dataset_version)
assert fp_run_a == fp_run_b or noted, 'config changed between runs'  # 两次对比必须来自同一次运行

Trap Addendum: Carrying Over Recommender Experience Wholesale

A frequent source of error is applying the evaluation intuitions of recommender systems to retrieval. The two do share something — both rank candidates — but the key differences are larger: a recommender's candidate pool is highly personalized and grows out of behavior logs, while a retrieval candidate pool is globally shared and comes from a knowledge base.

This produces three concrete biases. First, in recommendation "high exposure" does not mean "relevant," but in retrieval "no interaction" is usually just a neutral signal; so back-deriving negatives from click logs and porting that directly to retrieval introduces systematic noise — which is exactly why hard negative mining needs particular care. Second, recommendation optimizes long-term retention while retrieval needs to be right now; the objective functions are simply different. Third, constraints like diversity and diversification in recommendation usually do not hold in a retrieval scenario, and forcing them on makes offline metrics drift away from the real goal.

What is worth carrying over is the methodology that is independent of any particular business: stratified slices, confidence intervals, paired comparison, versioned regression. These are common properties of evaluation science itself, unrelated to the specific business, and can be borrowed without hesitation.

Trap Addendum: Metric Conventions Quietly Differ Across Tools

The last easily overlooked trap happens at the tooling layer: a metric all called "Recall@10" may not even be defined identically across different vector databases' sample code and different evaluation frameworks — some count after ranking by cosine similarity, some compute once each on normalized and unnormalized vectors and average them, some include results after filtering, some include results before filtering.

Such differences raise no error; they just make two numbers incomparable. The defense is: all metrics must be computed by exactly one reference implementation inside the project, and other tools' outputs are allowed only as intermediate quantities (such as raw ranking lists), never as final metrics. The report should also give several key intermediate quantities from that run, for example the top-10 hit count and the raw distance scores, so that even if the convention needs correcting later, historical data can be recomputed.

10. An Evaluation Pipeline You Can Run as-Is

10.1 The Data Layer

There is more to lock down in the data layer than you might think: a versioned document collection (recording chunking rules, chunking parameters, and the maximum-length truncation policy), a query collection carrying source labels and stratum labels, an annotation guideline plus an annotation agreement report, and a fixed split into a tuning set and a holdout set.

The document collection also needs versioning, because adjusting the chunking strategy invalidates all annotations. A reasonable approach: when chunking rules change, automatically trigger an annotation impact analysis that lists which annotations' answer documents no longer exist, or have been split apart, under the new chunking.

10.2 The Execution Layer

The core of the execution layer is a configuration-driven experiment runner: its input is a list of candidate models × a configuration matrix, and its output is a slice-grouped metric table + a latency/memory table + a configuration fingerprint. The runner must guarantee that within a single run, all configurations share the same document encodings and the same query order — otherwise paired comparison does not hold.

Do not let the experiment matrix explode all at once. A reasonable order is: first run one round of coarse screening with a fixed default index configuration (usually 3~5 candidates is enough) and eliminate the clearly trailing ones; then run a dimension sweep and an efSearch sweep on the survivors to find each one's quality-latency curve; and only then make the final comparison under an equal-latency budget. This order keeps the number of experiments within an acceptable range.

10.3 The Reporting Layer

A qualified evaluation report must contain at least six parts: the configuration fingerprint and dataset version; a slice-grouped metric table (with confidence intervals); significance and win/loss statistics for the paired comparison; a three-way trade-off chart of quality, latency, and memory; a list of key badcases with their attribution; and an explicit statement of which questions this round's conclusions cannot answer.

from collections import defaultdict
from typing import Dict, List


def slice_report(runs: Dict[str, float], meta: Dict[str, dict],
                 k: int = 10) -> List[dict]:
    """按来源 / 难度 / 语言 / chunk 长度分层汇总，禁止只报总体均值。"""
    buckets = defaultdict(list)
    for qid, value in runs.items():
        m = meta[qid]
        buckets[('source', m['source'])].append(value)
        buckets[('difficulty', m['difficulty'])].append(value)
        buckets[('lang', m['lang'])].append(value)
        buckets[('chunk_len', m['chunk_len_bucket'])].append(value)
    rows = []
    for (dim, value), vals in sorted(buckets.items()):
        lo, hi = bootstrap_ci(vals)
        rows.append({
            'dimension': dim,
            'slice': value,
            'n': len(vals),
            'recall_at_%d' % k: round(sum(vals) / len(vals), 4),
            'ci95': [round(lo, 4), round(hi, 4)],
        })
    return rows

The value of this pipeline lies not in the amount of code, but in the fact that it makes every conclusion reproducible, traceable, and regression-testable. The real output of an evaluation system is not that table of scores, but a mechanism that can answer, six months later, whether this change actually made things better.

10.4 Regression Gates: Making Evaluation Part of the Pipeline

An evaluation system starts to prove its value at the moment it begins blocking bad changes from shipping. That requires wiring evaluation into CI or the release process as a hard gate, rather than leaving it as a reference-only report.

Gate design must avoid a common mistake: turning every metric into an auto-blocking threshold. An evaluation set has only a few hundred queries, so metrics naturally fluctuate; overly strict thresholds cause frequent false alarms, and people eventually start habitually ignoring them — which is worse than having no gate at all. The reasonable approach is a two-tier gate: an absolute floor on critical metrics (usually a fairly loose hard threshold), plus a soft alert on relative movement against the baseline (when the drop exceeds a certain magnitude, require human confirmation accompanied by badcases), and then use the paired significance test described earlier to judge whether that alert is worth responding to.

The gate must also carry a data fingerprint. When the corpus version, chunking rules, or the evaluation set itself changes, metrics are not directly comparable with history; at that point the system should automatically skip the comparison or switch to a new baseline, rather than report a false alarm of a "large drop." Otherwise, after the first false alarm, people begin habitually ignoring gate output and the gate stops working.

10.5 An Execution Order You Can Copy

Compressing the above into an executable path goes roughly like this: first fix a system configuration baseline (model + normalization + metric + index + reranking) and record its fingerprint; then run one pipeline self-check on a public benchmark to confirm the implementation is sound; next build a 300~500 item stratified evaluation set, annotate it, and verify consistency; then run the first round of coarse screening with two or three candidate models, reporting by slice rather than only means; then sweep dimension and latency budget for the survivors and plot each one's trade-off curve; make the final comparison under an equal-latency constraint and use the paired test to confirm the difference is significant; and finally lock the configuration, wire this evaluation up as a regression gate, and agree on a usage-count cap for the holdout set.

The two steps in this path most likely to get skipped happen to be the two most valuable: slice-based reporting prevents one wrong selection, and the usage cap on the holdout set prevents the entire system from breaking down within half a year. Neither produces an immediately visible number, yet they determine whether every number that comes after them can be trusted.

10.6 A Recurring Misconception: Treating Evaluation as a One-Off Project

Many teams build their evaluation system during some special optimization effort, finish that round, and leave it sitting there. Three months later, when someone asks "would switching models help?", there is no usable evaluation set, and everyone is back to gut feel. That is precisely the root cause of the traps above recurring — they are not anyone's carelessness, but the inevitable symptom of a missing system.

Keeping it alive requires holding two lines. First, running evaluation must be cheap enough that you can run it as a matter of course after every retrieval-related change. Second, the results must actually be used to make decisions — that is, every selection round produces a written report, including conclusions like "this time we decided not to switch models." Recording the decision not to switch is especially important: it turns evaluation from "an engine for pushing model changes" into "a judgment on whether a change is warranted," and only then is it a trustworthy measurement.

Conclusion: Turn Model Selection into a Falsifiable Experiment

Looking back at the whole article, there is really only one sentence to take away: selecting an embedding model is not a knowledge problem, it is an experimental design problem. General benchmarks can help you narrow the candidate range but cannot replace the decision; leaderboard ranks cannot replace measurement on your own corpus; a single score run cannot replace paired comparison; and an isolated model metric cannot replace a system metric covering "embedding + reranking + index."

If I may leave only three actionable pieces of advice: first, before making any comparison, fix "model + normalization + distance metric + index parameters + quantization + reranking" into a single system configuration, and compare configurations rather than models. Second, report the evaluation set by slice, and rather let a conclusion be as conservative as "cannot be distinguished" than make decisions on noise. Third, let bottleneck diagnosis decide the order of investment — when the Recall curve is still climbing, go fix the index and the reranker; when Recall has saturated but nDCG will not move, go fix the reranker and chunking; and only when both are saturated should you consider changing the embedding model.

One last reminder: the evaluation system itself is an asset that has to be maintained. The corpus changes, queries change, upstream models change, annotations expire. Only by versioning the evaluation set and making regression checks part of CI does it actually protect you — rather than one day turning into a historical document nobody dares touch and that everyone believes selectively.

• Pre-launch checklist: configuration fingerprint recorded; evaluation set reported by slice rather than by mean alone; paired significance test performed; index parameters locked down and swept separately; quantization / compression recall degradation quantified; end-to-end answer quality sample verification passed; encoding configuration consistency self-check passed on both the document side and the query side; invalidated annotation samples flagged and removed.