AI Search Architecture: Engineering Trade-offs in Retrieval, Reranking, and Generation

1. Define What a Search System Actually Is

Retrieval Plus Generation Is Not a Search System

The easiest mistake in engineering is to equate the implementation of a search system with "vector retrieval plus a large model that summarizes the results." That looks perfectly complete at the demo stage: a user asks a question, the system finds a few passages, hands them to a model, and produces a fluent answer. But once it runs at real scale, it collapses in three places at once.

The first collapse point is recall. No matter how strong the generative model is, it can only digest the few hundred items the retrieval layer hands it. If recall keeps the right document out, every downstream optimization is just a model writing a fluent answer from the wrong material — and fluent wrong answers are far harder to debug than an outright error. The second collapse point is latency. Generation is a serial, non-parallelizable stage with a very heavy long tail, and its cost is determined jointly by output length and model size. Once a search system's P95 latency target is compressed into a few hundred milliseconds, any form of online generation blows that budget outright. The third collapse point is trustworthiness. People come to search because they need to confirm a fact; an answer with no citations, no sources, and no timestamps is indistinguishable from fabrication to a user, even when the content happens to be correct.

So the essence of a search system is a retrieval service that delivers information verifiably within a hard latency budget, and generation is only one optional delivery form. Once you see it that way, every architectural decision gets a single yardstick: any design must answer three questions — is recall good enough, is latency good enough, and will the user dare to trust it?

Five Layers and Their Responsibility Boundaries

A complete search system decomposes into five layers. The offline index construction layer turns raw corpora into a retrievable structure. It sits off the user request path and sets the ceiling on "can this be found at all." The online retrieval (recall) layer screens a few hundred candidates out of tens of millions of documents in milliseconds; it sets the ceiling on candidate-set quality. The reranking layer makes finer-grained and far more expensive relevance judgments on those candidates; it sets the precision of the final ordering. The generation layer organizes the ranked material into a natural-language answer; it sets the quality and trustworthiness of how the answer is expressed. The presentation layer decides what the user actually sees — summary cards, the answer body, the citation list, related follow-ups — and it sets whether the information gets consumed effectively.

The most important engineering consensus across these five layers is this: an upper layer can compensate for a lower layer's weakness, but cost and latency then rise non-linearly. When recall is weak, you can compensate with a stronger reranker and a longer context, but that turns a 200 ms request into an 8-second one. Conversely, when recall is good enough, reranking only has to process a very small candidate set, and generation can safely be constrained to a handful of high-confidence passages. The core work of architecture design is compressing each layer's capability requirement down to exactly what is needed, no more.

How to Allocate the Latency Budget

The latency budget has to be carved up before you write any code; otherwise every stage quietly spends a little more and the total runs away from you. Below is how a typical high-traffic search scenario might be budgeted. Generation is the optional stage here: if a precomputed answer is hit or a structured list is returned directly, the entire generation budget can be reassigned to recall and reranking.

Layer	P50 budget	P99 budget	Degradation action when the budget is exceeded
Query understanding	15 ms	60 ms	Skip rewriting and use the original query
Multi-lane recall	40 ms	120 ms	Drop the slowest lane, keep the fallback lane
Reranking	80 ms	200 ms	Degrade to ranking directly by coarse scores
Generation	600 ms	2500 ms	Return the top ten titles with snippets instead
Presentation and network	40 ms	100 ms	Stream the first screen and fill in the rest in the background

The real value of this table is not in the numbers themselves but in the last column. Every stage needs a pre-agreed answer to "what happens when the budget is gone." In practice, a degradation strategy that was never designed up front and improvised during an incident usually degrades into a hard error, whereas a carefully designed fallback chain still returns usable results under overload. Degradation is itself a code path that needs its own testing, and it tends to break more easily than the happy path precisely because it is so rarely exercised.

Also worth noting is the relationship between budget allocation and traffic shape. Search traffic has a pronounced peak-to-trough ratio, and the heavy computational stages — reranking and generation — become bottlenecks first at peak. A common approach is to adjust budgets dynamically based on the traffic water level: at low load, rerank and generate everything; at high load, return recall results first so users can start reading, then complete reranking and generation in the background and stream them back in. The user's perceived latency is anchored to the first screen at the recall stage, while the expensive work is demoted to async tasks nobody waits on.

2. The Offline Index Construction Layer: Forward Index, Inverted Index, and Vectors

What an Inverted Index Actually Looks Like in Production

The inverted index is the oldest and least replaceable component in a search system. Its core structure is a term-to-document-list mapping, paired with a forward index (documents to raw fields) used to fetch body text, titles, and timestamps after a hit. Forward and inverted are a complementary pair: the inverted index narrows the candidate set quickly, the forward index fills in the complete record. This combination exists in virtually every mature search engine, and it has never been replaced by vector retrieval.

There are a few practical realities about the inverted index worth calling out on their own. First, tokenization and normalization. In Chinese settings the choice of segmentation granularity directly determines recall quality: coarse segmentation recalls more broadly but carries more noise, while fine segmentation is more precise but can miss the document entirely when it cuts in the wrong place. The common combination in practice is "coarse cut, fine correction" — store one coarser segmentation on the index side, and do precise matching on key fragments at query time. Second, pruning stopwords and very high-frequency terms. Tokens like "的" (de), "怎么" (zenme), and "是什么" (shenme) appear in a large share of documents, and dropping them shrinks posting list size and query cost significantly. Third, field weighting. A hit in the title, a hit in a tag, and a hit in the body contribute very differently to relevance, so posting lists need to be stored grouped by field, or scored with different saturation coefficients.

Structural Choices for the Vector Index

Dense retrieval maps documents and queries into the same vector space and trades exactness for sub-linear lookup speed through approximate nearest neighbor (ANN) indexes. The choice of index structure directly shapes the recall-versus-latency balance, and that balance looks completely different at ten million documents than at a hundred million.

Below roughly a million documents, graph indexes (proximity-graph based methods, for example) usually beat inverted-style vector indexes on both recall and latency. The price is slow construction, a large memory footprint, and the fact that updating a single document means re-inserting it into the graph. At a hundred million documents, the memory and build costs of graph indexes become untenable, and quantized inverted schemes — quantizing float vectors into low-bit-width integers and then looking them up through an inverted structure — become the realistic choice. Latency is slightly higher, but the memory footprint drops to a fraction of the raw vectors, and both engineering complexity and operational cost drop a lot. The concrete performance gaps depend heavily on dimensionality, quantization bit width, filter conditions, and hardware; there is no universal conclusion, and you have to measure it on your own data and your own hardware.

Index Build Throughput Is a First-Class Concern

Many teams watch only query latency and then get tripped up by index build throughput after launch. The build cost of a vector index scales roughly with document count times vector dimension, and graph index construction adds edge maintenance on top of that. The build stage must support resuming from a checkpoint, otherwise one failure means starting over from scratch; it must support parallelism, otherwise a single machine's throughput becomes the bottleneck for content ingestion.

#增量索引构建的基本骨架（伪代码）
def build_delta(docs, base_index, changed_ids):
    #1.只处理发生变更的文档,先在基线索引里删除旧向量
    for doc_id in changed_ids:
        base_index.delete(doc_id)
    #2.批量计算新向量,批量写入,避免逐条提交
    vectors = embed_in_batches([d for d in docs if d.id in changed_ids], batch_size=256)
    base_index.upsert_bulk(docs, vectors)
    #3.写完后原子切换可见性标记,避免读到半成品
    base_index.commit_generation(new_generation)
    #构建必须可重入:任何一步中断后重跑都能得到一致结果

Three design points in this skeleton are easy to overlook and absolutely critical. First, deletes must precede writes, otherwise stale vectors linger and create a silent data error in incremental update scenarios — one document holding two bodies of content. Second, batch commits: writing records one at a time is typically one to two orders of magnitude slower than batching. Third, the generation switch mechanism — what the query side reads is always one complete generation, so even if the build is interrupted you never get a hole where the index has the original text but no vector. All three can be skipped in a single-machine script; none can be skipped in a continuously running online system.

3. The Online Retrieval Layer: Engineering Tradeoffs in Recall

Sparse Retrieval: BM25 and Its Variants

BM25 is still the strongest sparse baseline in a search system, a fact that has been re-verified in engineering practice over and over. Its core idea is this: higher term frequency contributes more but saturates, inverse document frequency (IDF) measures how discriminative a term is, and document-length normalization suppresses the natural advantage of long documents. BM25 has simple parameters, requires no training, is interpretable, and scores a single document without depending on global state — which makes it nearly irreplaceable in scenarios that need exact matching or retrieval of short strings such as order numbers and model codes.

Variants around BM25 push in three directions. First, weighting on the query side, treating query terms differently according to intent weight and assigning different weights to title terms and modifiers. Second, field-aware scoring on the document side, splitting BM25 into several field-specific versions and combining them linearly or non-linearly. Third, introducing semantic signals — for example, weighting query terms with entity links produced by a knowledge graph or a pretrained language model, so that ambiguous terms such as "苹果" (apple) get a more appropriate IDF estimate from context.

Dense Retrieval: Dual-Encoder Architecture and Training Objectives

The mainstream structure for dense retrieval is the dual encoder: the query and the document each pass through their own encoder into the same space, and vector similarity serves as the relevance criterion. The advantage is that query vectors can be precomputed offline, so serving requires only one approximate nearest neighbor lookup against the document vector store, which keeps latency under control. The disadvantage is just as clear: the query and the document cannot see each other during encoding, a property known as independence of representation. It inherently weakens fine-grained term matching, and it degrades noticeably in scenarios that require precise alignment.

The choice of training objective has a large impact on the final result. The most naive contrastive objective uses only positive and negative pairs and degrades easily; introducing hard negatives — documents that rank high in the retrieval results but are actually irrelevant — is critical for improvement, because they cover exactly the boundary regions the model tends to confuse. In addition, mixing in some lexical term-matching signal from the original text as a multi-task constraint during training can, to some degree, counteract the dulling of exact matching caused by the dual tower. These are consensus directions at the mechanism level; which particular combination pays off more depends on corpus composition and label quality, and it has to be validated in your own scenario.

How Multi-Lane Recall Is Fused

A mature search system never bets on a single recall lane. A typical configuration looks like this: a sparse lane for exact matching, a dense lane for semantic generalization, a rewrite lane for broadening the expression, a graph lane for entities and multi-hop associations, and finally a lane for popularity or collaborative recall based on behavioral logs. Each lane returns its own candidate set and scores, and they then enter the fusion layer.

The most naive fusion approach is score normalization followed by a weighted sum, but it has a hidden hazard: score distributions vary enormously across lanes. Sparse-lane scores typically cluster in one range while vector-lane cosine similarities cluster in another, so naive weighting effectively suppresses one of the lanes. The safer approach is to normalize by quantile or by statistical standardization first, and then weight; the safest approach is to learn a fusion model from a small batch of human-labeled queries and let the model learn the relative weights itself. Another key fusion decision is quota: capping how many candidates each lane may contribute before fusion, so that one lane with strong recall capability cannot squeeze out the others and turn multi-lane recall back into single-lane recall.

#多路召回的融合与配额(伪代码)
def retrieve_fused(query, plan, k=200):
    #plan:每一路的名称、权重、配额、是否启用、截止时间
    pools = []
    for lane in plan.lanes:
        if not lane.enabled or lane.over_budget():
            continue
        cands = lane.search(query, top_k=lane.quota, timeout=lane.timeout)
        pools.append((lane.name, lane.weight, normalize_by_rank(cands)))
    #分位数归一化后加权,避免不同路的分数尺度互相压制
    merged = defaultdict(float)
    for name, weight, cands in pools:
        for rank, (doc_id, _) in enumerate(cands):
            merged[doc_id] += weight * (1.0 / math.log2(rank + 2))
    return sorted(merged.items(), key=lambda x: -x[1])[:k]

The most notable thing in the code above is not the algorithmic detail but the over_budget() check and the timeout parameter. They embody a frequently underestimated architectural principle: online retrieval must be allowed to fail partially. When a vector index occasionally wobbles, or one shard of a sparse index times out, none of that should fail the entire request — it should show up as one lane's contribution going missing, with the remaining lanes and the degradation policy filling the gap. Scoring by reciprocal rank also demonstrates something: at the recall stage, the ordinal information of "rank" is more reliable than an "absolute score," because it is naturally immune to differences in score scale.

Why You Keep One Traditional Retrieval Lane as a Fallback

This lane often gets cut in the new systems many teams build, because semantic retrieval appears to cover most of the requirements. But the job it does is entirely different from semantic retrieval, and there are several classes of task it does structurally worse.

The first class is precise identifier retrieval. A user searches for an order number, a version string, a fragment of an email address, an error code. The semantic vectors of such tokens have almost no discriminative power, because in the training corpus they are not natural language. Here term matching is not merely more accurate but the only viable path. The second class is the long tail of rare entities. Semantic models understand high-frequency concepts well, while their grasp of low-frequency proper nouns depends entirely on whether those nouns appeared in pretraining and fine-tuning data — term matching is completely immune to that uncertainty. The third class is freshness. When a user searches for an event that has just happened, newly written index entries may not have vectors yet, whereas an inverted index is visible the moment it is written. The fourth is interpretability. Highlighting a term hit is the easiest relevance evidence to explain to a user and the most reliable feedback source for a correction system.

So keeping the traditional retrieval lane is cheap, and the return is that it drops the system's overall tail failure rate by an order of magnitude. This is one of the few design points in a search system that I believe will almost never be optimized away.

The Recall Versus Latency Tradeoff

The core tension in the recall stage is how complete you can make the candidate set within a given millisecond budget. The adjustable levers, ordered roughly by impact, are as follows.

First, the parameters of the index structure. The search breadth of approximate nearest neighbor search directly determines the recall-latency exchange rate: widening it by one notch buys a recall gain that is cheap in the low-recall region, while its marginal return decays rapidly in the high-recall region. Second, the per-lane candidate quota. Too small a quota truncates genuinely relevant documents; too large a quota significantly increases fusion and reranking cost. Third, whether to use two-stage recall — shrink the range with an extremely cheap filter first, then perform the precise lookup. Fourth, document length truncation: during vector retrieval, encoding only the first few passages is often enough to judge document-level relevance, whereas encoding the full text multiplies the cost.

What needs stressing is that recall here cannot be judged by "the model feels it is relevant." It must be computed from ground-truth labels on a labeled set. A practical approach is to first compute a theoretical upper bound with a relatively loose candidate set, then trim the configuration item by item and watch for the point where the curve begins to steepen noticeably — that is usually the optimal stopping point within the latency budget.

4. Query Understanding: The Most Underrated Source of Returns

Intent Classification and Query Routing

Query understanding is a layer with a very high return on investment inside a search system. The same few dozen characters of user input can carry completely different intents: looking for a specific page, understanding a concept, resolving an error message, comparing two options, tracking a development. The ideal retrieval strategy even differs across these — navigational intent needs exact matching and a single answer quickly, browsing intent needs diverse coverage, and exploratory intent needs semantic generalization rather than literal hits.

The value of intent classification lies in letting the system choose different processing paths, not merely in attaching a label. When intent is judged navigational, you can skip reranking entirely, take the top-scoring page, and highlight it; when it is judged freshness-sensitive, turn time filtering from a soft constraint into a hard one; when it is judged comparative, split the query into several sub-queries, recall each separately, and merge the results. Getting intent routing right often produces a more visible gain than upgrading the reranking model by one size class.

Query Rewriting and Expansion

Query rewriting covers several directions: correction (fixing the input into the words the user actually meant), normalization (dropping meaningless modifiers and turning colloquial phrasing into terminology), expansion (adding synonyms, hypernyms, hyponyms, and related entities on top of the original query), and splitting and merging (breaking a compound query into sub-queries, or merging several clauses into one more precise retrieval unit).

The biggest risk of rewriting is that it introduces bias. If an expansion term is not strongly tied to the original meaning, the rewritten query drags recall off course — and this kind of error is very hard for a user to recognize as "the retrieval got it wrong." They simply think "this system is really dumb." In practice, therefore, teams lean toward conservative rewriting: expand only when confidence is high enough, keep the original query as an independent recall input, and let the results of the original and rewritten queries converge in the fusion layer instead of having the rewrite replace the original outright. This "parallel rather than replacement" principle is more robust in the majority of scenarios.

Freshness Detection

Freshness detection treats time as a first-class signal rather than simply sorting by time descending. Freshness signals come from three sources: temporal adverbs in the query text ("最新" (latest), "2024 年" (2024), "上周" (last week)), the intrinsic rate of change of the entities a query involves (software versions, stock prices, and weather are inherently time-varying), and the document's own metadata update characteristics. The way these signals are handled is not to bolt on one time-decay factor across the board, but to layer them: queries that explicitly express a temporal constraint get a hard filter or a strong weight; the time-varying nature of entities gets a moderate smoothed decay; stable knowledge such as mathematical theorems or historical facts gets no time penalty at all.

The parameters of the time-decay function are extremely sensitive. Decay that is too aggressive suppresses documents that happen to be important but slightly older; decay that is too slow lets stale but keyword-perfect matches rank ahead. The safer approach is to treat the decay parameter as an experimentable variable and tune it separately for different query types, instead of letting one global parameter rule them all.

Carrying Context Across Turns

The difficulty in a multi-turn scenario is not "can it remember the last utterance," but determining which components of this utterance are inherited and which replace. A user asks about "the battery life of product A," then asks "what about B?" — the referent of the second query is implicit. A user asks about "the battery life of product A," then asks "how long does charging take?" — the subject of the second query has become charging rather than the product. One workable decision path is to examine the semantic completeness of the query itself: anything that parses independently into an explicit query is not inherited, and only queries that require outside information in order to parse trigger inheritance.

Context inheritance can be implemented at two levels, shallow and deep. The shallow level is query concatenation — splicing the previous turn's terms in and running one ordinary retrieval. The deep level is rewriting — having the model generate a complete, self-contained query from the conversation history. The latter is clearly higher quality because it handles coreference correctly, but it brings an engineering constraint: it introduces model call latency, and a rewrite error contaminates retrieval results directly. A reasonable compromise is to use rules first to decide whether inheritance is needed, call the model only when it is, and keep the original input as a fallback recall path for the rewritten result.

Spelling and Synonyms

These two problems look basic, but handling them poorly directly eats a meaningful share of long-tail traffic. Spelling correction has to distinguish between "must correct" and "must not correct": obvious input errors (typos, mixed pinyin and Latin input, mixed Chinese and English input) should be corrected, while "misspellings" of proper nouns, brand names, personal names, and model codes are often correct as typed, and correcting them rashly is worse than leaving them alone. The decision signal usually comes from how frequently that token appears in the corpus and from the user's historical behavior, rather than from a pure edit-distance threshold.

One practical principle for synonyms: synonym expansion can live in the index or in the query, but not both at once, otherwise you introduce double counting and weight imbalance. Index-side expansion is friendlier to long-tail queries, because the cost is paid only once at build time; query-side expansion is friendlier to new and trending terms, because you do not have to wait for a rebuild. The two can be tiered by how often a term updates.

5. The Reranking Layer: Where Precision Meets Cost

What a cross-encoder Actually Solves

The dual-tower structure of the recall stage has one flaw you cannot design around: the query and the document cannot see each other during encoding. The scoring function takes two independent vectors as input, so it cannot model whether a given term matters in this particular context. A cross-encoder instead concatenates the query and the document and feeds both into the model, so every attention layer can directly observe the interaction between them and make fine-grained relevance judgments. The price is that every query-document pair requires one full forward pass: it cannot be precomputed, it cannot be accelerated with approximate nearest neighbor search, and its latency is proportional to the size of the candidate set.

This property determines the positioning of a cross-encoder: it can only be applied to a candidate set that has already been reduced. A coarse stage shrinks a million candidates down to a thousand, reranking shrinks a thousand down to a few dozen, and the multiplicative effect across those stages is a leap in overall recall quality — while also making it the main consumer of the latency budget.

Why You Cannot Rerank Everything

Here is a rough calculation: a medium-sized cross-encoder doing one forward pass on a single query-document pair typically costs milliseconds on ordinary hardware. Reranking a full set of one thousand candidates makes that layer alone take seconds; ten thousand candidates pushes it into the low tens of seconds. Even if GPU parallelism lifts throughput by ten times, the queueing and scheduling overhead of an online service still leaves P99 completely out of control.

A deeper problem is that the marginal return of reranking diminishes non-linearly. What reranking has to solve for the candidate the coarse stage ranked first is a misjudgment; the candidate ranked five hundredth, however relevant it may be, is for the user merely a position difference within one screen. Spreading expensive compute evenly across the tail of the ranking is obvious waste. The sane allocation is to concentrate reranking capacity on the head and allow it to expand on demand — for example, reranking only the top hundred candidates and leaving the rest in coarse order.

Cascade Reranking: Layering and Adaptation

Since a single reranking model cannot cover the full candidate set, the most practical approach is a cascade. The first layer uses a lightweight reranking model over a relatively large candidate set. It can be a distilled language model, a model with layers truncated away, or a purpose-trained relevance classifier, and it needs to produce an ordering reliable enough to build on. The second layer then applies a high-cost strong model to the top few dozen candidates. The third layer steps in under extreme conditions, for example performing fine-grained relevance judgments on the few most relevant candidates or generating span-level matches.

Cascade layer	Candidate scale	Model form	Responsibility
Coarse ranking	10⁵～10⁶	Sparse and vector indexes	High recall, tolerating a large amount of noise
Lightweight reranking	500～2000	Distilled cross-encoder or low-layer truncation	Squeeze out noise and form a trustworthy head
Strong reranking	30～100	Full cross-encoder	Head-level precision ranking, decides the final order
Fine judgment	5～10	Fine-grained matching or span-level decision	Span localization and extraction of relevant passages

The candidate-scale ranges in the table are only order-of-magnitude references; the real boundaries should be determined by curves on a labeled set. Exactly how many candidates each layer takes depends on the actual discriminative power of the model at that layer, on the online latency budget, and on hard business requirements such as "the first few results have to be right."

Above the cascade there are two enhancement techniques worth considering. First, adaptive cascade: use a cheap discriminator to judge how hard the current query is, let simple queries such as navigational ones skip reranking entirely, and send only hard queries through the full chain. This lowers average cost while losing very little quality. Second, distillation alignment: have the lightweight reranking model fit the output distribution of the strong reranking model, so that it approximates the strong model's behavior in a cheaper form, which in turn makes the first layer of the cascade more reliable.

Signals Beyond Relevance Reranking

Relevance reranking is not the only factor determining the final order. The final ranking is usually a multi-objective fusion: textual relevance, content quality (source authority, freshness, structural completeness), user personalization (past interests, the current session), and business objectives (click-through rate prediction, conversion prediction). The weights among these signals have to be determined through online experiments; hardcoding a "relevance first" ordering in practice tends to drown relevance in everything else.

One easy trap is making business signals too strong. Once the click-through model is strong enough, it learns to put whatever users are most likely to click at the very top, and that is systematically biased relative to "what the user actually needs" — clickbait-style entries always win on CTR. The mitigation is to cap business signals and fold negative feedback such as "did the user keep consuming after the click" into the model, rather than optimizing the click alone.

6. Generation: Hallucination and Citation Traceability

Where Hallucination Comes From

In a search setting, generative model hallucination has several distinct sources, and discussing them together sends optimization in the wrong direction.

The first class is retrieval-starved hallucination: retrieval supplied no relevant information, so the model had to answer from parametric memory, and parametric memory is unreliable on the long tail and on anything time-sensitive. The root cause here lies in recall, and the correct move is to repair recall rather than to rewrite the prompt. The second class is constraint-failure hallucination: retrieval supplied the right material, but the model ignored it while generating. This usually stems from context being too long, which dilutes the middle portion of the input — the so-called lost-in-the-middle effect — or from instructions that never explicitly said "answer only from the provided material." The third class is synthesis hallucination: the material itself is fragmentary, inconsistent, or ambiguous, and the model makes its own tradeoffs and fills gaps while synthesizing.

The three classes call for completely different handling. The most valuable principle from an engineering standpoint is this: make "no answer" a legitimate answer state. When retrieval quality falls below a threshold, the system should explicitly return "we could not find enough relevant content" together with a suggested direction for rewriting, rather than letting the model answer regardless. A search system that dares to say "I don't know" is far more useful than one that never refuses but frequently fabricates.

How to Make Citations More Than Decoration

In practice, citations are frequently reduced to pure decoration: the model is told to "cite a number," the model emits a number, and the document behind that number has nothing to do with the sentence. The user clicks through, finds the mismatch, and never trusts a citation again. This is the classic failure path of a citation system, and a worse outcome than having no citations at all.

To make citations trustworthy, the key is constraining generation to a verifiable granularity. A few approaches work. One is sentence- or paragraph-level citation, where every output sentence must be bound to a specific input passage and the binding is enforced during generation rather than inferred afterward. Another is structured fact templates, where the answer is rendered as a conclusion sentence plus an evidence passage, and the model is responsible only for extraction and paraphrase within the given passages, not for free-form organization. A third is post-generation verification, where each citation goes through an independent entailment check to decide whether the sentence really can be derived from the cited material; only passing citations are displayed, and failing ones are degraded to unreferenced output.

#引用对齐与校验的流程(伪代码)
def generate_with_citations(query, materials, k=5):
    context = build_context(materials[:k])
    sentences = llm_generate(query, context, require_span_refs=True)
    verified = []
    for sent in sentences:
        ref_ids = sent.span_refs
        #1.引用必须指向真实存在的片段
        ref_ids = [r for r in ref_ids if r in context.span_index]
        #2.句子必须真的能从该片段推出,而不是编号存在即可
        if not ref_ids:
            verified.append(sent.without_refs())
            continue
        ok = entailment_check(sent.text, context.spans[ref_ids[0]].text)
        if ok:
            verified.append(sent.with_refs(ref_ids))
        else:
            #校验不过不静默丢弃,降级为无引用并保留内部告警
            verified.append(sent.without_refs())
            metrics.inc("citation_rejected")
    return verified

Several details in this flow deserve emphasis. Citation numbers must be checked for existence, because a model will very naturally produce a number that does not exist, and in long contexts this is nearly unavoidable. Verification must rest on entailment rather than on the validity of the number, because entailment is what users actually care about. If a failed verification is handled by silently dropping the sentence, the system quietly loses correct content; the correct approach is to keep the text but degrade it to unreferenced and record a metric at the same time. That way no information is lost while the health of the citation system remains observable.

From a product perspective, citations must also be clickable, locatable, and highlightable. Showing a link that jumps to the top of an entire document has very little value; users need to land directly on the paragraph that supports the sentence. This capability looks like a front-end concern, but it actually depends on whether the index layer preserves stable span identifiers and span offsets. Span-level segmentation therefore has to be done offline and cannot be improvised at query time.

Latency and Cost of Generation

The cost of generation is strongly correlated with output length, while long outputs are precisely what is needed to express complex answers accurately. There is a structural contradiction here: the more a question requires generation, the longer the answer, the higher the latency, and the more likely a degradation is triggered. There are three main mitigation directions. First, streaming output, which pushes time-to-first-byte as low as possible so the user sees the skeleton first. Second, on-demand expansion: give the conclusion sentence and a bullet list first, and defer the evidence details until the user expands or asks a follow-up. Third, routing: use a short-output policy for simple questions and reserve the long-output path for genuinely complex ones.

On cost, generation is the one component in a large search system that can exceed the total cost of retrieval. Common optimizations are tiered model routing (small models handle the majority of simple queries), context trimming (feeding in only genuinely relevant passages instead of always padding to a fixed length), and answer caching for high-frequency queries. The returns from these are substantial, but they introduce consistency problems: a cached answer may no longer match the latest data, so time-sensitive answers must carry a short TTL, and the expiry policy has to be aligned with index freshness.

7. Index Freshness and Consistency

Getting Incremental Indexing Right

The core challenge of incremental indexing is not "being able to write it in" but "not producing inconsistencies once it is written in." There are three main classes of risk.

First, partial writes. If a document's vector has already been written but its forward fields have not been updated yet, a query may retrieve incomplete data. The remedy is generational switching or a double-write buffer: complete the write of all fields in an invisible staging area first, then switch atomically to the visible state once it is done. Second, unsynchronized deletion. Source data is deleted but the index is not, which lets content that no longer exist still be recalled — a serious problem in permission and compliance scenarios. Delete events must enter the same processing pipeline as write events, and deletes must be retryable and verifiable. Third, multiple versions of some documents. If an old version is retrieved while a document is being updated but the user expects the new version, you get intermittent bugs that are hard to reproduce, usually resolved by validating the document version number at read time.

The Real Cost of a Full Rebuild

A full rebuild looks like the silver bullet that solves every inconsistency problem, yet its cost is routinely underestimated. It includes reading all source data, re-tokenizing and rebuilding the postings (sparse), re-encoding and rebuilding the graph (dense), writing to storage, and holding two copies of storage during the switchover. For sparse indexes a full rebuild may take only minutes; for vector indexes, rebuild time is proportional to data volume times dimension, and building the graph index is serial and hard to parallelize. The more troublesome part is the service impact at the moment of cutover — either old and new indexes coexist and peak storage doubles, or the cutover introduces a brief window of write blocking.

The pragmatic approach is a hybrid strategy: sparse indexes use a full rebuild plus incremental correction (because it is cheap, and inverted indexes already handle incremental writes well), while vector indexes are incremental-first with periodic full rebuilds (to eliminate the fragmentation and quality drift that accumulate), and full rebuilds are scheduled during traffic troughs. In practice this combination is more pragmatic than chasing "an index that is absolutely consistent at all times," because the latter is economically infeasible.

How to Set the Consistency Window

The consistency window is the time from a source data change to that change being visible to retrieval. This metric should be tiered by business scenario rather than held to one site-wide standard. Internal data such as support tickets and order status needs to be visible within minutes or even seconds; a public content library may tolerate hours; archived data and static knowledge bases can live with days. Once the tiers are set, splitting different data domains into separate index instances with their own update frequency and retention policy is far simpler and far cheaper than forcing one globally consistent index.

One special case also needs handling: a user's own content. Someone who searches right after publishing often cannot find what they just posted, which is an extremely jarring experience — and it is easy to fix with a bypass that "lets freshly published content through directly by user identity," without waiting for the full indexing pipeline to finish. Such a bypass must also be paired with permission filtering, or it becomes a backdoor for unauthorized access.

8. Non-Functional Requirements: Multi-Tenancy, Permissions, and Cost

Three Granularities of Multi-Tenant Isolation

Multi-tenancy has three isolation granularities in a search system, and each carries a completely different implementation cost.

The coarsest is index-level isolation, one independent index instance per tenant. Isolation is the most thorough because data is not shared physically at all, the implementation is the simplest, and permission filtering needs no extra mechanism. The cost is low resource utilization — a small tenant's index will never reach ideal economies of scale, and once the tenant count reaches millions, the overhead of managing instances becomes unacceptable.

In the middle is collection-level isolation, where multiple tenants share one index but each tenant's data carries a tenant identifier field that is used as a filter condition at query time. This is the most common approach, because resource utilization is high and the implementation is not complex. The cost is that filtering must be pushed down to the index layer to guarantee that nothing leaks; if the filter is applied after full scoring, it leaks the existence of information that should have stayed hidden.

The finest is document-level or field-level isolation, which shares the index structure but requires per-result permission checks before results are returned. This is unavoidable when permission rules are complex (role-based, group-based, row- and column-level rules), but it pushes the performance problem from the index layer up into the application layer, where it easily becomes a production bottleneck.

The selection criteria in practice are clear: few tenants with highly uneven usage calls for index-level isolation; many tenants with even usage calls for collection-level isolation; and when the permission rules themselves are complex (the norm in enterprise settings), add a layer of precise verification on top of collection-level isolation, mixing approaches by department or business line.

Permission Filtering Must Be Pushed Down to the Retrieval Layer

This is the place in a search system where security incidents are most likely to happen. The common but wrong approach is to skip filtering during retrieval, pull the results first, and then use permission logic to strip out whatever is not visible. This has three problems. First, it leaks the existence of information: a user can infer from changes in response time or result counts that hidden content exists. Second, it wastes retrieval compute — returning ten visible results may require pulling back hundreds of candidates and checking each one. Third, in approximate nearest neighbor scenarios it may be outright infeasible: if filtering happens after the vector lookup, a user may recall only a handful of results and never fill the quota.

The correct approach is to push the permission condition down as a filter rather than a post-condition on vector retrieval. Approximate nearest neighbor indexes natively support filtered queries, and there are three implementations: filter-then-search, search-then-filter, and a hybrid of the two. The third is best: run one lookup with the filter applied, and if too few candidates remain after filtering, run a wider lookup and filter again, widening iteratively until the quota is met. The sparse lane likewise supports short-circuiting on the filter field during inverted-list reads, which is far more efficient than retrieving everything and filtering afterward.

Two more points about permission filtering are easy to overlook. First, filter conditions must be trusted at the request level rather than taken directly from user input — this is a textbook injection surface. Second, there is a delay between a permission change and an index update, so a document whose permissions were just revoked still carries its old identifier in the index. The mitigation is to maintain a watermark table of "permission version numbers," carry the user's permission version with each request, and route requests with an older version through a verification-and-upgrade path.

Storage and Cost at Large Index Scale

Storage cost is usually not a problem below ten million documents, but at the hundred-million scale it becomes the primary tension. What needs to be worked out are a few products: document count times vector dimension times bytes per dimension gives the raw vector volume; the size of the inverted index depends on the average token count of the corpus and the document count; the size of forward storage depends on the total field volume. Forward storage is often the largest of the three, because it holds the original text and usually has several redundant copies.

Common ways to cut storage cost, ranked by cost-effectiveness: vector quantization (compressing floats into low-precision integers, the most direct lever), hot-warm tiering (migrating infrequently accessed data to cheaper high-capacity storage), loading forward fields on demand (storing only the frequently accessed fields and putting low-frequency fields in object storage), and separating document body text from the metadata search needs (search only needs the index and a summary, while the body is read from primary storage on demand). Combined, these levers usually make a noticeable difference, because the vector portion often accounts for less than half of total storage — the real space hogs are forward storage and the original text.

Another frequently underestimated cost is caching. Query-level result caches and document-level metadata caches both consume memory, and hit rates for both are typically high in a search system. When designing caches you have to be clear about invalidation triggers: result cache invalidation is driven by index generation changes, and document cache invalidation is driven by document updates. Missing an explicit invalidation trigger means users see stale content, and this kind of bug almost never shows up in a test environment.

Reliability and Failure Domains

A search system is a textbook case of "partial failure, total unavailability": a recall lane timing out looks like nothing more than degraded quality, but if the degradation logic is not done properly it becomes a blanket 500. The key to reliability design is therefore giving every recall lane its own timeout and circuit breaker, so that a single-lane failure affects only that lane's contribution. The circuit breaker trigger should look at the latency distribution, not just the error rate — backlogs caused by long-tail latency eventually avalanche, and at that moment the error rate may still look normal.

9. The Evaluation System: The Disconnect Between Offline and Online Metrics

What Each Offline Metric Actually Measures

Recall@k measures whether the correct document made it into the candidate set; it reflects the completeness of the recall layer and is unaffected by improvements to reranking or generation. Mean Reciprocal Rank (MRR) measures how high up the first correct result appears; it focuses more on the head than recall does, but says nothing at all about the distribution from second place onward. Normalized Discounted Cumulative Gain (nDCG) applies position weighting while also accounting for graded relevance across multiple relevant results, making it the most comprehensive offline metric for ranking quality.

These three metrics answer different questions, so using them for decisions yields different conclusions. For example, when you improve generation quality, all three offline metrics stay flat while you may still see significant online gains. The reverse is also true: a reranking model that gains two or three points offline may change nothing online. The reason is that all of these metrics assume "the user reads carefully and judges by relevance," whereas real user behavior is clicking, scanning, and abandoning — each of which reads the semantics of ranking completely differently.

How the Disconnect Shows Up

The disconnect between offline and online usually appears in a few forms. The first is annotation distribution bias: queries in the labeled set are all carefully constructed, while the long tail in real online traffic is far heavier, so the gains from offline optimization concentrate on a subset of users that rarely touch it. The second is click bias: users only click what ranks near the top, which gives training signals built from click logs a built-in position bias, and the click-through rate model then inherits and amplifies that bias. The third is "the correct answer is not in the candidate set": a user's real intent is compositional and no single correct document can answer it, but the annotation labels only one document, so online users remain unsatisfied anyway. The fourth is latency being ignored entirely: no offline evaluation counts latency, so a scheme that pushes latency from 200 milliseconds to 2 seconds still "wins" on the offline metrics.

How to Design an A/B Test

There are several specificities about A/B tests for search systems that deserve particular attention.

Traffic must be split by user, not by query. Search has a strong repeated-query phenomenon, so splitting by query causes the same user to see two versions, which creates confusion and makes results hard to attribute.

The experiment period must cover a complete usage cycle. Behavior patterns within a week differ sharply between weekdays and weekends, so a two-day experiment easily produces conclusions that contradict the long term. Holidays and major event contamination windows must also be avoided.

Metrics must be composite. A reasonable combination: success rate (the user does not immediately bounce back after the first result) as the guardrail metric, effective click-through rate or post-click dwell time as the primary metric, and sampled human satisfaction as the calibration metric. When a guardrail metric misses its bar, the change does not get ramped — and this rule has to be fixed before the experiment starts, to avoid selectively interpreting results afterward.

Sample size must be estimated up front. Take click-through rate as an example: the baseline value determines the sample size needed to detect a small lift, and the higher the baseline the more sample is required. In high-click-through search scenarios, a difference of a few percentage points often needs a considerable amount of traffic to reach statistical significance, which means many seemingly small changes cannot be validated in the short term. This is itself an important architectural constraint: changes that can only be confirmed through long-running experiments do not belong on the critical path.

The Guardrail Metrics You Should Not Skip

Beyond the primary and guardrail metrics, an A/B test should also keep a fixed set of "safety" metrics under monitoring. These do not need to improve, but the moment they degrade it means something is wrong with the system: P95 and P99 latency, retrieval failure rate (including timeouts, circuit breaker trips, and the proportion of degradation triggers), the no-result rate, and the citation verification rejection rate. These metrics form the safety net for architectural change and are more effective than any post-mortem.

10. Several Directions of Progress in 2026

Fusion in Hybrid Retrieval Keeps Moving Forward

The fusion of sparse and dense retrieval started as result-level score weighting and has now advanced to several finer granularities. A few directions are particularly interesting. First, term-level hybrid: instead of weighting two independent rankings, explicitly fuse term-matching signals and vector similarity as two features inside the same scoring function, letting the reranking model learn the relative weight between them by itself. Second, replacing sparse retrieval's inverted index with a learnable sparse representation, so that sparsity no longer comes from tokenization rules but from training — preserving exact matching while sidestepping tokenization errors. Third, end-to-end training that optimizes sparse, dense, and reranking within a joint objective so that each layer's output distribution is separable from the others. The size of the payoff from these three paths depends heavily on corpus characteristics: the more irregular the vocabulary, the more obvious the gains from term-level hybrid and learnable sparsity tend to be.

The Real Gains and the Bill for Agentic Search

Letting a model autonomously decide how many rounds of retrieval to run and what to retrieve is the fastest-moving direction right now. Its appeal is direct: single-round retrieval can only match explicitly stated semantics, while complex questions require decomposition — look up the entity first, then the relationships, then counterexamples, and sometimes switch to a different retrieval approach after a failure. This is the key capability that upgrades "search" into "research."

But the cost has to be accounted for carefully, and it comes in two parts.

The first is latency. Multi-round retrieval is strictly serial, with each round containing at least one model inference plus one retrieval, so more rounds means latency growing roughly linearly. On a search entry point targeting hundreds of milliseconds, two or three rounds are already over budget. This directly determines that Agentic Search is more suitable as a "deep mode" than as the default — a mode the user actively switches into, where waiting tens of seconds is acceptable.

The second is cost and lack of control. The number of rounds is decided by the model itself, so there is no guaranteed upper bound on cost, and a pathological query may trigger a large number of useless retrievals. Moreover, every round is a decision opportunity, decision quality determines the final outcome, and wrong decisions cascade. The necessary engineering constraints are: a hard cap on rounds, a retrieval quota per round, and an explicit "do we have enough information right now" judgment between rounds to avoid meaningless continuation. A fallback also has to be prepared — when the round budget runs out, generate an answer from the material at hand with an explicit statement of uncertainty, rather than pretending the information is sufficient.

Mode	Latency magnitude	Cost magnitude	Applicable to
Single-round hybrid retrieval	Hundreds of milliseconds	Low	Factual queries, navigation, exact match
Query rewrite plus single round	Sub-second	Medium	Colloquial input, aliases, multi-turn references
Generative summary	Seconds	Medium-high	Survey-style queries, judgments requiring synthesis
Agentic multi-round	Tens of seconds or more	High and uncapped	Compositional questions, comparisons or multi-party verification

The key information in this table is not in the specific numbers but in the fact that latency and cost span orders of magnitude across these four modes, so they have to be different product entry points rather than different branches inside one interface. The vast majority of traffic should go through the first two tiers, reserving the latter two for the users who genuinely need them — which is also the most effective way to control overall cost.

Semantic Cache and Reuse Across Similar Queries

A semantic cache uses vector similarity to decide whether a new query is semantically equivalent to a historical one, and reuses the result accordingly. Its gains are substantial in scenarios with high repetition rates (customer support, enterprise internal knowledge bases, hot events), because in these scenarios a large number of queries are semantically synonymous. But its risks are equally clear:

First, staleness risk. Once an answer about some event is cached and the event develops further, a cache hit serves outdated content. Caches must be bound to an expiry time and an index generation, and invalidated proactively when the index updates. Second, permission risk. Different users' queries may be semantically equivalent while their permissions differ, so reusing results directly causes unauthorized access. The cache key must include the user's permission context, or simply cache only insensitive public queries. Third, error propagation risk. Once a wrong answer is cached it keeps hitting, and correcting it is expensive, so only results that pass a high-confidence judgment should enter the cache.

In practice the safer approach is to place the semantic cache at the layer of result titles rather than answer bodies — reusing the top result list for identical queries is much safer, while generated answers are recomputed each time, which greatly narrows the exposure to staleness and error propagation.

The Boundary Between Search and Agents

As agents become the new entry point, the question "what form should search take" gets raised again and again. One usable criterion is to look at what the interface promises.

If the interface promises "give one authoritative answer that can be traced to sources," it is a search system, and it must have strict recall, a latency budget, citation verification, and an evaluation system. If the interface promises "complete a task requiring multiple steps of action and judgment," it is an agent system, and its output is a side effect — possibly a series of actions, a decision, a deliverable.

The two have different engineering centers of gravity: the core tension in a search system is latency versus quality, while the core tension in an agent system is autonomy versus reliability. The more reasonable architecture is therefore two systems coexisting and calling each other — the search system is used by the agent as a tool, exposing a structured, traceable result interface with latency guarantees; the agent handles decomposition and decisions, while search handles retrieval and verification. Letting the search system take on multi-round autonomous decisions leaves it with neither the flexibility of an agent nor the reliability of search; letting the agent access raw data sources directly destroys evaluability. This boundary is, in my view, an engineering consensus that still holds in 2026.

11. A Workable Reference Implementation

The code below strings the layers discussed above together into a complete request flow. It is not runnable production code but a skeleton that makes the key decision points explicit: the timeout, degradation, parallelism, and confidence threshold of each layer are all written into the code, because these are exactly the places where architectures actually diverge when they are built.

def search_pipeline(req):
    #1.查询理解层:意图、时效、改写,各自独立超时,失败可跳过
    intent = classify_intent(req.q, timeout=15ms)
    fresh = detect_freshness(req.q, intent)
    variants = [req.q]
    if intent.rewrite_needed:
        rw = llm_rewrite(req.q, history=req.history, timeout=60ms)
        #原查询与改写并行召回,而不是替换
        variants.extend(rw.queries)

    #2.召回层:各路并行,按计划分配配额与超时
    plan = build_plan(intent)  #每路的开关、配额、权重来自配置
    pools = parallel_collect([
        sparse_search(req.q, plan.sparse, filter=acl_filter(req)),
        vector_search(req.q, plan.vector, filter=acl_filter(req)),
        sparse_search(variants[1:], plan.expansion, filter=acl_filter(req)),
        hot_search(req, plan.behavior, filter=acl_filter(req)),
    ], timeout=120ms)

    #3.融合层:按配额截断后做分位数归一化加权
    candidates = fuse(pools, quota=plan.quota, target=1000)

    #4.置信度闸门:这一层决定生成是否会引入幻觉
    quality = judge_quality(candidates, intent)
    if quality.score < THRESHOLD[intent.type]:
        #明确返回无解,而不是让模型硬答
        return no_answer(retry_hint=quality.hint, candidates=candidates[:5])

    #5.重排层:轻量层处理千级,强模型处理头部,自适应跳过简单查询
    if intent.simple:
        ranked = candidates[:10]
    else:
        coarse = light_rerank(req.q, candidates[:800], timeout=200ms)
        ranked = strong_rerank(req.q, coarse[:80], timeout=180ms)
        ranked = diversify(ranked)  #去掉同源近重复

    #6.呈现:多档延迟预算,按预算选交付形态
    if budget.remaining() < GENERATION_BUDGET:
        return render_list(ranked[:10])
    if budget.remaining() < GENERATION_BUDGET * 0.5:
        return render_summary_with_stub(ranked)  #首屏流式
    return render_full_answer(req, ranked, budget=budget)

There are five positions in this skeleton worth calling out on their own; they are where every conclusion in this article lands.

First, the rewritten variants in step 1 run in parallel with the original query rather than replacing it, because rewriting introduces bias and keeping the original path is the only fallback. Second, every recall lane in step 2 carries the same permission filter, pushed down to the index layer to avoid the information leak and candidate starvation that post-filtering causes. Third, the confidence gate in step 4 turns both "retrieval found nothing" and "retrieval found something but it isn't good enough" into legitimate no-answer responses, which makes it the single most effective design point for suppressing generation hallucination. Fourth, the adaptive skip in step 5 means simple queries consume no reranking budget, which directly determines the average cost of cascaded reranking. Fifth, step 6 chooses the delivery form based on the remaining budget instead of producing a full answer and then truncating it on timeout.

The Order to Build Things In

If you are building from scratch, the experiential priority order is this: first build hybrid retrieval in the recall layer (one sparse lane plus one dense lane, plus a traditional fuzzy-matching lane as a fallback), establishing accuracy on an acceptable baseline; then add lightweight reranking aligned with a strong model via distillation, which usually yields the largest gains of the whole project; then layer on query understanding (intent and freshness detection first, rewriting can come later); then build the evaluation system — this step must come before any fine-grained tuning, otherwise you cannot tell whether a change is good or bad; next come citation verification and generation, making sure the confidence gate sits before generation; and only then optimizations like Agentic deep mode and semantic caching.

The logic behind this order is that the early steps have deterministic gains, low risk, and offline metrics that can validate them; the later steps have gains that depend on the specific scenario and require online experiments, so investing in them too early easily produces something that cannot be evaluated.

12. Conclusion

Architectural decisions for an AI search system are, in substance, about finding a balance point among four mutually conflicting demands: completeness of recall, the latency budget, trustworthiness of generation, and control of cost. These four cannot all be optimal at once, so the core work of architecture design is really to make explicit "in which scenario do we sacrifice which one."

A few judgments running through the whole article can be summarized as follows: every layer needs a degradation path, because behavior under overload says more about architectural quality than peak performance does; keeping a traditional retrieval lane during the recall stage costs very little and pays off enormously, so it should never be optimized away; permission filtering must be pushed down to the retrieval layer, because this is a security issue, not an optimization issue; a citation only counts once it has passed independent verification — a number existing does not mean the citation holds; the evaluation system must precede fine-grained tuning, otherwise all optimization is flying blind; and the latency budget must be carved up before the first line of code is written, with every stage deciding in advance what happens when it goes over budget.

One last point deserves emphasis: the quality gap between search systems rarely comes from how advanced the algorithms themselves are, and much more often from how complete the engineering details are — the consistency of index writes, the implementation of degradation paths, the quality of the evaluation set, the sufficiency of experiment periods. None of this work has a shortcut, and none of it can be substituted by swapping in a stronger model. This is also why the same technical approach yields wildly different real-world results in different teams.

13. Several Engineering Details That Are Easily Underestimated

How Long Documents Are Handled Determines the System's Ceiling

The impact surface of chunking far exceeds intuition. It simultaneously determines index size, recall granularity, the completeness of evidence for generation, and the locatability of citations, making it one of the few decision points where "change it once, and the payoff shows up in every layer." Common approaches are fixed-length chunking, semantic-boundary chunking, and structural chunking (heading hierarchy, sections, list items).

Fixed-length chunking is the simplest to implement, but it cuts a complete argument in the middle, leaving both chunks without context, so neither the reranking model nor the generation model can judge their relevance. Chunking along semantic boundaries (paragraphs, headings) is the more reasonable choice, because a paragraph is already the smallest complete unit of expression the author produced. The cost is uneven chunk length: long paragraphs may exceed the effective range of the vector encoding and need another layer of oversized-chunk splitting.

What is more worth investing in is letting chunks carry structural information: the owning document ID, the heading path (the chain of h2/h3 levels), the ordinal position within the document, and references to neighboring chunks. This metadata has three uses: reranking features can perceive where a chunk sits in the document (a chunk near the beginning of a document usually represents the topic better than one near the end), generation citations can display a full path ("Document title › Level-2 heading › Level-3 heading" is far more useful than an isolated sentence), and result deduplication can operate by document rather than by chunk.

Chunk overlap is a question that has to be decided by content type. Overlap can be small or even zero for technical documentation and question-answer pairs, because each passage is self-contained on its own; narrative text and long arguments need some overlap to preserve information that spans paragraphs. A rising overlap ratio directly inflates both index size and the rate of duplicate candidates at recall time, so it should be configured per document type rather than as one global value.

Deduplication and Diversity

Vector retrieval has a well-known property: high-scoring results often come from several adjacent chunks of the same document. That is a good thing during generation (the material is coherent) and a bad thing in result presentation (the user sees ten nearly identical results). Therefore there must be a deduplication step before the presentation layer.

The strength of deduplication has to vary by scenario. If the user wants a survey, keeping multiple chunks from the same source gives generation more complete material; if the user wants to find one document, deduplicating by document and showing only the most relevant chunk per document is the correct approach. In implementation, doing both layers of deduplication at once is safer: first a hard deduplication by document ID to cap the share of any single document, then semantic near-duplicate detection over the remaining candidates (if the similarity between two chunks exceeds a threshold, keep only one). The latter requires an extra similarity computation, an acceptable cost because the candidate count is already very small at that point.

Designing the No-Result Case Matters More Than the Results Case

The no-result scenario is the most underestimated part of a search system. It does not occur very often, but it directly determines how much the user trusts the whole system — users do not think "nothing found" is normal; they think the system is broken or the content was not indexed.

Sensible handling has three layers. The first is expanded recall: when every current lane returns empty, relax the constraints and retry, including dropping overly strict time filters, lowering the similarity threshold for vector search, removing low-weight modifiers from the query, and turning off strong filter conditions identified by intent recognition. The second is escape rewriting: when there is still no answer, decompose the query into simpler parts and search them separately, or use vector retrieval to find the semantically closest available content. The third is explicit notification, but it has to give specific, actionable suggestions rather than a bare "not found" — suggested alternative terms, conditions that could be relaxed, or related popular content.

The no-result rate itself should be a continuously monitored metric. A sudden rise in the rate usually means something is wrong on the index side (a failed build, a stuck generation switchover) rather than that user queries changed, so it also serves as a good health check signal.

Gray Release and Index Compatibility

An index structure change (a new tokenizer, a new vector model, a new chunking strategy) means the old and new indexes cannot coexist, while the rebuild window can never be zero. The engineering approach that works is dual-write plus dual-read: create a parallel index instance, write content to both, distribute read traffic between old and new in proportion, compare the two result sets batch by batch, and once the difference stabilizes, cut over to the new index and discard the old one.

A key constraint in this process is that the index generation identifier must run all the way through into the results, because reranking and generation evaluations have to be aggregated by index version; otherwise the quality of the old and new indexes gets mixed together and no trustworthy conclusion can be drawn. Another constraint comes from the query understanding layer's assumptions about index format — for example, tokenization on the query side must match the index side, and when the tokenizer changes, that assumption breaks in many scattered places. Converging all tokenization calls into a single module is therefore about making this class of change touch only one place.

Observability of Cost

The cost distribution of a search system is often the opposite of intuition: the queries with the highest share of traffic may consume only a small share of cost, while a few complex queries consume most of it. To make cost manageable, cost has to be broken down and recorded by query dimension, covering at least: the share of requests per intent, the candidate count of each recall lane, the number of candidates reranking actually processed, the number of input and output tokens generated, and the proportion of cache hits.

Only with this data can you make grounded decisions such as "switch a certain intent to a cheaper retrieval mode." Without it, cost optimization can only rely on global parameter tuning, which typically harms high-value users while delivering no obvious benefit.

14. The Evidence Chain from Retrieval to Answer

The Evidence Chain Matters More Than Citations

Citations solve the question of "can a given sentence in the answer be clicked open and verified," but what users actually need to trust is an entire complete evidence chain: starting from the user's question, through how query understanding interpreted it, what material retrieval found, how that material was filtered and ranked, and where each sentence of the final answer came from. If any link in this chain is hidden, users can only see the conclusion and have no way to judge its reliability.

Making this chain explicit has one concrete implementation value — it makes errors locatable. When a user reports a wrong answer, the system can answer "this retrieval understood the query as X, recalled Y candidates, the correct document sat at position Z among them, and it was excluded for reason W." Without this chain, troubleshooting depends on manual reproduction, and since the query distribution of a search system is extremely long-tailed, reproduction rates are very low. Building troubleshooting capability into the architecture is far cheaper than retroactively adding logs.

The specific record items that need to be in place include: the original query, the intent classification result, freshness parameters, the recall count and quota of each lane, the top rankings after fusion, the change in reranking score for each candidate (a candidate whose score was sharply lowered during reranking is often a clue to the problem), the final list of chunks selected into context, and the citation binding and verification result for each answer sentence. These records can be sampled on only a portion of requests to control storage cost, but no-result and low-confidence requests should be logged in full, because they carry the highest information density.

Span-Level Evidence Localization

For citations to be genuinely usable, localization precision has to reach the span rather than the document. One workable design is to require the model to output span identifiers rather than document identifiers during generation, then restore clickable links afterward through a span-to-document mapping. The additional benefit of this is support for span-level highlighting — when a user clicks a citation they see the passage that supports that sentence directly, rather than having to search a long document themselves. This is the key step that turns "citations" from a formal marker into a usable feature.

The stability of span identifiers also needs consideration. If a change in chunking strategy causes all identifiers to change, historical citations are all invalidated. A workable approach is to use "document ID plus content hash" rather than "document ID plus ordinal" as the identifier basis, so that even after rechunking, chunks with identical content can still be matched, or at least old citations can be migrated to new spans through content similarity.

15. The Cost of Building an Evaluation Set

The Evaluation Set Is the Scarcest Asset a Search System Has

Every decision in a search system depends on the evaluation set, but the cost of constructing a trustworthy one is far higher than writing the code. A minimally usable evaluation set needs three things: a batch of queries with a realistic distribution, a set of human-annotated relevance judgments, and a stable stratification scheme (at minimum by intent and difficulty, because an unstratified aggregate score can barely guide optimization).

The most practical source of queries is sampling from online logs rather than having humans construct them. Hand-constructed queries skew heavily toward the "reasonable and answerable" end, while the long tail in real online traffic is far heavier than that. More importantly, queries in real logs carry context (who the user is, when they searched, which page they came from), and that context determines how the correct answer should be judged — the correct result for the same query may differ across users.

The step most prone to problems during annotation is inconsistent granularity. Give annotators the binary question "is this passage relevant to the query" and inter-annotator agreement is usually not great. A more effective question format is to present a ranked candidate list and ask the annotator to reorder it (a relevance preference judgment), because this task is more natural for humans and closer to real ranking needs. The specific annotation method and agreement-checking approach depend on the annotation tooling; the trade-off between annotation cost and quality has no universal answer.

Stratified Evaluation Is More Useful Than an Aggregate Score

For an evaluation set containing all kinds of queries, the aggregate score has almost no guiding value — a ten-point rise in the aggregate could be the result of one query category improving sharply while others degrade slightly. What is actually useful is stratified results: metrics viewed separately by intent category (navigational, factual, exploratory, comparative), and metrics viewed separately by difficulty (head queries, long-tail queries, emerging queries).

Once you have stratified results, the optimization direction becomes clear. For example, if navigational recall is close to one hundred percent while exploratory recall is only sixty percent, the problem lies in semantic generalization rather than overall recall; if head queries perform poorly while the long tail does well, the problem lies in head-level reranking precision. These judgments cannot be reached from an aggregate score, and they directly determine which layer to change in the next round.

Building a Weakly Supervised Set from Online Signals

A human-annotated set is always too small to drive model iteration. Another route is to construct weak supervision signals from online behavior: a user clicking a result means they at least consider it relevant, not bouncing back immediately after clicking means the content quality is acceptable, skipping a result ranked higher means that result is probably not relevant. These signals can be used to train a reranking model, but position bias must be handled — the most direct approach is to trust only the click and skip patterns of the first few positions, and to model obviously navigational clicks separately (immediate bounce-back after the click and going straight to the target page).

A weakly supervised set cannot replace a human-annotated set; the two have different uses. The weakly supervised set is large and broadly covering, which suits model training; the human-annotated set is small but accurate, which suits serving as a non-negotiable evaluation baseline. Keeping only one of them easily produces bias between the two that goes unnoticed.

16. Quality Regression and Online Operations

Configuration as Code, Configuration as Quality Parameters

A search system has a large batch of parameters that determine quality: per-lane recall quotas, fusion weights, freshness decay coefficients, the number of reranking layers, deduplication thresholds, threshold gates per intent, cache expiry times. Scattering them across the code and various configuration centers makes questions like "why did quality get worse last week" impossible to answer. Converging them into a single versioned configuration, and tying every configuration change to a traceable release record, is the first engineering task of search system operations.

Only after centralizing configuration can you do parameter-sweep style regression validation: fix the evaluation set, sweep each parameter across a reasonable range, observe how the stratified metrics change, and aggregate the results into a table. The value of this process lies not only in finding optimal parameters but in understanding the sensitivity of each one — which parameters move the needle substantially on a small nudge (changes to these must be tightly controlled), and which are insensitive across a wide range (these can be handed over to business teams to adjust freely). The value of this sensitivity understanding is severely underestimated in practice; it directly determines where a team should spend its review effort.

Health Metrics That Must Be Monitored at All Times

Beyond the guardrail metrics already mentioned, the anomalies in the following set of metrics usually surface problems before users complain, and they are worth listing separately.

On the index side, monitor: the empty-result ratio of each recall lane (a lane suddenly dropping to zero usually means a build or service anomaly), index generation lag time, and index build failure rate and retry counts. On the recall side, monitor drift in the candidate count distribution — a sudden contraction in the distribution means some category of query hit an undetected failure. On the generation side, monitor the no-answer return rate, the citation verification rejection rate, and average generated output length. On the presentation side, monitor the click position distribution of results.

What these metrics share is that "they are not quality metrics but system state metrics" — their anomalies almost always point at a specific failure rather than quality fluctuation, so alert precision is very high. By contrast, short-term fluctuation in relevance metrics is hard to judge as failure or noise, which makes their alerting value low. These two kinds of signal should be monitored separately and not mixed together.

Failure Drills

A search system's degradation paths are rarely exercised in normal operation, so they have to be drilled proactively. Scenarios worth drilling include: the vector index service being completely unavailable, a recall lane whose latency spikes enough to cause a global timeout, a batch of reranking model timeouts, the generation service being rate-limited, index build tasks hanging long enough to cause generation lag, and the permission service failing so that the filter condition cannot be obtained.

Each drill has to produce a clear answer to one question: what will the user see, and is that outcome acceptable. The last scenario is especially critical — the default behavior when the permission service is unavailable must be "deny" rather than "allow," and such defaults are often written backwards in the code, because they only live on a branch that is unreachable in normal operation. The value of the drill is precisely in actually walking through those normally unreachable branches.

17. The Criteria Behind Architecture Choices

What Scale Requires What Complexity

The complexity of a search architecture should match its scale; here is a rough set of dividing lines. With a corpus under a hundred thousand documents, a single process plus an in-memory index plus an in-memory vector table is enough, with no sharding, no generational switching, and no separate query understanding service. At the million scale, the index needs persistence and sharding support, vector retrieval needs an approximate nearest neighbor structure, and reranking needs a lightweight model deployed independently. At the hundred-million scale, all of the above become mandatory, plus storage tiering, hot-cold separation, and a separate index build cluster.

The significance of these dividing lines is that crossing a given scale means solving several problems at once. Implementing these mechanisms ahead of time before reaching that scale means the maintenance cost drags on the team continuously. Using the crudest possible approach until you reach the next dividing line is usually the correct engineering decision during the small-to-medium scale stage.

Where the Build-or-Buy Line Sits

Retrieval-related components (engines, vector databases, reranking models, generation models) all have usable open-source options and commercial services today, so the decision should rest on four questions: can data leave the domain, do the performance metrics meet hard requirements, is the iteration speed sufficient, and is the total cost of ownership genuinely lower.

Among these, "iteration speed" is often underestimated yet frequently decisive. Quality differences between search systems come mainly from a large volume of fine-grained tuning — chunking strategy, weights, thresholds, the construction of training data — and all of it requires high-frequency iteration. If every change to the chosen option has to go through a long process or depends on an external schedule, it cannot produce those tuning gains, and the final result is actually worse than an option you can freely modify. Any calculation of total cost of ownership has to include engineering headcount and the loss of iteration speed; counting only license fees and machine costs often leads to the wrong conclusion.

One more point needs adding: handing one layer to an external option does not mean giving up understanding of that layer. The concrete implementations of the recall, reranking, and generation layers differ enormously across options, and without understanding their internal mechanisms you cannot tell which layer the problem is in when they underperform — whether the parameters are wrong or the option is a poor fit — leaving you to switch vendors over and over instead of solving the problem. Maintaining an understanding of each layer's mechanism is the precondition for keeping room to optimize.