Multimodal Document Understanding: From Layout Parsing to Semantic Retrieval

I. Start by Defining the Problem: Document Understanding Is Not OCR

When a team takes on a new requirement, the sentence it repeats most is "we need to read these PDFs." That sentence never lands on anything concrete, because it fuses four subtasks with completely different failure modes into a single deliverable — and those four subtasks barely overlap in cost structure, in viable alternatives, or in the debugging path you take after something breaks. What you actually owe your caller is never "what characters are printed on this image," but "what structured facts exist in this document, what are the source coordinates of each fact, how trustworthy is each one, and which regions did we fail to read altogether." Those four questions together are the system contract.

1.1 The four subtasks that get conflated

The first is character recognition: turning pixels into a character sequence. This is a closed problem — there are labeled datasets, there are well-defined metrics, and the optimization space has essentially been mined out. The second is layout understanding, which answers "which characters belong to the same logical block, what are the geometric and containment relations between blocks, and what is the reading order of the page." That is a structural problem, and it is the one most likely to fail catastrophically: even if recognition accuracy reaches 98%, if the two wrong characters happen to land in an amount field or an ID field, the entire record is wrong.

The third is element-level semantic classification, translating geometric blocks into business concepts: is this a heading or body text, is this a table or a bordered paragraph misdetected as one, is this a figure caption or a footnote, is this a running header or the document title. The fourth is cross-element structural reasoning — for instance, a table whose header sits on the previous page while its data rows sit on the next, or a formula reference embedded in a paragraph whose meaning is only clear after you resolve the reference. Of these four, only the first has been treated as a "core capability" worth investing in for many years; the other three were usually filed under "preprocessing" or "postprocessing." That is the primary source of incidents in document parsing pipelines.

1.2 What layout understanding actually covers

Broken apart, layout understanding is at least four things that have to be right independently. First, region localization: which blocks exist on the page, and what is each block's bounding rectangle. Second, type classification: which of heading, body text, list, table, figure, caption, header, footer, formula, or references each block belongs to. Third, structural relations: parent–child relations between blocks (a table cell belongs to a table), ordering relations (reading order), and adjacency and containment relations. Fourth, continuation across pages and across columns: is the bottom of this page and the top of the next page the same table, or the same paragraph continuing across columns.

Of these four, only the first has a reasonably mature off-the-shelf solution. The other three are exactly the parts that look simple but whose errors are completely invisible to the layer above. A pipeline that reverses the order of a body block and a caption block still emits text that reads fine, and you will get no warning at all — yet downstream retrieval will keep recalling figure numbers out of captions as if they were body-text entities. Only layout-level metrics can surface errors like this.

Subtask	Typical input	External symptom on failure	Is a single-stage failure visible?
Character recognition	Cropped text-line images	Wrong characters, merged lines, formulas degraded into plain characters	Yes — there are metrics, easy to catch
Layout structure	Full-page bitmap or vector primitives	Reading order scrambled, tables shattered, cross-page breaks	Often invisible
Element semantics	Layout blocks and their geometric attributes	Captions treated as body text, headers mixed into body, tables read as paragraphs	Invisible
Structural reasoning	All of the above plus document metadata	Lost headers, misaligned cells, failed coreference resolution	Invisible

The last two columns of this table are the point I want to emphasize: the higher up a stage sits, the more likely its errors are caught automatically; the lower it sits, the more the output "looks right," and it can only be surfaced through end-to-end business metrics or manual sampling. Engineering effort should be allocated in that reverse order, not by how novel the technology happens to be.

1.3 Why reading order cannot be treated as post-processing

In many implementations, layout detection runs first and produces all the blocks, then the blocks are sorted by their top-left coordinates — y first, x second — and that sorted list is declared the reading order. This works on single-column pages with a single text line and clear whitespace between blocks. It falls apart on real layouts like these: in a two-column paper, the left and right columns interleave heavily in y, so sorting by coordinates mixes lines from the two columns; in a magazine layout with a sidebar, the sidebar text happens to land in y right between two body paragraphs; inside a table, the coordinate relationships between cell texts are completely different from those of paragraphs, so mixing them splices table content into the middle of body text.

The correct approach is to model reading order as an explicit structural problem rather than a sorting problem: first do region grouping (which blocks belong to the same column, the same table, the same figure), then order the blocks within each region, and finally order the regions themselves by their layout semantics. These steps cannot be swapped, because deciding "which blocks belong to the same region" itself requires joint geometric and semantic judgment. A more pragmatic compromise is to let an end-to-end model directly emit Markdown or JSON already organized in reading order, pushing the ordering problem down into the model — but that means accepting stability worse than an explicit geometric pipeline, and paying a higher per-call cost.

II. The Four Things Layout Understanding Has to Get Right

2.1 Region localization: keep the noise out

The real difficulty of region localization is not detection accuracy but noise filtering. Binding shadows on scans, paper creases, fax speckle, handwritten annotations in the margins, and seals straddling the gutter all produce low-confidence "phantom blocks." If those phantom blocks get through, every downstream stage pays for them: the recognition engine runs over a meaningless multi-line text, the layout type classifier emits a wrong label, and the semantic extraction stage may drop it as a footer — or splice it into the result as body text.

There are a few cheap rules worth implementing first: filter out tiny fragments by area and aspect ratio; put a lower bound on the confidence threshold — prefer a miss to a false positive, because a miss shows up downstream as "this text does not exist" and is easy to catch, whereas a false positive shows up as "there is some extra content" and is very hard to catch; run a quick check on color variance and local contrast, and label obviously image-like blocks as figures instead of passing them downstream. The bar for this stage is not accuracy — it is whether the post-filter false positive rate is low enough to ignore.

2.2 Type classification: the ambiguity comes from the document's own conventions

Type classification looks like a ten-or-so-class categorization problem. The actual difficulty is that in real documents the same visual form carries multiple semantics. A centered, bold, larger line of text might be the document title, or a section heading, or merely a line inside a decorative divider. A bordered rectangular region might be a table, or a code block, or an image placeholder, or even a single-line callout box. Small text under a figure might be a caption, or it might be a body sentence that simply happens to be typeset smaller.

What actually improves accuracy is not swapping in a bigger classification model — it is bringing in evidence from outside the layout. Document-level metadata (section structure, bookmarks, the table-of-contents tree, page-number positions) provides strong priors for block types: large centered text on a table-of-contents page is far more likely to be a section name; a block that appears at the same vertical position on every page, with content that repeats heavily across pages, is almost certainly a header or footer; short text near the bottom of the page matching a "Figure X" or "Table X" prefix pattern is more likely a caption. Writing these priors explicitly into the rule layer is usually far more stable than letting a model guess from pixels alone — and it costs almost nothing.

2.3 Structural relations: turn blocks into a tree or a graph

A set of blocks on its own means nothing; structure has to be built. Most implementations model the layout as one of two things: a hierarchical tree, or an annotated relation graph. A tree naturally carries reading order (a preorder traversal is enough) and expresses relations like "this caption sits under this figure" or "this table belongs to this section" very naturally. Its drawback is that elements needing two-dimensional structure, such as tables, are hard to fit into a tree; they usually degrade into a flat "list of table cells," and the row and column information is thrown away. A relation graph is more flexible — it can express "belongs to the same table," "is the caption of the previous block," "is a two-column sibling of the left neighbor," and more in a single vocabulary of edges — but you have to define the traversal order yourself, and the complexity is noticeably higher.

A pragmatic compromise is to model the document as a "two-level hybrid structure": the top level is a tree of regions (sections, paragraph groups, column blocks, tables, figures), and each region hangs its own local structure beneath it (a table's interior is a two-dimensional grid, a paragraph's interior is a sequence of lines). Reading order is then decided by the top-level tree, while a table's row and column information is preserved in the local structure and is not destroyed by global sorting. This hybrid structure is directly reusable in both of the later stages — table reconstruction and retrieval chunking — which makes it one of the highest-leverage design decisions available.

2.4 Cross-page and cross-column continuation: the step most teams miss

Cross-page continuation has two detection signals, and in practice you should use both. The first is geometric: the top of the next page contains a region that closely matches the table at the bottom of this page in column count, column widths, and rule-line style, while the table at the bottom of this page is clearly unfinished (the last row has content in only some cells, or the bottom border is missing). The second is content-based: the header text of the two tables is identical or highly similar, and the header does not appear on the first page. Only join when both signals fire — that is far more reliable than looking at either one alone.

Two extra problems show up after the join. The first is imperfect column alignment: scanning skew can make the same column slightly different in width across the two pages, and forcing pixel alignment misaligns the content; aligning logically by header text and column index is usually more robust. The second is headers and footers inserted mid-table: some typesetting setups repeat the header row when they continue a table, so you must deduplicate explicitly before merging. Cross-column continuation is simpler but easier to overlook: a paragraph in a two-column layout breaks at the bottom of one column and continues at the top of the next. If you process page by page, the two columns become two independent text segments, the sentence gets cut in half, and on the retrieval side it turns into two low-quality chunks.

III. Where the Line Between Traditional Pipelines and End-to-End Approaches Actually Sits

3.1 The classic four-stage pipeline

The traditional approach breaks document processing into a set of cascaded stages: page preprocessing first (rotation correction, shadow removal, binarization), then layout detection and classification, then optical character recognition (OCR) on each text block, then table structure reconstruction, and only then semantic extraction and field filling. The upside is that every stage is replaceable, cacheable, and independently deployable: swapping the recognition model does not touch layout detection, and changing a layout rule does not require re-running the expensive recognition step. Each stage can be evaluated in isolation to localize problems, and per-page processing cost is low, so throughput can be very high.

Its downsides are just as clear: cascade error compounds. If layout detection misses a block, that block's text never enters the result and nothing raises an error. If table reconstruction gets one row wrong, downstream field extraction attaches that row's numbers to the wrong column names — and field extraction itself looks perfectly normal. These "silent errors" are the largest hidden cost of traditional pipelines in enterprise settings, and they typically surface only when a business team notices that one quarter's report numbers do not add up, then has to trace backward to find them.

3.2 The end-to-end approach: reading the page image directly

The shape of the end-to-end approach is this: render a whole page as an image, hand it straight to a vision-language model, and require it to emit the page's text content, its structural labels, and even an HTML representation of any tables. The biggest change in this line of work in recent years comes from the improving capability of vision document models: on table reconstruction, reading order, and caption association, their performance has moved from "dazzling in demos, unusable in production" to "usable on most standard scenarios, still needs a fallback on hard ones." How far that actually goes depends on the model version, the resolution budget, and the prompting strategy, and it is not something to summarize with a single conclusion.

The structural advantage of the end-to-end approach is that it handles geometric and semantic information at the same time. A traditional pipeline has to classify the layout before it can decide what to do next, whereas a sufficiently strong vision model can infer from the overall context that "this small text should be a caption," even though its geometric shape is identical to body text. It also requires no rule set, and behaves more smoothly on layouts it has never seen. There are three costs: per-page cost is significantly higher than in a traditional pipeline; errors are often unexplainable, because when the model reads a two-row table as a paragraph you get no intermediate state to inspect; and stability is worse than a geometric pipeline — the same input can yield different-quality output after a model version bump or a prompt edit, which is a hard blocker for audit scenarios that require reproducibility.

3.3 Comparing the failure modes of the two routes

Put the failure modes side by side and the logic behind route selection becomes obvious. The typical failure of a traditional pipeline is "missing" and "misaligned": an entire passage of text is lost, or a row of data is attached to the wrong column. Its signature is that the errors are stable and reproducible — once one appears you can fix it by adding a rule. The typical failure of the end-to-end route is "overconfident smoothing": it completes structures that do not exist, replaces blurry characters with words that fit the context, and merges two unrelated tables into one narrative. Its signature is that the errors are unstable and can only be mitigated by constraining the output and sampling multiple times.

Dimension	Traditional four-stage pipeline	End-to-end page reading
Per-page cost	Low, self-hostable, high throughput	High, depends on a model service, subject to rate limits
Error explainability	Strong — intermediate state visible, localizable to a specific stage	Weak — only the final output, hard to localize
Layout generalization	Weak — a new layout needs new rules or new training data	Strong — behavior varies with context, smoother on unseen layouts
Output stability	Strong — deterministic input yields deterministic output	Affected by version, prompts, and sampling parameters
Two-dimensional table structure	Depends on dedicated structure reconstruction; complex tables are error-prone	Can emit table representations directly, but mismatches still happen
Long-document global context	Naturally paginated, supports incremental processing	Often constrained by single-page or single-image context limits

One point deserves separate mention: end-to-end models are not naturally good at "global consistency across very long documents." What they are good at is understanding within a single page. Cross-page header inheritance, continuous section numbering, and coreference resolution still require an explicit document-level structural layer to catch them. Conflating single-page understanding with document-level assembly is the most common misjudgment when an end-to-end approach lands in production.

3.4 In production, the hybrid form is far more common

The systems that actually hold up in production are rarely pure versions of either route; they draw an explicit line between "who does localization" and "who does understanding." One common and effective split is this: geometric work (shadow removal, rotation correction, header/footer stripping, column detection, cross-page table pairing) goes to deterministic image and geometry processing, because those tasks have clear rules, are cacheable, and produce repairable errors; semantic work (element type classification, table semantics reconstruction, field extraction, question answering) goes to the multimodal model, because the rules are hard to enumerate and depend heavily on context. The recognition stage in between can be invoked selectively per region: text blocks go through OCR, table regions and figure regions are sent straight to the model as raw images, avoiding the compounding loss of "OCR first, then let the model understand the OCR output."

A different split is dual-path parallelism: the traditional pipeline produces one structured baseline, the end-to-end model produces another, and a lightweight arbitration layer decides whom to believe. Arbitration can be confidence-based or consistency-based: if the two paths disagree on a given field, mark that field as low confidence and send it to a human queue rather than forcing a binary choice. This combines the generalization of the end-to-end route with the explainability of the traditional pipeline, at the cost of maintaining arbitration logic and accepting some volume of manual review. For compliance-heavy scenarios where a single wrong row is enormously expensive, this path is worth the investment. For high-volume, low-value scenarios, a single end-to-end path plus sampled quality checks is enough.

IV. Tables: The Hardest Link in the Whole Pipeline

4.1 The first layer of difficulty: cell boundaries are a geometry problem, not a recognition problem

The first step of table reconstruction is determining the grid. The classic approach detects horizontal and vertical lines in the table region image, projects them into row and column boundaries, and then infers cell extents by checking whether adjacent boundaries connect through. This works very well on cleanly printed, sharply ruled tables exported electronically, but real documents break it in two common ways. The first is the partially ruled table: many reports draw only the outer frame and horizontal rules and never draw vertical rules, so column boundaries have no visual basis at all and can only be inferred from horizontal alignment of content — and as soon as one column's content length distribution is heavily skewed, all the column boundaries shift. The second is the unruled table, where cells are separated purely by whitespace; here even row boundaries have to come from gap clustering, and clustering fails outright when row heights are uneven or cells span multiple rows.

For these two degraded structures, a sound reinforcement strategy is "geometry first, content as fallback": use the lines to give high-confidence row and column anchors, then use the distribution of cell-text centroids and the whitespace gaps to fill in the boundaries that have no anchor, and finally run a consistency check against the document's template profile — across pages of the same template, tables usually share the same row and column counts or follow a fixed grouping pattern, so if the column count inferred on one page disagrees with the document-level mode, that should trigger recomputation or degradation. This consistency check at the layout level intercepts a large volume of silent column misalignment.

4.2 The second layer of difficulty: merged cells and nested headers

Handled badly, merged cells are the direct cause of column-name mismatch. There are two common shapes: horizontal merging (the first row has one "Total Revenue" heading spanning three columns, and the second row holds the three specific line items) and vertical merging (a "2024" label in the first column spans two rows, with revenue and cost in the other two columns). If you treat them as ordinary cells, the table degenerates into a two-dimensional array with holes in it, column names are lost, and during semantic extraction the model or the rules have no idea which label the value "2024 revenue" should carry.

The right way to model this is to preserve the merge relations themselves: resist the urge to flatten the grid into a clean rectangle, and first emit an intermediate structure that carries span information, with every cell recording its own row start/end, column start/end, and text. From that structure you then derive business-facing views, for example expanding rows into a list of records of the form "year + line item + value." Even when the header is nested multiple levels deep, the expanded records still carry the full path above them. Nested headers also involve two special structures that need separate recognition: first, a header that itself spans a logical page (the previous page carries a "Quarterly Data" super-header, while this page has only the quarter columns); second, a header that contains both dimension columns and measure columns (measure columns typically have a totals row at the bottom, dimension columns do not). These two pieces of information determine which columns should become dimensions and which should become measures when the table is later sliced.

4.3 Three routes from table pixels to structured data

Combining grid, text, and merge relations into a final structure comes down to roughly three routes, each with a clear applicability boundary. The first is pure geometry plus rules: on the detected grid, walk every cell and fill the text blocks in by containment, identifying merged cells via "no vertical rule connecting through" or "horizontal rule connecting through" rules. This route is fast, stable, and reproducible, but struggles badly with in-cell line breaks, nested sub-tables inside a cell, and unruled tables. The second is geometry plus object detection: first detect a bounding box for each text line, then fill the table using a "which grid intersection does this belong to" assignment rule, while training a separate model specifically to recognize merge spans. This route is more accurate on complex tables, at the cost of requiring annotations for your table style and having limited cross-template transfer. The third is handing it straight to a multimodal model and parsing the HTML table it emits. This route is a relief for unruled tables and nested headers, but it converts a geometry precision problem into a model generation problem, and once the cell count the model emits disagrees with the visual row count, the error enters downstream in a highly hidden way.

In practice I lean toward the combination of "geometry for boundaries, model for semantics": let the geometric pipeline produce a grid carrying merge information, send the grid together with cropped cell images into the model, and give the model exactly one job — filling in a semantic label for each cell (line item, period, amount, unit, is-total) plus unit normalization. The model then never has to re-infer geometry, so the room for hallucination is compressed down to the label level, which is precisely where the model is strongest. Here is a simplified implementation skeleton.

class TableStructurer:
#输入是页面位图与版面树中的表格区域节点
    def restore(self, page_image, region):
        grid = self.grid_detector.infer(page_image, region.box)
#线条缺失时用空白间隙补边界，并对不上文档级列数时重算
        grid = self.gap_fallback.fill(grid, page_image, region.box)
        grid = self.template_check.reconcile(grid, region.doc_profile)
        cells = self.text_filler.assign(page_image, grid)
#合并跨度在这里被保留，不压平
        spans = self.span_detector.detect(page_image, grid, cells)
        return Grid(cells=cells, spans=spans, doc_profile=region.doc_profile)

    def semanticize(self, grid, page_image):
#只让模型填标签与单位，几何结果不交给模型重推
        crops = grid.crop_each(page_image)
        raw = self.vlm.label_batch(crops, prompt=self.LABEL_PROMPT)
        labels = self.schema_validator.repair(raw, allowed=self.dimension_hint)
        return grid.to_records(labels)

Two easily overlooked details matter a great deal here. The first is unit normalization: if the header says "10,000 yuan" and the value is "1234.5," failure to normalize at this stage corrupts every cross-table comparison downstream — and the error is completely invisible within a single table. The second is handling total rows: total rows should generally be excluded from the detail records, yet at the same time they are the most valuable validation anchor — if the detail rows do not sum to the total, some cells were misread or missed, and this check should be written into the pipeline as a mandatory output quality gate rather than left to a human reader.

V. Charts: First Decide Whether You Should Read Them at All

5.1 Two routes: convert to data, or go through question answering

Chart handling has two routes with entirely different properties. The first is chart-to-table, whose goal is to extract the numbers out of line charts, bar charts, and pie charts into a data table. The value of this route lies in its output being structured numbers that can be computed on, aggregated, and compared, and that can be hit precisely once they enter a retrieval system. But its technical difficulty is high: converting axis scales and units, decomposing the heights of stacked bars, matching colors to legends across multiple series, handling error bars and confidence intervals, correcting the proportional distortion caused by truncated axes — each of these needs dedicated handling, and if any one of them is wrong, an entire column of numbers is wrong.

The second route is chart question answering: hand the chart and a natural-language question to the multimodal model and let it answer "which year had the highest revenue" or "what does this trend indicate." This route works well for judgment-type questions — trends, comparisons, extrema — because the model is already doing visual reasoning and you never have to restore the numbers explicitly. It is clearly unsuited to cases that need exact values, such as "list every point's value and sum them": the numbers the model produces cannot be self-checked visually, the error may reach several percentage points, and nothing reveals it.

5.2 Routing signals you can actually decide on

In practice, do not rely on humans to judge which route each chart takes; construct a set of routing signals that can be extracted automatically. The strongest signal is chart type: bar charts and line charts are relatively friendly in terms of numeric legibility; pie charts and donut charts allow proportions to be estimated but make exact values hard; scatter plots and box plots are fundamentally unsuitable for exact data conversion; multi-series stacked charts are the hardest. The second signal is axis information: if the axes have explicit tick labels and start at zero, numeric conversion is highly reliable; if the axes are truncated, lack tick labels, or the axis labels are relative metrics (growth rate, share of total), the risk of data conversion rises sharply.

The third signal is the downstream use, and it is often more important than the first two. If the downstream consumer only needs an overview for a human, or needs the chart as supporting context during retrieval, take the question-answering route and emit a natural-language description. If the downstream consumer needs numeric computation, cross-chart alignment, or a data warehouse, you must take the chart-to-table route and accept a higher per-image cost plus a certain failure rate in transcription. Write these three signals into a routing function that returns one of three outcomes — "convert to data / question answering / degrade to an image description" — and the result is far more stable than applying one uniform strategy. In particular, the "degrade to an image description" tier must exist: it guarantees that when a chart genuinely cannot be handled reliably, the pipeline explicitly marks it as low value instead of producing a plausible-looking but wrong number.

5.3 Do not lose the caption-to-body relationship

Elements other than the chart itself also need correct attribution. A caption is typically a numbered short sentence containing the chart type, the time range, the statistical caliber, the unit, and the source. All of these are prerequisites for the data being usable: time series with different calibers cannot simply be concatenated, and values with no source are unusable in compliance settings. A caption should therefore not be treated as ordinary body text and spliced into a paragraph; it should be bound to its figure as a single retrieval unit, so that "figure region + caption text" is indexed and recalled together. This is especially valuable in RAG settings: when a user asks "the sales share by region in 2024," what should come back is the figure block carrying the caption "Figure 3, Sales Share by Region (2024, tax-inclusive, unit: 10,000 yuan)" — not some sentence in the body that happens to contain the words "share."

There is also a boundary condition that is easy to get wrong in caption attribution: when a page holds several charts, a caption may appear above the figure rather than below it, and when two figures sit side by side, the caption may hug only one of them. When deciding attribution, combining "nearest neighbor + layout direction + numbering pattern" is more reliable than distance alone. And when the numbering is missing (no "Figure X" in the caption), text content becomes the primary signal, misattribution becomes more likely, and this falls into the category of cases that need sampled manual verification.

VI. Scans and Phone Photos: Which Stage of the Pipeline Collapses First

6.1 Degradation is tiered, not gradual

The most important thing to understand when handling real input is this: image-quality degradation is not a continuum but a small number of discrete "steps." A scan may be clean overall yet the whole page is tilted five degrees; it may have very high resolution yet a binding shadow running the full height of the page. Different types of degradation hit different stages of the pipeline and get worse in stages until one link fails completely. Treating all degradation as a single "overall quality score," or handling every case with one uniform preprocessing pipeline, is the main reason results are unstable.

6.2 Each category of degradation and where it breaks

Skew and perspective distortion hit layout detection first. When text lines are not horizontal, gap detection based on horizontal projection fails, and row boundaries are split incorrectly; at the same time skew makes glyphs on adjacent lines approach each other, so within-line segmentation during recognition breaks too. Rotation correction should happen before recognition, and it must distinguish "whole-page rotation" from "local perspective" — the former only needs a global angle estimate, while the latter is common on contracts and receipts photographed with a phone and requires a perspective transform; and the margin cropping after the transform has to be conservative, or you will cut off content near the page edge.

Shadows and uneven illumination hit the recognition threshold. If binarization uses a global threshold, strokes in shadowed regions get flattened or turn into solid white. The usual countermeasures are background estimation and normalization before binarization, or simply switching to an adaptive threshold in shadowed regions. A more practical move is this: if the pipeline does not need morphological analysis or line detection (both of which table structure depends on), consider skipping binarization entirely and feeding the raw image into the recognition engine — modern recognition engines usually normalize enough internally, and hand-rolled binarization can easily do more harm than good. This is a frequently overlooked simplification.

Low resolution hits small text and tables. Below a certain threshold the recognition error rate does not rise gradually, it collapses, and it does so especially on densely packed small-text tables, footer notes, and multi-digit identifiers. The right move at that point is not to grind through the full pipeline anyway, but to run a quick resolution and font-size estimate first and flag regions predicted to be unreliable as "needs human review" ahead of time, so that garbage never enters downstream as valid data. Better to fail explicitly than to silently emit wrong numbers.

Handwriting hits the recognition model itself. Most recognition engines trained on printed text degrade substantially on handwriting, especially cursive, hurried, and mixed forms (handwritten annotations inserted into printed body text). Recognition accuracy for handwriting mixed with print depends heavily on the specific implementation, and handwriting-specific models vary enormously in availability and cost, so the realistic move is to recognize the handwritten regions at the layout stage and route them separately, rather than expecting one engine to cover both.

Mixed-language content hits the language-identification and tokenization assumptions. Recognition engines usually embed the assumption that "this block of text is all one language," and when they meet an English term embedded in a Chinese paragraph, or a page mixing Chinese, English, Japanese, and Korean, they easily substitute visually similar characters for one another (for instance full-width versus half-width forms, or Latin letters that differ only in case). These errors may still read sensibly semantically, which makes them nearly invisible downstream. The effective countermeasures are block-level language classification, reprocessing mixed-language blocks with an engine that supports multiple scripts, and keeping a language tag in the output so that later stages can choose tokenization and retrieval strategies.

6.3 The degradation ladder: turning failure into an observable state

Every category of degradation should have its own fallback path, rather than sharing one "quality too low, discard it" path. A workable degradation ladder usually looks like this: start with the full geometric pipeline plus recognition plus structure reconstruction; when geometric confidence is insufficient, fall back to having the multimodal model emit the page structure directly; when the model's output fails validation, fall back to plain-text recognition plus sequential stitching, explicitly tagged "reading order unreliable"; and when recognition confidence is too low, stop degrading — emit an empty result plus the coordinates of the problem region and send it to a human queue.

The core of this ladder is not "always produce a result" but "always produce an explainable result." Every degradation level has to leave a trace in the output: which path was taken, which regions were degraded, and why. This metadata becomes extremely useful later — it lets a problem like "retrieval recalled a passage of wrong text" be traced back to a specific page and a specific degradation decision, instead of becoming an undebuggable black box. A pipeline with no degradation markers can only re-run everything when something goes wrong, and a full re-run is expensive, so in practice people choose not to re-run at all.

6.4 A practical front-door quality gate

Running one cheap gate check before formal processing usually pays off well. The checks include: whether the page dimensions fall in a reasonable range (too small may mean a thumbnail, too large may mean a high-resolution scan was inserted); whether the estimated text-line density is abnormal (too low may mean a blank page or an image-only page, too high may mean photo mode was chosen and moiré patterns appeared); whether large solid-black or solid-white regions exist (possibly a capture mistake or a scanner lid left open); and whether the page shows obvious skew beyond eight degrees. None of these need a model — plain image statistics suffice, and the runtime is negligible. Making them a batch pre-filter blocks a substantial share of invalid jobs, and the compute saved often exceeds a thousandfold the cost of running the checks themselves.

VII. Output Shape: Structured Output Is Worth More Than Long Text

7.1 Why you should not emit a single long string

Many document understanding pipelines ultimately emit a single string. That looks simple, but in practice it flattens a large amount of information, and flattening costs three things. First, traceability is lost: neither the user nor the system can get from the result back to the source page, and the most basic question — "on which page and in which block does this conclusion come from?" — goes unanswered. Second, recomputation becomes impossible: once downstream discovers that some class of field was extracted wrongly, the only option is to re-run the whole pipeline from the original file. Third, retrieval quality degrades: the same chunk of text mixes headings, body text, captions, and table content, so plain-text chunking can only cut by length, and the resulting pieces are often semantically incomplete.

A more valuable output is a set of element-level structured objects: each element records its own type, its coordinates on the page, the section path it belongs to, its text content, its confidence, and its relations to other elements. A document is then these elements plus the structural relations among them. With coordinates you get traceability and an entry point for human review; with types you get the possibility of type-differentiated retrieval; with relations you get a carrier for cross-page continuation and table reconstruction.

7.2 Where structured output meets constrained decoding

When using a model for document extraction, the biggest pain point is unstable formatting: fields go missing, types drift, extra fields show up, and downstream parsers crash as a result. Structured-output constrained decoding turns the output grammar into a hard constraint of the decoding process — the model can only pick tokens that form a legal structure, which eliminates format drift by construction. Its junction with document parsing is this: the thing being constrained is not a stretch of free text but a structural definition that conforms to the document model — element types come from the layout-type enum you defined, coordinates come from the page grid you marked up, and section paths come from legal nodes on the document tree. These three constraints map exactly onto the three places in document understanding that are both most error-prone and most in need of validation.

It is worth noting that constrained decoding changes the model's output distribution. Layering semantic constraints on top of grammatical ones (for example requiring that an element of type "caption" have a text length under some cap, or that table elements carry column-count information) compresses illegal combinations further, but it can also make the model converge prematurely to a barely legal output on a handful of edge samples. The paired validation layer is therefore not optional: validation should run as an independent pass after decoding and pick out samples that are "syntactically legal but semantically unreasonable." Here is a skeleton of an output contract and its validation.

DOC_SCHEMA = {
"type": "object",
"required": ["page", "elements"],
"properties": {
"page": {"type": "integer"},
"elements": {"type": "array", "items": {
"type": "object",
"required": ["kind", "bbox", "text"],
"properties": {
"kind": {"enum": ["heading", "para", "table", "figure", "caption",
"header", "footer", "formula"]},
"bbox": {"type": "array", "items": {"type": "integer"}, "minItems": 4},
"text": {"type": "string"},
"confidence": {"type": "number"}
}}}}}

def extract(doc, page_image, vlm, validator):
#语法层由解码器保证，这里不再解析自由文本
    obj = vlm.generate(image=page_image, schema=DOC_SCHEMA)
    issues = validator.check(obj)
#语义层独立校验：长度、坐标合理性、覆盖率
    issues += check_coverage(obj, page_image)
    issues += check_context(obj, doc.chapter_tree)
    for el in obj["elements"]:
        el["route"] = pick_route(el, issues)
    return obj

7.3 Make coordinates a first-class citizen

Leaving coordinates in a first-class position pays off in many places downstream. A manual review UI can jump straight to the erroneous region; diffing tells you what this pipeline version recognized that the previous one did not, and vice versa; confidence can be weighted per region instead of applied uniformly across the page; and rendering back onto the original image for sampled visual inspection needs no extra coordinate mapping work. All of these capabilities come for free out of structured output, provided you leave the coordinate field in from the start — you cannot add it later, because adding coordinates later means re-running recognition.

There is one more easily overlooked use: coordinates are a diagnostic signal for whether the model is genuinely reading the image. If a vision model's output coordinates are heavily concentrated in one place, or cover the entire page uniformly, it is probably not doing layout-aware perception at all but treating the problem as text generation and guessing. Monitoring the trustworthiness of coordinates as an independent evaluation dimension lets you catch capability regressions the moment you upgrade a model.

VIII. Plugging Into Retrieval: How Parsing Results Become Recallable Units

8.1 Chunking units are not fixed-length windows

If the output of document parsing is chunked at a fixed length and pushed straight into the index, retrieval quality will be poor. There are three reasons: a paragraph cut in the middle yields a semantically incomplete chunk; a table cut apart separates header rows from data rows, leaving the chunk meaningless; and losing the heading hierarchy strips the chunk of its topical context. For the same content, splitting along semantic units is markedly better than splitting by length, and this is one of the largest payoffs of doing parsing properly.

The recommended chunk granularity is "section path + complete paragraph or complete table." Concretely, let each chunk carry its full path on the document tree (for example "Chapter 3 Cost Analysis > 3.2 Unit Cost > some paragraph"), while the chunk body holds only the paragraph itself. When retrieval hits, splice the path into the context as well, and the model then knows which section it is answering about. For tables, prefer chunking by "header + several rows," and make sure every row carries the expanded form of the column names, so the chunk remains self-explanatory once detached from the original table. For figures, the chunk is "caption + a model-generated description of the image," not the image itself.

8.2 Hybrid retrieval: why document scenarios especially need BM25

Document scenarios differ sharply from casual chat scenarios in what they need from retrieval: documents are full of proper nouns, identifiers, model numbers, clause numbers, and statutory citations — "exact tokens." Vector retrieval is strong at semantic similarity but fumbles exact matching: a user searching for "A320-200 model, clause 3.3" may get back a pile of semantically related paragraphs whose identifiers are all wrong. Term-matching methods like BM25 perform steadily on exact matching, and the two are complementary — which is exactly why hybrid retrieval is close to mandatory in document scenarios.

There are several ways to fuse hybrid retrieval results, the simplest and most common being a weighted sum: take the vector-recall and keyword-recall result lists and accumulate them weighted by reciprocal rank. A slightly more careful option is reciprocal rank fusion, which depends only on rank and not on score distribution, making it insensitive to how differently different retrievers scale their scores — in practice more stable. A third option is to rewrite the query first, expand it into several phrasings, recall each separately, and aggregate the results; this is especially effective on documents with mixed languages, because the query language and the document language may differ. The choice of fusion weights depends on the corpus and is not something to copy from someone else's experience, but a useful starting point is to weight keywords slightly higher, because the share of exact tokens in document queries is usually higher than in natural-language questions.

def hybrid_search(query, index, w_vec=0.45, w_kw=0.55, topk=50):
vec_hits = index.knn(query.embed(), topk)
kw_hits = index.bm25(query.analyze(), topk)
score = {}
for rank, h in enumerate(vec_hits, 1):
        score[h.id] = score.get(h.id, 0.0) + w_vec / (60 + rank)
for rank, h in enumerate(kw_hits, 1):
        score[h.id] = score.get(h.id, 0.0) + w_kw / (60 + rank)
#片段自带路径与表格展开，命中即可拼上下文
    return sorted(score.items(), key=lambda x: -x[1])[:topk]

8.3 What has to be done on top of hybrid retrieval

The ceiling on recall quality usually does not lie in the retrieval algorithm but in a handful of details at index construction time, and every one of those details depends on parsing quality. First, metadata filtering: retrieval should be able to filter by document type, section, time, and confidence, which requires the parsing stage to emit reliable type and section information. Low-confidence chunks can optionally be kept out of the index or down-weighted — recall less rather than recall wrong. Second, self-contained chunks: a chunk must carry enough context (section path, expanded header, caption), otherwise after recall it gets spliced into the prompt and the model still sees subject-less fragments. Third, numeric table columns have to be reachable by retrieval: if an amount in a chunk is formatted as a string with thousands separators and a currency symbol, exact retrieval will hit it but semantic retrieval may not understand it, so store both representations.

A fourth point is often overlooked: content in documents that is "important but not sentence-shaped," such as parameter tables in appendices, restriction notes in footnotes, or a "Note: figures exclude tax" line below a table. Such content has an enormous influence on answer correctness, yet it is often dropped during chunking because the block is too small or contains no sentence breaks. The parsing stage should give this kind of content an explicit label, so that it is either merged into the paragraph it belongs to or preserved as an independent short chunk.

8.4 Traceability from retrieval all the way to the answer

Since the parsing stage already preserves coordinates and section paths, you should carry them through to the final answer. Make the generation stage cite chunk provenance (page number plus section path), have the front end turn that into clickable traceability links, and let a click jump directly to the corresponding region of the source document with highlighting. This is not merely a UX nicety — it gives a wrong answer a fast path to being spotted and corrected, and it separates the two failure classes "the system is wrong" and "the model misunderstood." In compliance scenarios, traceability is often a hard requirement rather than a bonus.

IX. Evaluation: Why Extraction Accuracy Alone Will Fool You

9.1 Build the evaluation set by working backward from real failures

The easiest place for document understanding evaluation to go wrong is starting annotation from a clean, self-selected sample of documents. Such an evaluation set yields a number that looks good but carries no decision value: it cannot tell you which stage to optimize, nor whether this change actually helped. A more effective approach is to derive the evaluation set backward from production traffic — pull out every case that downstream has questioned, every case that human review marked as wrong, and every case the retrieval system recalled and judged irrelevant, and have the business side help pick a representative subset for careful annotation. An evaluation set built from real failures may only contain a hundred documents, but every metric movement it produces can support one explicit optimization decision.

The evaluation set also needs stratified sampling rather than random sampling. Randomly drawing a hundred documents may well produce ninety samples of the same template, in which case the metric you measure only reflects how well the system fits that one template. Reasonable stratification dimensions include: source channel (electronic export, phone photo, scanned archive, third-party system push), layout complexity (single column, two columns, multi-column, newspaper-style mixing), content type (contracts, financial reports, manuals, receipts, papers, forms), and quality tier (clean, skewed, shadowed, low resolution). Reporting metrics per tier is the only way to see which layer is the weak one.

9.2 What each layout-level metric measures

The value of layout-level metrics lies in localization, but their correlation with end-to-end results is limited, so they have to be used as a group. Region localization metrics can be computed from geometric overlap, and improvements here often do not track business value — cutting layout blocks finer makes geometric overlap look better, but over-cutting destroys sentence completeness. Type classification metrics (per-class precision and recall) need a per-class confusion view, with attention on the confusable pairs: caption versus body text, header versus body text, table versus bordered paragraph. Reading-order metrics either have humans annotate the order and measure consistency, or are reflected indirectly through a downstream proxy metric — for example, splitting the output into sentences and computing sentence completeness rate. That sentence completeness metric is crude, but it is highly sensitive to reading-order problems and is worth monitoring routinely.

Table structure metrics have to be split into two layers. Cell assignment accuracy measures whether text landed in the wrong cell; structure reconstruction accuracy measures whether merge relations and row/column counts are right. In practice the latter predicts downstream failures better than the former: many misalignments come from matching cell counts with wrong row-to-column mapping, which cell assignment accuracy alone will never reveal. Cross-page continuation should be judged by both continuation accuracy and erroneous-continuation rate, and the latter matters more — forcibly merging two unrelated tables yields a dataset that looks complete and is entirely wrong.

9.3 End-to-end metrics: answer questions, do not diff characters

End-to-end metrics measure "can this document support the business question." There are several forms: correctness on QA pairs (given a question and a reference answer, check whether the system output contains the correct answer), field-level precision and recall on field extraction (for structured business fields), and relevance metrics on the retrieval side (for a given query, is the correct chunk recalled, and how is it ranked). These three serve different purposes and cannot replace one another. A pipeline with very high extraction accuracy may retrieve nothing useful at all because of how it chunks; a pipeline with very high QA accuracy may simply be performing well on a small set of easy documents.

How you construct end-to-end metrics matters just as much. Pairing multiple questions of differing difficulty with the same document discriminates between strong and weak systems far better than one question per document; asking only factual questions rewards systems that "will answer anything," so unanswerable questions must be included to test refusal ability; and for numeric content, questions should be designed so that answering requires computation (for example "by how much did the third quarter grow compared with the second"), because such questions expose recognition errors and extraction errors at the same time — an error that would be masked by merely asking "what was the third-quarter value" if the model fills it in from memory or from context.

9.4 Why character-level similarity is a weak metric

Evaluating document parsing with character-level metrics like edit distance or BLEU is, in essence, optimizing against an objective function that does not fit the problem. Such a metric scores "the table's columns are in the wrong order but every character is present" as a high score, and scores "a table was correctly converted into structured records" as a low score, because it cannot handle structural differences. Worse, it pushes optimization in the wrong direction: teams end up optimizing the stages that affect character similarity (such as glyph recognition) while ignoring the structural stages that actually drive the business. To measure whether the document was "read correctly," what you should be measuring is structure and semantics, not the closeness of character sequences.

9.5 A workable evaluation dashboard

If I were standing up a minimal evaluation system, I would build it in this order: first the end-to-end QA set and the retrieval relevance set, because those two map directly to business value; then layout-level metrics as a diagnostic panel, to answer "this end-to-end score dropped — which stage did it drop in"; and finally a confidence calibration assessment, meaning whether the confidence the model reports actually agrees with its real accuracy. Confidence calibration is routinely ignored and yet extremely important — if confidence is not calibrated, you cannot use it for degradation routing, for prioritizing manual review, or for on-demand sampling, and the entire confidence-based engineering design is built on sand. A rough observation: the same model's confidence distribution differs a great deal across document types, and is typically noticeably inflated on handwriting and low-resolution input, which is exactly the portion the manual review queue should handle first.

X. Engineering It: Throughput, Retries, Caching, and Cost

10.1 Parallelize by page, not by file

The most easily overlooked parallel design problem is the granularity at which you split the work. Document-level parallelism looks the most natural, but the costs are obvious: a hundred-page report is held by one task and you wait for it to finish before you get results; any single page failing means re-running the whole document; and when task durations vary widely, the load is badly imbalanced. Page-level parallelism is finer-grained, naturally suited to parallel scheduling and failure isolation, and it supports streaming results out. The only extra work it creates is cross-page continuation, because page-level parallelism means the continuation logic has to live in a higher-level coordinator that collects every page's intermediate results before merging. That coordinator's implementation complexity is manageable, because it depends only on two lightweight features — "table region candidates" and "header fingerprints" — and never needs to see the whole document's content.

The real constraint in scheduling is often not CPU but the rate limits of external model services. So concurrency control has to be layered: local tasks such as geometry and recognition can be scaled by CPU core count plus some headroom; tasks that call vision models must be rate-limited by the service quota, and need independent failure retries with backoff. Put the two classes into separate queues rather than sharing one pool, because their resource profiles are completely different — mixed together, local tasks end up throttled by an external service's quota, or local tasks consume all process memory and cause model calls to fail.

10.2 Idempotence, caching, and incremental updates

Re-running a document pipeline is expensive, so idempotence and caching have to be designed from the start, otherwise the cost of retrofitting them later is severe. There are three baseline requirements. First, content addressing: use the file hash plus the parse configuration version jointly as the processing key, so that a configuration change automatically triggers a full recompute while a plain failure retry hits the cache. Second, stage-level caching: persist each stage's intermediate result separately with its own version number, so that swapping the recognition model only re-runs everything after recognition, and changing a layout rule only re-runs everything after layout. Third, stable output: identical inputs must produce byte-identical outputs, avoiding the situation where traversal order or timestamps cause the cache to never hit.

The hard part of incremental update is that "the document has changed" is not a simple judgment. A document's modification time may change because of a format conversion while its content did not change; a document may have only its revision date updated in one place while its conclusions, amounts, and headers have all changed. The sound approach is layered comparison: compare content hashes first and reuse immediately on a hit; on a miss, run a lightweight diff to localize the change (for example comparing render fingerprints page by page) and re-run only the changed pages and their neighbors; and if the document structure is complex (say it involves cross-page tables), widen the scope to the whole related section rather than re-running only that one page. The payoff of incremental updates is substantial for documents that change periodically — but to get it right you first have to have stage-level caching in place, because otherwise there is no way to know which stages can be skipped.

CREATE TABLE parse_stage_cache(
doc_hash       text NOT NULL,
pipeline_ver   text NOT NULL,
page_no        integer NOT NULL,
stage          text NOT NULL,
stage_ver      text NOT NULL,
payload        bytea,
conf_avg       real,
degraded       boolean DEFAULT false,
primary key(doc_hash, pipeline_ver, page_no, stage, stage_ver));

CREATE TABLE parse_doc_state(
doc_hash       text primary key,
source_version text,
doc_version    integer,
parse_ver      text,
status         text,
updated_at     timestamptz);
--重跑前先按页比对渲染指纹，只重算变化页及其邻居

10.3 Failure retries: separate retryable from non-retryable

Treating every failure as retryable is a common form of waste. Retryable failures include external service timeouts, rate limiting, and temporary resource exhaustion; these should be retried with exponential backoff and a maximum attempt count. Non-retryable failures include corrupted images, unparseable formats, and empty page content; retrying them any number of times will not succeed, so they should be marked failed immediately and routed to an exception queue. There is a third, most troublesome category: failures that "succeeded on the surface but produced untrustworthy results," such as a recognition engine returning empty text without raising an error, or a model returning elements that are structurally legal but semantically empty. Retries cannot solve these; only output validation can catch them and trigger degradation or human intervention. Retry policy therefore has to be paired with output validation: retries solve "it did not finish," and validation solves "it finished but the result is unusable."

10.4 Cost control: tiered scheduling by page

The most effective cost control lever is not compressing per-page cost but letting different pages travel different-cost paths. A workable tiering looks like this: pages the quality gate judges clean and with simple layouts run the full geometric pipeline plus local recognition, never calling a vision model and costing essentially nothing at the margin; pages with complex layouts but clean images skip the geometric pipeline and call the vision model on the raw image, which is expensive but accurate; degraded pages get one round of geometric preprocessing to improve usability, and only then decide which route to take; and regardless of route, high-value pages (contract cover pages, financial statement pages, clause-dense pages) should be marked high priority and processed with full redundancy, such as dual-path parallelism plus human confirmation.

The caching strategy has to match the tiering. Geometry-plus-recognition results on clean pages are highly reusable and should be cached long-term; vision-model call results have relatively lower caching value (because they are expensive, so cache them even if the hit rate is not high), but they need a short validity period so that template changes do not keep returning stale results. Another frequently overlooked cost item is the storage of logs and debugging artifacts: rendered page images, intermediate masks, and annotated bounding-box visualizations easily become the largest share of storage, they have no value for the online service, and they should be cleaned up on a retention schedule after sampled quality checks, keeping only a small but representative subset.

XI. Progress and Capability Boundaries in 2026

11.1 The real directions of progress in vision document models

Setting aside specific product marketing, if you watch how vision document models have evolved you can see several clear directions. The first is document-specific training: these models are trained specifically on layout-dense and table-dense corpora, which makes them more reliable than general-purpose vision-language models on table output and reading order. The second is scheduling of resolution and visual token budget: rather than brute-forcing input resolution upward, the model adaptively decides which regions to look at and at what magnification. That direction fits document scenarios extremely well, because within one page the table region needs high magnification while the header and footer need almost none. The third is native document input: some models accept document pages directly instead of first rendering them to bitmaps, thereby preserving the information in the vector text layer — an advantage when handling hairline table rules and mathematical formulas. The fourth is deep integration with structured-output constrained decoding, so that model output can be consumed downstream directly without fragile regex parsing.

There is one more direction that tends to be underestimated: document-level rather than page-level input. I covered earlier the limits of single-page understanding, and the recent approach is to feed a section or a set of pages as input so the model can maintain structural consistency over a longer context. This alleviates cross-page continuation and section numbering continuity, but it places real demands on cost and context length, and in practice it is usually enabled only for high-value documents.

11.2 Capability boundaries: when you still should not hand it to the model

Despite clear progress, the boundaries of several task categories remain sharp, and they are not things to hand entirely to the model. The first is bulk extraction requiring exact values, especially financial statements and receipts, where being wrong by a single digit is very costly; the model can help with localization and cross-checking, but the value that finally enters the database should come from a structured path with geometric grounding and explicit units. The second is any process that must be reproducible and auditable: a model's intermediate reasoning is not reproducible, so once an output is disputed there is no way to produce evidence. The third is extreme layouts — handwritten historical archives, badly warped scans of old bound volumes — where geometric pipeline failures are predictable and repairable, while model failures here are usually silent.

Another boundary is poorly formatted semi-structured content, such as extracting task lists out of chat screenshots, forwarded emails, or social media screenshots. Such inputs have no stable layout, traditional pipelines have nothing to work with, and the model is genuinely good at them — but that belongs to a different problem domain (UI understanding and information extraction), whose evaluation methods and failure modes are entirely different from document parsing, so do not force it into the same pipeline assumptions. The test is simple: if the notion of a "page" does not exist, and elements are grouped visually rather than by typographic structure, then it is not a document understanding problem.

11.3 Where constrained decoding and document parsing converge

Over these past years structured-output constrained decoding has moved from "produce a syntactically valid format" toward "produce a structurally self-consistent result," and this convergence matters for document parsing because it shifts part of the quality guarantee out of offline validation and into the decoding process itself. A document model can first predict the type and hierarchy of a region, then generate content within that region, so structure and content are no longer two independent steps. In theory this reduces "correct format but misplaced content" errors, because it couples the assignment of content to positions with text generation. Still, stay clear-headed about this: constrained decoding guarantees conformance to a structural definition, not conformance to reality. It ensures output is always parseable; it cannot guarantee the facts are right. Treating constrained decoding as a quality-assurance mechanism rather than a formatting mechanism is the cognitive bias most worth avoiding when using this technology.

11.4 How to evaluate when you swap the model

A model upgrade is usually the change with the highest payoff and the highest risk in such a system, because every downstream metric moves — and the reason for the move is implicit. The sound approach is this: run both the old and the new model on a fixed evaluation set, compare end-to-end metrics against the stratified diagnostic metrics, and do not just look at whether the total went up; watch carefully for capability migration, such as a higher total alongside a large regression on the handwriting tier, which is completely invisible in the total; check whether confidence calibration still holds, because a new model's confidence distribution is usually entirely different from the old one and reusing the old thresholds blindly will invalidate all the routing logic; and finally run a sampled blind human evaluation, because automatic metrics are often insensitive to newly introduced failure modes of the "confidently wrong" variety.

XII. The Human Review Loop: Making Errors Discoverable and Attributable

12.1 How to order the review queue

Human review is part of the document pipeline, not an add-on. But "throw everything uncertain at a human" is not workable; confidence scores and degradation markers have to keep the review volume within an operable range. The ordering priority for the review queue is usually determined by several signals: lowest confidence first; anything processed through a degraded path next (because degradation itself means the main path failed); anything touching high-value fields (amounts, identifiers, dates) next; and anything downstream has already questioned goes first — this last signal is the most valuable, because it represents real business impact rather than the model's own doubt. The remainder is filled in by random sampling, used to estimate overall accuracy. Random sampling and targeted review must be reported separately, and the gap between their accuracy rates is itself a measurement of quality problems.

12.2 The review UI determines review efficiency

The design of the review UI directly determines unit review cost. The minimum requirement is three things: side-by-side display of the original image and the parse result, with clickable linkage; highlighting and jumping to the region in question; and the ability for corrections to flow straight back as new annotation data. With a well-designed review UI, a reviewer can process a page in less than half the time it takes just to look at the image — and review speed directly determines how many samples you can cover. The annotations that review produces are also a continuous source of evaluation data and the direct corpus for fine-tuning layout and recognition models. If those annotations do not flow back structurally into the training and evaluation workflow, human review degenerates into an expensive log viewer. Treating "structural feedback" as a first-class requirement when designing the review UI is far less effort than trying to retrofit it later.

12.3 Drive optimization priority with an error taxonomy

The most valuable use of review data is not computing an accuracy rate but attributing and ranking errors. I recommend maintaining an error taxonomy, filing every error confirmed in review into a category such as geometric misalignment, character misrecognition, type misclassification, cross-page continuation errors, chart numeric errors, or model hallucinated completions. For each category, record the scope of impact (how many documents are involved, which templates). Optimization priority should be ordered by "blast radius × cost of fixing that one category," not by current frequency. There is a counterintuitive conclusion here: an issue that appears only on one specific template but can be fixed by adding a single rule may deserve to be tackled before a recognition issue that appears across the whole site but requires re-annotating training data. Re-ranking this priority table periodically improves the actual user experience more sustainably than chasing average accuracy.

XIII. A Pre-Launch Checklist and Common Failure Triage

13.1 A checklist you can actually use

Checking in this order blocks most of the rework that otherwise shows up after launch. Parsing layer: is the type and coordinates of every element preserved; does cross-page table continuation have logic and test cases covering both the "continuation" and the "erroneous continuation" scenarios; are captions bound to their figures rather than merged into body text; and does header/footer stripping have cross-page consistency verification (header text stripped out of the same document should repeat heavily — if different pages strip out different content, the rules are unstable). Structure layer: do tables preserve merge-span information rather than flattening it; are values unit-normalized; is the totals-row check enforced as a gate? Output layer: are low-confidence and degraded regions explicitly marked; are empty results distinguished from failures; and do two runs on the same input produce exactly identical output?

Retrieval layer: do chunks carry their section path; do table chunks carry the expanded header; is the chunk length distribution sane (if the vast majority of chunks are all very short or all very long, the chunking logic is wrong); and is numeric content stored in two representations? Cost layer: is scheduling tiered by page; is stage-level caching in place; are rate limiting and backoff configured; and is a retention period set for logs and intermediate artifacts? Operations layer: can you reproduce any individual parse result from a document hash plus a configuration version; can you re-run only selected pages; and is there a regression evaluation process for model version changes?

13.2 Triage paths for common failures

A few high-frequency failures have fairly fixed investigation paths. "Retrieval recalled a semantically related but entirely wrong chunk": look first at that chunk's type label and confidence; if low-confidence chunks got into the index, it is a filtering rule problem; if the chunk's type is caption or header but it was treated as body text, it is an element classification or attribution problem. "Field extraction keeps getting a certain number wrong": first confirm whether the column names of the table containing that number were expanded correctly; if all column names are right, then check the merge span of that cell; if the column names are misaligned, it is a table structure reconstruction problem, not an extraction problem — and following this order saves a great deal of pointless fiddling with model prompts. "An entire document comes out in scrambled order": first check whether it is a two-column or multi-column layout, then check whether rotation correction took effect, and only then suspect the model. "Only certain customers' historical documents parse badly": you can essentially go straight to the degraded-quality path, configuring a separate degradation strategy and manual review ratio for this class of input.

13.3 When to change architecture instead of tuning parameters

There are three situations in which you should stop and consider an architectural adjustment. First, the same class of error recurs and cannot be fixed with rules or data, which means the current representation cannot carry that information and the output structure has to change. Second, optimizing one class of document visibly degrades another, which means the single existing representation conflicts across layouts, and you should consider splitting the pipeline by document type. Third, end-to-end metrics plateau while every per-stage diagnostic looks normal — this usually means the problem is a cross-stage interaction (recognition correct but order wrong, or order correct but chunking shatters the semantics), and you should go look at the intermediate artifacts rather than keep optimizing any single stage. The most important single test is this: if every optimization requires changes in several places that then interact with each other, it is time to rethink the design of the intermediate representation.

Finally, back to the opening line: the difficulty in document understanding is not the model, it is the contract. You have to be able to state clearly which structured facts exist in a document, what the coordinates and trustworthiness of each fact are, and how the system behaves when parsing fails. Define those three things clearly, and everything else is just engineering effort.

XIV. Key-Value Extraction: Forms, Receipts, and Contract Clauses

14.1 Three layout families for key-value extraction

Alongside tables and charts, key-value extraction is the highest-frequency document understanding task in the enterprise: buyer and seller on an invoice, amount and reason on an expense claim, party information and subject matter on a contract cover page, applicant field and purpose on an application form. Their layouts fall roughly into three families with completely different extraction strategies. The first is the right-aligned label type: the label sits on the left, the value on the right, separated by whitespace or an underline; once you recognize the label text, you simply look to the right by geometric relation to find the value. This is the simplest and most reliable family. The second is the inline label type: the value follows the label after a colon or a list separator, sometimes with several key-value pairs on one line, so you split on punctuation and then use language and position to assign ownership. The third is the boxed and nested type: the value sits inside a bordered cell, and the label may be outside the box or drawn inside it — this is where "value treated as label" and "label treated as value" swaps are most likely to occur.

14.2 Why key-value extraction is especially prone to swap errors

Swap errors are the most typical and hardest-to-detect failure of key-value extraction, because the extracted fields are usually all well-formed: a date landed in a date field, an amount landed in an amount field — only the owning entity is wrong. Swap errors typically arise from overly permissive geometric reasoning. The assumption that "within a line, sort all text blocks by x, and the left one is the label while the right one is the value" fails outright when a line holds two key-value pairs; and when a wide gap separates label from value and exceeds the intended distance threshold, the value goes unclaimed and degrades into "the next orphan text block." The effective countermeasures come in three layers: first, anchor on a label vocabulary (match known labels by regex or dictionary, then claim the nearest value geometrically), and explicitly mark a label whose value could not be claimed as empty rather than grabbing something arbitrary; second, type-check claimed values (date format, amount format, ID-number check digit), and treat a claim whose type does not match as a failure; third, run a duplicate-key check over the whole line, keeping both values and flagging a conflict when the same field appears twice in a document, instead of silently taking the first.

14.3 For long text like contract clauses, the hard part is locating, not extracting

Extracting contract clauses looks simple — find the section on "liability for breach" and pull the text out of it — but in implementation there is a common structural misconception: what is actually hard is usually not "which passage to extract" but "what counts as a passage." A contract clause typically consists of a heading, several paragraphs of body text, several sub-items, and several exception notes. The heading hierarchy is expressed through numbering and indentation but may have no visual font difference at all. Splitting crudely on blank lines cuts one logical clause into three; splitting on fixed length mixes the exception notes in with the main clause. The sound approach is to use numbering patterns and indentation levels to rebuild a clause tree, and number recognition in contract-style documents is usually more reliable than layout detection, because numbering comes from disciplined typesetting. When numbering is absent, indentation depth and punctuation patterns (opening connectives such as "provided that" or "except as otherwise agreed in this clause") can serve as supplementary signals.

14.4 Receipts and forms: rules beat models

Invoices, train tickets, ID cards, and business licenses — highly standardized instruments — are where traditional rules have the best cost-performance ratio. The reason is simple: the layout is nearly fixed, field positions are predictable, and regulatory or industry standards often impose explicit layout requirements. The most effective approach here is to anchor the layout first (locating the instrument type with geometry or a small number of keypoint detections), then extract fields with template rules, using the model only as a fallback when the rules fail and for generalizing to non-standard variants. By contrast, free-layout extraction starting from zero really is more dependent on the model. Running these two classes of input through one pipeline with the same strategy is a common design mistake.

There is also a strong validation opportunity in instrument scenarios that tends to be overlooked: different instrument types impose arithmetic constraints on amounts. The tax-inclusive total should equal the sum of the pre-tax amount and the tax amount, and check digits can be recomputed to verify. Such checks catch a large volume of recognition and extraction errors without depending on any model at all, cost almost nothing, and produce very few false positives. Contract documents likewise carry cross-field constraints usable as gates — for example, the signing date should not precede the effective date, and the uppercase and lowercase forms of an amount should agree. Writing this kind of business constraint out as a validator suite is the highest-return single step in improving pipeline reliability, and it is usually far more effective than swapping models.

XV. The Data Contract: Making Parse Results Long-Term Consumable

15.1 Parse results are a data asset and need versioning

Once a parse artifact has entered a downstream index or a data warehouse, it has become a data asset rather than some pipeline's temporary output. That means that as soon as it is published you have to think about versioning: a change in the meaning of a schema field (a field goes from string to object), a deprecated field, a newly added field, or an extended enum all affect every consumer. Without version management, upgrading the pipeline means re-running the entire corpus and hoping nothing downstream breaks as a result. So from day one, manage "structural version" and "content version" separately: the structural version increments when the format definition changes, the content version increments when the parse result updates, and the two are independent. That way "this re-run only fixed two field values" and "this re-run happened because of a structural upgrade" become clearly distinguishable.

15.2 Field definitions must state provenance and confidence

Every externally exposed field should carry three things: where in the document it came from (page number plus coordinates, or a block identifier), how it was obtained (geometric rules, recognition plus rules, or model extraction), and its confidence tier. The confidence tier does not have to be very precise, but it should at least distinguish "has geometric grounding and passed business validation" from "model-generated and unvalidated." These three pieces cost very little, yet they determine the debugging path when something goes wrong downstream. A field that carries only a value leaves you unable to judge whether to trust it; a field with provenance and confidence lets you locate the problem region in seconds. Treating such metadata as a first-class field rather than something tucked into an auxiliary log is the dividing line between parse artifacts that remain usable over time and ones that do not.

15.3 Migration strategy for structural change

When the structure changes, the sound approach is dual-write plus gradual migration rather than a one-shot cutover. Concretely: a newly added field is write-only at first, and the read-side logic switches over only after it handles both the old and new structures; when a field's semantics change, do not redefine them in place — add a new field, dual-write during a transition period, and deprecate the old field only after all downstream consumers have switched; extending an enum is a low-risk change that can ship directly, but you still need downstream to confirm it tolerates unknown values appearing, otherwise it will trigger parse failures. Deprecating a field must wait until all downstream consumers confirm they no longer depend on it, and you must keep a sufficiently long rollback window. This process looks cumbersome, but it turns "one change blows up all downstream consumers" — a high-risk operation — into a series of small, observable, rollbackable operations.

15.4 The external interface should expose uncertainty

Many systems design the parse interface as "give a value, get a value," hiding uncertainty completely. The problem with such an interface is that it pushes the entire decision about whether human intervention is needed onto downstream, and downstream does not understand how document quality is tiered, so it ends up either trusting everything or trusting nothing. A more reasonable interface exposes low confidence, degradation, and failure as first-class return states, and lets callers set thresholds according to their own risk appetite. That way one parsing service can support both "bulk-load the database, a small number of errors is tolerable" and "compliance archiving, errors must be blocked" — two completely different scenarios — without maintaining two separate systems.

XVI. After Recall: Reranking, Deduplication, and Context Assembly

16.1 Recalled does not mean usable

Hybrid retrieval produces a candidate set, and getting from candidates to something usable takes three more steps that have to be designed explicitly. The first is reranking: use a stronger model such as a cross-encoder to score candidate pairs in fine granularity and push the genuinely relevant chunks to the front. In document scenarios the payoff from reranking is usually obvious, because many relevant chunks land outside the top ten during dual-vector recall due to chunk granularity or wording differences, and reranking can pull them back up. The second is deduplication: adjacent paragraphs of the same document are often highly similar in content and will occupy multiple recall slots, so use text fingerprints or semantic similarity to drop redundancy and free positions for chunks from different sources. The third is context assembly: concatenate in score order while respecting the context length budget and structural constraints.

16.2 Three decisions inside context assembly

The most consequential decision in the assembly stage is "do we supply adjacent content." When a user asks for a specific number, giving only the hit chunk leads to follow-up questions that often need neighboring context; but automatically padding in adjacent chunks eats into the budget. There is a practical compromise: supply on demand — only add adjacent content when the hit chunk is obviously incomplete (a table chunk missing its header, a paragraph chunk beginning with a pronoun, no section path), and otherwise do not. This decision can be made automatically from chunk metadata, with no model involvement.

The second decision is the ordering of chunks from the same document. If multiple chunks were recalled from one document, they should be concatenated in their in-document positional order rather than by relevance, because humans read documents in order too, and scrambled order makes the model misjudge relations between paragraphs. The third decision is how to handle conflicting content: when different documents give different answers to the same question, do not let ranking decide and let the model choose on its own — explicitly keep multiple sources with annotations, and have the model present both along with their respective sources when answering. Otherwise the generated answer will sound certain while actually being unreliable. These three points are the main source of "it can recall but it answers badly" in many RAG systems.

16.3 Citations and traceability

Citation information should be prepared at assembly time: give every chunk the document title, page number, section path, and coordinates, write them into the chunk's hidden fields, and require the model to cite them when generating in the prompt template. A common practical problem is unstable citation formatting, with the model occasionally fabricating page numbers. A sturdier approach is to stop letting the model write page numbers at all: pre-assign citation identifiers and place them at the start of each chunk, so the model only has to cite an identifier. This is the constrained-decoding idea applied once more to the citation problem — turning "generate a citation" from free generation into selecting an existing identifier, which substantially improves reliability.

16.4 Offline evaluation of the recall layer is worth more than evaluation of the generation layer

If you can only do one thing, I recommend starting with offline evaluation of the recall layer: prepare a batch of real queries, annotate the correct chunk for each, and continuously monitor recall rate and ranking metrics. The reason is that problems in the recall layer can be localized, quantified, and improved along a clear path, whereas generation-layer problems are often just "the answer is wrong" with no way to tell whether recall missed or the model failed to read. Monitoring recall-layer metrics over the long term also yields an extra benefit: when a new document template or a new parse pipeline version is rolled out, you see the change in retrieval quality immediately, without waiting for a business user to report that "search seems inaccurate lately."

XVII. Multilingual Documents: The Double Challenge of Recognition and Retrieval

17.1 Document-level language versus block-level language

The first difficulty with multilingual documents is the granularity of language identification. A document that is entirely in one language is relatively easy to handle, but what is far more common in real corpora is a mismatch between document-level and block-level language: English terms and code snippets embedded in a Chinese report, a Chinese signature page attached to an English contract, an English product model number inside a Japanese document. Handing a whole page to a recognition engine built on a monolingual assumption then produces visually similar characters substituted for one another, and such errors are extremely hard to detect downstream, because the resulting text still reads coherently. The pragmatic approach is to cut out text blocks at the layout stage first, then classify language at the block level, route mixed-language blocks separately to the corresponding recognition channel, and keep a language tag in the output. The language tag looks like a mere auxiliary field, but it determines the tokenization, retrieval, and even summarization strategies that follow; without it, every later step can only fall back on one averaged-out strategy for wildly different content.

17.2 Language mismatch on the retrieval side

When the document language and the query language differ, pure vector retrieval still works (multilingual embedding models map semantically similar cross-lingual text to nearby positions), but keyword retrieval almost completely fails — a user searching in Chinese for "liability for breach" gets zero term matches from BM25 against an English contract. There are two remediation paths here. The first is query-side expansion: use translation or multilingual rewriting to expand the query into several language variants, recall each separately, and fuse the results, at the cost of extra encoding calls. The second is index-side handling: store an additional machine-translated proxy text per document for keyword indexing, while keeping the original for display and citation. Each path has a cost: the former is cheap but depends on translation quality, while the latter significantly inflates index size and cost on multilingual corpora. Which one to choose depends on your query language distribution — if queries concentrate in a few languages, query-side expansion is the better deal; if document languages are extremely scattered, index-side handling is more stable.

17.3 Adapting retrieval for identifiers, codes, and proper nouns

Finally, a class of content deserves a separate warning: identifiers, model numbers, clause numbers, statutory citations, email addresses, and URLs. What these have in common is that users almost always search for them exactly, while vector retrieval is basically ineffective on them and sometimes actively harmful — because "Article 3.3" and "Article 3.5" nearly overlap in semantic space while being completely different in fact. For this class of content, the keyword channel is the only reliable one. Three engineering recommendations follow. At ingestion, preserve the original written form — do not normalize identifiers into a tidier format, because users' queries use the original form. Do not tokenize identifier-like tokens too aggressively, and do not filter them out as stopwords or low-frequency terms. When chunking, do not separate an identifier from the thing it modifies: in a phrase like "in accordance with GB/T XXXX," the identifier and the standard name must stay in the same chunk, otherwise exact matching will hit half the content while losing the meaning.

There is a companion retrieval-side problem for this kind of content: what to display after a hit. When a user searches for an identifier, what they usually want is the context in which it appears rather than an entire chapter, so when the main value of a hit chunk is that it provides a locating coordinate, the interface should return the coordinate directly instead of stuffing the whole passage into the context and letting the upper layer jump to the right spot in the source document itself. Recognition and retrieval meet again here: the coordinates preserved at the layout stage ultimately become traceability inside the retrieval result.