1. Lay out the whole data flow first
Most people first meet a vision-language model and think of it as a chat model that can see images. That impression is both useful and harmful. It is useful because the external interface really is the same. It is harmful because it hides the three places in the chain where things actually go wrong: how an image becomes tokens, how tokens are read by the language model as words, and which parameters are actually moving during training. This article walks the chain in that order, and every section ends in one concrete engineering judgement instead of a glossary of names.
A full inference pass is wired together roughly like this: image pixels go through a vision encoder and produce a set of vectors; those vectors pass through a projection layer (also commonly called a projector, connector, or resampler) which maps them into the language model’s word embedding space; the vectors are then treated as visual tokens, concatenated with the text tokens into one sequence, and fed into the autoregressive decoder; the decoder generates the answer token by token exactly the way it handles ordinary words. Nowhere in this process is there a switching between an image branch and a language branch — there is only a single move from the visual coordinate system to the language coordinate system.
The three interfaces that genuinely require design decisions in this chain. The first interface is what shape of vector the vision encoder outputs: resolution, patch size, whether hierarchical features are kept, and whether the vision tower itself has been pretrained with image-text contrastive learning. The second interface is how many tokens the projection layer keeps, that is, how wide the information bottleneck is opened. The third interface is who is being trained, and it directly decides what the model can learn and what it will throw away.
What matters here is that these three interfaces are not independently tunable. Raise the input resolution from 224 to 672 and the token count grows by close to nine times; the compute and activation memory of the projection layer rise with it, and so does the effective sequence length during training. At that point you usually have to change the projection layer structure and the training mix in step, otherwise you easily end up with a gap where the training loss looks perfectly normal but at inference time a single sentence spliced in front of several thousand visual tokens runs past the context length or blows up memory.
Another common misconception is treating the vision encoder as a pure feature extractor that merely spits out vectors. Reality is different: if the vision tower itself has never been pretrained for image-text alignment, its output space is purely visual and its semantic axis has almost nothing to do with the language model’s semantic axis. At that point, no matter how large the projection layer is, it can only perform a limited amount of transport rather than translation — it cannot invent correspondences that do not exist in the training data. This is the premise for understanding every training stage that follows.
This article tries to state each judgement explicitly rather than handing you a term-by-term glossary. Wherever a specific paper index, a specific benchmark score, or a specific ablation number is involved, I describe only the mechanism itself, because those details vary enormously between implementations and hard-coding them tends to mislead readers.
2. Choosing a vision encoder: what CNN and ViT are actually competing over
The history of vision encoders is far longer than that of large language models. In the image classification era the main line was CNN: from early networks to the residual family, and on to modern convolutional designs such as EfficientNet and ConvNeXt. ViT entered halfway, replacing convolution with patch splitting plus global self-attention, and was for a while assumed to make convolution obsolete. But in the vision-language model setting the conclusion is not that one side simply wins — hierarchical and flat architectures each occupy their own ecological niche.
2.1 The effective range of convolutional backbones
The strength of convolution is locality and efficiency. Convolutional kernels share parameters and are extremely sensitive to high-frequency detail, so signals such as small text, thin strokes, and texture are captured very solidly; the parameter count is small, memory footprint is low, and inference is fast. In on-device deployment, real-time video streams, and scenarios demanding extremely low latency, a convolutional backbone is still the most economical choice. That is a purely economic judgement, not technological backwardness.
The weaknesses of convolution are just as obvious. Growing the receptive field depends on stacking layers, and deep semantic extraction is less efficient than self-attention; more importantly, convolution naturally outputs multi-scale hierarchical features, while a language model’s input is a flat string of tokens, and mapping hierarchical features onto a token sequence is not natural — it usually requires an extra fusion module.
In real projects you rarely see a purely convolutional vision tower. What is more common is a hybrid: a convolutional backbone handles early-stage detail capture, followed by one or two layers of self-attention for global fusion. This structure is especially common on inputs such as documents, tables, and UI screenshots, because convolution responds to fine-grained text edges more sharply than self-attention with a fixed patch split.
2.2 Four directions of modification in the self-attention family
The first direction replaces global attention with window attention or approximate linear attention, aiming to bring complexity down from quadratic to near-linear while making high resolution feasible in terms of memory. The second direction changes how patches are embedded — for example using convolution for patching, or performing one round of pixel rearrangement and aggregation before the input enters the backbone, turning the input into fewer but wider tokens. The third direction is adapting the positional encoding, because interpolated absolute position embeddings tend to break down when the resolution changes drastically, and interpolatable relative position bias or 2-D rotary embeddings usually have to take over. The fourth direction decouples training resolution from inference resolution, that is, doing most of the pretraining at a lower resolution and continuing training at the target high resolution.
These four directions are essentially orthogonal in engineering terms, and in practice two or three of them are often used together. Combining low-resolution pretraining with crop-style fine-tuning is currently the least-effort way to serve many input formats from a single vision tower.
On whether the vision tower should keep hierarchical features, there is a disagreement that is often overlooked yet has a large impact. Taking only the last layer gives highly semantic features, few tokens, and low overhead, but spatial detail is lost badly — text, coordinates, and small objects blur into a single blob. Taking the last layer plus several intermediate layers recovers part of that detail, at the cost of multiplying the token count, and the semantic granularity of different layers is inconsistent, so they must be projected separately and then concatenated or fused with weights.
2.3 The split between hierarchical and single-layer features
My practical judgement is this: if the input is mostly natural scenes — recognition, captioning, general question answering — single-layer features plus enough resolution are usually sufficient, and the gain from hierarchical features does not pay for doubling the tokens. If the input is mostly text-dense images — receipts, papers, screenshots, tables, UI — you almost have to take the multi-layer feature path. The last-layer feature has already smoothed away stroke-level information, and no matter how strong the projection layer is or how high the resolution goes, that cannot be recovered; multi-layer features preserve information that will never be available again later on.
| Vision tower form | Strengths | Costs | Better suited to |
|---|---|---|---|
| Convolutional backbone with late-stage attention | Sharp detail, high throughput, memory-efficient | Weak global modelling, needs an extra fusion module | On-device, real-time, document detail |
| Pure self-attention with single-layer features | Strong global modelling, few tokens, scalable | Weak detail, steep cost at high resolution | Natural scenes, general QA |
| Self-attention with multi-layer feature fusion | Balances detail and semantics | Token count multiplies, projection structure complex | Documents, tables, charts, grounding |
| Low-resolution pretraining plus high-resolution continued training | One set of weights serves many input formats | Continued-training data and compute cost | General systems with heterogeneous inputs |
2.4 A quick selection guide
Condensed into four sentences: for general visual question answering and stable throughput, initialise the vision tower with mature open-source image-text aligned weights — least effort and least likely to produce structural errors. For document and UI understanding, pick a vision tower that has been further trained on OCR-heavy supervised data, and definitely go through multi-layer features. For real-time or on-device work, return to the convolutional family and accept a higher token count. For maximum accuracy with plenty of compute, go straight to high-resolution self-attention plus multi-layer features, and prepare for a more expensive serving setup.
One more reminder: the provenance of the vision tower weights affects the final result more than architectural details do. The same self-attention backbone initialised with general image-text aligned weights and initialised with pure image self-supervised weights differ enormously in convergence speed and alignment difficulty. When compute is tight, spending budget on a better set of vision tower initialisation weights is usually a better deal than enlarging the language model.
2.5 Resolution adaptation of the vision tower
The input size a vision tower was trained with is often not the size it is used with, and that is something every multi-format input system has to handle. The most common approach is to use schemes that can be interpolated by resolution, such as 2-D rotary embeddings, rather than absolute position embeddings on a fixed grid; the former accepts non-square inputs without retraining, while the latter goes out of alignment as soon as the resolution changes, and the model quietly outputs results that look plausible but are entirely mislocated.
Another easily overlooked point is the coupling between receptive field and size. The effective receptive field of a self-attention backbone shrinks in relative terms as the input grows: the same window attention configuration can cover the whole image at low resolution but sees only a local region at high resolution. If you do not adjust the window size or the number of attention layers accordingly, global understanding degrades noticeably on high-resolution inputs, and the degradation shows up as narrower answer scope and no cross-region reasoning — which is easily misdiagnosed as a data problem or a model capability problem.
There is also a compromise path worth remembering: keep the vision tower internally at low resolution to control compute, and perform a single spatial upsampling after the tower output but before it enters the language model. That amounts to paying the high-resolution cost only along the very short path before entering the language model — a pragmatic approach that still yields decent results when compute is limited. The price is that the recovered detail is extremely limited, which helps little for text-dense inputs, so it suits natural scenes only.
3. Resolution, visual token count, and the information bottleneck
The essence of the resolution decision is finding the balance point between seeing clearly and being able to afford it. The visual token count is roughly image height times width divided by patch area, divided again by the spatial downsampling factor. Doubling either side makes the token count four times larger and the sequence length inside the autoregressive decoder four times larger, so both attention cost and KV cache footprint step up — in long-context scenarios that cost often exceeds the model’s own compute.
More troublesome is how widely the resolution requirement varies in text-dense scenarios. A landscape photo is adequately described at a fairly low resolution, while a full page of tightly typeset scanned document becomes a blur at the same resolution. A fixed-resolution approach is guaranteed to waste something at both ends: compute wasted at one end, and outright illegibility at the other.
So the mainstream thinking today is no longer “what resolution do I use” but “how many tokens do I budget for this image”, with a policy deciding how that budget is spent. The shift sounds like a mere change of wording, but it moves the design problem from the architecture layer to the policy layer, and every dynamic-resolution scheme since then has been developed inside that framework.
There are three common budget allocation policies. The first is a fixed budget plus a thumbnail: downsample the whole image into one small view that preserves global semantics and object layout, then attach a high-resolution local view that preserves detail, with both sharing the budget. The second is dynamic tiling: run one lightweight content judgement first, cut the image along the effective region, and give few or even zero tokens to blank and solid-colour areas. The third is native dynamic resolution: cut the image directly into several strips by aspect ratio, each strip with a fixed height and a variable width, so each strip’s token count is determined by the height; a 2-D positional encoding then encodes the strip positions back in, letting the model know where that strip sits in the original image.
- Thumbnail plus local view: easiest to implement, predictable in behaviour, friendly for debugging — good for getting a system running first.
- Dynamic tiling: the most token-efficient and the most effective for layout-heavy documents and UIs, but the tiling policy is itself an independent quality bottleneck.
- Native dynamic resolution: the most expressive, with no wasted thumbnail, but it puts high demands on the positional encoding and on training data coverage, so both implementation and training need more care.
On resolution and text recognition there is one judgement that has to be stated clearly: below how many pixels of glyph height in the image does recognition start to become unreliable? This empirical threshold varies quite a bit across models, but the overall pattern is stable — below eight pixels it is basically unreliable, and twelve to sixteen pixels is the usable lower bound. Applying that standard in reverse, if a full A4 page is squeezed into an input of a few hundred pixels per side, each character probably gets only a few pixels; at that point no architecture can save you. The problem is not the model, it is that you did not give it enough pixels.
There are only two remedies: either tile the image and read it block by block so every character falls into a usable pixel range; or simply run an OCR pass first and feed the recognized text together with the original image into the model, letting textual information travel over the text channel instead of being read raw out of pixels. In practice the latter is often a better deal than piling on resolution without limit, and much more stable.
# 视觉 token 预算与实际字高估算（工程里上线前先算这个）
def estimate_tokens(h, w, patch=14, merge=2, extra_patches=0):
gh, gw = h // patch, w // patch
n = (gh * gw) // (merge * merge)
return n + extra_patches # extra_patches 用于缩略图等附加 token
def estimate_text_pixels(h, w, lines=45, chars_per_line=38):
# 满页密排文档的粗略字高估算
return h / lines
for res in (224, 336, 448, 672, 896):
side = res * res
n = estimate_tokens(res, res)
px = estimate_text_pixels(res, res)
print(f"res={res} tokens={n} text_h={px:.1f}px")
# 结论：整页密排文档想直接靠像素读，
# 输入边长至少要 1120 以上才勉强进可用区间，
# 这时 token 已经上万，必须切块。There is another performance fact that is often overlooked: the relationship between visual token count and training throughput is not linear. Once the sequence on the language model side gets longer, peak memory during training rises quickly, while the vision tower’s share of the compute actually shrinks. By the time you push the resolution high enough that the vision tower is no longer the bottleneck, you have effectively walked into a pure language-model memory problem, and the right optimization direction is sequence parallelism, gradient checkpointing, or KV cache quantization — not more tinkering with the vision tower.
4. The projection layer: from visual space to language space
What the projection layer does is convert coordinate systems. The vectors output by a vision encoder typically have a dimension of around one thousand, are strongly local, and their semantic axis leans toward vision; the hidden dimension of a language model is often in the low thousands or higher, with a completely different semantic axis. What the projection layer has to do is not simply stretch dimensions but restructure the semantic geometry of the former space into the semantic geometry of the latter. If this step is botched, no later model size can rescue it.
There are currently three mainstream designs. A linear layer is just a matrix multiplication: the fewest parameters, the highest information fidelity, and essentially no overfitting risk, but limited expressiveness. An MLP is two fully connected layers with an activation, the most common compromise. Query-based designs use a set of learnable queries that pull a fixed number of vectors out of the visual features through cross-attention, decoupling the output token count from the input token count; structurally these are often called resamplers or Q-Former-like structures.
4.1 Trade-offs among the three structures
A linear layer applies in one specific situation: when the visual features are already highly aligned — for example when the vision tower was alignment-trained on a large amount of high-quality image-text data — and the projection layer only needs to perform a coordinate transformation. In that case adding a non-linear layer brings no benefit and only increases overfitting risk. Many practical systems of modest size simply stop at two linear layers, and the results are not bad at all.
The MLP is the choice of the overwhelming majority of production systems. It adds one non-linearity over the linear layer, lifting expressiveness a clear notch, and the parameter increase stays completely under control. The one thing to watch is not to widen the hidden layer blindly, because activation memory grows linearly with the number of visual tokens: doubling the hidden dimension means doubling the memory of that stage.
The core value of the query-based design is decoupling and compression. Feed in several thousand visual tokens and you get a fixed few hundred out, which cuts memory and inference cost substantially, and the fixed output length makes batching and caching at the system layer much easier. The price is an information bottleneck. Past a certain compression ratio, tasks that need precise alignment clearly lose points — counting, coordinate grounding, reading small text. A second problem is that query vectors tend to degenerate during training into extracting only global semantics while ignoring local detail, which shows up as the model describing the overall image fluently yet failing to answer any question about a specific small detail at a specific location.
| Projection layer type | Parameter count | Information fidelity | Overfitting risk | Applicable boundary |
|---|---|---|---|---|
| Single linear layer | Minimal | Highest | Lowest | Already well-aligned vision tower, small models |
| Two-layer MLP | Moderate | High | Moderate | Default choice for production systems |
| Multi-layer MLP | Large | High | Somewhat high | Returns saturate quickly, not recommended as a default |
| Query resampler | Moderate | Bounded by compression ratio | High | High resolution, long context, memory-sensitive |
4.2 The parameter count trap
There is an easy trap here. The parameter count of the projection layer grows only linearly with the visual feature dimension, so it looks small and is easily underestimated; but its compute and activation memory grow with the number of visual tokens. A matrix going from four thousand dimensions to eight thousand is not large by itself, but if four thousand visual tokens pass through it one after another, the activation memory eats a very substantial share of the budget. So in a dynamic high-resolution system, the projection layer design must be considered together with the token budget, instead of fixing the structure first and then raising the resolution.
There is also a depth question worth separating out. Early practice distinguished deep and shallow projections: a deep projection chains several non-linear transformations, a shallow projection is a single linear layer. The repeatable conclusion is fairly consistent — past two or three layers the returns decay rapidly, while a shallow projection converges faster early in training and uses less memory. So the more common practice today is to stay at a shallow to moderate depth of one or two layers and spend all the saved compute on higher resolution and multi-layer features, which is a far better deal than deepening the projection.
import torch
import torch.nn as nn
class VisionProjector(nn.Module):
def __init__(self, vis_dim=1280, llm_dim=4096, n_layer=2):
super().__init__()
self.pre_norm = nn.LayerNorm(vis_dim) # 抑制异常大维度
layers, d = [], vis_dim
for _ in range(n_layer):
layers += [nn.Linear(d, llm_dim), nn.GELU()]
d = llm_dim
self.mlp = nn.Sequential(*layers)
self.post_norm = nn.LayerNorm(llm_dim)
def forward(self, feats):
# feats: [B, T, vis_dim]，T 为视觉 token 数
x = self.pre_norm(feats)
x = self.mlp(x)
return self.post_norm(x) # 输出维度对齐 LLM 词嵌入空间
# 高分辨率动态切块时的预算分配
def merge_budget(feature_list, budget=576):
# 多层特征按层号加权拼接，超预算时从最深层开始降采样
merged = torch.cat(feature_list, dim=-1)
if merged.shape[1] <= budget:
return merged
keep = budget // len(feature_list)
return torch.cat([f[:, :keep] for f in feature_list], dim=-1)4.3 The small detail of normalization
Whether visual features should be normalized before projection, and whether another normalization layer should follow the projection, has a significant effect on training stability, especially under half-precision training. Unnormalized visual features often contain one or two dimensions with abnormally large values, and those dimensions push the inner product in attention up to the point of overflow, which shows up as the loss suddenly turning into NaN in the first few hundred steps. The safe recipe is one normalization before the projection and one after it: both are cheap, and together they save a great deal of debugging time.
4.4 Common structural variants of the projection layer
In real engineering you will also see several variants. The most common attaches a type embedding for visual tokens after the MLP, plus a temporal or image-index embedding, to tell the model which image and which frame each token belongs to. This step can be skipped entirely for single-image tasks, but on multi-image and video tasks the model basically cannot learn cross-image and cross-frame reference relations without it.
The second variant combines the MLP with a query resampler: first use the MLP to map dimensions into the language space, then run a cross-attention extraction with a set of queries to output a fixed number of tokens. The benefit is that the problems on both ends are solved separately; the drawback is that once the chain gets longer debugging becomes difficult, and when something breaks it is hard to tell whether the dimension mapping is wrong or the compression is wrong.
The third variant changes per-position mapping into grouped mapping, for example using different small projection layers per spatial block to handle features produced at different scales. This works naturally in multi-layer feature fusion, because the statistical distributions of different layers are different to begin with, and pushing them through the same projection parameters usually needs a longer warmup before training converges.
One general piece of advice: the structural choice for the projection layer should be settled early in the project and then frozen. Do not expect to rescue results late by changing the projection layer. Once that structure moves, both the pretrained vision tower weights and the fine-tuned state of the decoder have to be re-adapted, and that cost is far higher than picking a reasonable but unimpressive structure from the start.
5. Three-stage training: what each stage is actually learning
The most common recipe today has three stages: visual alignment pretraining, instruction fine-tuning, and preference optimization. The goal, the unfrozen scope, and the data type of each stage all matter, and merging them is the most common source of performance loss.
5.1 Stage one: visual alignment pretraining
This stage trains only the projection layer and freezes both the vision tower and the language model. The goal is extremely narrow: teach the projection layer to map visual space into language space so that visual tokens become readable to the language model. The data is usually very large-scale, coarsely captioned image-text pairs.
Why train only the projection layer? Because at this point the language model’s next-token prediction ability is already formed, while the text supervision in large-scale image-text pairs is weak and extremely noisy. If you unfreeze the whole model here, the overwhelming majority of the gradient comes from the text side, the language model gets dragged along by a huge amount of coarse text first, and the visual part is drowned out before it has learned anything. Freezing both ends and letting only the connector in the middle learn is the optimal compromise in both cost and stability, and it is the highest return per unit of effort in the whole chain.
But pure freezing has a price too. If the vision tower is general contrastive-learning weights, its feature space is not sensitive to text-dense images or fine-grained relations, and the projection layer cannot conjure that information out of nothing. So the more mature practice is to gradually unfreeze the last few blocks of the vision tower late in this stage and adapt the features with a clearly smaller learning rate; paired with that, raise the share of high-resolution, text-bearing images in the image-text pairs used for alignment pretraining. The criterion is simple: if the text-related validation tasks drop rather than rise after unfreezing, the adaptation has gone too far, so fall back to unfreezing only the projector.
5.2 Stage two: instruction fine-tuning
In this stage the language model is unfrozen (usually with full parameters), the projection layer keeps training, and the vision tower stays frozen or is fine-tuned with an extremely low learning rate. The data switches from coarse captions to structured instructions: multi-turn dialogue, visual question answering, reasoning problems, text recognition, grounding, and refusal samples.
The most common failure in this stage is not convergence but inconsistent data formats. Within one dataset, some samples place the image in the user turn, some in the system prompt, and some insert a placeholder in the middle of the text; multi-image samples also differ in ordering and in separator conventions. If the formats are inconsistent, what the model learns is a positional bias rather than the ability to look at the image, and the symptom is a clear drop the moment you rephrase the question. The fix is to push all data through one unified preprocessing template — images always at the start of the user turn, or always through the same placeholder — and to manually spot-check a few hundred samples before training.
The second issue is the data mix. In instruction fine-tuning the ratio between visual data and pure text data is extremely sensitive. Too little pure text and the language ability collapses, with the model starting to emit incoherent sentences; too much and the model overfits to the short-answer style of “look at the image and speak”, hurting complex reasoning and long-form answers. In practice a visual share in the tens of percent range is fairly stable, but tune it per task: for vertical-domain question answering raise it, for a general assistant lower it and keep more pure text samples as a floor.
The third issue is long sequences and packed training. Concatenating multiple samples into one long sequence for efficient training is standard practice, but it goes wrong easily when visual tokens make up a large share: the position indices of visual tokens must be renumbered continuously after concatenation and cannot reuse the numbering from inside a single sample; and if the attention mask implementation only clips by text length, it will mask out visual tokens or let padding leak through. Both bugs present as training loss that looks normal combined with inference results that are inexplicably bad, and neither is easy to catch by staring at the loss curve.
# 三阶段训练的解冻范围与数据来源（伪代码）
STAGES = [
dict(name="align_pretrain",
trainable=["projector"], # 只训接头
data="web_captions_pretrain", # 大规模粗标注
lr_proj=1e-4, lr_vision=0.0, lr_llm=0.0),
dict(name="vision_adapt",
trainable=["projector", "vision_top_blocks"],
data="highres_text_captions", # 提高含文字高分辨率样本比例
lr_proj=1e-5, lr_vision=1e-6, lr_llm=0.0),
dict(name="sft",
trainable=["projector", "llm"], # 全参解冻语言模型
data="instruction_tuning_mix", # 多轮/推理/OCR/定位/拒答
lr_proj=1e-5, lr_vision=0.0, lr_llm=2e-5),
dict(name="preference",
trainable=["llm"],
data="vision_pref_pairs", # 幻觉对比 + 拒答 + 不确定性
lr_proj=0.0, lr_vision=0.0, lr_llm=2e-6), # 学习率压到 SFT 的十分之一
]
for st in STAGES:
for name, p in zip(["proj", "vis", "llm"], ["projector", "vision", "llm"]):
p.requires_grad_(name in st["trainable"])
set_lr(opt, p, st[f"lr_{name}"])
run(st["data"])| Stage | Unfrozen scope | Main data | What is really being learned |
|---|---|---|---|
| Alignment pretraining | Projection layer only | Massive coarsely captioned image-text pairs | The coordinate mapping from visual space to language space |
| Vision adaptation | Projection layer plus the last blocks of the vision tower | High-resolution text-bearing images | Making fine detail features usable |
| Instruction fine-tuning | Full language model | Structured multimodal instructions | Task format, turn behaviour, answer style and boundaries |
| Preference optimization | Language model | Visual preference pairs | Suppressing hallucination, calibrating refusal and uncertainty expression |
5.3 Stage three: preference optimization
This stage uses preference data for direct preference optimization or one of its variants: given the same prompt, supply a preferred and a rejected answer and move the model’s probability mass toward the preferred side. Constructing visual preference data is harder than doing so for text, because on visual tasks which answer is better is often far more debatable, and a high share of automatically constructed data introduces noisy labels.
The most valuable category in visual preference data is hallucination contrast. The recipe is to construct a visually nonexistent detail and have the model explicitly refuse or correct it. This directly suppresses hallucination, meaning the tendency of a model to fill in what looks like a plausible but factually nonexistent description using language priors when the visual evidence is insufficient. This is one of the most stubborn defects of vision-language models, and it is especially severe under short-answer formats.
Another high-yield practice is refusal and uncertainty training, which teaches the model to say it cannot see clearly or that the image does not contain it when the image is blurry, the resolution is insufficient, or the question goes beyond the image content, instead of making something up. This capability is strongly coupled to the token budget design discussed in section 3: with a small budget you cannot see clearly anyway, and if the model was never taught what to say when it cannot see clearly, it will confidently output wrong details. So in budget-constrained systems refusal training is not a nice-to-have but a necessary companion.
One warning: the gains from preference optimization on visual tasks are usually smaller than on text tasks, and it easily introduces over-refusal: the model becomes uncertain about everything and starts hedging when you ask what is in the image. The more stable recipe is to cut the learning rate to one tenth to one fifth of instruction fine-tuning, deliberately retain a portion of well-evidenced samples that require a definite answer for balance, and track the over-refusal metric curve on the validation set.
5.4 Data: the half that really decides the ceiling
Architecture decides the ceiling; data decides how much of it you can actually touch. Here is a counterintuitive piece of experience: during alignment pretraining the data volume can be enormous and the quality can be sloppy, whereas in instruction fine-tuning the data volume is often two orders of magnitude smaller but the format correctness of every single example decides success or failure. Under the same compute budget, expanding instruction fine-tuning data from one hundred thousand samples to three hundred thousand crudely generated ones usually gains far less than unifying the format of those one hundred thousand samples and spot-checking them by hand.
Data sources roughly fall into three categories. The first is weakly labelled data crawled from the web or from video frames, where instructions are generated by automatic captioning or from existing subtitles: large in volume, noisy, and used mainly to support alignment pretraining. The second is manually constructed high-quality instructions covering multi-turn dialogue, reasoning, grounding, and refusal: small in volume, but every sample has a clear behavioural target, and this is the backbone of instruction fine-tuning. The third is synthetic data, where a strong model generates questions, answers, and preference pairs in bulk, sitting between the other two.
Synthetic data is the part that has changed the most in 2026. Use a strong model to generate a dozen questions from different angles on one image, then have another strong model produce the preferred and rejected answers; preference pairs built this way work extremely well for hallucination suppression, because the rejected answer can be a manually injected stereotypical hallucination pattern such as stating a nonexistent object with great confidence. But synthetic data carries a fatal risk of stylistic convergence: if tens of thousands of samples all come from one model, the sentence patterns, lengths, and degree of hesitation in the answers become highly uniform, and the model learns the style of that generator rather than genuine diversity. The mitigation is to mix several generators and to forcibly inject a certain proportion of human-written samples and format-anomaly samples into the synthetic data.
Three details in data processing deserve separate emphasis. First, deduplication — especially semantic deduplication of image-text pairs, because duplication rates among web pairs are extremely high, and duplicated samples give the model an abnormally strong bias toward certain phrasings. Second, mix balance: no single task category should dominate the data, and in particular the ratio between multi-turn dialogue and single-turn question answering directly affects whether the model keeps asking or answers directly when it meets a multi-turn conversation. Third, length distribution: the ratio between long-answer and short-answer samples determines the model’s default output length, and the root cause of a model being too verbose or too terse lies here rather than in the model.
Finally, the coupling between data and architecture. A concrete example: if your system relies heavily on tiling, the training data must contain the multi-image samples produced by the same tiling logic, otherwise at inference the model sees one distribution and during training another, and the symptom is strong single-image tasks but weak multi-image tasks. The same applies if you use an OCR dual channel at inference: training should include a comparable proportion of OCR-text-plus-image samples. Architectural decisions and the data pipeline have to change together, or half of the change will never take effect.
6. Special input forms: multi-image, video, and text-dense images
Everything above describes the standard single-image chain. The input forms that actually break most often in production are the ones below, and their difficulty lies not in the model itself but in how the sequence is organized.
6.1 Multi-image input
Multi-image input is far more fiddly than it looks, and there are three things to solve. First is the layout of image tokens in the sequence: put them all up front before the text, or insert placeholders at the corresponding turns. The behaviour of the two schemes differs clearly — the front-loaded version makes the model more sensitive to image order, while the placeholder version supports interleaved image-text instructions better but complicates preprocessing. Second is the continuity of position encoding: when several images are concatenated into one sequence the position indices must be numbered continuously overall, otherwise the model reads the boundary between images as a semantic break, and the symptom is that cross-image reasoning (comparing two images) is clearly weaker than single-image tasks. Third is image-order sensitivity: many implementations are highly sensitive to image order, so swapping two images changes the answer, which is a serious defect on image-text comparison tasks and requires explicitly adding order-perturbed samples to the training data to build robustness.
There is also a capacity problem with multi-image input: N images mean N times the tokens, and the budget tightens immediately. The common practice is to give the main image a full budget and compress the secondary images, or going further, allocate a full budget only to the images explicitly referenced in the question and route the rest through thumbnails. The essence of that policy is using the reference information on the language side to guide budget allocation on the visual side, and the effect is often far better than allocating uniformly.
6.2 Video input
The cheapest way to implement video input is to sample frames into a pseudo-image sequence at a common rate of one to four frames per second and push the whole thing through the image path. The big advantage is that you directly reuse the image-side pretraining and the entire engineering chain; the disadvantage is that you lose inter-frame continuity and the real temporal ordering, so what the model actually sees is a set of unrelated still photos.
The real direction of improvement is to introduce 3-D positional encoding inside the vision tower and turn attention into a spatio-temporal separated form — for example local window attention along the time dimension and full attention along the spatial dimension, or axial attention the other way around. Such designs let the model answer questions about action order, state change, and cross-frame consistency instead of only describing single frames. The cost is a significant rise in memory and implementation complexity, and the training data requirement shifts from image-text pairs to video plus high-quality subtitles, whose supply is an order of magnitude smaller.
6.3 Text-dense images: documents, tables, and UI
Documents, tables, charts, and UI screenshots are the scenarios that separate models most sharply, and they are also the easiest ones to fail on. The typical failure modes are these: insufficient resolution blending strokes into blocks of colour; multi-column layouts treated as single-column so the stitched reading order is wrong; the row and column alignment of tables destroyed so the model reads adjacent cells as continuous text; chart legends and axis labels ignored as background noise; and rotated text, artistic fonts, or handwriting being unrecognized outright.
There are four corresponding engineering measures. The first is raising effective resolution, that is, tiled block reading, which is a physical necessity — the glyph height threshold was already computed earlier. The second is feeding OCR-extracted text in as auxiliary input alongside the original image, so textual information travels over the text channel rather than being read raw out of pixels, which saves tokens and improves accuracy at the same time. The third is requiring the model to output coordinates, that is, grounding: training text blocks bound to positions with detection-style intermediate supervision, which is almost indispensable for downstream table extraction. The fourth is the plainest and most effective option: render the image into structured text with an existing tool first, then let the model read the structured text.
Table and numeric extraction deserve special emphasis: if the goal is to get the exact value of every single cell, the model must output cell-level localization information, because simply letting it read out the table content will inevitably produce line-merging and misalignment. This is also why, in the training data of document models, coordinate supervision is worth far more than plain text supervision — coordinate supervision actually gives the model a strong constraint that it must align with visual evidence.
6.4 High-dimensional output tasks
Beyond one-sentence answers, a vision-language model can output segmentation masks, depth maps, human pose, and detection boxes. That turns visual tasks from discriminative into dense prediction, and the corresponding output head designs are completely different: either attach a decoding head after the language side, or conversely use the visual tokens as queries to drive a downstream segmentation network, or generate a sequence of normalized coordinates on the text side and parse it. The first two are usually more accurate, the third is more flexible but needs extra coordinate decoding and post-processing.
The key to cost control is letting high-resolution features enter the downstream network only when needed. If the segmentation head needs full resolution, take the high-resolution path only there while the text generation path keeps using compressed tokens: the two paths share the vision tower but not the feature resolution. This is close to the standard practice of every multi-task vision-language system.
7. Several new trends in 2026
7.1 Native multimodality and a unified token space
The central claim of this line of work is: do not bolt a projection layer on the back any more, but from the very start of training let images, text, and audio share one token space and one set of model weights. The benefit is direct — modalities can complete each other, long-document understanding with interleaved image and text becomes markedly stronger, and reasoning on the language side is better constrained by visual evidence. The costs are extremely high training expense, an extremely hard data mix to tune, and the fact that weak data quality in any single modality drags down the whole.
By 2026 this line has moved from the paper stage into real deployed systems, but full-scale pretraining is no longer realistic for small and mid-sized teams. The more realistic path is domain continued training on top of the weights of an open-source unified model, focusing on the modality you lack most in your own scenario — for instance a large number of specialized charts, or documents in a specific format — instead of trying to train a unified model from scratch. One pragmatic judgement on this path is that data quality in domain continued training matters more than data volume: a few tens of thousands of high-quality domain samples usually beat several million generic ones.
7.2 Video and spatio-temporal modelling
The centre of gravity in video understanding is shifting from frame-sampled question answering toward spatio-temporal reasoning: action order, state change, cross-frame consistency, long-range events. That requires the model not to treat frames as independent images but to have explicit temporal position encoding and a temporal attention structure. There is a very simple test for whether a system genuinely does spatio-temporal modelling: shuffle the order of the input frames; if the answer does not change at all, then it is basically not doing temporal modelling, it is just doing multi-image description.
7.3 The limits of visual token compression
Reducing visual tokens is currently one of the directions with the most engineering value, because the payoff is direct: a shorter sequence, less memory, higher throughput, and more concurrent requests served per card. The main means include merging similar tokens, spatial pooling, a query resampler, and content-adaptive allocation of token counts. The risk lies in the trade-off between compression ratio and accuracy on detail tasks, and if the compression policy is a non-learnable hard rule, the model needs extra training to adapt to it. My judgement is that placing the compression at the late stage of the vision tower, before the language model, is the position where gains and losses balance best; doing dynamic token counts inside the language model is more aggressive and still fairly fragile.
7.4 Visual sparsity and mixture of experts
Carrying the sparse-expert idea from the language side over to the visual side is the key means of turning high resolution from a hard cost line into an option: high-resolution features are computed heavily only in the few selected experts, while the other tokens take a shared lightweight path. The more mature approach in 2026 puts the sparse selection in the late stage of the vision tower rather than across the whole tower, keeping the early stage dense so that low-level detail is not lost. The direct basis for this judgement is that low-level features carry edge and texture extraction, which is information unfriendly to sparse selection, while high-level semantics are what suits sparsification.
7.5 From being able to see, to being able to use tools
One clear trend at the architecture level is that models are starting to learn to call external vision tools: zoom and crop, call OCR, call a detector, call a renderer. In essence this moves the resolution budget problem from the architecture layer to the inference strategy layer — instead of stuffing every pixel into a single forward pass, the model decides for itself where to look, how many looks to take, and how many rounds to spend. The direction is valuable, but the training difficulty it brings is not small: you need trajectory data where the model calls a tool before answering, the supervision signal is far sparser than ordinary instruction data, and the latency cost of multi-round calls at inference has to be acceptable on the business side. It already looks more like an agent training paradigm than a pure architecture question.
7.6 Evaluation and diagnosis: how to tell which link broke
Evaluating multimodal systems has one special difficulty: when results are bad it is hard to tell whether the vision tower did not see clearly, the projection layer compressed away the information, or the language model never learned to use it. Tweaking things at random is the least efficient diagnostic method. An actionable diagnostic order is to test the feature quality of the vision tower first, then the information fidelity of the projection layer, and only suspect the language model last.
Step one is to push the input image through the vision tower alone and see whether the feature map responds clearly in the text regions. If the response is already weak, the problem is in the vision tower or the resolution: adjust the resolution or swap the tower immediately and stop looking elsewhere. Step two is to check feature similarity before and after projection; if after projection the tokens of different regions become nearly indistinguishable, the problem is the compression ratio or the normalization in the projection layer, so swap in a shallower projection and test again. Only step three looks at the behaviour of the language model — whether it understands the question at all, or is simply guessing from language priors without looking at the image.
There is one extremely useful controlled experiment: replace the image with a pure white image or a random noise image and ask the same question. If the model’s answer is nearly the same as when it saw the real image, it is not using visual information at all, and the problem is in the first two links rather than the language model. This test costs almost nothing, yet it eliminates the link most often misdiagnosed in one shot, and it deserves to be written into the debugging procedure as a standard check.
How the evaluation set is organized also matters. Stratifying by task type is more diagnostic value than stratifying by data source: natural-scene question answering, text reading, counting, spatial grounding, fine-grained recognition, and refusal calibration, with each layer scored separately. An aggregate score can rise while the text-reading layer drops, and that kind of change is simply invisible inside the aggregate — by the time it shows up in production it is already too late.
8. An engineering selection checklist and common failure points
Compressing the judgements above into one executable checklist and going through it in order is more useful than reading ten architecture surveys.
- Compute the token budget first, then fix the resolution. Estimate glyph height and token counts on real business images, and confirm that a pure-pixel approach is physically sufficient.
- Prefer a mature, already-aligned set of vision tower weights; unfreeze only the last few blocks when necessary, and verify real gains only on text tasks.
- Default to a two-layer MLP projection, shallow to moderate depth first, and spend the saved compute on resolution and multi-layer features.
- Add one normalization layer before and after the projection to avoid NaN collapse early in half-precision training.
- Unify the data format template before instruction fine-tuning, spot-check samples by hand, and renumber positions continuously after concatenation.
- Start the visual share of the mix at a few tens of percent, then raise it per task.
- Refusal and hallucination preference data is mandatory; without it a resolution-constrained model will confidently fabricate detail.
- Cut the preference optimization learning rate to about one tenth of instruction fine-tuning and watch the over-refusal metric on its own.
- For text-dense scenarios, prefer the OCR-text-plus-image dual channel instead of piling on resolution without limit.
- Table and numeric extraction tasks must add coordinate supervision; text supervision alone will merge lines and misalign cells.
A few high-frequency failure points deserve separate emphasis. The first is running alignment pretraining and instruction fine-tuning together, trying to tune the data mix without a single baseline, which makes any improvement unattributable. The second is raising resolution without adjusting the long-sequence training configuration in step, so training is fine but inference exceeds the context length. The third is not renumbering positions continuously for multi-image samples, leaving cross-image reasoning inexplicably weak. The fourth is unfreezing the vision tower too much, scrambling an already well-aligned feature space so that general ability actually degrades.
There is also a pragmatic judgement about deployment: more visual tokens does not necessarily mean better results, but it almost certainly means more expensive inference. When writing a design document you should give three numbers at once — token count, per-request memory footprint, and estimated throughput — instead of only a quality comparison. Those three numbers are what let the business side make a decision.
| Scenario | Recommended vision tower | Resolution strategy | Projection layer |
|---|---|---|---|
| General image question answering | Mature self-attention weights | Moderate fixed resolution | Two-layer MLP |
| Documents and tables | Multi-layer features with strong text supervision | Dynamic tiling plus dual-channel OCR | Shallow projection to preserve detail |
| Real-time video stream | Convolutional backbone with attention | Low-resolution frame sampling | Linear to shallow MLP |
| Long video understanding | Self-attention with 3-D positional encoding | Temporal windows plus spatial compression | Query resampler |
| High-resolution image editing | Self-attention preserving layered detail | Highest resolution | Multi-layer feature fusion projection |
8.1 The real memory and throughput bottleneck
Once resolution is pushed up, the bottleneck almost always shifts from the vision tower to the sequence length on the language side. The effective optimization order at that point is: first gradient checkpointing and a frozen cache for the vision tower, then sequence parallelism or bucketing to cut padding waste, and quantization only last. The return diminishes at every step, but the first step usually solves most of the problem, because the peak memory of alignment pretraining and of instruction fine-tuning come from completely different sources — the former is determined by the large-batch high resolution of the projection layer, the latter by the full-parameter language model over long sequences.
8.2 Governing visual tokens on the inference and deployment side
The token budget fixed during training is amplified once more into a cost problem at deployment, for a simple reason: images from online requests are often harder than those in the training set and need more tokens to see clearly, while users are unwilling to wait. The two measures used on the deployment side are caching and tiered budgets.
On caching, the most typical problem in multi-turn dialogue is the same image being re-encoded on every turn. The correct approach is to cache the visual tokens in the prefix and only append text tokens in later turns, so the prefix cache hits and the vision encoding cost is paid once. The precondition is that the visual tokens are ordered before the text, or inserted with a placeholder while keeping the same prefix unchanged; if the image is inserted at a different position every turn, the cache will never hit. The gain from this single implementation detail is often larger than any vision-side optimization, and it barely touches the model.
Tiered budgets are the more proactive approach: quickly judge image difficulty and size with a lightweight discriminator or a rule, answer simple images directly on a small budget, and give complex images a full budget and the complete pipeline. For inputs like document screenshots that almost certainly need a high budget, this judgement is almost never wrong; for simple scenarios such as casual chat images, the compute saved is very substantial.
Compression can also be done on the inference side, and it is often a better deal than compressing during training because it requires no retraining. The recipe is to keep the first and last visual tokens, add several representative tokens chosen by similarity, discard the rest, and let the model adapt to this sparse input at inference time. But if the model has never seen sparse input, results will still drop, so the safer order is: caching first, tiered budgets next, and inference-side trimming only as a last resort.
The three metrics to watch at the end of the deployment side are the P95 length of visual tokens, per-request peak memory, and cache hit rate. Those three numbers determine concurrency capacity and tail latency, and they are far more useful than the average token count, because essentially all the stutter users perceive comes from the long tail.
9. Conclusion
Looking back at the whole chain, the architectural choices of a multimodal model actually converge on three decisive questions: what shape of vector the vision tower outputs, how many tokens the projection layer keeps, and which parameters are moving during training. The answers to these three questions jointly determine the capability boundary and the cost structure of a model, and every remaining detail is a local optimization underneath those three answers.
For the vast majority of engineering teams the lowest-effort path is: a mature, already-aligned vision tower plus a two-layer MLP projection plus an explicit token budget plus a disciplined three-stage training process, with the saved energy spent on data quality and on handling special input forms. What truly creates differentiation is not making some structure fancier, but handling the hard input forms — documents, multi-image, video — seriously.
One judgement worth remembering is that architecture can compensate for the problems caused by insufficient resolution, but it cannot compensate for never having given enough pixels in the first place. The return on investment for text-dense tasks should be prioritised ahead of almost every architectural change: spending twice the compute to raise a full page of document input from 448 to 896 usually works better than deepening the projection layer from two layers to four.
This article only discusses the architecture layer. Real multimodal application development, system prompt engineering, and tool-call orchestration belong to a different main line, are essentially orthogonal to the trade-offs discussed here, and mixing them in would leave both sides unclear. What architecture gives you is a base model with a clear ceiling and a predictable cost; how far above that ceiling you can go depends on engineering and data.
Another easily underestimated judgement is that the capability ceiling of a multimodal system is usually set by its weakest link rather than its strongest. A system with a very advanced vision tower but an over-compressing projection layer will lose to a plain system with an ordinary vision tower and sufficient resolution on text tasks. So when reviewing an architecture, you should be looking for the shortest board, not the longest one.
And looking back from the vantage point of 2026, three lines — the convergence of native multimodality, the filling in of spatio-temporal modelling, and models learning to actively call vision tools — are moving toward the same destination: vision is no longer a one-shot input to the model but a dynamic resource that can be drawn on demand, zoomed on demand, and switched in representation on demand. Understanding this point matters more than remembering any single concrete structure.
