World Models and Embodied AI: Technical Routes and Engineering Challenges

I. What Problem Are World Models Actually Solving

Agents in the software world have an implicit benefit: actions are reversible. If an agent calls the wrong API, writes the wrong file, or walks into a dead end, the worst case is starting over. That ability to "start over" is the hidden foundation behind the fast iteration of software agents over the past few years.

Embodied AI has no such benefit. A robot arm that pushes a cup off the table spills the water; a mobile base that hits a person causes a loss that cannot be rolled back; a bolt tightened past spec is scrap. So before a robot performs an action, it must answer a question that a software agent never has to answer: if I do this, what happens next?

A world model is the engineering answer to that question. It is not "a model that can generate video," nor "a vision model that recognizes objects," but a combination of three components:

• State representation: compressing "the world at this moment" into some representation the model can reason over. It can be a neural network latent vector, a 3D point cloud, Gaussian points, an occupancy grid, or an organized scene graph.
• Transition dynamics: given the current state and an action, predict the next state. This is the hardest and most fragile part of the entire system.
• Action consequence mapping: translating the abstract intent emitted by a human or by a policy into conditional signals the world model can consume. This piece has been underestimated for a long time, yet it decides whether a world model can actually be connected into a control loop.

The key judgment is this: the overwhelming majority of projects that call themselves "world models" have in fact delivered only one or two of these three. Some do representation only (video understanding models), some do generation only (video generation models), some do offline prediction only (given one trajectory, predict the next one). They remain a considerable distance away from "usable as a simulator inside a closed loop." The job of this article is to take this roadmap apart and examine how solid each piece actually is.

1.1 Where the Three Technical Routes Diverge

Putting the publicly visible work from 2024 through 2026 side by side, it splits into three routes according to "in what space prediction happens, and what is being predicted." They are not mutually exclusive — frontier systems in 2026 are converging toward hybrid architectures — but the underlying philosophies differ enormously, and choosing the wrong one carries an extremely high engineering cost.

Route A: representational world models (predict abstractions, not pixels)

The core claim is that what an agent needs for decision-making is not "what the next frame looks like" but "what the next frame means." So predict directly in a latent representation space, throwing away entirely the high-frequency degrees of freedom that are irrelevant to decisions — pixels, textures, lighting, the flutter of leaves.

There are two typical approaches on this track. One is the joint embedding predictive architecture (JEPA): maintain a context encoder and a target encoder, encode the current frame and several future frames into latent space separately, and place the prediction loss in latent space. The target branch is normally updated by exponential moving average with stop-gradient applied to the context branch — otherwise the model finds the degenerate shortcut of "encode every input into the same vector." The other is recurrent state-space models (the RSSM family): explicitly splitting out a deterministic state and a stochastic state, using the stochastic state to absorb the uncertainty of observation and the deterministic state to carry the dynamics, and pairing it with a KL balancing loss that pulls the prior toward the posterior, so that dozens of steps of imagined rollout happen inside latent space.

The strength of this route is unmistakable: it is fast. Latent dimensions run two to three orders of magnitude below pixel space, so rolling out hundreds of steps in latent space is affordable — which is precisely why it is naturally suited to model-based RL. The weakness is just as clear: it produces no visible image, so when it goes wrong the error is hard to localize. That makes it hard to use standalone as an interpretable simulator, and hard to use directly to generate training data.

Route B: generative world models (treating a video generation model as a physics simulator)

The bet on this route is: since video generation quality is already high enough, add action conditioning and it becomes approximately a renderable physical environment. Its selling point is not abstract reasoning but fidelity, renderability, and utility as a data engine.

Technically, the bar on this route has been pushed very high: diffusion or flow-matching architectures, high-dimensional latent video tokenizers, massive video pretraining, and a conditioning injection mechanism able to ingest actions. It transplants the big generative-quality problem straight into robotics, so "is the generated physics correct?" becomes the only question that matters — and that is exactly where current generative models are least stable.

The strength here is that once it works, you have an unlimited simulator: you can manufacture extreme samples a real robot could never collect, and you can fill in the long tail. The weakness is that real-time performance and controllability are both hard. Multi-step denoising or flow-matching inference latency is naturally in the range of hundreds of milliseconds to seconds, while a robot closed loop demands millisecond latency. At the same time, "the frame is plausible" and "the action consequences are correct" are two different things — a good-looking frame does not imply the latter.

Route C: latent space prediction and planning (the engineering compromise between the two)

The approach on this route is: do not chase full pixel-level reconstruction. Instead, learn an action-conditioned dynamics model in latent space, then choose actions with short-horizon rolling plus value guidance. The typical structures are the imagined actor-critic that evolved out of the Dreamer line, and the TD-MPC line that combines latent space modeling with model-predictive control and imposes planning feasibility constraints inside latent space.

It deliberately compresses the prediction horizon very short — usually single digits to low-teens steps. This is an extremely important engineering judgment: the value density of a world model decays rapidly with time steps, so rather than build a model that cannot hold up for 200 steps, build one that is accurate within 10 and let feedback control absorb the remaining uncertainty. It is also the most engineering-mature of the three routes, and the easiest one to actually get running in production today.

1.2 Side-by-Side Comparison of the Three Routes

Dimension	Representational	Generative	Latent-space planning
Prediction target	Latent representation	Pixels / latent video	Latent state
Main benefit	Abstract reasoning, sample efficiency	Fidelity, data generation	Closed-loop usability
Main bottleneck	Latent collapse, no visual interpretability	Physical consistency, inference latency	Limited long-horizon extrapolation
Inference real-time performance	Good	Poor	Good
Usable as a data engine	No	Yes (if the physics is trustworthy)	No
Typical failure mode	Latent variables carry no geometry, so planning and perception come apart	Object drift, corrupted occlusion, inverted causality	State prediction diverges outside the training action set
Deployment maturity	Medium	Low to medium	High

The row most easily misread in this table is "deployment maturity." It does not mean the representational and generative routes are unimportant; it means that in cost-sensitive, high-consequence robotic scenarios, the value of something that actually works far exceeds that of something that merely looks more beautiful. This also explains the actual direction of travel in 2026: more teams no longer pursue a single pure route, but adopt a hybrid stack in which "latent space handles planning, generative models handle rendering and data, classical control handles fallback and real-time execution."

II. From Pixel Prediction to Representation Prediction: Why the Target Must Change

2.1 Three Structural Failure Modes of the Pixel-Level Objective

Treating "predict the next frame" as the definition of a world model is very close to the universal mistake of all early attempts. Its failure is not that training was insufficient — it is that the objective function itself penalizes the right thing. It decomposes into three independent failure modes.

First, the unpredictable component eats most of the model capacity. In real video, the distribution of pixel energy is extremely skewed: large background regions and slow lighting variation are low-frequency and predictable, while leaf flutter, splashing water, glass reflections, fabric wrinkles, and sensor noise are high-frequency and essentially random. If you train with a pixel-level reconstruction loss, the gradient gets dominated by those unpredictable components, and the model either wastes capacity fitting noise or underfits the predictable low-frequency structure. This is why early "video frame prediction" models often produce a high-fidelity but physically meaningless background, yet blur to mush wherever objects actually move.

Second, per-pixel regression converges to the conditional mean. Under L1 or L2 loss, the optimal prediction is the average of all possible future pixels. When "the cup is at A in the next frame" and "the cup is at B" are equally likely, the model outputs a blurry superposition of A and B. This directly causes trajectory divergence: a few pixels of blur, amplified repeatedly through the rollout, and the object "melts away" within a handful of steps. Visually it appears as fogging; physically it appears as an implausible collapse of uncertainty.

Third, frame rate does not match physical events. Video is sampled at a fixed frame rate, but physical events vary enormously in duration: the critical moment of a single grasp might last only tens of milliseconds. The model gets trained to optimize "average per-frame correctness," so it spends much of its capacity on redundant information where frames change slowly, while staying insensitive to the contact instant that decides success or failure.

2.2 How Latent Space Prediction Sidesteps All Three

The representational route answers with three corresponding moves: move the objective function into latent space, so that unpredictable high-frequency detail never enters the loss; use predictive rather than regression-style targets, with online bootstrapping to avoid regressing directly onto the final outcome; and give the prediction head a causal mask, so it predicts only the currently invisible portion instead of copying the input, which rules out the trivial solution.

The engineering details here matter far more than the method names, and are worth spelling out one by one.

On preventing representation collapse: if the context encoder is free to encode all frames into arbitrary vectors, then the objective "predict the target encoder's output" can be satisfied trivially. The two most common countermeasures are to stop gradient backprop into the context branch, so that the target is updated only slowly through the EMA target branch; and to introduce a variance regularizer that encourages different dimensions of the latent to spread out rather than all collapsing to the mean. More aggressive JEPA variants additionally apply layer-wise normalization to the target representation, further erasing scale information and forcing the model to preserve structure.

On keeping the prediction horizon short: representational models become essentially unreliable after a dozen frames or so. The engineering consensus is not to force a longer horizon but to decompose long tasks into a "plan one layer, execute a few frames, replan" loop. In essence, this treats the world model as a local model rather than a global one.

On the representation must carry geometry: this is the most overlooked and also the most fatal of the three. A representation that performs well on classification tasks has not necessarily preserved metric structure. If the vector displacement corresponding to "move the cup 10 cm to the left" in latent space is approximately the same direction across all scenes, then that representation can support grasping. But if the displacement vector depends on viewpoint and on background, the policy collapses the moment the scene changes. So the test for whether a representational world model is usable is not its reconstruction metric, but whether performing a linear interpolation or perturbation in latent space and then rendering or decoding the result is physically plausible.

2.3 Applicability Boundary: What the Representational Route Is Good At, and Bad At

The strengths of the representational route concentrate in three places. First, planning: rolling out in latent space is cheap, which makes large-scale sampling-based search affordable. Second, sample efficiency: on tasks with very little real interaction, a small number of trajectories plus imagination-based training already yields a usable policy. Third, multi-task sharing: a shared latent space naturally provides a transfer interface between tasks.

The weaknesses are just as clear. It produces nothing you can look at, so when something goes wrong there is nothing to diagnose from. Its latent space can be "gamed" by the policy — the policy finds states that are unreachable in the real world but carry low loss in latent space — and in implementation this has to be plugged with feasibility constraints. And it is hard to use directly to manufacture training data, because without a decoder there are no pixels.

A pragmatic way to combine them is: use the representational model for planning and cost evaluation, and the generative model for rendering and data synthesis. The former answers "will it be done correctly"; the latter answers "does it look right, and can it manufacture new scenarios." The interface design of this combination often determines success or failure more than the individual quality of the two models.

III. Generative World Models: The Question of Substance Beyond Image Quality

The core bet of the generative route can be put in a single sentence: video generation models have already implicitly learned physics, and if you connect actions into them, it becomes approximately a renderable interactive environment. This bet does hold in some scenarios and fails in many others, but its appeal is strong enough — because once it holds, it solves precisely the most expensive problem in embodied AI: real robot data.

But this bet contains three layers that need to be examined separately, and much of the discussion conflates them into one:

• Visual coherence: whether consecutive frames connect smoothly without flickering. This is the easiest layer to achieve, and the least important one.
• Semantic persistence: whether the same object is still that same object, keeping a constant size, color, count, and identity. This is medium difficulty, and it sits right around the "passing mark" of most current generative models.
• Correctness of action consequences: if I push the table, does the table actually move along the direction of the force; if I pick up the cup, does the cup actually leave the tabletop. This layer almost single-handedly determines whether it is useful in robotics, and it is currently its weakest.

A common but fatal failure mode is that the model has learned correlation rather than causality. In video, if a hand moving to the left usually comes with the cup moving to the left, the model only has to learn that statistical association in order to generate short clips that "look right." But once it enters a closed loop — executing dozens of actions in a row — that shortcut is fully exposed: each time it produces a next frame that satisfies marginal statistics but violates accumulated constraints, so objects systematically "leak" mass or drift in position. On offline video this is almost invisible, because there is no feedback.

3.1 What Putting a Video Generation Model to Work as a Physics Simulator Is Actually Betting On

3.2 Latent Space + Flow Matching: The Current Mainstream Technical Choice

The 2026 generative world models have converged heavily at the architectural level, and the typical choice is "a high-compression latent video tokenizer plus a flow-matching generator plus multiple conditioning injections." Here is why each piece is chosen, and what each one costs.

Why a latent video tokenizer: running diffusion directly in pixel space multiplies timestep, spatial resolution, and channel count together into a compute explosion. A video tokenizer compresses a span of frames into a low-dimensional latent variable, shrinking the problem size into an acceptable range. The cost is two lossy compressions stacked on top of each other: tokenizer artifacts and decoding loss blend into the generation error, which is especially unfriendly to fine geometry and contact reasoning. This cost is often ignored, because perceptual metrics in offline evaluation mask it.

Why the shift from diffusion to flow matching: diffusion has to iterate for many steps (dozens to start), while flow matching can reach acceptable results with fewer sampling steps, which directly improves real-time performance. But reducing the number of sampling steps for flow matching aggravates distribution drift — the fewer the steps, the worse the consistency over long sequences. This is the same contradiction as the three-layer split in 3.1: quality, real-time performance, and consistency cannot all be had at once, and every current system picks a balance point inside that triangle.

What conditioning injects: this is the key to correctness of action consequences. There are three common approaches: injecting the action embedding via cross-attention along the time dimension; concatenating the action as an extra channel into the latent representation; or encoding the action as spatio-temporal modulation (similar to the modulation schemes used in diffusion models) applied across layers. Each approach has its own preference, but for "contact-rich" tasks — grasping, insertion, screwdriving — all of them face the same dilemma: the information content of the action is extremely low at the moment of contact (a 7-dimensional vector carries almost no information), yet the alignment before and after contact must be extremely precise; being off by a few millimeters means grasping empty air.

3.3 The Timestep Gap Between Offline Video and Online Interaction

This is the most engineering-shaped and also the most easily overlooked wall facing the generative route. The timestep of offline video generation is "generate an entire future sequence," which permits bidirectional attention and globally consistent information. Online interaction demands something different: "given the current frame and one action, produce a result within the next control period." The direction of information flow in these two modes is completely different.

The time budget is even more troublesome. The control period of a typical mobile manipulation task is on the order of 20 to 50 milliseconds, while a multi-step flow-matching latent video model, even after extensive engineering optimization, typically lands in the hundreds of milliseconds to seconds. That is more than a tenfold gap. The engineering means available to close it boil down to a few options: lower the generation resolution and the number of frames (sacrificing detail), distill to fewer steps (sacrificing diversity), generate only keyframes and interpolate the intermediate frames (sacrificing responsiveness to continuous motion). None of them comes for free.

This also explains why the robots actually deployed in 2026 mostly adopt a "hybrid stack": the generative model handles offline data generation and high-level semantic reasoning, while the real-time part is handed over to classical model-predictive control, trajectory optimization, or a purely feedforward policy. The position of the world model on a real production line is far lower than it is in the papers.

IV. Interactive World Models: Action Conditioning and Latent Action Spaces

To make a world model genuinely "interactive," the core difficulty is where the action comes from. Generating video only requires "what happens next," while interaction requires "Y happens because I did X." That Y is exactly where the difficulty lies.

4.1 Explicit Action Injection: Requires Paired State-Action-Outcome Data

The most direct approach supplies the action explicitly: input the current state and the action, predict the next state. This is clean and strongly causal, but the cost is that it requires large-scale, heavily annotated action data. Video on the public internet carries almost no action labels — you can see a cup sliding off the table, but you do not know who applied the force, how large it was, or in what direction.

This is exactly why robot data is expensive, and it is also where the entire promise of the "learn from human video" route lives. If action labels have to be collected through teleoperation, then every single frame requires a human sitting in front of the robot operating it by hand, and the cost runs to several times that of pure video data, not less.

4.2 Latent Actions: Self-Supervised Action Extraction from Unlabeled Video

The idea behind latent actions is to bypass explicit labels: from monocular human video, train an encoder that maps consecutive frames into a low-dimensional action space, and let the model perform action-conditioned prediction inside that space. The action dimensionality itself (compressing a continuous high-dimensional quantity down to, say, 2 to 8 dimensions) is a constraint imposed on complexity by construction — a low-dimensional action space rules out the degenerate solution of "hard-coding whole-frame motion into the action," forcing the model to extract the genuine control variables.

The core of training such a model is a clever self-supervised objective: make the interpolation between the current frame and some future frame predictable from the learned actions. The intuition is that an action is essentially "a displacement along the flow of time": specify "how many steps from now into the future" as a weighted combination of a few learnable codebook tokens, and in order to predict coherently, the model must align the action tokens with temporal position. That removes any need for manual action labels and lets you extract a sequence of discrete latent actions from raw video for free.

The capability boundary of this method has to be stated clearly: the actions it extracts are visual "motion displacements," not physical "forces". For motions that mainly manifest as displacement — navigation, pushing and pulling, throwing — latent actions are quite effective. But for pulling up (which requires judging whether the grasp point is actually gripped), for rotation (which involves contact and friction), and for any force-dependent task, visual motion contains almost none of the required information. So latent actions suit coarse-grained action-conditioned pretraining, not direct fine-tuning of precision manipulation.

4.3 Three Necessary Properties for Training Latent Actions into a Closed Loop

Whether actions come from teleoperation or from latent actions, for an action-conditioned model to serve as the "environment" inside a closed loop, three properties must hold, and they are the root cause of many failures:

• Real-time performance: a single prediction has to complete within the control period, otherwise the model only slows the system down. This constraint in turn limits model size and the number of prediction steps.
• Re-entrancy: repeating the same action under the same state must produce the same outcome. If the model's prediction is stochastic, a closed-loop controller cannot converge and will jitter continuously. This is a fatal problem when many generative world models are used directly as simulators.
• Controllable error growth: error growing as prediction steps increase is not frightening; error blowing up superlinearly is. Many current models still look reasonable within 3 to 5 steps and become completely distorted past 10 steps. So in real systems the world model is rarely rolled out for long — the correct approach is a short horizon plus high-frequency replanning.

4.4 A Minimal Latent Space Dynamics + Imagination Training Skeleton

To make the discussion above concrete, the following Python code shows the core training loop of an interactive world model: quantile representation encoding, RSSM-style stochastic dynamics, and imagined rollout. It is not complete production code, but it marks the seams where "representational + latent space + closed loop" meet. Pay attention to the engineering trade-offs in the key comments.

import torch
import torch.nn as nn
import torch.nn.functional as F


class RSSM(nn.Module):
    """递归状态空间模型：确定性状态承载动力学，随机状态承接观测不确定性。

    关键取舍：随机维度不能太小，否则 KL 项会平凡地把先验拉向后验，
    世界模型退化成"抄观测"；也不能太大，否则先验学不动、rollout 很快发散。
    """

    def __init__(self, obs_dim: int, act_dim: int, deter: int = 512, stoch: int = 64):
        super().__init__()
        self.deter, self.stoch = deter, stoch
        # 先验分支：只吃历史状态和动作，用来在"没有观测"时也能往前推（想象时用）
        self.prior = nn.GRUCell(act_dim + stoch, deter)
        self.prior_head = nn.Linear(deter, 2 * stoch)   # 均值 + log 方差
        # 后验分支：吃先验状态和当前观测，用来在"有观测"时校正（训练时用）
        self.posterior = nn.GRUCell(obs_dim + deter, deter)
        self.posterior_head = nn.Linear(deter, 2 * stoch)
        # 观测解码头：把潜状态还原成观测，逼迫潜空间真的携带世界信息
        self.obs_head = nn.Sequential(nn.Linear(deter + stoch, 512), nn.SiLU(),
                                      nn.Linear(512, obs_dim))

    def _dist_params(self, h):
        """把高斯参数切片拆出来，并夹住方差防止数值爆炸。"""
        mean, logstd = h.chunk(2, dim=-1)
        return mean, logstd.clamp(-8.0, 4.0)

    def prior_step(self, h_prev, z_prev, a):
        """纯动力学推进：世界模型在推理/想象时走的唯一路径，必须可重入。"""
        h = self.prior(torch.cat([a, z_prev], dim=-1), h_prev)
        mean, logstd = self._dist_params(self.prior_head(h))
        z = mean + torch.randn_like(mean) * logstd.exp()
        return h, z

    def observe(self, obs, h_prev, z_prev, a):
        """有观测时校正后验；训练用 KL 把先验拉向后验，这就是"动力学先验"的训练信号来源。

        注意后验必须吃先验的 h（而不是 h_prev），这样 KL 约束的才是
        "仅靠动力学预测的状态" 与 "融合了观测的状态" 的差距。
        """
        h_prior, z_prior = self.prior_step(h_prev, z_prev, a)
        h = self.posterior(torch.cat([obs, h_prior], dim=-1), h_prior)
        mean, logstd = self._dist_params(self.posterior_head(h))
        z_post = mean + torch.randn_like(mean) * logstd.exp()
        kl = self._kl((mean, logstd), self._dist_params(self.prior_head(h_prior)))
        recon = self.obs_head(torch.cat([h, z_post], dim=-1))
        return h, z_post, {"kl": kl, "recon_mse": F.mse_loss(recon, obs)}

    @staticmethod
    def _kl(q, p):
        """两个对角高斯之间的 KL；free-bits 阈值防止后验坍塌到先验（反过来的坍塌）。"""
        qm, qs = q
        pm, ps = p
        return (qs - ps + (ps.exp() + (pm - qm) ** 2) / (2 * ps.exp()) - 0.5).sum(-1).clamp(min=1.0).mean()


def imagine_and_actor_critic(rssm: RSSM, start_state, actor, critic, horizon: int):
    """在潜空间想象 H 步并回传策略梯度（REINFORCE 风格，可换成 TD(λ)）。

    工程要点：
    1) horizon 要短（通常 3~15）。长时域想象会让误差超线性爆炸，
       收益递减而算力线性增长，短时域 + 高频重规划是更稳的工程选择。
    2) actor 的梯度只依赖潜状态轨迹，critic 提供基线降低方差。
    3) 想象过程中不反传 observation 分支（no_grad），省显存。
    """
    h, z = start_state
    traj = []
    for _ in range(horizon):
        a = actor(torch.cat([h, z], dim=-1))
        with torch.no_grad():
            h, z = rssm.prior_step(h, z, a)   # 想象阶段走先验，不看观测
        traj.append((h, z, a))

    # 逆向累积折扣回报，lambda-return 平滑 bootstrap；critic 已给出基线，
    # 所以这里 G 已经是"超额回报"，不需要再减去基线。
    G, lam, actor_loss = 0.0, 0.95, h.new_zeros(())
    for step, (h, z, a) in enumerate(reversed(traj)):
        with torch.no_grad():
            r = critic(torch.cat([h, z], dim=-1)).mean() - 0.01 * (a ** 2).sum(-1)
        G = r + lam * G
        logp = torch.log_softmax(actor(torch.cat([h, z], dim=-1)), dim=-1)
        # 逐步权重 0.9^t：靠后的步对梯度贡献更大，等价于短时域的隐式折扣
        actor_loss = actor_loss + (0.9 ** step) * (-G * logp.sum(-1)).mean()

    # 世界模型的 KL / 重构项由训练循环另行回传，这里只返回策略损失
    return actor_loss

This skeleton deliberately leaves parts "incomplete" (for example, the posterior correction branch is left with a hook). The reason is very practical: a complete trainable RSSM plus actor-critic would turn this section from "explaining the roadmap" into "transcribing an implementation," which would blur the point. What is genuinely worth taking away are the judgments in those three comments — the magnitude of the stochastic dimension, the preference for short horizons, and running imagination through the prior branch. Those three have not been overturned as of 2026.

V. Spatial Intelligence and 3D Representation: Which Layer Is Hard

5.1 From 2D Observation to a 3D World: Ambiguity Is Not a Precision Problem but an Information Problem

Monocular video-to-3D reconstruction is hard not because of the optimization solve, but because of insufficient information. From a single viewpoint, a blurry silhouette could be a single solid cylinder or two distant pillars overlapping. This ambiguity is mathematically impossible to remove from the observation itself; it can only be filled in with priors (objects are usually some particular shape, the ground is usually flat). And the stronger the prior, the more the reconstruction tilts toward a "neat-looking" fictional geometry, which is actively harmful in precision manipulation.

This yields an implication that is often overlooked: in embodied tasks, the goal of a 3D representation is usually not "reconstruct it truthfully" but "make it good enough to decide with". Judging the trajectory of a cup falling off a table requires only its approximate position, velocity, and rough scale; it does not require millimeter-level surface normals. Judging whether the arm can insert into a slot, on the other hand, requires local high precision, which is exactly the opposite requirement. In other words, there is no unified "3D representation precision" metric — only precision requirements defined per task.

5.2 Metric Accuracy and Topological Structure: Two Different Capabilities

Spatial intelligence has to be split into at least two capability layers, and their technical paths barely intersect.

• Metric geometry: where an object is, how large it is, how far away, which way it faces. Under monocular input this is fundamentally unrecoverable, so in practice it depends on multiple viewpoints, a known baseline (IMU odometry, action logs), or priors. This is the home turf of classical SLAM and multi-view geometry.
• Topology and relations: what is inside what, what supports what, which handle connects to which cavity, which side of the door frame the switch is on. This kind of information is, precisely, directly visible in a single-frame 2D observation — because it is a semantic relation in the image plane and requires no depth.

Interestingly, the second category (topological relations) is learned implicitly in large quantities during vision-language model pretraining, while the first requires dedicated 3D data. So here is a counterintuitive conclusion: the semantic priors of large models are strong on topological relations and weak on metric geometry, because image-text data from the internet contains almost no precise metric supervision. This explains why so many 3D methods inject 2D semantic priors into the geometric pipeline rather than the other way around.

5.3 Four Engineering Carriers for Explicit 3D Representation

To represent the 3D state of the world inside a model, the mainstream carriers in 2026 fall into four categories, each with a very clear applicability boundary.

Representation form	Strength	Weakness	Applicable scenarios
Dense occupancy grid / occupancy field	Unifies geometry and semantics, directly usable for collision checking and planning	Memory and compute grow cubically with resolution, hard to cover long corridors and outdoor space	Autonomous driving, tabletop near-field work, primary obstacles for navigation
3D Gaussian points (the 3D Gaussian Splatting family)	High rendering quality, fast training and inference, differentiable	Represents a surface rather than occupancy, collision checking requires an extra conversion, updating dynamic scenes is awkward	Novel view synthesis, digital twins, re-rendering for data augmentation
Multi-view point cloud / depth map fusion	Interfaces directly with sensors, short pipeline, easy to attach to classical stacks	Accuracy is dragged down by multi-view calibration and pose precision, fails in poorly textured regions	Grasp pose estimation, point cloud planning, robot arm workspace
Implicit fields (occupancy / distance / radiance field functions)	Resolution decoupled from storage, continuous and differentiable, friendly to optimization	Slow to train, random queries, real-time rendering depends on acceleration structures	Offline reconstruction, research baselines, tasks requiring continuous optimization

There is only one core question in choosing: is your downstream task "looking" or "touching"? For "looking" (simulation rendering, semantic queries, data augmentation), Gaussians or implicit fields offer the best cost-performance ratio. For "touching" (collision checking, grasping, reachability analysis) you must fall back to occupancy or point clouds, because the former has no reliable answer to "is there something at this position."

5.4 What Feed-Forward 3D Reconstruction Actually Changes

The most practically useful advance of the past two years is not the end-to-end implicit 3D model, but feed-forward 3D reconstruction: no per-scene optimization, using a Transformer to predict a point map, or depth plus camera parameters, in one shot from several views, compressing what used to be tens of minutes to several hours of optimization into a single forward pass.

It has not solved the fundamental monocular ambiguity problem, but it has solved a very practical engineering bottleneck: turning 3D reconstruction from "offline batch processing" into "an online component". For a world model this matters enormously — only when reconstruction is fast enough can it supply a 3D state for every frame inside the closed loop, and only then does "imagining in 3D" become possible instead of something you can run once offline. This is also the precondition for attaching a 3D representation to a latent space planner.

Its limits should be kept clearly in view, though: feed-forward methods perform well in texture-rich natural scenes, and tend to make mistakes in symmetric, transparent, reflective, and repetitively textured industrial and lab settings — which is precisely where robots spend most of their time. Its proper role is that of "fallback geometry": take a well-calibrated depth camera or multi-camera rig as the primary source, and use feed-forward methods to fill blind spots and raise the frame rate.

VI. Long-Horizon Consistency and Controllability: Two Underestimated Hard Problems

6.1 Long-Horizon Consistency: Error Does Not Accumulate Linearly

Almost every world model evaluation is short-horizon: given 8 to 16 frames of real history, predict some number of future frames. But deployment needs task horizons of minutes to tens of minutes. The gap between these two is the most underestimated deployment obstacle of 2026.

Error accumulation is superlinear because of a feedback mechanism: once the model's state estimate drifts away from the true state, the state it occupies has left the training distribution — during training it only ever saw "real, plausible" intermediate states, while rollout produces "plausible but unreal" intermediate states. The model's error function is unconstrained in the out-of-distribution region, so each step's error exceeds the previous one and positive feedback forms. This is why so many models look nearly perfect for the first 10 steps under open-loop evaluation, collapse completely after 20 steps, and produce meaningless predictions past the point of collapse.

Effective mitigations are few today, and are mostly a combination of three things: short horizon plus high-frequency replanning (the most practical), long-term memory constraints (anchoring key states, for example keeping object positions in a map from being overwritten by the rollout), and uncertainty estimation and reset (falling back to a conservative policy or requesting a re-observation when the model is unsure of itself). The third is the most reliable in engineering terms but also the most troublesome, because "knowing that you don't know" requires an additional training signal.

6.2 Controllability: Three Levels of Increasing Difficulty

Breaking controllability down into three levels makes it much clearer where current progress has actually stalled.

Level	Control target	Current status	Key difficulty
Instruction-level control	Semantic goals such as "put the red block into the box"	Relatively mature	Instruction ambiguity, scene ambiguity, long-horizon semantic memory
Trajectory-level control	Specifying the path or target pose of the end effector	Medium, engineering-feasible	Collision feasibility of the path against obstacles, closed-loop replanning
Contact-level control	Specifying force, torque, and contact state, phase control for insertion and screwdriving	Immature	Force information is invisible to vision, the contact instant has an extremely short time constant

Most generative world models claim controllability, but what they actually achieve is instruction-level plus part of trajectory-level. Almost nobody has crossed into contact-level, for the reason already stated in the first section: contact is a physical event that is invisible to vision. Video carries neither force nor tactile signal, so the model can only see the outcome after contact (whether the cup was picked up or not). This is purely a missing-information problem, not a compute problem.

If the task genuinely does require contact-level capability (assembly, insertion and removal, precision screwdriving), the pragmatic route today is a hybrid policy combining tactile and force-torque sensing with vision: let the world model handle the high-level planning up to just before contact, and hand the last few centimeters to classical force control. This is not technological conservatism but division of labor: let the model do the part where it has an information advantage.

VII. Data Cost: The Real Hard Constraint of Embodied AI

7.1 Real Robot Data: The Cost Curve Is Steeper Than Most People Estimate

When discussing embodied AI, people habitually say "real robot data is expensive." The judgment is right, but it is not specific enough. More concretely, the cost of real robot data is determined by three multiplicative factors, and they do multiply:

Scene setup cost × collection throughput × reset cost. The vast majority of people estimate only the first and the second and ignore the third — but the reset cost is often the dominant term. In a grasping task, if the success rate is only 40%, then behind every 10 "valid successful demonstrations" there are 15 failures and resets. Each reset means returning objects to their initial pose (manually, or automatically with the arm), confirming nothing was damaged unintentionally, and recalibrating if necessary. This makes the labor cost of a single valid trajectory several times its nominal duration.

More importantly, marginal cost rises steeply with generalization requirements. What you need to cover is not the "success path" but "every possible failure," and the failure space is combinatorially explosive: object position × orientation × material × lighting × distractors × the robot's own state. This explains why the strategy at large companies is always teleoperation collection combined with crowdsourcing and multi-site parallelism: the only effective lever is to outsource scene setup and turn humans into one station on an assembly line.

There is also a frequently overlooked time dimension: the "hours" of real robot data are the most expensive resource there is, and the pace of model iteration is bound by how fast data arrives. The real bottleneck for many teams is neither compute nor data volume, but the fact that the arrival rate of data cannot keep up with the pace of architectural iteration.

7.2 Simulation: The Domain Gap Has Explicit Structure, Not a Vague "It Doesn't Look the Same"

Saying "there is a gap between simulation and reality" carries no information. The domain gap can be split into three categories whose countermeasures are completely different, and this distinction directly determines how engineering effort gets allocated.

Visual domain gap: differences in the distributions of texture, lighting, material, and camera noise. This is the easiest category to handle — domain randomization (randomizing texture, lighting, camera pose, background, object appearance) can push it down to an acceptable level, which is why simulated visual RL is relatively mature in 2026.

Dynamics domain gap: friction coefficients, joint backlash, actuator latency, compliance, and structural deformation that differ from reality. This is the most fatal category, because domain randomization is almost useless against it — you cannot randomize "simulated rigid-body contact" into "real compliant contact." There are only two effective responses: system identification (fitting dynamics parameters backwards from real robot data so the simulation itself becomes accurate), or residual learning (layering on top of the simulation a residual model trained on real robot data).

Sensor and actuator gaps: depth cameras failing on reflective metal, encoder quantization error, torque droop as battery voltage drops with charge, thermal drift. These usually do not exist at all in simulation, and they are the hardest to expose during testing — because their failures are tied to the state of the device.

This yields a counterintuitive but important conclusion: the real value of simulation is not to replace real robot data, but to generate samples a real robot could never collect — long-tail failures, extreme perturbations, dangerous operating conditions. For safety training and red-team testing, simulation is irreplaceable; for replacing ordinary behavioral data, the domain gap makes the payoff decay rapidly.

7.3 Video Transfer: What Free Data Can and Cannot Support

Internet video is the only "free" large-scale temporal data, and its value ceiling is set by one physical fact: force information is not visible in video. We know the cup is moving when it slides off the table, but we do not know whether the tabletop friction coefficient is 0.3 or 0.6, nor whether the cup's center of mass sits forward or backward. Video records kinematics, not dynamics.

This absence determines the applicability boundary of video transfer, and that boundary is quite sharp:

• Transfer works: navigation and coarse motion (roughly which direction to move, when to turn), reversible and highly fault-tolerant tasks (the rough trajectories of wiping and tidying), and semantic-level priors (what objects usually appear together, common scene structures).
• Transfer is inefficient but not zero: tasks requiring precise geometry (grasp pose), where video can give "roughly where it is and roughly what pose," but the precision has to be supplied by real sensors.
• Transfer is essentially useless: fine contact tasks (insertion and removal, screwdriving, cable handling), any task requiring force or torque feedback, and manipulation that depends on friction and compliance. The reasons these tasks fail are simply not visible in video.

So the most important engineering role for a vision-language-action model is pretraining and initialization: pour the semantic and coarse-motion priors learned from video into the policy, then adapt to the target task with a small amount of real robot data. It lowers the order of magnitude of the real robot data required, but it does not reduce that requirement to zero. Treating a VLA as a "data-free solution" is one of the most widespread misjudgments of 2026.

Data source	Unit cost	Dynamics fidelity	Ability to cover the long tail	Best-fit role
Real robot teleoperation	Highest	Real	Limited by labor and time	Final adaptation data for precision manipulation
Human egocentric video	Nearly zero	None (no force or friction)	Natively covers long-tail scenarios	Pretraining priors and semantic initialization
Physics simulation	Low	Bounded by system identification quality	Extremely strong, actively constructible	Red-teaming and safety training for dangerous conditions
Generative synthesis	Medium	Depends on the physical consistency of the generative model	Controllable and programmable	Filling in long-tail combinations missing from real data

7.4 Data Mix Ratio: The Knob Engineers Are Actually Tuning

In reality, nobody relies on a single data source. Almost every serious team is tuning a mixing ratio, and a badly tuned ratio often costs far more than switching to a larger backbone. The following code spells out the mixing logic, and the step most easily overlooked in it is "safety filter first, then oversample."

from dataclasses import dataclass
from typing import Dict, List

@dataclass
class Bucket:
    name: str
    items: List[dict]
    sample_weight: float      # 过采样权重：低质量但高覆盖的数据给更高权重
    max_ratio: float          # 占比上限：防止某个源主导训练，灾难性遗忘的保险丝

# 四类数据源，weight 的含义是"同一条样本被看到的相对次数"
BUCKETS: Dict[str, Bucket] = {
    "real_teleop":   Bucket("真机遥操作", [], sample_weight=1.0, max_ratio=0.45),
    "human_video":   Bucket("人类视频",   [], sample_weight=0.35, max_ratio=0.50),
    "sim":           Bucket("仿真",       [], sample_weight=0.5, max_ratio=0.60),
    "generated":     Bucket("生成合成",   [], sample_weight=0.6, max_ratio=0.40),
}

def apply_safety_filter(items: List[dict]) -> List[dict]:
    """先剔除坏样本再过采样。

    顺序很关键：如果先过采样再过滤，坏样本会被反复放大，
    一个污染样本的梯度会被重复施加 N 次，这是训练不稳定的常见根因。
    """
    out = []
    for it in items:
        # 关节超限、时间戳倒流、力矩削顶的片段都应剔除
        if it.get("max_torque") >= it.get("torque_limit", 1.0):
            continue
        if it.get("dt", 0) <= 0:
            continue
        out.append(it)
    return out

def mix(buckets: Dict[str, Bucket], batch_size: int, rng) -> List[dict]:
    """按权重采样并强制占比上限。返回打乱后的一个 batch。"""
    for name, b in buckets.items():
        b.items = apply_safety_filter(b.items)

    live = [b for b in buckets.values() if b.items]
    picks, weights = [], []
    for b in live:
        n = min(int(batch_size * b.max_ratio), len(b.items))
        if n == 0:
            continue
        picks += [b.items[rng.randrange(len(b.items))] for _ in range(n)]
        weights += [b.sample_weight] * n

    # 权重高斯归一化后做无放回抽样，保证每个来源都进 batch
    total = sum(weights) or 1.0
    norm = [w / total for w in weights]
    idx = rng.choices(range(len(picks)), weights=norm, k=min(batch_size, len(picks)))
    return [picks[i] for i in idx]

The point of this code is not the implementation, but two ordering judgments: filtering before oversampling, and ratio caps before weighted sampling. The former prevents contaminated samples from being amplified through the gradient; the latter prevents one data source from dragging the model off course during multi-stage training. These two matter far more than tuning sampling implementation details, yet almost no tutorial emphasizes them.

VIII. Vision-Language-Action Models and the Classical Control Stack: The Interface Is the Real Problem

8.1 Action Representation: From Discrete Tokens to Continuous Action Chunking

The thinking behind VLA-style models is very clear: reuse the entire pretraining of a vision-language model (massive image-text semantics, instruction understanding, commonsense) and swap only the output head for actions. The problem lies in the design of that output head, and in 2026 three approaches each have their own wins and losses.

Discrete tokenization: quantizing continuous actions into tokens by bucketing each dimension, reusing the language model's decoding head. The advantages are reuse of the training and inference stack and a natural quantization noise that acts as a regularizer. The disadvantages are that quantization error directly eats accuracy on precision manipulation, and that sequence length grows linearly with dimensionality, making inference latency noticeably worse. It suits low-dimensional, medium-precision multi-joint control.

Continuous regression: directly outputting normalized continuous actions through a Gaussian mixture or a diffusion head. Accuracy is good, but inference requires multiple samples and the output carries multi-modal uncertainty, which is less stable in engineering terms than the discrete option.

Action chunking: the current mainstream answer. Emit a sequence of N future actions at once, let the actuator advance through the sequence, and request a new chunk when it runs out or is interrupted. It solves three problems at once: inference frequency can sit far below control frequency (10 Hz inference with 50 Hz execution is entirely sufficient), inference latency is amortized across multiple actions, and the temporal locality of actions is explicitly modeled (actions in adjacent frames are naturally correlated).

8.2 Frequency and Latency: Engineering Details in the Middle Layer Decide the Feel

Action chunking introduces a new problem: discontinuity at chunk boundaries. The first action computed for a new chunk and the last action of the previous chunk may jump, which shows up as robot arm jitter (action jerk). This kind of jitter is not only ugly; it also significantly shortens hardware lifetime and reduces grasp success rate.

The standard fix is to do temporal alignment and smoothing at the execution layer. The following code demonstrates a minimal chunk executor, with the focus on how the resampling mode is chosen and how jitter is suppressed.

import numpy as np


class ChunkExecutor:
    """把低频推理的动作分块，平滑地执行到高频控制回路里。

    三个关键取舍：
    1) 用重采样而不是保持，配合一次低通滤波，消除块边界跳变；
    2) 剩余步数不足一半就请求新 chunk，让重规划发生在动作平缓段，
       而不是等到块用尽、在接缝处硬切；
    3) 滤波强度自适应：低速段可以重滤波，高速段必须保相位，
       否则反而会让动作滞后于指令。
    """

    def __init__(self, ctrl_hz: float, infer_hz: float, smooth: float = 0.25):
        self.ctrl_dt = 1.0 / ctrl_hz
        self.steps_per_chunk = int(round(ctrl_hz / infer_hz))
        self.smooth = smooth              # 0 = 不滤波，1 = 完全跟随上一动作
        self.buffer = np.zeros((0, 0))
        self.idx = 0
        self.last = None

    def load_chunk(self, chunk: np.ndarray) -> None:
        """chunk 形状为 [T, D]，T 应等于 steps_per_chunk 或其整数倍。"""
        self.buffer = np.asarray(chunk, dtype=np.float64)
        self.idx = 0

    def step(self) -> np.ndarray:
        # 剩余步数不足一半时提前补块，让重规划发生在动作平缓处
        if self.idx >= len(self.buffer) - max(1, self.steps_per_chunk // 2):
            raise NeedsNewChunk()

        target = self.buffer[self.idx]
        self.idx += 1

        if self.last is None:
            out = target
        else:
            # 速度自适应的低通：速度越大，滤波越弱，保住跟随性
            speed = float(np.linalg.norm(target - self.last))
            alpha = 1.0 - self.smooth * float(np.exp(-speed))
            out = self.last + alpha * (target - self.last)

        self.last = out
        return out


class NeedsNewChunk(Exception):
    """执行器主动请求新推理结果；由上层控制循环捕获并调用模型。"""

The most valuable engineering intuition here is this: the timing of replanning is itself a control design decision. Always waiting until a chunk is exhausted before replanning produces two jerks, one at the end of the chunk and one at its start. Replanning early, during a smooth segment, makes the jerk land where the motion itself is already smooth. This is the dividing line between "the model can be used" and "the feel of it can be used," and it is where systems of this kind differ the most between the lab and the production line.

8.3 Safety: Neither a VLA nor a World Model Is a Safety Controller

There is one reality that has to be stated plainly: the confidence of a language model is untrustworthy and cannot be used as a safety criterion. Letting a VLA decide for itself whether "the risk of my current action is acceptable" is equivalent to assigning a safety function to a model that carries no out-of-distribution guarantee. The usual approach on a production line is an independent shielding layer, and nothing on that path depends on any learned model:

• Hard constraints: joint position, velocity, and torque limits, workspace boundaries, singular configuration avoidance.
• Collision monitoring: independent detection based on real-time depth or joint encoders; on trigger, stop immediately, without consulting the policy.
• Watchdog: on inference timeout or out-of-range output, enter a preset safe pose as soon as the timeout fires, rather than holding the previous action.
• Human-robot interaction degradation: when a person is detected entering the safety zone, switch to a protective stop, without entering the model inference loop.

There is one requirement here that can never be tested in simulation: the system must handle "equipment anomalies," a category that does not exist in simulation — an encoder glitch on one joint, an air pressure drop halfway through a grasp, a collaborative arm being dragged by an external force. A substantial share of real deployment failure modes belongs to this category, and simulation training covers them with zero coverage. This is why acceptance testing on a real production line has to deliberately design "fault injection" cases.

IX. The Gap Between Benchmark Scores and Real Deployment

In papers about embodied AI, success rate is a nearly information-free metric. The reason is not that the metric was badly designed; it is that the semantics it carries are far too broad, and everything people actually care about has been averaged away by it.

9.1 Systematic Misalignment Across Three Dimensions

Offline versus closed loop. The vast majority of benchmarks are measured under an offline protocol: either "given a stretch of real observation history, predict some number of future frames," or "given a task, output an action sequence." Real deployment is closed loop: actions influence subsequent observations, and observations determine the next action. This feedback loop amplifies distribution shift from "a small perturbation" into "a structural difference." Offline metrics have very weak predictive power for closed-loop risk.

Average versus long tail. A benchmark reports one average success rate, but a production line cares about a lower quantile of the distribution — the worst 10% of scenarios. A system with "90% average success" may have far lower shop-floor usability than one with "75% average but evenly dispersed failures," if its failures concentrate on one specific reflective object type or one specific placement angle.

Completion versus usefulness. "Success" in a benchmark is usually a task-level binary judgment: did the thing end up in the right place. But industrial scenarios care about process quality — were there knocks, is it out of tolerance, did it affect the neighboring component, is extra rework needed. A system with a higher success rate that frequently causes slight damage may actually be worse in real cost terms.

What benchmarks usually report	What the production line really cares about	Judgment bias caused by the misalignment
Task-level average success rate	Lower quantiles of bucketed success rates	Overestimates reliability under specific disturbances
Offline prediction error (pixel / latent distance)	Correctness of action consequences in the closed loop	Mistakes visually plausible results for physically correct ones
Single-attempt success rate	Whether the system can self-recover after a failure	Overestimates real usability, because the production line has no unlimited retries
Fixed initial condition distribution	The real distribution of initial conditions on site	The benchmark distribution was cherry-picked, so extrapolation is overestimated
Whether it completed (boolean)	Process damage and tolerance	Ignores hidden quality costs

9.2 The Evaluation Misalignment of Generative World Models Is Especially Severe

Video-generative world models are currently evaluated mostly with visual quality metrics (fidelity, structural similarity, video perceptual distance, and the like). The correlation between that family of metrics and "correctness of action consequences" is quite weak, and may even be negative: a video in which the cup clips through the hand and drops onto the table has excellent image quality but is completely wrong physically.

A method of evaluation that reflects real capability much better is the counterfactual intervention test: fix the same initial scenario, change only one action, and check whether the difference in the model's output obeys physical laws. These tests can be written directly as automated criteria. One class is direction monotonicity: the larger the force applied by the action, the larger and more consistent in direction the object's displacement should be; if you see "more force, less displacement" or a reversal of direction, the model is fitting that region by correlation alone. Another class is irreversibility consistency: performing the same action twice from the same state should give the same result; if the two outputs differ significantly, the model lacks the determinism a closed loop requires. A third class is conservation relations: in stacking and collision scenarios, whether the total count of objects and the support relations are preserved. These criteria are closer to "can this model be used as a simulator" than any visual metric is.

It is worth emphasizing that these three tests are extremely cheap — no annotations, no human raters, only the qualitative relations that physical common sense gives you for free. They should be a standard acceptance item for any generative world model that claims interactive capability, and their actual adoption rate in 2026 is still very low.

9.3 An Evaluation Perspective Closer to Deployment

If only one thing could be changed, I would suggest switching evaluation from "report one success rate" to "report a success rate curve bucketed by initial-condition difficulty," and additionally reporting three metrics: the average number of human interventions per completed task, the worst-quantile duration of a single task, and the self-recovery rate after a failure.

Only together do these three metrics constitute "usability." The first measures how many people have to stand by; the second measures whether the cycle time can hold up on the production line; the third measures whether the system has any basic resilience. And they are precisely the metrics least often reported in papers — because inside a controlled benchmark environment they carry no meaning. This is the most plain and also the most effective dividing line between research and deployment.

X. If You Want to Ship Today: A Staged Engineering Path

10.1 Stage One: Do Not Introduce a World Model, Earn the Cost Back First

The vast majority of tasks that can be deployed today do not need a world model. When the task distribution is narrow, fault tolerance is high, and precision requirements are moderate, a feedforward vision-language-action policy plus a chunk executor, limiters and a collision shield, and a watchdog is enough to reach a usable level.

The core deliverable of this stage is not the model, but the data pipeline and the failure list. The most valuable information you get after really running for a few hundred hours is "where the task actually fails," and that list decides whether to invest in real robot data, in synthetic data, or in better perception. Time spent building that closed loop returns far more than chasing updates to model scale.

10.2 Stage Two: Where the World Model Starts Compounding

Once the data closed loop of stage one is running, three kinds of investment begin to pay off clearly:

Synthetic data filling the long tail. Turn the failure samples discovered in stage one into reproducible scene descriptions and manufacture them in bulk through generation or simulation. This application has the lowest bar — it does not require the world model to be "very true," only roughly plausible within the neighborhood of the failure, and that is far easier than chasing global physical fidelity.

System identification driving the simulation. Use real robot data to back-calibrate friction, latency, and compliance parameters, pushing the simulation from "it looks like" to "it moves like." This is pure engineering work, requiring no model innovation, but the payoff is direct and verifiable.

Short-horizon latent space planning. Use the world model as the policy's "internal simulator," doing a small number of branches of planning in latent space in place of single-point greedy actions. This brings real gains on tasks with occlusion, with latency, and where the consequences of an action have to be anticipated, and because the horizon is short and the dimensionality low, it is not dragged down by error accumulation.

10.3 When It Becomes Worth Betting on a Generative World Model

There are three signals that justify a heavy investment in the generative route, and the more of them you satisfy, the more you should commit: real robot collection is dangerous or impractical (energized, high temperature, toxic, requiring high-speed collisions); you need large-scale parallel environment interaction that real robots simply cannot provide; failure samples are more valuable than success samples (for example when training "avoid" behaviors, actively generating failures is far easier than collecting them).

Conversely, if your task's failures can be solved by simply collecting more data, if precision requirements are extremely high, and if genuine contact feedback is needed, then putting your weight on a generative world model today will most likely produce a system that looks beautiful in research and is useless in practice.

10.4 A Selection Checklist

Before committing to a technical route, the following questions are worth answering one by one, because their answers essentially determine which route you belong to:

• Can the initial condition distribution of the task be accurately enumerated? If so, build a feedforward policy first and do not touch a world model.
• Is the main cause of failure insufficient perception, or insufficient action precision? If it is the latter, a world model does not help.
• Does the task run longer than one minute? If so, solve error accumulation first rather than improving single-step prediction accuracy.
• Do you need samples that cannot be safely collected on real hardware? If not, the generative route has no irreplaceable reason to exist yet.
• Is there a reliable arm pose and force control interface? Without one, even a good VLA has nothing to exert itself on; fill in the stack first.
• Can the team absorb the system identification workload of sim-to-real? If not, put the budget into real robot data.

10.5 Three Things Not to Do

Last are a few explicit negative recommendations, all of them options whose cost gets underestimated.

Do not treat a world model as a fully automatic replacement for data augmentation. If its output lacks a filtering stage, whether manual or automatic, it will pour physically wrong samples straight into the training set — and because those wrong samples look very much like real data in distribution, the training metrics look normal and the problem only surfaces at deployment time.

Do not let a VLA drive the actuator directly without a safety shielding layer. A language model's output distribution has no hard boundary, while an actuator does. Adding limiters costs a few lines of code; going without them costs one incident.

Do not use visual quality metrics to endorse an interactive world model. This is the most common misuse of an evaluation metric, and because the metric is off the shelf and the report looks beautiful, it is highly persuasive in papers and slide decks — which is exactly what makes it dangerous.

XI. Conclusion: The Convergence Direction of the Three Routes

Returning to the three routes at the top, the end state most likely to hold in 2026 is a layered hybrid stack rather than any single route winning outright.

The bottom layer is a reliable 3D and semantic state representation, provided jointly by the priors of a vision-language model and by multi-view geometry. This layer aims for "good enough," not for "exact." The middle layer is short-horizon action-conditioned prediction, responsible for reasoning a few seconds ahead in latent space to support high-level decisions; its value comes from speed rather than from fidelity. The top layer is the classical control and safety stack, responsible for millisecond-level execution, contact-level manipulation, and failure handling. It does not learn, but it is the last link that makes the system trustworthy. Generative models play the role of data engine and scene orchestrator within it, rather than that of a real-time simulator.

The basis for this judgment has recurred throughout the article: physical information is missing from visual data, contact information is missing from any passive data, and long-horizon prediction is unreliable under current model capacity. These three are not temporary difficulties caused by insufficient engineering investment; they are boundaries dictated by the nature of the data itself. So the pragmatic path is not to wait around for an all-purpose world model, but to place every stage where it has an information advantage and to explicitly defend the stages where it does not.

The lesson for researchers is this: rather than chasing an all-purpose model that can roll out for hundreds of steps, it is better to spend the effort on one narrow and accurate capability — for example precise pre-contact pose prediction, or a correct response to a single counterfactual intervention. Narrow capabilities can make it into products; all-purpose models are still stuck in papers.

XII. The Engineering Stack: Making World Models Actually Run on a Robot

All of the preceding discussion of routes ultimately has to pass a very plain test: can this system, within the compute budget of the target robot, with sufficient determinism, deliver on time inside a closed loop? This section discusses no methods, only the few hard constraints that keep recurring when you turn methods into systems. They look trivial, but most production failures come from here rather than from the model architecture.

12.1 The Latency Budget Must Be Broken Down Before It Is Optimized

"Inference is too slow" is a complaint with no direction for action. The correct move is to decompose the latency budget first and write down where every millisecond of the control period goes: sensor capture and exposure, image preprocessing, vision encoding, world model rollout, action chunk post-processing, actuator command dispatch and communication, and the queueing waits at each of these stages. Only after quantifying these separately do you know what is worth optimizing.

Once the budget is broken down, you usually arrive at a counterintuitive conclusion: optimizing the single forward pass of the large model yields limited returns, and what actually needs optimizing is often the least conspicuous data movement. Copying images from device memory to the compute units, moving hidden states between modules, synchronization points on the host side — added up, these often eat more than a third of total latency, and the optimization measures are very direct: preallocating tensors, eliminating implicit synchronization, compiling fixed-shape preprocessing into the graph.

Stage	Typical magnitude	Optimization measures	Upper bound on gains
Image capture and exposure	Same order as the frame rate, not compressible	Raise the sensor frame rate, shorten exposure	Low, but often a hard floor
Preprocessing and data movement	Severely underestimated	Pinned memory, fixed buffers, avoid implicit synchronization	High, and cheap
Vision and backbone forward pass	Grows with model scale	Quantization, structured pruning, operator fusion	Medium to high, at the cost of accuracy
World model rollout	Grows with horizon and branch count	Shorten the horizon, reduce branches, cache	High, but it directly weakens capability
Actuator communication and dispatch	Depends on the bus and the protocol	Hard real-time bus, batched command dispatch	Medium, dependent on the hardware

The "upper bound on gains" column is the core reference for selection: note that shortening the world model horizon is at once a latency optimization measure and a capability reduction measure. This is not a defect; it is a reminder that every compromise you make to satisfy a latency budget should be made with a clear idea of what you are trading for what. Many teams quietly cut model capability in half here while believing they are doing performance optimization.

12.2 Asynchrony: The Most Important Architectural Decision in a Closed-Loop System

Making "world model rollout" a synchronous blocking call is the most common architectural mistake, and the design most likely to survive a demo environment and fall apart in production. The correct move is to split the system into two independent loops: perception and rollout (asynchronous, allowed to be delayed, allowed to drop frames) and control execution (synchronous, hard real-time, never waits).

The key point of this architecture is that the control loop must have the ability to keep going independently of the rollout result. When the next action chunk is not ready, what does the system execute? The answer to that question determines the resilience of the whole system. There are three common options: keep executing the tail of the previous chunk (simple, but it may already be stale), hold the last dispatched hold command (simple, but it breaks contact tasks), or switch to an independent conservative controller that takes over (complex, but the safest). For contact-type tasks, the third is usually mandatory.

There is one more easily overlooked point: rollout results must carry a timestamp and be checked for staleness. An action chunk computed several hundred milliseconds ago, when the closed loop has already advanced dozens of frames, has changed in meaning from "a plan for the next several frames" into "a late reaction to something that already happened." The system has to recognize this case and discard the chunk, rather than blindly executing it.

12.3 Determinism and Reproducibility: The Dividing Line for Debugging Experience

The debugging cost of a robotic system depends enormously on whether a failure can be reproduced. Floating-point differences caused by a nondeterministic reduction order, thread scheduling differences, timing differences under multi-stream execution — all of these make the same piece of code produce slightly different actions on two runs, which is exactly what produces "it worked this time and failed that time." Debugging on such a system becomes unacceptably slow.

Feasible measures include fixing the deterministic options for threads and operators on the inference side (at a certain performance cost), threading the random seed through the entire chain of data sampling and training, recording complete input state snapshots rather than just action logs, and distinguishing "observation data" from "internal model state" inside the logs. The last one is especially useful — if you log only actions and not internal model state, then when something goes wrong you can almost never tell whether perception, rollout, or execution was at fault.

12.4 A Pragmatic Resource Judgment

Finally, let us quantify "whether it is worth it." The judgment framework is simple: the value a world model brings depends on the number of real robot interactions it replaces multiplied by the cost of a single interaction, minus its training cost, its inference compute cost, and its long-term maintenance cost.

If the cost of a single real robot interaction on your task is extremely high (dangerous, scarce, requiring extensive manual resets), the numerator term is large and the return on a world model can be considerable. If the task can be solved with a few hundred real robot interactions, then a world model will most likely never catch up — because the number of interactions and iterations needed to train a good enough world model already exceeds the number needed to simply collect real robot data directly. This is an accounting problem many people refuse to work out, but working it out changes decisions.

XIII. Quick Reference: Common Failure Modes

The preceding sections discussed mechanisms; this section singles out a few of the most expensive failures, because they share one characteristic: the training metrics look completely normal, and the problem only surfaces at deployment. After each one, an observable signal is given that can catch it early.

13.1 Latent Space Collapse: Beautiful Loss, Latent Variables Carrying No Information

The most dangerous failure of a representational world model is representation collapse. Once the context encoder discovers that encoding every input into similar vectors drives both the reconstruction loss and the prediction loss very low, the model at that point gives nearly the same prediction for any input, which means it has learned nothing. Typical symptoms are the variance of the latent variable decreasing monotonically with training, or the prediction loss plateauing at a low value within the first few epochs.

There are two observable signals. First, check whether the variance of the latent variable stays healthy across different scenes; the standard approach is to compute the per-dimension variance of the latent space features, and if most dimensions are already near zero, collapse has begun. Second, look at the EMA update rate of the target branch: if it is too fast, the latent space degrades together with the context branch; if it is too slow, nothing gets learned. The order in which these two parameters should be adjusted is to increase the strength of the variance regularizer first, and only then slow down the target branch.

13.2 Suicide Outside the Distribution: One Wrong Step, and Everything After It Is Lost

Every rollout-style world model has this property: outside the distribution of real states, error grows superlinearly. The consequence is a cascading system collapse — an early perception error pushes the state out of the training distribution, the world model's prediction distorts accordingly, the policy acts on the wrong prediction, the real state drifts further away, and positive feedback forms.

The observable signal is "the correlation between predicted confidence and actual error": if the actual error is larger precisely in the region where the world model is confident, then the model has not learned to recognize its own ignorance. A workable mitigation is to add out-of-distribution detection to the latent variable and, on trigger, fall back to a conservative policy or request one fresh observation, instead of continuing to trust the prediction.

13.3 Generative World Models Treating Statistical Association as Causation

Offline video looks smooth, but inside the closed loop objects systematically drift, clip through geometry, and change count — the typical signature of a model that has learned correlation instead of causality. It is almost impossible to expose during offline evaluation, because there is no feedback loop.

The cheapest detection method is the counterfactual intervention mentioned earlier: fix the scenario, change only one action, and check whether the difference in output matches physical intuition (more force means more displacement, repeating an action gives the same result, the total number of objects is unchanged after stacking). These three criteria require no annotation at all, a few lines of script can run them automatically, and they should be an acceptance item for any interactive world model.

13.4 The Instruction-Following Trap of VLAs

A vision-language-action model inherits the instruction-following training of the language model, which is an advantage in a demo and a liability on the production line. When the scene conflicts with the instruction — the instruction says "put the cup on the table" but a cup of the same kind is already there, or the object the instruction describes is not in view at all — the model's output often drifts randomly among several plausible interpretations instead of clearly refusing to execute.

The production-line response is not to add logic inside the model, but to add a upfront feasibility and state confirmation stage in the architecture: first confirm with independent perception that the target exists and is unique, then confirm with rules or pose constraints that it is reachable, and only then hand an unambiguous instruction to the VLA. The VLA is responsible only for "how to do it," not for "whether it can be done."

13.5 Self-Contamination of the Data Loop

Once synthetic data enters the training set, the next batch the model generates carries the bias of the previous round, and the feedback loop amplifies it continuously. This is a risk shared by every self-generating data pipeline, and unlike ordinary overfitting it does not surface on a validation set — because the validation set comes from the same generator.

There are three defenses, and they must be used together: keep a real data anchor that bypasses the generator entirely, sampled from at every stage to monitor performance degradation; physical rule filtering, discarding samples that clearly violate conservation or geometric constraints; and periodic manual spot checks, because the filters in the first two can themselves fail, and automated filtering cannot replace random spot checks. Without the real data anchor, the other two only hide the contamination.

XIV. Conclusion: Compressing Everything Into a Handful of Decidable Statements

A world model is not a single technical category, but a combination of three capabilities — abstract representation, prediction of action consequences, and spatiotemporal memory — whose maturities currently differ, and whose information requirements differ as well. Having interactivity, real-time performance, and physical self-consistency all hold at once is, in 2026, still not the true state of any single system.

This compresses into six judgments that can be checked directly during selection. First, the representational route is already in products, while the generative route mainly plays a role in datasets and pretraining — not because the latter is less advanced, but because closed loop requires it to satisfy physical fidelity and millisecond-level inference at the same time, and those two conditions are currently mutually exclusive in cost. Second, the value of a 3D representation depends on whether the downstream task is "looking" or "touching": pick Gaussians or implicit fields for rendering, and fall back to occupancy or point clouds for collision and grasping, because the cost of choosing the wrong representation is far larger than the cost of insufficient precision.

Third, the difficulty of long-horizon consistency comes from distribution shift rather than numerical error, so the solution is a short horizon plus high-frequency replanning, not a bigger model. Fourth, contact-level capability is limited by missing information: vision contains neither force nor touch, this problem will not be solved by a larger model and can only be closed by adding sensors, so the realistic answer for precision assembly tasks is still classical force control plus a high-level model. Fifth, the cost bottleneck of real robot data lies in resets and scenario coverage, and misdiagnosing a data problem as a model problem is the most common and most expensive mistake on this route.

Sixth, the gap between benchmarks and deployment comes mainly from the evaluation protocol rather than from model capability, and swapping average success rate for a difficulty-bucketed success rate curve, and swapping visual metrics for counterfactual intervention tests, are the two highest-leverage actions available for improving decision quality.

For readers in different roles, the conclusion lands slightly differently. Those doing research should acknowledge that a narrow, accurate capability is easier to adopt than broad, vague generality, so when choosing a topic they should look first for scenarios where the available information already suffices for validation, rather than picking the hottest direction and then hunting for data. Those doing engineering will find that the most worthwhile investment is often not the model itself, but the data loop that collects real failures back and makes every iteration show the same failure list. Those making decisions should require that whenever any new system is evaluated, the first question asked is under which conditions it fails and how it knows it is failing, rather than looking first at its ranking on standard benchmarks.

Taking these six together with the route discussion above yields one unified judgment: the current bottleneck of embodied AI is not model capacity, but information and verification. Physical information is not in passive data, contact information is not in any cheap data, and the true distribution of failures is not in benchmark sets. These three points determine why a large amount of apparently leading capability cannot be converted directly into usable systems — and why teams that put their engineering effort into closing these information gaps (real robot collection, system identification, fault injection, real data anchors) usually make more solid progress than teams that simply scale up the model.