Schema-Adaptive Action-Conditioned JEPA for Cross-Machine CNC Transfer under Partial Sensor Overlap
arXiv cs.LGarXiv:2609.16071v1 Announce Type: new
Abstract: Cross-machine deployment of industrial world models requires transfer across changes in dynamics, sensing interfaces, sampling regimes, and control units. We study a schema-adaptive action-conditioned Joint-Embedding Predictive Architecture (SAAC-JEPA) for CNC dynamics, where the source machine has 17 canonical sensor channels and the target shares only 10. Evaluation uses group-disjoint source splits, source-only normalization, held-out self-supervised validation, unit audits, and a sealed target test after model locking. Across five seeds, JEPA pretraining gives no clean-source forecasting gain: scratch and pretrained-body models obtain \(\mathrm{RMSE}=0.811\pm0.022\) and \(0.813\pm0.022\). A source-only search over 20 candidates selects a schema-consistent action-conditioned JEPA after seven-seed stability checks. On the confirmatory target pass, the locked model reaches zero-shot \(\mathrm{RMSE}=0.546\), \(R^2=0.012\), and \(\mathrm{NLL}=0.52\), outperforming persistence but not RevIN-equipped PatchTST and iTransformer baselines (\(0.503\) and \(0.498\)). A pre-declared paired ablation shows that RevIN in the same architecture improves RMSE to \(0.495\pm0.004\) over three seeds, but degrades target calibration (\(\mathrm{NLL}=20.6\)) on stationary context windows. A pre-lock adaptation sweep further reduces RMSE to \(0.520\) with limited target support. These results show that source-domain forecasting accuracy alone is insufficient to assess industrial predictive representations, and that cross-machine adaptation under partial sensor overlap is a distinct evaluation axis.
Mem-World: Memory-Augmented Action-Conditioned World Models for Persistent Robot Manipulation
arXiv cs.CVarXiv:2606.18960v3 Announce Type: replace
Abstract: Action-conditioned world models have emerged as a promising paradigm for robot learning, offering a scalable alternative to costly real-world experimentation by generating action-consistent video rollouts. However, persistent world modeling remains challenging in manipulation: frequent end-effector occlusions and rapid wrist-camera motion make the current observation insufficient for predicting future views, causing models to forget or hallucinate scene details seen in earlier frames. Existing memory retrieval strategies often fail to identify informative history in dynamic manipulation scenarios. To address this limitation, we propose Mem-World, a memory-augmented multi-view action-conditioned world model. At its core, we present W-VMem, a 4D wrist-view-centered surfel-indexed memory that anchors historical observations to temporally evolving surface elements. By explicitly modeling when and where scene elements are observed, W-VMem enables geometry-aware retrieval of relevant history frames conditioned on future actions. During generation, relevant history frames are selected via surfel-based rendering and scoring, providing informative and non-redundant context for prediction. Extensive experiments show that Mem-World generates persistent rollouts in complex manipulation scenarios, enables more reliable policy evaluation than Ctrl-World, improving the Pearson correlation with real-world performance by 14.5\%, and supports effective policy improvement through synthetic data generation, increasing success rates from 58\% to 72\% on long-horizon tasks.
World-Action Models for Robot Learning and Control: A Survey
arXiv cs.CVarXiv:2609.16074v1 Announce Type: cross
Abstract: Robots operating in open environments act under partial observability, physical constraints, and dynamic task contexts. Beyond mapping observations and language instructions to actions, they must anticipate how candidate actions may affect future states and task-relevant outcomes. Recent advances in world models, video generation, and Vision-Language-Action (VLA) policies have motivated the development of World-Action Models (WAMs), which couple future world prediction with executable action generation. This survey provides a robotics-oriented review of WAMs. We clarify their scope relative to conventional world models, model-based reinforcement learning, action-conditioned video generation, and reactive VLA policies, and organize existing methods through a unified taxonomy covering representations, transition modeling, action interfaces, architectures, training pipelines, data modalities, and scaling strategies. We further review applications of WAMs in manipulation, navigation, and autonomous driving, and we summarize the datasets, benchmarks, metrics, and protocols used to evaluate WAM systems. Finally, we discuss key challenges in action alignment, world-action factorization, spatial and multi-view consistency, long-horizon memory, neural simulation for closed-loop policy learning, and efficient inference. Taken together, this survey aims to provide a concise technical foundation for integrating predictive world modeling with action generation, toward more reliable embodied robot intelligence. Project page: https://rcl-robotics.github.io/Awesome-World-Action-Models.
Unifying Semantic Priors and High-Frequency Traces: Enhancing V-JEPA with Mixture-of-Experts for Robust Synthetic Image Forensics
arXiv cs.CVarXiv:2609.16778v1 Announce Type: new
Abstract: The unchecked proliferation of manipulated images on social media platforms has increased the spread of misinformation, posing a severe threat to public trust and information integrity. Modern deepfake detectors typically rely on Vision Transformers (ViTs) to capture the low-level inconsistencies that characterize fully synthetic or locally tampered images. However, the global understanding of such foundation models is not enough to discriminate alone between real and fake multimedia content, especially in challenging scenarios where images are compressed or transmitted through social media. In this paper we pioneer the application of Joint-Embedding Predictive Architecture (JEPA) models to deepfake detection, taking advantage of the generalized representation of visual reality that such World Models have exhibited. We hypothesize, and empirically demonstrate, that the intrinsic world understanding of JEPA models can be used as a strong prior for a deepfake detector. To fully exploit JEPA capabilities, we propose MoE-JEPA, a dual-stream architecture for deepfake detection. By enhancing a V-JEPA 2 backbone with a Residual Mixture-of-Experts (MoE) mechanism, along with a noise stream branch, our model dynamically internalizes forensic knowledge. Furthermore, a Gated Attention Multiple Instance Learning (MIL) module is employed to ensure precise spatial semantic understanding. Evaluated on the SID-Set benchmark, comprising 300K AI-generated, tampered and authentic images, MoE-JEPA establishes a new state-of-the-art with an accuracy of 95.54%, successfully outperforming vastly larger models.
Neuro-Symbolic Synergy for World Modeling
arXiv cs.CLarXiv:2602.10480v4 Announce Type: replace
Abstract: Large language models (LLMs) exhibit strong general-purpose reasoning capabilities, yet they frequently hallucinate when used as world models (WMs), where strict compliance with deterministic transition rules--particularly in corner cases--is essential. In contrast, Symbolic WMs provide logical consistency but lack semantic expressivity. To bridge this gap, we propose Neuro-Symbolic Synergy (NeSyS), a framework that integrates the probabilistic semantic priors of LLMs with executable symbolic rules to achieve both expressivity and robustness. NeSyS alternates training between the two models using trajectories inadequately explained by the other. Unlike rule-based prompting, the symbolic WM contributes candidate-level scores through log-linear reranking, without requiring the LLM to interpret rule text. Rule-guided sampling prioritizes transitions that are weakly covered by symbolic rules, using 35--60% of the training pairs while outperforming full-data supervised fine-tuning in five of six settings. Experiments on ScienceWorld, WebShop, and PlanCraft demonstrate consistent gains in WM prediction accuracy and data efficiency; one-step lookahead on open-ended WebShop also improves agent reward. Our models, rules, and code are available at https://github.com/tianyi-lab/NeSyS.
对话蚂蚁灵波 CEO 朱兴：机器人还吃不了「粗粮」
爱范儿
机器人打拳跳舞火了一年，它什么时候才能替我们上班？
前段时间宇树发布人形机器人格斗演示，称首次实现由世界模型实时驱动的全自主搏击。据其介绍，UnifoLM-X2-1.0 能实时预测未来状态，完成规划、决策和动态交互。
自主格斗让我们看到机器人临场反应的进步，但如果换成在药房里取放商品这样的场景，另一组难题又来了：包装各异的药品、有人走动的货架通道、长时间运行中不断出现的意外。
这些日常任务，需要机器人把一次次判断变成持续、可靠的工作。
APPSO 在 2026 外滩大会现场跟蚂蚁灵波 CEO 朱兴进行了交流。聊到机器人为什么连小卖部都还没大规模落地，他直言：「小卖部其实也很难落地。我懂你的焦虑，今天还是能力和成本两个都有问题。」
与许多围绕整机产品展开研发的本体厂商不同，灵波把重心放在具身「大脑」上，希望同一套基础模型能够适配不同构型、不同任务，再通过后训练降低具体场景的使用门槛。重载作业和轻量分拣可以用不同的身体，智能则尽可能复用。
生成可交互世界的 LingBot-World 也是灵波技术探索的一部分。它还训练了面向具身的视频生成基座 LingBot-Video，并在此基础上训练动作模型 LingBot-VLA 2.0，同时建设数据管线、后训练和部署工具。
朱兴关心的是，这些投入能否让机器人用更少的数据学会干活。
我们在外滩大会上，也看到灵波大脑的机器人展示了药品拣选、物流分拣和工业上下料等操作。药房演示中的货架通道仅 80 厘米宽，相关方案已在国大药房零售门店部署。
灵波也保留了自己的 R 系列本体，用于研发和生活服务探索。做具身大脑，仍然得与硬件、具体场景一起磨合。
在现场交流中，朱兴谈了不少尚未解决的问题：小模型能完成特定任务，为什么还要花钱训练基模？数据越多，效果为什么可能越差？商业化又该推进到哪一步？
对于日常生活中产生的数据，他的判断很有意思：今天的机器人还吃不了「粗粮」。
以下内容整理自蚂蚁灵波 CEO 朱兴的媒体沟通会，APPSO 略经编辑。
一、小卖部，也没那么容易
Q：药房、上下料看起来都很普通。为什么不选一些更能体现技术水平的场景？
朱兴： 我们更在意的是跟生态合作伙伴一起，做一些实实在在的落地。今天要让它做一个很炫的事情，那也是一个 demo。我让团队搞两个月，可以让四个机器人打掼蛋，展示高精度、触觉等非常多的技术，也没太大意义。
我们看重的是真实应用、数据的产生。在这个过程中，让模型迭代循环起来，知道还有哪些不足，要提升什么。
产业也想搞既能规模化应用、又更帅的东西。做不了，模型能力还没到那一步，或者成本还没到那一步。
Q：连抓取、放置这样的简单操作，也很难做到 100% 成功吗？
朱兴： 要看场景、环境和任务本身操作序列的复杂度。
小卖部其实也很难落地。我懂你的焦虑，今天还是能力和成本两个都有问题。 模型能力、硬件稳定性，有些就是够不着。
今天整个模型还在冷启动阶段。基模干什么？是把小学生培养成初中生、高中生、本科生，现在可能还在比较早的阶段。
相对客观地说，现阶段想做一定规模的落地，环境不能太开放，任务不能太长程。任务太长，异常就多。今天主流可落地的产品基本还是模仿学习为主，最怕异常，因为没见过。还有成本约束，你创造的价值不够，自己又贵，长期也玩不转。
Q：既然如此，先把一个小模型训到能稳定干活，为什么还要投入通用基模？
朱兴： 如果阶段性只做一个非常特定的任务，也不追求规模化复制，直接怼一个小模型，也不是不能做。但你要看，规模化是不是意味着跨场景、跨任务、跨场所复制，这里面有泛化的挑战。
基模能力跟成功率不冲突。基模很大的价值，就是让后训练的少样本表现更好：用更少的数据达到同样的效果，或者用同样的数据达到更好的效果。 这个效果就包含成功率。
基模提升是水涨船高，跟大语言模型的基模发展，带动生态应用是一样的道理。
Q：从一家门店复制到另一家，现在卡在哪儿？
朱兴： 我们过去一段时间把模型从实验室拿出来，跟很多伙伴做落地，确实会遇到一个门店和多个门店复制的问题。现在复制成本很高，本质上还是泛化性不足。不同药房、不同便利店，环境有很多不一样。
短期可以先选没那么开放的环境，压缩泛化的挑战；任务不要太长程，先偏单任务。模型可以先驱动一部分环节，再到更多地由模型主导，能力成长要有一个爬坡过程。
二、矿泉水不能跳着舞落到手上
Q：现在很多做视频、游戏的公司都在谈世界模型。它们的能力，能直接用来控制机器人吗？
朱兴： 有些词的概念很泛。具身的世界模型，跟数字世界、游戏里的模型，客观讲也不是一种生物。以视频生成路线为基础的这类世界模型，底层用的是视频生成的原理，但数字世界和物理世界差别比较大。
数字世界的模型，满足的是内容需求：丰富、好看、创新、特效、画质。只要好，用户可以接受延迟。
机器人需要什么？它不需要那么好看，也不需要特效，需要的是合理，符合物理规律，因为最终要变成动作，控制机器人。
比如我现在想喝这瓶矿泉水，矿泉水在天空中跳个舞，落到我手上，肯定好看，但不符合物理规律，没意义。
机器人第一在乎是否符合物理规律，第二需要高性能、实时。 它在物理世界里不断输入、输出，不可能等你帮它生成一个很好的故事。
Q：灵波也尝试过在通用视频模型上微调。为什么后来决定从头做？
朱兴： 1.0 系列的时候，我们也是基于数字世界的通用视频生成模型，比如 Wan，通过微调把它硬调到适合机器人。
一个问题是上限不高。另一个问题是，调到最后会破坏前面模型的先验和知识，副作用就是泛化性降低。走到后期，我们真的是撞到南墙了。
所以我们很早就启动了更原生的研发。LingBot-Video 训练时加入了大量机器人真实数据，再基于它训练 LingBot-VLA 2.0。
我们相信物理智能需要基于物理真实数据，从 0 到 1 训练自己的基模。数字智能，包括多模态、大语言模型的发展，可能会给具身很多助力，但内核部分还要靠具身本身解决。
Q：从头训练基模，投入更大、风险也更高，为什么还要选？
朱兴： 因为我们希望长期发展，选择上限更高的路线。从 0 到 1 大规模训基模挺费钱的。
大语言模型兴起之后，有的选择基于别人的基模微调，有的冒更大风险投入预训练。选择后者也不见得跑得出来。今天具身也进入了「百模大战」初期，你选一个更高上限的路线还不见得做得出来，但不选择的话，肯定做不大。
Q：LingBot-World 2.0 开放了 1.3B 小模型。模型小了，幻觉和效果损失怎么办？
朱兴： 不可能说幻觉完全没有。模型小了这么多，效果完全保持一致，也不现实。
我们之前说过会开放更小的版本，希望对高交互世界模型感兴趣的人能更多地玩起来，这是想给技术社区做的贡献。
这次也开放了一部分训练技术。比如怎样训练单向因果的能力，机器人的时间一定是单向的，在线预测要按时间顺序使用已经发生的观测，不能依赖未来的真实帧。从现在到未来，不可能穿越。这些技术本身也有门槛，希望能帮助开发者和学术界。
三、「炼丹」背后，是数据的配方
Q：行业总在谈百万、千万小时数据。你相信 Scaling Law 吗？
朱兴： 我也相信 Scaling Law。但不能这么机械地看数据量。
现在已经有这种现象：一个模型用的数据比另一个多了 3 倍、5 倍，效果却差，但在某些类型的场景任务上表现得特别好，肯定是数据分布大量不均衡。我们做基模，谈的是一定的通用性。
量重要，但更要谈质量和分布。大家老说「炼丹」，炼丹背后就是配方，数据的配方。
我知道大家特别想大力出奇迹，觉得怼到多少小时，智能就涌现了，具身智能新时代就来了。这个事情不太简单。
Q：有人说数采厂 90% 的数据无效。问题出在数据供应商，还是模型公司？
朱兴： 我不知道这个数字是谁说的，怎么统计出来的。确实，不管哪种数据源、哪种采集方式，都有大量场景重复。过度重复的数据，意义没有那么大。
但作为模型方，你有义务定义需要什么数据、什么质量，不能把这件事赖到数采厂。
数据需求是模型训练来定义的：要什么类型，什么分布，覆盖哪些场景、哪些任务。难采的先试采，形成比较好的标准作业流程，再交给供应商上量。这是模型厂商很核心的经验和能力。
供应商则要把执行做好。要求采抓药瓶，就要把轨迹质量、平滑性、适当的泛化性做好。各有各的责任。
Q：灵波具体怎么做？
朱兴： 我们很早就转向定制化采集，不怎么购买成品数据了。成品数据容易有技术质量差、大量重复的问题，导致最终能用的比例很低。
我们跟供应商在作业流程、端到端数据管线的自动化集成上，做了很多工作。内部从原始数据到能进入模型训练的数据漏斗，可用率比较早就从 15% 左右拉到了 90% 以上。
还有一个特别关注的问题：数据从生产出来，到进入训练，到底要多久，是一周还是三天？我们跟供应商基本采用流式交付，不等一大批数据攒齐再交。
Q：想把质量握在手里，是不是最好自己建数采厂？
朱兴： 没必要非黑即白。
专业供应商有大规模用工的能力。管理很多生产员，怎么考核、激励，产能怎么保留弹性，这也是能力。至少灵波不擅长，也不想把自己搞得那么复杂。
未来数据要进入更多场景，进入场景本身也需要运营。我们把数据管线做得更高效，对数据理解得更深，再跟专业服务商配合，是 1 加 1 大于 2。
Q：第一人称视角的 Ego 数采很热。戴个头环、眼镜拍下人干活，就够了吗？
朱兴： Ego 是一个很好的思想，以人为中心，更便携，更容易进入开放场景，能够增加场景和任务的多样性。但要看具体怎么采。
头环加裸手，当前比较大的问题是精度。手的轨迹是重建出来的，误差过大的时候，数据价值就打折了。
我比较看好结合高精度、便携的触觉手套。既保留 Ego 的优势，又借助硬件和算法解决精度问题。还有一个重要的点，视觉和触觉的信息天然可以对齐，只要误差不要太大。这类数据对下一步模型发展意义很大。
至于设备有两个、四个还是六个相机，视角不同，都有用途。关键还是数据满足什么标准，轨迹精度够不够。
Q：互联网视频、仿真和真机数据，到底应该怎么配？
朱兴： 不要轻易谈不同数据源的配比。互联网视频操作数据那么多，配那么一点点真机数据，不就淹死了吗？
要分训练阶段。越往预训练，我们越希望获得场景泛化，所以用无本体数据、互联网视频等更容易扩大规模的数据。越到后训练，越要针对具体场景、具体任务把成功率拉上去，贴近机器人的数据就越有价值。落地前还要采数据，就是让模型熟悉机器人的构型、任务和环境。
仿真数据也有价值。我认为今天更多体现在中后训练，尤其是针对某个具体任务，强化成功率。每种数据都有阶段性的价值，不能混在一起只看一个总量。
四、真正的飞轮，要能接住失败
Q：数据一直要靠人专门去采。什么时候才能让机器人边工作，边给自己积累数据？
朱兴： 今天还很早，是冷启动，靠各种数采强制喂。再往前走，有一点应用规模了，至少部分场景里的异常数据飞轮可以转起来。
这时候看重的是异常数据，成功数据没有那么大意义。
Q：再往后，真正的数据飞轮会是什么样？
朱兴： 我认为什么才是具身智能真正的数据飞轮？其实每个人的日常生活或者工作过程中，都在为机器人生产数据。
Q：这些日常数据，现在能直接给机器人用吗？
朱兴： 你现在给它，它也吃不了，吃不了「粗粮」。
Q：既然关键在数据，是不是模型架构已经没那么重要了？
朱兴： 数据现在很重要，未来更重要。但模型范式没有完全收敛。世界模型的能力怎么用好，机器人未来的学习范式，都还要发生变化，必须持续探索。
不要把这个跟 Transformer 那一层的计算架构混为一谈。计算架构一换，大家都会换。模型范式有新东西出来，别人也会对齐。
所以要把数据管线做得更高效，对数据的理解更深。离开这个东西谈模型，没有意义。
Q：那大语言模型公司下场，会不会很快把这些优势抹平？
朱兴： 我打个比方，如果今天 OpenAI 或者 Anthropic 下场做自动驾驶，能短时间颠覆特斯拉吗？
大模型厂商在训练能力等方面有优势。但自动驾驶更核心的壁垒，还是数据：什么是好数据，对数据的理解，以及数据管线。
通用大模型可以提升机器人特定阶段的研发效率，是很好的工具支持。但具身面临的数据挑战还在那里，还是要自己解决。
五、人形，不必成为执念
Q：为什么把适配不同机器人构型，看得这么重？以后不会收敛到一种通用的人形机器人吗？
朱兴： 构型泛化是长期存在的问题，也是产业需要。
先看生产力场景。高负重的重载作业，和很轻的分拣，为什么要同一套硬件配置？为什么一定要同一种构型？有的单臂用六轴就可以，有的需要七轴甚至更多。要服务效率，背后是成本。
再看家庭，每家的第一需求也不一样，有的要老人看护，有的要陪伴，喜好也不同。人都有高矮胖瘦。
我觉得机器人没必要长得像人、什么都像人。 我们更应该看具身智能作为一项技术，能在社会里创造什么价值。把「人形机器人必须更像人」当成第一性原理，我不觉得是科学的方法。
Q：外骨骼这类产品好像不太需要大脑。一定要用大模型，才算好的机器人吗？
朱兴： 我们谈具身智能、谈大脑，一般希望它能泛化。有的场景不需要怎么泛化，完全可以用其他技术更稳健地解决，这是没问题的。
技术都是手段，最终满足场景需要是最关键的。 过去工业机器人基本上把能穷举的规则都做了，再往前，有些场景就做不了了。
Q：灵波做大脑，为什么还要做自己的 R 系列机器人？
朱兴： 研发肯定需要软硬一体。模型需要数据，数据由硬件产生，自己不做，也得找别人定制。
另外，蚂蚁集团的核心业务沉
An immune world model for multiscale forecasting and therapeutic hypothesis generation
arXiv cs.LGarXiv:2609.14709v1 Announce Type: new
Abstract: Immune therapies act across cell-intrinsic programs, tissue ecosystems, and patient-specific immune states, yet most predictors address these scales separately. We used a governed evolutionary AI Scientist to construct the Immune World Model, an action-conditioned model that learns how interventions move immune states across cellular, tissue, and individual levels. The Immune World Model--building Scientist searched candidate architectures and workflows, and the resulting world model was frozen before independent confirmation. The frozen model generalized to unseen interventions and biological contexts, recovered intervention-specific cellular programs, integrated cell and tissue information to improve ecosystem and patient-response prediction, and forecast unseen perturbation combinations. Immune World Model--guided analysis then combined measured perturbations with cross-axis inference to nominate IL-36$\gamma$ plus SIRP$\alpha$ inhibition as a complementary-axis therapeutic hypothesis, whereas a governed self-correction audit rejected every screened cytokine pair. The Immune World Model provides a framework for multiscale immune simulation that connects AI Scientist-driven model construction, intervention forecasting, and the generation of prospectively testable therapeutic hypotheses.
Exploring napping paradigm for Recurrent Spiking Neural Networks
arXiv cs.LGarXiv:2609.13927v1 Announce Type: new
Abstract: Biological organisms minimize free energy by balancing two competing demands on their internal world model: it must be accurate enough to predict sensory input, yet simple enough to generalize beyond it. Two mechanisms regulate this balance offline: sleep reduces complexity through gradual synaptic downscaling, while stochastic noise attenuates precision, relaxing the constraint sensory input imposes on synaptic reorganization. Engineered Spiking Neural Networks (SNNs) leave this balance unaddressed, favoring instantaneous, noiseless weight normalization instead. This paper investigates the hypothesis that a biologically inspired micro-sleep paradigm, napping -- combining proportional weight scaling with continuous stochastic membrane activity -- can replicate the stability of normalization while shedding model complexity. We evaluate this in an unsupervised recurrent SNN trained via trace-based spike-timing-dependent plasticity (STDP) on Gabor-preprocessed MNIST. We tune napping across three regularization regimes by sweeping its duration and membrane noise level, then compare the best configuration against weight normalization. Across all three regimes, well-tuned napping matches the accuracy of normalization: accuracy peaks at brief durations and low noise, then declines monotonically as either grows. Clustering diverges, with the strongest geometric separation arising at longer durations and higher noise -- the two terms of free energy pulling apart, accuracy rewarding data fit and structure rewarding the simpler representation that gradual, noisy downscaling induces. This gain carries a simulation cost normalization avoids, so napping is most compelling where representational structure, rather than raw classification efficiency, is the priority.
AlayaVista: Streaming World Modeling from Panoramic States to Perspective Video
arXiv cs.CVarXiv:2609.14462v1 Announce Type: new
Abstract: Interactive video world models must maintain broad scene context under camera motion while producing high-fidelity observations with low latency. Existing approaches face a representation trade-off: perspective models operate on local views and must preserve off-screen content over long rollouts, whereas broader spatial coverage is typically obtained by synthesizing full-sphere videos or constructing explicit 3D representations. Motivated by the complementary roles of global context and selective local acuity in visual perception, we present AlayaVista, a camera-controllable streaming video world model that decouples panoramic world evolution from perspective observation synthesis. Given a single perspective image, AlayaVista constructs a 360-degree scene prior using a pretrained panorama expansion model and then evolves the scene as a camera-conditioned panoramic latent state. A latent viewport renderer maps this state to the requested perspective video latents, while a perspective refiner restores details, suppresses artifacts, and performs super-resolution. To support efficient streaming, we adapt the panoramic generator to chunk-autoregressive generation and distill both panoramic generation and perspective refinement into few-step processes. To provide the supervision required by this design, we construct MUGEN, a large-scale real-world panoramic video dataset containing 1,318 hours of videos at resolutions of at least 4K, together with rich semantic and geometric annotations.
DreamQAS: Learning a Decision-Useful World Model for VQE-Efficient Quantum Architecture Search
arXiv cs.AIarXiv:2607.29491v2 Announce Type: replace-cross
Abstract: Reinforcement-learning-based quantum architecture search (RL-QAS) repeatedly invokes a variational quantum eigensolver (VQE) after each gate addition even though circuit transitions and action legality are known. DreamQAS preserves these exact dynamics and learns only expensive post-VQE feedback through a recurrent ensemble that predicts a frontier-relative feedback score without requiring the exact ground-state energy, enabling uncertainty-controlled multi-step imagination. Under a common 15,000-episode budget and frozen evaluation, DreamQAS has the lowest reported mean error among RL methods on all five main molecular tasks. At fine-error targets reached by all seeds of DreamQAS and a matched non-imaginative control, it uses 1.6-2.0 times fewer real VQE calls on four tasks. Holding LiH-4q feedback-model weights fixed, its imagined-policy actor attains 0.073 mHa, versus 4.280 mHa and 4.434 mHa for greedy and beam deployment. Learned-transition and end-to-end predictor controls further show that preserving exact circuit structure and using feedback through policy learning are both important. Counterfactual action-ranking improves throughout training on all five probed tasks, while ensemble disagreement improves risk-coverage over random rejection on three tasks. DreamQAS therefore learns decision-useful feedback for QAS without modeling already-known circuit dynamics or requiring the exact ground-state energy.
Policy and World Modeling Co-Training for Language Agents
arXiv cs.AIarXiv:2606.02388v2 Announce Type: replace-cross
Abstract: Reinforcement learning (RL) improves large language model (LLM) agents by teaching them which actions lead to high rewards, but provides little supervision on what those actions do to the environment. World modeling (WM) can fill this gap, yet existing approaches often require separate simulators, extra training stages, or additional inference-time computation. We observe that on-policy RL rollouts already contain the needed signal: each transition pairs an action with its resulting next observation. Based on this observation, we propose PaW, a Policy and World modeling co-training framework that adds auxiliary WM supervision to the same policy during RL, without changing the inference paradigm. To make auxiliary WM supervision informative and stable, PaW introduces three components: action-entropy-based WM data selection, noise-tolerant WM loss, and reward-adaptive loss balancing. Experiments on three agentic task benchmarks show consistent improvements over strong RL baselines across models and RL algorithms. These results suggest that standard RL rollouts are a practical source of WM supervision for language-agent training.
The Theory of Mind Utility: A Formal Account of Mentalizing
arXiv cs.AIarXiv:2606.12721v3 Announce Type: replace
Abstract: Inferring another person's beliefs requires reconstructing their information access history: what they encountered, in what order, from whom, and with what credibility. Existing formal accounts of theory of mind generally treat beliefs as given. We introduce the Theory of Mind Utility (ToM-U), a computational-level theory of how beliefs are formed from information-access history. ToM-U represents a person's epistemic situation as a Local Epistemic World Model (LEWM): a directed typed graph linking agents, information, and belief-like states. ToM-U generates candidate LEWMs, limits how deeply a person reasons about another's reasoning, and evaluates each candidate against observed behavior. A sufficiently coherent candidate is accepted, while a failed one leaves a structured residue that shapes later reasoning. Together, these components formalize a distinct cognitive process that operates across social reasoning. The architecture supports testable predictions about when mentalizing will fail and how those failures will affect subsequent attempts.
When the World Lies: Backdoor Attacks on Latent World Models for Downstream Control
arXiv cs.AIarXiv:2609.15781v1 Announce Type: cross
Abstract: Pretrained world models, learned simulators that encode an observation into a latent state and predict how it evolves under actions, are beginning to be reused as off-the-shelf dynamics backbones for control, like pretrained encoders and language models are reused today. We show that this reuse opens a supply-chain backdoor: an adversary who controls only a released checkpoint can hijack the downstream controller, even though the victim trains and evaluates entirely on clean data and never sees the trigger. The attack encodes no explicit trigger-to-action rule. Instead, the poisoned model routes trigger-bearing observations into a chosen latent region and reshapes the local dynamics there, so that the victim's own optimization (Dreamer-style actor training in imagination, or MPC/CEM planning over predicted futures) re-discovers the attacker's target action on its own. Across several control tasks and trigger families, the trigger steers the controller's action toward the attacker's target, controlling every action dimension and hijacking 100\% of triggered steps on the strongest settings. The checkpoint still passes the clean-data diagnostics a victim would run before deployment, with clean-task success retaining at least $\sim$75\%. The effect is temporally gated: it appears only while the trigger is present and disappears when the trigger is removed. Trigger-blind repair is budget-dependent: moderate clean fine-tuning can preserve clean utility while leaving the triggered failure intact, whereas sufficiently aggressive adaptation can remove it only after substantially degrading clean control. The world-model backbone itself is therefore an emerging and underexamined attack surface for control. The full code and artifacts are available in our repository.
Legislating World-Model-Based Planning with Legal Reasoning
arXiv cs.AIarXiv:2609.15113v1 Announce Type: cross
Abstract: As robotic systems grow more general, legal norms are needed to integrate them into society. This paper extends the isomorphism problem of aligning legal source texts with their encodings, and measures two key challenges to robot normative control: (1) the \textit{grounding isomorphism gap}, where perception error grounds false atoms for legal reasoning, and (2) the \textit{ontological isomorphism gap}, where one legal conclusion admits many faithful translations into planning constraints. The paper introduces a legal planning stack that employs Defeasible Deontic Logic (DDL) to constrain a motion planner. The stack leverages learned world models to plan and to provide legal context, enabling \textit{ex ante} governance that intervenes before an illegal action is executed. It was deployed on a simulated robot arm pushing a cube across a $3\times3$ grid. The findings were (1) the legislated agent abided substantially more often than the non-legislated one, and modeling perception uncertainty lifted abidance even further, (2) the legal reasoning ran efficiently at runtime and its verdicts were auditable, and (3) the stack adapted to exogenous signals and endogenous rule changes. Both gaps were measured: (4) world model and probe error corrupted the factual input for the DDL reasoner, and (5) a single law admitted several faithful metric interpretations yielding drastically different abidance. Thus, \textit{ex ante} legislation functions as intended, and closing these gaps with a standardized mapping from the law to runtime constraints and improved fact grounding from perception will yield robust laws that align robot behavior with society's norms.
LPA-CWM: A Learned Physical Adjudicator for Motion Reasoning with Counterfactual World Models
arXiv cs.AIarXiv:2609.14073v1 Announce Type: cross
Abstract: Counterfactual world models (CWM) extract motion from pretrained video predictors by comparing factual and intervened predictions. However, responses generated under different target-frame masks vary in reliability, while uniform aggregation weights them equally. We formulate response aggregation as candidate reliability learning and propose LPA-CWM with a lightweight Learned Physical Adjudicator (LPA). Trained on dense MOVi-F trajectories, the 3.0M-parameter LPA compares visual context and response structure across an unordered candidate set to predict relative weights, while the CWM predictor and intervention generator remain frozen. The weighted responses undergo windowed localization and one paired re-evaluation to recover motion. We also introduce Completeness-aware Motion Correspondence (CMC), a ground-truth-anchored evaluation protocol that jointly measures localization, trajectory completeness, visibility, and continuity, counting missing predictions as failures on visible dynamic points. On the evaluated DAVIS and Kinetics subsets, LPA-CWM improves $\mathrm{DCA}_{\mathrm{avg}}$ over Uniform CWM by 60.0\% and 29.0\%, respectively, and also improves tracking accuracy under TAP-Vid First. A quick overview is available at https://LPA-CWM.github.io.
LePlanner: An Iterative Amortized Controller For World Models
arXiv cs.AIarXiv:2609.13845v1 Announce Type: cross
Abstract: World models trained with joint-embedding predictive architectures learn compact, structured latent representations from physical interaction, yet planning in these latent spaces typically relies on one of two costly approaches. Search-based planners such as CEM, MPPI, and iCEM optimize action sequences through many predictor rollouts, achieving strong performance at the cost of high per-decision compute and latency. Policy-based methods amortize inference into a single forward pass but can degrade on contact-rich tasks where the demonstration distribution is multimodal. We propose LePlanner, an amortized iterative controller that learns to construct and refine latent action sequences through a frozen world-model predictor. LePlanner is trained with an arrival-and-hold objective that encourages the controller to reach the goal at the earliest feasible horizon and remain there. This addresses horizon-reset procrastination, a failure mode in which repeated receding-horizon replanning continually postpones goal arrival. An additional action-Gaussian loss keeps generated actions near the support of the offline dataset. Across navigation, contact-rich manipulation, and continuous-control environments, LePlanner matches or exceeds search-based planners while requiring an order of magnitude fewer predictor evaluations and 3-49x lower wall-clock time per decision. It achieves success rates of 98% on PushT, 100% on Reacher, 100% on TwoRooms, and 92% on the OGBench Cube task. These results show that much of the structure discovered through online search can be amortized into a lightweight iterative policy, enabling fast, horizon-aware, nonlinear physical control without online optimization.
Sampling headroom is not selection gain: a compute-value audit of test-time scaling for video world models
arXiv cs.AIarXiv:2609.13257v1 Announce Type: cross
Abstract: Test-time scaling (TTS) can improve generation only when additional compute produces better candidates and the system can reliably identify them. This distinction is especially important for video world models, where a wider sample pool may contain stronger rollouts without improving the output that is ultimately selected. We introduce the Compute-Value Audit (CVA), a sequential framework that asks whether extra sampling creates opportunity, observable signals provide a reliable state, that state supports a beneficial action, and the resulting gain exceeds the full entry fee of generation and verification. On 192 Physics-IQ scenes, expanding the pool from 4 to 16 candidates increases oracle quality by +9.23 IQ (95% CI [+7.44, +11.14]), but Flow, Cycle, and VideoReward fail to recover this headroom reliably. Across three generators, none of twelve adaptive-depth policies outperforms uniform compute; they recover only 42-69% of the measured entry fee. A matched-60-NFE Predict-and-Perturb intervention on VideoPhy2 is likewise negative across three fresh-seed replicas. These negative results are not universal: anchor-explorer passes all four stages in a sparse PRM800K setting, MMLU-Pro exposes the gap between predictive state and useful action, and a privileged paired future establishes a positive video upper bound. Together, these results show that sampling headroom has deployment value only when it can be converted into a reliable decision whose benefit survives the complete compute charge. Code is available at https://github.com/YuhuaJiang2002/sampling-headroom-is-not-selection-gain.
Diagnosing Faults in Reinforcement Learning Simulators and World Models with Canonical Polynomial Invariants
arXiv cs.AIarXiv:2609.13194v1 Announce Type: cross
Abstract: A large literature builds physical structure into learned dynamics on the premise that models respecting the underlying physics predict better. We test that premise using exact polynomial invariants recovered from trajectories and canonicalised as reduced Gr\"obner bases over $\mathbb{Q}$. On Acrobot, exactness provides little benefit for prediction: a consistency regulariser reduces algebraic residual while leaving rollout fidelity essentially unchanged, and a shaping potential recovered from a system with a 100% mass error accelerates learning as effectively as the correct potential. Exact canonical invariants instead prove valuable for diagnosis. We develop two procedures: screening, which identifies the violated physical constraint, and attribution, which recovers the faulty invariant and identifies the responsible physical parameter. To enable this, we introduce normal-form deflation and quotient-space recovery. Across fifteen injected faults, screening localises every broken constraint with no false alarms, whereas observation-space baselines do not localise any; attribution recovers the responsible parameter on all seven parameter faults. Paired difference tests detect all faults, showing that the advantage is localisation rather than detection. Perturbing reference generators by $10^{-4}$ preserves 14--15/15 localisations, showing that screening does not require exactness, whereas ideal-equality decisions distinguish perturbations of only $10^{-12}$, showing that exactness is required for algebraic comparison. Applied to 350 release pairs across eleven RL environments, the diagnostic finds no evidence of changed simulator dynamics, instead revealing properties of the benchmark implementations themselves.
When Should a World Model Move? Loss-Conditioned State Execution
arXiv cs.AIarXiv:2609.15801v1 Announce Type: new
Abstract: We introduce loss-conditioned state execution, a model-agnostic method that decides whether to execute a world model's fixed feasible proposal or retain the current state. Predictive informativeness alone, however, does not establish whether an update will reduce downstream loss. Occurrence ranking can approach perfection while persistence remains the unique absolute-loss Bayes action. Two transition laws can also share occurrence information and conditional variance yet require opposite absolute-loss decisions. We formalize state movability as the existence of a loss-reducing feasible correction and distinguish it from the benefit of a particular proposal. Our method constructs a loss-specific feasible proposal from a predictive distribution and evaluates its groupwise bounded-loss gain over persistence on independent calibration units. The proposal is executed only in groups with a positive simultaneous lower confidence bound. For fixed proposals and groups with bounded unit losses, we prove that every accepted group has lower expected loss than persistence with high probability when calibration units are i.i.d. draws from the target population. Experiments on public forecasting and action-conditioned dynamics benchmarks show supported updates and a trade-off between certification and coverage. On 28,684 held-out M4 Monthly series, the method executes the proposal for 14.0% of series and achieves bounded loss 0.588, compared with 0.599 for persistence and 0.621 for always executing the proposal. The paired 95% bootstrap intervals for both comparisons lie below zero. In constrained forecasting of six unhealthy-inventory types from JD$\mbox{.}$com, a leading e-retailer in China, strong occurrence-ranking signal coexists with a loss-based preference for persistence, illustrating why event predictability and state execution must be evaluated separately.
One Model, Two Physical Stories: Auditing Misalignment in Multi-Modal World Modeling
arXiv cs.AIarXiv:2609.14833v1 Announce Type: new
Abstract: World models, systems that generate what happens next given current environmental conditions, are increasingly being implemented with multi-modal generation in mind. However, generating multiple modalities simultaneously, such as visual simulations alongside physical state predictions in the form of text, introduces the risk of cross-modal inconsistency. Tested separately, both outputs may look convincing while still disagreeing: a model can calculate that a ball should rebound in one modality, then generate no rebound in another modality, to say nothing of diverging from real-world dynamics entirely. In this work we focus on two failures explicitly: \emph{Internal misalignment}, the disagreement between the world model's generated video and the same world model's prediction in a different modalities, and \emph{external misalignment} the disagreement between the world model's generation and an analytic physical environment. We derive common contracts of event, magnitude, timing, and construct a physics grounded pipeline to make comparisons measurable in both external and internal settings. We then ask whether progressively supplying the model's own contract (the A ladder for the internal setting) or a corrected physical contract (the B ladder for the external setting) closes the respective gaps. Across four mechanisms and 20 settings, we find that while language answers all 22 text probes correctly with respect to the true environment, the neutral video is often in disagreement, suggesting that the current unified backbones may not be capable of correct reasoning, internal consistency, and external physical fidelity all at once.
继续滚动加载更多…