NeoWorld-3: From Pixels to Code-Defined Physical Worlds

研究动机：像照片还不够

Motivation: matching a photo is only the start

如果机器人要把碗放进微波炉，模型除了“看起来像微波炉”，还必须有可用的内部空间、独立的门、合理的转轴和不会互相穿透的碰撞体。NeRF、3D Gaussian Splatting 与生成式网格解决了部分外观或形状问题，但这些输出本身不自动提供上述结构。

To place a bowl inside a microwave, a robot needs more than a microwave-looking model: it needs usable interior space, a separate door, a plausible hinge, and collision bodies that do not intersect. NeRF, 3D Gaussian Splatting, and generated meshes address aspects of appearance or shape, but their outputs do not automatically supply this structure.

现有 coding agents 能写建模代码并根据渲染结果迭代，却容易猜错精确尺寸，也可能输出开口网格、错误关节或相互穿透的零件。NeoWorld-3 的思路是把离散结构决策交给模型，把连续尺寸拟合交给优化器，把结果接纳交给确定性的检查程序。

Existing coding agents can write modeling code and iterate from renders, but they can guess dimensions poorly and produce open meshes, incorrect joints, or intersecting parts. NeoWorld-3 assigns discrete structural decisions to the model, continuous dimension fitting to an optimizer, and acceptance of results to deterministic checks.

依据：引言与相关工作。

Source: Introduction and related work.

核心方法：保留程序，再优化程序

Method: retain the program and optimize through it

1. 术语落到具体对象上

1. What the terminology means here

2. 用共享参数编码形状关系

2. Encode shape relationships with shared parameters

重建状态写作 P = (S, θ, T)，执行源码 S(θ) 得到几何。比如柜体与门共享同一个宽度参数时，优化这个参数会同时改变两者，而不会像自由移动网格顶点那样轻易破坏配合关系。添加缺失把手、改变连接方式或增加凹槽仍然需要修改 S，优化器无法凭空创造程序没有表达的结构。

The reconstruction state is P = (S, θ, T), with geometry generated by executing S(θ). If a cabinet and its door share a width parameter, fitting that parameter changes both together and helps preserve their relationship, unlike unconstrained vertex edits. Adding a missing handle, changing a connection, or introducing a recess still requires editing S: the optimizer cannot invent structure outside the program’s representation.

在主文公式中，程序、位姿和相机固定，只拟合有界参数：

In the main-text formulation, the program, poses, and cameras remain fixed while bounded parameters are fitted:

\[\theta^\star=\arg\min_{\theta\in\mathcal B}\mathcal L_{\mathrm{obs}}\left(\{\mathcal R_v(\mathcal E_S(\theta),T)\}_{v\in\mathcal V},\mathcal O\right).\]

这里 E 是程序的可微执行，R 是各相机视角的渲染，损失比较轮廓、边界、特征及可用深度。共享参数通过同一条依赖链影响多个零件。附录中的实际搜索先用 cross-entropy sampling，再从优选样本运行 Adam；场景路径还可联合拟合受允许的位姿及相机修正，因此不能把整个系统简化成固定相机下的一次梯度下降。

Here E is differentiable program execution, R renders each camera view, and the loss compares silhouettes, contours, features, and available depth. Shared parameters affect multiple parts through the same dependency chain. The implementation first uses cross-entropy sampling, then Adam from elite samples; the scene route can also fit permitted poses and camera corrections jointly. The complete system therefore goes beyond one gradient descent run with fixed cameras.

3. 在快速几何副本中搜索，在 Blender 中确认

3. Search in a fast geometry replica, confirm in Blender

1. 建立基线：在 Blender 生成原生几何，并检查 tensor twin 与其一致。Torch 执行受支持的操作，nvdiffrast 批量渲染候选参数。 Establish a baseline: build native geometry in Blender and check that the tensor twin agrees. Torch executes supported operations and nvdiffrast renders candidate parameter batches.
2. 有界搜索：只改变选中的参数，并遵守参数范围与已执行分支的限制。不受支持的操作会返回明确失败；任意 Blender Python 并不会自动变得可微。 Search within bounds: change selected parameters while respecting their ranges and the calibrated program branches. Unsupported operations return explicit failures; arbitrary Blender Python does not become differentiable.
3. 物化与接纳：在 Blender 中构建选出的候选；平均 silhouette IoU 至少增加 0.002，最弱视角最多下降 0.02，其他适用的轮廓、深度和网格拉伸检查也必须通过。失败就保留旧状态。 Materialize and admit: build selected candidates in Blender. Mean silhouette IoU must rise by at least 0.002, the weakest view may fall by at most 0.02, and applicable contour, depth, and mesh-stretch checks must also pass. Failure preserves the previous state.

这套规则限制的是测得的退化；它既不保证每个视角都改善，也不证明未观测到的背面正确。物体接回场景时另有规则：平均 IoU 不下降、最弱视角下降不超过 0.02，并验证周围未被委派的物体没有改变。

These rules limit measured regressions; they neither guarantee improvement in every view nor establish correctness of unseen surfaces. Reintegration of an object into a scene uses a separate rule: mean IoU must not fall, the weakest view may fall by at most 0.02, and surrounding objects outside the delegation must remain unchanged.

4. 分层重建与物理检查

4. Hierarchical reconstruction and physical checks

场景协调器负责整体布局、相机与物体身份，将某个物体的多视角裁剪、掩码及遮挡证据交给建模助手。助手返回独立程序，协调器在暂存环境中检查并接入。重复物体可共享建模程序，同时保留各自位姿。RoboCasa365 输入是 16 张 RGB 图和已知相机，没有传感器深度；所用深度由 MoGe-3 预测并做跨视角尺度拟合。

The scene coordinator manages layout, cameras, and object identities, delegating an object’s multi-view crops, masks, and occlusion evidence to a specialist. The specialist returns a separate program, which the coordinator checks in a staging environment before integration. Repeated objects can share one construction program while retaining individual poses. RoboCasa365 inputs contain 16 RGB images and known cameras, without sensor depth; MoGe-3 predicts depth, followed by cross-view scale fitting.

物理相关检查分层进行：网格连接性检查及明确的接缝修复、关节运动采样、导出运动学一致性检查，以及带碰撞体和物性假设的仿真。接缝修复只重接可识别的重合顶点索引，不填洞，也不合并独立运动部件。关节扫描可以发现矛盾；当信息缺失时，“没有被已有测量否定”不等于真实机构已被验证。

Physical checks operate at different levels: mesh connectivity and explicit seam repair, sampled joint motion, exported-kinematics consistency, and simulation with collision bodies and assumed physical properties. Seam repair reconnects identifiable coincident vertex indices; it does not fill holes or merge independently moving parts. Joint sweeps can expose contradictions, but when evidence is missing, “not refuted by available measurements” does not establish the real mechanism.

5. RSI：把重复代码变成候选工具

5. RSI: turn repeated code into candidate helpers

这里的 recursive self-improvement 是跨任务的工具开发流程：收集执行轨迹及任务后建议，识别反复出现的代码，提出纯算术 Python helper，再验证原生执行、tensor twin、导出和配对任务表现。运行中的任务始终使用冻结版本的库。附录 A.5 的初步试验没有产生被采纳的扩展：一次没有修改，另一次 helper 通过执行验证但未在配对任务中使用。因此 RSI 是已描述的机制，不是已证实的性能增长来源。

Recursive self-improvement here is a tool-development process across tasks: collect execution traces and post-task advice, identify repeated code, propose pure arithmetic Python helpers, and test native execution, tensor replay, export, and matched-task behavior. Each running task uses a frozen library snapshot. The initial trials in Appendix A.5 produced no adopted extensions: one made no change, while another helper passed executable validation but was unused in the paired task. RSI is a described mechanism, not an established source of performance growth.

依据：附录 A.2–A.6；实现细节另见附录 C。

Source: Appendix A.2–A.6; implementation details also appear in Appendix C.

主要实验结果：改善来自哪里

Results: where the gains appear

以下数字来自 v1 的表 2–5，均为作者报告，未在本研究中重新运行。物体级比较使用 GPT-5.6 Terra，场景级比较使用 GPT-6 Sol；每个设置内各方法使用相同模型。除 Exec 外，3DCodeBench 指标只在成功执行的资产上计算。粗体表示表中最优值，包含并列。

The following numbers are author-reported results from Tables 2–5 of v1; they were not rerun for this study. Object-level comparisons use GPT-5.6 Terra and scene-level comparisons use GPT-6 Sol, with the same model across methods within each setting. All 3DCodeBench metrics except Exec are calculated only on successfully executed assets. Bold marks the best value in each column, including ties.

SigLIP-2 与 DINOv3 比较图像特征相似性；Uni3D 比较 3D–3D 或图像–3D 表征；Framed IoU 衡量配准后的轮廓重叠。它们共同提供外观、语义与形状证据，但任何单项都不能证明真实物理行为。

SigLIP-2 and DINOv3 compare image-feature similarity; Uni3D compares 3D–3D or image–3D representations; Framed IoU measures aligned silhouette overlap. Together they provide appearance, semantic, and shape evidence, but no single metric establishes real physical behavior.

1. 静态物体：212 个 3DCodeBench 样本

1. Static objects: 212 instances from 3DCodeBench

相对 VIGA，Framed IoU 从 0.6547 升至 0.7550，增加 0.1003；CD 从 0.0680 降至 0.0365，按两值计算下降约 46.3%。CD 是该基准协议中的几何距离，不能直接当成厘米。Exec 表示程序能执行，并不表示物体在物理上正确。

Relative to VIGA, Framed IoU rises from 0.6547 to 0.7550, an increase of 0.1003; CD falls from 0.0680 to 0.0365, a calculated reduction of about 46.3%. CD is a geometry distance under this benchmark’s protocol, not a measurement directly interpretable in centimeters. Exec measures whether the program runs, not whether the object is physically correct.

2. 可动关节物体：77 个 PartNet-Mobility 样本

2. Articulated objects: 77 PartNet-Mobility instances

AS-gIoU 在此定义为 1 − gIoU，是误差，所以越低越好。AOR 衡量部件重叠，报告的 0.0000 对应作者在评估中未检测到穿透；它不保证所有连续运动、未知物体或真实动力学都无碰撞。PartNet 输入没有已知相机，因此该路径无法使用与照片配准的数值细化，这也限制了把所有关节改进归因于梯度优化的解释。

AS-gIoU is defined here as 1 − gIoU, an error for which lower is better. AOR measures part overlap; the reported 0.0000 accompanies the authors’ statement that no penetration was detected in the evaluation. It does not guarantee collision-free continuous motion, unseen objects, or real dynamics. PartNet inputs lack known cameras, so photograph-registered refinement is unavailable in that route; this also limits attributing every articulation gain to gradient optimization.

3. 完整场景：10 个 RoboCasa365 场景

3. Full scenes: 10 RoboCasa365 scenes

Complete、Layout 和 Function 是多模态模型评审分数，分别针对完整性、布局及功能合理性，不能当成机器人成功率。P-Pen 是重建的可动关节物体中存在部件相交的比例；VIGA 没有对应结构，因此不适用。NeoWorld-3 的 Complete 并非最佳，且 OCD 仍为 25.9445 cm；场景平均指标不能替代抓取所需的局部精度。

Complete, Layout, and Function are multimodal-model ratings of completeness, spatial arrangement, and functional plausibility; they are not robot success rates. P-Pen is the fraction of reconstructed articulated objects with intersecting constituent parts; VIGA lacks the required structure, so the metric is inapplicable. NeoWorld-3 is not best on Complete, and its OCD remains 25.9445 cm. Scene-average metrics cannot substitute for the local precision needed to grasp an object.

4. 工具替换实验：比“多一个 agent”更具体

4. Tool replacement: evidence beyond adding another agent

完整系统相对工具替换版本，DINOv3 从 0.7644 升至 0.8142，OCD 从 47.4549 cm 降至 25.9445 cm。这个实验同时替换 NeoMCP 和 NeoSDK，并非分别移除梯度细化、接缝修复和 RSI 的因果消融；表 5 不能单独确定每个组件贡献多少。

Compared with the tool-replacement configuration, the full system improves DINOv3 from 0.7644 to 0.8142 and reduces OCD from 47.4549 cm to 25.9445 cm. This experiment replaces NeoMCP and NeoSDK together. It does not individually remove gradient refinement, seam repair, and RSI, so Table 5 cannot isolate each component’s contribution.

依据：第 4 节，表 2–5。

Source: Section 4, Tables 2–5.

机器人实验：几何提供计划，反馈补偿误差

Robotics: geometry supplies the plan, feedback handles error

平台是两台 AgileX Piper 机械臂及腕部 RGB-D 相机。首先用标定板建立相机、场景与机器人之间的米制坐标关系，再在 MuJoCo 中规划和检查碰撞。执行前用腕部观察修正目标位姿；执行中以 100 Hz 控制循环和电机电流估计的接触力进行反馈。质量、摩擦和执行器参数需要另行配置。

The platform uses two AgileX Piper arms with wrist RGB-D cameras. A calibration board establishes a common metric frame for the cameras, scene, and robot, followed by planning and collision checks in MuJoCo. Wrist observations correct target poses before execution; a 100 Hz control loop uses contact forces estimated from motor currents during execution. Mass, friction, and actuator parameters are configured separately.

开门使用顺应性拉动：初始只指定方向和距离，控制器根据估计的接触力转动拉动方向，不需要准确的铰链模型。真实微波炉门扣释放时需要 10–15 N，之后为 0–2 N。放碗与关门使用针对该布置录制的示教动作，所以“training-free”或“zero-shot”不应被解读为没有标定、示教或反馈。

Door opening uses a compliant pull: only an initial direction and distance are specified, and the controller rotates the pulling direction from estimated contact force without requiring an accurate hinge model. The real microwave latch releases at 10–15 N, followed by 0–2 N pulling force. Bowl placement and door closing use demonstrations recorded for this layout, so “training-free” or “zero-shot” should not be read as absence of calibration, demonstrations, or feedback.

依据：附录 B.1–B.3；主文第 4.6 节。

Source: Appendix B.1–B.3; main-text Section 4.6.

局限与研究判断

Limitations and assessment

怀疑者会如何概括它？

What would a skeptic reduce it to?

一个合理的怀疑是：“这是一套 LLM 驱动的程序化 CAD、逆图形优化与机器人集成系统，很多底层组件都已存在。”这并没有否定价值。本文较具体的贡献在于让同一份可编辑程序贯穿模型修改、tensor 搜索、Blender 物化和受检查的场景整合，将视觉拟合与可交互资产的结构要求连接起来。这是本研究对贡献的判断，不是论文已经证明所有组成部分都首次出现。

A reasonable skeptical reading is: “This integrates LLM-driven procedural CAD, inverse graphics optimization, and robotics, using many existing components.” That does not erase its value. The concrete contribution is carrying the same editable program through model edits, tensor search, Blender materialization, and checked scene integration, connecting visual fitting to the structural requirements of interactive assets. This is this study’s assessment, not proof that every component is unprecedented.

• 外观无法确定隐含动力学。质量分布、重心、摩擦和隐藏关节限制无法只从外观确定。benchmark 导出、benchmark 评估器与 physics-in-the-loop 还分别使用不同质量假设：1000 kg/m³、500 kg/m³，以及每个物体 1 kg。低穿透率并不证明真实接触动力学被恢复。 Appearance underdetermines hidden dynamics. Mass distribution, center of mass, friction, and hidden joint limits cannot be established from appearance alone. Benchmark export, benchmark evaluation, and physics-in-the-loop also use different mass assumptions: 1000 kg/m³, 500 kg/m³, and 1 kg per body, respectively. Low penetration does not establish recovered real contact dynamics.
• “simulation-ready”取决于导出路径。visual URDF 只有视觉和运动学元素，没有碰撞、惯性或动力学元素。场景 benchmark 的 --watertight 检查默认关闭；完整 metric solid certification 是独立 SDK 接口，不能把能力支持表当成每个结果都经过完整认证。 “Simulation-ready” depends on the export route. Visual URDF includes visual and kinematic elements but no collision, inertial, or dynamics elements. The scene benchmark’s --watertight check is off by default, and full metric solid certification is a separate SDK interface. A capability table does not mean every output received full certification.
• 证据规模和归因有限。场景评估只有 10 个场景，部分评分依赖多模态模型；表 2–5 未提供置信区间。物理试验是开发演示，RSI 初步试验没有被采纳的扩展，工具替换实验也没有拆开各组件贡献。 Evidence scale and attribution are limited. Scene evaluation covers only 10 scenes, some scores depend on a multimodal model, and Tables 2–5 provide no confidence intervals. Physical trials are development demonstrations, initial RSI trials adopted no extensions, and tool replacement does not separate individual component contributions.
• 几何表达与延迟限制应用。可微执行只覆盖受支持的建模操作，错误的形状结构仍需代码修改。报告的场景配置允许 64 次作者调用及 10,800 s 时间上限；这是预算而非平均耗时。快速位姿更新也不能代表从采集到重建的全流程实时性。 Representation and latency constrain deployment. Differentiable execution covers supported modeling operations, and an incorrect shape structure still needs a code edit. The reported scene configuration permits 64 author calls and a 10,800 s wall-time limit; this is a budget, not mean runtime. Fast pose updates likewise do not establish real-time capture-to-reconstruction performance.

下一步最值得验证什么？

What should be tested next?

作者提出用主动观察、推碰或触觉估计隐藏物性，并通过缓存程序先验和局部更新减少重建延迟。作为额外评估建议，应固定模型、输入和预算，分别消融数值细化与物理检查，报告实际端到端耗时，并在新布局上统计无需重新示教的任务成功率；只有这样才能区分重建质量、控制器补偿与人工先验各自的作用。

The authors propose active observation, pushing, or tactile sensing to estimate hidden physical properties, and cached program priors plus local updates to reduce reconstruction latency. Additional evaluation should hold the model, inputs, and budgets fixed, ablate numerical refinement and physical checks separately, report actual end-to-end latency, and measure task success in new layouts without new demonstrations. This would separate reconstruction quality from controller compensation and human priors.

依据：第 5 节；相关限定见附录 A.4、A.5 和 B。

Source: Section 5; related qualifications in Appendices A.4, A.5, and B.

来源与阅读路线

Sources and reading guide

本研究以 2026-10-20 发布的 arXiv v1 为准。先读第 3 节理解程序表示与优化，再读表 2–5 评估改善，最后对照附录 A.4、A.5 和 B，核对物理验证、RSI 与实机迁移的范围。插图均来自原论文；交互图由表 5 的数值重绘。

This study follows arXiv v1, submitted on 2026-10-20. Read Section 3 for representation and optimization, Tables 2–5 for measured improvements, then Appendices A.4, A.5, and B for the scope of physical validation, RSI, and robot transfer. Images come from the paper; the interactive chart reproduces values from Table 5.

作者提供的 NeoWorldStudio 代码链接与项目页面可用于继续探索；本研究的实验结论来自论文，没有进行代码复现或独立硬件验证。

The authors’ NeoWorldStudio code link and project page provide starting points for further exploration. Experimental conclusions here come from the paper; no code reproduction or independent hardware validation was performed.