RoboSnap: One-Shot Real-to-Sim Scene Generation for Generalizable Robot Learning and Evaluation

研究动机：漂亮的重建为什么还不够？

Motivation: why a good-looking reconstruction is insufficient

如果杯子看起来在桌上，碰撞网格却悬空或陷入桌面，机械臂的抓取就会失败。只追求输入视角的像素相似度，还可能把背景保留成静态图像，使相机移动后的观测失真。机器人学习需要同时满足接触合理、视角可变、场景可编辑。

A cup can look correctly placed while its collision mesh floats above or intersects the table, causing a grasp to fail. Optimizing only input-view similarity can also leave a static background that becomes incorrect as the camera moves. Robot learning needs plausible contact, movable viewpoints, and editable scenes together.

关键设计取舍是只给交互区域配置可碰撞几何，把周围环境交给可重新渲染的视觉层。这减少了对整个房间进行精确物理重建的需求，但也限定了机器人可以可靠交互的范围。 §3

The key tradeoff is to assign collision geometry to the interaction region and a re-renderable visual layer to the surroundings. This reduces the need for physically precise reconstruction of the entire room, while limiting the region in which reliable interaction is supported. §3

核心方法：从照片到可执行场景

Method: from a photograph to an executable scene

术语速查：每个词在这里具体指什么

Terminology: what each term means in this system

1. 先恢复可接触的物体，再统一坐标

1. Recover contactable objects and align coordinates

VLM 识别交互区域中的物体和支撑平台；SAM 3 提取实例 mask；SAM 3D 为每个实例生成带纹理网格、尺度和初始位姿。VGGT 预测相机几何及稠密点图；系统在 mask 内选取高置信度点，通过固定尺度的粗到细 ICP调整旋转和平移，拒绝过大的更新。

A VLM identifies objects and the support platform; SAM 3 extracts instance masks; SAM 3D generates a textured mesh, scale, and initial pose for each instance. VGGT predicts camera geometry and a dense point map. High-confidence points within each mask guide coarse-to-fine ICP with fixed scale, correcting rotation and translation while rejecting excessive updates.

对支撑平台进行 RANSAC 平面拟合，再以最小二乘细化，把法向对齐到世界坐标的 +z，平台中心放在原点。DROID 的机器人基座使用数据集提供的相机到基座标定；未标定照片则需依据平台及指定的机器人朝向初始化。因此，“单图”描述的是场景重建输入，不意味着部署时完全不需要标定或先验。

RANSAC plane fitting followed by least-squares refinement aligns the support normal with world +z and places the platform centroid at the origin. For DROID, robot placement uses the dataset’s camera-to-base calibration; uncalibrated photographs require initialization from platform geometry and a specified robot-facing direction. Thus, “single image” describes the reconstruction input, not deployment without calibration or priors.

2. 背景负责看起来真实，关节结构依赖类别先验

2. Reconstruct visual context and use priors for articulation

去除交互区域后，Gemini-2.5-flash-image 补全背景；论文使用生成式 world model（参考文献中的 Marble）创建 Gaussian-splat 场景，再经稀疏对应和 ICP 对齐到相同世界坐标。对微波炉等可动关节物体，P3-SAM 分割部件，系统从标准关节物体数据集中检索同类别的运动学参数。隐藏表面和关节结构包含生成与检索先验，并非全部从照片直接测得。

After removing the interaction region, Gemini-2.5-flash-image completes the background. The paper uses a generative world model, identified as Marble in its references, to create a Gaussian-splat scene, then aligns it to the shared world frame using sparse correspondences and ICP. For articulated objects such as a microwave, P3-SAM segments parts and the system retrieves category-level kinematic parameters from an articulated-object dataset. Hidden surfaces and joint structures therefore depend on generative and retrieval priors rather than direct measurement from the photograph.

3. 交替做几何纠错与重力沉降

3. Alternate geometric correction and gravity settling

GPT-4V 在 5 次随机 Set-of-Mark 标注上预测物体关系，以多数票构建 Support 和 Contact 边。只支撑别人、不被其他物体支撑的根节点固定不动；其余物体可调整位姿。先修复网格并预计算 SDF，再用 V-HACD 生成凸碰撞包络。

GPT-4V predicts object relations from 5 randomized Set-of-Mark overlays; majority voting produces Support and Contact edges. Roots that support others without being supported are fixed, while the remaining objects can change pose. Meshes are repaired and SDFs precomputed, and V-HACD constructs convex collision hulls.

1. SDF 阶段：优化非根物体的平移和旋转，兼顾穿透、支撑、接触与偏离初始布局的惩罚。 SDF phase: optimize non-root translations and rotations using penetration, support, contact, and deviation-from-initial-layout penalties.
2. Physics 阶段：把更新后的位姿送入 SAPIEN，清零速度，在重力下沉降；初期强烈抑制水平运动，避免碰撞冲量把物体弹飞。 Physics phase: load the updated poses into SAPIEN, reset velocities, and settle under gravity; suppress early horizontal motion to prevent collision impulses from launching objects.
3. 重复：把沉降后的位姿作为下一轮起点，直到位移足够小或达到轮数上限，得到最终场景。 Repeat: use the settled poses to initialize the next round until displacement is sufficiently small or the round limit is reached.

\[\mathcal L=w_{\rm pen}\mathcal L_{\rm pen}+w_{\rm sup}\mathcal L_{\rm sup}+w_{\rm con}\mathcal L_{\rm con}+w_{\rm reg}\mathcal L_{\rm reg}.\]

这相当于先把“杯子穿进桌面”这样的几何错误修掉，再检查它是否真的能站稳。正则项限制修正幅度，尽量保持原始布局。 Appendix B

This first corrects geometric errors such as a cup intersecting the table, then checks whether the result can actually settle. Regularization limits the correction to preserve the recovered arrangement. Appendix B

4. 用同一场景生成观测、示范与评测

4. Use the same scene for observations, demonstrations, and evaluation

Isaac Sim 渲染物理前景的 RGB、深度和覆盖区域；Gaussian splatting 从对应相机视角渲染背景。逐像素比较深度，只有存在前景且前景不被背景遮挡时才取前景颜色。这样相机可移动，前景物体也可独立移动。

Isaac Sim renders foreground RGB, depth, and coverage, while Gaussian splatting renders the background from the corresponding camera. Per-pixel depth comparison selects foreground color only where foreground exists and is not occluded by the background. Cameras can therefore move and foreground objects can move independently.

\[I_{\rm out}(u)=m(u)I_{\rm fg}(u)+(1-m(u))I_{\rm bg}(u),\quad m(u)=\mathbf1[\alpha_{\rm fg}(u)>0\land D_{\rm fg}(u)\le D_{\rm bg}(u)].\]

InternDataEngine 读取任务、机器人、相机和技能模板；AnyGrasp 提供抓取候选，技能模块输出 6D 末端路点，cuRobo 转换为避碰的密集关节动作，只保存成功示范用于训练。评测时则让策略根据连续观测闭环输出动作。真实轨迹回放、规划器生成示范、策略闭环评测是三个不同实验，不能混为一种成功率。 §3

InternDataEngine reads the task, robot, cameras, and skill templates. AnyGrasp supplies grasp candidates, skill modules produce 6D end-effector waypoints, and cuRobo converts them to dense collision-aware joint actions; successful demonstrations are retained for training. Evaluation instead runs a policy in a closed observation–action loop. Real-trajectory replay, planner-generated demonstrations, and closed-loop policy evaluation are three distinct experiments, not one success metric. §3

主要实验结果：逐项看证据强度

Results: assess each claim against its evidence

实验范围与训练预算

Experimental scope and training budget

DROID-Sim 共构建 564 个场景，但细致视觉与稳定性对比只使用固定的 10 个场景；回放从中随机抽取 5 个。真实机器人训练与评测另使用 4 个布置、10 个任务，硬件为 Franka Research 3、Robotiq 2F-85，以及腕部和外部 RealSense D435i 相机。微波炉开门与放碗是连续长任务的两个单独计分阶段。

DROID-Sim contains 564 constructed scenes, but detailed visual and stability comparisons use a fixed subset of 10; replay samples 5 of those. Real-robot training and evaluation use a separate set of 4 setups and 10 tasks with a Franka Research 3, Robotiq 2F-85, and wrist/exterior RealSense D435i cameras. Opening the microwave and inserting a bowl are separately scored stages of one sequential task.

每个任务有 30 条真实示范，以及 2,000 条普通仿真示范和 2,000 条增强仿真示范。R1–R3 表示流式采样权重，而非数据池的数量比例。所有设置使用相同预训练初始化、batch size 16 和 15,000 次更新；余弦学习率峰值 2×10⁻⁵、warmup 100 步。每个策略／任务评测 30 次真实试验，分成 3 组，每组 10 次；论文的 ± 是三组的标准误，不是多个独立训练种子的误差。 Appendix A

Each task has 30 real demonstrations and separate pools of 2,000 ordinary simulated and 2,000 augmented simulated demonstrations. R1–R3 are streaming sampling weights, not pool-size ratios. All settings use matching pretrained initialization, batch size 16, and 15,000 updates, with a cosine learning-rate schedule peaking at 2×10⁻⁵ and 100 warmup steps. Each policy/task is tested in 30 real trials grouped into 3 batches of 10; the paper’s ± values are standard errors across those batches, not uncertainty across independent training seeds. Appendix A

1. 稳定性明显改善；像素指标并非全面领先

1. Stability improves substantially; pixel metrics do not all improve

表 1：在 10 个场景中运行 300 个物理步后统计。平均位移相对 RoLA 从 0.2713 m 降至 0.0474 m（按表值计算约减少 82.5%）；去掉 refinement 时退化到 0.3126 m。这里“倾倒”指上方向变化超过 45°；“碰撞”指标按相邻采样中心位移超过 0.05 m 或向上跳跃超过 0.03 m 标记，并非直接测量所有网格穿透体积。

Table 1: measurements after 300 physics steps in 10 scenes. Mean displacement falls from 0.2713 m for RoLA to 0.0474 m, an approximately 82.5% reduction calculated from the table; removing refinement raises it to 0.3126 m. “Falling” means an up-axis change exceeding 45°. The collision metric flags center displacement above 0.05 m between sampled poses or upward jumps above 0.03 m; it does not directly measure all mesh-intersection volumes.

视觉对比中，RoboSnap / RoLA 的 SSIM 为 0.4907 / 0.4521，LPIPS 为 0.4958 / 0.4996，SIFT match ratio 为 0.1226 / 0.0664；但 PSNR 为 13.25 / 13.40，略低。结论应是部分结构与分布指标更好，而不是所有视觉指标领先。回放结果为 5/5 对 2/5；真实轨迹只用于测试，不参与修正物体布局，样本量仍很小。 §4

For RoboSnap / RoLA, visual metrics are SSIM 0.4907 / 0.4521, LPIPS 0.4958 / 0.4996, and SIFT match ratio 0.1226 / 0.0664, but PSNR is slightly lower at 13.25 / 13.40. The supported conclusion is improvement in some structural and distributional metrics, not superiority on every visual metric. Replay succeeds in 5/5 versus 2/5 scenes; real trajectories are used only for evaluation, not to repair object layouts, but the sample is small. §4

2. 仿真数据适合作为真实数据的补充

2. Simulated data works best as a supplement to real data

R2 相对真实数据基线分别提升 9.0 和 13.4 个百分点，不是 9.0% 和 13.4% 的相对提升。π0.5 合上笔记本电脑从 50.0% 提升到 80.0%；但挂杯子从 23.3% 降至 16.7%，微波炉放碗从 6.7% 降至 3.3%。平均收益不代表每个任务受益，也不能把 R3 叫作“从未见过真实数据”的模型。

R2 gains 9.0 and 13.4 percentage points over the real-only baselines, not relative increases of 9.0% and 13.4%. For π0.5, closing a laptop improves from 50.0% to 80.0%, but hanging a mug declines from 23.3% to 16.7%, and placing a bowl in the microwave declines from 6.7% to 3.3%. Average gains do not imply improvement on every task, and R3 is not a model that has never seen real data.

3. 扰动下跌幅变小，但最难的扰动仍明显

3. Perturbation losses shrink, but difficult shifts remain

单位核查：论文 §4.4 使用“relative degradation”和“13% → 8%”措辞，但表 3／图中的值由原始成功率减去扰动成功率得到，是百分点。上述六项的算术平均约为 13.25 → 8.48 个百分点；作者将其概括为 13 → 8。扰动是逐项施加，不能据此声称对组合扰动也同样有效。

Unit check: §4.4 calls this “relative degradation” and describes “13% → 8%,” but Table 3 and its plot subtract perturbed success rates from original success rates, yielding percentage points. The arithmetic means of the six displayed drops are about 13.25 → 8.48 points; the authors summarize them as 13 → 8. Perturbations are applied separately, so these results do not establish equal robustness to combined shifts.

4. 相关性支持初步代理评测，不代表绝对准确

4. Correlation supports preliminary proxy evaluation, not absolute accuracy

作者将仅真实数据微调的 π0.5 策略直接放入对应仿真场景，比较跨任务成功率。Pearson r 衡量共同变化；MMRV 衡量排序反转所对应的真实成功率差距，越小越好。结果支持该模型在这些任务上的相对难度代理；它没有验证跨模型家族、跨大量检查点或全部 DROID-Sim 场景的稳定排序。

The authors run real-only-fine-tuned π0.5 policies in their corresponding simulated scenes and compare success rates across tasks. Pearson r measures co-variation; MMRV measures real-success gaps associated with ranking inversions, where lower is better. This supports a relative-difficulty proxy for this model on these tasks, not a validated ranking across model families, many checkpoints, or all DROID-Sim scenes.

按图 6 已四舍五入的表值计算，平均绝对差为 8.35 个百分点（本研究计算，非作者单独报告的指标）；合上笔记本电脑为真实 50.0%、仿真 73.3%。因此，高相关仍可伴随明显的绝对成功率偏差。

Calculated from the rounded values in Figure 6, mean absolute error is 8.35 percentage points (our calculation, not a separately reported author metric); closing a laptop scores 50.0% in reality and 73.3% in simulation. High correlation can therefore coexist with substantial absolute success-rate error.

局限与展望：实际新意与证据边界

Limitations and outlook: novelty and evidence boundaries

审慎定位：怀疑者可以把它看成“已有分割、单图 3D、背景生成、关系推断和机器人规划器的组合，再做一次物理清理”。这是合理的系统层面描述。实际贡献是把可碰撞前景与可移动视角背景的分层场景，结合关系约束下的交替几何／物理修正，形成可重复使用的场景，并验证其训练和评测价值；不是提出一个全新的机器人策略模型。

Critical positioning: a skeptic could describe this as existing segmentation, single-image 3D, background generation, relation inference, and robot planners combined with physical cleanup. That is a reasonable system-level description. The concrete contribution is a reusable scene combining collision-aware foreground and a background supporting moving viewpoints with relation-constrained alternating geometric/physical refinement, plus evidence of its training and evaluation value; it is not a new robot policy architecture.

• 输入与物体范围：作者指出严重遮挡、极端光照和含混材质会降低可靠性；当前面向刚体及关节物体，不覆盖软体、颗粒或流体。 Input and object regime: the authors identify severe occlusion, extreme illumination, and ambiguous materials as failure sources. The system targets rigid and articulated objects, not deformable, granular, or fluid materials.
• 动力学并未被识别：质量、摩擦依赖先验，关节参数从同类别资产检索；附录的布局沉降还使用统一物理参数。稳定站住不代表接触力、摩擦或关节阻尼与真实物体一致。 Dynamics are not identified: mass and friction rely on priors, and joint parameters are retrieved from same-category assets; the appendix also uses uniform parameters during layout settling. A stable arrangement does not establish correct contact forces, friction, or joint damping.
• 归因仍不充分（本研究判断）：去掉 refinement 的消融很有价值，但缺少分别移除场景图、SDF、物理沉降、背景层或不同增强类型的完整实验，难以量化每个模块对真实任务收益的独立贡献。 Attribution remains incomplete (our assessment): the no-refinement ablation is useful, but there is no complete set of isolated tests removing the scene graph, SDF, physics settling, background layer, or individual augmentations. Each module’s independent contribution to real-task gains remains unclear.
• 泛化证据有限：564 是构建规模，不是完整成功验证规模；10 个任务、30 次试验及一种主要机器人配置不足以建立广泛部署可靠性。R2 后平均成功率仍约 42%，长期任务第二阶段尤其困难。 Generalization evidence is limited: 564 is the construction scale, not the scale of complete success validation. 10 tasks, 30 trials, and one primary robot configuration do not establish broad deployment reliability. Average success after R2 remains around 42%, with the second stage of the long task particularly difficult.

下一步最有价值的验证（建议）：对更大的保留场景集测量构建失败率和回放成功率；用同一组任务比较多个策略及检查点的排序；报告成功率校准误差和不确定性；分别消融视觉层与物理修正；将学习到的真实物理参数与类别先验比较。这些实验能区分“可用的特定场景工具”与“可泛化的评测基础设施”。

Most informative next tests (proposed): measure construction failures and replay success on a larger held-out scene set; rank multiple policies and checkpoints on the same tasks; report calibration errors and uncertainty; ablate the visual layer and physical refinement separately; compare identified physical parameters with category priors. These experiments would distinguish a useful tool for specific scenes from a broadly generalizable evaluation infrastructure.

复现与来源：哪些已提供，哪些仍待验证

Reproduction and sources: available components and remaining checks

论文附录报告单场景自动构建耗时约 12–25 分钟，5–10 个物体的中等复杂度场景平均约 17 分钟；这些是作者运行估计，不是本研究复测。合成示范生成在 8 张 RTX 4090 上约为每天 10,000 条任务轨迹，不能与单场景构建耗时混为同一性能指标。 Appendix C · Appendix A

The appendix reports approximately 12–25 minutes for automatic scene construction, averaging about 17 minutes for moderately complex scenes with 5–10 objects. These are author estimates, not independently reproduced timings. Synthetic demonstration generation reaches about 10,000 task trajectories per day on 8 RTX 4090 GPUs; this is a separate throughput metric from scene-construction latency. Appendix C · Appendix A

截至本研究核查日 2026-10-22，公开仓库 README列出 GUI、自动分层场景生成和真实机器人部署教程已提供；Evaluation code 与 DROID-Sim 仍为未勾选条目，项目页的数据集也标为 Coming soon。README 中的预计发布日期不能当作实际发布证据。本研究检查了公开说明和论文，未运行完整重建或机器人训练。

As checked on 2026-10-22, the public repository README lists the GUI, automatic layered scene generation, and real-robot deployment tutorial as available, while evaluation code and DROID-Sim remain unchecked. The project page also labels the dataset “Coming soon.” A planned release date in the README is not evidence of an actual release. This study inspected the public documentation and paper but did not run the full reconstruction pipeline or robot training.

论文与当前代码应区分版本：论文引用 Marble 作为背景 world model，而公开代码还出现 Lyra 相关组件；不能默认当前默认后端完全复现 v1 的实验条件。复现时应记录代码提交、模型权重、背景后端、物理参数、标定和任务成功判定。

The paper and current code should be versioned separately: the paper cites Marble as the background world model, while the public code also includes Lyra-related components. Current defaults should not be assumed to reproduce the v1 experimental configuration. Reproduction should record the code commit, model weights, background backend, physics parameters, calibration, and task-success definitions.