GS-Agent: Creating 4D Physical Worlds With Generative Simulation

研究动机：把生成问题变成制作与调试问题

Motivation: turn generation into production and debugging

一个“巧克力流到蛋糕上”的请求同时要求正确的几何、黏性、碰撞、时间尺度和镜头。传统制作由人反复调整这些要素；视频生成模型则从数据中合成外观与运动，可能出现对象不一致、运动不合理或镜头不听指令的问题。

A request for chocolate flowing over a cake simultaneously requires suitable geometry, viscosity, collisions, timing, and camerawork. Traditional production relies on people repeatedly adjusting these components. Video models synthesize appearance and motion from data, but can produce inconsistent objects, implausible dynamics, or camera motion that misses the instruction.

关键想法：把动态状态交给物理引擎，把场景配置与修正交给语言模型。相机、材质和运动成为可明确编辑的变量，模型可以用实际运行结果检查计划。这里的 4D 指三维空间加时间，不是第四个空间维度。

Key idea: let a physics engine evolve the dynamic state while language models configure and revise the scene. Cameras, materials, and motion become explicit editable variables, and the model can test its plan against execution feedback. Here, 4D means three spatial dimensions plus time, not a fourth spatial dimension.

核心方法：三种职责，一个可执行世界

Method: three roles, one executable world

术语速查：这些抽象词具体是什么

Glossary: what the abstract terms mean here

flowchart LR
 U["Text request"] --> M["Manager Agent"]
 M --> E["Entity Agent"]
 M --> R["Render Agent"]
 E --> S["Genesis simulation"]
 R --> S
 M --> S
 S --> F["State and visuals"]
 F --> M
 M --> O["Code and video"]

1. Manager：分解任务、配置求解器、控制时间

1. Manager: decompose tasks, configure solvers, and control time

Manager 一次向一个专家分配一个子任务，接收结果后检查并决定是否返工。它设置时间步长、子步数、重力、MPM 网格与边界或 SPH 粒子尺度，通过 AdvanceScene 和 ResetScene 推进或重置场景。视频通常作为 AdvanceScene 的结果返回，随后在当前上下文中自我审查。

The Manager assigns one subtask at a time to a specialist, checks the returned result, and decides whether to revise it. It sets time steps, substeps, gravity, MPM grids and bounds, or SPH particle scales, and uses AdvanceScene and ResetScene to advance or reset the scene. Video is returned by AdvanceScene and then reviewed within the Manager’s current context.

2. Entity：从资产到可运动的材料

2. Entity: turn assets into moving materials

资产优先从 BlenderKit 与 PolyHaven 的整理库中检索，索引结合文本和 CLIP 图像嵌入，并返回预览、得分与轴对齐包围盒。检索失败时调用 Meshy 生成，仍失败则用基础几何体。代理根据包围盒设定尺度、朝向与位置，再用实际加载后的状态和图像检查接触面与穿插。

Assets are first retrieved from a curated BlenderKit and PolyHaven library using text and CLIP image embeddings, with previews, scores, and axis-aligned bounds returned. Retrieval failure triggers Meshy generation, followed by primitive geometry if generation also fails. The agent uses bounds to set scale, orientation, and placement, then checks contact surfaces and interpenetration against loaded state and images.

物理参数与外观参数分开：Young’s modulus、Poisson’s ratio 和屈服应力影响变形，而 Surface 决定视觉效果。Entity 通过试运行后观察实际形变来调参，并使材料尺度与求解器分辨率相容。运动用每步执行的代码实现，包括 PD 控制、位置/速度/力命令以及粒子发射器。

Physics parameters are separate from appearance: Young’s modulus, Poisson’s ratio, and yield stress affect deformation, while Surface controls the visual result. Entity tunes parameters by observing actual deformation after trial runs and matching material scales to solver resolution. Per-step code implements motion through PD control, position/velocity/force commands, and particle emitters.

3. Render：把镜头与物理时间对齐

3. Render: synchronize cameras with simulation time

Render 将“从右侧看”等指令映射到引擎坐标系，设定相机内外参、光源和取景。相机环绕、推进和跟随是显式轨迹代码，采样与录制时间和仿真步同步。提示词要求先使用快速 Rasterizer 检查物体位置和运动，确认后再用 RayTracer 完成最终灯光与视频。

Render maps instructions such as “view from the right” into engine coordinates and sets camera intrinsics/extrinsics, lighting, and framing. Orbits, dollies, and tracking are explicit trajectory code, with capture synchronized to simulation steps. The prompts call for checking placement and motion with a fast Rasterizer before using RayTracer for final lighting and video.

4. 工具接口与修正循环为何重要

4. Why tools and the revision loop matter

专门工具把配置、实体操作、状态查询和渲染组织为明确动作，但部分工具仍接收代码；这不是完全消除自由代码的系统。论文给出的代表案例是浴缸网格漏水：代理观察视频，判断碰撞几何不密封，在角部添加刚体补丁并再次检查。它证明了一种可行的修正行为，而不是整体错误恢复成功率。

Specialized tools organize configuration, entity operations, state queries, and rendering into explicit actions, although some tools still accept code; the system does not eliminate free-form programming. A representative case is a leaking bathtub mesh: the agent inspects video, diagnoses non-watertight collision geometry, adds rigid patches at the corners, and checks again. This demonstrates a possible recovery behavior, not an aggregate recovery success rate.

最关键的一句话：用可执行仿真把“生成—观察—修改”闭环中的每次修改落实为对象、材料、控制或相机参数；三种角色负责把这个搜索空间拆小。

The essential mechanism: executable simulation makes each change in the generate–observe–revise loop an explicit edit to objects, materials, controls, or cameras; the three roles divide this search space into smaller tasks.

主要实验结果：物理与控制领先，审美有取舍

Results: gains in physics and control, with aesthetic tradeoffs

论文使用两套不同的评测：24 个 NewtonGen 场景覆盖 12 类物理规律，用于 PIS；另有作者收集的 30 个复杂场景，包含多物体交互与动态镜头，用于文本对齐与美观度。默认代理骨干为 gpt-5，仿真引擎为 Genesis；输出统一为 720p。以下是论文报告结果，本研究未重新运行实验。

The paper uses two distinct suites: 24 NewtonGen scenes covering 12 physical laws for PIS, and 30 author-collected complex scenes with multi-object interactions and moving cameras for alignment and aesthetics. The default agent backbone is gpt-5, the engine is Genesis, and outputs are standardized to 720p. The following are paper-reported results; this study does not rerun the experiments.

如何读 PIS：保持不变量，不等于验证全部物理

Reading PIS: preserving invariants does not verify all physics

Video-PIS 从 SAM2 分割后的物体中心与形状估计运动；State-PIS 使用引擎内的三维质心运动，避免视频测量误差。NewtonGen 的定义为 PIS = 1 / (1 + σ(C) / (|μ(C)| + ε))，其中 ε = 10⁻⁵，C 是理论上应保持不变的量，例如抛体运动的竖直加速度。分数在 0–1，越高表示该量随时间越稳定。公式见 NewtonGen 第 5 节，式 7；GS-Agent 正文将它简写为相对标准差指标。

Video-PIS estimates motion from SAM2-segmented object centers and shapes; State-PIS uses 3D center-of-mass motion inside the engine, avoiding video measurement error. NewtonGen defines PIS = 1 / (1 + σ(C) / (|μ(C)| + ε)), with ε = 10⁻⁵, where C is a theoretically invariant quantity, such as vertical acceleration in projectile motion. Scores range from 0–1; higher means that quantity is more stable over time. See NewtonGen Section 5, Equation 7; GS-Agent describes it in shorthand as a relative-standard-deviation metric.

State-PIS 是仿真内部状态的精确读数，不是真实世界测量。没有状态输出的视频模型无法得到此分数，表中的缺失值不是 0；也不能把 GS-Agent 的 State-PIS 0.83 与 Sora2 的 Video-PIS 0.62 当成同一测量直接比较。恒定但数值错误的加速度也可能具有较高不变性分数。

State-PIS is an exact reading of internal simulation state, not a real-world measurement. Video models without state outputs cannot receive this score; missing entries are not zero. GS-Agent’s State-PIS of 0.83 should not be directly compared with Sora2’s Video-PIS of 0.62 as if they were the same measurement. Even a constant but incorrect acceleration can have a high invariance score.

主结果：论文表 1

Main results: paper Table 1

对齐指标使用 Perception Encoder 特征相似度；美观度来自 VBench。它们不是任务成功百分比。GS-Agent 的 Video-PIS 比 Sora2 高 0.09，对齐分数高 1.6；但自动美观度低于 Wan2.2 和 Sora2。

Alignment uses Perception Encoder feature similarity; aesthetics uses VBench. These are not task-success percentages. GS-Agent exceeds Sora2 by 0.09 Video-PIS and 1.6 alignment points, but its automated aesthetic score is below both Wan2.2 and Sora2.

人类评价：论文表 2

Human evaluation: paper Table 2

15 名参与者对随机排序的五种方法视频按 1–5 分评价，得到 270 份有效回应。下表是平均分；270 不是参与者人数。论文没有提供这些均值的置信区间，因此细小差距不能直接解读为统计显著优势。

Fifteen participants rated randomly ordered videos from five methods on a 1–5 scale, yielding 270 valid responses. The table reports means; 270 is not the participant count. The paper does not provide confidence intervals for these means, so small differences should not be treated as statistically established advantages.

GS-Agent 的物理、镜头和内容平均分最高，但内容对齐相对 Sora2 仅高 0.05。人类美观度偏好 Sora2，而自动指标偏好 Wan2.2；两种评估不应合并为单一“画质冠军”。

GS-Agent has the highest mean physics, camera, and content scores, but its alignment lead over Sora2 is only 0.05. Humans favor Sora2 aesthetically, whereas the automated metric favors Wan2.2; these evaluations should not be collapsed into one universal visual-quality ranking.

消融：分工与接口均有贡献，但因果隔离有限

Ablations: roles and tools help, with limited causal isolation

把工具集中到一个代理后，State-PIS 从 0.83 降到 0.42，支持职责拆分的价值；接口消融降到 0.57。不过论文用 SWE-Agent w/ Visual 作为接口消融比较，其代理结构与工具组织也不同，因此不能将全部差距只归因于接口。论文也未给出匹配 token、工具调用或耗时预算后的比较。

Consolidating tools into one agent lowers State-PIS from 0.83 to 0.42, supporting the value of role separation; the interface ablation reaches 0.57. However, the paper uses SWE-Agent w/ Visual for the interface comparison, which also differs in agent structure and tool organization. The entire gap therefore cannot be attributed solely to the interface. A comparison with matched token, tool-call, or runtime budgets is not reported.

局限与展望：自动化制作的价值及其边界

Limitations and outlook: the value and limits of automated production

批判性定位：怀疑者可以将它概括为“语言模型驱动的仿真脚本工作流”。真正值得关注的新意是针对动态、多材料场景，把三种专家职责、专门仿真工具和运行后视觉检查组成完整闭环；不是首次用语言模型写图形代码，也不是新的物理定律或求解器。论文此前工作的比较包括静态场景代理及物理引导生成，但没有在本实验中直接比较所有这类方法。

Critical positioning: a skeptic could describe this as a language-model-driven simulation scripting workflow. The concrete contribution is the complete loop combining three specialist roles, dedicated simulation tools, and visual checks after execution for dynamic, multi-material scenes. It is neither the first use of language models for graphics code nor a new physical law or solver. The related-work discussion includes static-scene agents and physics-guided generation, but the experiments do not directly compare against every such approach.

• 仿真不等于真实：论文讨论中的“保证物理与时间一致性”应理解为在给定引擎、材料模型、控制与数值设置下的状态演化。错误材料、粗糙碰撞网格、不稳定步长和强制轨迹仍会造成偏差；没有相对于真实实验的全面校准。
• Simulation is not reality: the discussion’s claim of guaranteed physical and temporal consistency should be read within the chosen engine, material models, controls, and numerical settings. Wrong materials, coarse collision meshes, unstable time steps, and imposed trajectories can still introduce errors; comprehensive calibration against real experiments is not demonstrated.
• 自我审查会失败：附录 C.3 报告三类失败：长计划污染上下文，让模型误以为目标已经实现；误判根因，反复调整错误参数；臆造不存在的 API、材料属性或坐标约定。引擎在环并不能自动让视觉判断可靠。
• Self-critique can fail: Appendix C.3 reports plan-induced context contamination that makes the model believe its intended result occurred, root-cause misdiagnosis that repeatedly changes the wrong parameter, and hallucinated APIs, material properties, or coordinate conventions. An engine in the loop does not automatically make visual judgment reliable.
• 证据范围有限：24 个物理场景与 30 个复杂场景支持有限测试域中的结论；缺少完整的生成耗时、API 成本、失败率和多次运行方差，无法据此判断大规模生产的吞吐量与可靠性。
• Evidence is bounded: 24 physics scenes and 30 complex scenes support conclusions within a limited test domain. Comprehensive generation time, API cost, failure rate, and repeated-run variance are not reported, preventing a firm assessment of production throughput and reliability.
• 下游价值仍需验证：可执行场景可以提供深度、分割、法线和粒子状态，也便于改变镜头或初始条件。但论文没有报告由这些数据训练机器人后得到的实机收益；强化学习与 sim-to-real 是潜在用途，不是已验证结果。
• Downstream value still needs testing: executable scenes can supply depth, segmentation, normals, and particle states, and make camera or initial-condition changes accessible. The paper does not report real-robot gains from training on these data; reinforcement learning and sim-to-real transfer are potential uses, not validated outcomes.

本研究的判断：最有前景的方向是把物理仿真作为可检查、可编辑的数据生成底座。下一步最有信息量的实验是固定预算与同一骨干的接口/分工对照、跨运行失败率与成本统计，以及真实物理量或下游机器人任务的外部验证。

This study’s assessment: the most promising direction is physics simulation as an inspectable, editable basis for data generation. The most informative next experiments would isolate tools and roles under a fixed budget and shared backbone, report failure rates and costs across runs, and validate against real physical measurements or downstream robotics tasks.

来源与版本

Sources and version

主要依据 GS-Agent v1（2026-07-23），尤其第 3–5 节、表 1–4、附录 B 与 C。图像来自论文 HTML 的图 3、5、6；流程图是本页的解释性重绘，柱图按表 1 数值绘制。

Primary source: GS-Agent v1 (2026-07-23), especially Sections 3–5, Tables 1–4, and Appendices B and C. Images come from Figures 3, 5, and 6 in the paper HTML; the flowchart is this study’s explanatory reconstruction and the bar chart uses Table 1 values.

辅助核对：作者项目页提供视频、模态示例与示例仿真代码；所查页面没有单独的 GS-Agent 代码仓库链接。NewtonGen v2 仅用于澄清 PIS 定义，没有将其模型实验数字并入本页。

Additional checks: the authors’ project page provides videos, modality examples, and example simulation code; the inspected page does not link a separate GS-Agent code repository. NewtonGen v2 is used only to clarify the PIS definition; its model results are not mixed into this study.