4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes

为什么普通代码测试和视频相似度不够

Why ordinary code tests and video similarity are insufficient

普通代码题可以用输入输出判断对错，但这里要从像素反推物体形状、遮挡、材质、接触和运动。把每帧画得相似仍可能藏着错误的三维结构；即使几何正确，也可能靠逐帧安排位置来模仿碰撞。液体、柔性物体和断裂还会改变形状或拓扑，不能只靠给刚体设置位姿解决。 [1]

Ordinary coding problems can be checked through inputs and outputs. Here the agent must infer shape, occlusion, materials, contact and motion from pixels. Similar frames can conceal incorrect 3D structure; even correct geometry can use prescribed positions to imitate a collision. Fluids, soft objects and fracture also change shape or topology, exceeding simple rigid-object pose estimation. [1]

核心贡献是把动态视频重建变成可执行、可检查的三维程序任务：真实视频提供复杂观测，合成视频提供已知的三维几何和运动作为标准答案。这是一套 benchmark、数据和评测协议，不是新的视频生成网络或新的物理求解器。 [1]

The central contribution is turning dynamic video reconstruction into an executable, inspectable 3D programming task: real videos supply complex observations, while synthetic videos supply known geometry and motion for evaluation. This is a benchmark, dataset and evaluation protocol, rather than a new video-generation network or physics solver. [1]

定位判断：视频转代码的想法本身已有先例。这里更具体的新意，是多种材料与相互作用、统一的显式 4D 导出格式，以及与人类偏好对照的分项评测组合。上表依据本文第 5 节归纳，不代表对各项先前工作的独立复现实验。 [1]

Assessment: video-to-code already has precedents. The more specific contribution is the combination of varied materials and interactions, a unified explicit 4D export format, and decomposed evaluation checked against human preferences. The table summarizes Section 5 of this paper; it does not represent independent replication of the earlier projects. [1]

任务与评测：究竟输入、生成和检查什么

Task and evaluation: what goes in, comes out and gets checked

1. 只给视频，让智能体自行选择场景表示

1. Give only video and let the agent choose a scene representation

每个模型配置在每个场景运行一次。输入是 RGB 视频和统一的任务／输出格式说明，不提供场景名称、物体列表、语义描述或参考三维世界。每次运行使用独立容器和一块 GPU，提供 Blender、数值与模拟库、离线文档以及格式检查器；论文描述没有人为步数上限，并停止卡住或超过 6 小时的运行；当前公开 Stirrup runner 另设 500 turns，详见下方源码审计。 [1]

Each model configuration runs once per scene. It receives an RGB video and fixed task/output instructions, without scene names, object lists, semantic descriptions or reference 3D worlds. Each run uses an isolated container and one GPU, with Blender, numerical and simulation libraries, offline documentation and a format checker. The paper describes no artificial step cap and termination of stalled or over-six-hour runs. The current public Stirrup runner separately imposes 500 turns; see the source audit below. [1]

4D 就是三维空间随时间变化。智能体可反复写代码、渲染、比较和修改；运动既可来自 Blender 或自写的模拟器，也可来自解析公式和关键帧。最终程序必须独立重建结果，不能再读取参考视频或把视频像素复制成纹理。渲染分辨率、帧数和帧率必须与输入一致。 [1]

4D means 3D space evolving over time. Agents may repeatedly code, render, compare and revise. Motion can come from Blender or custom simulation, analytic formulas, or keyframes. The final program must regenerate the result independently, without reading the reference video or copying its pixels into textures. Output resolution, frame count and frame rate must match the input. [1]

2. 导出可检查的几何和运动，而不只是视频

2. Export inspectable geometry and motion alongside video

solution/build.sh 重建 world/：其中 camera.json 描述相机，render.mp4 是渲染视频，meshes/ 保存逐帧表面，dynamics/ 保存持久物质点的位置。网格顶点数量与拓扑允许变化，因此断裂并不要求维持固定网格；无持久粒子身份的 Eulerian 液体还可导出 solver/ 缓存，供评测时恢复轨迹。 [1] [3]

solution/build.sh regenerates world/: camera.json describes the camera, render.mp4 is the video, meshes/ stores per-frame surfaces, and dynamics/ stores persistent material-point positions. Vertex counts and topology may change, so fracture need not preserve a fixed mesh. Eulerian liquids without persistent particle identities can also export solver/ bakes for trajectory extraction at evaluation time. [1] [3]

3. 两类数据提供不同强度的证据

3. Two data sources provide different kinds of evidence

数据集包含 100 个真实视频和 100 个物理模拟场景，涉及刚体／关节系统、可变形固体、布料／绳子等低维结构，以及颗粒／流体。83% 的场景含多个动态物体，66% 含多种材料；类别可重叠。真实片段优先选择相机稳定、遮挡较少、无剪辑的事件，并排除人类和动物；帧率上限为 60 fps，真实视频最多 300 帧。 [1]

The dataset contains 100 real videos and 100 physics-simulated scenes: rigid/articulated systems, deformable solids, cloth/rope and other thin structures, and granular/fluid materials. Multiple dynamic objects appear in 83% of scenes and multiple materials in 66%; categories can overlap. Real clips favor stable cameras, limited occlusion and continuous events, and exclude humans and animals. Frame rates are capped at 60 fps, and real videos at 300 frames. [1]

真实视频没有真实的三维运动答案，所以部分参考量由视觉模型估计；合成视频有已知世界，可直接比较三维表面与轨迹。两者互补，但不同分项并不都覆盖同一组场景。 [1]

Real videos lack ground-truth 3D motion, so vision models estimate some reference quantities. Synthetic videos have known worlds, enabling direct surface and trajectory comparisons. These sources complement each other, but metric families do not all cover the same scenes. [1]

4. 五个分项构成 Overall；VQA 与 Elo 单列

4. Five families form Overall; VQA and Elo remain separate

网站把各指标映射到预先声明的 [0,1] 范围，而非根据参评模型做 min–max 缩放；一个分项内部等权平均，Overall 再对五个分项等权平均。因此 Overall = 0.79 不是“79% 场景通过”。原始 Depth error 的范围是 [0,2]，越低越好，不能直接当成表中越高越好的深度分项。 [2] [3]

The site maps metrics to declared [0,1] scales rather than min–max scaling over participating models. Metrics receive equal weight within each family; Overall then equally averages the five families. Thus Overall = 0.79 does not mean “79% of scenes passed.” Raw Depth error ranges from [0,2] with lower being better; it must not be confused with the higher-is-better depth family score in the leaderboard. [2] [3]

失败会被计入：无有效视频会使所有指标失败；视频有效但世界格式无效，则相关几何指标受罚。缺少参考信号的指标会对所有模型统一排除对应场景；另有 6 个含透明主体的真实场景被排除出部分栅格化比较。不能把网站的“覆盖所有 200 场景”理解为每个指标都恰好有 200 个有效参考。 [1]

Failures count: invalid video fails all metrics; valid video with an invalid world incurs penalties on geometry-dependent metrics. Scenes missing a prerequisite reference signal are excluded consistently across models for that metric. Six real scenes with transparent main objects are also excluded from certain rasterization comparisons. The site’s “all 200 scenes” coverage language should not be read as exactly 200 usable references for every metric. [1]

系统源码审计：谁做什么

System audit: responsibilities and data flow

直接结论：4DCodeBench 是视频→图形程序的评测系统。外层调度器启动一个 coding agent；模型自行决定如何观察、写代码、渲染和修改；结束后，独立 scorer 读取交付物，VLM judge 再提供单列的语义与偏好评测。仓库没有实现新的物理求解器，也没有把评测分数自动反馈给 agent 的训练循环。 README.md

Direct answer: 4DCodeBench evaluates video-to-graphics programs. An outer scheduler launches one coding agent; the model decides how to inspect, code, render and revise. After it exits, a separate scorer reads the delivery, while a VLM judge provides separately reported semantic and preference evaluations. The repository implements neither a new physics solver nor a training loop that automatically feeds evaluation scores back to the agent. README.md

本次源码审计固定到 aa05fb426273b59f47fac71825f77ca1b7a64c08（2026-10-07 读取）。下方保留原有论文结果；新增部分以该 commit 的执行代码为准，并单独标出文档与代码的差异。Stirrup 内部提示词来自镜像明确固定的 PyPI 包 stirrup==0.2.0，并非猜测其他 CLI 的内部提示词。 harness/agent/Dockerfile

This audit pins the source to aa05fb426273b59f47fac71825f77ca1b7a64c08, retrieved 2026-10-07. The existing paper results remain below; the new sections follow this commit’s executable code and separately identify documentation discrepancies. Stirrup’s internal prompts come from the image’s pinned PyPI package stirrup==0.2.0; they are not guesses about other CLIs’ internal prompts. harness/agent/Dockerfile

flowchart LR
  J["Jobs and runtime"] --> H["Inference harness"]
  T["Task documents"] --> A["Coding agent"]
  V["Reference video"] --> A
  H --> A
  A --> W["World and solution"]
  V --> P["Prepare estimates"]
  G["Hidden reference world"] --> S["Scorer"]
  M["Masks and checkpoints"] --> P
  M --> S
  P --> S
  W --> S
  S --> O["Metrics and viewer"]
  W --> Q["VLM judge"]
  V --> Q
  Q --> E["VQA and Elo"]

输出契约与相机约定

Output contract and camera convention

/input/reference.mp4                  # read-only input
/task/TASK.md                         # rendered initial task
/task/world-format.md
/task/render-settings.md
/workspace/solution/build.sh          # executable; no arguments
/workspace/solution/**/*.py, **/*.sh
/workspace/world/camera.json          # K: 3x3; E: 4x4 camera-to-world
/workspace/world/render.mp4
/workspace/world/meshes/<id>.npz       # vertices_TTTT: float32; faces_TTTT: int32
/workspace/world/dynamics/<name>.npz  # pos: (F,N,3) float32; ids: uint16
/workspace/world/solver/              # only identity-less liquid caches

所有数据使用米、Z-up 和同一 F 帧时间轴。相机固定，局部坐标为 x 向右、y 向下、z 向前。网格可改变顶点数和拓扑；pos[t,j] 的 j 列必须始终代表同一物质点，出生／消失之外用 NaN。静态物体不需要 dynamics；Mantaflow 这类重新播种粒子的液体应提交 bake 与七字段 manifest，scorer 最多平流 16,384 个 tracer。 harness/agent_context/world-format.md

All data use metres, Z-up and one F-frame timeline. The camera is fixed, with local x right, y down and z forward. Mesh vertex counts and topology may change; column j of pos[t,j] must represent the same material point, with NaN outside its lifetime. Static objects need no dynamics archive. Reseeded liquids such as Mantaflow supply a bake and seven-field manifest; the scorer advects up to 16,384 tracers. harness/agent_context/world-format.md

build.sh 的协议要求是从空目录重建 world，不能读取输入视频、联网、安装依赖或载入随程序附带的资产。最终使用 Blender Cycles、原分辨率、128 samples、denoising 和 OptiX GPU；迭代阶段文档允许半分辨率 EEVEE。渲染必须与参考具有相同帧数、帧率和宽高。此处区分“任务要求”与后文实际 gate 检查范围。 harness/agent_context/render-settings.md

The build.sh contract requires rebuilding world from scratch without reading the input video, networking, installing packages or loading bundled assets. Final rendering uses Blender Cycles, native resolution, 128 samples, denoising and OptiX GPU; the guidance allows half-resolution EEVEE while iterating. Render frame count, frame rate, width and height must match the reference. These task requirements are distinct from the implemented gate checks discussed below. harness/agent_context/render-settings.md

全部执行循环、工具与 skill 边界

Execution loops, tools and skill boundaries

“循环”有不同层级。外层调度与失败重试由程序定义；代码—渲染—检查的具体策略由模型选择。代码中没有固定的“先规划 N 次、再让 critic 打分、最后择优”的统一流水线。也没有跨场景学习、参数更新或自动进化 skill 的实现。 harness/runtime/agents.py

There are several kinds of loop. Scheduling and failure retries are program-defined; the code–render–inspect strategy is model-chosen. The code defines no shared pipeline of N planning rounds, critic scoring and best-candidate selection. It also implements no cross-scene learning, parameter updates or automatic skill evolution. harness/runtime/agents.py

flowchart LR
  I["Read task"] --> O["Inspect video"]
  O --> C["Write scene code"]
  C --> R["Run and render"]
  R --> V["Inspect output"]
  V -->|"revise"| C
  V --> K["Public checker"]
  K -->|"format errors"| C
  K --> F["Finish or timeout"]
  F --> X["Collect delivery"]

循环依据：harness/stirrup/runner.py、harness/runtime/launch.py、scorer/execution.py、vlm_judge/judge.py；Stirrup dependency 的源文件随提示词资产保留。

Loop evidence: harness/stirrup/runner.py, harness/runtime/launch.py, scorer/execution.py, vlm_judge/judge.py; Stirrup dependency source is retained in the prompt assets.

工具与“skill”：仓库实际上提供什么

Tools and “skills”: what the repository actually supplies

在该 commit 的 tracked tree 中未发现 SKILL.md、AGENTS.md、技能注册表或 benchmark 自己配置的子 agent。最接近可复用 skill 的内容是 TASK/world-format/render-settings 三份说明、离线 Blender API 文档、checker 和 preview；它们没有技能触发器或自动路由。Stirrup 库支持 sub-agent，但本 runner 只注册 shell、view_image 与 finish。Codex 启动命令还明确关闭 plugins 与 remote_plugin。 harness/runtime/agents.py

The tracked tree at this commit contains no SKILL.md, AGENTS.md, skill registry or benchmark-configured sub-agent. The reusable guidance consists of TASK/world-format/render-settings, offline Blender API documentation, checker and preview; these have no skill trigger or routing mechanism. Stirrup supports sub-agents as a library feature, but this runner registers only shell, view_image and finish. The Codex launcher explicitly disables plugins and remote_plugin. harness/runtime/agents.py

基础镜像提供 CUDA 13.0.2、Blender/bpy 5.2.0、Python 3.13、PyTorch、NumPy、SciPy、Taichi、Warp、Trimesh、Pillow、FFmpeg 和 ImageMagick。这些是可调用的通用能力，不是每次都强制使用的模拟方法。agent 网络默认开启以访问模型 provider；“build.sh 不联网”是交付程序的协议要求，不能等同于整个 agent container 没有网络。 harness/base/Dockerfile

The base image provides CUDA 13.0.2, Blender/bpy 5.2.0, Python 3.13, PyTorch, NumPy, SciPy, Taichi, Warp, Trimesh, Pillow, FFmpeg and ImageMagick. These are available capabilities rather than mandatory simulation algorithms. Agent networking is enabled by default to reach model providers; the no-network build.sh contract does not imply the entire agent container is offline. harness/base/Dockerfile

四种 agent 的实际启动配置

Actual launch configuration for four agents

"""The headless command line of each agent, and the agent registry.

All agents run in one image. `infer.py` adds each agent's environment and mounts
only that agent's credential file.
"""

from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class Agent:
    """One agent: its name, accepted efforts (empty: any), and transcript dir under $HOME."""

    name: str
    efforts: tuple[str, ...]
    session_dir: str

    def command(self, task_path: str, workspace_path: str, model: str, effort: str, timeout_sec: int) -> list[str]:
        raise NotImplementedError


@dataclass(frozen=True)
class ClaudeCode(Agent):
    def command(self, task_path: str, workspace_path: str, model: str, effort: str, timeout_sec: int) -> list[str]:
        return [
            "bash", "-c",
            (
                f'claude -p "$(cat {task_path})" --model {model} --effort {effort} '
                f"--permission-mode dontAsk --allowedTools Bash Edit Write Read Glob Grep NotebookEdit "
                f"--output-format json"
            ),
        ]


@dataclass(frozen=True)
class Codex(Agent):
    def command(self, task_path: str, workspace_path: str, model: str, effort: str, timeout_sec: int) -> list[str]:
        return [
            "bash", "-c",
            (
                f'codex exec "$(cat {task_path})" --model {model} '
                f'-c model_reasoning_effort="{effort}" '
                # plugin sync is off: it copies bundles into ~/.codex at startup and can block exit
                f"--disable plugins --disable remote_plugin "
                f"--dangerously-bypass-approvals-and-sandbox --skip-git-repo-check -C {workspace_path} --json"
            ),
        ]


@dataclass(frozen=True)
class Antigravity(Agent):
    def command(self, task_path: str, workspace_path: str, model: str, effort: str, timeout_sec: int) -> list[str]:
        return [
            "bash", "-c",
            (
                f'agy -p "$(cat {task_path})" --model {model} --effort {effort} '
                f"--dangerously-skip-permissions --output-format json --print-timeout {timeout_sec}s"
            ),
        ]


@dataclass(frozen=True)
class Stirrup(Agent):
    """Artificial Analysis' Stirrup agent loop: harness/stirrup/runner.py at /opt/stirrup."""

    def command(self, task_path: str, workspace_path: str, model: str, effort: str, timeout_sec: int) -> list[str]:
        return ["bash", "-c", f"python3 /opt/stirrup/runner.py {task_path} {workspace_path} {model} {effort} {timeout_sec}"]


AGENTS: dict[str, Agent] = {
    # stirrup takes any effort and sends it as reasoning_effort
    "stirrup": Stirrup(name="stirrup", efforts=(), session_dir=".stirrup"),
    "claude": ClaudeCode(
        name="claude",
        efforts=("low", "medium", "high", "xhigh", "max"),
        session_dir=".claude/projects",
    ),
    "codex": Codex(
        name="codex",
        efforts=("low", "medium", "high", "xhigh", "max"),
        session_dir=".codex/sessions",
    ),
    "antigravity": Antigravity(
        name="antigravity",
        efforts=("low", "medium", "high"),
        session_dir=".gemini/antigravity-cli/brain",
    ),
}


def agent(name: str) -> Agent:
    if name not in AGENTS:
        raise KeyError(f"unknown agent {name!r}; known: {sorted(AGENTS)}")
    return AGENTS[name]

全部公开提示词与问题库

All public prompts and the question bank

完整性范围：本节给出仓库自己的全部固定任务／judge 提示词、三份 agent 文档、Stirrup 0.2.0 的系统／摘要提示词与运行时补充消息，以及所有 200 场景的 VQA 问题。原始提示词以可复制的 code/data 保持原文，两种页面语言共享；解释均完整翻译。未发布的 CLI 系统提示词、工具 schema 和 provider 内部消息不在本仓库中，不能声称已恢复。

Completeness scope: this section includes the repository’s fixed task/judge prompts, all three agent documents, Stirrup 0.2.0 system/summary prompts and runtime additions, and the VQA questions for all 200 scenes. Literal prompts remain copyable original code/data shared between languages; explanations are fully translated. Unreleased CLI system prompts, tool schemas and provider-internal messages are not present in this repository and cannot be claimed as recovered.

初始 coding 消息由 TASK.md 路径替换生成，然后作为各 CLI 的 task/user 输入。不会把全部参考视频帧自动附到初始消息；agent 自行用工具观察视频。world-format 与 render-settings 以只读文件提供，并由 TASK 指示阅读。源文件开头仍写 input.mp4，但实际 mount 和后续路径是 /input/reference.mp4；这是源码中的命名残留。 harness/runtime/infer.py

The initial coding message is TASK.md with paths substituted, passed as each CLI’s task/user input. The harness does not automatically attach all video frames to that message; the agent inspects the video through tools. world-format and render-settings are read-only files referenced by TASK. The source opening still says input.mp4, although the actual mount and later path are /input/reference.mp4; this is a naming inconsistency in the source. harness/runtime/infer.py

`input.mp4` is a video of a dynamic scene.
Your task is to write graphics code that, when rendered, reproduces the observed 4D scene.

Your reconstruction should model the scene and its dynamics explicitly.
It should be both visually and physically consistent with the target.
Render the result in Blender, matching the input's dimensions, framerate, and camera.

## Scoring
Your solution will be scored frame-by-frame.
It will also be evaluated against the ground-truth 3D geometry.
Accordingly, solutions that represent motion or appearance in 2D or copy from the video instead of explicitly modeling the scene will be strongly penalized.
The autograder will fail submissions that copy pixels from the input video into the Blender scene for any purpose, such as to texture objects; all must be programmatically defined.

## Output
The output should contain two directories:
1. `/workspace/solution` is the program, an executable `build.sh`, and the Python it runs.
2. `/workspace/world` is its output, holding the camera, the render, the geometry at every frame, and the 3D state of the moving matter over time.

`build.sh` takes no arguments, does **not** read the video file, and writes all of `/workspace/world` from nothing.
Both are required to verify that the reconstruction is reproducible.

Run `python -m checker` to confirm that what you have written is in the correct format.
The packaged checker does not contain the privileged autograder.

## How to build the scene and its motion

One option is to do all of it in Blender: describe the 3D scene and its motion
through time there, and render that 4D sequence. You can also write external
graphics code or programs using the libraries provided to describe the motion of the scene
through time, and use Blender only to render the final 4D sequence. NumPy,
SciPy, PyTorch, Taichi, Warp and Trimesh are installed, and anything else on the
machine is fair game.

However the motion is computed, the evolution of the scene has to be rendered
with Blender, through the camera you export.

The rendered geometry in `meshes/` may change freely from frame to frame, vertex count and
topology included, as happens with liquids or fractures. `dynamics/` is where the
frame-to-frame correspondence lives: column `j` of `pos` is the same piece of
material at every frame, so `pos[t, j]` and `pos[t+1, j]` are that matter one
frame apart. Geometry rebuilt independently at each frame cannot give you that.

## Output format

The full format of `/workspace/world` is in `/task/world-format.md`.
It lists every file with its dtype and shape, the camera conventions, and what to return for a liquid.
Read it before you write your export.

### Program: `/workspace/solution`

`/workspace/solution` should contain an executable `build.sh` and the `.py` and `.sh` files it runs, in any layout.
Running `bash build.sh` should write the complete `/workspace/world`, replacing anything already there.
It cannot read the video, access the network, install packages, or load assets shipped alongside it.
Anyone with this environment should be able to rebuild your world by running it.

### World: `/workspace/world`

```
world/
  camera.json          {"intrinsics": K, "extrinsic": E}, the fixed camera
  render.mp4           the rendered video
  meshes/<id>.npz      each rendered object's triangles at every frame
  dynamics/<name>.npz  each piece of moving material's positions over time
  solver/              a liquid solver's own bake, if it keeps no particle identity
```

`meshes/` is the geometry that gets rendered, and its vertex count and topology may change from frame to frame.
`dynamics/` describes how the 3D geometry of the moving objects moves through time: `pos` has shape `(F, N, 3)`, and column `j` is the same piece of material at every frame.
Static objects need only a `meshes/` archive.
The points in `dynamics/` are the vertices of your simulation mesh for a rigid body, cloth or soft body, and the particles for MPM, SPH or sand.

All positions are in world-space metres, with Z up.
`render.mp4` must match `/input/reference.mp4` in frame count, frame rate and resolution, or it will not be scored.
Every file in `meshes/` and `dynamics/` uses the same `F` frames.
Each frame of `render.mp4` should show exactly the triangles in `meshes/` and nothing else.

### Environment

The environment has FFmpeg, FFprobe, ImageMagick, Python and Blender.
You can run Blender as the `blender` executable or import it in Python as `bpy`.
NumPy, SciPy, PyTorch, Taichi, Warp, Trimesh, Pillow and the checker are available in Python.
The Blender Python API reference is in `/opt/blender-docs`.
`/task/render-settings.md` has the render settings we expect.

### Checking your work

`python -m checker` reports every format error, and a world that fails it scores zero.
`python -m checker.preview` saves `/workspace/tmp/preview_0000.png`, with frame 0 of `render.mp4` next to your `meshes/` projected through `camera.json`.
If the two don't line up, your camera or meshes don't match what you rendered.
Neither tool checks whether your world matches the video.
Put intermediate files in `/workspace/tmp`, not in `world/` or `solution/`.

# The World Package Format

The exact format of the world directory you return. TASK.md has the short
version and the rules that cost the most when they are wrong; this is the full
contract. `python -m checker` enforces all of it and prints every violation, and
a world that does not pass the checker scores zero.

Four entries: `camera.json`, `render.mp4`, `meshes/`, `dynamics/`. Plus
`solver/` for a liquid whose solver keeps no particle identity. Coordinates are
world-space metres, Z-up. All frame-indexed data share one timeline of `F`
frames, `F` being the frame count of `render.mp4`: frame `t` of the video, of
every `dynamics/` array and of every mesh archive is the same instant.

```
world/
  camera.json          {"intrinsics": K, "extrinsic": E}
  render.mp4           the rendered video
  meshes/<id>.npz      vertices_TTTT (V_t,3) float32, faces_TTTT (T_t,3) int32
  dynamics/<name>.npz  pos (F,N,3) float32, ids (M,) uint16; faces (T,3) int32 for a mesh, tets (K,4) int32 for a tetrahedral mesh
  solver/manifest.json  and the solver's own cache, for an identity-less liquid only
```

## `camera.json`

`camera.json` carries `intrinsics`, a 3×3 pinhole matrix in pixels, and
`extrinsic`, a 4×4 camera-to-world transform in metres. Both are shared by every
frame: the camera does not move. Camera axes are +x right, +y down, +z
forward, so a world point `p` projects proportionally to `K R^T (p - t)`; pixel
`(i, j)` has centre `(i+0.5, j+0.5)` and the principal point is `(W/2, H/2)`.
The scorer projects your geometry exactly this way, so getting it wrong shows up
as geometry sitting off the render.

## `meshes/`

One archive per rendered object, `<id>.npz`, `<id>` a positive integer unique
within the world. **Every rendered object needs one, including a liquid.** One
member pair per frame of the object's lifetime, which is one contiguous interval
below `F`: `vertices_TTTT` `(V, 3)` float32, finite world-space metres, and
`faces_TTTT` `(T, 3)` int32 indexing that frame's vertices; `TTTT` is the
zero-padded frame index. A frame where the object has no geometry holds empty
arrays. Vertex count and topology may change between frames; the archive
carries no cross-frame correspondence.

An object no `dynamics/` archive lists is static: identical `vertices` and
`faces` at every frame of its lifetime. The exception is an identity-less liquid,
which is listed by no archive and is not static; its ids go in
`solver/manifest.json` under `liquid_ids`.

## `dynamics/`

The state your description produced: one archive per piece of moving material,
any unique file name. Every rendered object that moves is listed in the `ids` of
some archive, except an identity-less liquid, which is named in
`solver/manifest.json` instead.

- `pos` `(F, N, 3)` float32. Column `j` is one piece of material through the whole
  video: `pos[t, j]` is its position at frame `t`, finite over one contiguous
  interval of at least one frame (its lifetime) and NaN outside it. A column is
  the same material at every frame it exists; it is not rebuilt per frame.
- `ids` `(M,)` uint16, `M ≥ 1`: the `meshes/` ids that display this material. One
  material may be displayed by several objects, and one object may display
  several materials.
- Connectivity, the same at every frame, indexing columns of `pos`. At most one
  member, `faces` or `tets`, according to how your dynamics represent the
  material:
  - surface mesh (rigid body, cloth, shell, soft-body surface): `pos` holds the
    dynamics mesh's vertices, `faces` `(T, 3)` int32 its triangles;
  - tetrahedral mesh (FEM): `pos` its vertices, `tets` `(K, 4)` int32 its
    tetrahedra;
  - particles (MPM, SPH, granular, PBD): `pos` holds every particle by slot, no
    connectivity member; a slot is NaN where the particle is not active;
  - a liquid whose solver keeps no particle identity, such as Blender's fluid
    (Mantaflow), which reseeds its particles every frame: write no archive and
    build no tracer columns. Ship the bake under `solver/` and the tracks are
    extracted from it afterwards. Its per-frame mesh still goes in `meshes/`
    like anything else, and every other moving object still needs its archive.

Every `pos` array has exactly `F` rows, and every mesh frame index falls below
`F`.

## `solver/`

The solver's own bake, so the tracks can be extracted afterwards. Required only
for an identity-less liquid; omit it otherwise.

```
world/solver/
  manifest.json        one JSON object, the seven members below
  <cache>/             the solver's cache directory, exactly as written
```

For Blender's `FLUID` domain: set `cache_type = "ALL"` before baking, or nothing
is written to disk and there is no bake to ship. Bake the full timeline, copy the
`cache_directory` whole with its `config/`, `data/` and `mesh/` subdirectories,
and do not clear it afterwards.

`manifest.json` carries the seven things the cache does not:

- `cache`, the directory name;
- `fps` and `time_scale`, the domain's own, so one frame is known to advance
  `time_scale / fps` seconds of solver time;
- `metres_per_unit`;
- `domain_min`, the domain object's own box corner in world metres. Take it from
  the object's box, not its evaluated mesh, which is the liquid surface and
  shrinks with the pool;
- `frames`, first and last covered;
- `liquid_ids`, the `meshes/` ids rendering it.

## `render.mp4`

An H.264 MP4 with the same frame count, resolution and frame rate as the input
video, rendered through `camera.json`. Those four are checked before anything
else is looked at, and a world that misses any of them is refused whole: not one
metric is read and the attempt scores nothing. So confirm what you actually
wrote, rather than what you meant to write:

```bash
ffprobe -v error -select_streams v:0 -count_frames \
        -show_entries stream=nb_read_frames,width,height,r_frame_rate \
        -of csv=p=0 world/render.mp4
```

and check it against the same reading of the input video. At every frame each
rendered object is exactly the triangle mesh stored under its id in `meshes/`,
and no geometry appears that is not in `meshes/`, so convert curves, volumes and
point clouds to triangles before rendering them. Lights, materials and world
shading are not geometry and are yours to choose.

## What the checker checks

`python -m checker` prints every violation of the format above: missing entries,
wrong dtypes and shapes, indices out of range, lifetimes that are not
contiguous, a rotation block that is not orthonormal, a frame count that
disagrees with the video, an id in `ids` with no mesh archive, a static object
whose arrays change.

# Render settings this runtime expects

The task asks for `render.mp4` at the input video's frame count, resolution and
frame rate, rendered through the camera you export. How you get there is your
business; these are the settings this machine is set up for, and the ones the
reference worlds were rendered with.

## Final render

Cycles, at the input video's own width and height, `resolution_percentage = 100`,
128 samples, denoising on. Rendering smaller and scaling up changes what the
image-based metrics see, so render at the size you deliver.

## GPU

The runtime provides an NVIDIA GPU. Cycles falls back to the CPU silently when
OptiX is not configured, which turns a twenty-minute render into an overnight
one, so configure it explicitly and fail loudly if it is missing:

```python
scene.render.engine = "CYCLES"
preferences = bpy.context.preferences.addons["cycles"].preferences
preferences.compute_device_type = "OPTIX"
preferences.get_devices()
optix_devices = [device for device in preferences.devices if device.type == "OPTIX"]
if not optix_devices:
    raise RuntimeError("No OptiX device is available")
for device in preferences.devices:
    device.use = device.type == "OPTIX"
scene.cycles.device = "GPU"
scene.render.resolution_x = input_width
scene.render.resolution_y = input_height
scene.render.resolution_percentage = 100
scene.cycles.samples = 128
scene.cycles.use_denoising = True
```

## Writing the video

Two things about `render.mp4` that fail loudly only after a long render, and are
worth setting before you start one.

`file_format = "FFMPEG"` is rejected unless the media type is set first. Blender
5 splits `ImageFormatSettings` by media, and the default is `IMAGE`, so the video
formats are not in the enum until you ask for them:

```python
scene.render.image_settings.media_type = "VIDEO"     # or FFMPEG is not selectable
scene.render.image_settings.file_format = "FFMPEG"
scene.render.ffmpeg.format = "MPEG4"
scene.render.ffmpeg.codec = "H264"
scene.render.fps = round(input_fps)                  # and fps_base for a fractional rate
```

Blender names the file after the frame range, so a `filepath` of `.../render`
becomes `render0001-0251.mp4` and the delivery is missing the `render.mp4` the
format asks for. Spell the name out and turn the suffix off:

```python
scene.render.use_file_extension = False
scene.render.filepath = str(world / "render.mp4")
```

Check what you actually wrote before you deliver: `ffprobe -count_frames` on your
`render.mp4` and on the input should agree on frame count, width, height and
frame rate. The gate compares exactly those four and refuses the world outright
when they differ, before any metric is read.

## While you are iterating

EEVEE at half size is much faster and good enough to check framing, motion and
timing against the video:

```python
scene.render.engine = "BLENDER_EEVEE"
scene.render.resolution_percentage = 50
```

Switch back to the settings above for what you deliver. A long Cycles render is
worth waiting for; let the command run to completion rather than polling it.

Stirrup 的系统提示与自动消息

Stirrup system prompt and automatic messages

runner 没有传入自定义 system_prompt，因此库加入 max_turns=500 的默认 base prompt，并附加不可与用户交互的说明。输入文件不通过 session(input_files=...) 提供，而由 container mount 提供。达到剩余 20 turns 时加入余量提示；没有工具调用且下回合没有余量提示时，追加 continue 消息。图像工具返回内容默认被移到额外的 user message。

The runner passes no custom system_prompt, so the library adds its default base prompt with max_turns=500 and guidance that user interaction is unavailable. Input files arrive through container mounts rather than session(input_files=...). Warnings begin at 20 remaining turns. When a response has no tool call and the next turn will not contain a warning, a continue message is appended. Image tool results are moved into additional user messages by default.

You are an AI agent that will be given a specific task. You are to complete that task using the tools provided in {max_turns} steps. You will need to call a finish tool as your last step, where you will pass your finish reason and any required final fields for that tool.

The context window is approaching its limit. Please create a concise summary of the conversation so far to preserve important information.

Your summary should include:

1. **Task Overview**: What is the main goal or objective?

2. **Progress Made**: What has been accomplished so far?
   - Key files created/modified (with paths)
   - Important functions/classes implemented
   - Tools used and their outcomes

3. **Current State**: Where are we now?
   - What is currently working?
   - What has been tested/verified?

4. **Next Steps**: What still needs to be done?
   - Outstanding TODOs (with specific file paths and line numbers if applicable)
   - Known issues or bugs to address
   - Features or functionality not yet implemented

5. **Important Context**: Any critical details that shouldn't be lost
   - Special configurations or setup requirements
   - Important variable names, API endpoints, or data structures
   - Edge cases or constraints to keep in mind
   - Dependencies or relationships between components

Keep the summary concise but comprehensive. Do not use any tools. Focus on actionable information that will allow smooth continuation of the work.

**Context Continuation**

Due to context window limitations, the previous conversation has been summarized. Below is a summary of what happened before:

---

{summary}

---

You should continue working on this task from where it was left off. All the progress, current state, and next steps are described in the summary above. Proceed with completing any outstanding work.

IMPORTANT: Respond with the summary as plain prose text only. Do NOT call any tools — a tool call cannot serve as a summary and will cause the summarization to fail.

# Turn-budget messages
"This is the last turn. Please finish the task by calling a finish tool."
f"You have {number_of_turns_remaining} turns remaining to complete the task. Please continue. Remember you will need a separate turn to call a finish tool."
# Continue / summary acknowledgement
"Please continue the task"
"Got it, thanks!"
# Image transport messages
f"Here is the image for tool call {tm.tool_call_id}"
f"Done! The User will provide the image for tool call {tm.tool_call_id}"
# Third summarization attempt (after base summary and prose-only warning)
f"{text_only_prompt}\n\nTools are disabled for this response. For reference, the tools available earlier in the conversation were:\n{tool_docs}"
# tool_docs
"\n".join(f"- {tool.name}: {tool.description}" for tool in self._active_tools.values())

code_exec(cmd: str)
Description: Execute a shell command in the execution environment. Returns exit code, stdout, and stderr as XML. Use `uv` to manage packages.
cmd: Shell command to execute (bash syntax). IMPORTANT: Use only relative paths. Do not use absolute paths (starting with / or ~) or reference directories outside the working directory.

view_image(path: str)
Description: View an image file from the execution environment's filesystem.
path: Path to the image file within the execution environment filesystem (supports .png, .jpg, .jpeg, .gif, .bmp, .tiff, .psd)

finish(reason: str, paths: list[str])
Description: Signal task completion with a reason. Use when the task is finished or cannot proceed further. Note that you will need a separate turn to finish.
reason: Reason for finishing.
paths: List of file paths created or modified. Do not include directories, only files.

注意两处实际含义：code_exec schema 仍要求相对路径，但本 runner 覆盖了 absolute-path 检查，使 container 成为边界；finish 的描述说需要单独一轮，而该版本执行器允许普通工具先运行、finish 最后运行。上述文本如实保留，不把提示文字等同于实现约束。工具错误、无效参数、文件不存在与图像转换错误也会进入会话；完整生成逻辑见下方源码资产。 harness/stirrup/runner.py

Two implementation details matter: code_exec’s schema still asks for relative paths, but this runner overrides absolute-path checks and treats the container as the boundary. Finish’s description asks for a separate turn, whereas this library version executes ordinary tools first and finish last within a turn. The literal wording is preserved without equating it with enforcement. Tool errors, invalid arguments, missing files and image-conversion errors also enter the conversation; the source assets below retain their complete generators. harness/stirrup/runner.py

Judge：固定提示词、图像顺序与完整问题库

Judge: fixed prompts, image order and complete question bank

两个 judge 都默认用 google/gemini-3.8-flash、temperature=0，通过 OpenRouter 发送单个 user message；没有独立 system message。VQA 先放按时间排序的 16 张 JPEG（768 px 宽，包含首尾），最后放编号问题提示；短于 16 帧的视频使用全部帧并更新 n_frames。pairwise 放 reference 的 3 fps / 512 px 帧，再放 A、B，最后放统一偏好提示。 vlm_judge/judge.py

Both judges default to google/gemini-3.8-flash, temperature=0, sending one user message through OpenRouter with no separate system message. VQA places 16 chronological JPEGs first (768 px wide, including both endpoints), then the numbered-question prompt. Videos shorter than 16 frames use all frames and update n_frames. Pairwise sends reference frames at 3 fps / 512 px, then A and B, followed by the common preference prompt. vlm_judge/judge.py

def vqa_prompt(questions, n_frames):
    head = (f"These are {n_frames} frames from one video, in order. The first is the video's "
            "first frame and the last is its final frame; the rest are sampled evenly from "
            "in between.\n\n"
            f"Answer each of the following {len(questions)} questions about that video. "
            "Each is a yes/no question about what is physically visible.\n\n")
    body = "\n".join(f"{i}. {q['question']}" for i, q in enumerate(questions, 1))
    return (head + MISSING_RULE + "\n" + body +
            "\n\nReply with one line per question, in order, formatted exactly as:\n"
            "1. Yes\n2. No\n3. Missing\n(and so on). Give no other text.")

These frames come from a 3D reconstruction that may be incomplete, so an object a question names may be absent from the video entirely.
If an object a question asks about is NOT IN THE VIDEO AT ALL, answer 'Missing' for that question.
Missing is only about objects that are not there. If the objects ARE there but the thing described does not happen, or looks different from the description, that is an ordinary No, not Missing.
Only answer Yes or No when you can see the things the question is about.

You are shown a reference recording of a physical event, then two attempts by coding models to reproduce it: video A and video B.

Which attempt is closer to the reference? Consider the geometry of the objects in the scene, how they move, the physics, the collisions, the deformations, where things end up.

Answer with one word: A or B.

f"The REFERENCE, {len(fr)} frames in order:"
f"VIDEO A, {len(fa)} frames in order:"
f"VIDEO B, {len(fb)} frames in order:"

发布问题库共有 200 场景、1,251 个计分问题与 200 个 presence 问题。计分问题类别：first_frame 371、last_frame 354、key_event 304、contact_response 218、material 4。presence Q0 仅作 gate。问题选项和答案不会附给 judge；实际 prompt 只加入 question 字符串。每个 scene 的内容不同，模板本身固定。 vlm_judge/vqa_questions_and_ontology_final.json

The released bank contains 200 scenes, 1,251 scored questions and 200 presence questions. Scored categories: first_frame 371, last_frame 354, key_event 304, contact_response 218, material 4. Presence Q0 is a gate only. Options and gold answers are not sent to the judge; the prompt inserts only question strings. Scene-specific content varies while the template stays fixed. vlm_judge/vqa_questions_and_ontology_final.json

选择任一 scene 可查看其完整展开的原始英文 VQA 输入；此处按 16 帧生成，实际短片或中断续跑时会分别改变帧数或未回答的问题子集。所有问题（含 ontology、类别、答案）与 200 个展开 prompt 均可下载。

Select any scene to inspect its complete expanded original English VQA input. These examples assume 16 frames; shorter clips or resumed runs change the frame count or unanswered question subset. Downloads include all questions with ontology, categories and answers, plus all 200 expanded prompts.

These are 16 frames from one video, in order. The first is the video's first frame and the last is its final frame; the rest are sampled evenly from in between.

Answer each of the following 10 questions about that video. Each is a yes/no question about what is physically visible.


These frames come from a 3D reconstruction that may be incomplete, so an object a question names may be absent from the video entirely.
If an object a question asks about is NOT IN THE VIDEO AT ALL, answer 'Missing' for that question.
Missing is only about objects that are not there. If the objects ARE there but the thing described does not happen, or looks different from the description, that is an ordinary No, not Missing.
Only answer Yes or No when you can see the things the question is about.

1. In the first frame, is there something that looks like a crumpled pink cloth? Answer Yes even if it is crude, roughly shaped, or a different colour.
2. At any point, does the cloth move while no gripper is touching it?
3. At any point, does the cloth move while no gripper is closed on it?
4. In the first frame, is a crumpled pink cloth in the top right corner of the white table, sitting partly inside a white basket?
5. In the first frame, do two white robot arms with black grippers stand at the front of the table, one on the left and one on the right?
6. Does the right arm reach into the corner, grip the crumpled cloth and drag it to the middle of the table?
7. Do both grippers grab two corners and lift the cloth off the table, so it hangs stretched between them?
8. Do the arms lay the cloth back down flat and then fold it over on itself several times?
9. In the final frame, is there only one pink cloth on the table, folded into a small square bundle?
10. In the final frame, are both grippers pulled back, with neither of them touching the folded cloth?

Reply with one line per question, in order, formatted exactly as:
1. Yes
2. No
3. Missing
(and so on). Give no other text.

manifest.json · Stirrup 0.2.0。没有恢复三种外部 CLI 的内部 system prompt；镜像安装命令没有版本 pin，因此当前安装得到的 CLI 也不能证明论文当时使用的提示词。

manifest.json · Stirrup 0.2.0. Internal system prompts for the three external CLIs were not recovered. Their image installation commands have no version pin, so a current installation cannot establish which prompts the paper used.

You are an AI agent that will be given a specific task. You are to complete that task using the tools provided in 500 steps. You will need to call a finish tool as your last step, where you will pass your finish reason and any required final fields for that tool.
 You are not able to interact with the user during the task.

该完整消息还明确禁止与用户交互。库支持 skills_dir，但 runner 的 session 只传 output_dir，没有加载 skills_dir，也没有父 agent 可继承 skill。

The complete message also states that user interaction is unavailable. The library supports skills_dir, but this runner’s session passes only output_dir, loads no skills_dir and has no parent agent from which to inherit skills.

指标公式：与源码逐项对应

Metric equations: mapped to source code

以下公式按当前仓库的 score key 对照实现。带帽变量表示提交，未带帽表示参考；F 是完整视频帧数，T 是该指标实际读取的帧数。相似度越高越好，只有 raw depth_error 越低越好。scorer 输出逐项读数，不自行输出 paper Overall。 docs/spec.md

The equations below follow the current repository’s score keys and implementations. Hats denote submissions; unaccented variables denote references. F is the full video frame count and T the number of frames actually read by a metric. Higher similarities are better; only raw depth_error is lower-is-better. The scorer emits per-metric readings rather than the paper’s Overall. docs/spec.md

共同算子：时间采样、SW、DTW 与三维归一化

Shared operators: time sampling, SW, DTW and 3D normalization

DINOv3、TIPS、GeoPhys、Flow、Track2D、Depth 使用该采样时间轴。Dynamic IoU 与 mesh quality 使用完整帧；Occupancy DTW 另取最多 16 个均匀帧。最终 render 仍必须交付全部 F 帧。 config/sampling.py

DINOv3, TIPS, GeoPhys, Flow, Track2D and Depth use this sampled timeline. Dynamic IoU and mesh quality use all frames; Occupancy DTW uses up to 16 evenly spaced frames. The delivered render must still contain all F frames. config/sampling.py

SW 是源码中的 sliced Wasserstein-2 近似：固定 seed=0 的随机单位方向，排序投影后线性插值到 8,192 quantiles，再取所有差值的 RMS。m 是每坐标分量 RMS，不是向量范数的 RMS；Flow 用 d=2，EMD step 用 d=3。名称 EMD 不代表代码求解了完整最优运输。 scorer/metrics/distances.py

SW is the implementation’s sliced Wasserstein-2 approximation: seed-0 random unit directions, sorted projections linearly interpolated to 8,192 quantiles, then an RMS over gaps. m is per-coordinate-component RMS, not RMS vector norm; d=2 for Flow and d=3 for EMD step. The EMD name does not mean the code solves full optimal transport. scorer/metrics/distances.py

DTW 先选择总成本最小的单调路径 π*，再除以该路径长度；不是直接优化平均成本。它允许时间对齐变形，因此不能把高分等同于逐帧时机完全正确。三维坐标经 similarity registration，并除以参考 frame-0 可见点云的 RMS radius σ。配准比较 frame-0 camera-visible ICP 与跨 16 帧动态表面 multi-start ICP 两个候选。 scorer/metrics/dtw.py

DTW first chooses the monotone path π* of minimum total cost, then divides by that path’s length; it does not directly optimize average cost. It tolerates temporal warping, so high scores do not imply exact frame timing. 3D coordinates undergo similarity registration and normalization by the reference frame-0 visible cloud’s RMS radius σ. Registration compares frame-0 camera-visible ICP with a multi-start fit of dynamic surfaces across 16 frames. scorer/metrics/dtw.py

frame-0 ICP 最多 40 次，剔除最高 20% 配对成本，采样最多 8,192 点；动态 multi-start 候选使用多个旋转初值与质心拟合。最后按参考相机视角下的配准误差选择候选，不能把这种尺度／位姿对齐后的分数解释成绝对尺度恢复。 scorer/metrics/registration.py

Frame-0 ICP uses up to 40 iterations, discards the costliest 20% of pairings and samples up to 8,192 points. Dynamic multi-start fitting uses multiple rotation seeds and a centroid fit. Candidate selection uses registration error under the reference viewpoint; a score after this scale/pose alignment does not establish absolute-scale recovery. scorer/metrics/registration.py

semantic_dinov3; semantic_tips

e 是逐帧全局 embedding：DINOv3 使用 class token，TIPS 使用其图像表示。先计算同一采样帧的 cosine，再平均并映射到 [0,1]。前者是主 Perceptual 分项，后者单列。它们主要看画面／语义，不直接验证物质对应。 scorer/metrics/semantic.py

e is a per-frame global embedding: DINOv3 uses the class token and TIPS its image representation. Average cosine similarities for corresponding sampled frames, then map to [0,1]. DINOv3 forms the main Perceptual family; TIPS is additional. These measure appearance/semantics rather than directly validating material correspondence. scorer/metrics/semantic.py

dynamic_iou

M 是参考动态物体 mask（真实数据手工标注，合成数据 rasterize）；提交 mask 来自完整场景的 z-buffer，静态物体能遮挡动态物体。忽略参考边界 band B；半径 max(1, round(0.0025√(W²+H²)))。真实数据先以相同尺度对提交 mask 做 closing。去掉 band 后 union 为空则该帧 IoU=1。 scorer/metrics/dynamic.py

M is the reference dynamic-object mask: manually annotated for real scenes, rasterized for synthetic ones. The submission mask comes from a full-scene z-buffer, so static objects can occlude dynamic ones. Ignore reference boundary band B with radius max(1, round(0.0025√(W²+H²))). Real-scene submission masks are first morphologically closed at the same scale. An empty union after band removal scores 1. scorer/metrics/dynamic.py

depth_error

真实场景：参考为 Video Depth Anything disparity，提交为几何 z-buffer 的 1/z。双方在 covered pixels 上各自减 median，再除以 mean absolute deviation（下限 10⁻⁸）。E 是去掉最大 1% residual 后的平均绝对误差；c 为 coverage。无覆盖帧记 1，整个 metric 的最坏 gate penalty 为 2。 scorer/metrics/depth.py

Real scenes: reference disparity comes from Video Depth Anything; submission disparity is 1/z from geometry. Normalize each side over covered pixels by subtracting its median and dividing by mean absolute deviation, floored at 10⁻⁸. E averages absolute residuals after discarding the largest 1%; c is coverage. A no-coverage frame scores 1, while the metric’s worst gate penalty is 2. scorer/metrics/depth.py

flow_distribution

真实场景：RAFT 参考 flow 与解析几何 flow，在两侧都有效且任一侧位移超过 0.5 px 的像素上比较。每步最多取相同的 4,096 像素位置，但比较时把向量视为无序集合。K 包含正常步骤及缺提交覆盖的惩罚步骤；后者 gap=1。无参考 coverage 或静止步骤跳过。 scorer/metrics/flow.py

Real scenes: compare RAFT reference flow with analytic geometry flow at pixels valid on both sides where either displacement exceeds 0.5 px. Sample at most 4,096 shared pixel locations per step, but compare the vectors as unordered sets. K includes valid steps and missing-submission-coverage penalty steps, whose gap is 1. Steps with no reference coverage or no motion are omitted. scorer/metrics/flow.py

track2d_dtw

真实场景：CoTracker3 跟踪参考 query；提交侧跟随 query 下方的三维材料并投影。参考轨迹需 textured seed，并在去除共享 camera drift 后距起点的最大位移超过图像对角线 δ 的 2%。D_i 是对应轨迹在 δ 单位下的 length-normalized DTW；κ=0.03。其路径配对由 query 固定，不再 Hungarian 匹配。 scorer/metrics/track2d.py

Real scenes: CoTracker3 follows reference queries; the submission tracks underlying 3D material and projects it. Reference tracks need a textured seed and maximum displacement from the seed above 2% of image diagonal δ after shared camera drift removal. D_i is length-normalized DTW between corresponding paths in units of δ; κ=0.03. Query locations fix path correspondence, without Hungarian matching. scorer/metrics/track2d.py

Track2D seed 的灰度标准差在 11×11 patch 上须大于 0.5；若某条参考轨迹没有任何可用的提交帧，该路径成本为 cap。camera drift 使用每帧的 median displacement；有可用 background query 时优先用背景，避免把主体运动减掉。scorer/metrics/track2d.py

Track2D seeds require grayscale standard deviation above 0.5 over an 11×11 patch. A reference path with no answered submission frame costs the cap. Camera drift uses per-frame median displacement, preferring available background queries to avoid subtracting the main object’s motion. scorer/metrics/track2d.py

scene_3d

合成场景：使用 frame 0 的可见表面点云与配准后的提交，距离以 σ 为单位。对每点最近邻距离先 cap，再求双向平均；任一 cloud 为空时 Chamfer=κ，分数为 0。此项不比较后续帧，也不覆盖所有隐藏面。 scorer/metrics/distances.py

Synthetic scenes: compare frame-0 visible surface clouds after registration, in units of σ. Cap each nearest-neighbor distance before averaging both directions. An empty cloud gives Chamfer=κ and score 0. This metric does not compare later frames or all hidden surfaces. scorer/metrics/distances.py

trajectory_dtw

合成场景：用共同 voxel size 按物质出生时刻采样，跟随持久材料点，视锥外／生命周期外记 NaN，比较时压紧 finite frames。两侧通常下采样到相同数量，上限 8,192 paths。Hungarian 对 capped DTW cost 找一一对应；未匹配参考路径成本 κ=1。它检查材料轨迹但容许时间重排。 scorer/metrics/trajectory.py

Synthetic scenes: sample material at birth using a shared voxel size, follow persistent points, mark out-of-view/lifetime positions NaN and compact finite frames for comparison. Both sides are normally subsampled to equal counts, capped at 8,192 paths. Hungarian assignment minimizes capped DTW costs; unmatched reference paths cost κ=1. It checks material paths while allowing temporal warping. scorer/metrics/trajectory.py

emd_step

合成场景：复用轨迹样本，比较相邻完整帧的三维位移集合，不使用轨迹间 assignment。按整段总 motion 归一化，避免近静止帧主导；参考位移少于 8 点的步骤跳过，提交少于 8 点时 distance 取该步 normalizer。总 motion 为 0 时无数值，状态及原因另存。 scorer/metrics/dynamics.py

Synthetic scenes: reuse trajectory samples and compare unordered 3D displacements between consecutive full frames, without trajectory assignment. Normalize by total sequence motion so near-static frames do not dominate. Skip steps with fewer than eight reference displacements; if the submission has fewer than eight, set distance to that step’s normalizer. Zero total motion produces no number, with state/cause stored separately. scorer/metrics/dynamics.py

附加指标与几何质量的精确范围

Additional metrics and geometry-quality scope

geophys_dinov3

取 DINOv3 第 18 层 patch tokens 的空间平均 z_t。五项 φ 分别为速度长度的 std、转角 mean、转角 std、二阶差分平方范数 mean，以及下一个 z 到前 6 帧 affine span 的距离 mean。仅平均双方都 finite 的项 J；不足 8 帧可能不可计算。 scorer/metrics/geophys.py

Take spatially averaged DINOv3 layer-18 patch tokens z_t. The five φ descriptors are standard deviation of speed magnitude, mean turn angle, standard deviation of turn angle, mean squared second-difference norm, and mean distance of the next z from the affine span of the preceding six frames. Average only descriptors finite on both sides, J; fewer than eight frames can be uncomputable. scorer/metrics/geophys.py

uni3d_point_scene; uni3d_moge_scene

Uni3D point 用合成场景配准后的 frame-0 clouds；Uni3D MoGe 用真实视频 frame 0 的 MoGe-3 point map 与提交可见表面。各自算 Uni3D embedding cosine，并映射到 [0,1]；不是额外的像素误差。 scorer/metrics/uni3d/__init__.py

Uni3D point uses registered frame-0 clouds on synthetic scenes. Uni3D MoGe compares the real video’s frame-0 MoGe-3 point map with the submitted visible surface. Both map Uni3D embedding cosine to [0,1]; neither is an additional pixel error. scorer/metrics/uni3d/__init__.py

occupancy_dtw

合成场景：最多 16 个均匀时刻，每一参考点集与每一提交点集算 SW，cap=0.5，再在 frame×frame cost 矩阵上求总成本最小的单调路径，并除路径长度。无须材料身份；参考无可见材料的帧略去，提交空帧成本为 cap。 scorer/metrics/occupancy.py

Synthetic scenes: at up to 16 uniform times, compute SW for every reference/submission point-set pair, cap at 0.5, then find the minimum-total-cost monotone path through the frame×frame matrix and divide by its length. Material identity is unnecessary. Omit reference frames without visible matter; empty submission frames cost the cap. scorer/metrics/occupancy.py

mesh_watertight; mesh_manifold; mesh_clean_faces; mesh_no_self_intersection

关键实现细节：实际聚合单位是object×frame，不是直接对所有 connected components 等权平均。每个 object 内各 component 的 verdict 先做 AND，再对所有 object-frame 行平均。文档使用 components 的说法容易混淆。检查包含清理／小洞处理及薄面判断；不是要求所有布料都成为封闭实体。 scorer/metrics/mesh.py

Important implementation detail: aggregation units are object×frame, not equally weighted connected components. Component verdicts are ANDed within each object, then averaged over all object-frame rows. The documentation’s “components” wording can obscure this. Checks include cleanup/small-hole handling and thin-surface logic; they do not require all cloth to be a closed solid. scorer/metrics/mesh.py

interpenetration

每个非空 frame 先平均各 object 的埋入顶点比例，再跨 frame 平均，最后取 1−该值；不是把整个视频所有顶点放在一个分母。埋入判定需另一 object 有非零体积、|winding|>0.5、到其表面距离超过 0.03×两物体较小 bounding-box diagonal，并通过 bounding-box 检查。 scorer/metrics/interpenetration.py

For each nonempty frame, average buried-vertex fractions across objects, then average across frames and subtract from 1. This is not a single fraction over all video vertices. Burial requires another object of nonzero volume, |winding|>0.5, distance from its surface above 0.03 times the smaller object bounding-box diagonal, and bounding-box tests. scorer/metrics/interpenetration.py

验证、VQA／Elo 与聚合公式

Validation, VQA/Elo and aggregation equations

flowchart LR
  R["Delivered render"] --> V{"Video valid?"}
  V -->|"no"| X["Applicable metrics error"]
  V -->|"yes"| P["Render metrics"]
  V -->|"yes"| W{"World valid?"}
  W -->|"no"| E["Geometry metrics error"]
  W -->|"yes"| G["Geometry metrics"]
  P --> S["Reward and detail"]
  G --> S
  E --> S
  X --> S
  S --> A["Explicit aggregation"]

适用范围、失败与分母

Eligibility, failures and denominators

源码：scorer/gate.py、scorer/outcome.py、scorer/score.py。Geometry quality 在协议上单独报告，不加入五族 Overall；几何完整性、外观和动态匹配是不同问题。

Source: scorer/gate.py, scorer/outcome.py, scorer/score.py. Geometry quality is reported separately and is not part of the five-family Overall; geometry validity, appearance and motion fidelity answer different questions.

Paper Overall 与 local viewer 不是同一聚合器

Paper Overall and the local viewer use different aggregation

此式表述论文／官方 leaderboard 的规则：相似度沿用 [0,1]，Depth 映射为 1−D/2；在适用的 case 集合 C_m 中，失败代入最坏值，统一排除无效参考；先对指标平均，再在族内等权平均，最后五族等权平均。Perceptual={DINOv3}；2D Dynamics={IoU,Flow,Track2D}；2.5D={Depth}；3D Geometry={Scene3D}；3D Dynamics={Trajectory,EMD}。VQA、Elo 和附加指标不进入 Overall。

This equation expresses the paper/official leaderboard rule: similarities retain [0,1], and Depth maps to 1−D/2. Within each eligible case set C_m, failures receive worst values and unusable references are excluded consistently. Average each metric, then equally average metrics within a family, then equally average five families. Perceptual={DINOv3}; 2D Dynamics={IoU,Flow,Track2D}; 2.5D={Depth}; 3D Geometry={Scene3D}; 3D Dynamics={Trajectory,EMD}. VQA, Elo and additional metrics are excluded from Overall.

实现差异：scorer 只写各 metric。viewer/static/app.js 的 caseSummary 先筛选 agent_exit=0、delivered 且 status≠gate_failure 的 runs，仅平均 finite reward；再对 case 平均。它还提供 shared-case 交集模式，但没有重建上述 failure-penalized Overall。直接读 viewer 表不能当作论文 leaderboard 的等价复现。现有结果表来自保留的 official JSON snapshot。 viewer/static/app.js

Implementation distinction: the scorer writes individual metrics. viewer/static/app.js caseSummary first keeps runs with agent_exit=0, delivered and status≠gate_failure, averages finite rewards, then averages cases. It offers shared-case intersections, but does not reconstruct the failure-penalized Overall above. Its tables are not an equivalent reproduction of the paper leaderboard. The retained results table uses the official JSON snapshot. viewer/static/app.js

VQA：raw accuracy、balanced accuracy 与 gates

VQA: raw accuracy, balanced accuracy and gates

C_Y/C_N 是 gold Yes/No 的正确计数，先应用 gates。1,251 个计分问题中 Yes=1,079，No=172；全答 Yes 的 raw baseline 为 0.8625，balanced baseline 为 0.5（假设通过 gates）。Missing、不合法回答、无 render 或未得到答案均不计正确；200 个 presence Q0 不进分母。已记录且不正确的 presence 行使整场景其他答案判错；当前实现若完全缺失 presence 行，并不会额外推断 gate 失败。 vlm_judge/judge.py

C_Y/C_N count correct gold-Yes/No answers after gating. Of 1,251 scored questions, 1,079 are Yes and 172 No. Always-Yes yields raw accuracy 0.8625 and balanced accuracy 0.5, assuming gates pass. Missing, invalid answers, absent renders and unanswered questions earn no correct credit; the 200 presence Q0 questions are outside the denominator. A recorded incorrect presence row invalidates that scene’s other answers; the current implementation does not infer gate failure from a completely absent presence row. vlm_judge/judge.py

Layer 1 在 API 之前以每 8 帧采样检查图像结构：edge density<1.0% 或平均 dominant quantized color share>0.95 则 gate。edge 在 160×90 RGB→BT.601 灰度上，以水平／垂直相邻像素差>6 计数；颜色在 96×54、每通道右移 5 bits（每通道 8 个级别） 后计主色占比。VQA 可 --no-layer1，score 可 --no-presence；pairwise CLI 没有 --no-layer1 选项。 vlm_judge/judge.py

Layer 1 samples every eighth frame before API invocation: gate if edge density<1.0% or mean dominant quantized-color share>0.95. Edges use 160×90 RGB converted to BT.601 grayscale and count horizontal/vertical neighbor differences>6. Colors use 96×54 frames with each channel shifted right by five bits (eight levels per channel). VQA accepts --no-layer1 and score accepts --no-presence; the pairwise CLI has no --no-layer1 option. vlm_judge/judge.py

Pairwise → Bradley–Terry → Elo

Pairwise → Bradley–Terry → Elo

w_i 包含每次胜利 1 和每次 draw 0.5，n_ij 为有效对局数。初始 p_i=1，默认迭代 9,000 次，额外 prior 相当于与强度 1 的虚拟对手平局；每轮几何均值归一化使 rating 均值为 0。官方网页使用 1000 anchor 显示，所以与 judge CLI 存在平移。默认 200 次 bootstrap 以 scene 为重采样单位，取 2.5/97.5 百分位；不是独立重采样每个 comparison。 vlm_judge/judge.py

w_i includes 1 per win and 0.5 per draw; n_ij counts valid games. Initialize p_i=1 and run 9,000 iterations by default. The prior acts as a draw against a virtual opponent of strength 1. Geometric-mean normalization each iteration centers ratings at zero. The official website displays a 1000 anchor, an offset from the judge CLI. The default 200 bootstrap draws resample scenes and use 2.5/97.5 percentiles, rather than independently resampling comparisons. vlm_judge/judge.py

调度以 round-robin 生成每场景的一轮，优先选此前比较最少的 pairs（默认 3 对），并平衡各 system 在 A 位置的次数。--pairs 0 取该轮全部 pairs，不是该场景所有可能组合。单方缺 render 自动输；两方都有 render 且都被 Layer 1 gate 才记 draw；两方都没有 render 时在排程阶段直接跳过。API 错误／无法解析胜方从 Elo 拟合排除，不记平局。 vlm_judge/judge.py

Scheduling generates one round-robin round per scene, prioritizes least-compared pairs (three by default) and balances A-slot exposure across systems. --pairs 0 includes every pair in that round, not every possible pairing for the scene. One missing render loses automatically. Two existing renders both failing Layer 1 draw; two absent renders are skipped at scheduling time. API failures or unparseable verdicts are excluded from Elo fitting, not counted as ties. vlm_judge/judge.py

源码审计发现与复现边界

Audit findings and reproduction boundaries

以下是静态源码审计得到的具体边界，不是重新运行 3,600 次实验得出的结果。这些发现补充论文已有的单视角歧义、指标代理误差和“重播不等于物理理解”局限。

The following boundaries come from static source inspection, not a rerun of the 3,600 experiments. They supplement the paper’s single-view ambiguity, proxy-metric errors and the distinction between replay and physical understanding.

上述依据：scorer/gate.py、checker/solution.py、harness/stirrup/runner.py、vlm_judge/judge.py、viewer/static/app.js。

Evidence: scorer/gate.py, checker/solution.py, harness/stirrup/runner.py, vlm_judge/judge.py, viewer/static/app.js.

可复现一次运行时应保留什么

What to preserve for a reproducible run

建议保留 repo commit、容器 image digest、CLI 版本、完整 jobs/runtime（不含凭据）、模型与 reasoning effort、三份已渲染 task 文件、session/logs、run.json、solution/world、reward.json 与 reward.detail.json，以及 prepare cache/checkpoint 版本。若要验证独立重建，应在额外的干净环境执行 build.sh 并比对输出；这是本报告建议的补充验证，不是仓库现有 scorer 自动完成的步骤。

Preserve the repo commit, image digest, CLI versions, complete jobs/runtime without credentials, model and reasoning effort, three rendered task files, sessions/logs, run.json, solution/world, reward.json and reward.detail.json, plus prepare-cache/checkpoint versions. To verify independent regeneration, execute build.sh in an additional clean environment and compare outputs. This is a recommended extra check, not an action automatically performed by the repository scorer.

# Repository entry points; fill local jobs/runtime first.
python harness/runtime/infer.py --jobs .local/jobs.toml --runtime .local/runtime.toml --dry-run
python harness/runtime/infer.py --jobs .local/jobs.toml --runtime .local/runtime.toml
python harness/runtime/score.py --jobs .local/jobs.toml --runtime .local/runtime.toml --stage prepare,score,viz
python3 vlm_judge/judge.py vqa --system <agent-model-effort> --out answers.jsonl
python3 vlm_judge/judge.py score answers.jsonl
python3 vlm_judge/judge.py pairwise --out pairs.jsonl --max-tokens 256
python3 vlm_judge/judge.py elo pairs.jsonl --boot 200

本次验证范围：读取源码与 pinned dependency，核对固定文本、question 数量、指标常量、数学定义和两种语言页面结构；未调用 judge API、未执行 GPU inference/scoring，也未声称复现作者分数。

Validation performed here: source and pinned-dependency inspection, checks of fixed text, question counts, metric constants, mathematical definitions and both language structures. No judge API calls or GPU inference/scoring were run, and the authors’ scores were not independently reproduced.

结果：静态外观更强，动态仍是短板

Results: stronger static reconstruction, weaker dynamics

以下精确数值来自核查当日的官方 leaderboard JSON，与论文图 4 的舍入结果一致；均为作者报告，本研究未重新运行 3,600 次评测。点击列标题排序，窄屏可横向滚动。模型名与推理档位采用论文标注。 [1] [2]

The precise values below come from the official leaderboard JSON on the check date and agree with the rounded results in paper Figure 4. These are author-reported results; this study did not rerun the 3,600 evaluations. Click column headers to sort; scroll horizontally on narrow screens. Model names and reasoning settings follow the paper. [1] [2]

Astra [Max] 综合领先，Opus 5.5 的三维分项更高。前者 Overall 为 0.791，后者为 0.778；Opus 的 3D Geometry / 3D Dynamics 为 0.937 / 0.746，高于 Astra 的 0.922 / 0.735。Overall 与 VLM Elo 的排序并不相同：Astra [High] 的 Elo 高于 Opus 5.5，但 Overall 更低。最强开源权重模型也取决于指标：DeepSeek 的 Overall 为 0.597，Qwen 的 VQA 和 Elo 更高。 [2]

Astra [Max] leads Overall; Opus 5.5 scores higher in the 3D families. Their Overall scores are 0.791 and 0.778. Opus reaches 0.937 / 0.746 on 3D Geometry / 3D Dynamics, above Astra’s 0.922 / 0.735. Overall and VLM Elo also order models differently: Astra [High] has higher Elo but lower Overall than Opus 5.5. The strongest open-weight model depends on the metric too: DeepSeek scores 0.597 Overall, while Qwen has higher VQA and Elo. [2]

0.91 对 0.67 的含义：论文把 Perceptual 与 3D Geometry 平均为“静态”，把 2D Dynamics 与 3D Dynamics 平均为“动态”；这里没有把深度分项计入静态。Astra [Max] 的这两组均值分别约为 0.91 和 0.67。不同指标虽都在 [0,1]，其阈值与含义不同，因此这是评测上的差距，不是统一物理单位下的误差比。 [1]

What 0.91 versus 0.67 means: the paper averages Perceptual and 3D Geometry as “static,” and 2D Dynamics and 3D Dynamics as “dynamic.” Depth is not included in this static grouping. Astra [Max] averages approximately 0.91 and 0.67 in these groups. Although the metrics share a [0,1] range, their thresholds and meanings differ; this is an evaluation gap, not an error ratio in a common physical unit. [1]

推理投入的变化：同一模型内有改善，跨模型不成立

Reasoning effort helps within a model, but token volume does not rank models

从 Low 到 Max，Overall 增加约 0.064，报告的每任务费用约为原来的 3.24 倍（本研究根据表值计算）。这是一次推理档位变化实验，不是等预算消融。跨模型的总 token 与步数和 Overall 几乎不相关，论文报告 |ρ| ≤ 0.10；全部 token 中 97% 是缓存读取，因此“几百万 token”不能当作几百万新生成的推理 token。费用是该实验的计费估计，开源权重模型采用 OpenRouter 价格并假定 95% 缓存命中率。 [1] [2]

From Low to Max, Overall rises by about 0.064 while reported cost per task increases approximately 3.24× (calculated here from the table). This varies reasoning settings rather than holding budget fixed. Across models, total tokens and steps barely correlate with Overall: the paper reports |ρ| ≤ 0.10. Cache reads account for 97% of all tokens, so “millions of tokens” does not mean millions of newly generated reasoning tokens. Costs are estimates for this experiment; open-weight models use OpenRouter prices with an assumed 95% cache hit rate. [1] [2]

策略与数据差异，比单一冠军更有信息

Strategies and scene differences reveal more than a single winner

所有解法中，67% 使用解析运动，19% 自写模拟，10% 使用 Blender 物理，3% 使用关键帧；少量其他方法与舍入使总数不一定等于 100%。Opus 5.5 的自写／Blender 模拟合计为 76%；Astra 从 Low 到 High 到 Max 的模拟比例为 13% → 22% → 32%。这些是代码策略的统计，并不能证明使用模拟导致更高分。 [1]

Across solutions, 67% use analytic motion, 19% custom simulation, 10% Blender physics and 3% keyframes. Small residual categories and rounding mean the shares need not sum to 100%. Opus 5.5 uses custom/Blender simulation in 76% of solutions; Astra’s simulation share rises 13% → 22% → 32% from Low to High to Max. These describe implementation choices and do not establish that simulation causes better scores. [1]

真实视频更难：Astra [Max] 的真实／合成 VQA 为 0.853 / 0.898，Opus 5.5 为 0.775 / 0.880。论文还发现多材料场景的外观和 VQA 较弱，布料／绳子类的二维运动和深度更难。应称为跨场景来源的表现差异，不能据此断言未见过的物理干预也能泛化。 [1] [2]

Real footage is harder: Astra [Max] scores 0.853 / 0.898 VQA on real/synthetic scenes, versus Opus 5.5’s 0.775 / 0.880. The paper also finds weaker appearance and VQA on multi-material scenes, and harder 2D motion and depth for cloth/rope. These are performance differences across scene sources, not evidence of generalization to unseen physical interventions. [1] [2]

自动评分与人类偏好的一致性

Agreement between automated scores and human preferences

76 位参与者完成 3,587 次两两判断，覆盖 17 个模型配置；Opus 5.5 因发布时间未参加人类研究。模型层面 Human Elo 与 VLM Elo 的 Spearman ρ = 0.980，与 Overall 的 ρ = 0.96。在 916 个共同比较上，人类与 VLM 的一致率为 89.3%，人类间重复比较一致率为 92.1%。但相差 50 Elo 以内的模型，单次裁判判断接近随机，因此不能用总体高相关来保证相邻名次可靠。 [1]

Seventy-six participants supplied 3,587 pairwise judgments over 17 model configurations; Opus 5.5 was absent from the human study because of its release timing. At the model level, Human Elo correlates with VLM Elo at Spearman ρ = 0.980 and with Overall at ρ = 0.96. Human–VLM agreement is 89.3% on 916 shared comparisons, versus 92.1% agreement between humans on repeated comparisons. However, individual judge decisions are near chance for models within 50 Elo, so high aggregate correlation does not guarantee reliable adjacent ranks. [1]

局限与判断：这是否测到了物理理解？

Limitations and assessment: does this measure physical understanding?

• 最关键的欠约束：同一视频可能由多种几何和运动程序生成。按时间安排位置也能取得高分；测试没有系统改变初态、外力或材料，也没有要求预测观察窗口之后的事件。因此证据支持“动态场景重建能力存在差距”，不支持“高分模型已经学会可迁移的物理规律”，也不能仅凭某模型常用解析运动就判定其完全不懂物理。 [1]
• The central ambiguity: Multiple geometry and motion programs can explain one video. Prescribed positions can score well; the test does not systematically intervene on initial conditions, forces or materials, or demand prediction beyond the observed window. Evidence supports a gap in dynamic reconstruction, not proof of transferable physical laws in high-scoring models. Frequent analytic motion also cannot alone establish a complete absence of physical understanding. [1]
• VQA 的类别偏斜：论文图 5 的始终回答 Yes 基线为 87.5%，Astra [Max] 为 87.6%。这个基线不是一个能生成合格世界的智能体，因此不能替代完整 benchmark；但它说明 87.6% 不能按平衡二元问答解读为强物理推理证据。VQA 应与门控、各类问题、Elo 和几何／运动分项一起阅读。 [1]
• VQA answer imbalance: Paper Figure 5 gives an always-Yes baseline of 87.5%, versus 87.6% for Astra [Max]. This baseline is not an agent producing a valid world and cannot replace the full benchmark. It does show why 87.6% should not be read like accuracy on balanced binary questions or as strong physics-reasoning evidence. Read VQA alongside gates, question categories, Elo and geometry/motion metrics. [1]
• 模型、工具框架和预算混在一起：闭源模型运行各自原生 CLI，开源权重模型使用 Stirrup；单场景仅一次运行，且没有统一 token／费用预算。结果评估的是具体智能体系统，不能纯粹归因于模型权重。场景 bootstrap 的区间也不能替代重复运行来测量生成随机性。 [1]
• Models, harnesses and budgets vary together: Proprietary models use their native CLIs; open-weight models use Stirrup. There is one run per scene and no shared token or dollar budget. Results evaluate particular agent systems rather than model weights alone. Bootstrapping scenes does not replace repeated runs to measure generation variability. [1]
• 参考与指标都有选择：真实数据依赖估计深度／光流，3D Geometry 只比较配准后的初始可见表面，DTW 容忍部分时序差异，五分项平均也隐含权重选择。有限、偏稳定视角的 200 场景不能覆盖全部日常物理。低成本几何有效性检查又可能奖励简单但错误的物体，因此它们未进入 Overall。 [1]
• References and metrics make choices: Real references depend on estimated depth/flow; 3D Geometry checks aligned initial visible surfaces; DTW tolerates some timing differences; equal family averaging embeds a weighting choice. Two hundred scenes favoring stable views do not span everyday physics. Cheap mesh-validity checks may reward simple but inaccurate objects, so they remain outside Overall. [1]

值得保留的结论：这是一个有用的“从视觉到可检查程序”的测试平台，最有价值的信号是分项错误与策略差异。下一步应固定初始程序，改变重力、摩擦、物体位置或施加外力，并检查更长时间与新视角；再用等预算、重复运行比较解析动画和模拟。前一类干预与长程预测是作者提出的方向，后面的控制设计是本研究建议，均不是论文已验证的结果。 [1]

The useful conclusion: this is a valuable testbed for translating vision into inspectable programs, with diagnostic errors and strategy differences providing its strongest signals. Next, keep the inferred program fixed, vary gravity, friction, object positions or external forces, and test longer horizons and new views; compare analytic animation and simulation under matched budgets and repeated runs. Interventions and longer-horizon prediction are author-proposed directions; the additional controls are this study’s suggestions. None are established results of the paper. [1]

如何使用：先检查一个场景的完整交付物

Using it: inspect one scene’s complete deliverable first

1. 获取数据与环境：官方仓库提供推理容器、checker、scorer、VLM 裁判和可视化工具；Hugging Face 提供数据。固定代码版本、模型版本、CLI、推理档位和预算后，先运行一个场景。
2. Get data and environment: the official repository provides inference containers, checker, scorer, VLM judge and visualization tools; Hugging Face hosts data. Record code/model versions, CLI, reasoning setting and budget, then start with one scene.
3. 验证程序与世界：确认 build 程序能独立重建；使用 checker 检查格式，用预览确认导出相机与网格和渲染一致。checker 不判断重建是否真的匹配视频。
4. Validate the program and world: check that the build independently regenerates the result. Use the checker for format validity and its preview to check camera/mesh alignment with the render. The checker does not determine whether the reconstruction matches the reference.
5. 准备参考，再评分：scorer 的 prepare 阶段生成可复用的参考估计，score 阶段输出各指标。既看具体错误帧和分项，也保留失败状态；需要语义事件或偏好比较时再运行 VQA／Elo。
6. Prepare references, then score: the scorer’s prepare stage generates reusable reference estimates; its score stage writes per-metric outputs. Inspect failing frames and individual metrics while retaining failure states; add VQA/Elo for semantic events or preference comparisons.
7. 按用途选模型：复现视频外观、重建可编辑几何和推断可迁移模拟是不同目标。先确定关心的输出与分项，再看预算；不要仅凭一个 Overall 数字把模型当作通用物理引擎。
8. Select for your objective: recreating video appearance, recovering editable geometry and inferring transferable simulations are different goals. Choose the outputs and metrics that matter, then compare budgets; an Overall score alone does not qualify a model as a general physics engine.

以上流程依据公开 README 和格式规范整理；本次只阅读并核对源码文档与发布结果，未安装或执行完整 GPU 基准。 [3]

This workflow follows the public README and format specification. This study inspected source documentation and reported results; it did not install or execute the full GPU benchmark. [3]

资料、版本与证据边界

Sources, version and evidence boundary

1. 4DCodeBench — arXiv:2610.03715v1

   2026-10-02 提交。方法与结论见第 2–6 节；指标见附录 D，人工评测见 E，策略与运行统计见 F.4 和 G。PDF 首页另印有 10-5-2026，本页发表日期采用 arXiv 提交记录。

   Submitted 2026-10-02. Methods and conclusions: Sections 2–6; metrics: Appendix D; human study: E; strategies and runtime statistics: F.4 and G. The PDF also prints 10-5-2026 on its first page; this study uses the arXiv submission record for publication date.

   PDF
2. 官方项目Official project · 排行榜 JSONLeaderboard JSON

   在 2026-10-07 获取。排行榜、费用和图表使用此快照；实时网站后续可能变化。原始数据另存为随页资产，便于复查。

   Retrieved 2026-10-07. The table, costs and chart use this snapshot; the live site may subsequently change. The raw data is included as a page asset for inspection.

   JSON
3. 官方仓库Official repository · 评测规范Evaluation specification

   在 2026-10-07 阅读 README 与格式／评测规范；数据入口为 4DCodeBench on Hugging Face。

   README and format/evaluation specification read on 2026-10-07; data entry point: 4DCodeBench on Hugging Face.

图片来自原论文，保留原图标签；图 5 由公开数值重新绘制。解释、计算与批判性判断已和作者实验结果区分。本页提供完整中英双语正文；右下角语言按钮可切换。

Images come from the paper with original labels preserved; Figure 5 is replotted from public values. Explanations, calculations and critical assessments are distinguished from author-reported experiments. This page contains full Chinese and English versions, switchable using the language button at the lower right.

本次扩充为固定 commit 的系统审计：harness/runtime/infer.py、harness/runtime/agents.py、docs/spec.md、vlm_judge/judge.py。Stirrup 0.2.0 源码来自 PyPI wheel，hash 见 prompt manifest。

This extension audits a fixed commit: harness/runtime/infer.py, harness/runtime/agents.py, docs/spec.md, vlm_judge/judge.py. Stirrup 0.2.0 source comes from the PyPI wheel, with its hash in the prompt manifest.