WorldCraft: From Camera Navigation to Object Trajectories — Data Requirements for Manipulable Worlds

Executive summary

WorldCraft extends interactive video world models from camera navigation to object-level trajectory actions. The user clicks an object, sketches a path, and the model generates future frames in which that object follows the path while the camera continues to move. The paper reports three mechanisms: Normalized World Trajectory (NWT), Spatial-Pathway LoRA (SP-LoRA), and Trajectory-Anchored State Persistence (TASP). The backbone is WorldPlay / HunyuanVideo-1.5.

The paper reports that screen-space trajectories entangle object motion with ego-motion, that naive full fine-tuning hurts camera fidelity, and that autoregressive memory stores stale locations if an object moves off-camera. Those three failure modes are data problems as much as model problems.

We verified the arXiv page and project page on 2026-08-24. We did not find a public training-code repository we could confirm. No paper license was independently verified.

Why this research matters

A world model that can only move a camera is a viewer. Agents, games, and generative-video directors need to move entities. WorldCraft’s action is a continuous trajectory attached to a selected object — closer to a robotics push than to a discrete game button, and closer to a VFX layout pass than to a text prompt.

For data teams, the paper is a reminder that “actions” are not one type. ForgeWM’s actions are device packets. ActWorld’s actions are verbs plus phases. WorldCraft’s actions are 3D paths. Mixing those schemas without saying so produces datasets that look rich and train poorly.

The problem the paper addresses

The authors evaluate a literature split. Interactive world models (Genie 3, WorldPlay, GameCraft, Matrix-Game 2.0) navigate cameras autoregressively without object actions. Trajectory-guided video methods (DragAnything, Wan-Move) move objects, often with a static camera and without long-horizon memory. WorldCraft aims to compose both.

Three coupled challenges, as the authors state them:

1. Camera-trajectory coupling in pixel space.
2. Preserving a pretrained camera controller while adding object control.
3. Off-camera state: the object’s location must change even when it is not visible.

How the method works

NWT lifts a user path into a camera-invariant world coordinate frame, then re-projects it under the current pose. Depth is estimated, with an iterative refinement ablation reported as helpful under large rotations.

SP-LoRA injects the trajectory through the spatial-control pathway (action_in and ProPE projections) with about 50M trainable parameters. The paper reports that full fine-tuning lowers trajectory error but raises rotational RPE, damaging camera control.

TASP treats the world trajectory as persistent spatial state and refreshes autoregressive memory after trajectory-conditioned generation so moved objects can reappear in the right place after leaving the view.

Conditioning reuses displaced first-frame latents at target positions — an in-context trajectory channel compatible with the pretrained patch embed. Our interpretation: the dataset must provide trajectories in a coordinate system that still makes sense after the camera leaves. Pixel polylines are not that system.

What data the method requires

Appendix Table 6 reports 27,027 filtered clips, standardized to 30 fps and 97 frames (~3.2 s). Camera parameters are converted to per-frame 3×3 intrinsics and 4×4 world-to-camera matrices. Object displacement statistics are reported. Subject identification methods vary by source (the table distinguishes estimated versus provided cameras and how the target object is identified).

A practical implementation would still need collision flags. A sketched path that tunnels through a wall is a valid WorldCraft action and a failed robotics or game-eval action. The method optimizes trajectory following, not physical feasibility.

Dataset and supervision signals

The authors report WorldCraft achieving the lowest trajectory error and strong pixel/semantic metrics on 50 static-camera clips, and camera RPE of 0.131 at 61 frames versus 0.120 for WorldPlay, improving to 0.123 at 253 frames versus WorldPlay 0.130. Full fine-tuning’s camera regression is an experimental result, not a slogan: adding object supervision through the wrong parameters overwrites the camera skill.

Training versus evaluation again diverge. Training uses short standardized clips. The interesting claim — off-camera persistence — needs longer rollouts and a held-out revisit protocol. If the revisit view leaked into training, TASP looks better than it is.

What the reported results show

The paper reports accurate object control, preserved camera fidelity, and maintained object state across autoregressive rollouts with off-camera excursions. Ablations credit world-space trajectories over pixel-space paths, iterative depth, SP-LoRA over full FT, and a Static-BI → Dynamic-AR curriculum.

Our interpretation: WorldCraft shows that object actions should be stored in world coordinates with a camera track beside them. It does not show that a click-and-drag path is a complete action space for agents. Grasping, articulation, and contact still sit outside a 3D polyline.

Limitations and open questions

• Appendix D discusses limitations; we do not copy it. Independently: short 3.2 s training clips versus long interactive use; monocular depth errors; single-object emphasis; no verified public training dump; no verified license.
• Project page is available; code was not independently confirmed.
• “First to combine both action modalities” is the authors’ claim. We treat it as their positioning, not as a surveyed proof.
• Trajectory error in pixels is not task success.

From research to production

If you already log cameras for a NeRF or Gaussian-splat pipeline, you are halfway to a WorldCraft-like dataset. The missing half is object-centric.

Production checklist:

1. Stable instance IDs across frames, including occlusion.
2. World-space transforms, not only crop boxes.
3. Camera extrinsics in the same world frame.
4. Explicit off-camera intervals.
5. A holdout of revisit shots.
6. Rights on assets being moved — moving a licensed character mesh is not the same as moving a reconstructed sofa.

QA should replay the stored path in the engine or DCC tool and measure 3D error, then compare to the generated video. If you only measure 2D path error under a static camera, you are testing DragAnything’s problem, not WorldCraft’s.

{
  "sequence_id": "wc-00491",
  "fps": 30,
  "frames": 97,
  "camera": [{"t": 0, "K": "...", "E": "..."}],
  "object_id": "crate_12",
  "world_traj": [{"t": 0, "xyz": [1.2, 0.0, 3.4]}],
  "visible": [{"t": 0, "on_screen": true}]
}

GaussMathematics perspective

The paper reports that object-level trajectories can be added to an autoregressive camera world model by lifting paths into world space, adapting only the spatial-control pathway, and persisting the trajectory as state. The authors evaluate static-camera trajectory following and camera-only fidelity. Those are the authors’ results.

Our interpretation is that WorldCraft is a coordinate-systems paper. Teams still labeling object motion as 2D polylines on MP4s are creating data that fights the camera. Teams that already have engine transforms are sitting on a better action encoding than the research pipeline can extract from video.

From a production-data perspective, the dangerous shortcut is to treat generated click-and-drag videos as robotics or game-agent supervision. A path that is easy for a video model may be kinematically illegal for a manipulator and legally undeclared for a character asset. Licensed 3D worlds can emit WorldCraft-like fields without guessing depth.

A practical implementation would still need topology and collision metadata if the downstream task is spatial intelligence rather than visual following. That is the bridge to 3D asset datasets.

We do not claim the reported RPE numbers will survive a different backbone. We do claim that camera-object composability should be a dataset acceptance test: hold the object path fixed, vary the camera, and fail the clip if the object annotation was stored only in pixels.

References

• Gu et al., *WorldCraft: From Camera Navigation to Object Manipulation in Interactive Video World Models*, arXiv:2605.25077, 24 May 2026. https://arxiv.org/abs/2605.25077
• Project page: https://nevsnev.github.io/WorldCraft/
• WorldPlay; HunyuanVideo-1.5; DragAnything; Wan-Move; Matrix-Game 2.0; GameCraft; Yume — comparison systems named by the authors