Game Data for World Models: Video, Actions, State, and Telemetry

World-model training data from games is a synchronized record of video, actions, state, events, telemetry, and camera — joined on a single clock. Video alone is not enough. A video shows what appeared on screen. It does not reliably show what the agent commanded, what the simulation believed, what happened off-screen, or whether two frames are 33 milliseconds or 200 milliseconds apart after an encoder dropped time.

Interactive world-model papers keep rediscovering this the hard way. Action-conditioned generators need frame-aligned controls (ForgeWM). Object interaction needs event-dense labels, not only walkthroughs (ActWorld). If your capture pipeline emits MP4s and a separate, unsynced input log, you are not collecting game data for world models. You are collecting two datasets that happen to be about the same session.

The join is the product

Temporal alignment is the difference between a training record and a documentary. If actions are stored at 60 Hz and video at 24 fps without a mapping, distillation recipes that pack controls into VAE hops will silently jitter the supervision. The model then “fails to follow WASD” when the dataset never agreed on WASD.

Why only video fails

Video-only corpora teach appearance and some implied physics. They do not distinguish:

• a camera pan from an object slide (the WorldCraft problem);
• a scripted cutscene from a player action;
• a blocked jump from a missed input;
• a desync from a real world change.

Early neural game simulators such as GameNGen trained on aligned observation-action pairs from an engine, not on YouTube rips of the same title (Valevski et al., 2024). That design choice is still the correct default.

Failure mode: upsampling a streamer’s encode, running a VLM to guess keys, and calling it action-conditioned data. You can get a demo. You cannot get a trustworthy closed-loop evaluator.

A practical sync contract

Pick a master clock. Engine tick is better than capture-card PTS. If you must use a capture card, record both and measure lag.

{
  "session_id": "wm-harbor-00077",
  "master_clock": "engine_tick",
  "tick_hz": 30,
  "video": {"path": "obs.mp4", "fps": 30, "pts_to_tick": "lut.parquet"},
  "actions": "actions.jsonl",
  "state": "state.jsonl",
  "events": "events.jsonl",
  "telemetry": "telemetry.jsonl",
  "camera": "camera.jsonl",
  "objects": "objects.jsonl"
}

Every JSONL row shares tick. If a stream is slower, repeat the last value or mark it missing. Do not interpolate actions you did not measure.

Camera and object metadata

Camera fields should include intrinsics, extrinsics or engine camera pose, FOV, and projection type. Object metadata should include instance IDs, classes, visibility, and preferably world transforms. Without those, you can still train a video model. You cannot train spatial intelligence or off-camera persistence without guessing, which is exactly when generated worlds start teleporting furniture.

Telemetry is not vanity

Record frame time, dropped frames, resolution changes, and — for networked titles — RTT. A hitch is a dynamics event. If you drop it, the model learns impossible motion. Telemetry also explains eval disagreements: the policy looked wrong because the capture machine thermal-throttled.

Training data versus evaluation data

Training can tolerate some missing state if the objective is appearance. Evaluation cannot. An eval episode should freeze the build, the seed, the action interface, and the metric. Preferred metrics:

• action follow-through on held-out traces;
• state prediction on fields you actually logged;
• loop-closure / revisit tests for memory;
• rights-safe replay, not a public YouTube URL.

Do not train on the eval traces. World-model papers that share first frames and action scripts for comparison only work if those scripts never entered the training mix.

QA

1. Cross-correlation of input onsets with animation onsets on a sample.
2. Tick monotonicity.
3. Video duration versus tick count.
4. Null rate per field.
5. Holdout scene list.
6. Human spot-check of desync, especially mouse look.

If QA is “we watched a few clips,” you will ship a generative-video set with a world-model label. Pair this capture standard with the broader training-ready checklist.

Capture architectures that actually stay aligned

Three patterns show up in production. Only the first two are trustworthy for action-conditioned training.

Engine-side recorder. The game writes observations, actions, and state on the simulation tick. Video is either an in-engine buffer or a lossless frame dump. This is what GameNGen-style work assumes. Latency between input and display is known because it is the engine’s latency.

Instrumented replay. You store a deterministic replay and re-render later at the resolution the model needs. Alignment is excellent if the replay is actually deterministic. Many networked titles are not. If you cannot seed physics and netcode, stop calling it a replay and start calling it a log.

Capture card plus input hook. Useful when you cannot modify the build. Measure the delay with a calibration scene (flash on button-down). Store the delay. If you cannot measure it, you cannot train look-axis models.

Failure mode: mixing the three without labeling which episode used which. Distillation then sees a random control delay and concludes the student is unstable.

What “synchronized” means numerically

Pick tolerances and write them down:

• action-to-tick: 0 (same tick) for engine recorders;
• video-to-tick: ≤ 1 frame after the LUT;
• camera pose-to-tick: same as video;
• event timestamps: integer ticks, not wall-clock strings;
• dropped frames: explicit gaps, never interpolated actions.

If a clip violates the tolerance, quarantine it. Training on “almost synced” mouse look is worse than dropping the clip, because the error is systematic in yaw.

Object metadata without boiling the ocean

You do not need every actor. You need a documented subset: player, held items, interactables in view, and anything named in the eval rubric. Identity must persist across occlusion if off-camera state is in scope. If you cannot afford IDs, do not advertise spatial intelligence.