What Makes a Game Dataset Training-Ready?

A game dataset is training-ready when a downstream team can rebuild the same experience, map every sample to a documented schema, prove the rights envelope, and fail a sample in QA without guessing. Specification, source, dependencies, runtime validation, gameplay loop, world state, actions, metadata, licensing, provenance, and QA are not polish. They are the dataset. A zip of Unity projects or a pile of MP4s can be a useful ingredient. It is not, by itself, training data for models that understand the world.

The distinction matters because world-model and agent teams now ask for “game data” the way computer-vision teams once asked for “images.” Images at least share a rectangle of pixels. Games do not share a runtime. If the title does not boot, or boots into a different physics build, every later annotation is fiction.

A folder of game files is not a dataset

Game files become a dataset only after they are bound to a learning objective. The objective might be action-conditioned video, code-agent playability, spatial layout, or policy evaluation. Each objective needs a different record. The shared minimum is that another engineer, six months later, can answer: what ran, what the agent saw, what it did, what the world did in return, and whether they are allowed to train on it.

Research systems such as GameNGen and later interactive video world models already assume an aligned action-observation stream, not an unstructured project dump (Valevski et al., 2024). If you cannot reconstruct that stream, you do not have world-model data. You have a backup.

The acceptance contract

Specification and source

Write the spec as if the model team will never open the editor. Include the intended observation, the legal action set, episode length, reset rules, and the fields that constitute success. Pin source. A dataset whose scenes silently track an internal branch is not versioned, even if the export folder has a date.

If the program includes code for coding agents, the spec must say whether hidden tests, shaders, or third-party plugins are in scope. Playability and compilation are different bars; that split is covered in playable game datasets.

Dependency versions and runtime validation

Record engine version, platform, graphics API, and package lockfiles. Then boot. Runtime validation means a clean machine can start the build and reach a known spawn without a human clicking around the editor. If validation needs a GPU-local cache that is not in the dataset, document it or the dataset is incomplete.

Failure mode: shipping content that only loads because an artist’s local cache still holds shaders. The sample looks present. Training jobs crash. QA that only opens the project in the original workspace will not catch this.

Gameplay loop, world state, and actions

A training-ready loop has a clock. On each tick — or each recorded frame — you should be able to join:

• observations (video, UI, proprioception, or buffers);
• actions (keys, pads, API calls);
• state (transforms, flags, scores, inventories);
• events (deaths, triggers, scene loads).

If you only have video, you have a generative-video ingredient. You do not yet have game data for world models. If you only have code, you may have a coding-agent ingredient. You do not yet have a dynamics dataset.

{
  "episode_id": "titleA-ep-01842",
  "build": "0.9.4+sha.9f2c",
  "tick_hz": 30,
  "seed": 441,
  "scene": "harbor_night",
  "actions": [{"tick": 12, "type": "key", "name": "Jump", "down": true}],
  "state": [{"tick": 12, "player_z": 1.8, "on_ground": false}],
  "events": [{"tick": 48, "name": "checkpoint", "id": "dock_02"}]
}

This schema is a contract, not a file format religion. JSONL, protobuf, or engine replays are all fine if the join key is explicit.

Metadata, licensing, provenance

Metadata should make sampling possible: biome, lighting, difficulty, camera rig, control scheme, language, accessibility settings. Without it, you cannot build a balanced split.

Licensing is a first-class field. Engine terms, marketplace assets, music, and voice lines have different rules. Training-ready means the manifest says what may be trained on, what may be redistributed, and what must stay in a confidential engine. This is not legal advice; it is dataset hygiene. The companion note is licensed game data for AI training.

Provenance answers a later audit: capture machine, operator or automated policy, random seed, and parent dataset version. If a clip is regenerated after a physics hotfix, it needs a new version, not a silent overwrite.

QA that can fail a sample

Minimum QA:

1. Boot and reach spawn.
2. Replay stored actions and compare hashes of selected state fields.
3. Schema validation.
4. Rights coverage: every referenced asset ID exists in the manifest.
5. Split integrity: eval scenes do not appear in train.
6. Human play sample for “this is the intended game,” because automation will certify a broken jump if the spec forgot the jump.

Failure mode: QA that only checks file presence. A complete folder can still be an unplayable, unlicensed, desynchronized record.

Production view

Training-ready is a property of the record, not of the title’s quality. A small, ugly prototype with a pinned build, a tick log, and a rights manifest can beat a beautiful unreleased game whose capture pipeline dropped inputs.

Do not promise that any capture is complete. Promise that failures are visible. The cheapest time to discover a missing lockfile is before the model team starts a run, not after they report that “the game data does not generalize.”

A delivery layout that survives handoff

A training-ready drop is boring on purpose:

dataset/
  SPEC.md
  MANIFEST.json
  builds/linux-x64/
  lockfiles/
  episodes/
  splits/train.txt
  splits/eval.txt
  qa/boot.jsonl
  qa/replay_hashes.jsonl

If SPEC.md and MANIFEST.json are missing, the rest is raw material. If splits are missing, the model team will invent them and contaminate eval. If qa/ is missing, you will rediscover boot failures in GPU hours.

What changes when the consumer is a world model versus a coding agent

World models care about tick-aligned actions and cameras. Coding agents care about build graphs and play oracles. A dataset can serve both only if both contracts are present. Shipping a playable prototype without a tick log does not satisfy a world-model RFP. Shipping trajectories without a bootable build does not satisfy a coding-agent RFP. Name the consumer in the spec so leftover files are not misread as completeness.

External anchors worth keeping next to an internal spec include the glTF 2.0 specification when 3D payloads ship with the game, and the original interactive-video papers that assumed aligned actions rather than project archives. The research trend is moving toward denser controls (ForgeWM); production datasets should get stricter, not looser.