Why passive video is not enough
A camera pointed at a game can produce useful pixels. It cannot, by itself, tell a world model which command was issued, which object moved, or whether a collision was an event or a rendering artifact. Engine-native capture keeps observation, action, and state on one clock so later training and evaluation do not have to reverse-engineer the session.
One environment, synchronized signals
We work inside customer-authorized game and simulation environments. The unit of delivery is a record: frame-aligned video, recorded inputs, camera parameters, telemetry, engine events, task labels, and provenance. Not every environment exposes every signal. The schema is scoped to what the engine can emit and what the model actually needs.
Data layers
Each layer has example fields, a reason the model may need it, and a check we can run. Fields are illustrative of the record structure, not a claim that every program includes every key.
| Layer | Example fields | Why the model needs it | How we verify |
|---|---|---|---|
| Observation | frame_id, rgb, optional depth / segmentation, view_id | Gives the model the visual evidence it is expected to predict or condition on. | Resolution, colorspace, and missing-frame checks against the capture clock. |
| Action | control_vector, discrete_command, duration, agent_id | Records what was actually issued, not what a later model infers from pixels. | Action presence on every tick in the policy window; illegal command flags. |
| State | object_id, pose, velocity, inventory, flags, relations | Exposes world variables that video compresses away: occupancy, identity, latent goals. | Schema conformance and identity continuity across ticks. |
| Camera | extrinsics, intrinsics, fov, projection, attached_to | Makes multi-view geometry recoverable and keeps camera motion from being hallucinated. | Frustum consistency and timestamp alignment with the observation stream. |
| Telemetry | engine_tick, fps, physics_step, latency_ms | Separates simulation time from wall time so long-horizon models can stay causal. | Monotonic tick and bounded jitter against the declared timestep. |
| Events | event_type, actors, payload, tick | Marks collisions, interactions, successes, and failures as first-class labels. | Engine-emitted events reconcile with state deltas at the same tick. |
| Task | task_id, instruction, success_criteria, horizon, split | Ties an episode to a learning objective instead of an unlabeled play session. | Success criteria evaluate on recorded state, not on a hidden side channel. |
| Time | tick, timestamp_utc, frame_index, episode_t | Is the join key. Without a shared clock, the rest of the stack is a pile of files. | Join completeness: every observation tick has action, camera, and state rows. |
| Provenance | environment_id, authorization_id, build, capture_job, version | Keeps rights, source, and replacement history attached to every delivered record. | Manifest completeness and version identity on the delivery package. |
Training and evaluation use cases
Action-conditioned video
Train or evaluate generators that must respect issued controls, not only visual style.
Playable world models
Provide the action-state traces a model needs if later rollouts are expected to stay interactive.
Long-horizon modeling
Keep identity, inventory, and task progress available beyond a few seconds of pixels.
Spatial intelligence
Use camera, pose, and occupancy so layout is a recorded fact rather than a guessed one.
Agent post-training
Supply trajectories and outcomes scoped to a defined policy interface.
Evaluation
Hold out scenarios with success criteria that can be re-run on the same environment build.
Recovery behavior
Capture failure, interruption, and recovery instead of only successful demonstrations.
Counterfactual scenarios
Replay the same initial state with a controlled action change when the environment supports reset.
These are data uses, not promised model metrics. Outcomes depend on the architecture, split, and authorization terms of each program.
Controlled scenario production
When the environment supports reset, we can produce repeatable tasks: the same initial state, a defined action interface, and success criteria evaluated on recorded state. Counterfactual variants are possible only when the engine can restore the snapshot. We do not invent scenarios the customer cannot authorize.
Delivery formats
Typical packages include frame-aligned records, action-state trajectories, camera paths, event logs, and replayable episodes, plus a schema and QA report. Formats follow the training stack: parquet or JSONL for tabular streams, versioned media for observations, and a manifest that joins them on tick.
Quality and provenance
Alignment, missing ticks, identity continuity, and license completeness are checked before release. Source and authorization identifiers travel with the dataset version. See Data Quality and Licensing & Provenance.
Passive capture versus engine-native records
| Passive capture | Engine-native record |
|---|---|
| Video only | Video aligned with action and state |
| Inputs inferred | Inputs recorded |
| State estimated | State captured from the environment |
| Events guessed | Events emitted by the engine |
| Limited replay | Repeatable scenarios |
| Unclear provenance | Source and authorization records |