Playable Game Datasets for Training and Evaluating Coding Agents

A playable game dataset for coding agents is a bundle of prompt, specification, source, build, runtime, and judged gameplay — including win, lose, and restart — such that an agent’s patch can be scored for more than “it compiled.” Compilation success is a necessary gate. It is not playability. Playability means a human or a scripted player can enter the loop, take legal actions, hit documented outcomes, and recover from failure without undefined behavior.

That definition is stricter than a GitHub repo with a README. It is also the only definition that stays honest when agents start generating games instead of sorting lists. Software-engineering benchmarks such as SWE-bench showed that real repositories need fail-to-pass tests, not star counts (Jimenez et al., 2024). Games add a second oracle: the loop must run.

What the dataset must contain

Prompt and specification

The prompt is not the spec. The prompt is the agent-facing instruction, with the same information constraints you will use at eval time. The spec is the grader-facing contract: input bindings, camera, win conditions, forbidden libraries, and performance budgets if any.

Failure mode: a prompt that says “make a platformer” and a spec that secretly requires coyote time. The agent did not fail the game. You failed the dataset.

Keep a hidden test spec if you must, but version it. Silent spec edits after release contaminate leaderboards the same way they contaminate SWE-bench-style sets.

Code, build, and runtime

Pin language, engine, and dependency lockfiles. Provide a headless or CI build. If the engine needs a license server, the evaluation harness — not the model card — has to say so.

Runtime is a separate artifact. A successful dotnet build that never opens a window is still a coding result. For a playable dataset, launch must reach a known spawn. Record the command, display size, and whether audio is required. Many CI machines have no audio device; if the game deadlocks waiting for a device, your eval is a hardware test.

Gameplay, win, lose, restart

Define the legal action interface: keyboard, virtual pad, or API. Define at least one winning trajectory that a reference player can complete, one losing trajectory, and a restart that restores a documented state rather than leaking health across runs.

episode:
  spawn: dock_start
  actions: [Move, Jump, Interact]
  win: flag_reached == true
  lose: player_hp <= 0
  restart: reload_scene(spawn)
  timeout_s: 180

If restart is “press R” but R does not clear a global singleton, agents will harvest points from leaked state. That is not intelligence. It is a dirty eval.

Compilation is not playability

Automated checks should always run first. They catch missing scenes, broken scenes, and leaked state. Human play catches the rest: unreadable collision, impossible jumps, UI that covers the only button. Neither replaces the other. Author-only play is the most expensive lie, because the author navigates by memory.

VibeWorlding-style construction agents show a parallel lesson in 3D: layout success is not the same as a living world (VibeWorlding). Playable game evals should not copy that mistake in engine form.

How to score without inventing a leaderboard

Publish the harness. Freeze seeds. Separate public smoke tests from hidden tests if contamination is a concern, and say so. Report compile rate, launch rate, smoke pass rate, and human play pass rate as four numbers. A single “success %” will hide a dataset that never launches.

Do not claim the set measures general game engineering. It measures the spec you wrote. That humility is part of training-ready game data.

Production and QA

Production means the reference implementation, broken variants, and the grader ship together. QA means a second machine, a second person, and a replay hash. If physics cannot be seeded, stop pretending the replay is a unit test; switch to invariant checks (score monotonicity, no NaNs, restart clears HP).

Rights still apply. A coding-agent eval that embeds a marketplace character without a training-and-eval license is not an eval you can share. Pair this with licensed game data.

A reference harness, not a mystery grader

Publish, at minimum:

• the exact commands for build, launch, smoke;
• the mapping from process exit codes to scores;
• the timeout;
• the display size;
• whether GPU is required;
• the seed policy.

If the harness is private, say so and freeze it. Moving from “compile in CI” to “play in a VM with audio” without versioning the harness is how two labs report incomparable numbers.

Task types that should not share a score

Keep these as separate slices. A single “game coding accuracy” number will be dominated by compile rate.

Human play protocol

Give reviewers a script, not a vibe:

1. Fresh install from the dataset archive.
2. Follow the prompt the agent saw — no extra hints.
3. Attempt the documented win and the documented lose.
4. Restart twice.
5. File a structured bug (cannot launch / cannot complete / undefined behavior / spec mismatch).

Authors of the reference game must not be the only reviewers. They will jump gaps they forgot to specify.