From generated code to functioning worlds
A coding agent can emit HTML that looks like a scene. Evaluation needs a world that launches, exposes state, accepts interaction, and can be reset. We package browser-native environments so specification, source, assets, contracts, and traces are one inspectable object — not a screenshot of a demo.
What an executable world contains
Specification
A natural-language task, constraints, and success criteria that a coding or planning agent can be asked to implement.
Source
Browser-native world code, typically Three.js, with a documented runtime interface rather than a sealed binary.
Licensed virtual assets
Referenced geometry and materials with per-asset provenance. Rights are scoped to the program; redistribution is never assumed.
Runtime-state interface
A readable world state surface — for example a documented window interface — so verification does not scrape pixels.
Behavioral contracts
Checkable claims: object exists, goal is reachable, interaction mutates the declared field, reset restores the initial snapshot.
Interaction traces
Recorded actions, state samples, and events so a run can be replayed and compared against the contract.
Runtime-state interfaces
Pixel-only grading collapses the moment a camera angle changes. A documented runtime interface — object poses, flags, inventories, and task fields — lets verification read the world the way an agent would. The exact surface is named in the schema for that pack; we do not assume a single global API across customers.
Behavioral contracts
Contracts are checkable claims, not slogans: the goal exists, an interaction mutates a declared field, a blocked path fails, reset restores the initial snapshot. Automated checks catch schema drift and broken interaction chains. They do not replace human playability review.
Asset and source validation
Referenced GLB packages and source files are listed in manifests. Loading is checked in the target runtime. Licensed virtual assets keep per-asset provenance. Training rights and redistribution rights are recorded separately; a file that loads is not thereby cleared for public re-release.
Interaction traces and replay
Traces store actions, sampled state, and events so a run can be replayed against the same build. Replay is evidence, not a claim of bitwise determinism on every browser. Where reset is supported, we measure whether the snapshot returns.
Automated verification and human playability review
Automation covers launch, schema, contracts, and basic interaction. A reviewer still plays the world: camera, controls, stuck states, and whether the specification is honestly implemented. See Data Quality for the gate sequence.
Delivery structure
The tree below is an illustrative layout for an evaluation world pack.
WORLD-001/ ├── specification.md ├── source/ ├── assets/ ├── state-schema.json ├── behavioral-contract.json ├── interaction-traces/ ├── qa-report.json ├── asset-manifest.json └── license-manifest.json
Illustrative delivery layout. Actual packages are adapted to the runtime, authorization terms, and evaluation protocol of each program.
Verification loop
- 01 Specification
- 02 Build
- 03 Launch
- 04 Interact
- 05 Read state
- 06 Verify behavior
- 07 Reset
- 08 Replay
- 09 Record evidence
Who this is for
- Coding Agent labs
- text-to-game teams
- multimodal agents
- browser agents
- game-generation models
- world-generation teams
- benchmark teams
- evaluation teams