WorldCoder-Bench Explained: Why Training-Ready Three.js Worlds Matter for AI Agents

Executive summary

WorldCoder-Bench is a 2026 benchmark for whether a coding model can synthesize a physically grounded, executable Three.js world from a natural-language task — not merely a scene that looks plausible. The paper reports 2,026 expert-curated canonical tasks across Simulation, Rendering, and Application, with optional .glb assets and hidden behavioral contracts. StateProbe runs the generated HTML in a sandboxed browser, reads a runtime interface such as window.__3D_STATE__, and checks mutation-hardened contracts. The authors report that the strongest evaluated system reaches 27.8% verification coverage on WorldCoder-Core and 19.9% on WorldCoder-Robust, with failures dominated by state-schema drift and broken interaction chains.

The public GitHub snapshot we checked on 2026-08-24 is not the same object as the paper’s 2,026-task claim. The repository README states a 2,004 unique-task snapshot (core_205 + hf_snapshot_1799) and explicitly says those directory names are not a new claim about the canonical 2,026-task split. This note keeps those numbers separate.

Our interpretation is that WorldCoder-Bench is a specification for training-ready Three.js data, not only a leaderboard. A rendered scene is not yet a functioning world. Training-ready Three.js data must connect visible behavior, hidden state, interactions, assets, and executable validation. That sentence is GaussMathematics analysis, not a quotation from the authors.

Why WorldCoder-Bench matters now

Public benchmarks indicate that coding agents are being asked to ship interactive 3D, while evaluators still often score pixels, DOM nodes, or static tests. The research direction helps explain a production failure we already see in game-data programs: a world can compile, render, and still be unusable as agent training or evaluation data because the hidden state never matched the visible controls.

We should not assume that every buyer is responding to the same paper. WorldCoder-Bench does not prove a market causal chain. It does give a concrete, inspectable contract for browser-native 3D generation: Three.js page, optional GLB, runtime schema, and a probe that can fail the sample.

Related GaussMathematics notes on playable game datasets for coding agents and training-ready game datasets make the same compile-versus-play split in engine form. WorldCoder-Bench makes the split measurable inside a WebGL canvas.

From web generation to executable 3D worlds

The paper reports a shift from static web generation to worlds that must integrate assets, obey spatial and physical constraints, and keep user-facing controls synchronized with hidden runtime state. Three.js worlds render into a canvas. Screenshots and DOM walkers see buttons and pixels. Collision, energy, animation phase, and interaction flags live in JavaScript.

The authors evaluate that mismatch: they report DOM-based scoring as essentially uncorrelated with hidden state-level correctness on their paired comparison, and they report that an expensive source-probing agent still grants passing marks to a large share of severely defective outputs. We do not treat those figures as a general law of all 3D tools. They are a warning about this evaluation surface.

Why the benchmark uses Three.js

The authors describe browser-native 3D, commonly built with Three.js, as a natural substrate because generated programs must run as a page: load, render, and respond. The paper does not claim Three.js is a universal standard for all world models. From a production-data perspective, Three.js is one executable target among others (Godot in GameCraft-Bench, tutorial-derived engine edits in GameDevBench, web games in OpenGame). The transferable lesson is the oracle, not the library: if the product is an interactive world, evaluation must read runtime state.

What the 2,026 tasks contain

The paper reports 2,026 finalized canonical tasks, organized into three macro-categories by user intent:

• Simulation — observing system evolution and dynamics.
• Rendering — controlling visual presentation.
• Application — completing an interactive goal.

The authors further describe 15 fine-grained domains, difficulty levels, and asset tiers (primitive-only, single-.glb, multi-asset). Splits reported in the paper include:

27.8% in the paper is Verification Coverage (V-Cov) for the strongest reported system on WorldCoder-Core, not on all 2,026 tasks. 19.9% is V-Cov for that system on WorldCoder-Robust. Mixing those percentages with the 2,026 count would be a reporting error.

The GitHub README we verified describes a rebuttal/research snapshot: 205 + 1,799 = 2,004 unique tasks, plus a note that three metadata-declared asset paths were missing from the source archives. Use the paper for the 2,026 claim. Use the repository for what you can actually download today.

Assets, state, and behavioral contracts

Each task is a directory: a natural-language brief (task.json) and optional assets/ with pre-authored 3D resources such as .glb. The model must emit a self-contained HTML file. Hidden from the model are the Scene Interaction Graph (SIG), action sequence, thresholds, and assertions. In the public snapshot, that hidden bundle is stored as icg.json for archive compatibility; the README says “ICG” is not a separate benchmark term.

The paper is licensed CC BY 4.0 on arXiv (verified 2026-08-24). The GitHub snapshot carries a provisional research-evaluation notice, not a blanket CC BY grant, and it warns that GLB files in data/assets/shared should not be assumed redistributable. Those are different legal objects. This article analyzes the paper; it does not republish the snapshot assets.

How StateProbe evaluates hidden runtime behavior

StateProbe, as the authors describe it, runs generated HTML in headless Chromium (Playwright in the public code), with a local version-locked Three.js archive. It first checks executability: page load, live WebGL, render loop, no early JavaScript exceptions. Failure is Runtime_Crash.

For executable programs it applies deterministic probes: snapshot, act, snapshot, check hidden assertions over the delta. Outcomes include Runtime_Crash, Check_Fail, and Check_Pass. Coverage metrics include Affordance, State, Transition, and Verification coverage. Contracts are admitted only after mutation testing (deleted updates, scaled constants, swapped event targets).

The SIG models interaction paths. Combined with window.__3D_STATE__, this is how the benchmark pierces the canvas. Our interpretation: any production Three.js dataset that wants to train or evaluate agents should expose an equivalent, versioned state interface even if it never uses StateProbe’s exact scripts.

What the reported results show

The paper reports nine frontier models; the best V-Cov on WorldCoder-Core is 27.8%, with no system above 30% on hidden contracts despite programs that load and visually resemble the target. On WorldCoder-Robust, the same leading system is reported at 19.9% V-Cov. The authors describe failures as dominated by state-schema drift and broken interaction chains rather than missing scene elements.

They also report utility metrics (Return on Automation, Time Efficiency Multiplier) arguing that cheaper or faster models can still provide quality-adjusted value on easier domains. We do not convert those labor-rate assumptions into a GaussMathematics pricing claim.

Why successful rendering is not enough

This table is ours. It operationalizes the paper’s failure modes for a data-QA checklist.

What a training-ready Three.js world should contain

From a production-data perspective, a training-ready record is closer to WorldCoder’s task directory than to a zip of pretty GLBs:

{
  "task_id": "harbor-slicer-004",
  "spec": "nl_brief.md",
  "entry": "index.html",
  "assets": ["mug.glb"],
  "state_schema": ["positions", "velocities", "mode", "flags"],
  "runtime_interface": "window.__3D_STATE__",
  "reset": "reload_spawn",
  "rights": { "code": "customer", "glb": "licensed" }
}

A practical implementation would still need license manifests, holdout tasks whose contracts never leak into training prompts, and human play on a sample. Benchmarks hide contracts for leaderboard integrity. Production datasets must version both the visible spec and the private oracle.

Implications for game-data production

The research direction helps explain why “Three.js dataset” is an incomplete purchase order. You may be buying:

1. GLB catalogs (identity, materials, units);
2. executable pages (code + assets + loop);
3. agent evaluation tasks (spec + hidden contract + traces).

Those are different SKUs. Pixel-generative world models need a different stack again — see playable video models versus executable Three.js worlds.

Public benchmarks indicate that surface evaluators can invert rankings. A commercial-scale data program would additionally need engine-side or runtime probes, not only review stills.

GaussMathematics perspective

The paper reports a Three.js world-synthesis benchmark, a 2,026-task canonical scale, StateProbe, and low V-Cov even for strong models. The repository reports a 2,004-task inspectable snapshot. Those are the authors’ and maintainers’ facts.

Our interpretation is that the missing product in many “AI 3D” programs is the join between GLB, code, state, and a failing test. We do not claim WorldCoder-Bench caused companies to purchase Three.js assets. We do not claim all world-model teams now need Three.js.

From a production-data perspective, customer-authorized game environments and licensed virtual assets can emit WorldCoder-like fields without copying the benchmark: a spec, an executable, a state schema, interaction traces, and rights. Playable Three.js environments are one delivery shape among others.

A commercial-scale data program would additionally need provenance for every GLB, a reset contract, and an eval split that never sees the hidden assertions during training.

Limitations

• We did not re-run StateProbe or reproduce the nine-model table.
• 2,026 versus 2,004 must not be collapsed; the snapshot also notes missing declared assets.
• Paper CC BY 4.0 does not automatically license repository GLBs.
• Mutation-hardened contracts are only as good as the injected defect battery.
• Human time and $60/hour assumptions in RoA/TEM are the authors’ construction, not our audit.
• Browser-native Three.js is not a substitute for Godot/Unreal engine-grounded benches.

References

• Lu, Xu, Yu, Jiang, Yu, Wang, Yang, Zhang, Wang, He, and Liang, *WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesis*, arXiv:2606.01869, submitted 1 June 2026, updated 8 June 2026. https://arxiv.org/abs/2606.01869. Licensed CC BY 4.0. Changes: summarized and independently analyzed by GaussMathematics.
• Code and benchmark materials: https://github.com/shuolucs/WorldCoder-Bench
• Related coding-agent game benches discussed elsewhere: GameCraft-Bench (arXiv:2606.17861), GameDevBench (arXiv:2602.11103), OpenGame (arXiv:2604.18394)