How to Build Diverse Virtual Asset Datasets Without Near-Duplicates

A diverse virtual asset dataset is a collection whose structural coverage matches the task, while cosmetic variants are labeled as variants, exact duplicates are removed, and near-duplicates are either grouped or excluded from evaluation. Diversity is not “more files.” A marketplace scrape can be huge and still be one sofa in thirty fabrics. Training will memorize the sofa and the eval will leak through the fabrics.

The first 150 words are the rule: decide what counts as a different object, measure similarity in geometry and texture, impose a taxonomy, and freeze train/test so two near-copies cannot sit on both sides of the split.

Structural diversity versus cosmetic variation

If the task is material generation, cosmetic variation is signal. If the task is layout or retrieval, it is noise unless grouped. VibeWorlding-style agents that retrieve by category will look better than they are if the catalog is 40 near-identical trees (VibeWorlding).

Exact duplicates and near-duplicates

Exact: identical hashes after a canonical export (applied transforms, stripped author metadata). Near: high geometric similarity (Hausdorff on simplified cages, occupancy IoU, Chamfer on samples) or high texture similarity (perceptual hashes, CLIP on turntable views) with the same topology.

Do not use a single threshold for all categories. Wheels are supposed to look alike. Sculpted creatures are not.

Failure mode: deduping only on filename. chair_lod0 and chair_high survive, eval dies.

Geometry similarity and texture similarity

Run geometry first, then texture. Two crates with swapped labels should match on mesh. Two unrelated objects with a shared wood albedo should not.

Store the features. Diversity reports that cannot be regenerated are decorations.

{
  "asset_id": "furn.chair_12",
  "cluster_id": "chair.windsor.v3",
  "role": "canonical",
  "geom_nn": [{"id": "furn.chair_18", "chamfer": 0.004}],
  "view_nn": [{"id": "furn.chair_19", "clip": 0.92}]
}

role: canonical | variant | exclude_eval is the production switch. Variants may train. They must not quietly occupy the test split.

Taxonomy and distribution

A taxonomy lets you see that 60% of “furniture” is chairs. Balance at the level the model is judged: if the customer cares about industrial props, a beautiful furniture long tail is the wrong diversity.

Publish histograms: category, triangle count, bbox volume, material type. If bbox volumes collapse because everything was unit-normalized, you have another problem — see 3D asset training readiness.

Train/test contamination

Contamination in 3D is sneakier than in images. A test chair can be a rotated train chair, a LOD, or a kitbash that still shares 80% of the hull. Split by cluster_id, not by asset_id. If a cluster is in train, all variants are in train.

For environment-level tasks, also split by scene kit. Two rooms dressed from the same 12 unique meshes are not independent worlds. That is the bridge to synthetic environments versus static collections.

Computer-vision work on near-duplicate contamination in image benchmarks is the same pathology in a simpler medium; the lesson travels even when the papers are about photos rather than meshes. Treat cluster-aware splits as a default, not as an advanced extra.

QA

1. Hash-level exact dedup.
2. Cluster review for the fat categories.
3. Held-out human “same object?” sample.
4. Retrieval test: query a train canonical, fail if a test variant returns at rank 1.
5. Rights: variants inherit the parent license; do not assume a recolor is a new work you own.

Do not claim a set is “fully diverse.” Claim the taxonomy, the cluster method, and the known remaining collisions.

How clusters should be built

A workable production recipe:

1. Canonicalize exports (apply transforms, freeze scale).
2. Simplify to a cage with a fixed triangle budget.
3. Sample points, compute Chamfer both ways.
4. Render three orthogonal views with a clay material; embed with a frozen image encoder.
5. Union-find with category-specific thresholds.
6. Human-review the largest clusters and all cross-category merges.

Do not cluster across categories automatically. A barrel and a drum may be geometrically close and semantically different. Conversely, a folded and unfolded cloth may be semantically one asset and geometrically far — those belong in an articulation group, not a near-dup cluster.

Distribution reports worth shipping

Buyers should receive:

• count per taxonomy node;
• unique clusters per node;
• variant multiplier (assets / clusters);
• bbox volume quantiles;
• triangle-count quantiles;
• list of clusters larger than N.

If variants/clusters is 12 in “seating” and 1.1 in “tools,” you do not have a diverse furniture-and-tools set. You have a chair factory with a tool drawer.

Contamination that survives hashes

Kitbashing is the usual leak: a test radio built from a train speaker plus a train antenna. Mesh hashes differ; retrieval still cheats. Mitigations: part-level IDs when you have them, and a rule that any test asset whose parts overlap a train cluster is reassigned. If you cannot afford part graphs, at least forbid shared source files across the split.