Voxel-to-Text: Combining LLM Geometry with Diffusion Detail
An ongoing experiment log on turning Minecraft voxels into structured text so that a language model can reason about 3D shape — the first step toward a hybrid that gets sharp geometry from LLMs and fine detail from diffusion.
Contents
1. Background
Current 3D generation is dominated by diffusion-based pipelines. They are excellent at modeling fine surface detail and material appearance, but they frequently produce inaccurate geometry and artifacts — disconnected surfaces, floating fragments, topology errors. Language models, on the other hand, can generate sharp, well-articulated 3D geometry (boxes, loops, symmetry, joinery) because they reason about structure the way they reason about code. The geometry is clean, but it lacks the surface richness that diffusion provides.
The research question: can a hybrid system get sharp geometry from an LLM and fine detail from diffusion, in the same pipeline?
Both capabilities exist, but they speak different languages. Closing that gap requires a representation that both sides can consume. This experiment log tracks the attempt to build that bridge starting from the voxel side.
2. How an LLM Can Understand Mesh
Any pipeline that uses language-model reasoning about 3D must pass some representation through the model. Two approaches exist:
Convert mesh to text
Describe geometry as code, coordinates, or structured notation. No new tokens needed. Works with any frozen LLM.
Add 3D tokens to LLM
Encode geometry into a new embedding space and expand the model's vocabulary. Requires re-training or fine-tuning.
Chosen: text
Text (code) is human language's own compressed embedding — dense, universal, and format-stable. Selected for its information density and zero retraining cost.
Text code is human-engineered to be maximally dense: a single function call can describe a wall, a loop can describe a tower, a noise call can describe terrain. It carries similar information density to a learned embedding, but with the additional property that the format never changes and is universally understood — by the model, by a human reviewer, and by any executor that runs it. Learned embeddings, by contrast, are opaque to the model's pre-trained knowledge and require paired training data to remain meaningful.
3. Voxel-to-Text Compression
The full mesh-to-text problem is hard. A simpler, well-scoped version of the same question is: how do you compress a 3D voxel grid to text? Voxels are already discrete (a natural fit for code), their occupancy is binary or block-typed, and the output can be verified by re-running the code and comparing voxels.
The objective takes the MDL (Minimum Description Length) form:
\[ P^{*} \;=\; \arg\min_{P}\; \underbrace{\lVert V - \mathrm{Exec}(P)\rVert}_{\text{reconstruction loss}} \;+\; \lambda \cdot \underbrace{\lvert P \rvert}_{\text{text length}} \]Pushing \(\lambda\) high forces the agent away from block-by-block copying toward abstract rules (loops, noise, symmetry). Pushing it too low permits copying and learns nothing. The goal is a code short enough to generalize yet faithful enough to rebuild the structure.
4. The Seed That Cannot Be Guessed
Before pursuing the voxel-to-text approach, a critical failure mode needed examination: the "Seed That Cannot Be Guessed" problem (full analysis here).
The core issue: many 3D structures are built with procedural noise whose seed is hidden. The shortest program to rebuild such a structure is a single noise call — except it requires the exact seed used at build time, which is unknowable. Exact reconstruction demands storing the seed, which is just copying in disguise.
\[ V = G(s,\, z), \qquad s = \text{seed (hidden)}, \quad z = \text{style settings} \]The shortest code cannot rebuild a noisy shape exactly unless the seed is already known. This is the same barrier as GAN inversion: many latents map to one output, so recovering the input from a single output is undefined.
The experiment concludes that this problem is inconclusive as a hard blocker — it is a trap for exact reconstruction, not for approximate structural understanding. Whether the overall approach succeeds depends on the balance between reconstruction loss and text-length penalty:
| Regime | Reconstruction loss | Text length | What happens |
|---|---|---|---|
| Too loose on \(\lambda\) | near zero | huge | copies blocks verbatim, no structure learned |
| Too tight on \(\lambda\) | high | tiny | invents fake patterns, underfits |
| \(\lambda\) off entirely | zero | ∞ allowed | everything becomes "unknown", no learning |
| Balanced | small | compact | loops, noise families, structural rules emerge |
The resolution is to target a distributional objective — learn the space of valid programs over many builds, not the single best program for one build. The seed becomes a range, not a fixed number. This mirrors exactly what diffusion does at the pixel level: instead of recovering a sample, it learns the shape of the probability field.
5. Part-by-Part Reconstruction
A flat "compress the entire voxel grid to one program" task is extremely difficult for several reasons:
- Context length: a 64×64×64 chunk exceeds any model's working memory without external tools.
- RL reward sparsity: a single reward on the whole structure provides no useful gradient signal early in training.
- Code structure: the best human-written code is modular. A building is a roof, a wall, a floor — not a monolith.
If the best code is structural and RL needs laddering, part-by-part reconstruction is the natural entry point.
The proposal: decompose the voxel grid into semantic parts, ask the LLM to write a reconstruction program for each part independently, then assemble the parts. This reduces each sub-task to a size the model can handle, provides dense intermediate rewards, and mirrors how humans describe construction ("first the foundation, then the walls, then the roof").
flowchart LR
A["3D voxel grid\n(full structure)"] --> B["Semantic\ndecomposition"]
B --> C1["Part A\n(e.g. roof)"]
B --> C2["Part B\n(e.g. walls)"]
B --> C3["Part C\n(e.g. floor)"]
C1 --> D1["LLM writes\nprogram A"]
C2 --> D2["LLM writes\nprogram B"]
C3 --> D3["LLM writes\nprogram C"]
D1 --> E["Assemble\n+ verify"]
D2 --> E
D3 --> E
E --> F["Reconstructed\nvoxel grid"]
This decomposition also provides natural curriculum laddering for RL: simple geometric parts (flat floor, rectangular wall) provide easy early wins, while complex organic parts (curved roofs, irregular terrain) can be deferred to later training stages.
6. Part Dataset Creation
To train a part-aware reconstruction model, a dataset of labeled semantic parts is needed. The first step is generating semantic masks from rendered 2D images of the 3D structures.
The key observation: image generation models (specifically image-to-image with GPT-4o / image-2) can produce high-quality semantic segmentation masks directly from rendered Minecraft screenshots. This is promising because it bypasses the need for manual annotation — the same model that will eventually reason about parts can also generate the training signal for what a "part" looks like.
Below: the original rendered view of a snow house structure vs. the semantic mask produced by image-2. Each color region corresponds to a distinct semantic part (roof, wall segment, door, snow layer, etc.).
Slide to compare: original render (left) vs. image-2 semantic mask (right). Drag to reveal.
The workflow is: render the 3D structure from multiple views → pass each rendered image to image-2 for semantic segmentation → collect consistent part labels across views. Because Minecraft structures have clear block boundaries, the masks tend to align well with actual voxel boundaries, making the 3D back-projection step tractable.
Note that ChatGPT will crop any transparent region before feeding into image editing. So make sure to color transparent regions for size matching. And for the background, we find a grey color that is the furthest away from the existing pallette in the image.
7. Inverse Ray Tracing to 3D
Having a 2D semantic mask is not enough — each colored pixel needs to be mapped back to the specific 3D voxel it corresponds to. This is done by inverse ray tracing: for each pixel \((u, v)\) in the rendered image, cast a ray from the camera through that pixel into the voxel grid, and record the first voxel hit. Pixels sharing the same mask color map to the same semantic part in 3D space.
The renderer already stores the camera matrices used to produce each render, so the ray direction for any pixel can be computed analytically:
\[ \mathbf{r}(t) \;=\; \mathbf{o} + t\,\mathbf{d}, \quad \mathbf{d} \;=\; K^{-1}\begin{pmatrix} u \\ v \\ 1 \end{pmatrix}, \quad t > 0 \]where \(K\) is the camera intrinsic matrix and \(\mathbf{o}\) is the camera origin in world coordinates. The first voxel whose bounding box the ray intersects is the block owning that pixel.
The interactive viewer below has the snow house structure pre-loaded. Hover over the rendered image to see the ray trace highlight the corresponding block in the 3D view. This is the exact mechanism used to map 2D mask pixels back to 3D voxel coordinates.
3D Schematic
Loadsnowhouse.litematic — auto-loaded
Input Image
Loadsnowhouse-render.png — hover to ray trace
Camera JSON
Loadsnowhouse-render.json — auto-loaded
Hover over the image to trace a ray into the 3D view and highlight the corresponding block. The JSON stores per-view camera matrices; the renderer uses them to compute the exact ray for each pixel.
8. Next Steps
The current status: the 2D-to-3D part mapping pipeline is working. The next step is to produce a large-scale part dataset — rendered images, semantic masks, and corresponding 3D voxel part labels — across many different Minecraft structures. Once the dataset exists, the LLM part-reconstruction training loop can begin.