mcsegment — Pipeline Algorithmic Report

Overview

flowchart TD
  LIT[("litematic")]
  RENDER[("render PNG")]

  subgraph S1["Stage 1 · export_geometry.ts"]
    s1a["for each non-air block\nb.pos, blockName, blockProps"]
    s1b["getMesh + SpecialRenderers\nCull.none() → all 6 faces incl. interior"]
    s1c["quad → 2 tri\n[0,1,2] and [0,2,3]  fixed winding"]
    s1d["pickId = x·sY·sZ + y·sZ + z + 1\nmat4.translate → world-space float32"]
    s1e["positions.bin [N,3,3] f32\nblock_ids.bin [N] i32\nmeta.json {size:[sX,sY,sZ] nTriangles}"]
    s1a-->s1b-->s1c-->s1d-->s1e
  end

  subgraph S2["Stage 2 · camera.py"]
    s2a["viewProjectionMatrix[16]\nstored column-major (WebGL)"]
    s2b["reshape(4,4, order='F')\n→ correct row-major VP [4,4]"]
    s2c["placements[0]\ntile_x  tile_y  W  H  mirror"]
    s2a-->s2b-->s2c
  end

  subgraph S3["Stage 3 · rasterize.py  ×N views"]
    s3a["VP @ [x,y,z,1]T → clip[4]\nw ≤ 0 → discard entire tri (behind camera)"]
    s3b["ndc = clip / w\npx = (ndcX+1)·W/2\npy = (1−ndcY)·H/2   ← Y-flip"]
    s3c["signed area = det of screen-space edge vecs\narea ≤ 0 → skip (backface or degenerate)"]
    s3d["AABB per tri\nix ∈ [floor(min_px), ceil(max_px)] ∩ [0,W)\niy ∈ [floor(min_py), ceil(max_py)] ∩ [0,H)"]
    s3e["per pixel in AABB:\nedge fns w0 w1 w2 — all ≥ 0 → inside"]
    s3f["depth = (w0·z0 + w1·z1 + w2·z2) / area\ndepth below zbuf[y,x] → write block_id"]
    s3g["mirror=true → id_buf[:,::-1].copy()"]
    s3h["id_buf [H,W] int32\n0 = bg   else = pick_id of frontmost block"]
    s3a-->s3b-->s3c-->s3d-->s3e-->s3f-->s3g-->s3h
  end

  subgraph S4["Stage 4 · coverage.py"]
    s4pre{"--render\nprovided?"}
    s4s["simple path\nmask_mode: L / R / redness\n→ uint8 packed_mask"]
    s4r1["redness = max(0, R_mask − max(G,B)_mask)"]
    s4r2["diff = mean |mask_rgb − render_rgb| per pixel"]
    s4r3["combined = (redness above t_r) AND (diff above t_d)"]
    s4r4["erode: MinFilter (2p+1)×(2p+1)\nremoves boundary bleed"]
    s4b["extract tile per view\npacked[ty:ty+H, tx:tx+W]"]
    s4c["bool_sel = tile.ravel() above mask_threshold"]
    s4d["per view accumulate\ntotal_visible  += bincount(ids_flat)\ntotal_selected += bincount(ids_flat[bool_sel])"]
    s4e["score = selected / visible\nbid→(x,y,z): idx=bid−1\nx=idx//(sY·sZ)  y=(idx mod sY·sZ)//sZ  z=idx mod sZ"]
    s4f["scores.json  {'x,y,z': score}"]
    s4pre-->|"no"|s4s-->s4b
    s4pre-->|"yes"|s4r1-->s4r2-->s4r3-->s4r4-->s4b
    s4b-->s4c-->s4d-->s4e-->s4f
  end

  subgraph S5["Stage 5 · remove_blocks.py"]
    s5a["assert: exactly 1 region\nassert: palette[0] = air  →  air_idx = 0"]
    s5b["decode BlockStates stretches mode\nbits = max(2, ceil(log2(palette_size)))\nIDs straddle 64-bit long boundary\nsigned i64 → unsigned via bitmask"]
    s5c["seeds = positions where score above threshold"]
    s5d{"--propagate?"}
    s5e["BFS 6-connected, match palette_id\nbbox = seed_bbox expanded ±1 block\nprevents flood into same-type walls"]
    s5f["arr[y·sX·sZ + z·sX + x] = 0  (air)"]
    s5g["encode back: stretches bit-pack\n→ signed i64 longs"]
    s5h["output .litematic"]
    s5a-->s5b-->s5c-->s5d
    s5d-->|"yes"|s5e-->s5f
    s5d-->|"no"|s5f
    s5f-->s5g-->s5h
  end

  LIT --> s1a
  LIT --> s5a
  s1e --> s3a
  s2c --> s3a
  s2c -.->|"tile coords"| s4b
  s3h --> s4d
  RENDER --> s4pre
  s4f --> s5c
        

Stage 1 · Geometry Export scripts/export_geometry.ts

Iterates every non-air block, extracts mesh triangles, and writes three binary files. The block ID encoding here must be exactly reproduced in Python.

Block ID Encoding

Matches src/Picking.ts → encodeBlockPosToPickId:

0 is background. Size \((s_X, s_Y, s_Z)\) comes from structure.getSize() = bounding box of all blocks in the litematic region.

Mesh Extraction

Two mesh sources per block, each wrapped in independent try/catch:

Cull.none() exports all 6 faces including interior ones. The rasterizer's backface cull removes non-visible interior faces naturally.

Coordinate Space

getMesh() returns geometry in 0–1 block-unit space (scaled by 1/16 from 0–16 texel space). mat4.translate(t, identity, b.pos) shifts each block to its grid position. Final positions are in world-space block coordinates (integer grid, float-precise).

Quad → Triangle Convention

Each quad is split into 2 triangles with fixed winding:

Winding determines which face direction is "front" for backface culling in Stage 3.

Output Files

Stage 2 · Camera Loading mcsegment/camera.py

VP Matrix Convention

The camera JSON stores viewProjectionMatrix as a 16-element flat array in column-major order (WebGL / gl-matrix convention). NumPy's default reshape is row-major (C order). Wrong order here silently transposes the matrix and projects all blocks to wrong pixels.

Verification: with correct reshape, objectCenter from the JSON projects to tile center ± 0.5 px.

Tile Extraction

Each view occupies a tile inside the packed image. Position comes from placements[0]:

For a 3×3 nineview at 418×418 px per tile, the packed image is 1254×1254. If tile coordinates are wrong, mask pixels don't align to rasterized block IDs → incorrect coverage.

Mirror Flag

Some views have mirror: true in the JSON. This is stored in ViewMetadata.mirror and applied in Stage 3 after rasterization — not during projection.

Stage 3 · Rasterization mcsegment/rasterize.py

Produces an int32 [H, W] image where each pixel holds the pick_id of the frontmost triangle. 0 = background.

Vertex Projection

All N×3 vertices projected in one vectorized batch:

Triangles where any vertex has \(w \le 0\) (behind camera) are discarded entirely.

NDC → Pixel Coordinate

Y-axis flip: NDC +Y is up; image +Y is down. The (1 − ndcY) term is mandatory. Wrong sign maps the top half of the structure to the bottom of the image.

Backface Culling

Signed area of the projected triangle in screen space:

If area ≤ 0 the triangle is back-facing (or degenerate) → skipped. This is what makes Cull.none() in Stage 1 harmless: exported interior faces are culled here.

Z-Buffer Rasterization

Per triangle, over its axis-aligned bounding box clamped to the image:

1. Compute edge functions \(w_0, w_1, w_2\) for each pixel center.
2. If all \(\ge 0\) → pixel is inside triangle.
3. Interpolate depth: \(\displaystyle d = \frac{w_0\,z_0 + w_1\,z_1 + w_2\,z_2}{\text{area}}\)
4. If \(d < z_{\text{buf}}[y,x]\): write block_id and update \(z_{\text{buf}}\).

Depth is NDC-z (not view-space z). Smaller value = closer to camera for a standard WebGL projection.

Mirror Application

Must be applied after rasterization, not during projection. The mask tile for a mirrored view is stored un-flipped in the packed image; the flip makes the rasterized id_buf align to it.

Output

One int32 [H, W] id_buf per view. Pixel value = pick_id of frontmost block, or 0 for background / empty space.

Block–Pixel Attribution

The rasterizer stores block_id per pixel, not triangle index. A single block exports up to 12 triangles (6 faces × 2 triangles each via Cull.none()). The Z-test resolves which triangle is frontmost at each pixel, then discards the triangle identity — only the block_id carried by that triangle is written to id_buf.

Consequence for Stage 4: bincount(ids_flat) accumulates pixel counts per block across all its visible triangles automatically. No triangle bookkeeping needed downstream.

Stage 4 · Coverage Computation mcsegment/coverage.py

Mask Preprocessing (Enhanced Mode)

When --render is provided, a 4-step pipeline replaces simple thresholding. Each step is applied to the full packed image before tile extraction.

Stage Images — 雪屋 chimney run

Concrete parameters: --mask-threshold 50 --diff-threshold 30 --erosion-px 3

Simple Mode (no --render)

Falls back to single-step threshold on mask_mode channel:

Binarization

In enhanced mode: output of step 4 is binary (0 or 255); threshold 127 correctly selects 255s.

Bincount Accumulation

For each view, two integer histograms over the id_buf:

ids_flat[bool_sel] selects only the block IDs at mask-positive pixels. bincount counts how many pixels each block contributed. Both accumulators are int64.

Score Formula

Where \(b = \text{pick_id}(x,y,z)\) and the sum is over all views. Blocks with 0 visible pixels across all views are absent from the output (not score 0).

ID → Coordinate Decode

Exact inverse of Stage 1 encoding:

Size \((s_X, s_Y, s_Z)\) loaded from geometry_meta.json must match TS structure.getSize() exactly. A mismatch silently maps IDs to wrong coordinates.

Output

scores.json — keys "x,y,z", values float in [0, 1].

Stage 5 · Block Removal mcsegment/remove_blocks.py

Score Threshold

Blocks with score > threshold become seeds for removal. With enhanced mask preprocessing (redness + diff + erosion), all score > 0 blocks are already high-confidence — bleed is eliminated upstream. Use 0.5 with enhanced mode; use 0.3–0.5 as safety net with simple mode to exclude bleed noise (score 0.003–0.3).

Coordinate Normalization

Litematic regions have independent Position offsets. Must replicate the TS normalization from Schematics.ts:

Scores use world coords (from Stage 4). Litematic stores region-local coords. Mismatched coordinate system → remove wrong blocks or remove nothing.

Litematic Index Order (YZX)

Block index inside region's BlockStates long array:

This is YZX order (Y outermost, X innermost). Confirmed by src/Schematics.ts. Wrong order reads/writes wrong block positions.

Bit-Packing: Stretches Mode

Litematic uses "stretches" packing — block IDs are packed densely across 64-bit longs, with IDs allowed to straddle long boundaries:

Minimum 2 bits regardless of palette size. The signed→unsigned conversion (& 0xFFFF…) is required because Python's nbtlib yields signed int64 longs, but bit ops need unsigned semantics.

Preconditions (enforced with ValueError)

Two invariants are asserted at load time:

Optional Propagation (--propagate)

BFS from seed blocks to connected same-type neighbors, constrained to seed bounding box + 1 margin:

Bbox constraint prevents same-type flood into house walls. Useful when rasterizer only sees the outer shell of a hollow structure (e.g., chimney bricks).

Risk: if chimney block type (bricks) also appears in house walls within the bbox, those wall blocks are also removed. Use high score threshold (0.5) to keep seeds tightly in the chimney column.

Concrete Example · 雪屋 (Snow House)

Full chimney-removal run on a real Minecraft snow house. 9-view pack, 418×418 px per tile, 1254×1254 packed image.

Camera JSON Render & Mask Litematic

View 0 of 9 from images/雪屋.json. Column-major flat array → reshape as order='F':

view 0 — azimuth 0°, elevation 20°, no mirror

{ "viewIndex": 0, "azimuthDegrees": 0, // camera angle around Y axis "elevationDegrees": 20, // camera tilt above horizontal "mirror": false, // no horizontal flip for this view "viewProjectionMatrix": [ // 16 floats, COLUMN-MAJOR (WebGL gl-matrix) 3.732050895690918, 0, 0, 0, 0, 3.5069806575775146, -1.1543179750442505, -0.3420201241970062, 0, -1.2764365673065186, -3.1714625358581543, -0.9396926164627075, -44.784610748291016, -19.430234909057617, 83.09712982177734, 106.40138244628906 ], "placements": [{ "x": 0, // tile left offset in packed image (px) "y": 0, // tile top offset in packed image (px) "width": 418, // tile width "height": 418 // tile height }] }

Python reshape:

vp = np.array(view['viewProjectionMatrix']).reshape(4, 4, order='F') # vp[0] = [3.732, 0, 0, -44.78] ← column 0 of WebGL matrix # vp[1] = [0, 3.507, -1.276, -19.43] # vp[2] = [0, -1.154, -3.171, 83.10] # vp[3] = [0, -0.342, -0.940, 106.40]

Tile extraction for view 0: mask_tile = packed_mask[0:418, 0:418]. After enhanced preprocessing (redness + diff + erosion), packed_mask is already binary (0/255). bool_sel selects the 255-pixels.

Before and after chimney removal. Command used:

# Step 1: export geometry (once per litematic) yarn vite-node scripts/export_geometry.ts \ --input test/fixtures/litematics/雪屋.litematic \ --output /tmp/雪屋_geo # Step 2: compute per-block coverage scores (enhanced mode) mcsegment coverage \ --geometry /tmp/雪屋_geo/雪屋_geometry_meta.json \ --camera images/雪屋.json \ --mask imgs/雪屋_mask.png \ --render imgs/雪屋_render.png \ --mask-mode redness \ --mask-threshold 50 \ --diff-threshold 30 \ --erosion-px 3 \ --output /tmp/雪屋_scores.json # → 1557 block scores written # → score>0: 71 blocks (was 1104 without enhanced mode) # Step 3: remove blocks (score > 0.5, BFS-fill same-type interior) mcsegment remove-blocks \ --litematic test/fixtures/litematics/雪屋.litematic \ --scores /tmp/雪屋_scores.json \ --output /tmp/雪屋_no_chimney.litematic \ --threshold 0.5 \ --propagate # → 52 seed blocks (score > 0.5, all brick types, 0 non-brick) # → 99 total removed (52 seeds + 47 interior bricks via BFS propagation)

Before — 雪屋 with chimney

After — chimney removed (99 blocks)

Correctness Parameters

Every parameter in this table affects which blocks end up in scores.json or get deleted. A wrong value silently changes the output — no runtime error.

Known Pitfalls

Parameter Quick Reference

Recommended Defaults