3D Point-Cloud Segmentation Harness — Implementation Plan

2026-08-12 · Miro · designed by Sol, critiqued by Opus, reviewed & finalized by Fable

Summary

Build the 3D analog of the 2D line-bitmap training harness in the new repo 3dai.iolabs.pointcloud.mlsegmentation: semantic segmentation of roadside assets (guardrails, signs/gantries, trees, walls, fences, poles, terrain, structures) on dense highway MLS LAS corridors. Lane markings stay out of scope.

Every model/training method is one experiment config (E1–E14 from the research plan). Deliverable one makes E2 (CPU voxel-size oracle) and E1 (Superpoint Transformer pilot scaffold, oracle-gated) runnable; E3–E14 ship as schema-valid templates or gated configs that fail closed with a named unmet gate.

SSOT: shared ML code lives in the existing iolabs-ml-harness package (bumped to 0.2.0, image deps optionalized); pre-annotations come from the classical seg3d fusion package plus the ReCap annotation round-trip. Full spec: docs/plans/2026-08-12-harness-implementation-spec.md (in-repo, ~1,700 lines).

Key decisions

Mirror the 2D repo's conventions exactly: one YAML per experiment under configs/, strict schema (unknown keys are errors), TensorBoard under runs/, DVC data lanes 00_external → 04_predictions, Azure ML command jobs on the shared T4 workspace, uv with an ml extra.
mlharness 0.2.0 is additive: typed FactoryRegistry, rank-agnostic SegmentationStats (ignore-aware, point tensors OK), masked CE/focal/focal-Tversky losses, shared provenance.py; OpenCV/SMP move to an [image] extra with lazy compat facades. The 2D consumer must pass its tests unchanged in a clean environment before publish — proven by a dedicated gate.
External frameworks are checkouts, not dependencies: SPT and Pointcept run in their own environments via pinned-commit runner scripts; this repo ships only thin overlay bridges (frameworks/spt, frameworks/pointcept). No spconv, no FlashAttention on T4, published NC checkpoints are deny-by-default policy.
One adapter, two emitters, exact identity: CorridorAdapter streams LAS/COPC bounded, joins pre-annotation labels by exact 1 mm keys, tiles deterministically along the corridor axis, and emits canonical NPZ + SPT raw + Pointcept scenes. Every point carries a stable origin_index; prediction remapping is index-based (never nearest-neighbor) with overlap-logit blending.
Ontology is frozen at 9+1: eight interest classes + trained other_background, contiguous 0..8; 9 = void (never predicted). Deciding macros use the split-frozen supported interest subset so structurally-empty classes cannot dilute comparisons.
Locked promotion-test corridors are unreachable without an explicit --promotion --promotion-authorization approval file; a YAML flag alone grants nothing. Promotion needs non-inferiority CIs plus a predeclared superiority condition.
TensorBoard carries everything: native framework training curves plus an eval/ event stream from the evaluator — per-class IoU/P/R/F1, macros, confusion matrix, FP/km, continuity, seam-stratified metrics, and rendered 3D pred-vs-gold views (worst-tiles contact sheets). JSON reports are never the only surface of a metric.
E2 runs on full-resolution labels only: the oracle regenerates fused labels in-memory via the upstream package (written NPZ artifacts are ~6.67 cm representatives and are rejected as oracle input) and decides 2/3/5 cm voxel size from purity, mixed-voxel rate, and thin-class survival.

Phases

WP1 — mlharness 0.2.0 parallel

Branch feat/3d-generalization in the mlharness repo. Optionalize image deps behind [image], add registry.py, generalized metrics, masked losses, provenance.py; keep the image Lightning loop untouched. Acceptance: core suite green without OpenCV/SMP, full suite green with [image], and the 2D shared-harness worktree passes its existing tests + fast-dev-run against the built wheel in a freshly created environment.

WP2 — 3D repo scaffold parallel

pyproject.toml (core = NumPy/SciPy/laspy/DVC + upstream seg3d package; ml extra = torch stack + mlharness), DVC config and data lanes, Azure scripts (env.sh, submit/run/sync/promote/tb), conda environment template, README, CLAUDE.md, dev script overlaying the sibling mlharness checkout. Interim editable path sources with TODO markers until 0.2.0 is published to the private index.

WP3 — Phase-0 contracts parallel, freezes first

Frozen ontology YAML (train IDs, LAS maps, colors, void policy), three-tier corridor splits with leakage validation and locked-tier authorization guard, metrics (delegating classification wrapper + FP/km, continuity, corridor-clustered bootstrap), provenance manifest wrapper. ontology.py/splits.py interfaces freeze before WP4 starts implementing against them.

WP4 — Ingest + adapter after WP3 freeze

Bounded LAS/LAZ/COPC readers with stable source-row identity, seg3d/ReCap artifact validation and exact 1 mm label join (unmatched → void, ≤2% ceiling), deterministic corridor-axis tiling with pinned origins, canonical NPZ schema, SPT/Pointcept NumPy emitters, overlap-logit blending and exact index remapping, plus prepare_dataset.py, ingest_preannotations.py, and the adapter round-trip verifier.

WP5 — Experiments + runners after WP1/3/4

All 14 experiment configs + experiment cards, strict config schema with typed study/gate constructs, dispatcher (fail-closed statuses), SPT/Pointcept overlays and runner shell scripts (Hydra --config-dir + searchpath; Pointcept registry import preflight), E2 voxel-oracle implementation, train.py / evaluate.py with the TensorBoard evaluation contract.

Integration & review

Integration agent: uv sync, full CPU suites in both repos, lock checks, cross-WP import smoke, fix breakage.
Adversarial review: Opus + Sol in parallel over the full diff, findings fixed by a dedicated agent, then a final Fable pass.
Nothing is committed or published without explicit approval; mlharness work stays on its feature branch.

Experiment matrix (E1–E14)

ID	What	Status
E1	SPT pilot, superpoint-purity oracle gate before training	implement-now
E2	CPU voxel-survival oracle at 2/3/5 cm, full-resolution labels	implement-now
E3–E6	Feature ablation, rare-class losses, imbalance factorial, multiclass-vs-OVR	template-only
E7–E9	EZ-SP scale-up; corridor tiling matrix; Pointcept challengers (SpUNet screen, PTv3/LitePT T4 gates)	gated / template
E10–E14	Pseudo-labels, gold+pseudo, weak supervision, own-corridor SSL, active learning	gated-later

Risks

high Upstream label resolution: written seg3d NPZ artifacts are voxel representatives (~6.67 cm); any E2 shortcut through them silently produces wrong oracle numbers. Mitigated by hard preflight: E2 accepts only the manifest-proven in-memory full-resolution fuse product.
high SPT/Pointcept API drift: both moved in 2026. Commits are pinned per experiment, bridges are contract-tested where a checkout exists, and inverse-map behavior must be verified at the pinned commit before E1 acceptance.
med 2D consumer regression from the dependency split (OpenCV arrives transitively today): caught by the mandatory clean-environment install gate in WP1, not by reused environments.
med Package bootstrap: 0.2.0 and the upstream seg3d package may not be on the private index yet; interim editable path sources with TODO markers keep local work unblocked, publication is an explicit operator step.
med T4 constraints: no FlashAttention (Turing), spconv wheels capped at cu126 — enforced by runner preflights and the E9 operational gates instead of trust.
low Parallel-agent conflicts: five work packages own disjoint file sets (every __init__.py explicitly assigned), CLI and DVC stage contracts frozen before parallel work.

Open questions

Pilot corridor IDs: which real corridors seed corridor_splits_v1.yaml (train / validation / first locked promotion pair)? Placeholders ship until then; E1/E2 real runs need them.
New LAS codes for pole/structure (proposed 72/73): needs an upstream ontology decision; until then those classes learn only from updated ReCap exports.
Annotation tooling for the first gold corridor (ReCap round-trip vs alternatives) — E14's workflow decision, not blocking the harness.
DVC remote capacity for dense point tiles — confirm before the first dvc push.