Fleet refactor: single source of truth, Google pyguide, bloat removal

2026-08-12 · Miro / AI3D pipeline · recon by 25+ agents (Grok explorers, Codex analysts+critic, Opus gap-fill), adversarially verified

Summary

Nine repos: seven pipeline stages (3dsegmentation, asphaltedge, guardrails, verticalsigns, tablecloth, maskclustering, lastools) plus two shared libraries (iolabs-common, iolabs-geometry-*). Adoption of the shared libs is uneven: asphaltedge/guardrails/verticalsigns already consume them, tablecloth partially, and seg3d, maskclustering, lastools consume nothing — they carry local copies of axis, config-loader, geoshift, raster-frame and CRS machinery.

The refactor has three strands, done per-repo in one migration wave each: (1) single source of truth — collapse ~10 verified duplicate clusters into iolabs-common (generic IO/config/logging) and iolabs-geometry-geometry/-raster (geometric algorithms, raster frames); (2) Google pyguide compliance — one shared ruff config, Google-style docstrings, kill broad except Exception, split giant modules, py.typed everywhere; (3) bloat removal — ~80 verified findings, roughly 31k LOC deletable (mostly stale investigation dirs), including all tests that assert default-config literals.

Blocking prerequisite: the 3dai.common checkout (0.3.2) is behind what three consumers already require (≥0.4.0 on Nexus); config_loader exists only in a shared-extraction worktree. Resolve that branch ambiguity before anything else.

Key decisions

Phases

Phase 0 — Groundwork 1–2 days
  • Resolve 3dai.common source of truth: reconcile the 0.3.2 checkout vs Nexus ≥0.4.0 vs the shared-extraction worktree holding config_loader; merge to main, tag. Also land the stranded feat/segment-points-io branch (worktree .claude/worktrees/segment-points-io) — it already implements the geoshift loader and most of the run3-NPZ discovery/concat SSOT (clusters C+D); highest-leverage single action in the whole plan.
  • Clone the split-out 3dai.iolabs.geometry.raster and 3dai.iolabs.geometry.visualization repos — both already exist on Bitbucket (AI3D-344 split is complete remotely); locally only the tombstoned pre-split copies exist. Never edit the archive.
  • Add iolabs-common as a dependency of seg3d — it currently has none, and that single missing edge is the precondition for six of the eight IO clusters.
  • Write characterization tests for the five frozen contracts (hash-join dtype/rounding, axis degenerates, spline root policies, raster half-pixel, ground modes concat/streaming). These live in the shared-lib repos.
  • Land the shared ruff config + py.typed + CI check in iolabs-common and iolabs-geometry-geometry first (they are the template).
  • Fix version-metadata drift now (guardrails 0.2.0 vs __version__ 0.1.0; maskclustering 0.2.4 vs 0.1.0) — cheap and unblocks clean publishing.
Phase 1 — Publish shared libraries 2–4 days

iolabs-common 0.5.0 gains: config_loader (packaged-default JSON + deep merge + --set overrides + unknown-key rejection), segment_points_io (from the feat/segment-points-io branch: geoshift loaders, run3-NPZ discovery/validated concat with Record boundaries and .part skipping, chunk streamer from tablecloth), point_hash (1 mm hash-join SSOT, float64→int64 spec), point_masks_io (spec'd point_masks.npz reader/writer, spec doc moves alongside), atomic_io (atomic_savez + cache_is_fresh), crs (WKT1-by-EPSG table, looks_georeferenced, optional pyproj), run_stats (stats.json conventions + numpy-safe JSON encoder), segments (range parsing 066-074,038, dir resolution). Also: split the 528-line memory_guard.py twin orchestration functions, and delete the deprecated memory_guard copy in geometry (pre-existing duplication between the two targets).

iolabs-geometry-geometry 0.10.0 gains: hardened Axis as the only station/offset implementation (asphaltedge's copy lacks validation — it adopts, not merges), polyline resample/densify/bbox-clip, corridor-from-edge-polylines (from seg3d pavement.py), policy-complete spline/plane intersection API (three root policies exposed explicitly — they are different algorithms, not one function), hash_join SSOT module with spec'd dtype+rounding, low-level HighwayData XML schema layer.

iolabs-geometry-raster 0.2.0: published from its existing split repo; no API change needed — guardrails/verticalsigns/asphaltedge already delegate to it. Prune never-imported exports (build_ground_model_concat/streaming top-levels, grid_dims, draw_halo_label) per bloat audit.

iolabs-geometry-visualization 0.7.0 (current release 0.6.2): published from its existing split repo as the QC render/overlay home — gains seg3d's orthographic splat rasterizer, tablecloth's overlay renderer, percentile normalization, label placement, dashed-line drawing; prunes its zero-consumer legacy exports (apply_*_colors, calculate_zy_rotation, lineset_from_spline, downsample helpers) once lastools' dead VISUALIZE branch is removed. open3d becomes an optional extra (iolabs-geometry-visualization[open3d], approved) — the new QC helpers are numpy+PIL only, so 2D-QC consumers don't inherit the heavy 3D dependency.

Each lib release also removes its own bloat (safe unused-public-API items: geometry's camera variants, deprecated memory_guard, polyline_hygiene.AxisFrameexcept axis.Axis and cutting_planes, which look unused only because their consumers migrate in Phase 2).

Phase 2 — Per-repo migration wave 1–2 weeks, ordered

Order: tablecloth → maskclustering → asphaltedge → guardrails → lastools → verticalsigns → seg3d. Per repo, one branch does all three strands: swap local modules for lib imports (leave one-cycle re-export shims), delete safe+probable bloat, apply ruff + docstring pass on touched files. Pipeline stays runnable after every repo because consumers read disk artifacts, not imports.

  • tablecloth: adopt common config_loader + stats; NPZ chunk-reader moves to common (it and guardrails hold near-verbatim copies); publish to Nexus so verticalsigns can drop its editable-sibling dependency.
  • maskclustering: replace hand-rolled types.py/raster_frame.py with iolabs-geometry-raster (last hand-roller); adopt common config_loader; typed config instead of nested dict[str,Any]; delete test_packaged_defaults_load + version-pin test.
  • asphaltedge: adopt geometry Axis (gains validation — characterization tests catch behaviour change); hash-join module (dtype decision applies here); biggest package-code cut: dead CRF pair-state branch (~900), zchannels format support (~1.1k), failed experiment gates (~550), boundary.py+shapely, ~25 unused re-exports, tqdm dep; stale experiment scripts (~950) move to dev/.
  • guardrails: corridor code adopts geometry Axis/frames; chunk reader from common; move analysis/ (~1.8k) + probe.py (~750) to dev/; delete dead corridor/geometry helpers; split 1963-line detect.py while touching it.
  • lastools: most surgical — live consumers outside the fleet (3dai.lanefinder imports las2open3d); its lock resolves geometry 0.6.1 (legacy). Migrate spline/plane code to the new policy-complete API last among producers; delete dead chain (~240 LOC incl. hard-disabled VISUALIZE branch, which also drops the iolabs-geometry-visualization dep); deprecate contrast.py (Miro's call): DeprecationWarning + drop from README public API, removal in a later release.
  • verticalsigns: after tablecloth publishes: swap editable sibling for Nexus wheel; adopt camera/projection from geometry; move out_eval/ (~19k) and tools/ (~3.7k) to dev/ (preserved, not deleted); trees.py + conifer wiring stays (Miro: not dead — keep as-is); split 1723-line config.py (297-field dataclass, 807-line from_mapping).
  • seg3d (this repo): last, biggest import swap: config.py→common config_loader, crs.py→common crs, geoshift+stats→common, lines_xml.py resample/clip→geometry, pavement.py corridor→geometry, hash-join→common point_hash, render.py rasterizer→geometry-visualization; delete test_config.py default-parity tests and test-only cluster helpers; scripts/orbit_video.py and smoke_seg066.py stay in scripts/ (Miro: orbit video is a recurring QC tool — supported script, imageio stays in the dev dependency group).
Phase 3 — IO/CRS duplication detail (8 verified clusters) the technical core

Verified by side-by-side code reading with numeric experiments. Recommended micro-order within Phases 1–2: land feat/segment-points-io → B → H → F → C → E → G(crs) → A → D; the lastools LAS-writer consolidation (G, part 2) is a separate later project.

ClusterTargetRiskEssence
A — 1 mm hash-join keyscommon point_hashmedseg3d float64→int64 is canonical; asphaltedge float32→int32 is a bug (1–4% dropped matches measured; int32 overflows at UTM northing). asphaltedge's match rate will rise after fix — diff surface_match_rate on real segments. Delete its dead _match_surface_kdtree alternative.
B — NPZ chunk streamercommon segment_points_iolowguardrails/tablecloth copies are near-verbatim (_read_points_header byte-identical). Take tablecloth's (.copy() = writeable chunks); pass chunk_points as int, not a config object.
C — run3_geoshift.json loadcommon segment_points_iolowFive implementations, three path conventions, two error policies. The unmerged feat/segment-points-io loader is canonical; add find_geoshift/_or_none variants + geoshift_from_mapping. Never bake in a sign — consumers legitimately add, subtract, or ignore the shift.
D*_run3_points.npz discovery+concatcommon segment_points_iomed-high~9 hand-rolled glob+concat loops with four dtype policies, only tablecloth skips .part files (everyone else can crash on partial writes). Migrate repo-by-repo with golden outputs; keep decimation policies OUT of the loader. Add seg3d's Record boundary table.
Epoint_masks.npz sidecarcommon point_masks_iolow-medTwo writers, one reader, spec only in prose. guardrails writes an identity instance_json_index (carries no info) and doesn't dedupe overlapping instances. Encode the spec in code; move point_masks_spec.md next to it; allow_pickle=False.
F — tablecloth kept_mask readtablecloth (producer owns reader)lowguardrails and seg3d read the same file with opposite polarity, opposite failure modes, and different path resolution (guardrails tries 4 layouts, seg3d 1 — seg3d silently misses masks guardrails finds). Unify on kept-sense + guardrails' resolution + seg3d's accounting.
G — LAS CRS/WKT + write conventionscrs→common; LAS writer→lastools (later)low / highNot a duplicate — an asymmetric gap: lastools has zero CRS handling and reads the legacy scan_angle_rank field, so it cannot read the PDRF7 files seg3d writes. Promote seg3d crs.py verbatim to common now; consolidating a LAS writer into lastools requires first splitting lastools into a thin las_io (laspy+numpy only — today it drags torch/open3d/visualization).
H — atomic NPZ writescommon atomic_iolowThree compression policies + two temp-naming schemes; tablecloth's own write_masks_npz is non-atomic in the same file that defends against truncated files. One atomic_savez with <name>.<pid>.part; fold in cache_is_fresh.

Deliberate non-unifications (documented, do not "fix"): the three RasterFrame variants encode different origin conventions (maskclustering's geoshift-invariant mapping is intentional); asphaltedge's twin assert_aligned_* functions guard different geoshift conventions — merging them would make the wrong call silently possible; the four asphaltedge raster loaders share shape but have distinct missing-file contracts tied to operational reality.

Phase 4 — Style compliance sweep ongoing per-repo + one final pass

Systemic fixes (fleet-wide, mostly automated): shared ruff config (line 100) + CI in all nine repos; Google-style docstrings on public API (convert NumPy-style in geometry and lastools; write missing ones — guardrails 23/92 undocumented, verticalsigns 40/155, maskclustering 13/33; seg3d is already closest to the Google standard — use it as the fleet baseline); eliminate broad except Exception (23 in one common file, two silent optimizer fallbacks in geometry fit_spline, batch runners fleet-wide — catch narrow, log+record in continue-on-fail loops); module-vs-symbol import hygiene; py.typed in every published wheel (annotation coverage already 445/452 — cheap win); version SSOT via importlib.metadata (kills the drift class: guardrails 0.2.0 vs 0.1.0, maskclustering 0.2.4 vs 0.1.0, lastools no __version__ at all).

Fleet-wide antipatterns from the second audit pass (seg3d/asphaltedge/tablecloth/lastools):

  • Underscore-module-as-public-API: asphaltedge's _config.py re-exported via __all__, _rate_limit_run in public API, tablecloth's _read_points_header imported cross-module (fixed by cluster B's promotion to common). Rename to public modules or add thin public wrappers.
  • lastools needs a hygiene pass before any migration: broken docstrings (dead strings not in first-statement position, ×3), identifier typos (trajecotry_*, OUTLIER_REMOVVE_*), fully untyped public functions, function-local ALL_CAPS temps.
  • tablecloth: filter.py shadows the builtin (rename to runner.py during migration); argparse default=[] shared mutable default; bare cleanup except Exception in reader/io.py:159.
  • Megamodule priority order: asphaltedge image_edge.py (3650 lines, worst in fleet — ImageEdgeConfig alone is ~687) → guardrails detect.py (1963) → verticalsigns config.py (1723) + detect.py (1420) → geometry fit_spline.py (1272) → seg3d fuse.py (628, fuse_segment ~346) → tablecloth ground.py (626) → lastools las2open3d.py (427). Split by responsibility only when the module is already being touched in Phase 2.
Phase 5 — Verification & closeout gate per repo + final
  • Per-repo gate: tests green, characterization suites green, wheel published, downstream stage re-run on one real segment (A1 branch_000 reference segment) with byte-identical or explained-diff outputs.
  • Final: full pipeline run on a reference Abschnitt; compare stats.json chain against pre-refactor baseline; delete all re-export shims; archive tombstoned repo; wrap-up per repo (version bump, Nexus publish, Jira-tagged commits).

Risks

Decisions log (owner answers, 2026-08-12)


Sources: workflow sot-refactor-recon (15 agents: 8 Grok repo maps, 2 Codex style audits, 4 Codex duplication themes, 1 Codex critic, 718k tokens), workflow sot-refactor-bloat (5 Codex analysts, 66 findings), gap-fill agents (Codex tablecloth/maskclustering bloat; Opus 5 IO/CRS 8-cluster analysis with numeric verification; Grok style audit of seg3d/asphaltedge/tablecloth/lastools). Owner decisions incorporated 2026-08-12. Full agent outputs in session task files.