Fleet refactor: single source of truth, Google pyguide, bloat removal
Summary
Nine repos: seven pipeline stages (3dsegmentation, asphaltedge, guardrails, verticalsigns, tablecloth, maskclustering, lastools) plus two shared libraries (iolabs-common, iolabs-geometry-*). Adoption of the shared libs is uneven: asphaltedge/guardrails/verticalsigns already consume them, tablecloth partially, and seg3d, maskclustering, lastools consume nothing — they carry local copies of axis, config-loader, geoshift, raster-frame and CRS machinery.
The refactor has three strands, done per-repo in one migration wave each: (1) single source of truth — collapse ~10 verified duplicate clusters into iolabs-common (generic IO/config/logging) and iolabs-geometry-geometry/-raster (geometric algorithms, raster frames); (2) Google pyguide compliance — one shared ruff config, Google-style docstrings, kill broad except Exception, split giant modules, py.typed everywhere; (3) bloat removal — ~80 verified findings, roughly 31k LOC deletable (mostly stale investigation dirs), including all tests that assert default-config literals.
Blocking prerequisite: the 3dai.common checkout (0.3.2) is behind what three consumers already require (≥0.4.0 on Nexus); config_loader exists only in a shared-extraction worktree. Resolve that branch ambiguity before anything else.
Key decisions
- Two homes only.
iolabs-common= config loading, geoshift/CRS, stats-JSON conventions, numpy JSON serializer, segment-range CLI parsing, logging identity.iolabs-geometry-geometry= axis/station-offset frames, polyline ops, splines, corridor construction, hash-join spec.iolabs-geometry-raster= raster frames + ground models (already exists; maskclustering is the last hand-roller).3dai.io_geometry.geometryis tombstoned (AI3D-344, 2026-07-16) — archive-only; finish the split rather than touching it. - Contract-first migration. Characterization tests freeze behaviour before code moves: 1 mm hash-join dtype policy (seg3d float64→int64 vs asphaltedge float32→int32 — a real divergence, needs a spec'd SSOT module, not a shared constant), axis degenerate-input handling, all three spline-intersection root policies, raster half-pixel conventions, both ground modes.
- Additive releases, shim window. Publish
iolabs-common0.5.0rc1→0.5.0 andiolabs-geometry-geometry0.10.0rc1→0.10.0 without deleting existing symbols; migrated repos keep local modules as thin re-export shims for one release cycle, then delete. - Migration order (dependency-driven, critic-verified): resolve common branch → publish shared libs → tablecloth → maskclustering → asphaltedge → guardrails → lastools → verticalsigns (depends on tablecloth as editable sibling today) → seg3d last (pure consumer, most local copies).
- Unify plumbing, not policy. Config-loader plumbing (packaged-default JSON, deep merge,
--setoverrides, unknown-key rejection) goes to common; per-repo coercion/validation stays local. Same for lanes-XML: share a low-level HighwayData schema layer, keep the three parser strategies' call sites intact. Documented non-unify list stays in the plan appendix. - Bloat policy. Dead code inside package modules: safe findings delete in the migration commit of each repo; probable delete after a grep+test pass; remaining needs-owner-check items get a one-line decision from Miro first (list below). Stale investigation scripts and dirs (out_eval/, analysis/, probe.py, one-off scripts) are moved to a
dev/folder inside their repo — excluded from the wheel, lint, and CI — rather than deleted, per Miro's decision (2026-08-12). All default-config assertion tests (literal value asserts,__version__pins) are removed fleet-wide; schema-validation and override-merge tests stay. - Style baseline is tooling, not heroics. One shared
ruffconfig — line length 100 (Miro's call, matching fleet de-facto style over pyguide's 80), import order, docstring checks — copied into every repo +py.typedmarkers + Google-style docstring conversion on touched modules only. No repo currently has any lint/formatter config, so the diff is enforced by CI from day one. iolabs-geometry-visualizationlives and becomes the QC home. The shared QC render/overlay helpers (seg3drender.pyrasterizer, tableclothoverlay.py, p2–p98 percentile intensity normalization, collision-free label placement, dashed polyline drawing) unify there — not deferred (Miro's call). Its dead legacy API (zero-consumer exports) is pruned in the same release. Consumers declare it as an optional dependency (pytest-style: an optional extra such asmyrepo[qc]or the dev dependency group), never a hard runtime dep — QC rendering is not needed in production pipeline runs, and render code paths guard the import with a clear "install with [qc]" error.
Phases
Phase 0 — Groundwork 1–2 days
- Resolve
3dai.commonsource of truth: reconcile the 0.3.2 checkout vs Nexus ≥0.4.0 vs theshared-extractionworktree holdingconfig_loader; merge to main, tag. Also land the strandedfeat/segment-points-iobranch (worktree.claude/worktrees/segment-points-io) — it already implements the geoshift loader and most of the run3-NPZ discovery/concat SSOT (clusters C+D); highest-leverage single action in the whole plan. - Clone the split-out
3dai.iolabs.geometry.rasterand3dai.iolabs.geometry.visualizationrepos — both already exist on Bitbucket (AI3D-344 split is complete remotely); locally only the tombstoned pre-split copies exist. Never edit the archive. - Add
iolabs-commonas a dependency of seg3d — it currently has none, and that single missing edge is the precondition for six of the eight IO clusters. - Write characterization tests for the five frozen contracts (hash-join dtype/rounding, axis degenerates, spline root policies, raster half-pixel, ground modes concat/streaming). These live in the shared-lib repos.
- Land the shared ruff config +
py.typed+ CI check iniolabs-commonandiolabs-geometry-geometryfirst (they are the template). - Fix version-metadata drift now (guardrails 0.2.0 vs
__version__0.1.0; maskclustering 0.2.4 vs 0.1.0) — cheap and unblocks clean publishing.
Phase 1 — Publish shared libraries 2–4 days
iolabs-common 0.5.0 gains: config_loader (packaged-default JSON + deep merge + --set overrides + unknown-key rejection), segment_points_io (from the feat/segment-points-io branch: geoshift loaders, run3-NPZ discovery/validated concat with Record boundaries and .part skipping, chunk streamer from tablecloth), point_hash (1 mm hash-join SSOT, float64→int64 spec), point_masks_io (spec'd point_masks.npz reader/writer, spec doc moves alongside), atomic_io (atomic_savez + cache_is_fresh), crs (WKT1-by-EPSG table, looks_georeferenced, optional pyproj), run_stats (stats.json conventions + numpy-safe JSON encoder), segments (range parsing 066-074,038, dir resolution). Also: split the 528-line memory_guard.py twin orchestration functions, and delete the deprecated memory_guard copy in geometry (pre-existing duplication between the two targets).
iolabs-geometry-geometry 0.10.0 gains: hardened Axis as the only station/offset implementation (asphaltedge's copy lacks validation — it adopts, not merges), polyline resample/densify/bbox-clip, corridor-from-edge-polylines (from seg3d pavement.py), policy-complete spline/plane intersection API (three root policies exposed explicitly — they are different algorithms, not one function), hash_join SSOT module with spec'd dtype+rounding, low-level HighwayData XML schema layer.
iolabs-geometry-raster 0.2.0: published from its existing split repo; no API change needed — guardrails/verticalsigns/asphaltedge already delegate to it. Prune never-imported exports (build_ground_model_concat/streaming top-levels, grid_dims, draw_halo_label) per bloat audit.
iolabs-geometry-visualization 0.7.0 (current release 0.6.2): published from its existing split repo as the QC render/overlay home — gains seg3d's orthographic splat rasterizer, tablecloth's overlay renderer, percentile normalization, label placement, dashed-line drawing; prunes its zero-consumer legacy exports (apply_*_colors, calculate_zy_rotation, lineset_from_spline, downsample helpers) once lastools' dead VISUALIZE branch is removed. open3d becomes an optional extra (iolabs-geometry-visualization[open3d], approved) — the new QC helpers are numpy+PIL only, so 2D-QC consumers don't inherit the heavy 3D dependency.
Each lib release also removes its own bloat (safe unused-public-API items: geometry's camera variants, deprecated memory_guard, polyline_hygiene.AxisFrame — except axis.Axis and cutting_planes, which look unused only because their consumers migrate in Phase 2).
Phase 2 — Per-repo migration wave 1–2 weeks, ordered
Order: tablecloth → maskclustering → asphaltedge → guardrails → lastools → verticalsigns → seg3d. Per repo, one branch does all three strands: swap local modules for lib imports (leave one-cycle re-export shims), delete safe+probable bloat, apply ruff + docstring pass on touched files. Pipeline stays runnable after every repo because consumers read disk artifacts, not imports.
- tablecloth: adopt common config_loader + stats; NPZ chunk-reader moves to common (it and guardrails hold near-verbatim copies); publish to Nexus so verticalsigns can drop its editable-sibling dependency.
- maskclustering: replace hand-rolled
types.py/raster_frame.pywithiolabs-geometry-raster(last hand-roller); adopt common config_loader; typed config instead of nesteddict[str,Any]; deletetest_packaged_defaults_load+ version-pin test. - asphaltedge: adopt geometry
Axis(gains validation — characterization tests catch behaviour change); hash-join module (dtype decision applies here); biggest package-code cut: dead CRF pair-state branch (~900), zchannels format support (~1.1k), failed experiment gates (~550),boundary.py+shapely, ~25 unused re-exports, tqdm dep; stale experiment scripts (~950) move todev/. - guardrails: corridor code adopts geometry Axis/frames; chunk reader from common; move
analysis/(~1.8k) +probe.py(~750) todev/; delete dead corridor/geometry helpers; split 1963-linedetect.pywhile touching it. - lastools: most surgical — live consumers outside the fleet (
3dai.lanefinderimportslas2open3d); its lock resolves geometry 0.6.1 (legacy). Migrate spline/plane code to the new policy-complete API last among producers; delete dead chain (~240 LOC incl. hard-disabled VISUALIZE branch, which also drops theiolabs-geometry-visualizationdep); deprecatecontrast.py(Miro's call):DeprecationWarning+ drop from README public API, removal in a later release. - verticalsigns: after tablecloth publishes: swap editable sibling for Nexus wheel; adopt camera/projection from geometry; move
out_eval/(~19k) andtools/(~3.7k) todev/(preserved, not deleted);trees.py+ conifer wiring stays (Miro: not dead — keep as-is); split 1723-lineconfig.py(297-field dataclass, 807-linefrom_mapping). - seg3d (this repo): last, biggest import swap:
config.py→common config_loader,crs.py→common crs, geoshift+stats→common,lines_xml.pyresample/clip→geometry,pavement.pycorridor→geometry, hash-join→commonpoint_hash,render.pyrasterizer→geometry-visualization; deletetest_config.pydefault-parity tests and test-only cluster helpers;scripts/orbit_video.pyandsmoke_seg066.pystay inscripts/(Miro: orbit video is a recurring QC tool — supported script, imageio stays in the dev dependency group).
Phase 3 — IO/CRS duplication detail (8 verified clusters) the technical core
Verified by side-by-side code reading with numeric experiments. Recommended micro-order within Phases 1–2: land feat/segment-points-io → B → H → F → C → E → G(crs) → A → D; the lastools LAS-writer consolidation (G, part 2) is a separate later project.
| Cluster | Target | Risk | Essence |
|---|---|---|---|
| A — 1 mm hash-join keys | common point_hash | med | seg3d float64→int64 is canonical; asphaltedge float32→int32 is a bug (1–4% dropped matches measured; int32 overflows at UTM northing). asphaltedge's match rate will rise after fix — diff surface_match_rate on real segments. Delete its dead _match_surface_kdtree alternative. |
| B — NPZ chunk streamer | common segment_points_io | low | guardrails/tablecloth copies are near-verbatim (_read_points_header byte-identical). Take tablecloth's (.copy() = writeable chunks); pass chunk_points as int, not a config object. |
| C — run3_geoshift.json load | common segment_points_io | low | Five implementations, three path conventions, two error policies. The unmerged feat/segment-points-io loader is canonical; add find_geoshift/_or_none variants + geoshift_from_mapping. Never bake in a sign — consumers legitimately add, subtract, or ignore the shift. |
D — *_run3_points.npz discovery+concat | common segment_points_io | med-high | ~9 hand-rolled glob+concat loops with four dtype policies, only tablecloth skips .part files (everyone else can crash on partial writes). Migrate repo-by-repo with golden outputs; keep decimation policies OUT of the loader. Add seg3d's Record boundary table. |
E — point_masks.npz sidecar | common point_masks_io | low-med | Two writers, one reader, spec only in prose. guardrails writes an identity instance_json_index (carries no info) and doesn't dedupe overlapping instances. Encode the spec in code; move point_masks_spec.md next to it; allow_pickle=False. |
F — tablecloth kept_mask read | tablecloth (producer owns reader) | low | guardrails and seg3d read the same file with opposite polarity, opposite failure modes, and different path resolution (guardrails tries 4 layouts, seg3d 1 — seg3d silently misses masks guardrails finds). Unify on kept-sense + guardrails' resolution + seg3d's accounting. |
| G — LAS CRS/WKT + write conventions | crs→common; LAS writer→lastools (later) | low / high | Not a duplicate — an asymmetric gap: lastools has zero CRS handling and reads the legacy scan_angle_rank field, so it cannot read the PDRF7 files seg3d writes. Promote seg3d crs.py verbatim to common now; consolidating a LAS writer into lastools requires first splitting lastools into a thin las_io (laspy+numpy only — today it drags torch/open3d/visualization). |
| H — atomic NPZ writes | common atomic_io | low | Three compression policies + two temp-naming schemes; tablecloth's own write_masks_npz is non-atomic in the same file that defends against truncated files. One atomic_savez with <name>.<pid>.part; fold in cache_is_fresh. |
Deliberate non-unifications (documented, do not "fix"): the three RasterFrame variants encode different origin conventions (maskclustering's geoshift-invariant mapping is intentional); asphaltedge's twin assert_aligned_* functions guard different geoshift conventions — merging them would make the wrong call silently possible; the four asphaltedge raster loaders share shape but have distinct missing-file contracts tied to operational reality.
Phase 4 — Style compliance sweep ongoing per-repo + one final pass
Systemic fixes (fleet-wide, mostly automated): shared ruff config (line 100) + CI in all nine repos; Google-style docstrings on public API (convert NumPy-style in geometry and lastools; write missing ones — guardrails 23/92 undocumented, verticalsigns 40/155, maskclustering 13/33; seg3d is already closest to the Google standard — use it as the fleet baseline); eliminate broad except Exception (23 in one common file, two silent optimizer fallbacks in geometry fit_spline, batch runners fleet-wide — catch narrow, log+record in continue-on-fail loops); module-vs-symbol import hygiene; py.typed in every published wheel (annotation coverage already 445/452 — cheap win); version SSOT via importlib.metadata (kills the drift class: guardrails 0.2.0 vs 0.1.0, maskclustering 0.2.4 vs 0.1.0, lastools no __version__ at all).
Fleet-wide antipatterns from the second audit pass (seg3d/asphaltedge/tablecloth/lastools):
- Underscore-module-as-public-API: asphaltedge's
_config.pyre-exported via__all__,_rate_limit_runin public API, tablecloth's_read_points_headerimported cross-module (fixed by cluster B's promotion to common). Rename to public modules or add thin public wrappers. - lastools needs a hygiene pass before any migration: broken docstrings (dead strings not in first-statement position, ×3), identifier typos (
trajecotry_*,OUTLIER_REMOVVE_*), fully untyped public functions, function-local ALL_CAPS temps. - tablecloth:
filter.pyshadows the builtin (rename torunner.pyduring migration); argparsedefault=[]shared mutable default; bare cleanupexcept Exceptioninreader/io.py:159. - Megamodule priority order: asphaltedge
image_edge.py(3650 lines, worst in fleet —ImageEdgeConfigalone is ~687) → guardrailsdetect.py(1963) → verticalsignsconfig.py(1723) +detect.py(1420) → geometryfit_spline.py(1272) → seg3dfuse.py(628,fuse_segment~346) → tableclothground.py(626) → lastoolslas2open3d.py(427). Split by responsibility only when the module is already being touched in Phase 2.
Phase 5 — Verification & closeout gate per repo + final
- Per-repo gate: tests green, characterization suites green, wheel published, downstream stage re-run on one real segment (A1 branch_000 reference segment) with byte-identical or explained-diff outputs.
- Final: full pipeline run on a reference Abschnitt; compare stats.json chain against pre-refactor baseline; delete all re-export shims; archive tombstoned repo; wrap-up per repo (version bump, Nexus publish, Jira-tagged commits).
Risks
- high Hash-join dtype unification changes outputs. seg3d (float64→int64) vs asphaltedge (float32→int32) currently interoperate by accident of data ranges. Unifying changes join behaviour at mm boundaries. Mitigation: characterization tests + one real-segment diff before/after.
- high Nexus vs checkout drift. Consumers require iolabs-common ≥0.4.0 that no local branch cleanly provides; publishing 0.5.0 from the wrong base could regress what's on Nexus. Mitigation: Phase 0 branch reconciliation before anything.
- low Moved investigation dirs may be referenced by absolute path. BattleBox/Azure scripts outside grep reach may point at old locations of
out_eval/,analysis/etc. Mitigation: they move todev/(not deleted) in dedicated commits — a path fix or revert is trivial. - medium Out-of-fleet consumers.
3dai.lanefinderand Azure ML pipelines import lastools/maskclustering symbols and pin old wheels (lastools lock resolves geometry 0.6.1). Mitigation: shim window + keep-flag list is part of each repo's migration checklist. - medium Behavioural "hardening" is a silent change. Adopting geometry's validated Axis makes asphaltedge stricter (rejects inputs it previously accepted). Mitigation: run on real data early; treat new rejections as findings, not failures.
- low Style-pass churn. Docstring/import rewrites bloat diffs and mask logic changes. Mitigation: style commits separate from logic commits, enforced by convention in each PR.
Decisions log (owner answers, 2026-08-12)
- verticalsigns
out_eval/+tools/: not deleted — move todev/.trees.py+ conifer wiring: keep (not dead). - seg3d
orbit_video.py+smoke_seg066.py: keep inscripts/— orbit video is a recurring QC tool. - lastools
contrast.py: deprecate (warning + README removal now, delete later). - Raster/visualization homes: the split repos already exist on Bitbucket (
3dai.iolabs.geometry.raster,3dai.iolabs.geometry.visualization) — clone and publish from them; tombstoned workspace stays archived.-visualizationis kept and becomes the QC render/overlay home. hash_round_units_per_m: becomes a real config parameter — the sharedpoint_hashmodule takesunits_per_m(default 1000); repos wire it through their configs instead of validating it against a fixed constant. Producer and consumer of a join must agree — per-run stats record the value so mismatches are detectable.- QC render/overlay helpers: unify now into
iolabs-geometry-visualization(not deferred, not into common). Consumers take it as an optional dependency (extra/dev group, pytest-style), not a runtime dep. - Ruff line length: 100.