Fix S6 OOM on junction segments (A3 segment_145)

2026-07-15 · Miro / AI3D LaneFinder · generated by Claude

Summary

Yes — it is a memory issue, now confirmed at both the infra and code level. s6_mask_clustering was OOM-SIGKILLed on A3 segment_145 (both attempts), and the AML job still reported Completed because the parallel component sets mini_batch_error_threshold: -1 — failed segments are silently abandoned.

Why this segment: it sits on a road junction and pools step-3 point chunks from both roads' scan runs — 2.52 GB compressed npz vs 1.4–1.65 GB for neighbors — on CPU-Compute-4x nodes with only 7.9 GB RAM.

The code-side trigger: the deployed package (v0.2.2) enables intensity separation by default, and that path loads the entire segment point set at once (pipeline.py:166 → load_step3_points): every chunk fully materialized simultaneously, points upcast to float64, a full-copy np.concatenate, a chained data.append that reallocates channel arrays per chunk, plus full-size projection index arrays. Peak RSS is a small multiple of the uncompressed input — several times what the node has free.

Fix strategy: (1) fail loudly (mini_batch_error_threshold: 0), (2) memory-lean loading in this package (v0.2.3) — load only what separation needs and preallocate, (3) patch A3 by re-running just segment_145 and merging outputs, then refresh the N:-drive deliverables.

Memory analysis (why it OOMs)

The per-segment flow in process_segment (pipeline.py) has three passes over the step-3 npz chunks:

Pass	Where	Memory behavior
Extrema scan	`pipeline.py:90–98`	lean one chunk at a time, discarded after min/max
Point→component assignment	`pipeline.py:139` → `assign_point_chunks`	mostly lean generator-fed, keeps only mask-hit points (small fraction); repeated `concatenate`/`append` per chunk is quadratic-ish but on small arrays
Intensity separation prep	`pipeline.py:165–169` → `load_step3_points`	the bomb loads ALL chunks fully at once, then copies

Inside load_step3_points (input_io.py:53–61):

chunks = [load_step3_file(p) for p in paths] — the whole uncompressed dataset resident at once; each file's points upcast to float64 (input_io.py:32).
np.concatenate([chunk.points ...]) — inputs + output alive simultaneously (transient ~2× on xyz).
data = data.append(chunk.data) in a loop — a fresh reallocation of the accumulated channel arrays on every iteration (quadratic total allocation churn).
Then frame.project_xy over all points produces full-length sep_rows/sep_cols index arrays (int64 → another 16 B/point).

All six npz arrays (points, scan_angle, intensity, red, green, blue) are loaded, but the separation path only consumes xyz + intensity (separate_cluster, sample_anchor_intensities); RGB and scan-angle are dead weight on this path (only the save_padded_clusters debug option touches them).

Quantified peak (Grok analysis, cross-checked)

Step-3 files are written with plain np.savez (uncompressed — verified in s3_segment_mapper.py:873), so 2.52 GB on disk ≈ raw bytes. At ~33 B/point (xyz float64 = 24, intensity uint16, RGB uint16×3, scan-angle int8) that is N ≈ 76 M points for segment_145.

Allocation	Size	Where
All chunks resident + concat/append transients	~4.3 GB (~57 B/pt)	`input_io.py:54–60`
Full-length int64 `sep_rows`/`sep_cols`	~1.2 GB (16 B/pt)	`pipeline.py:168`
Full-length float64 intensity casts per cluster	+0.6–1.2 GB spike	`intensity_separation.py:198–236` (`sample_anchor_intensities`)

Sum ≈ 5.5–6.5 GB against a 7.9 GB node already at ~70–77% baseline → SIGKILL. Neighbors at 1.4–1.65 GB input scrape through; the junction's +55% tips it over. Open3D neighbor search operates on local padded subsets only — not a contributor.

Both the 2026-07-12 new-model run and the 2026-07-02 run show the identical 2-blob hole — consistent with v0.2.x (separation enabled since v0.2.0, ~Jul 1) being deployed for both. The deployed env pins iolabs-point-cloud-mask-clustering==0.2.2.

Key decisions

Decision	Choice	Rationale
Silent failures	Set `mini_batch_error_threshold: 0` in `components/s6_mask_clustering.yml` (consider same for s7a)	One-line change; a hole in the deliverable is never acceptable, so the job should fail loudly. do first
Primary OOM fix	Memory-lean loading in this package → v0.2.3	Junction segments are a permanent feature of highways; SKU-bumping just moves the cliff. Lean loading makes 2.5 GB-compressed segments fit the existing 7.9 GB nodes. recommended
Bigger SKU	Fallback / belt-and-braces only	Use a high-mem compute for the one-off seg145 patch rerun (no package release needed); optionally keep as pipeline default later if lean loading proves insufficient. costs money, hides the class of bug
A3 patch scope	Re-run s6 for segment_145 only, merge into existing step6 output, re-run 7a→7c	Steps 3–7 are deterministic; non-failed segments reproduce identically, so don't re-run them. s8/ACC export not needed unless Miro asks.
Lean-loading design	Load only xyz+intensity for separation; preallocate from known totals; fill per chunk	`source_count` is already computed in the extrema pass — preallocation is free. Keeps float64 xyz → bit-identical results. Details in Phase 3.

Phases

Phase 1 — Sweep for other silent holes cheap, do first

For each s6 job of A1/A2/A3 (and optionally the 2026-07-02 runs): download artifacts/logs/sys/job_report/remaining_mini-batches.txt + processed_mini-batches.csv via az ml job download --name <s6-guid> --download-path <dir> (terminal jobs → allowed; logs only, no --all).
Cross-check blobs: list step6_output/branches/*/lane_points/segment_*/ and flag segments with ≤2 blobs (dir markers only). A1 has 2 branches.
Deliverable: table run × failed segments. A3-145 is known; junctions elsewhere are the suspects.
Handles: A3 s6 = 77f50279-1cce-485f-b277-71cb7cd3e802; A1/A2 run names = same prefix …_1/…_2, children via az ml job list --parent-job-name <run>. az is slow (60–120 s/call → timeouts ≥200 s) — prefer azcopy + SAS for blob listing; battlebox may still have ~/.ai3d_container_sas.

Phase 2 — Patch A3 segment_145 (and any other holes found)

Filtered tasks folder: copy 0145_branches_branch_000_lane_points_segment_145.task.json from azureml/6c90e71c…/step6_segment_tasks/ to a new blob folder; bind as step6_tasks.
Patch pipeline: author pipelines/helpers/abschnitt_3_from_step6_seg145.yaml binding step5_output = azureml/2139a64c…/step5_output/, step3/step4 from the reused GUIDs; instance_count: 1.
Compute: run s6 on a higher-memory SKU (check az ml compute list; OOM is per-node, so only more RAM per node — or the lean v0.2.3 build — helps). This one-off doesn't need to wait for the package release if a 16–32 GB SKU is available.
Merge problem: step-7 prep scans all segments under one step6_output root. Either azcopy-merge old step6_output + patch output into a new blob folder and run prepare_7/7a/7b/7c against it, or re-run 7a only for segment_145's task and re-run the 7b gather over combined snapshots. Verify against how prepare_data_for_step_7 builds tasks before choosing.
Deterministic reruns → only the patched segment changes; spot-check a neighbor segment hash to confirm.

Phase 3 — Prevent recurrence (package v0.2.3 + component changes)

a) Fail loudly (orchestrator repo): in components/s6_mask_clustering.yml set mini_batch_error_threshold: 0 (job fails if any segment fails after retries); review s7a for the same pattern. Version-bump the component per repo rules.

b) Memory-lean loading (this repo → v0.2.3):

pipeline.py:165–169: replace load_step3_points with a separation-specific loader that reads each npz once and keeps only points (float64, needed for anchor geometry → bit-identical) and intensity; never load RGB/scan-angle on this path (except when save_padded_clusters debug is on — keep the fat path behind that flag).
Preallocate: source_count is known from the extrema pass — allocate (N,3) float64 + intensity + row/col arrays once and fill per chunk; kills the concatenate doubling and the quadratic append chain. Est. peak on seg145: ~4.3 GB → ~2.0 GB.
Project per chunk into preallocated int32 row/col arrays (raster indices fit easily) — saves ~0.6 GB vs int64 on seg145.
Required, not optional: fix sample_anchor_intensities (intensity_separation.py:198–236) to stop casting the full-length intensity array to float64 per cluster — cast only the masked subsets. Without this, the 0.6–1.2 GB per-cluster spike can still OOM even after lean loading.
input_io.load_step3_points keeps its API for other callers but is no longer on the hot path; optionally fix its append-chain too.
Rejected options: float32 xyz (risks non-bit-identical separation cutoffs) and per-chunk polygon cropping (separation intentionally samples the asphalt ring outside the mask — cropping changes results, see comment at pipeline.py:162–164).
Add a regression guard: unit test asserting the separation path never materializes more than one chunk's channel set (e.g. via a counting loader stub), plus a peak-RSS smoke note in the changelog.

c) Release dance: bump to v0.2.3, publish to Nexus (/wrap-up flow), then in the orchestrator: bump environments/s6_mask_clustering/conda.yml pin → register new env version → bump component env pin (3-step env-version dance; the env-pin guard test in tests/ must pass).

d) Optional insurance: if the sweep finds junction segments >4 GB compressed anywhere, also raise the s6 compute SKU in the pipelines.

Phase 4 — Re-deliver A3

Refresh XML + overlays on the shared drive: N:\02_Modelling\05_Rink\02_AI 3D modeling\03_Working folder\260703_ML_Abschnitt_3\260713_step7_xml_overlays.
Battlebox is the delivery workhorse (azcopy fast, N: automount); its C: is low (~6.8 GB) — don't stage large data there.
Tell Miro when done so the colleague gets pinged. No s8/ACC export unless requested (429-throttled, not part of last delivery).

Risks

merge The step6-output merge for step-7 prep is the trickiest part of the patch — verify prepare_data_for_step_7's scanning logic before choosing merge-blob vs partial-7a rerun.
determinism Lean loading must keep float64 xyz and the same chunk ordering (sorted(paths)) or separation cutoffs could shift; keep dtype changes out of anything feeding compute_cutoff.
threshold mini_batch_error_threshold: 0 makes transient node failures fail the whole job; max_retries: 1 may deserve a bump to 2 alongside it.
downstream RAM Confirmed: sample_anchor_intensities re-inflates memory (full-length float64 intensity casts per cluster) — lean loading alone is not sufficient; ship all three fixes (streaming loader, int32 indices, masked-subset casts) together and verify peak RSS on seg145 during the patch run.
az az CLI 60–120 s/call through ssh; JMESPath filters flaky through ssh quoting — dump full lists instead.

Open questions

Which higher-memory CPU SKUs exist in the workspace (az ml compute list) — or does one need to be created for the patch run?
Threshold policy: hard 0, or a small tolerance (e.g. 1–2) with an alert? Hard 0 recommended given deliverable = per-segment XML.
Should the 2026-07-02 old-model runs be patched too, or are they superseded by the new-model delivery?
Does Miro want the padded-clusters debug output kept? If never used, deleting it simplifies the lean loader.