Handoff — Production Step 7A/7B with dash fusion (modelling-lines 0.13.0) from existing Step 6 clusters

2026-09-04 · continuation agent on Battlebox, working in ~/dev/3dai.iolabs.orchestrator (WSL Ubuntu26, user ai) · AI3D-318 · written by Claude Fable 5.1

Summary

Goal. Produce Step 7A (skeletonize, parallel on T4 GPUs) and Step 7B (connect + road axis, CPU) outputs on Azure ML for Abschnitte 1, 2 and 3, reusing the already-finished Step 6 mask clusters, with the new boundary dash fusion: dashes that straddle a segment plane are exported as one <line> fitted from both point clusters instead of two touching halves. The helper pipelines pipelines/helpers/abschnitt_{1,2,3}_from_step6.yaml run exactly this graph (Prepare 7 → 7A → 7B → 7C overlay; no Step 8/10).

State. Package iolabs-point-cloud-modelling-lines 0.13.0 is on Nexus (tag v0.13.0). Orchestrator master @ 6de8934 pins env s7_cluster_stepper:23 in both Step 7 components. todo Env v23 is not registered in the workspace (checked 2026-09-04: az ml environment show --name s7_cluster_stepper --version 23 → "No environment exists"). Register it first; submitting before that fails. Battlebox checkout was 2 commits behind master when this was written.

Deliverable. One successful *_from_step6 run per Abschnitt on env v23, with the verification in §Verification done (sidecars present, fusion summary > 0 pairs, XML line count lower than the previous run, control plots sane), and run names + child GUIDs recorded in docs/handoff_abschnitt_1_step3_planes.md §3 style (or a new docs note) and reported back to Miro.

Current state code done nothing submitted yet

Inputs per Abschnitt (child-job GUIDs on workspaceblobstore)

Mount form is mandatory: azureml://datastores/workspaceblobstore/paths/azureml/<guid>/<output>/. Raw https:// blob URLs fail with ScriptExecution.StreamAccess.Authentication. Keep step3 and step6 from the same chain: prepare_data_for_step_7 joins step3_output/manifests/branches.json against step6_output/<geometry_dir>/lane_points/segment_*/clusters_mask/run6_cluster_*.npz.

Abschnittstep3_outputstep4_output (7C only)step6_outputstep1 trajectory_outputPrevious step7_output (baseline)
A1, July chain (what the helper YAML points at today)77f7ac77-a89a-446c-911c-37182f8caf4d1e5a9e8f-e2aa-4fe7-ade5-e8e413839ff59d21f6cf-59aa-46e7-a5ef-c3d037caa2d917853b44-4e50-422d-a3b2-f3c5049bf04b7b3292b5-ec5b-488e-81fc-f7c795c19883 (run …from_step6_20260722082250)
A1, return-count / no-angle-filter chain (AI3D-382, 2026-09-03, Step 6 on T4 nodes)e2b345bc-f1a2-434b-b02d-ebb944735b0a05482cf3… prefix9fe47e47… prefixreuse 17853b44…none yet
A2c678ba94-9691-421c-a62b-9963e3e74227f9d7d5da-f4aa-46b7-95c1-f865d2bb2d247d177605-2380-4a1c-a7ac-934ec90ecf63f5757d3b-9930-4c14-af44-be16717677185361bf44-540f-4182-9b70-f94e7f7477b7
A3f290eb36-6390-4a74-83a0-1aaf751973900e616d05-0099-4a08-bcf4-f5569fad8a3cf51a37f7-ec11-46c5-967c-b2e5860f74c68d3072e4-3acf-45cc-ae35-67e2810dc888look up in the A3 from_step6 helper history / job list

The A2/A3 helper YAMLs already carry their GUIDs; verify them against the table before submitting (grep existing_ pipelines/helpers/abschnitt_2_from_step6.yaml). For the A1 return-count chain the full Step 4/6 GUIDs must be resolved first (see step 3 below); the source table is docs/handoff_abschnitt_1_step3_planes.md §3.

Run procedure (on Battlebox, in the orchestrator repo)

  1. Sync the repo. 
    cd ~/dev/3dai.iolabs.orchestrator && git status --short && git pull --ff-only origin master
git log --oneline -1          # expect 6de8934 or newer
grep -n "s7_cluster_stepper" components/s7a_skeletonize.yml components/s7b_connect_axis.yml   # both :23
.venv/bin/python -m pytest -q tests/test_env_pins.py
  2. Register env v23 (skip only if show succeeds). The first job on a new env builds the image (~10–20 min); validate/submit after registration. 
    az ml environment show --name s7_cluster_stepper --version 23 -o table \
  || az ml environment create --file environments/s7_cluster_stepper/environment.yml
az ml environment list --name s7_cluster_stepper --query "[0].version" -o tsv   # expect 23
    Do not print conda.yml in logs or chats: the pip index line embeds Nexus credentials.
  3. Decide the A1 input chain (open question 1). If Miro wants the return-count chain, resolve the full GUIDs: 
    az storage blob list --account-name ai3dlfmlwpc011354919387 \
  --container-name azureml-blobstore-919dd9b5-5c3a-4888-80a3-6b3f79f2fc48 \
  --prefix "azureml/9fe47e47" --delimiter / --auth-mode key --query "[].name" -o tsv
# same for azureml/05482cf3 ; or: az ml job list --parent-job-name <from_step5_to_step6b parent> -o table
    Note: on 2026-09-04 the blob listing for prefix azureml/9fe47e47 returned no entries, so the Step 6 output of that chain is probably stored under a different job GUID than the prefix in the doc; resolve it via az ml job list --parent-job-name on the AI3D-382 from_step5_to_step6b parent run (experiment lanefinder_abschnitt_1, 2026-09-03) and check the step6_output blob folder exists before submitting. Then override at submit time instead of editing the YAML (--set inputs.existing_step6_output.path=azureml://datastores/workspaceblobstore/paths/azureml/<guid>/step6_output/, likewise step3 and step4).
  4. Validate, then submit one Abschnitt at a time (7A takes all 6 T4 instances; A1 has 131 + 11 segments across two branches). 
    export RUN="lanefinder_abschnitt_2_from_step6_$(date -u +%Y%m%d%H%M%S)"
az ml job validate --file pipelines/helpers/abschnitt_2_from_step6.yaml -o json
az ml job create --file pipelines/helpers/abschnitt_2_from_step6.yaml --set name="$RUN" \
  --query "{name:name,status:status,studio:services.Studio.endpoint}" -o json
    --set name=… is mandatory (fixed name: in the YAML, AML rejects duplicates). Suggested order: A2 first (smallest risk, has a baseline), then A3, then A1.
  5. Monitor. 
    az ml job show --name "$RUN" --query "{status:status}" -o tsv
az ml job list --parent-job-name "$RUN" --query "[].{n:display_name,s:status,id:name}" -o table
az ml job stream --name <7A child id>        # live logs
    Child display names: Prepare Data for Step 7, Step 7A - Skeletonize, Step 7B - Connect and Axis, Step 7C - Overlay XML on Topdown Tiles. Kibana (data view logs-iolabs-*): ai3d.pipeline_step: "s7a" / "s7b", correlate with ai3d.aml_root_run_id. Expected duration from earlier runs: 7A about 1–2 h per Abschnitt on 6 T4 instances; 7B minutes.
  6. Record run name, 7A/7B child GUIDs and the step7_output path per Abschnitt (needed later by Step 8/10 helpers), and report to Miro with the verification numbers.

Verification (what "done" means)

  1. Env actually used. In the 7A/7B child job details the environment must be s7_cluster_stepper:23; in 7B logs the XML metadata software_version / pkg_version("iolabs-point-cloud-modelling-lines") must read 0.13.0.
  2. Sidecars exist. In the 7A output (azureml/<7A child>/step7a_snapshots/<geometry_dir>/lane_points/) expect NNNN_lane_state_partial.json and NNNN_lane_state_partial_boundary_clusters.npz for most segments (segments without boundary dashes legitimately have no sidecar). 7A logs: event step7_partial_boundary_clusters per segment. 
    az storage blob list --account-name ai3dlfmlwpc011354919387 \
  --container-name azureml-blobstore-919dd9b5-5c3a-4888-80a3-6b3f79f2fc48 \
  --prefix "azureml/<7A child>/step7a_snapshots/" --auth-mode key \
  --query "[?ends_with(name,'.npz')] | length(@)"
  3. Fusion happened. 7B log event step7_dash_fusion_summary per branch: fused_count > 0, refit_count ≈ fused_count, axis_count 0 or near 0. On the local A1 subset the ratio was 19 pairs per 22 segments; whole A1 should be in the order of 100+ pairs. If fused_count = 0 with sidecars present, stop and diagnose (flags not facing → check boundary_end values in a partial JSON).
  4. XML sanity. Download step7_output/branches/<branch>/run7_lanes_*.xml and compare with the baseline run of the same Abschnitt: <line> count lower by roughly the fused-pair count; feature (lane) count about equal; no line longer than ~12 m for dashed features (a 20 m+ dash means two neighbours got fused; the gates should make this impossible — report it). 
    az ml job download --name <7B child> --download-path /tmp/ai3d-aml/$RUN --all
grep -c "<line" /tmp/ai3d-aml/$RUN/named-outputs/step7_output/branches/*/run7_lanes_*.xml
  5. Control plots. run7_control_plot_<branch>_*.pdf present per branch and the 7C overlay collages look like the previous run (dashes continuous across segment limits, no doubled strokes at plane positions).
  6. Manifest. step7_output/manifests/step7_branches.json present with all branches (Step 10 hard-requires it).

Risks & open questions

Suggested skills

Key artifacts & handles