Handoff — Perspective render across all of Abschnitt 1/2/3

Summary

Time estimate — method & measured inputs

Measured per-task time (real, from the validated run)

From run lanefinder_abschnitt_1_render_perspective_20260708072345 (child d05ac804-710f-4872-9a12-b448b85ee612), the full production config (run7b axis + append-next-segment + chunked v0.3.3), 12 two-segment tasks on 4 nodes. Per-task elapsed seconds:

branch_000 seg 0..5 : 22.9  34.1  47.5  58.4  58.0  58.9
branch_001 seg 0..5 : 20.9  40.1  61.7  81.5  90.3  98.3
mean ~56 s · min 21 s · max 98 s (grows with segment size)

These are segments 0–5, which are on the smaller side. The per-segment NPZ blob-size audit showed interior segments reach ~0.9–1.2 GB each (a paired task ≈ up to ~2.3 GB ≈ 190M points). Large pairs will run longer than 98 s — use a blended planning mean of ~90 s/task for a full Abschnitt, and refine after the first ~30 real tasks complete.

Formula

wall_clock ≈ ceil(N_tasks / P_nodes) × t_task  +  overhead
  N_tasks   = total segments in the Abschnitt (append adds NO tasks)
  P_nodes   = s3c_instance_count (parallel workers, 1 proc/node)
  t_task    ≈ 90 s  (blended; measured 21–98 s on small segments)
  overhead  ≈ 5–10 min: node cold-spinup + prepare_data step
              (env IMAGE ALREADY BUILT for v4 → no rebuild wait)

Segment counts (the N you must confirm)

Segments are 100 m (s3_segment_length_m: 100.0). A2/A3 are highway sections of similar order to A1; a working guess is ~120–250 each, but count them exactly — it is the dominant term. Grand total is plausibly ~400–550 tasks.

Scenario table (per Abschnitt, refine N and P)

All-three total: the three Abschnitte are independent pipelines. Run sequentially → sum the rows (~3–4 h at 4 nodes; ~1–1.3 h at 16 nodes). Run concurrently → roughly the single-Abschnitt time, but you need 3 × P nodes free at once and enough F4s_v2 quota. The cluster CPU-Compute-4x is Standard_F4s_v2 (8 GB); confirm its scale_settings.max_instances and the subscription core quota before promising a node count (the earlier az ml compute show did not return a max — verify).

First two numbers to nail down (they determine the answer)

# 1) exact segment counts A2, A3 — from each Abschnitt's step3_output blob list
#    (see "How to run" for locating the step3 child), then:
grep -oE 'branch_[0-9]+/lane_points/segment_[0-9]+' <bloblist> | sort -u | wc -l
# 2) cluster ceiling
az ml compute show --name CPU-Compute-4x -g AI3D-rg -w ai3d-lf-mlw-pc-01 \
  --query "{size:size,min:scale_settings.min_instances,max:scale_settings.max_instances}" -o json
az vm list-usage --location polandcentral --query "[?contains(name.value,'FSv2')]" -o table

Current state A1 done end-to-end

• Rasterizer iolabs-image-analyzer-rasterizer v0.3.3 on Nexus, master pushed (Bitbucket ioholding/3dai.iolabs.imageanalyzer.rasterizer), tags v0.3.0..v0.3.3. v0.3.3 = chunked accumulation for process_perspective: peak RAM flat vs total points (scales only with the largest single NPZ), byte-identical outputs. 94 tests green.
• Orchestrator branch ai3d-two-stage-step7 pushed (Bitbucket ioholding/3dai.iolabs.orchestrator). New step s3c_render_perspective: wrapper scripts/render_perspective/render_perspective.py, component components/s3c_render_perspective.yml (v2), env environments/s3c_render_perspective/ registered as azureml:s3c_render_perspective:4 (pins rasterizer 0.3.3).
• Validated on Azure, twice completed: plane-axis run (child d37201a7…, 12/12, 252 imgs) and the run7b-axis + append run (child d05ac804…, 12/12, 290 imgs). Both limited to segment 0–5. Images downloaded to D:\ai3d-aml\perspective-views\ and D:\ai3d-aml\perspective-views-axis\.
• Axis XML exists locally for all three: /home/miro/dev/3dai.lanefinder/data/00_external/260703_Abschnitt_{1,2,3}/run7b_axis/branch_XXX/run7_lanes_*.xml. Only A1's is uploaded to blob so far (lanefinder/aux/abschnitt_1_run7b_axis/).

Render config already dialed in (keep these)

camera_spacing 10 m · camera_height 10 m · pitch_down 25° · fov 70°
image 1600×900 · color rgb · save_depth true · append_next_segment true
axis: Step-7 HighwayData XML "Central Axis" polyline (geoshift-relative,
      == run3 geoshift; projects each segment's plane points onto the axis)
aggregation topmost (nearest-wins) · outputs: RGB + 16-bit-cm depth PNG + per-seg JSON

How to run all three — step by step

The Abschnitt-1 helper pipeline is the template: /home/miro/dev/3dai.iolabs.orchestrator/pipelines/helpers/abschnitt_1_render_perspective.yaml. It binds an existing step3_output (mounted, datastore form) + an axis_root, runs prepare_data_for_step_3b (which honours min/max_segment_index) then s3c_render_perspective.

1. Confirm env is current: the pushed component references azureml:s3c_render_perspective:4; env v4 is registered. No rebuild needed unless you bump the rasterizer again.
2. Locate each Abschnitt's Step-3 step3_output. A1 uses child 77f7ac77-a89a-446c-911c-37182f8caf4d (from full run lanefinder_abschnitt_1_pipeline_20260702070612). For A2/A3, the recent *_from_step3_* runs bind an existing step3 rather than producing one, so read their YAML/inputs to get the real path, or find the s3_segment_mapping child of the last full pipeline for each. Recent completed candidates: lanefinder_abschnitt_2_from_step3_20260702165047, lanefinder_abschnitt_3_from_step3_20260702203744 (inspect az ml job show --query inputs). Grab the datastore-form path azureml://datastores/workspaceblobstore/paths/azureml/<child>/step3_output/.
3. Count segments for A2/A3 from those step3 blob lists (command in the estimate section) → fills the N in the time estimate.
4. Upload A2/A3 axis XML to blob (A1 pattern), one XML per branch dir:

   for A in 2 3; do for b in branch_000 branch_001; do
     F=$(ls /home/miro/dev/3dai.lanefinder/data/00_external/260703_Abschnitt_$A/run7b_axis/$b/run7_lanes_*.xml)
     az storage blob upload --account-name ai3dlfmlwpc011354919387 \
       --container-name azureml-blobstore-919dd9b5-5c3a-4888-80a3-6b3f79f2fc48 \
       --name "lanefinder/aux/abschnitt_${A}_run7b_axis/$b/$(basename $F)" \
       --file "$F" --auth-mode key --overwrite; done; done
   # NB: confirm each Abschnitt's branch count — A2/A3 may not both have exactly 2 branches.

5. Clone the helper pipeline to abschnitt_2_render_perspective.yaml / abschnitt_3_render_perspective.yaml. Change: experiment_name, existing_step3_output.path, existing_axis_root.path (to the A2/A3 aux path), set s3c_min_segment_index: -1 and s3c_max_segment_index: -1 (= ALL segments), and raise s3c_instance_count to your chosen P.
6. Validate then submit:

   cd /home/miro/dev/3dai.iolabs.orchestrator
   az ml job validate --file pipelines/helpers/abschnitt_2_render_perspective.yaml -o json
   RUN="lanefinder_abschnitt_2_render_perspective_$(date -u +%Y%m%d%H%M%S)"
   az ml job create --file pipelines/helpers/abschnitt_2_render_perspective.yaml --set name="$RUN" -o json

7. Monitor (background poll, matched to cache windows — do NOT sleep 300s). On failure, pull child logs and check sys/job_report/processed_mini-batches.csv for per-task status; a silent worker death + fast fail + clean stderr = OOM (unlikely now with chunking, but a single >60M-pt NPZ could still stress 8 GB).
8. Collect to D drive with az storage blob download-batch (destination dir must pre-exist; use --overwrite true; flatten the deep azureml/<child>/step3c_output/branches/*/lane_points/perspective_views/ path into per-branch folders — see prior runs under D:\ai3d-aml\).

Auth note: uploading env/conda embeds the Nexus index credential; Miro has approved that upload. Pushing to master / default branch needs Miro's OK (auto-mode blocks it). Nexus publish uses UV_INDEX_NEXUS_USERNAME/PASSWORD env vars via UV_PUBLISH_USERNAME/PASSWORD.

Key artifacts & handles

Project memory (already written): ~/.claude/projects/-home-miro-dev-3dai-iolabs-imageanalyzer-rasterizer/memory/perspective-deployment-state.md and perspective-camera-spec.md — read them for the full frame contract and the OOM-triage note.

Risks & open questions

• confirm A2/A3 branch count & step3 path. The _from_step3 runs reuse an existing step3; verify the real datastore path and that each has the branch layout the axis upload assumes. Do not assume 2 branches.
• confirm Axis-vs-points frame per Abschnitt. A1's XML Geoshift matched run3_geoshift.json bit-exact. Re-check for A2/A3 — the wrapper logs a WARNING (does not fail) if they differ; a mismatch would misplace cameras.
• watch Off-axis segments. The A1 offline sweep flagged segments ~11–19 projecting 20–50 m off the branch axis (likely a ramp / second carriageway; the axis is per-branch, one carriageway). The wrapper warns when projection >20 m. Expect similar spots in A2/A3; eyeball a few mid-Abschnitt renders.
• memory 8 GB nodes. Chunking bounds peak to ~largest single NPZ; a single >60 M-pt NPZ (~5 GB transient) could still be tight. If a specific big segment OOMs, lower points_per_chunk in the rasterizer or the node-side; do not raise error_threshold to mask it.
• quota Node ceiling unknown. Confirm max_instances and FSv2 core quota before committing to 16/32 nodes or to running all three concurrently.
• Cost note: F4s_v2 fleet was chosen over GPU for cost; keep it CPU. The whole job is compute-cheap (minutes of core-time per Abschnitt), so node width is the only real lever on wall-clock.

Suggested skills

azure-cli (AI3D Azure ML ops — has the constants, child-job and blob recipes), and this repo's wrap-up if the rasterizer needs another Nexus release.