Handoff — Create the perspective images

Summary

What the images are

Pinhole-camera renders of a point cloud: each point is projected into a camera, nearest point wins per pixel (occlusion), 1 px per point. Output per camera = an RGBA PNG (1600×900) + a 16-bit depth PNG (centimeters). Cameras are placed along the road axis, ~1 per 10 m, 10 m above the road, pitched 25° down, looking ~21 m ahead along the axis curve. World-up is +Z; image-v grows downward.

The rasterizer library (iolabs_image_analyzer_rasterizer) provides the kernel: CameraPose, project_perspective, cameras_from_road_axis, and TopdownRasterizer.process_perspective(...). The Step-7-XML axis parsing + two-segment appending live in the orchestrator wrapper (scripts/render_perspective/render_perspective.py), not the library.

Create locally CPU, no Azure compute

1 · Install the package (Nexus)

Python 3.11–3.12. Depends on torch and iolabs-geometry-visualization (also on Nexus). Use your Nexus creds from env — do not inline them:

uv venv && source .venv/bin/activate       # or any venv
uv pip install --extra-index-url \
  "https://$NEXUS_USER:$NEXUS_PASS@nexus.iolabs.ch/repository/pypi-private/simple/" \
  "iolabs-image-analyzer-rasterizer==0.3.3" pillow
# (private index is authenticate=always; iolabs-geometry-visualization resolves from it too)

2 · Get one segment's inputs from blob

Use the azure-cli skill's blob recipes. For an Abschnitt-1 segment (Step-3 child 77f7ac77-a89a-446c-911c-37182f8caf4d), download into seg/:

ACC=ai3dlfmlwpc011354919387
CT=azureml-blobstore-919dd9b5-5c3a-4888-80a3-6b3f79f2fc48
BASE="azureml/77f7ac77-a89a-446c-911c-37182f8caf4d/step3_output/branches/branch_000/lane_points"
mkdir -p seg/segment_000
# per-branch geoshift + planes:
az storage blob download --account-name $ACC --container-name $CT --auth-mode key \
  --name "$BASE/run3_geoshift.json" --file seg/run3_geoshift.json
az storage blob download --account-name $ACC --container-name $CT --auth-mode key \
  --name "$BASE/run3_planes.npz" --file seg/run3_planes.npz
# the segment's point clouds (list then pull the *_run3_points.npz ones):
az storage blob download-batch --account-name $ACC --source $CT --auth-mode key \
  --destination seg/segment_000 \
  --pattern "$BASE/segment_000/*_run3_points.npz"   # NB: recreates the blob path under dest

Each NPZ has keys points,red,green,blue,intensity. points are geoshift-relative (already raw − geoshift); pass the geoshift so the renderer does not re-shift.

3 · Minimal render script (plane-straight axis)

Self-contained via the public library — axis = plane-boundary point i → i+1. Good enough for a quick look:

import glob, numpy as np
from pathlib import Path
from PIL import Image
from iolabs_image_analyzer_rasterizer.rasterization.perspective import cameras_from_road_axis
from iolabs_image_analyzer_rasterizer.rasterization.topdown_rasterizer import (
    TopdownRasterizer, load_geoshift_json)

D = Path("seg"); SEG = 0
geoshift = load_geoshift_json(D / "run3_geoshift.json")        # (3,) float64
planes   = np.load(D / "run3_planes.npz")                       # pointNNN / normalNNN
p0 = planes[f"point{SEG:03d}"]; p1 = planes[f"point{SEG+1:03d}"]
axis = np.stack([p0, p1])                                       # (2,3), geoshift-relative

cams = cameras_from_road_axis(axis, spacing=10.0, height=10.0,
        pitch_down_deg=25.0, fov_deg=70.0, image_size=(1600, 900))

npz = sorted(glob.glob(str(D / f"segment_{SEG:03d}" / "*_run3_points.npz")))
r = TopdownRasterizer(device="cpu", aggregation="topmost")
merged, depths, per_file, meta = r.process_perspective(
        [Path(p) for p in npz], cams, color_modes=["rgb"], geoshift=geoshift)

Path("out").mkdir(exist_ok=True)
for cam_id, modes in merged.items():
    Image.fromarray(modes["rgb"]).save(f"out/{cam_id}.png")     # HxWx4 uint8
    d = depths[cam_id]                                          # HxW float32 m, 0=empty
    Image.fromarray(np.clip(d*100,0,65535).astype("uint16")).save(f"out/{cam_id}_depth.png")
print(meta["cameras"][0]["position"], len(merged), "cameras")

Runs in seconds/​camera on CPU. device="cuda:0" if a GPU is present — the code is device-agnostic.

4 · Faithful reproduction (real Step-7 axis + appended next segment)

To match the production images exactly, reuse the wrapper's helpers instead of cameras_from_road_axis. From an orchestrator checkout (3dai.iolabs.orchestrator, branch ai3d-two-stage-step7), the functions in scripts/render_perspective/render_perspective.py are import-safe without Azure:

• _get_axis_data(axis_root, branch_dir_name) → parses the run7_lanes_*.xml "Central Axis" polyline (+ verifies its Geoshift matches run3_geoshift.json).
• _segment_axis(planes, seg_index, lookahead) and _cameras_from_step7_axis(...) → project the segment's plane points onto the axis and emit the CameraPose list that follows the curve.
• _next_segment_rel_path(...) → the sibling segment to append; concatenate its *_run3_points.npz into the file list before process_perspective.

Axis XML for a local run: pull from blob lanefinder/aux/abschnitt_1_run7b_axis/branch_XXX/run7_lanes_*.xml, or (if this machine has the lanefinder repo) …/3dai.lanefinder/data/00_external/260703_Abschnitt_{1,2,3}/run7b_axis/branch_XXX/. Then build cameras with those helpers and call process_perspective as above.

Create on Azure production, whole Abschnitte

The s3c_render_perspective parallel pipeline renders every segment across nodes. Use the azure-cli skill for constants and job ops. Template pipeline (already validated on A1 segments 0–5):

cd 3dai.iolabs.orchestrator            # branch: ai3d-two-stage-step7
# pipelines/helpers/abschnitt_1_render_perspective.yaml
az ml job validate --file pipelines/helpers/abschnitt_1_render_perspective.yaml -o json
RUN="lanefinder_abschnitt_1_render_perspective_$(date -u +%Y%m%d%H%M%S)"
az ml job create --file pipelines/helpers/abschnitt_1_render_perspective.yaml --set name="$RUN" -o json

It binds an existing step3_output + an axis_root, runs prepare_data_for_step_3b (honours min/max_segment_index; set both to -1 for ALL segments), then the render component. Outputs land at azureml/<child>/step3c_output/branches/*/lane_points/perspective_views/ (RGB + _depth.png + per-segment JSON with absolute camera poses). Collect with az storage blob download-batch (destination must pre-exist; use --overwrite true).

For A2/A3 and full-run timing, see the companion handoff: handoff-perspective-full-abschnitte.

Inputs & where they live

Storage acct ai3dlfmlwpc011354919387, container azureml-blobstore-919dd9b5-5c3a-4888-80a3-6b3f79f2fc48 (key auth). Workspace: sub ba81b555-…, rg AI3D-rg, ws ai3d-lf-mlw-pc-01.

Config & gotchas

Dialed-in settings (keep)

spacing 10 m · height 10 m · pitch 25° · fov 70° · 1600×900
color rgb · save_depth true · append_next_segment true
aggregation topmost (nearest-wins) · axis: Step-7 "Central Axis" curve

• frame Cameras must be in the same frame as the points. With Step-3 NPZ, points are geoshift-relative → pass geoshift= and keep cameras relative (the plane points and the XML axis already are). Never mix a raw-absolute camera with relative points.
• memory v0.3.3 chunks accumulation so peak RAM is flat vs total points (scales only with the largest single NPZ). Locally this matters only for very large single files; the Azure nodes are 8 GB (F4s_v2) — that's why chunking exists. If a huge file strains RAM, lower the rasterizer's points_per_chunk constructor arg.
• axis The XML Geoshift must match run3_geoshift.json (A1 matched bit-exact). The wrapper logs a WARNING, not an error, on mismatch — a mismatch misplaces cameras. It also warns when a segment projects >20 m off-axis (A1 segments ~11–19 do — likely a ramp/second carriageway; the axis is per-branch, one carriageway).
• cosmetic 1-px points → dithering near the camera and "ghost" trails from vehicles that moved during the scan (data property, not a bug). Splatting is the eventual fix.

Repos & suggested skills

• Rasterizer: Bitbucket ioholding/3dai.iolabs.imageanalyzer.rasterizer (master, tags v0.3.0–v0.3.3). API docs in its CLAUDE.md.
• Orchestrator: Bitbucket ioholding/3dai.iolabs.orchestrator, branch ai3d-two-stage-step7.
• Skills: azure-cli (blob/job ops), wrap-up (if the rasterizer needs another Nexus release).