Perspective projection & multi-segment merge

2026-07-07 · iolabs-image-analyzer-rasterizer · generated by Claude

Summary

Both requests are feasible. (1) Perspective rendering works by transforming points before rasterization: world → camera frame → frustum cull → perspective divide, then reuse the existing PyTorch scatter kernel — the topmost (max-Z) aggregation becomes "nearest to camera" by feeding negated depth as Z. One safety fix is mandatory first: the kernel clamps out-of-canvas points onto border pixels instead of dropping them, which is harmless for top-down bbox framing but smears garbage in perspective. (2) Multi-segment merging already works today — process_tile accepts any list of NPZ files and frames them jointly — as long as all files share one geoshift frame. Confirmed 2026-07-07: segments of a run do share one run3_geoshift.json, so merging needs no code changes; a rebasing safety net stays in the plan as optional. Camera poses are trajectory-derived: ~1 per 10 m along the road axis, ~10 m above it, pitched slightly down — generated by a cameras_from_road_axis helper. A per-pixel depth map ships as a second output (free from the existing max-Z buffer).

Performance requirement (added 2026-07-07): 10+ camera positions per segment, deployed on cheap Azure CPU nodes (the earlier GPU→CPU move stands: fleet of CPU nodes beats fewer GPU nodes on cost). Consequence: the projection must be written in torch, not NumPy, so the whole per-camera loop runs vectorized on whatever device the rasterizer was constructed with — fast on CPU via MKL/OpenMP today, and a device="cuda:0" flag flip away from GPU if the economics ever change. The speed win comes from amortization: stage points + colors on the device once per segment, then each camera costs only a (3×3)·(3×N) matmul + cull + scatter. Per-camera NumPy round-trips are the anti-pattern to avoid.

Feasibility analysis

Q1 — Perspective via camera position: will it work?

Yes, because rasterize_tile is secretly projection-agnostic. Its core is: map (x, y) to a pixel index, resolve collisions by keeping the point with maximum Z (scatter_reduce_ amax). Nothing in that kernel knows about "top-down" — it renders whatever 2D coordinates you hand it. So perspective = a pre-transform:

1. p_cam = R @ (p_world − camera_pos)        # extrinsics
2. keep only p_cam.z > z_near                # frustum cull (mandatory!)
3. u = fx · x/z + cx ;  v = fy · y/z + cy    # perspective divide
4. rasterize (u, v, −z) with a pixel-unit framing
   → amax on −z ≡ nearest-to-camera wins → correct occlusion

Three things do not transfer and need handling:

Quality caveat Points are 1 px each. Near the camera the cloud spreads out and the image gets holes; far away it's dense. Acceptable for a first version; a splat radius (render each point as a small disk) is the standard follow-up if needed.

Q2 — Two+ segments in one image: possible today?

Yes, mostly already built. process_tile(npz_files=[...]) takes any list of files, concatenates all points, computes one shared framing, and renders one merged image (plus aligned per-file images with save_each). gather_segment_files(from_seg=…, to_seg=…) already collects a segment range — just flatten its dict values into one list:

files_by_seg = gather_segment_files(base, from_seg=16, to_seg=17)
all_files = [f for fs in files_by_seg.values() for f in fs]
merged, per_file, meta = rasterizer.process_tile(all_files, geoshift=gs)

The one gap: Step-3 NPZs are geoshift-relative, and each segment can carry its own run3_geoshift.json (centroid of its trajectory splines). Concatenating two segments with different geoshifts misaligns them by the difference of their origins — silently. The fix is a small rebasing helper: pick a common origin (first segment's geoshift), shift every other segment's points by (own_gs − common_gs) in XY(Z), then proceed exactly as today. If all files come from the same run and share one geoshift JSON, the current code is already correct with zero changes.

Key decisions

DecisionChoiceWhy
How to add perspective Projection as a pre-transform + reuse scatter kernel; new perspective.py module, new PerspectiveRasterizer (or method) Kernel already does the hard part (vectorized occlusion resolve on CPU/CUDA); no fork of the rasterization code
Device strategy for 10+ cameras/segment Projection written in torch on self.device; points/colors staged once per segment, reused across all cameras; NumPy only at load and final image export Same code is fast on CPU nodes (current Azure deployment) and GPU-ready via the existing device param; amortizing the upload/conversion makes per-camera cost ≈ one matmul + scatter
Occlusion / depth test Feed −depth as Z into existing topmost aggregation amax(−z) = nearest wins; zero kernel changes; "average" mode stays available too
Out-of-view points Filter in projection step (frustum cull incl. z ≤ z_near); keep kernel clamp as-is Kernel untouched → top-down behavior provably unchanged; behind-camera points must die before divide anyway
Camera specification CameraPose dataclass: position + look-at (3×3 rotation kept as escape hatch), vertical FOV, image size, z_near Confirmed use case is trajectory-derived poses, which map naturally to position + look-at; FOV+size derive fx/fy/cx/cy
Camera generation answered cameras_from_road_axis(axis_points, spacing=10.0, height=10.0, pitch_down_deg=…) helper: one camera per ~10 m along the road axis, ~10 m above it, looking slightly down along the driving direction Per Miro 2026-07-07: cameras come from the road axis, not free-flying; a helper keeps the geometry in one tested place
Depth map output answered Yes — return per-pixel distance alongside RGBA. In perspective mode the max_z buffer holds −depth, so depth = −max_z; empty pixels → 0 (or NaN), decided in implementation Nearly free (buffer already computed); useful downstream
save_each for perspective answered Reuse the existing save_each flag; per-file perspective renders share the same camera Miro has no preference — cheapest option is reusing the flag, no new config surface
Multi-segment merge Keep process_tile as the merge engine; add rebase_geoshift helper + optional per-file geoshift list Merging & shared framing already exist and are tested; only the frame mismatch is new
Metadata for perspective New camera sidecar dict, not framing_metadata Geo sidecar consumers must never receive fake geo data from a perspective render

Phases

Phase 1 — Multi-segment merge: safety net only S · downgraded

Confirmed 2026-07-07: all segments of a run share a single run3_geoshift.json, so cross-segment merging works today with zero code changes — flatten gather_segment_files() output into one process_tile call. This phase shrinks to a guard: accept an optional per-file geoshift sequence, and if they differ, rebase to the first file's frame (p_common = p_own + (gs_own − gs_common)) with a warning log. Implement opportunistically or skip until a mixed-run merge actually appears.

Tests: two synthetic NPZs with different geoshifts whose absolute points touch → merged image shows them adjacent, not offset; single-geoshift path byte-identical to current output.

Phase 2 — Projection module + CameraPose (torch, device-agnostic) M

New file rasterization/perspective.py. The projection math is torch, executed on the rasterizer's device — NumPy appears only at the NPZ boundary. This keeps the per-camera loop free of host↔device transfers and makes CPU execution vectorized (MKL/OpenMP) rather than interpreter-bound:

@dataclass
class CameraPose:
    position: np.ndarray        # (3,) world (or geoshift-relative) coords
    look_at: np.ndarray | None  # convenience; else rotation
    rotation: np.ndarray | None # (3,3) world→camera
    fov_deg: float = 60.0       # vertical FOV
    image_size: tuple[int, int] = (1920, 1080)
    z_near: float = 0.1

def project_perspective(pts: torch.Tensor, cols: torch.Tensor,
                        camera: CameraPose):
    """pts/cols already staged on device.
    → (proj_pts (M,3): u, v, −depth), cols (M,4), all on device.
    Culls z ≤ z_near and u/v outside the image."""

Unit-testable without any rasterization: a point straight ahead lands at image center; nearer of two collinear points has larger −depth; behind-camera point is culled; off-FOV point is culled. Decide the up-vector convention here (Z-up world, image v grows downward) and document it in the docstring.

Camera generation helper (confirmed use case — poses derived from the road axis):

def cameras_from_road_axis(axis_points,          # (M,3) polyline, geoshift-relative
                           spacing: float = 10.0,   # one camera per ~10 m
                           height: float = 10.0,    # meters above the axis
                           pitch_down_deg: float = 15.0,
                           **camera_kwargs) -> list[CameraPose]:
    # resample axis at `spacing` arc-length; at each sample:
    #   position = axis_pt + (0, 0, height)
    #   look_at  = point ahead along the axis, dropped by tan(pitch) · dist
    # → forward-facing drive-through views, slightly downward

Tests: straight synthetic axis → evenly spaced cameras, all looking forward-down; curved axis → look-at follows the curve. Exact pitch/FOV defaults are tuning knobs, not architecture.

Phase 3 — Batch multi-camera rasterization + camera sidecar M

First a small refactor: extract the scatter core of rasterize_tile into a tensor-native _scatter_rasterize(pts_t, cols_t, W, H) that takes tensors already on device (rasterize_tile becomes a thin NumPy wrapper around it — top-down path byte-identical). Then the batch orchestrator, which is where the 10+-cameras-per-segment speed lives:

def process_perspective(self, npz_files, cameras: list[CameraPose],
                        color_modes=None, geoshift=None):
    # per SEGMENT, once:
    #   load NPZs, rebase geoshifts (Phase 1), colors_for_mode,
    #   pts_t / cols_t = torch tensors on self.device
    # per CAMERA (cheap loop, all on device):
    #   proj, cols = project_perspective(pts_t, cols_t, cam)
    #   img_t, maxz_t = self._scatter_rasterize(proj, cols, W, H)
    #   images[cam_id] = img_t.cpu().numpy()   # only transfer per camera
    #   depths[cam_id] = (-maxz_t).cpu().numpy()  # per-pixel distance, ~free
    → (images: {camera_id: {mode: HxWx4}},
       depths: {camera_id: HxW float32},   # 0 where no point landed
       camera_metadata list)

Per-camera cost is one (3×3)·(3×N) matmul + divide + mask + scatter — no reload, no color re-conversion, no re-staging. On CPU nodes this is memory-bandwidth-bound and parallelizes across cores via torch; on CUDA the identical code applies. Check the Y-flip: rasterize_tile flips Y (north-up); with image-v-down projection coords, either pre-flip v in the projection or account for it — one deliberate sign choice, locked by a test (point above camera axis appears in the upper image half). Multi-segment perspective comes free via Phase-1 merging. Sidecar per camera: position (absolute world, i.e. geoshift added back), rotation, fov, image size, z_near. Add a benchmark script (N points × C cameras timing on CPU) so regressions are measurable.

Phase 4 — Integration tests, docs, release S

End-to-end test: synthetic scene (colored ground plane + a wall), camera looking at the wall → wall pixels occlude ground behind it; empty-input and all-culled inputs return the transparent background. Update CLAUDE.md/README (two projection modes, geoshift rebasing rules). Version bump + Nexus publish under the Jira tag.

Risks

RiskSeverityMitigation
Behind-camera / off-screen points clamped onto border pixels (kernel clamp) high Frustum + viewport cull in project_perspective before the kernel ever sees the points; dedicated test
Silent misalignment when merging segments with different geoshifts high Phase 1 rebasing; log a warning if per-file geoshifts differ and no rebase path was taken
Holes / see-through foreground near the camera (1-px points) med Accept in v1; document; follow-up: splat radius or downscale-render
Y-flip / handedness sign error → mirrored images med Asymmetric synthetic test scene (distinct colors left/right, up/down) locks orientation
float32 precision: absolute-world coords (~10⁶ m) through the projection med Do the camera transform in float64 relative coords (subtract geoshift first — Step-3 inputs already are), cast to float32 only after
CPU throughput too low for 10+ cameras × tens of millions of points med Amortized staging keeps per-camera work minimal; benchmark in Phase 3 sets the baseline; escape hatches: pre-cull points to a per-camera radius before projecting, chunk the point array, or flip device to CUDA — code is already device-agnostic
Per-camera host↔device round-trips sneaking in (NumPy in the hot loop) med Projection API takes/returns tensors only; single .cpu().numpy() per finished image; benchmark would expose regressions
Memory when merging many segments (points concatenated per color mode) low Existing pattern already re-concatenates per mode; fine for 2–5 segments, revisit only if whole-run merges appear

Open questions

Resolved 2026-07-07