Handoff: better runtime estimates for planned harness experiments
A management presentation is being built with a timescale slide: how long does a 30–50-epoch training run over ~50 corridor segments take? The current numbers are extrapolated from your 3-epoch smoke. You ran the actual training and know the pipeline internals — check the extrapolation below, correct it where you can, and answer the open questions. Write your answers to
/tmp/harness-runtime-answers-20260813.md.
What the presentation currently claims (check this)
Anchors measured from your artifacts (runs/E9/version_5 train, version_6 eval, train.log, provenance timestamps):
- Steady-state SpUNet epoch on 3 tiles (~1.9 M decimated pts, batch 1, RTX 3080 Ti Laptop): ~16–20 s/epoch after a ~105 s one-time warmup; total 3-epoch train run 5 m 02 s.
scripts/evaluate.py: 71 m 33 s wall for 3 val tiles (single-core; bootstrap 2000 + viz dominated).- Canonical prepare: 2 m 43 s for 5.1 M points.
Extrapolation used in the deck (linear in tile count):
| Scenario | Epoch time | 30 epochs | 50 epochs |
|---|---|---|---|
| 50 segments (~150 tiles), decimated 6.67 cm, local 3080 Ti | ~15 min | ~7.5 h | ~12.5 h |
| Same on Azure single-node 2×T4 (assumed ×1.5–2 slower) | ~22–30 min | ~11–15 h | ~19–25 h |
| Full-resolution at 3 cm grid (assumed ×3–5 training points) | — | ~1–2 days local | ~2–5 days T4 |
Open questions (highest value first)
- Linear scaling: is epoch wall time ≈ linear in tile count at batch_size 1, or do dataloader workers / RAM (25 GB local WSL) / caching break that at ~150 tiles? Is a 50-segment local Pointcept run even feasible in 25 GB RAM, or does it force the AML cluster regardless of GPU speed?
- Full-res multiplier: once E2 unlocks full-resolution data at a 2–5 cm working voxel, what is the realistic training-point multiplier vs the 6.67 cm decimated smoke data? Is ×3–5 fair at 3 cm?
- T4 factor: better estimate than "×1.5–2 vs 3080 Ti Laptop" for spconv SpUNet fp16? Does the single-node 2×T4 job actually use both GPUs, or one?
- E1 SPT: expected wall time for the SPT pilot (partition preprocessing + 30–50 epochs) at 50-segment scale on the T4 cluster. Literature anchor in the vault is S3DIS ≈ 3 GPU-h; anything better from the SPT configs/docs you pinned?
- Per-epoch validation cost: at 50 segments the val set grows too — does per-epoch val eval stay negligible or start to dominate?
- evaluate.py scaling: 71 min for 3 tiles single-core — linear in corridors? Any estimate of the post-vectorization cost (you flagged it as a vectorization candidate alongside the E2 oracle treatment)?
Key artifacts
runs/E9/version_5/train.log,provenance*.json— epoch timings, hardware block.runs/E9/version_6/— eval wall time,checkpoint_selection.json,reports/remap_report.json(point counts per tile).configs/e01_spt_pilot.yaml,configs/e09_pointcept_challengers.yaml,configs/dev/a1_pointcept_smoke.yaml.- Memory notes:
~/.claude/projects/-home-miro-dev-3dai-iolabs-pointcloud-mlsegmentation/memory/harness-buildout-state.mdandpointcept-local-env.md. - The presentation being refined:
docs/pages/3d-harness-presentation-20260813.html(timescale section) — update its table directly if your numbers differ materially, keeping the "extrapolated, not measured" caveat unless you actually measure.
Constraints
- Do not launch long training runs without Miro's go-ahead; a short local timing probe (e.g. a few epochs on more tiles, if data exists) is fine if cheap.
- Never touch
promotion_testdata without explicit promotion authorization arguments. - The presentation is public-URL: no client names, dataset codenames, Azure account/cluster identifiers, or internal URLs in anything you write into it.