Final report — AI3D-338 retraining program

2026-07-11 · linebitmapsegmentation · 260624_confirmed_good · plan · round 1 · round 2

Summary

The program ran three rounds (7 Azure ML runs) on the 2× bigger 260624_confirmed_good dataset and hit its exit criterion: the bottleneck is demonstrably the data/label geometry, not the model, loss, schedule, or label stroke. Four independent levers each moved nothing outside the ±0.035 seed band: data volume (round 1), label-stroke realism and schedule length (round 2), background-dilution in the loss (round 3). The converged operating point on the 97-tile held-out test is coverage 0.92/0.94 · clDice 0.975 · F1 ~0.87 (recall-first pick), with no structural failures — no spurious lines, no broken centerlines — in any reviewed overlay sheet. Under the recall-over-precision objective, the segmentation front-end is ready to feed the masks→3D→CAD-centerline stage; further gains live in the annotation layer, not in training.

Ship pick recall-first

ft_r34_260624 — focal_tversky + resnet34, round-1 run, epoch-54 checkpoint (epoch054-0.8442.ckpt, AML job linebitmapseg_ft_r34_260624_20260711t050704z).

  • Best line-completeness of the program: coverage 0.916 solid / 0.938 dashed, clDice 0.975 — fewest missed/broken lines, the recall-first criterion.
  • Over-paint (area 1.15/1.13) is harmless downstream — skeletonization is width-invariant.
  • Overlays: recovers faint dashes the dice family drops; zero spurious lines in the reviewed sheets.

Promoted to DVC under data/03_experiments/ft_r34_260624/ (checkpoint + report + test metrics + TB scalars). For a marginally better pixel-F1 at slightly lower coverage, dice_focal_260624_fg (0.8770) is the calibration-side alternative.

All runs (held-out 97-tile test, scalar-4 ruler unless noted)

roundrunlevercov S/DclDiceF1 fgverdict
1ft_r34_260624 ship2× data 0.916/0.9380.9750.8678= Phase-D score → volume null
1dice_focal_2606242× data 0.875/0.8890.9710.8750= Phase-D score → volume null
2ft_r34_260624_s5d3stroke {5,3} 0.920/0.913*0.961*0.8390*ruler moved, model didn't → null
2dice_focal_260624_s5d3stroke {5,3} 0.883/0.858*0.962*0.8529*confirms stroke null
2ft_r34_260624_e9090 epochs 0.911/0.9240.9750.8703schedule null
3ft_r34_260624_fgfg-only loss 0.912/0.9330.9750.8691bg-dilution null (FT side)
3dice_focal_260624_fgfg-only loss 0.877/0.8980.9730.8770right sign, inside band → null

* native s5d3 ruler (labels changed); yardstick re-scores in the per-experiment reports. Seed std ±0.035 (Phase-D measurement).

The four nulls (why this is a data verdict)

  1. Data volume — +96% reviewer-ok tiles reproduced Phase-D scores to three decimals on a different, harder test set.
  2. Label-stroke realism — models paint whatever stroke they're taught; width-invariant metrics flat on both loss families. (Bonus discovery: the rasterizer's odd-width quantization means the "4 px" config always painted 5 px — solids matched physical paint from the start.)
  3. Schedule — +50% epoch budget: ΔF1 +0.0025.
  4. Background dilution — fg-only class mean: inside the band on both families (best-of-program F1 and +0.009 dashed coverage on dice_focal, right sign, not significant).

Combined with Phase D's 10-config loss/encoder sweep (flat 0.86–0.87), every model-side axis is exhausted. The residual defects visible in overlays are the known label-geometry classes: wide-paint gores a centerline vector can't represent, genuinely ambiguous faint/dark tiles, and the asphalt-edge-line ambiguity.

Data work next (in expected-value order)

  1. Gore / wide-paint representation — annotate lane-split wedges as area or multi-line geometry (or widen the vector model). The single recurring structural failure since Phase D.
  2. Faint-tile confirmation pass — review the dark/mottled tiles and either confirm or drop ambiguous paint; these cap dashed recall (the faint adjacent-carriageway dashes are the one recall gap overlays still show). Hint from round 2: thin dashed labels concentrated gradient enough to catch some of these — worth folding into the annotation guideline.
  3. Asphalt-edge rule — decide when the outer edge line is real (multi-lane sections) vs spurious; the last class of potential false CAD lines.
  4. More curved/ramp sections — the geographic tail is still sparse there.

Model-side, only two things remain worth carrying forward as defaults: classes: [1, 2] on dice-family losses (harmless-to-positive) and the per-class stroke support (useful for future label experiments; keep training at scalar 4).

Artifacts