Final report — AI3D-338 retraining program
Summary
The program ran three rounds (7 Azure ML runs) on the 2× bigger 260624_confirmed_good dataset and hit its exit criterion: the bottleneck is demonstrably the data/label geometry, not the model, loss, schedule, or label stroke. Four independent levers each moved nothing outside the ±0.035 seed band: data volume (round 1), label-stroke realism and schedule length (round 2), background-dilution in the loss (round 3). The converged operating point on the 97-tile held-out test is coverage 0.92/0.94 · clDice 0.975 · F1 ~0.87 (recall-first pick), with no structural failures — no spurious lines, no broken centerlines — in any reviewed overlay sheet. Under the recall-over-precision objective, the segmentation front-end is ready to feed the masks→3D→CAD-centerline stage; further gains live in the annotation layer, not in training.
Ship pick recall-first
ft_r34_260624 — focal_tversky + resnet34, round-1 run, epoch-54
checkpoint (epoch054-0.8442.ckpt, AML job
linebitmapseg_ft_r34_260624_20260711t050704z).
- Best line-completeness of the program: coverage 0.916 solid / 0.938 dashed, clDice 0.975 — fewest missed/broken lines, the recall-first criterion.
- Over-paint (area 1.15/1.13) is harmless downstream — skeletonization is width-invariant.
- Overlays: recovers faint dashes the dice family drops; zero spurious lines in the reviewed sheets.
Promoted to DVC under data/03_experiments/ft_r34_260624/ (checkpoint +
report + test metrics + TB scalars). For a marginally better pixel-F1 at slightly
lower coverage, dice_focal_260624_fg (0.8770) is the calibration-side
alternative.
All runs (held-out 97-tile test, scalar-4 ruler unless noted)
| round | run | lever | cov S/D | clDice | F1 fg | verdict |
|---|---|---|---|---|---|---|
| 1 | ft_r34_260624 ship | 2× data | 0.916/0.938 | 0.975 | 0.8678 | = Phase-D score → volume null |
| 1 | dice_focal_260624 | 2× data | 0.875/0.889 | 0.971 | 0.8750 | = Phase-D score → volume null |
| 2 | ft_r34_260624_s5d3 | stroke {5,3} | 0.920/0.913* | 0.961* | 0.8390* | ruler moved, model didn't → null |
| 2 | dice_focal_260624_s5d3 | stroke {5,3} | 0.883/0.858* | 0.962* | 0.8529* | confirms stroke null |
| 2 | ft_r34_260624_e90 | 90 epochs | 0.911/0.924 | 0.975 | 0.8703 | schedule null |
| 3 | ft_r34_260624_fg | fg-only loss | 0.912/0.933 | 0.975 | 0.8691 | bg-dilution null (FT side) |
| 3 | dice_focal_260624_fg | fg-only loss | 0.877/0.898 | 0.973 | 0.8770 | right sign, inside band → null |
* native s5d3 ruler (labels changed); yardstick re-scores in the per-experiment reports. Seed std ±0.035 (Phase-D measurement).
The four nulls (why this is a data verdict)
- Data volume — +96% reviewer-ok tiles reproduced Phase-D scores to three decimals on a different, harder test set.
- Label-stroke realism — models paint whatever stroke they're taught; width-invariant metrics flat on both loss families. (Bonus discovery: the rasterizer's odd-width quantization means the "4 px" config always painted 5 px — solids matched physical paint from the start.)
- Schedule — +50% epoch budget: ΔF1 +0.0025.
- Background dilution — fg-only class mean: inside the band on both families (best-of-program F1 and +0.009 dashed coverage on dice_focal, right sign, not significant).
Combined with Phase D's 10-config loss/encoder sweep (flat 0.86–0.87), every model-side axis is exhausted. The residual defects visible in overlays are the known label-geometry classes: wide-paint gores a centerline vector can't represent, genuinely ambiguous faint/dark tiles, and the asphalt-edge-line ambiguity.
Data work next (in expected-value order)
- Gore / wide-paint representation — annotate lane-split wedges as area or multi-line geometry (or widen the vector model). The single recurring structural failure since Phase D.
- Faint-tile confirmation pass — review the dark/mottled tiles and either confirm or drop ambiguous paint; these cap dashed recall (the faint adjacent-carriageway dashes are the one recall gap overlays still show). Hint from round 2: thin dashed labels concentrated gradient enough to catch some of these — worth folding into the annotation guideline.
- Asphalt-edge rule — decide when the outer edge line is real (multi-lane sections) vs spurious; the last class of potential false CAD lines.
- More curved/ramp sections — the geographic tail is still sparse there.
Model-side, only two things remain worth carrying forward as defaults:
classes: [1, 2] on dice-family losses (harmless-to-positive) and the
per-class stroke support (useful for future label experiments; keep training at
scalar 4).
Artifacts
- Branch
worktree-azure-retrain-260624(Bitbucket): 7 configs, harness changes (per-class stroke, fg-only losses — both Grok 4.5 implementations, reviewed), per-experiment reports underdocs/experiments/round{1,2,3}_260624/, round reports + this page underdocs/plans/. - Ship checkpoint DVC-promoted:
data/03_experiments/ft_r34_260624/(pointer staged in the main checkout, commit pending). - All 7 AML runs in workspace
ai3d-lf-mlw-pc-01, experimentlinebitmapseg_training; TensorBoard:scripts/azure/tb_azure.sh <jobs>.