Iterative Azure retraining on 260624_confirmed_good

2026-07-11 · AI3D-226 · linebitmapsegmentation

Summary

Goal: run iterative rounds of U-Net training experiments on Azure ML against the new 260624_confirmed_good dataset (485 reviewer-OK tiles, 291/97/97 train/val/test — roughly 2× the 248-tile split Phase D exhausted), writing a report per experiment and per round, and proposing the next round from the evidence. Stop when the held-out test set is effectively solved (metrics saturated, no structural failures in overlays) or when the failure modes are demonstrably data/label-geometry again — the Phase-D verdict at test F1 ≈ 0.87, clDice ≈ 0.97.

Round 1 is in flight: ft_r34_260624 (Phase-D CAD pick, focal_tversky + resnet34) and dice_focal_260624 (sweep2 baseline anchor, dice_focal + resnet18), hyperparameters frozen at their Phase-D values so the delta isolates what the new data buys.

Key decisions

Train on Azure ML, evaluate + report locally. Jobs go through the existing wiring (scripts/azure/submit_train.sh → configs/jobs/train.yaml → run_train.sh, T4 cluster). The AML job only trains; evaluate.py (held-out test metrics, dual-checkpoint) and make_report.py (control curves + report.md) run locally on the downloaded run output, reusing the established sweep tooling and report format.
Round 1 changes exactly one variable: the data. Both configs are byte-identical to their Phase-D counterparts (vector labels @ 4px, seed 1337, lr 3e-4, batch 16, crop 512, early-stop patience 10) except the split paths. Two anchors — the recall-tilted CAD pick and the precision-leaning baseline — reproduce the two ends of the known precision/recall axis.
Later rounds pull levers Phase D already prioritized, chosen by round-1 failure modes, not speculation: per-class label stroke (solid 5 px / dashed 4 px — the measured paint widths), Tversky α dial, encoder depth, schedule length.
Harness code changes are delegated to Grok 4.5 in Cursor (per Miro's instruction), e.g. the per-class label_stroke_px support; Fable reviews and integrates.
Comparability caveat is explicit: the test set changed (97 tiles vs Phase-D's 50), so cross-leaderboard deltas are directional. Within-round comparisons are exact; a same-test-set re-eval of the old shipped checkpoint is the fallback if results are ambiguous.
Everything ships via git: configs and reports on branch worktree-azure-retrain-260624, draft PR at the end; checkpoint artifacts get DVC-promoted per the existing publish_experiment.sh pattern.

Phases

Phase 0 — Wiring & data done

Read Phase-D / sweep2 reports; confirmed the "data is the bottleneck" verdict and the two reference configs.
Dereferenced the symlink split (872 MB, 971 files) and mirrored it to the workspace blobstore under linebitmapsegmentation/data/data/02_processed/260624_confirmed_good; added the dir to sync_data.sh.
Validated both configs through HarnessConfig.from_yaml.
Fixed configs/jobs/train.yaml for the newer az ml CLI (literal inputs as plain scalars; non-empty extra_args default).

Round 1 — Data effect at frozen hyperparameters running

Experiment	Loss / encoder	Role	Phase-D reference (old 50-tile test)
`ft_r34_260624`	focal_tversky / resnet34	CAD pick — recall + topology	F1 0.868 · clDice 0.975 · cov 0.93/0.94
`dice_focal_260624`	dice_focal / resnet18	Baseline anchor — lowest FP, best calibration	F1 0.871 · area ≈ 1.0

AML jobs linebitmapseg_ft_r34_260624_20260711t050704z and linebitmapseg_dice_focal_260624_20260711t051016z; a status monitor polls until terminal. Per experiment afterwards: download runs/ output → evaluate.py → make_report.py → per-experiment report (metrics tables, control curves, overlay read). Then a round report answering:

Did 2× data move test F1 / clDice / coverage at all (vs seed std ≈ 0.035)?
Which Phase-D failure modes survived (gore over-paint, asphalt-edge FP, faint tiles), judged from the overlays?
Is the precision/recall split between the two anchors still the same shape?

Round 2 — Levers picked from round-1 evidence conditional design

Candidate levers, in Phase-D priority order — run the 2–4 that match observed failure modes:

Lever	When it's the right move	Cost
Per-class label stroke (solid 5 / dashed 4 px) — needs small harness change (Grok 4.5 / Cursor)	Tightness/IoU capped with paint visibly wider than label; under/over-paint disagreement between the two anchors persists	code + 2 runs
Tversky α ∈ {0.2, 0.3, 0.4} on the new data	Recall (coverage) is the shortfall; α is a pure precision↔recall dial	2–3 runs
Encoder r34 → r50 / unet++	Only if new-data curves suggest capacity finally binds (train F1 ≫ val F1 gone)	1–2 runs
Longer schedule / more crops per tile	Val curves still climbing at epoch 60 on 2× data	1 run
Seed repeats (s7, s42)	Round-1 deltas within ±0.035 — establish the new noise floor before believing any ordering	2 runs

Round 3+ — Iterate to the stop criterion

Repeat design → run → report. Stop and declare the data the bottleneck when:

metric spread across levers stays inside the seed band, and
remaining test-overlay defects are the known label-geometry classes (wide-paint gores, asphalt-edge ambiguity, faint tiles) rather than model errors, or
coverage/clDice saturate (≥0.97/≥0.99) with no structural breaks — "test set effectively perfect" for the width-invariant centerline goal.

Either exit produces a final program report with the recommended ship checkpoint and the concrete next data work.

Ship — reports, PR, DVC

Per-experiment + per-round reports under docs/experiments/ (git) — same structure as the sweep2 reports.
Draft PR from worktree-azure-retrain-260624: configs, job-YAML fix, sync_data.sh, reports, this plan.
Best checkpoints DVC-promoted to data/03_experiments/ from the main checkout (pointer files committed separately, as before).

Risks

stats Seed noise (±0.035 F1) can eat the data gain. Mitigation: two independent anchors per round; seed repeats before trusting any ordering; judge overlays and coverage/clDice, not headline F1 alone.
data New-tile label quality is unaudited by this program. The 260611 batch needed a repair pass (40% solid-as-dashed mislabels) — the 260624 "correction_2" batch is assumed fixed; a residual mislabel pocket would depress apparent model performance. Overlay review flags this early.
infra Test-set change breaks strict leaderboard comparability (97 vs 50 tiles). Directional reads only; fallback: re-evaluate the old shipped checkpoint on the new test split.
infra az ml CLI drift already bit once (schema rejection). Job YAML fixed; further drift is possible on env or download paths.
infra Single-T4 nodes; jobs may serialize if the cluster max is 1 node — slows rounds but changes nothing else.

Open questions

What counts as "perfect" on test? Working definition: coverage and clDice saturated with zero structural defects in overlays (pixel-F1 1.0 is unreachable against a rasterized 4-px vector label by construction). Confirm or tighten.
Should the old 50-tile Phase-D test subset be re-scored each round for an apples-to-apples series, at the cost of an extra eval per experiment?
DVC promotion cadence — every round's best checkpoint, or only the final ship candidate?