Guardrail experiments — run-all runbook

Work from the repo root. The repo is uv-managed; iolabs-ml-harness comes from the private Nexus index (credentials must already be configured — if uv sync fails on auth, stop and report rather than fiddling).

cd <repo root>   # branch worktree-guardrail-experiments (or master after merge)
uv sync --extra ml --extra detectors --extra dev
uv run --extra ml python -c "import torch; print('cuda', torch.cuda.is_available())"
uv run --extra ml --extra dev pytest -q     # must be all green before training

Expected: cuda True; full suite passes. runs/ and data/detection*/ are gitignored — never commit them.

Each config differs from configs/unet_baseline.yaml in exactly one variable (loss or architecture). Run sequentially; capture the [train] best checkpoint: … line each run prints at the end — you need those paths in Phase 4.

for c in unet_baseline unet_focal_tversky unet_generalized_dice \
         unet_dice_focal_cldice unetplusplus_baseline fpn_baseline; do
  uv run --extra ml python scripts/train.py \
      --config configs/$c.yaml --max-epochs 15 \
      2>&1 | tee runs/train_$c.console.log || exit 1
done
grep -h "best checkpoint" runs/train_*.console.log

Monitor (optional): uv run --extra ml tensorboard --logdir runs. Checkpoints land under runs/<experiment>/…/checkpoints/; the console line is authoritative. If a run early-stops after a few epochs that is expected behavior (val/loss patience 4), not a failure.

Both exports read the split from the baseline config, so detector runs see exactly the segmentation tiles. COCO gets --pad-to 384 (RF-DETR), YOLO stays at 256.

uv run --extra ml python scripts/export_detection.py \
    --config configs/unet_baseline.yaml --format coco \
    --out data/detection_coco --pad-to 384
uv run --extra ml python scripts/export_detection.py \
    --config configs/unet_baseline.yaml --format yolo \
    --out data/detection_yolo

Expected output: 64 train / 9 val tiles (seed 20260707, crop 256) for both.

Both wrappers train outside the Lightning harness (their libraries own the loop) but on the Phase 2 exports. First run downloads pretrained weights — needs network. Do not commit any downloaded *.pt/*.pth.

uv run --extra ml --extra detectors python scripts/train_rfdetr.py \
    --dataset data/detection_coco --out runs/rfdetr_nano --epochs 30

uv run --extra ml --extra detectors python scripts/train_yolo.py \
    --dataset data/detection_yolo --out runs/yolo26n --epochs 30

RF-DETR checkpoints land in runs/rfdetr_nano/ (checkpoint_best_ema.pth preferred). YOLO writes runs/yolo26n/agpl_experiment_only/weights/best.pt. The YOLO wrapper prints an AGPL experiment-only banner — keep that labeling in anything you report.

Gotcha: each --seg NAME=CKPT NAME must equal the config file stem (configs/<NAME>.yaml) — the evaluator rebuilds the net from that config before loading weights. Use the best-checkpoint paths captured in Phase 1.

uv run --extra ml --extra detectors python scripts/evaluate.py \
  --config configs/unet_baseline.yaml \
  --seg unet_baseline=<ckpt> \
  --seg unet_focal_tversky=<ckpt> \
  --seg unet_generalized_dice=<ckpt> \
  --seg unet_dice_focal_cldice=<ckpt> \
  --seg unetplusplus_baseline=<ckpt> \
  --seg fpn_baseline=<ckpt> \
  --rfdetr rfdetr_nano=runs/rfdetr_nano \
  --yolo yolo26n=runs/yolo26n/agpl_experiment_only/weights/best.pt

Output: one markdown table on stdout and in runs/evaluation.md — box AP@50 / P / R / F1 for every run; pixel IoU/F1 (guardrail class) for the seg rows; the YOLO row is labeled AGPL experiment-only. If the RF-DETR run used --model small or a non-384 resolution, pass matching --rfdetr-model/--rfdetr-resolution to evaluate.py.

Create docs/experiments/2026-07-07-synthetic-baseline-matrix.md containing: the evaluation table verbatim, the exact commands used (epoch counts included), per-run best-checkpoint metric from the console logs, and two mandatory caveats — (1) synthetic harness-proof round, real cross-section data pending; (2) YOLO26 row is AGPL, experiment-only. Commit only that file (plus a short “Experiments” pointer in CLAUDE.md if not already present). Never commit runs/, data/detection*/, or weights.

uv run --extra ml --extra dev pytest -q   # still green
git add docs/experiments/2026-07-07-synthetic-baseline-matrix.md
git commit -m "docs: record synthetic experiment-matrix round (models x losses x detectors)"

64 training tiles is tiny for detectors. For a larger round, change synthetic_tiles to the same value (e.g. 256) in all six configs, re-export both detection datasets, and rerun Phases 1–5. The split stays common as long as every run uses the same synthetic_tiles/synthetic_seed/ val_fraction. Do not mix runs from different tile counts in one table.