Azure ML Training Deployment

One-time upload of the training data to the workspace default datastore, matching the repo-relative layout so --data-root is a pure prefix swap:

AZ="--account-name ai3dlfmlwpc011354919387 \
    --container-name azureml-blobstore-919dd9b5-… --auth-mode key"

az storage blob upload-batch $AZ \
  --destination-path linebitmapsegmentation/data/00_external/260611_topdown_tiles \
  --source data/00_external/260611_topdown_tiles
# same for data/01_interim/260611_topdown_tiles_markings
# and data/02_processed/260611_confirmed_good (review sidecars + adjusted vectors)

Write a small scripts/azure/sync_data.sh wrapping this (idempotent re-sync when DVC data changes) and drop a MANIFEST.txt (dvc hash of each mirrored .dvc file) next to the blobs so it's always clear which DVC revision the mirror reflects. ~5 GB upload, done once.

Exit check: az storage blob list --prefix linebitmapsegmentation/data/ shows the expected tile counts.

New environments/train/ following the orchestrator two-file convention:

# environments/train/environment.yml
name: linebitmapseg_train
version: 1
image: mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu22.04
conda_file: conda.yml   # generated, gitignored

# environments/train/conda.template.yml (committed)
name: linebitmapseg-train
channels: [conda-forge]
dependencies:
  - python=3.12
  - pip
  - xorg-libx11, xorg-libxext, xorg-libxrender, libglib, libgl   # cv2 runtime
  - pip:
      - --extra-index-url https://download.pytorch.org/whl/cu124
      - --extra-index-url https://${NEXUS_USER}:${NEXUS_PASS}@nexus.iolabs.ch/repository/pypi-private/simple/
      - torch==2.6.0+cu124        # T4 driver ceiling — proven in orchestrator s4/s7
      - segmentation-models-pytorch, albumentations, lightning>=2.2
      - tensorboard, torchmetrics, pyyaml, opencv-python-headless
      - iolabs-image-analyzer-line-bitmap-inference>=0.1.1

scripts/azure/render_env.sh substitutes the creds from env vars into the gitignored conda.yml, then:

az ml environment create --file environments/train/environment.yml \
  -g AI3D-rg -w ai3d-lf-mlw-pc-01

Verify first that iolabs-common resolves from Nexus via pip alone (locally it's a path dep ../3dai.common; the inference package pulls it transitively — orchestrator envs already install Nexus packages this way, but confirm this specific chain with a local pip install --dry-run against the index before burning an AML image build).

Exit check: environment build succeeds in AML (check the build log in Studio); a trivial job on the GPU cluster prints torch.cuda.is_available() == True.

Two new optional CLI args (Codex writes, Fable reviews; TDD per house rules):

• --data-root DIR — re-roots every relative data/… path in data.pairs / val_pairs / test_pairs onto DIR (pure prefix join in HarnessConfig post-load; absolute paths left untouched).
• --log-dir DIR — overrides train.log_dir (default runs) so outputs land on the AML output mount.
• --num-workers N — override data.num_workers; configs say 8 but NC4as_T4_v3 has only 4 vCPUs.

Exit check: local uv run --extra ml python scripts/train.py --config configs/unet_baseline.yaml --data-root . --log-dir /tmp/x --fast-dev-run passes; existing behaviour without the flags is byte-identical (unit test on config re-rooting).

configs/jobs/train.yaml (command job, parameterised by experiment config):

$schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
type: command
code: ../..                      # repo snapshot (see .amlignore)
compute: azureml:Nvidia-TeslaT4-2x-GPU-Compute
environment: azureml:linebitmapseg_train:1
inputs:
  data:
    type: uri_folder
    mode: ro_mount
    path: azureml://datastores/workspaceblobstore/paths/linebitmapsegmentation/data
  config: configs/unetpp_r34_recipe2.yaml
outputs:
  runs: { type: uri_folder, mode: rw_mount }
command: >-
  python scripts/train.py --config ${{inputs.config}}
  --data-root ${{inputs.data}} --log-dir ${{outputs.runs}}
  --num-workers 4
experiment_name: linebitmapseg_training

Add .amlignore (exclude data/, runs/, .venv/, .dvc/cache, docs/, previews) so the code snapshot is a few MB, and scripts/azure/submit_train.sh <config> [overrides…] wrapping az ml job create --file configs/jobs/train.yaml --set inputs.config=… name=…$(date -u +%Y%m%d%H%M%S).

Exit check: a --fast-dev-run job completes green on the GPU cluster; downloaded named-outputs/runs/ contains the TensorBoard event file.

Submit the current experiment queue (e.g. unetpp_r34_recipe2, the confirmed-good splits arm, seed-variance arms) — each as its own job; the cluster runs up to 12 concurrently. Daily loop:

az ml job list --query "[?experiment_name=='linebitmapseg_training']…" -o table
az ml job stream --name <job>                     # live stdout
az ml job download --name <job> --download-path … --all
tensorboard --logdir runs                          # merged with local baselines

Downloaded run dirs drop into the local runs/ tree (same experiment naming), so Azure and local arms compare in one TensorBoard. Best checkpoints that matter long-term get dvc add-ed under data/03_experiments/ like the existing sweeps.

Exit check: one full config trains to early-stop on Azure with val/f1 within noise of the local reference run.

• MLflow logger alongside TensorBoard — AML tracks metrics natively in Studio, enabling az ml job-level metric queries without downloading events.
• AML sweep jobs for hyper-parameter search (lr, tversky α, stroke px) instead of hand-written config fans like sweep2d_*.
• Low-priority tier on a second cluster for cheap long sweeps (checkpoint/resume already works via Lightning's last.ckpt).
• Bigger SKU quota (NC8as_T4_v3 or A100) if the 4-vCPU dataloader measurably starves the T4.