Push dropped the correction JSONs — Windows MAX_PATH

2026-07-08 · 3dai.datasetsorter · AI3D-226 · generated by Claude

Summary

Push wrote the per-folder sidecar (the review marks) but silently dropped the per-tile *_vectors.adjusted.json correction files. On the shared remote, 260611_Abschnitt_3 had its sidecar but 0 of its 56 correction files. The cause is not in the sync logic — it is Windows' 260-character path limit (MAX_PATH). The atomic file writer finishes with os.replace(tmp, final), which fails with WinError 3 for any final path ≥ 260 chars when LongPathsEnabled = 0 (the machine's default). The sidecar path is short enough to land; the deeper correction paths are not — so only the corrections vanished.

The fix

DecisionChoiceWhy
How to beat MAX_PATH Prefix every filesystem path with the extended-length form \\?\ on Windows Bypasses the 260-char limit (up to ~32,767) without needing admin rights or a registry change on every colleague's PC
Where to apply it One helper store.long_path(), routed through load_json, atomic_write_json, atomic_write_bytes, cleanup_tmp, vectors.file_sha1 Single choke-point; every read/write/rename that could hit a deep tiles/ path is covered
Behaviour off Windows No-op (returns the path unchanged) Linux/macOS have no MAX_PATH; the change is invisible there
Guard against regression First unit tests in the repo (pytest) The push code had zero tests; now the sidecar-vs-correction contract is pinned

Walkthrough

1 · The symptom reported

A colleague pushed to the shared gDrive (N:) folder. The sidecars showed up, the marks were there — but a folder like Abschnitt 3 had none of its correction files. From the app it looked like "push works for sidecars but not the small JSON files."

Crucially, the push summary said "skipped N" — the writes were failing, but the failure was swallowed and shown only as a bland "skipped" count with no reason.

2 · Root cause MAX_PATH

Both the sidecar and the correction files are written by the same atomic writer in store.py. It writes to a short temp file, then atomically renames it into place:

fd, tmp = tempfile.mkstemp(prefix="~ds", dir=path.parent)
...write + fsync...
os.replace(tmp, path)      # <-- fails here if len(path) >= 260 on Windows

The old code kept the temp name short to stay under MAX_PATH — but that only protects the temp file. The final os.replace target is the real, long name, and that is what Windows rejects when LongPathsEnabled = 0.

Why only the corrections? Path length:

  • Sidecar: …/Abschnitt_3/dataset_review.sidecar.json ≈ 228 chars → under 260 → lands.
  • Correction: …/Abschnitt_3/tiles/<stem>_intensity_vectors.adjusted.json ≈ 291 chars → over 260 → fails.

The correction files sit one level deeper (tiles/) and have a long, descriptive name — just enough to cross the line.

3 · Evidence (measured on the real remote + Windows Python)

The app's own runtime is Windows Python 3.11 with LongPathsEnabled = 0. Reproduced directly: os.replace wrote fine at 245 chars and failed with WinError 3 at ≥ 260. Then, per folder on the shared remote:

FolderSidecar (chars)Adjusted in sidecarFiles on remotePath length
Abschnitt_1_short2344242 ✓256
Abschnitt_1_long2337474 ✓254
Abschnitt_4_5230~8080 ✓242
Abschnitt_3228560 ✗291
Abschnitt_2228231190 (old)281

The pattern is binary per folder: where the correction path stays under 260 (Abschnitt 1), every file lands; where it crosses 260 (Abschnitt 3 at 291), none do. The sidecar — always ~228 — lands in every case. That is exactly "sidecars sync, small JSONs don't."

4 · The code fix shipped

A single helper maps a path to Windows' extended-length form; every writer/reader goes through it.

def long_path(path):
    r"""Windows: map to the \\?\ extended-length form so file ops
    bypass the 260-char MAX_PATH limit. No-op off Windows."""
    s = os.fspath(path)
    if os.name != "nt" or s.startswith("\\\\?\\"):
        return s
    abspath = os.path.abspath(s)
    if abspath.startswith("\\\\"):            # UNC share
        return "\\\\?\\UNC\\" + abspath[2:]
    return "\\\\?\\" + abspath

# in the atomic writer:
os.replace(tmp, long_path(path))              # now survives long paths

Verified on the real Windows runtime: a 304-char path now round-trips (write → read → overwrite) where before it failed.

5 · Tests (first in the repo)

pytest was added to pyproject.toml; a new tests/ dir covers:

  • Push behaviour — corrections must land, not just the sidecar; the "folder left empty" regression; idempotent re-push; overwrite; conflict take-local / keep-dest; tags-config travel; delete propagation.
  • Long-path fix — long_path() mapping (drive / UNC / idempotent / off-Windows no-op) and the atomic writers surviving a > 260-char path.
uv run --with pytest pytest      # 20 passed

Delivered on branch AI3D-226-fix-push-longpath → PR #1.

Risks & caveats

RiskSeverityNote
\\?\ on the Google Drive (N:) mount medium Verified against a local Windows disk, not the shared drive (writing there is blocked for the agent). Should be confirmed with one throwaway write to N: before wide rollout.
Existing over-limit files already on the remote low Abschnitt_3's 56 corrections were never uploaded — re-push after the fix lands them. Folders whose files already exist (from earlier good pushes) are unaffected.
Read-side path checks (Path.exists()) still limited low For a > 260-char file that already exists, re-push treats it as "new" and overwrites — harmless for push (take-local policy, idempotent content).

Decisions & open questions