Teaching a Computer to Color Highway Point Clouds

2026-07-10 · research brief for a non-ML reader · Kirioll / highway digitization

Summary

We researched how to build our own version of Pointly's highway classifier — software that takes a raw laser scan of a highway and automatically labels every point: this is road, this is a guardrail, this is a tree. Verdict:

  • Feasible in ~2 months with one good graphics card and tools we already know (Python, PyTorch-based open-source frameworks).
  • The algorithm is not the hard part — excellent free, commercially usable models exist. The training data is. Most famous public datasets legally forbid commercial use; we verified which ones don't.
  • Classes nobody publishes data for (guardrails, drains, noise barriers) we must label ourselves — but research shows labeling ~5% of the data gets ~95% of the quality[1][2], so a few km is enough. That self-made data is also exactly what makes the result ours, the same moat Pointly has.

What the system does

A survey car drives the highway and records hundreds of millions of 3D points — like an extremely dense, uncolored 3D sketch. Today, turning that into 2D CAD plans means a person manually deciding what each region is. The classifier does the first, most tedious step automatically: it assigns every point a class. Press the button to see the difference:

Simplified cross-section of one highway corridor slice, drawn as actual points. Left: what the scanner delivers. Right: what the classifier produces — the same points, each with a class label. (Illustration, not real data.)

Downstream, our existing geometry code turns those labeled points into the CAD deliverables — edge lines, splines, symbols. The full pipeline:

🛣️Laser scanLAS files, ~billions of points
→
✂️Cut into tiles~50 m slices along the road
→
🧠Neural networklabels every point, on GPU
→
🧩Merge + cleanstitch tiles, remove vehicles, sanity rules
→
📐Vectors & planssplines, symbols, 2D plans (existing code)
The neural network replaces only the manual "what is this point?" step. Everything around it is normal engineering we already do (tiling, merging, geometry, Docker workers).

Pointly — the commercial benchmark — classifies up to 90 km of highway per hour this way, trained specifically on German and Austrian highways[3]. Nothing about it is magic: it's a standard recipe of a good open model + proprietary labeled data + solid pipeline engineering.

One spec, three different problems

Our target element list (the HBW 2D-plans spec, ~30 element types) looks like one big wish list, but technically it splits into three buckets — and only the first one needs the neural network:

Bucket 1 · neural network

Surfaces & structures

Big, texture-like regions the network classifies point by point:

  • road surface, other ground, sidewalks
  • guardrails, fences, walls, noise barriers
  • signs & gantries, poles & lights
  • vegetation, slopes, trenches, bridges
Bucket 2 · object detection

Small point objects

Things with only 20–100 points each — too small for point-wise classification. Found instead as objects on top-down intensity images:

  • manholes / shafts, drains
  • emergency call boxes, km signs
  • individual trees
Bucket 3 · pure geometry

Vector deliverables

No training data needed — computed from the labeled points with classical geometry:

  • center/edge line splines, road edges
  • slope edges, trench bottoms, tree lines
  • elevation annotations, cadastre refs

Our current classical road-marking extraction (intensity thresholding) stays — for markings it is still the standard, reliable approach in the literature[4][5], and the results get fused at the end.

How the machine learns — and why we don't need much labeled data

Think of the network as a junior draftsman. You don't teach them from zero: they arrive knowing what roads generally look like (pre-training on public datasets), you show them a few km of our highways done correctly (fine-tuning), and then they work while you only correct mistakes, which they learn from (the correction loop, "active learning"[6]):

📚1 · Pre-trainpublic data: general road knowledge
→
🎯2 · Fine-tune2–5 km of our own labeled corridor
→
🔁3 · Correct & retrainfix only errors on new sections; 2–3 rounds
→
✅Productionmodel tuned to our scanners & roads

The striking research result: dense labeling of everything is unnecessary. Peer-reviewed studies consistently show you keep almost all quality with a tiny fraction of labels:

Quality kept vs. labeling effort (% of fully-labeled model quality)

Label everythingthe brute-force baseline
100%
Rough scribbles on 8% of pointsScribbleKITTI, CVPR 2022 [2]
95.7%
Smart selection, <5% labeledLiDAL active learning, ECCV 2022 [1]
95%
Machine pre-labels, human correctsLATTE / PALF studies [7][8]
same quality, ~6× faster
One honest caveat: no published study covers our exact case (a few km of survey-grade highway data) — these numbers are from city/automotive datasets, so we verify on our own first segment.

Practically: annotate a handful of representative kilometres once (about 1–2 person-weeks with a good tool and machine pre-labels — pre-labeling alone speeds annotation ~6×[7][8]), and the correction loop does the rest.

The data problem — the real finding of this research

Everyone assumes the famous academic datasets are free to use. They are — for research only. For a commercial product, the license is the gate. We verified each candidate (2026-07-10):

✔ Commercially usable safe

HelixNet — 10 billion labeled points, CC-BY-4.0 · verified on Zenodo [9] PandaSet — 37 label classes · "academic and commercial use" per official site [10] ZOD (Volvo/Zenseact) — lane markings, signs · CC-BY-SA-4.0, commercial explicitly OK [11] National aerial scans — Switzerland ✔ verified; NL/German states very likely · free labels for ground, vegetation, buildings, bridges [12][13][14] Our own unlabeled scans — self-supervised pre-training · zero license risk, perfectly matches our domain [15]

⚠ Unclear — check first verify

DALES — aerial, ready-made configs · license page unreachable [16] nuScenes · commercial program exists — would need to ask Motional [17] A2D2 (Audi) — German roads · "NoDerivatives" clause, needs legal review [18] Rail datasets (OSDaR23 …) · guardrail/pole-like classes, licenses unverified [19]

✘ Forbidden for commercial use blocked

Toronto-3D · best class match — CC-BY-NC [20] Paris-Lille-3D · CC-BY-NC-ND [21] SemanticKITTI, KITTI-360 · CC-BY-NC-SA (even the German 73 km one) [22][23] Waymo, Argoverse 2, ONCE · Argoverse terms explicitly forbid training commercial models [24][25][26]
The key insight: guardrails, noise barriers, drains and call boxes appear in no commercially usable public dataset. Whoever labels their own few kilometres owns something nobody can download. That — not the algorithm — is Pointly's moat, and it can be ours: we already have the raw scans.

What it takes

1 GPU
RTX 4090-class card is enough — the recommended model has official configs even for 11 GB cards [27]
212k
parameters in the recommended model (Superpoint Transformer) — tiny by AI standards, trains in hours not weeks [28]
2–5 km
of our own corridor to label once, ~1–2 person-weeks with machine pre-labeling [7][8]
~8 weeks
to a production-usable first version, per the plan below
90 km/h
Pointly's throughput benchmark [3] — achievable with our Docker/GPU workers (a single GPU segmented 18M points in ~10 s in the paper [29])
0 CHF
software licenses — recommended frameworks are MIT open source (Python/PyTorch, config-driven) [27][30]

Model choice in one sentence: start with Superpoint Transformer[28] (built exactly for huge survey scans, runs on one GPU, MIT license) and keep the heavyweight Point Transformer V3[31] as the accuracy benchmark — both are Python frameworks whose configs read like normal engineering code, no PhD required.

Azure: what we already have vs. the one thing to request

ComputeStatusWhat it's good for
Existing Azure ML cluster — 12 nodes × Standard_NC4as_T4_v3 (1× Tesla T4 16 GB, 4 vCPU, 28 GB RAM; $0.66/h per node, autoscales 0→12) have it Covers most of the plan already. Superpoint Transformer fine-tuning fits (the framework ships official 11 GB-GPU configs[27]; expect ~3–5× slower than an A100, still hours per run). And for production inference the 12 autoscaling workers are ideal — one tile per node is exactly the Pointly-style parallelism. Caveat: the CPU-heavy preprocessing should run as its own parallel step (only 4 vCPU per node[34]).
One training node — Standard_NC24ads_A100_v4 (1× A100 80 GB), ideally as Spot request Needed only for the Point Transformer V3 accuracy-benchmark experiments: PTv3's fast mode requires FlashAttention[30], which needs an Ampere-or-newer GPU[32] — the T4 generation predates that, so PTv3 on T4 falls back to a slow mode not worth training on. Cost frame: the whole T4 cluster at full tilt is $7.92/h, so one A100 Spot node[33] for training runs is often cheaper per finished experiment.
Software side is unexciting on purpose: an Azure ML GPU environment or NVIDIA PyTorch container (Ubuntu 20.04+, CUDA ≥ 11.6, PyTorch ≥ 1.12) plus pip install of the two frameworks. LAS tiles live in Blob storage exactly like our existing ACC cache; preprocessed tiles get staged to the node's local disk. Multi-GPU training (PTv3's official 4-GPU outdoor configs) is not part of this plan.

The 8-week plan

W1
W2
W3
W4
W5
W6
W7
W8
Benchmark & class set (incl. Pointly free-tier test, license decision)
Annotate sample + first fine-tuning
Correction loop + tiling/merge pipeline
Post-processing, vectors, workflow integration
Same skeleton as the original memo, with two amendments from this research: the license decision moved into week 1 (don't build on forbidden data "temporarily"), and the single-GPU-friendly model goes first.

Honest risks & open questions

Mini-glossary (the only 6 terms you need)
  • Point cloud — millions of measured 3D points (x, y, z + reflectivity), the raw output of a laser scanner.
  • Semantic segmentation — assigning a class label to every point. The colored picture at the top.
  • Pre-training / fine-tuning — first teach general road knowledge on public data, then specialize on a small sample of our own.
  • Active learning — the model asks a human to label only what it's unsure about; each round of corrections retrains it.
  • Intensity — how strongly a surface reflects the laser. Paint reflects far more than asphalt — this is why road markings are easy for classical code.
  • mIoU — the standard 0–100 quality score for segmentation (overlap between predicted and true regions, averaged over classes). Good highway models score ~70–80 on public benchmarks.

References

  1. Hu et al., LiDAL: Inter-frame Uncertainty Based Active Learning for 3D LiDAR Semantic Segmentation, ECCV 2022 — arXiv:2211.05997
  2. Unal, Dai, Van Gool, Scribble-Supervised LiDAR Semantic Segmentation (ScribbleKITTI), CVPR 2022 — arXiv:2203.08537
  3. Pointly GmbH, Point Cloud Classifier for Highways, 03.10.2023 — pointly.ai (accessed 2026-07-10)
  4. Road-marking segmentation from LiDAR reflectivity (adaptive Otsu thresholding, highway experiments) — arXiv:2211.01105
  5. Lane-level map generation via multi-region Otsu thresholding on LiDAR intensity — arXiv:2101.05066
  6. Wu et al., ReDAL: Region-based and Diversity-aware Active Learning for Point Cloud Semantic Segmentation, ICCV 2021 — arXiv:2107.11769
  7. Wang, Wu, Wu, Keutzer, LATTE: Accelerating LiDAR Point Cloud Annotation, IEEE ITSC 2019 — arXiv:1904.09085
  8. Zhang et al., PALF: Pre-Annotation and Camera-LiDAR Late Fusion for the Easy Annotation of Point Clouds — arXiv:2304.08591
  9. Loiseau, Aubry, Landrieu, Online Segmentation of LiDAR Sequences (HelixNet), ECCV 2022 — arXiv:2206.08194 · license: Zenodo, CC-BY-4.0
  10. Xiao et al., PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving, IEEE ITSC 2021 — arXiv:2112.12610 · license statement: pandaset.org
  11. Alibeigi et al., Zenseact Open Dataset, ICCV 2023 — arXiv:2305.02008 · license: zod.zenseact.com/license (CC-BY-SA-4.0)
  12. swisstopo, Terms of use for free geodata and geoservices — swisstopo.admin.ch (commercial use permitted with attribution; accessed 2026-07-10)
  13. AHN (Actueel Hoogtebestand Nederland), Open data — ahn.nl/open-data ("free and without restrictions"; formal license name not stated on page)
  14. GovData, Datenlizenz Deutschland – Zero – 2.0 — govdata.de/dl-de/zero-2-0 (explicitly permits commercial use)
  15. Wu et al., Sonata: Self-Supervised Learning of Reliable Point Representations, CVPR 2025 — arXiv:2503.16429 (note: released weights CC-BY-NC; approach reproducible on own data)
  16. Varney, Asari, Graehling, DALES: A Large-scale Aerial LiDAR Data Set, CVPRW 2020 — arXiv:2004.11985 (license unverified as of 2026-07-10)
  17. nuScenes / Motional, Terms of Use — nuscenes.org/terms-of-use (page not machine-readable during research; commercial licensing offered separately)
  18. Audi, A2D2 dataset download & license — a2d2.audi/en/download (CC-BY-ND 4.0)
  19. OSDaR23: Open Sensor Data for Rail 2023 — arXiv:2305.03001 (data-portal license page unreachable during research)
  20. Tan et al., Toronto-3D, CVPRW 2020 — arXiv:2003.08284 · license: GitHub, CC-BY-NC-4.0
  21. Roynard, Deschaud, Goulette, Paris-Lille-3D, IJRR 2018 — arXiv:1712.00032 · license: npm3d.fr (CC-BY-NC-ND-3.0)
  22. Behley et al., SemanticKITTI, ICCV 2019 — arXiv:1904.01416 · license: semantic-kitti.org (CC-BY-NC-SA)
  23. Liao, Xie, Geiger, KITTI-360, TPAMI 2022 — arXiv:2109.13410 · license: cvlibs.net (CC-BY-NC-SA-3.0)
  24. Waymo, Waymo Open Dataset — License Agreement for Non-Commercial Use — waymo.com/open/terms
  25. Argoverse, About / Terms — argoverse.org/about.html (CC-BY-NC-SA-4.0; forbids training commercial models)
  26. ONCE dataset, Terms of Use — once-for-auto-driving.github.io (non-commercial only)
  27. Superpoint Transformer repository (MIT; 11 GB & 32 GB GPU configs, corridor tiling) — github.com/drprojects/superpoint_transformer
  28. Robert, Raguet, Landrieu, Efficient 3D Semantic Segmentation with Superpoint Transformer, ICCV 2023 — arXiv:2306.08045
  29. Robert, Raguet, Landrieu, Scalable 3D Panoptic Segmentation As Superpoint Graph Clustering (SuperCluster) — arXiv:2401.06704
  30. Pointcept repository (MIT; PTv3 reference implementation) — github.com/Pointcept/Pointcept
  31. Wu et al., Point Transformer V3: Simpler, Faster, Stronger, CVPR 2024 — arXiv:2312.10035
  32. FlashAttention repository — supported GPU generations (Ampere/Ada/Hopper) — github.com/Dao-AILab/flash-attention
  33. Microsoft Azure, NC A100 v4-series VM sizes — learn.microsoft.com
  34. Microsoft Azure, NCas T4 v3-series VM sizes — learn.microsoft.com · node specs also from our Azure ML workspace (12-node cluster, polandcentral)
  35. Jiang, Saripalli, LiDARNet: A Boundary-Aware Domain Adaptation Model for Point Cloud Semantic Segmentation — arXiv:2003.01174
  36. Wang et al., Test-time adaptation for geospatial point cloud semantic segmentation — arXiv:2407.06043
  37. Pointly, User Guide (LAS/LAZ support, classifier overwrite, free-tier export limits) — PDF · pricing

Full research trail with all verification notes: internal vault note 2026-07-10 Highway MLS semantic segmentation - models, public datasets and training strategy.