las-laz-js 0.4.0 → 0.6.1

package/CHANGELOG.md

@@ -1,5 +1,68 @@
 # Changelog
 
+## 0.6.1 (2026-10-22)
+
+- Demo: a read bigger than the draw cap (20 M) is drawn as an even spread over the whole cloud,
+  not its first 20 M points; selections and edits still act on every point read, and the
+  selection line says how many selected points are on screen. `?drawCap=N` lowers the cap.
+
+## 0.6.0 (2026-10-22)
+
+Voxel neighbour index and the sparse-point (dust) filter.
+
+- `selectSparse(cols, { radius, minNeighbours = 1, xy })` selects the points with fewer than
+  `minNeighbours` other points within `radius` (inclusive), for `removePoints`. `minNeighbours: 1`
+  keeps thin sparse features such as scanned blastholes. Checked point for point against PDAL's
+  `filters.outlier` (`method=radius`; PDAL's `min_k` is `minNeighbours`, and PDAL does not count a
+  point exactly `radius` away).
+- `buildIndex(cols, { cellSize })`: a sorted voxel grid on flat typed arrays (6 to 8 bytes a point),
+  over x / y / z or a k = 3 `positions` column, with exact queries: `nearestPoint` (3D, or `xy` in
+  plan; `maxDistance`), `pointsInRadius` (a selection) and `pointsNearRay` (a cylinder or a cone,
+  with each point's `t` along the ray and `d` from its axis, for a caller's snap). Rebuild it after
+  any edit of the columns.
+- `dedupeSelection(cols, tolerance, { z })` returns `dedupe`'s kept points as a selection and a
+  `map` from every input point to the kept point it merged into (its nearest within the tolerance).
+- Speed (Node): a 34-million-point LAS (Float32 after a local origin) indexes in about 1.1 s at
+  about 8 bytes a point, and `selectSparse` at 2 m takes about 2.1 s.
+- Demo: a Sparse points selection on the Edit tab.
+
+## 0.5.0 (2026-10-22)
+
+Point operations.
+
+- Operations on point columns (the object `readPoints` returns, or any `{ count, <name>: TypedArray }`
+  whose arrays are `count × k` long), with no dependencies. A selection is a `Uint32Array` of
+  strictly ascending point indices, checked on use (`INVALID_ARGUMENT` names the bad position).
+- Select: `selectBox`, `selectPolygon` (even-odd; `boundary: "touching"` keeps points on the edge,
+  `"inside"` drops them; `tolerance`; `minZ` / `maxZ`), `selectClasses`, `selectReturns` (`returns`,
+  `first`, `last`, `single`, any one matches), `thinSelection` (the reader's `maxPoints` rule,
+  exported), and `unionSelections`, `intersectSelections`, `subtractSelections`.
+- Apply: `keepPoints`, `removePoints`, `splitPoints`, `mergePoints`. Copies by default; `inPlace: true`
+  on keep and remove compacts each column inside its own buffer and returns views of it (columns that
+  share memory are refused). `mergePoints` takes the union of the columns, lists zero-filled ones
+  in `filled`, widens types (Float64 over Float32 over integers; Int32 or Uint32 with Float32 gives
+  Float64), and refuses a `k` mismatch, a column named `count` or `filled`, and signed with unsigned
+  integers unless the signed type is strictly wider.
+- Transform: `makeTransform({ shift, rotateZ, rotate, scale, about })` builds a row-major `{ m, t }`;
+  `transformPoints(cols, T, { inPlace })` applies it in Float64 and returns a new `{ count, columns }`
+  object (a copy has new x, y, z and shares the rest; `inPlace` returns the input's arrays, overwritten).
+- `interleavePositions(cols, { layout, into, at, origin, float32 })` packs x y (z) into a caller's
+  Float32 / Float64 buffer, shifting by `origin` in Float64 first.
+- Reads: `index` field (each point's record number; Uint32, Float64 above 4,294,967,295 points);
+  `polygon` / `boundary` / `tolerance` crop while reading, so only the region is decoded and held;
+  `transform` applies in the fixed order record → world → T → minus `localOrigin` → Float32.
+  Filters apply in file coordinates; the result's bounds and `dedupe` are in output coordinates,
+  and the worker's elevation colour range follows the transform.
+- Changed: an omitted (or `null`) `bounds` limit is now open. Before, a missing limit kept nothing.
+  `polygon: null` means no polygon. `selectBox` and the `minZ` / `maxZ` of `selectPolygon` read a
+  `null` bound as open too.
+- `dedupe` moves every column (k values per point, such as an interleaved `positions` or `rgb`) with
+  the kept points; other properties pass through. `transformPoints` with `inPlace` refuses x, y or z
+  that share memory. `mergePoints` ignores a zero-point input when widening types.
+- Demo: an Edit tab (polygon, box and class selections; keep, delete, split, transform, crop on
+  read) and LAZ export.
+- Docs: README "Point operations".
+
 ## 0.4.0 (2026-10-22)
 
 Writes LAZ.

package/README.md

@@ -7,8 +7,12 @@ runs in the browser and in Node — and built for files far larger than memory.
 **Status: early (0.x).** It works and is tested, but the API may still change before 1.0.
 
 - Working: `openLas` reads the header, VLRs / EVLRs and coordinate system of LAS 1.0–1.4;
-  `readPoints` and `chunks` decode every point format (0–10) a window at a time, with filters
-  and thinning; `colourize` colours points; `writeLas` and `createLasWriter` write LAS 1.0–1.4.
+  `readPoints` and `chunks` decode every point format (0–10) a window at a time, with filters,
+  thinning, a polygon crop, a coordinate transform and the original point index; `colourize`
+  colours points; `writeLas` and `createLasWriter` write LAS 1.0–1.4.
+- Point operations on columns: select by box, polygon, class or return; keep, remove, split and
+  merge; transform coordinates; interleave positions for a renderer; find sparse points (dust);
+  a voxel neighbour index with nearest, radius and near-ray queries. See Point operations.
 - LAZ: point formats 0–3 and 6–8, read exactly like LAS — the same calls, the same options,
   decoded chunk by chunk in plain JavaScript (no WebAssembly) — and written with
   `compress: true`, byte for byte what PDAL (LAZperf) writes (bar one lossless exception, see
@@ -35,8 +39,9 @@ Drop a LAS or LAZ file onto the page. It is read locally (never uploaded), decod
 library's Worker (LAZ on every core) behind a progress bar, then shown once in a Three.js viewer;
 colour changes recolour the points already read, without reading the file again. The side panel shows the header, VLRs, coordinate
 system and LAZ compression; reads with class, return, bounds, thinning and dedupe filters;
-colours by RGB, intensity, elevation or class; lists point statistics; and saves the points back
-out as LAS. To run it locally: `npm run demo`, then open http://localhost:8080/.
+colours by RGB, intensity, elevation or class; lists point statistics; selects points by polygon,
+box or class and keeps, deletes, splits, transforms or crops them (the Edit tab); and saves the
+points back out as LAS or LAZ. To run it locally: `npm run demo`, then open http://localhost:8080/.
 
 ## Reading
 
@@ -56,7 +61,11 @@ var pts = await las.readPoints({
 	fields: ["x", "y", "z", "classification", "red", "green", "blue"],
 	classes: [2],            // keep only these classification codes
 	returns: [1],            // keep only these return numbers
-	bounds: { minX, maxX, minY, maxY, minZ, maxZ },
+	bounds: { minX, maxX, minY, maxY, minZ, maxZ },   // any limit may be left out: an omitted limit is open
+	polygon: [x0, y0, x1, y1, x2, y2],   // crop: keep points inside it (file coordinates, XY, even-odd); null is none
+	boundary: "touching",    // polygon edge: "touching" keeps points on it (default), "inside" drops them
+	tolerance: 0,            // file units: within this of a polygon edge counts as on it
+	transform: T,            // { m, t } from makeTransform, applied to world coordinates (see Point operations)
 	maxPoints: 5000000,      // even thinning; 0 keeps every point
 	dedupe: 0.01,            // drop points within 1 cm (XY; dedupeZ: true for 3D)
 	localOrigin: { x: 478000, y: 6772000 }, float32: true,   // small Float32 coordinates for the GPU
@@ -64,7 +73,7 @@ var pts = await las.readPoints({
 	signal: abortController.signal
 });
 pts.count; pts.x; pts.y; pts.z;   // one typed array per field
-pts.bounds; pts.stats;            // world bounds, per-class counts
+pts.bounds; pts.stats;            // world bounds (after any transform), per-class counts
 ```
 
 Fields: `x y z intensity returnNumber numberOfReturns scanDirection edgeOfFlightLine
@@ -72,10 +81,192 @@ classification classificationFlags scannerChannel scanAngle userData pointSource
 red green blue nir`. Fields a point format does not hold are left out. `scanAngle` is in
 degrees for every format.
 
+`index` is one more field, for every format: the column is present for the points the filters and
+`maxPoints` keep, and each value is that record's position in the file (0-based; it is not
+renumbered after filtering). It is a `Uint32Array`, or a `Float64Array` for a file of more than
+4,294,967,295 points. It is an ordinary column, so keep, remove, split and merge carry it, and it
+tells you which records of the file are left.
+
 Filters run on the raw record before it is decoded, so a filtered-out point costs a byte read.
+`polygon` is a crop on read: only the points inside it are decoded and held, so a site region of
+a large file never costs the whole file in memory. It needs at least 3 vertices (a flat list of
+x, y pairs, implicitly closed) and combines with the other filters and `maxPoints`. A LAZ crop
+saves memory and field decoding but still decompresses every chunk (there is no spatial index), so
+its time is close to a full read.
 `maxPoints` keeps exactly that many of the points that pass the filters, spread evenly over
 the whole file; with filters set it first counts the passing points, which reads the file twice.
 
+In `bounds`, a limit that is left out (or `null`) is open; before 0.5.0 a missing limit kept
+nothing. `polygon: null` means no polygon. `boundary` and `tolerance` belong to `polygon`: a point
+within `tolerance` of an edge or vertex is on the boundary, kept by `"touching"` and dropped by
+`"inside"`. They are not `dedupe`'s tolerance, which is the value of the `dedupe` option.
+
+`transform` moves the points while they are decoded, in a fixed order: record → world (scale and
+offset) → `transform` (Float64) → minus `localOrigin` → Float32 (when `float32` and `localOrigin`
+are set). Raw scanner coordinates never pass through Float32 at full magnitude. The filters
+(`classes`, `returns`, `bounds`, `polygon`) apply in **file** coordinates, before the transform;
+the result's `bounds`, its stats and `dedupe` are in the output coordinates. To crop a scan in
+site coordinates, read it transformed and use `selectPolygon`. The Worker's elevation colour
+range follows the transform.
+
+## Point operations
+
+Zero-dependency operations on **point columns**: the object `readPoints` returns, or any object
+of the same shape, `{ count, <name>: TypedArray, ... }`. A property is a column when it is a typed
+array whose length is `count × k` for a whole number `k` of 1 or more (`k` is 1 for `x` or
+`classification`, 3 for an interleaved `positions` or `rgb`). Operations carry every column, `k`
+values per point, and nothing else: `bounds`, `stats` and the like are not copied, so rebuild them
+from the result. A **selection** is a `Uint32Array` of point indices, strictly ascending, each
+below `count`. Everything that takes one checks it in one pass and throws `INVALID_ARGUMENT`
+naming the first bad position; everything that makes one returns a valid one.
+
+### Select
+
+```js
+import { selectBox, selectPolygon, selectClasses, selectReturns, thinSelection,
+         unionSelections, intersectSelections, subtractSelections } from "las-laz-js";
+
+var inBox  = selectBox(pts, { minX: 477010, maxX: 477050, minZ: 295 });   // inclusive; a limit left out is open
+var inSite = selectPolygon(pts, poly, { boundary: "inside", tolerance: 0.001 });   // poly: [x0, y0, x1, y1, ...]
+var ground = selectClasses(pts, [2]);
+var last   = selectReturns(pts, { last: true, single: true });   // returns: [..], first, last, single; any one matches
+var thin   = thinSelection(pts.count, 1000);                     // exactly min(count, 1000) indices, spread evenly
+var wanted = intersectSelections(inSite, ground);                // also unionSelections, subtractSelections
+```
+
+`selectPolygon` takes a flat `[x0, y0, x1, y1, ...]` list of at least 3 vertices, closed for you,
+in the columns' own coordinates; the interior is the even-odd rule, so a concave polygon works.
+Points on the boundary are kept by `boundary: "touching"` (the default) and dropped by `"inside"`,
+so the two differ by exactly the boundary points, and `splitPoints` by one polygon loses and
+duplicates nothing. Coordinates are not exact in binary, so a point mathematically on an edge may not
+count as on it at `tolerance: 0`; use a small tolerance (for example 1e-6 of the units) to catch
+boundary points. `minZ` / `maxZ` add a height limit. `thinSelection` is the reader's
+`maxPoints` rule: `keepPoints(all, thinSelection(n, t))` gives the points
+`readPoints({ maxPoints: t })` does.
+
+### Keep, remove, split, merge
+
+```js
+import { keepPoints, removePoints, splitPoints, mergePoints } from "las-laz-js";
+
+var kept    = keepPoints(pts, wanted);              // new columns of the same types
+var without = removePoints(pts, wanted);
+var [inside, outside] = splitPoints(pts, inSite);   // always copies; every point lands in one result
+var both    = mergePoints([inside, outside]);       // concatenated in order; both.filled is []
+
+var crop = keepPoints(big, selectPolygon(big, poly), { inPlace: true });   // no second copy
+```
+
+`inPlace: true` (for `keepPoints` and `removePoints`) slides the kept values down inside each
+column's own buffer and returns views of that memory: the input is overwritten, and a big cloud is
+cropped without a second copy. Columns that share memory (the same array twice, or overlapping
+views) are refused, since compacting one would corrupt the other.
+
+`mergePoints` takes the union of the column names; a column missing from an input is zero for
+that input's points, and its name is listed in `result.filled`. A column named `count` or
+`filled` is refused. A column's `k` must agree across inputs. The result's type is the widest:
+Float64 over Float32 over the integer types; an Int32 or Uint32 with a Float32 gives Float64
+(Float32 holds integers exactly only to 2^24). Signed with unsigned integers merge only when the
+signed type is strictly wider (Int16 with Uint8 is Int16; Int8 with Uint8, or Int32 with Uint32,
+is `INVALID_ARGUMENT`). An input with no points is ignored when the type is chosen, so an empty
+Int8 column does not block a Uint16 one; a column only an empty input has keeps its type.
+Coordinates must already share one frame.
+
+### Transform
+
+```js
+import { makeTransform, transformPoints } from "las-laz-js";
+
+// p' = scale · R · (p − about) + about + shift; rotateZ is degrees anticlockwise about +Z
+var T = makeTransform({ rotateZ: 90, about: [477050, 6770015, 0], shift: [10, 0, 0] });
+// T = { m: [0, -1, 0, 1, 0, 0, 0, 0, 1], t: [7247075, 6292965, 0] }   row-major: p' = m·p + t
+
+var moved = transformPoints(pts, T);                     // { count, columns }: new x, y, z; the rest shared
+var same  = transformPoints(pts, T, { inPlace: true });  // the same x, y, z arrays, overwritten
+
+var local = await las.readPoints({                       // or while reading
+	fields: ["x", "y", "z"], transform: T,
+	localOrigin: { x: 477050, y: 6770015 }, float32: true
+});
+```
+
+A transform is a plain object, `{ m, t }`, with `m` nine numbers (a row-major 3×3) and `t` three.
+Any affine map fits, so a fitted registration can be passed the same way; it is not Three.js's
+column-major `Matrix4`. `makeTransform` takes `shift`, `about`, `scale` and either `rotateZ` or
+`rotate` (a row-major 3×3; not both). `transformPoints` needs `x`, `y` and `z`, computes in
+Float64 and stores in the columns' own types. It returns a new `{ count, columns }` object either
+way: with `inPlace` the arrays in it are the input's, overwritten (and `x`, `y` and `z` must not
+share memory, or it throws `INVALID_ARGUMENT`). On read, the order is the one
+above.
+
+### Interleave for a renderer
+
+```js
+import { interleavePositions } from "las-laz-js";
+
+var positions = interleavePositions(pts, { origin: { x: 477000, y: 6770000 } });   // Float32Array, x y z per point
+geometry.setAttribute("position", new THREE.BufferAttribute(positions, 3));
+
+// or into a buffer you own, starting at point 'at'; layout "xy" writes two values per point
+interleavePositions(pts, { into: buffer, at: 50000, layout: "xyz", origin: { x: 477000, y: 6770000, z: 300 } });
+```
+
+The origin is subtracted in Float64 before storing, so world coordinates keep their decimals in a
+Float32 buffer; an axis left out of `origin` is not shifted ({ x, y } keeps world Z). `float32`
+defaults to true and picks the type of a new array; `into` must be a Float32Array or Float64Array
+long enough, else `INVALID_ARGUMENT`.
+
+### Sparse points (dust)
+
+```js
+import { selectSparse, removePoints } from "las-laz-js";
+
+var dust  = selectSparse(pts, { radius: 2 });   // fewer than 1 other point within 2 m
+var clean = removePoints(pts, dust);
+var lone  = selectSparse(pts, { radius: 0.5, minNeighbours: 4, xy: true });   // counted in 2D
+```
+
+A point is sparse when fewer than `minNeighbours` (default 1) **other** points lie within
+`radius` of it, inclusive; `radius` is required, in the columns' units. The result is a selection,
+so it can be shown before anything is deleted. `minNeighbours: 1` keeps any point with a single
+neighbour inside the radius, so thin sparse features — a line of points down a scanned blasthole —
+survive; dust that clumps survives too. A higher `minNeighbours` removes small clumps, at the
+cost of thin features. It works on columns in memory (x / y / z, or a k = 3 `positions`): 34 million
+points take about 2 s in Node. With PDAL's `filters.outlier` (`method=radius`), `min_k` is the same
+as `minNeighbours`, but PDAL does not count a point exactly `radius` away.
+
+### Neighbour index
+
+```js
+import { buildIndex, nearestPoint, pointsInRadius, pointsNearRay } from "las-laz-js";
+
+var index = buildIndex(pts);                              // or { cellSize: 0.5 }; omitted: ~16 points a cell
+var i     = nearestPoint(index, [x, y, z]);               // point number, or -1
+var j     = nearestPoint(index, [x, y], { xy: true, maxDistance: 0.3 });   // nearest in plan, within 0.3
+var near  = pointsInRadius(index, [x, y, z], 5);          // a selection
+var hit   = pointsNearRay(index, origin, direction, { radius: 0.2, growth: 0 });
+// hit = { points, t, d }: every point within radius + growth · t of the ray, t along it, d from its axis
+```
+
+A sorted voxel grid on flat typed arrays, 6 to 8 bytes a point (34 million points build in about
+1 s). Results are exact — every candidate is distance-checked in Float64 — "within" is inclusive,
+and ties go to the lowest point number. `pointsNearRay` is a cylinder for `growth: 0` (an
+orthographic camera's pick ray) and a cone for `growth > 0` (a perspective camera's, `growth` the
+tangent of the half-angle); points behind `origin` are left out, and choosing a snap point from `t`
+and `d` is the caller's. The index refers to the coordinate arrays, it does not copy them: rebuild
+it after any keep, remove, merge or transform — after an `inPlace` edit an old index would read
+overwritten memory, and queries cannot tell.
+
+### Dedupe with an index map
+
+```js
+import { dedupeSelection } from "las-laz-js";
+
+var { keep, map } = dedupeSelection(pts, 0.01);   // { z: true } for 3D
+var unique = keepPoints(pts, keep);               // what dedupe(pts, 0.01) returns
+// map[i]: where input point i went in unique — a dropped point maps to its nearest kept point
+```
+
 ## LAZ
 
 A LAZ file is opened and read with the same calls as LAS. Each LASzip chunk (50,000 points in

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "las-laz-js",
-	"version": "0.4.0",
+	"version": "0.6.1",
 	"description": "Zero-dependency JavaScript reader and writer for ASPRS LAS point clouds, and reader and writer for LAZ (LASzip), for the browser and Node, built for files larger than memory.",
 	"type": "module",
 	"exports": {

package/src/LasReader.js

@@ -25,6 +25,7 @@ import { createChunkDecoder } from "./laz/decompressChunk.js";
 import { createLazPool, defaultThreads } from "./laz/pool.js";
 import { makeKeep, countKept } from "./filter/filters.js";
 import { dedupe } from "./filter/dedupe.js";
+import { checkTransform } from "./ops/transform.js";
 import { openScratchSyncWriter } from "./io/ScratchBinaryFile.js";
 
 var DEFAULT_WINDOW = 64 * 1024 * 1024;
@@ -179,7 +180,7 @@ LasReader.prototype._plan = async function () {
 
 /**
  * Iterate the selected points window by window.
- * @param {object} [opts] - fields, classes, returns, bounds, maxPoints, localOrigin, float32, onProgress, signal
+ * @param {object} [opts] - fields, classes, returns, bounds, polygon (crop; with boundary "touching" | "inside" and tolerance), maxPoints, transform ({ m, t } from makeTransform: world coordinates are transformed in Float64 before localOrigin is subtracted; filters apply in file coordinates), localOrigin, float32, onProgress, signal
  * @returns {AsyncGenerator<object>} PointColumns per window
  */
 LasReader.prototype.chunks = async function* (opts) {
@@ -192,6 +193,7 @@ LasReader.prototype.chunks = async function* (opts) {
 // makes decodeRecords write straight into readPoints' result columns.
 LasReader.prototype._scan = async function* (opts, into) {
 	var h = this.header;
+	if (opts.transform) checkTransform(opts.transform, "transform");
 	var plan = await this._plan();
 	var keep = makeKeep(opts, h);
 	var fields = resolveFields(h, opts.fields);
@@ -208,7 +210,7 @@ LasReader.prototype._scan = async function* (opts, into) {
 		var kept = 0, w = 0;
 		for await (var bytes of windowBytes(plan, pool, buf, opts.signal)) {
 			var win = list[w];
-			var cols = decodeRecords(bytes, h, { fields: fields, keep: keep, thin: thin, localOrigin: opts.localOrigin, float32: opts.float32, into: into ? into(win) : null });
+			var cols = decodeRecords(bytes, h, { fields: fields, keep: keep, thin: thin, localOrigin: opts.localOrigin, float32: opts.float32, transform: opts.transform, firstIndex: win.firstRecord, into: into ? into(win) : null });
 			kept += cols.count;
 			w++;
 			if (typeof opts.onProgress === "function") opts.onProgress(w / list.length, "read " + kept + " points");
@@ -257,7 +259,7 @@ LasReader.prototype._countPassing = async function (plan, pool, buf, keep, signa
 
 /**
  * Read the selected points into one set of columns.
- * @param {object} [opts] - chunks() options plus dedupe (tolerance) and dedupeZ
+ * @param {object} [opts] - chunks() options plus dedupe (tolerance) and dedupeZ; dedupe runs on the output coordinates (after transform and localOrigin)
  * @returns {Promise<object>} PointColumns plus bounds and stats
  */
 LasReader.prototype.readPoints = async function (opts) {
@@ -273,7 +275,7 @@ LasReader.prototype.readPoints = async function (opts) {
 	var cap = !filtered ? (opts.maxPoints > 0 ? Math.min(opts.maxPoints, h.pointCount) : h.pointCount)
 		: (opts.maxPoints > 0 ? Math.min(opts.maxPoints, h.pointCount) : 0);
 	var acc = {}, n = 0;
-	fields.forEach(function (f) { acc[f] = new (columnType(f, opts))(cap || 1024); });
+	fields.forEach(function (f) { acc[f] = new (columnType(f, opts, h))(cap || 1024); });
 	function into(win) {
 		// A known cap is never exceeded, so it never grows; filters alone grow by doubling
 		if (!cap && n + win.count > acc[fields[0]].length) {

package/src/filter/dedupe.js

@@ -5,18 +5,25 @@
 // tolerance-sized cells, neighbours checked), ported from point
 // objects to columns. XY by default (one Z per XY location, as a
 // 2.5D surface wants); { z: true } compares in 3D.
-// Every column follows the kept points.
+// Every column follows the kept points (k values per point).
+// dedupeSelection gives the same keep as a selection, plus a map from
+// every input point to the output point it merged into.
 //==========================================================
 
 import { LasError, ERROR_CODES } from "../errors.js";
+import { columnsOf, gather, sameType } from "../ops/columns.js";
 
 /**
+ * The points dedupe keeps, and where every input point went.
+ * map[i] is the output number (position in keep) of the kept point that point i
+ * merged into: for a dropped point, the nearest kept point within tolerance
+ * (lowest number on a tie); a kept point maps to itself.
  * @param {object} cols - PointColumns ({ count, x, y, z?, ... }).
  * @param {number} tolerance - distance below which points merge (> 0).
- * @param {{z?: boolean}} [opts]
- * @returns {object} PointColumns with only the kept points
+ * @param {{z?: boolean}} [opts] - z: compare in 3D.
+ * @returns {{keep: Uint32Array, map: Uint32Array}}
  */
-export function dedupe(cols, tolerance, opts) {
+export function dedupeSelection(cols, tolerance, opts) {
 	if (!(tolerance > 0)) throw new LasError(ERROR_CODES.INVALID_ARGUMENT, "dedupe tolerance must be > 0");
 	var use3d = !!(opts && opts.z && cols.z);
 	var tol2 = tolerance * tolerance;
@@ -43,6 +50,8 @@ export function dedupe(cols, tolerance, opts) {
 	var head = new Map();
 	var next = new Int32Array(n);
 	var kept = new Uint32Array(n);
+	var map = new Uint32Array(n);
+	var dropped = [];
 	var nKept = 0;
 	for (var i = 0; i < n; i++) {
 		var cx = Math.floor(x[i] / tolerance) - lo[0] + 1, cy = Math.floor(y[i] / tolerance) - lo[1] + 1;
@@ -61,21 +70,62 @@ export function dedupe(cols, tolerance, opts) {
 				}
 			}
 		}
-		if (dup) continue;
+		if (dup) { dropped.push(i); continue; }
 		var own = key(cx, cy, cz);
 		var prev = head.get(own);
 		next[i] = prev === undefined ? -1 : prev;
 		head.set(own, i);
+		map[i] = nKept;
 		kept[nKept++] = i;
 	}
 
+	// Each dropped point maps to its nearest kept point (the cell lists hold kept points only)
+	dropped.forEach(function (i) {
+		var cx = Math.floor(x[i] / tolerance) - lo[0] + 1, cy = Math.floor(y[i] / tolerance) - lo[1] + 1;
+		var cz = use3d ? Math.floor(z[i] / tolerance) - lo[2] + 1 : 1;
+		var best = -1, bd = Infinity;
+		for (var dx = -1; dx <= 1; dx++) {
+			for (var dy = -1; dy <= 1; dy++) {
+				for (var dz = -dzRange; dz <= dzRange; dz++) {
+					var k = head.get(key(cx + dx, cy + dy, cz + dz));
+					while (k !== undefined && k >= 0) {
+						var ex = x[k] - x[i], ey = y[k] - y[i];
+						var ez = use3d ? z[k] - z[i] : 0;
+						var d = ex * ex + ey * ey + ez * ez;
+						if (d <= tol2 && (d < bd || (d === bd && k < best))) { bd = d; best = k; }
+						k = next[k];
+					}
+				}
+			}
+		}
+		map[i] = map[best];
+	});
+	return { keep: kept.slice(0, nKept), map: map };
+}
+
+/**
+ * @param {object} cols - PointColumns ({ count, x, y, z?, ... }).
+ * @param {number} tolerance - distance below which points merge (> 0).
+ * @param {{z?: boolean}} [opts]
+ * @returns {object} PointColumns with only the kept points
+ */
+export function dedupe(cols, tolerance, opts) {
+	var sel = dedupeSelection(cols, tolerance, opts).keep, nKept = sel.length;
+
+	// Every column (k values per point) follows the kept points; anything else passes through
+	var ks = Object.create(null);   // a column may be named like an Object.prototype member
+	columnsOf(cols).forEach(function (c) { ks[c.name] = c.k; });
 	var out = { count: nKept };
 	Object.keys(cols).forEach(function (name) {
+		if (name === "count") return;
 		var col = cols[name];
-		if (!ArrayBuffer.isView(col) || col.length !== cols.count) { if (name !== "count") out[name] = col; return; }
-		var c = new col.constructor(nKept);
-		for (var q = 0; q < nKept; q++) c[q] = col[kept[q]];
-		out[name] = c;
+		var v = col;
+		if (Object.prototype.hasOwnProperty.call(ks, name)) {
+			v = sameType(col, nKept * ks[name]);
+			gather(col, v, ks[name], sel);
+		}
+		// defineProperty so a column named "__proto__" stays a column
+		Object.defineProperty(out, name, { value: v, enumerable: true, writable: true, configurable: true });
 	});
 	return out;
 }

package/src/filter/filters.js

@@ -2,8 +2,8 @@
 //==========================================================
 // Record filters that run on the RAW record, before decoding, so
 // rejected points cost one or two byte reads (Kirra's classFilter
-// trick, generalised): classification codes, return numbers and a
-// 3D box. Filters combine with AND.
+// trick, generalised): classification codes, return numbers, a
+// 3D box and a polygon (crop on read). Filters combine with AND.
 // maxPoints thinning is NOT Kirra's floor-stride-and-cap rule, which
 // kept only the start of the file and, run before the filters,
 // could leave nothing: decodeRecords thins the records that PASS,
@@ -11,12 +11,16 @@
 //==========================================================
 
 import { POINT_FORMATS } from "../format/PointFormats.js";
+import { makePolygonTest } from "../ops/polygon.js";
 
 var LE = true;
 
+function limit(v, open) { return v === undefined || v === null ? open : v; }
+
 /**
  * Build a keep(view, recordOffset) test for decodeRecords.
- * @param {object} [filter] - { classes?: number[], returns?: number[], bounds?: {minX,maxX,minY,maxY,minZ,maxZ} }
+ * @param {object} [filter] - { classes?: number[], returns?: number[], bounds?: {minX,maxX,minY,maxY,minZ,maxZ} (an omitted or null limit is open),
+ *   polygon?: number[] (null or omitted: no crop) [x0, y0, x1, y1, ...] with boundary ("touching" | "inside") and tolerance as selectPolygon, in the file's world coordinates }
  * @param {object} header - from parseHeader.
  * @returns {function(DataView, number): boolean|null} null when there is nothing to filter
  */
@@ -38,15 +42,32 @@ export function makeKeep(filter, header) {
 			? function (v, at) { return ret[v.getUint8(at + 14) & 15] === 1; } // bits-ok
 			: function (v, at) { return ret[v.getUint8(at + 14) & 7] === 1; }); // bits-ok
 	}
+	var sc = header.scale, of = header.offset;
 	if (filter.bounds) {
-		var b = filter.bounds, s = header.scale, o = header.offset;
+		// an omitted limit is open
+		var b = filter.bounds;
+		var minX = limit(b.minX, -Infinity), maxX = limit(b.maxX, Infinity);
+		var minY = limit(b.minY, -Infinity), maxY = limit(b.maxY, Infinity);
+		var minZ = limit(b.minZ, -Infinity), maxZ = limit(b.maxZ, Infinity);
+		tests.push(function (v, at) {
+			var x = v.getInt32(at, LE) * sc.x + of.x;
+			if (x < minX || x > maxX) return false;
+			var y = v.getInt32(at + 4, LE) * sc.y + of.y;
+			if (y < minY || y > maxY) return false;
+			var z = v.getInt32(at + 8, LE) * sc.z + of.z;
+			return z >= minZ && z <= maxZ;
+		});
+	}
+	if (filter.polygon !== undefined && filter.polygon !== null) {
+		// the file's world x, y of the raw record, as the bounds test computes them; boundary and tolerance as selectPolygon
+		var poly = makePolygonTest(filter.polygon, { boundary: filter.boundary, tolerance: filter.tolerance });
+		var inPoly = poly.test, pminX = poly.minX, pmaxX = poly.maxX, pminY = poly.minY, pmaxY = poly.maxY;
 		tests.push(function (v, at) {
-			var x = v.getInt32(at, LE) * s.x + o.x;
-			if (x < b.minX || x > b.maxX) return false;
-			var y = v.getInt32(at + 4, LE) * s.y + o.y;
-			if (y < b.minY || y > b.maxY) return false;
-			var z = v.getInt32(at + 8, LE) * s.z + o.z;
-			return z >= b.minZ && z <= b.maxZ;
+			var x = v.getInt32(at, LE) * sc.x + of.x;
+			if (x < pminX || x > pmaxX) return false;   // y is not decoded for a point outside the box in x
+			var y = v.getInt32(at + 4, LE) * sc.y + of.y;
+			if (y < pminY || y > pmaxY) return false;
+			return inPoly(x, y);
 		});
 	}
 	if (!tests.length) return null;