@harperfast/hnsw 0.1.0 → 0.2.1

package/Cargo.lock

@@ -46,6 +46,7 @@ dependencies = [
 name = "hnsw-plane"
 version = "0.0.1"
 dependencies = [
+ "libc",
  "memmap2",
  "napi",
  "napi-build",

package/DESIGN.md

@@ -96,18 +96,24 @@ One file per index (per slice, once C2 lands): `<index-path>.hnsw`.
 | freelist_head                     | u64 atomic | CAS push/pop; ABA-guarded with a 32-bit tag                  |
 | txn_watermark                     | u64        | last durably indexed transaction; advanced by msync cadence  |
 | clean_shutdown flag               | u8         | torn-state detection on open                                 |
+| invalidated latch                 | u8         | one-way (v7): watermark reads 0 on every handle, open refuses |
+| write_epoch                       | u64 atomic | bumped by every node write; re-arms the read-side repair probe |
 
 **Main region — layer-0 slots**, addressed `4096 + id × slot_size`:
 
-| Field                           | Size (768-d int8, cap 64)          |
-| ------------------------------- | ---------------------------------- |
-| seq (seqlock)                   | 4 B                                |
-| flags (valid/deleted) + level   | 2 B                                |
-| scale (f32) + invMag (f32)      | 8 B                                |
-| degree                          | 2 B                                |
-| vector (int8 × 768)             | 768 B                              |
-| neighbor ids (u32 × layer0_cap) | 256 B                              |
-| **total, padded**               | **1,040 B → 1 KB-aligned 1,088 B** |
+| Field                           | Size (768-d int8, cap 64)           |
+| ------------------------------- | ----------------------------------- |
+| seq (seqlock)                   | 4 B                                 |
+| flags (valid/deleted) + level   | 2 B                                 |
+| scale (f32) + invMag (f32)      | 8 B                                 |
+| degree                          | 2 B                                 |
+| vector (int8 × 768)             | 768 B (padded to a 4-byte boundary) |
+| neighbor ids (u32 × layer0_cap) | 256 B                               |
+| **total, padded**               | **1,040 B → 1 KB-aligned 1,088 B**  |
+
+The vector's trailing pad keeps the neighbor array 4-aligned for every `dims`, so the search
+hot path reads each neighbor id as one aligned volatile `u32`. Upper-layer id lists are padded
+the same way (`degree u16 + pad u16 + ids`).
 
 At 100M nodes: ~109 GB (int8). A binary-code v2 slot (96 B codes + ids) is ~384 B → ~38 GB.
 For comparison, today's encoding averages 1,425 B/node _plus_ RocksDB overhead — so v1 is
@@ -167,6 +173,24 @@ bounded-lag durability with deterministic catch-up. For an approximate index who
 truth (records + pk→nodeId) remains fully transactional, bounded lag is the right trade — it
 buys the entire performance model.
 
+**Invalidation (a plane the host cannot delete).** Disabling a plane deletes its file; when the
+unlink fails (Windows sharing violation while another process maps it) the file must not be
+adopted later at its nonzero watermark, or it silently serves searches missing every mutation
+made while mirroring was off. `invalidate_plane(path)` / `invalidate_file(&handle)` leave two
+markers, always attempting both: in band — `PlaneFile::invalidate` sets a one-way header latch,
+zeroes the watermark, and msyncs the header page alone (a whole-mapping flush cannot run inline
+on a multi-GB plane, and lowering the watermark is the safe direction) — then a `<path>.stale`
+sidecar, created with create-new semantics (a planted symlink is never followed) and fsync'd
+together with its directory entry (the directory fsync is skipped on Windows, where `std` has no
+directory handle and `FlushFileBuffers` on the marker covers its creation). The package enforces
+both markers: `open` refuses a file carrying either, `create` refuses a path with a leftover
+sidecar, and `watermark()` reads 0 on every handle while the latch is set — so a flush already
+in flight on another handle, which still stamps the word, cannot revive the plane. In band
+first: the sidecar is what a process that cannot map the file checks, the latch is what covers a
+plane whose sidecar a crash lost. A temporary handle opened for the in-band mark is unmapped
+and closed before the sidecar step — its own mapping would keep the file undeletable — and the
+call fails only when neither marker is durable, leaving the file exactly as found.
+
 **Backup/copy-db/reseed:** the file is node-local derived state. Backup either includes it
 (consistent-enough after an msync barrier) or marks the index rebuild-on-restore. Replica
 reseed = rebuild from records (C5 bulk construction makes this fast; until then, the existing
@@ -186,6 +210,60 @@ search(sliceHandles, queryVector: Float32Array, k, ef, filter?): Promise<{ids, d
 - Auto-ef / auto-efC read the node count from the header high-water minus freelist length —
   same semantics as today, minus the #2182 inflation (freed ids return to the pool).
 
+**Upper-layer descent is a beam, not hill climbing** (`beam_descend`, `DESCENT_EF = 16`). The
+textbook width-1 descent halts at the first upper-layer node no neighbor improves on. On a
+clustered corpus that local minimum can sit in the wrong basin, and layer-0 adjacency is
+intra-basin, so the layer-0 beam has no uphill edge with which to leave — the query's true
+nearest neighbor is then unreachable at *any* ef, and raising ef only expands the wrong basin.
+Measured on the `tests/concurrent.rs` corpus (8 000 nodes, 64-d, self-query every node at
+ef 256, insertion order fixed by seed): width 1 loses 125 nodes over 200 builds, width 4 loses
+20 over 200, width 8 loses 8 over 700, width 16 loses 0 over 700. Cost at 50 000 × 768-d:
+visits/query +17 % to +27 %, p50 +0.06 ms flat (0.15 → 0.21 ms at ef 16, 0.21 → 0.28 at ef 64,
+0.46 → 0.47 at ef 512 — the descent is a fixed cost, so it hurts most where ef is small), build
+throughput -20 %. recall@10 improves below ef 128 (0.844 → 0.903 at ef 16, 0.983 → 1.000 at
+ef 64) and is unchanged above.
+
+`beam_descend` is shared by the read and write paths deliberately: insert must route through
+the same graph its queries will, or nodes get their neighbors chosen from a basin searches
+never reach. The width is a compile-time constant rather than a parameter because it is a
+correctness floor, not a recall/latency dial — `ef` is the dial.
+
+Three facts worth keeping when working on this.
+
+The trap is a property of graph *shape*, not of concurrency: it reproduces single-threaded from
+a fixed insertion permutation, and concurrency only shuffles that permutation. It also needs the
+full corpus — no seed reproduces it at 32 dims, or at 2 000 / 4 000 nodes, so a shrunken repro
+is not evidence of a fix. `descent_width_sweep` in `tests/concurrent.rs` (ignored by default) is
+the harness behind the table above.
+
+Read and write descent widths must match. Measured over 200 builds per cell: width 1 both sides
+loses 125 nodes, read-only widening loses 37, **write-only widening loses 245 — worse than
+either**, and both sides widened loses 0. `insert` seeds each level's `search_layer` from the
+descent's landing point, so a graph wired under one routing policy and queried under another is
+less navigable than one where they agree. This is also the upgrade story: an existing plane file
+read by a new binary is the read-only row, improved but not repaired until its nodes are
+re-inserted.
+
+`HNSW_SWEEP_READ_EF` sets the sweep's query-side width; the build side is whatever `DESCENT_EF`
+is compiled as, so the four cells are two runs per value of the constant:
+
+```text
+HNSW_SWEEP_SEEDS=200 HNSW_SWEEP_READ_EF=1  cargo test --release --test concurrent \
+    descent_width_sweep -- --ignored --nocapture
+HNSW_SWEEP_SEEDS=200 HNSW_SWEEP_READ_EF=16 cargo test --release --test concurrent \
+    descent_width_sweep -- --ignored --nocapture
+```
+
+Do not add a per-level visit cap to the descent without re-measuring. The obvious ceiling,
+`DESCENT_EF * UPPER_CAP` = 1024, is already exceeded by ordinary queries: the worst of 3 000
+random queries visits 788 nodes at level 1 on a 50 000-node graph and 1 044 on a 500 000-node
+one. A cap that binds silently degrades recall, which is the defect this exists to fix. What
+bounds the pathological case instead is `search_layer`'s strict `d < worst`: with every distance
+tied — a zero query ties them all at exactly 1.0 — a full result set never admits another
+candidate, so the descent drains after `ef` expansions per level (measured 608 visits at 50 000
+nodes, 990 at 500 000). `a_tied_distance_descent_stops_at_its_visit_cap` fails if that `<` is
+ever relaxed.
+
 **Filtering** (predicate-aware / ACORN, `filteredSearch = true` today):
 
 1. **Bitset fast path.** RBAC allow-lists and companion-condition candidate sets are computed
@@ -272,6 +350,14 @@ Decided (Kris, 2026-08-31):
 
 Open:
 
+- **Atomic slot payloads.** Fields a concurrent reader acts on (flags, level, degree, scale,
+  invMag, neighbor and upper ids) are read through aligned `read_volatile`, which forbids the
+  reload/split/sink across the seqlock's validating fence that `lto = true, codegen-units = 1`
+  otherwise licenses. That is not the same as being race-free under Rust's memory model: only
+  making those fields `AtomicU8`/`AtomicU16`/`AtomicU32` in the slot layout would be, and that
+  is a format change deferred past phase 1. The stored vector stays an ordinary load on
+  purpose — `cosine_int8_raw` must keep autovectorizing, and a torn vector only perturbs a
+  distance the generation check discards.
 - **msync cadence default** — bounded-lag durability window vs write amplification; needs a
   workload measurement, not a guess.
 - **f32 (quantization:"none") slot variant** — 3,072 B vectors → 3.4 KB slots; supported by the

package/README.md

@@ -35,15 +35,18 @@ the host's authoritative record store. Approximate indexes don't need per-commit
 they need cheap, bounded catch-up. See [DESIGN.md](DESIGN.md) for the format, the
 concurrency model, measured baselines, and the reasoning behind every trade.
 
-## Install & build
+## Install
 
 ```bash
 npm install @harperfast/hnsw
 ```
 
-Prebuilds are not published yet: building requires a [Rust toolchain](https://rustup.rs)
-(`npm run build`, or automatically on install when cargo is available). Linux x86_64 is the
-performance target (AVX2); macOS works (scalar fallback); Windows is untested.
+Prebuilt bindings ship as platform-specific `optionalDependencies`
+(`@harperfast/hnsw-<platform>-<arch>[-glibc]`) for linux-x64, linux-arm64, darwin-arm64, and
+win32-x64. Platforms without a published binding (musl, darwin-x64, win32-arm64) build from
+source on install when a [Rust toolchain](https://rustup.rs) is present, and throw a clear
+error otherwise. Linux x86_64 is the performance target (AVX2 + kernel-lock crash recovery);
+macOS and Windows are functional (no lock takeover — bounded degradation instead).
 
 ## Usage
 
@@ -65,6 +68,13 @@ const predicated = await plane.searchWithPredicate(queryVector, 10, 512, (ids) =
 );
 ```
 
+A plane is derived state; when the host must stop maintaining one and cannot delete the file
+(Windows sharing violations while another process maps it), `invalidatePlane(path)` — or
+`plane.invalidateFile()` through a handle the host already holds — durably marks it
+unadoptable: a one-way in-band latch (watermark reads 0, `Plane.open` refuses) plus a fsync'd
+`<path>.stale` sidecar (`stalePathFor(path)`, which `open` also refuses). It throws only when
+neither marker lands. Hosts delete both files and rebuild.
+
 Full API in [index.d.ts](index.d.ts).
 
 ## Benchmarks
@@ -78,8 +88,8 @@ equal recall.
 
 ## Status
 
-Extracted from the Harper vector-index engine; the format (v2) and API are young and may
-change with a major version + reindex. Roadmap: prebuilds, binary-quantized slot format
+Extracted from the Harper vector-index engine; the format (v7) and API are young and may
+change with a version bump + reindex (an older format version fails to open; rebuild). Roadmap: prebuilds, binary-quantized slot format
 (~4× smaller traversal plane), Matryoshka dimension truncation, mremap growth, index
 slicing with native top-k merge.
 

package/build.mjs

@@ -10,12 +10,9 @@ import { fileURLToPath } from 'node:url';
 const crateRoot = dirname(fileURLToPath(import.meta.url));
 const ifNeeded = process.argv.includes('--if-needed');
 const artifact = join(crateRoot, 'hnsw-plane.node');
-const prebuilt = join(crateRoot, 'prebuilds', `${process.platform}-${process.arch}`, 'hnsw-plane.node');
-
-// a prebuild satisfies --if-needed only if it actually LOADS on this system (a newer-glibc
-// artifact exists but throws at require; fall through to a local build in that case)
+// a platform optionalDependency or a loadable local artifact satisfies --if-needed (a
+// binding that exists but fails to link falls through to a local build)
 function loads(path) {
-	if (!existsSync(path)) return false;
 	try {
 		createRequire(import.meta.url)(path);
 		return true;
@@ -23,7 +20,11 @@ function loads(path) {
 		return false;
 	}
 }
-if (ifNeeded && (loads(artifact) || loads(prebuilt))) {
+function platformPackageLoads() {
+	const libc = process.platform === 'linux' ? '-glibc' : '';
+	return loads(`@harperfast/hnsw-${process.platform}-${process.arch}${libc}`);
+}
+if (ifNeeded && (platformPackageLoads() || (existsSync(artifact) && loads(artifact)))) {
 	process.exit(0);
 }
 try {

package/index.d.ts

@@ -12,7 +12,10 @@ export interface SearchHit {
 export declare class Plane {
 	/** Create a new plane file. `maxNodes` is a sparse reservation — pages materialize on write. */
 	static create(path: string, dims: number, layer0Cap: number, maxNodes: number): Plane;
-	/** Open an existing plane file (format-version mismatch throws: rebuild the index). */
+	/**
+	 * Open an existing plane file. Throws on a format-version mismatch and on an invalidated
+	 * plane (header latch or `.stale` sidecar): delete the file and its sidecar, rebuild.
+	 */
 	static open(path: string): Plane;
 
 	/**
@@ -101,4 +104,42 @@ export declare class Plane {
 	flush(watermark?: number): void;
 	/** flush() on the libuv thread pool — a whole-map msync can stall its calling thread. */
 	flushAsync(watermark?: number): Promise<void>;
+	/**
+	 * In-band half of invalidateFile() only — no sidecar, so a process that cannot map the
+	 * file sees nothing; prefer invalidateFile(). Sets the one-way header latch, zeroes the
+	 * watermark, msyncs the header page (a 4 KB barrier, not a whole-mapping flush). From then
+	 * on every handle reads watermark 0, whatever a racing flush stamps, and open() throws.
+	 */
+	invalidate(): void;
+	/**
+	 * invalidatePlane() through this handle: the in-band mark via this mapping (no second open,
+	 * no second registry slot — on Windows this mapping is why the unlink failed) and the
+	 * `.stale` sidecar next to the path it opened. The path must not have been replaced since.
+	 */
+	invalidateFile(): InvalidationOutcome;
+	/** Whether the plane was invalidated, by any handle, since this one opened. */
+	invalidated(): boolean;
 }
+
+export interface InvalidationOutcome {
+	/** The watermark was zeroed and its header page msync'd. */
+	inBand: boolean;
+	/** `<path>.stale` exists and is fsync'd (on POSIX, so is its directory entry). */
+	sidecar: boolean;
+	inBandError?: string;
+	sidecarError?: string;
+}
+
+/**
+ * Make a plane file that could not be deleted unadoptable, durably, through a temporary
+ * handle that is unmapped and closed before this returns. Both markers are always attempted:
+ * the in-band latch and the fsync'd `.stale` sidecar; open() refuses a file carrying either.
+ * Throws only when neither marker became durable; nothing is deleted or renamed, and an
+ * in-band mark whose msync failed may still have landed in the shared mapping (the safe
+ * direction: it reads as incomplete). Idempotent. Synchronous (three small fsyncs on a cold path).
+ */
+export declare function invalidatePlane(path: string): InvalidationOutcome;
+/** invalidatePlane() on the libuv thread pool. */
+export declare function invalidatePlaneAsync(path: string): Promise<InvalidationOutcome>;
+/** The sidecar convention: `<path>.stale`. */
+export declare function stalePathFor(path: string): string;

package/index.js

@@ -1,30 +1,51 @@
-// Loads the native module: a bundled prebuild for this platform when present, else a
-// locally built artifact (`npm run build`, requires a Rust toolchain). Linux prebuilds are
-// glibc builds; musl systems build locally.
+// Loads the native module, mirroring @harperfast/rocksdb-js's model: a platform-specific
+// optionalDependency package when one exists for this platform (linux bindings are split by
+// libc), else a locally built artifact (`npm run build`, requires a Rust toolchain).
 'use strict';
 const { existsSync } = require('node:fs');
 const { join } = require('node:path');
 
-const candidates = [
-	join(__dirname, 'prebuilds', `${process.platform}-${process.arch}`, 'hnsw-plane.node'),
-	join(__dirname, 'hnsw-plane.node'),
-];
-// pick the first candidate that actually LOADS (a prebuilt binary can exist but fail to
-// link, e.g. built against a newer glibc than this system's — fall through to a local build)
-let native0;
-const failures = [];
-for (const candidate of candidates) {
-	if (!existsSync(candidate)) continue;
+function libcSuffix() {
+	if (process.platform !== 'linux') return '';
+	let isMusl = false;
+	try {
+		const { glibcVersionRuntime } = process.report?.getReport?.()?.header ?? {};
+		isMusl = !glibcVersionRuntime;
+	} catch {
+		// fall through to ldd probing
+	}
+	if (!isMusl) return '-glibc';
 	try {
-		native0 = require(candidate);
-		break;
-	} catch (error) {
-		failures.push(`${candidate}: ${error.message}`);
+		const { execSync } = require('node:child_process');
+		isMusl = execSync('ldd --version', { encoding: 'utf8', stdio: 'pipe' }).includes('musl');
+	} catch {
+		// ldd may not exist; keep the report-based verdict
+	}
+	return isMusl ? '-musl' : '-glibc';
+}
+
+const failures = [];
+let native0;
+// 1. platform package (published binding)
+try {
+	native0 = require(`@harperfast/hnsw-${process.platform}-${process.arch}${libcSuffix()}`);
+} catch (error) {
+	failures.push(`platform package: ${error.message}`);
+}
+// 2. local build (dev checkouts, source-build installs)
+if (!native0) {
+	const local = join(__dirname, 'hnsw-plane.node');
+	if (existsSync(local)) {
+		try {
+			native0 = require(local);
+		} catch (error) {
+			failures.push(`${local}: ${error.message}`);
+		}
 	}
 }
 if (!native0) {
 	throw new Error(
-		`@harperfast/hnsw could not load a native binary for ${process.platform}-${process.arch}. ` +
+		`@harperfast/hnsw could not load a native binding for ${process.platform}-${process.arch}. ` +
 			'Build one with `npm run build` in ' +
 			__dirname +
 			' (requires a Rust toolchain: https://rustup.rs).' +

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@harperfast/hnsw",
-	"version": "0.1.0",
+	"version": "0.2.1",
 	"description": "Persistent, incrementally-maintained, concurrently-searchable native HNSW for Node.js: a memory-mapped fixed-slot graph file with off-event-loop search, seqlock concurrency, int8 asymmetric distance, and bitset/predicate filtering.",
 	"license": "Apache-2.0",
 	"repository": {
@@ -12,7 +12,6 @@
 	"files": [
 		"index.js",
 		"index.d.ts",
-		"prebuilds",
 		"hnsw-plane.node",
 		"src",
 		"build.rs",
@@ -40,5 +39,11 @@
 		"mmap",
 		"napi",
 		"rust"
-	]
+	],
+	"optionalDependencies": {
+		"@harperfast/hnsw-darwin-arm64": "0.2.1",
+		"@harperfast/hnsw-linux-arm64-glibc": "0.2.1",
+		"@harperfast/hnsw-linux-x64-glibc": "0.2.1",
+		"@harperfast/hnsw-win32-x64": "0.2.1"
+	}
 }

package/src/bin/bench.rs

@@ -122,7 +122,7 @@ fn main() {
         let build_start = Instant::now();
         for i in 0..n {
             let v = corpus.row(&mut rng);
-            insert(&graph, &v, &params, &mut scratch);
+            insert(&graph, &v, &params, &mut scratch).expect("build insert");
             if (i + 1) % 50_000 == 0 {
                 let rate = (i + 1) as f64 / build_start.elapsed().as_secs_f64();
                 println!("  built {} ({:.0} inserts/s)", i + 1, rate);