@delali/narsil 0.1.7 → 0.1.8

package/README.md

@@ -135,10 +135,50 @@ BM25 scoring with field boosting, fuzzy matching via bounded Levenshtein distanc
 
 Indexes split into shards when they grow beyond a configurable threshold. Partition assignment uses FNV-1a hashing for deterministic routing. Background rebalancing redistributes documents across new partitions without blocking queries, using a write-ahead queue for zero-downtime resharding.
 
+**Single-process overhead.** Partitioning pays off when shards live on separate processes or hosts. Inside one Node.js process, going from 1 to 20 partitions costs about 14% of insert throughput, 28% of median search latency, and 27% at p95, with no scaling upside. Keep `partitions.maxPartitions` low (or unset) for single-process deployments and raise it once you fan partitions out across workers or distributed nodes.
+
+**Rebalance latency spikes.** While a partition reshape is running, worst-tick p95 latency can climb to about 25ms compared with around 11ms in steady state. Schedule large bulk inserts that trigger rebalancing during low-traffic windows, or pre-size the index with `partitions.maxDocsPerPartition` to avoid mid-load reshapes.
+
 ### Worker isolation
 
 Search operations move off the main thread through worker threads (Node.js, Bun) or Web Workers (browser, Deno). The engine starts in direct mode for small indexes and auto-promotes to workers once document counts cross a threshold. You keep using the same API throughout.
 
+### Memory reporting
+
+Narsil reports memory at three levels: per index, per partition, and per runtime. `getStats(indexName)` returns `estimatedMemoryBytes`, a formula-based estimate for the index's main-thread partitions and vector structures.
+
+```ts
+const indexStats = narsil.getStats('products')
+
+console.log(indexStats.estimatedMemoryBytes)
+```
+
+The estimate comes from document counts, posting lists, field indexes, and vector indexes. It excludes V8 object headers, allocator overhead, and other host runtime costs, so use it for comparing indexes inside one process.
+
+`getPartitionStats(indexName)` returns the same estimate for each partition. The partition values sum to `getStats(indexName).estimatedMemoryBytes`.
+
+```ts
+const partitions = narsil.getPartitionStats('products')
+
+for (const partition of partitions) {
+  console.log(partition.partitionId, partition.estimatedMemoryBytes)
+}
+```
+
+`getMemoryStats()` returns a runtime snapshot and worker reports. It is async because workers report their heap usage through the worker message channel.
+
+```ts
+const memory = await narsil.getMemoryStats()
+
+console.log(memory.estimatedIndexBytes)
+console.log(memory.process?.heapUsed)
+console.log(memory.workers)
+```
+
+`memory.estimatedIndexBytes` sums `estimatedMemoryBytes` across every index in this Narsil instance. `memory.process` comes from `process.memoryUsage()` when the runtime exposes it. It measures the whole host process, so two Narsil instances in one Node.js process report the same process numbers. Browser runtimes return `null` for `memory.process`.
+
+`memory.workers` lists `heapUsed`, `heapTotal`, and `external` for each active worker. The array is empty before worker promotion.
+
 ### Distributed BM25 scoring
 
 Three scoring modes handle the partition-skew problem:
@@ -149,7 +189,7 @@ Three scoring modes handle the partition-skew problem:
 
 ### Vector search
 
-Store and query high-dimensional embeddings with cosine similarity, dot product, or Euclidean distance. Small vector sets use brute-force linear scan. When a field exceeds 10,000 vectors, the engine builds an HNSW graph in the background and swaps the search backend transparently.
+Store and query high-dimensional embeddings with cosine similarity, dot product, or Euclidean distance. Small vector sets use an exact brute-force scan. Once a field crosses 1,024 vectors, the engine builds an HNSW graph and switches the field to approximate search. The threshold is configurable per index via `vectorPromotion.threshold`.
 
 ### Hybrid search
 
@@ -356,7 +396,7 @@ Ranking quality is measured against the [Cranfield Collection](https://ir-datase
 
 | Engine | nDCG@10 | P@10 | MAP | MRR |
 | --- | ---: | ---: | ---: | ---: |
-| **Narsil 0.1.1** | **0.3739** | **0.2458** | **0.2614** | **0.5638** |
+| **Narsil** | **0.3739** | **0.2458** | **0.2614** | **0.5638** |
 | Orama 3.1.18 | 0.2911 | 0.1836 | 0.1846 | 0.4821 |
 | MiniSearch 7.2.0 | 0.0077 | 0.0067 | 0.0027 | 0.0139 |
 
@@ -371,6 +411,8 @@ These scores fall within the expected range for BM25 on standard IR benchmarks (
 
 Reproduce: `pnpm -C packages/benchmarks bench --tiers quality`
 
+Narsil also runs as a search server. On the BEIR information-retrieval datasets it is measured against Elasticsearch, OpenSearch, Qdrant, Weaviate, Typesense, and Meilisearch across keyword, vector, and hybrid retrieval. See [the full benchmarks](https://github.com/assetcorp/narsil/blob/main/BENCHMARKS.md) for those results.
+
 ## Runtime support (TypeScript)
 
 | Runtime | Concurrency | Persistence | Invalidation |

package/dist/adapters/filesystem.d.ts

@@ -1,4 +1,4 @@
-import { P as PersistenceAdapter } from '../adapters-BbRkoiu9.js';
+import { b as PersistenceAdapter } from '../adapters-Bo2JQg8b.js';
 
 interface FilesystemPersistenceConfig {
     directory: string;

package/dist/adapters/filesystem.mjs

@@ -1,4 +1,4 @@
-import { NarsilError, ErrorCodes } from '../chunk-R2C2EQGO.mjs';
+import { NarsilError, ErrorCodes } from '../chunk-ALB35PRQ.mjs';
 
 // src/persistence/filesystem.ts
 var cachedFs = null;
@@ -30,6 +30,25 @@ async function resolveAndGuard(baseDir, key) {
   }
   return resolvedPath;
 }
+async function fsyncDirectory(dirPath, fs) {
+  let handle = null;
+  try {
+    handle = await fs.open(dirPath, "r");
+    await handle.sync();
+  } catch (err) {
+    const code = err.code;
+    if (code === "EISDIR" || code === "EPERM" || code === "EINVAL" || code === "ENOTSUP") {
+      return;
+    }
+    throw new NarsilError(ErrorCodes.PERSISTENCE_FSYNC_FAILED, `fsync failed for directory "${dirPath}"`, {
+      cause: err instanceof Error ? err.message : String(err)
+    });
+  } finally {
+    if (handle !== null) {
+      await handle.close().catch(() => void 0);
+    }
+  }
+}
 async function listDirectoryRecursive(dir, baseDir, pathMod, fs) {
   const results = [];
   let entries;
@@ -62,13 +81,38 @@ function createFilesystemPersistence(config) {
   }
   const baseDir = config.directory;
   return {
+    directory: baseDir,
     async save(key, data) {
       const resolvedPath = await resolveAndGuard(baseDir, key);
       const fs = await getFs();
       const pathMod = await getPath();
       const dir = pathMod.dirname(resolvedPath);
       await fs.mkdir(dir, { recursive: true });
-      await fs.writeFile(resolvedPath, data);
+      const tempName = `${pathMod.basename(resolvedPath)}.tmp-${process.pid}-${Date.now()}-${Math.random().toString(36).slice(2)}`;
+      const tempPath = pathMod.join(dir, tempName);
+      let handle = null;
+      try {
+        handle = await fs.open(tempPath, "w");
+        await handle.write(data);
+        try {
+          await handle.sync();
+        } catch (err) {
+          throw new NarsilError(ErrorCodes.PERSISTENCE_FSYNC_FAILED, `fsync failed for "${key}"`, {
+            key,
+            cause: err instanceof Error ? err.message : String(err)
+          });
+        }
+        await handle.close();
+        handle = null;
+        await fs.rename(tempPath, resolvedPath);
+        await fsyncDirectory(dir, fs);
+      } catch (err) {
+        if (handle !== null) {
+          await handle.close().catch(() => void 0);
+        }
+        await fs.unlink(tempPath).catch(() => void 0);
+        throw err;
+      }
     },
     async load(key) {
       const resolvedPath = await resolveAndGuard(baseDir, key);

package/dist/adapters/indexeddb.d.ts

@@ -1,4 +1,4 @@
-import { P as PersistenceAdapter } from '../adapters-BbRkoiu9.js';
+import { b as PersistenceAdapter } from '../adapters-Bo2JQg8b.js';
 
 interface IndexedDBPersistenceConfig {
     dbName?: string;

package/dist/adapters/indexeddb.mjs

@@ -1,4 +1,4 @@
-import { NarsilError, ErrorCodes } from '../chunk-R2C2EQGO.mjs';
+import { NarsilError, ErrorCodes } from '../chunk-ALB35PRQ.mjs';
 
 // src/persistence/indexeddb.ts
 var DEFAULT_DB_NAME = "narsil";

package/dist/adapters/memory.d.ts

@@ -1,4 +1,4 @@
-import { P as PersistenceAdapter } from '../adapters-BbRkoiu9.js';
+import { b as PersistenceAdapter } from '../adapters-Bo2JQg8b.js';
 
 declare function createMemoryPersistence(): PersistenceAdapter;
 

package/dist/{adapters-BbRkoiu9.d.ts → adapters-Bo2JQg8b.d.ts}

@@ -3,6 +3,7 @@ interface PersistenceAdapter {
     load(key: string): Promise<Uint8Array | null>;
     delete(key: string): Promise<void>;
     list(prefix: string): Promise<string[]>;
+    readonly directory?: string;
 }
 interface InvalidationAdapter {
     publish(event: InvalidationEvent): Promise<void>;
@@ -33,4 +34,4 @@ interface EmbeddingAdapter {
     shutdown?(): Promise<void>;
 }
 
-export type { EmbeddingAdapter as E, InvalidationAdapter as I, PersistenceAdapter as P, InvalidationEvent as a, PartitionStatistics as b };
+export type { EmbeddingAdapter as E, InvalidationAdapter as I, PartitionStatistics as P, InvalidationEvent as a, PersistenceAdapter as b };

package/dist/chunk-ALB35PRQ.mjs

@@ -0,0 +1,98 @@
+// src/errors.ts
+var ErrorCodes = {
+  SCHEMA_INVALID_TYPE: "SCHEMA_INVALID_TYPE",
+  SCHEMA_MISSING_FIELD: "SCHEMA_MISSING_FIELD",
+  SCHEMA_DEPTH_EXCEEDED: "SCHEMA_DEPTH_EXCEEDED",
+  SCHEMA_INVALID_VECTOR_DIMENSION: "SCHEMA_INVALID_VECTOR_DIMENSION",
+  SCHEMA_INVALID_GEOPOINT: "SCHEMA_INVALID_GEOPOINT",
+  DOC_NOT_FOUND: "DOC_NOT_FOUND",
+  DOC_ALREADY_EXISTS: "DOC_ALREADY_EXISTS",
+  DOC_VALIDATION_FAILED: "DOC_VALIDATION_FAILED",
+  INDEX_NOT_FOUND: "INDEX_NOT_FOUND",
+  INDEX_ALREADY_EXISTS: "INDEX_ALREADY_EXISTS",
+  PARTITION_CORRUPTED: "PARTITION_CORRUPTED",
+  PARTITION_REBALANCING_BACKPRESSURE: "PARTITION_REBALANCING_BACKPRESSURE",
+  WORKER_CRASHED: "WORKER_CRASHED",
+  WORKER_BUSY: "WORKER_BUSY",
+  WORKER_TIMEOUT: "WORKER_TIMEOUT",
+  PERSISTENCE_SAVE_FAILED: "PERSISTENCE_SAVE_FAILED",
+  PERSISTENCE_LOAD_FAILED: "PERSISTENCE_LOAD_FAILED",
+  PERSISTENCE_DELETE_FAILED: "PERSISTENCE_DELETE_FAILED",
+  PERSISTENCE_CRC_MISMATCH: "PERSISTENCE_CRC_MISMATCH",
+  PERSISTENCE_WAL_CORRUPT: "PERSISTENCE_WAL_CORRUPT",
+  PERSISTENCE_FSYNC_FAILED: "PERSISTENCE_FSYNC_FAILED",
+  SEARCH_INVALID_FIELD: "SEARCH_INVALID_FIELD",
+  SEARCH_INVALID_VECTOR_SIZE: "SEARCH_INVALID_VECTOR_SIZE",
+  VECTOR_DIMENSION_MISMATCH: "VECTOR_DIMENSION_MISMATCH",
+  SEARCH_INVALID_FILTER: "SEARCH_INVALID_FILTER",
+  SEARCH_INVALID_MODE: "SEARCH_INVALID_MODE",
+  SEARCH_INVALID_CURSOR: "SEARCH_INVALID_CURSOR",
+  LANGUAGE_NOT_SUPPORTED: "LANGUAGE_NOT_SUPPORTED",
+  ENVELOPE_VERSION_MISMATCH: "ENVELOPE_VERSION_MISMATCH",
+  ENVELOPE_INVALID_MAGIC: "ENVELOPE_INVALID_MAGIC",
+  PARTITION_CAPACITY_EXCEEDED: "PARTITION_CAPACITY_EXCEEDED",
+  EMBEDDING_FAILED: "EMBEDDING_FAILED",
+  EMBEDDING_DIMENSION_MISMATCH: "EMBEDDING_DIMENSION_MISMATCH",
+  EMBEDDING_NO_SOURCE: "EMBEDDING_NO_SOURCE",
+  EMBEDDING_CONFIG_INVALID: "EMBEDDING_CONFIG_INVALID",
+  DOC_MISSING_REQUIRED_FIELD: "DOC_MISSING_REQUIRED_FIELD",
+  CONFIG_INVALID: "CONFIG_INVALID",
+  QUERY_ROUTING_FAILED: "QUERY_ROUTING_FAILED",
+  QUERY_PARTIAL_FAILURE: "QUERY_PARTIAL_FAILURE",
+  QUERY_NODE_TIMEOUT: "QUERY_NODE_TIMEOUT",
+  QUERY_NO_ACTIVE_REPLICA: "QUERY_NO_ACTIVE_REPLICA",
+  ALLOCATION_NO_DATA_NODES: "ALLOCATION_NO_DATA_NODES",
+  ALLOCATION_INVALID_CONFIG: "ALLOCATION_INVALID_CONFIG",
+  ALLOCATION_FAILED: "ALLOCATION_FAILED",
+  CONTROLLER_LEASE_LOST: "CONTROLLER_LEASE_LOST",
+  CONTROLLER_NOT_ACTIVE: "CONTROLLER_NOT_ACTIVE",
+  CONTROLLER_METADATA_INVALID: "CONTROLLER_METADATA_INVALID",
+  NODE_BOOTSTRAP_FAILED: "NODE_BOOTSTRAP_FAILED",
+  NODE_ALREADY_JOINED: "NODE_ALREADY_JOINED",
+  NODE_NOT_JOINED: "NODE_NOT_JOINED",
+  COORDINATOR_DEPENDENCY_MISSING: "COORDINATOR_DEPENDENCY_MISSING",
+  REPLICATION_ENTRY_INVALID: "REPLICATION_ENTRY_INVALID",
+  REPLICATION_INSYNC_REMOVAL_FAILED: "REPLICATION_INSYNC_REMOVAL_FAILED",
+  REPLICATION_ROLLBACK_FAILED: "REPLICATION_ROLLBACK_FAILED",
+  SNAPSHOT_SYNC_UNAUTHORIZED: "SNAPSHOT_SYNC_UNAUTHORIZED",
+  SNAPSHOT_SYNC_REQUEST_INVALID: "SNAPSHOT_SYNC_REQUEST_INVALID",
+  SNAPSHOT_SYNC_INDEX_NOT_FOUND: "SNAPSHOT_SYNC_INDEX_NOT_FOUND",
+  SNAPSHOT_SYNC_TOO_LARGE: "SNAPSHOT_SYNC_TOO_LARGE",
+  SNAPSHOT_SYNC_CAPACITY_EXHAUSTED: "SNAPSHOT_SYNC_CAPACITY_EXHAUSTED",
+  SNAPSHOT_SYNC_SNAPSHOT_FAILED: "SNAPSHOT_SYNC_SNAPSHOT_FAILED",
+  SNAPSHOT_SYNC_DECODE_FAILED: "SNAPSHOT_SYNC_DECODE_FAILED",
+  SNAPSHOT_SYNC_FRAME_INVALID: "SNAPSHOT_SYNC_FRAME_INVALID",
+  SNAPSHOT_SYNC_HEADER_INVALID: "SNAPSHOT_SYNC_HEADER_INVALID",
+  SNAPSHOT_SYNC_HEADER_MISMATCH: "SNAPSHOT_SYNC_HEADER_MISMATCH",
+  SNAPSHOT_SYNC_CHUNK_OUT_OF_ORDER: "SNAPSHOT_SYNC_CHUNK_OUT_OF_ORDER",
+  SNAPSHOT_SYNC_CHUNK_OVERFLOW: "SNAPSHOT_SYNC_CHUNK_OVERFLOW",
+  SNAPSHOT_SYNC_CHUNK_SIZE_EXCEEDED: "SNAPSHOT_SYNC_CHUNK_SIZE_EXCEEDED",
+  SNAPSHOT_SYNC_CHUNK_MISSING: "SNAPSHOT_SYNC_CHUNK_MISSING",
+  SNAPSHOT_SYNC_END_MISSING: "SNAPSHOT_SYNC_END_MISSING",
+  SNAPSHOT_SYNC_CHECKSUM_MISMATCH: "SNAPSHOT_SYNC_CHECKSUM_MISMATCH",
+  SNAPSHOT_SYNC_PRIMARY_ERROR: "SNAPSHOT_SYNC_PRIMARY_ERROR",
+  SNAPSHOT_SYNC_NO_TARGETS: "SNAPSHOT_SYNC_NO_TARGETS",
+  SNAPSHOT_SYNC_TRANSPORT_FAILED: "SNAPSHOT_SYNC_TRANSPORT_FAILED",
+  SNAPSHOT_SYNC_SCHEMA_UNAVAILABLE: "SNAPSHOT_SYNC_SCHEMA_UNAVAILABLE",
+  SNAPSHOT_SYNC_RESTORE_FAILED: "SNAPSHOT_SYNC_RESTORE_FAILED",
+  SNAPSHOT_SYNC_RESTORE_CLEANUP_FAILED: "SNAPSHOT_SYNC_RESTORE_CLEANUP_FAILED",
+  SNAPSHOT_SYNC_TIMEOUT: "SNAPSHOT_SYNC_TIMEOUT",
+  SNAPSHOT_SYNC_ALLOCATION_UNAVAILABLE: "SNAPSHOT_SYNC_ALLOCATION_UNAVAILABLE",
+  SNAPSHOT_SYNC_NOT_ASSIGNED: "SNAPSHOT_SYNC_NOT_ASSIGNED",
+  SNAPSHOT_SYNC_ABORTED: "SNAPSHOT_SYNC_ABORTED"
+};
+var NarsilError = class extends Error {
+  code;
+  details;
+  constructor(code, message, details) {
+    super(message);
+    this.name = "NarsilError";
+    this.code = code;
+    this.details = details ?? {};
+  }
+};
+function createNarsilError(code, message, details) {
+  return new NarsilError(code, message, details);
+}
+
+export { ErrorCodes, NarsilError, createNarsilError };