@delali/narsil 0.2.4-next.55 → 0.2.4-next.59

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (52) hide show
  1. package/README.md +39 -28
  2. package/dist/adapters/filesystem.mjs +1 -1
  3. package/dist/adapters/indexeddb.mjs +1 -1
  4. package/dist/{capabilities-DlYw0Ov0.d.ts → capabilities-DSsOGz6F.d.ts} +2 -2
  5. package/dist/{chunk-BH5SYB5W.mjs → chunk-24RXHMEL.mjs} +604 -150
  6. package/dist/{chunk-D2S642IV.mjs → chunk-3LHGNRYE.mjs} +1 -1
  7. package/dist/{chunk-TVT5V4E7.mjs → chunk-B2TFROFX.mjs} +299 -168
  8. package/dist/{chunk-L225SHNB.mjs → chunk-L6TNPLAR.mjs} +19 -5
  9. package/dist/{chunk-U2AL3SON.mjs → chunk-THYRHHO2.mjs} +3 -3
  10. package/dist/{chunk-NTZYA3SJ.mjs → chunk-U7KYW4BA.mjs} +3 -3
  11. package/dist/{chunk-FRQVMFY7.mjs → chunk-USKFU7J5.mjs} +1 -0
  12. package/dist/{chunk-4NVPQ6S5.mjs → chunk-UXNWQZST.mjs} +12 -3
  13. package/dist/{chunk-RGS7FMCB.mjs → chunk-VMRR3DQU.mjs} +74 -16
  14. package/dist/{chunk-2GWVQWGA.mjs → chunk-YMZCMH5M.mjs} +45 -1
  15. package/dist/{chunk-UTOFOJOR.mjs → chunk-YNVIUGF5.mjs} +6 -2
  16. package/dist/{chunk-5T6AOR37.mjs → chunk-ZNZBXMP5.mjs} +601 -60
  17. package/dist/{chunk-XNQAIO74.mjs → chunk-ZZT3EAUC.mjs} +177 -309
  18. package/dist/{client-CQUqt9Ir.d.ts → client-BQK6Tf6A.d.ts} +3 -3
  19. package/dist/client.d.ts +8 -8
  20. package/dist/client.mjs +1 -0
  21. package/dist/{config-CNas6wJH.d.ts → config-BhldZgaK.d.ts} +2 -2
  22. package/dist/{config-bXxnJrQc.d.ts → config-DmXULPa-.d.ts} +15 -7
  23. package/dist/direct-executor-ALAVE7FC.mjs +12 -0
  24. package/dist/distribution/coordinator/etcd.d.ts +2 -2
  25. package/dist/distribution/coordinator/etcd.mjs +2 -2
  26. package/dist/distribution/coordinator/in-memory.d.ts +2 -2
  27. package/dist/distribution/coordinator/in-memory.mjs +1 -1
  28. package/dist/distribution/transport/grpc.mjs +1 -1
  29. package/dist/distribution.d.ts +8 -8
  30. package/dist/distribution.mjs +382 -162
  31. package/dist/embeddings/openai.mjs +2 -2
  32. package/dist/{engine-Dj4I91Jv.d.ts → engine-CIOyFozK.d.ts} +15 -12
  33. package/dist/index.browser.mjs +1565 -528
  34. package/dist/index.d.ts +7 -7
  35. package/dist/index.mjs +11 -11
  36. package/dist/invalidation/filesystem.mjs +1 -1
  37. package/dist/{memory-xrXOZMbw.d.ts → memory-yFaVn2xZ.d.ts} +199 -107
  38. package/dist/persistence/durability/checkpoint-worker.d.ts +1 -1
  39. package/dist/persistence/durability/checkpoint-worker.mjs +11 -11
  40. package/dist/react.d.ts +6 -6
  41. package/dist/react.mjs +1 -0
  42. package/dist/{schema-mLpXZBEh.d.ts → schema-d5Afu03A.d.ts} +7 -2
  43. package/dist/server.d.ts +10 -10
  44. package/dist/server.mjs +10 -10
  45. package/dist/{tasks-Df73Tjgq.d.ts → tasks-CH2Dod8w.d.ts} +1 -1
  46. package/dist/{thread-5UP6MGED.mjs → thread-DYHEQRVY.mjs} +14 -10
  47. package/dist/{types-BTIVO8DI.d.ts → types-DAK7mnUZ.d.ts} +1 -1
  48. package/dist/vector/search-worker.d.ts +1 -1
  49. package/dist/vector/search-worker.mjs +4 -4
  50. package/dist/workers/entry.mjs +4 -4
  51. package/package.json +9 -9
  52. package/dist/direct-executor-RBA5O7EN.mjs +0 -12
package/README.md CHANGED
@@ -25,6 +25,7 @@ Try it in your browser at [narsil.sondelali.com/demo](https://narsil.sondelali.c
25
25
  ## Contents
26
26
 
27
27
  - [Install](#install)
28
+ - [Add Narsil with a coding agent](#add-narsil-with-a-coding-agent)
28
29
  - [Quick start](#quick-start)
29
30
  - [Features](#features)
30
31
  - [Documentation](#documentation)
@@ -42,7 +43,17 @@ pnpm add @delali/narsil
42
43
 
43
44
  Narsil works in Node.js 22 or newer, and in Bun, Deno, and browsers. The [Runtime support](#runtime-support) table covers each runtime.
44
45
 
45
- On an arm64 or x64 machine under macOS, Linux, or Windows, the package manager also installs Narsil's native search core as an optional dependency, and the engine searches vector graphs through it. If you omit optional dependencies, Narsil searches those graphs through WebAssembly, with the same results, as it always does in a browser.
46
+ On an arm64 or x64 machine under macOS, Linux, or Windows, the package manager also installs Narsil's native search core as an optional dependency, so the engine searches vector graphs through it. If you omit optional dependencies, Narsil searches those graphs through WebAssembly, with the same results, as it always does in a browser.
47
+
48
+ ## Add Narsil with a coding agent
49
+
50
+ The [Narsil skill](https://github.com/assetcorp/narsil/tree/main/skills/narsil) tells a coding agent whether to embed Narsil or serve it, which packages to install alongside it, and which server settings keep a public address safe. Install the skill into the coding agents on your machine with the skills CLI:
51
+
52
+ ```bash
53
+ npx skills add assetcorp/narsil
54
+ ```
55
+
56
+ The skill has the agent read the Narsil version that your project installs, and it sends the agent to that version's types and documentation. On the [agents page](https://narsil.sondelali.com/agents), you can copy a prompt that sets a coding agent to work on one of six search tasks.
46
57
 
47
58
  ## Quick start
48
59
 
@@ -85,7 +96,7 @@ const results = await narsil.query('products', {
85
96
  })
86
97
  ```
87
98
 
88
- Every hit holds the document, its id, and its BM25 score. Beside the hits, `results.count` holds the total number of matching documents, `results.elapsed` holds the query time in milliseconds, and `results.coverage` counts the partitions that the search covered, so you can tell when a cluster returns results from part of its data.
99
+ Every hit holds the document, its id, and its BM25 score. Beside the hits, `results.count` holds the total number of matching documents, `results.elapsed` holds the query time in milliseconds, and `results.coverage` counts the partitions behind the answer, so you can tell when a cluster answers from part of its data.
89
100
 
90
101
  ## Features
91
102
 
@@ -99,43 +110,43 @@ Every hit holds the document, its id, and its BM25 score. Beside the hits, `resu
99
110
 
100
111
  **Scale.** The engine routes each document to a [partition](https://github.com/assetcorp/narsil/blob/main/docs/partitions-and-workers.md) by a deterministic hash. When you call `rebalance()`, the engine reshapes the partitions online and buffers incoming writes in a write-ahead queue. Once an index holds 1,000 documents, the engine serves keyword queries from worker copies on half its worker threads, while the HTTP server receives requests on the threads that hold those copies.
101
112
 
102
- **Operations.** The [HTTP server](https://github.com/assetcorp/narsil/blob/main/docs/http-server.md) subpath wraps an engine in a REST API with health probes, bulk NDJSON import, snapshot and restore endpoints, and task-based long operations. The [client](https://github.com/assetcorp/narsil/blob/main/docs/client.md) subpath calls every one of those routes from a browser or from Node under the engine's own method names. The [React](https://github.com/assetcorp/narsil/blob/main/docs/react.md) subpath exposes those methods to components as hooks. For observability, you get [events, plugins, and memory reporting](https://github.com/assetcorp/narsil/blob/main/docs/observability.md). [Language modules](https://github.com/assetcorp/narsil/blob/main/docs/language-support.md) cover 107 languages as separate entry points, including 20 African languages.
113
+ **Operations.** The [HTTP server](https://github.com/assetcorp/narsil/blob/main/docs/http-server.md) subpath wraps an engine in a REST API with health probes, bulk NDJSON import, snapshot and restore endpoints, and task-based long operations. The [client](https://github.com/assetcorp/narsil/blob/main/docs/client.md) subpath calls every one of those routes from a browser or from Node under the engine's own method names. The [React](https://github.com/assetcorp/narsil/blob/main/docs/react.md) subpath exposes those methods to components as hooks. For observability, you have [events, plugins, and memory reporting](https://github.com/assetcorp/narsil/blob/main/docs/observability.md). [Language modules](https://github.com/assetcorp/narsil/blob/main/docs/language-support.md) cover 107 languages as separate entry points, including 20 African languages.
103
114
 
104
115
  ## Documentation
105
116
 
106
117
  | Guide | What it covers |
107
118
  | --- | --- |
108
- | [Configuration](../../docs/configuration.md) | Every `createNarsil` option, worker tuning, analysis rebuilds, and the tokenizer cache |
109
- | [Indexes and documents](../../docs/indexes-and-documents.md) | Schemas, index management, inserts, reads, updates, removals, and batch operations |
110
- | [Full-text search](../../docs/full-text-search.md) | Term queries, fuzzy matching, prefix completion, thresholds, highlighting, scoring modes, and suggestions |
111
- | [Filters, facets, and pagination](../../docs/filters-facets-and-pagination.md) | Field, array, presence, and geo filters, facet counts, sorting, grouping, cursors, and pinning |
112
- | [Vector search](../../docs/vector-search.md) | Vector fields, distance metrics, HNSW promotion, quantization, the native search core, and graph maintenance |
113
- | [Hybrid search](../../docs/hybrid-search.md) | Reciprocal rank fusion and linear blending of text and vector rankings |
114
- | [Geosearch](../../docs/geosearch.md) | Radius and polygon filters, and the two distance formulas |
115
- | [Embedding adapters](../../docs/embedding-adapters.md) | Automatic embedding on insert and query, named adapters, the bundled ones, and custom ones |
116
- | [Persistence and durability](../../docs/persistence-and-durability.md) | Storage backends, the write-ahead log, checkpoints, recovery, and snapshots |
117
- | [Partitions and workers](../../docs/partitions-and-workers.md) | Partition routing, online rebalancing, worker copies, and multi-instance invalidation |
118
- | [Language support](../../docs/language-support.md) | The 107 language modules, analysis revisions and rebuilds, and named tokenizers and stop words |
119
- | [HTTP server](../../docs/http-server.md) | Wrapping an engine in a REST API, every route it serves, and long-running tasks |
120
- | [Cluster mode](../../docs/cluster.md) | Multi-node indexes: nodes and roles, replication, routed writes, distributed searches and reads, and the calls that fail in a cluster |
121
- | [Client](../../docs/client.md) | Reaching a server from a browser or Node, following a task, and the codes it raises |
122
- | [React](../../docs/react.md) | The hooks over the client, one shared request per key, and loading a corpus from a component |
123
- | [Observability](../../docs/observability.md) | Plugin hooks, engine events, and memory reporting |
124
- | [Errors](../../docs/errors.md) | Every error code and what throws it |
125
-
126
- The [specification](../spec/) defines the `.nrsl` format, the analysis pipeline, and the replication invariants that every implementation must uphold.
119
+ | [Configuration](https://github.com/assetcorp/narsil/blob/main/docs/configuration.md) | Every `createNarsil` option, worker tuning, analysis rebuilds, and the tokenizer cache |
120
+ | [Indexes and documents](https://github.com/assetcorp/narsil/blob/main/docs/indexes-and-documents.md) | Schemas, index management, inserts, reads, updates, removals, and batch operations |
121
+ | [Full-text search](https://github.com/assetcorp/narsil/blob/main/docs/full-text-search.md) | Term queries, fuzzy matching, prefix completion, thresholds, highlighting, scoring modes, and suggestions |
122
+ | [Filters, facets, and pagination](https://github.com/assetcorp/narsil/blob/main/docs/filters-facets-and-pagination.md) | Field, array, presence, and geo filters, facet counts, sorting, grouping, cursors, and pinning |
123
+ | [Vector search](https://github.com/assetcorp/narsil/blob/main/docs/vector-search.md) | Vector fields, distance metrics, HNSW promotion, quantization, the native search core, and graph maintenance |
124
+ | [Hybrid search](https://github.com/assetcorp/narsil/blob/main/docs/hybrid-search.md) | Reciprocal rank fusion and linear blending of text and vector rankings |
125
+ | [Geosearch](https://github.com/assetcorp/narsil/blob/main/docs/geosearch.md) | Radius and polygon filters, and the two distance formulas |
126
+ | [Embedding adapters](https://github.com/assetcorp/narsil/blob/main/docs/embedding-adapters.md) | Automatic embedding on insert and query, named adapters, the bundled ones, and custom ones |
127
+ | [Persistence and durability](https://github.com/assetcorp/narsil/blob/main/docs/persistence-and-durability.md) | Storage backends, the write-ahead log, checkpoints, recovery, and snapshots |
128
+ | [Partitions and workers](https://github.com/assetcorp/narsil/blob/main/docs/partitions-and-workers.md) | Partition routing, online rebalancing, worker copies, and multi-instance invalidation |
129
+ | [Language support](https://github.com/assetcorp/narsil/blob/main/docs/language-support.md) | The 107 language modules, analysis revisions and rebuilds, and named tokenizers and stop words |
130
+ | [HTTP server](https://github.com/assetcorp/narsil/blob/main/docs/http-server.md) | Wrapping an engine in a REST API, every route it serves, and long-running tasks |
131
+ | [Cluster mode](https://github.com/assetcorp/narsil/blob/main/docs/cluster.md) | Multi-node indexes: nodes and roles, replication, routed writes, distributed searches and reads, and the calls that fail in a cluster |
132
+ | [Client](https://github.com/assetcorp/narsil/blob/main/docs/client.md) | Reaching a server from a browser or Node, following a task, and the codes it raises |
133
+ | [React](https://github.com/assetcorp/narsil/blob/main/docs/react.md) | The hooks over the client, one shared request per key, and loading a corpus from a component |
134
+ | [Observability](https://github.com/assetcorp/narsil/blob/main/docs/observability.md) | Plugin hooks, engine events, and memory reporting |
135
+ | [Errors](https://github.com/assetcorp/narsil/blob/main/docs/errors.md) | The codes you handle most often, and the three sets that `NarsilError.code` covers |
136
+
137
+ The [specification](https://github.com/assetcorp/narsil/tree/main/packages/spec) defines the `.nrsl` format, the analysis pipeline, and the replication invariants that every implementation must uphold.
127
138
 
128
139
  ## Examples
129
140
 
130
141
  | Example | What it shows |
131
142
  | --- | --- |
132
- | [HTTP server](examples/http-server/README.md) | The launcher serves the engine as a REST service with durability, API-key auth, and Docker packaging, and every setting comes from the environment. |
133
- | [Browser](examples/browser/README.md) | The app embeds the engine in a browser with IndexedDB persistence and Web Worker search. |
134
- | [Server app](examples/server-app/README.md) | The app calls the HTTP server through the client SDK and the React hooks to load corpora as import tasks. In its Ask view, you can ask questions about those corpora. |
143
+ | [HTTP server](https://github.com/assetcorp/narsil/blob/main/packages/ts/examples/http-server/README.md) | The launcher serves the engine as a REST service with durability, API-key auth, and Docker packaging, while every setting comes from the environment. |
144
+ | [Browser](https://github.com/assetcorp/narsil/blob/main/packages/ts/examples/browser/README.md) | The app embeds the engine in a browser with IndexedDB persistence and Web Worker search. |
145
+ | [Server app](https://github.com/assetcorp/narsil/blob/main/packages/ts/examples/server-app/README.md) | The app calls the HTTP server through the client SDK and the React hooks to load corpora as import tasks. In its Ask view, you can ask questions about those corpora. |
135
146
 
136
147
  ## Distribution
137
148
 
138
- `@delali/narsil/distribution` holds Narsil's multi-node cluster mode: nodes and roles, replication, coordinator adapters for etcd and in-process testing, TCP and gRPC transports with mutual TLS, and distributed query routing. A cluster node can create, drop, clear, write, update, search, list, count, and suggest across every partition. The [cluster dashboard example](examples/cluster-dashboard) starts three nodes against etcd and shows the state of each partition while you cut the network links. Because the layer is experimental and we may change its APIs without notice, pin an exact version before you depend on it. The [cluster guide](../../docs/cluster.md) covers the API. For the contract that every implementation must uphold, read [`packages/spec/distribution`](../spec/distribution).
149
+ `@delali/narsil/distribution` holds Narsil's multi-node cluster mode: nodes and roles, replication, coordinator adapters for etcd and in-process testing, TCP and gRPC transports with mutual TLS, and distributed query routing. A cluster node can create, drop, clear, write, update, search, list, count, and suggest across every partition. The [cluster dashboard example](https://github.com/assetcorp/narsil/tree/main/packages/ts/examples/cluster-dashboard) starts three nodes against etcd and shows the state of each partition while you cut the network links. Because the layer is experimental and we may change its APIs without notice, pin an exact version before you depend on it. The [cluster guide](https://github.com/assetcorp/narsil/blob/main/docs/cluster.md) covers the API, including the packages that the etcd coordinator and the gRPC transport each need. For the contract that every implementation must uphold, read [`packages/spec/distribution`](https://github.com/assetcorp/narsil/tree/main/packages/spec/distribution).
139
150
 
140
151
  ## Search quality
141
152
 
@@ -148,7 +159,7 @@ We measure ranking quality on the [BEIR](https://github.com/beir-cellar/beir) Sc
148
159
  - **MAP** tracks precision at every rank where a relevant document appears. A higher MAP means that relevant documents cluster near the top of the ranking.
149
160
  - **MRR** measures how soon the first relevant result appears. A higher MRR means that the first relevant document appears nearer the top.
150
161
 
151
- Continuous integration executes a separate [SciFact regression test](src/__tests__/relevance/scifact.test.ts) on the same corpus. That test fails the build when ranking quality drops below calibrated thresholds.
162
+ Continuous integration executes a separate [SciFact regression test](https://github.com/assetcorp/narsil/blob/main/packages/ts/src/__tests__/relevance/scifact.test.ts) on the same corpus. That test fails the build when ranking quality drops below calibrated thresholds.
152
163
 
153
164
  Reproduce these scores with `pnpm --filter benchmarks bench -- --tiers relevance`. [BENCHMARKS.md](https://github.com/assetcorp/narsil/blob/main/BENCHMARKS.md) holds the full quality, throughput, and latency tables for all three engines.
154
165
 
@@ -163,7 +174,7 @@ Narsil also works as a search server. On the BEIR datasets, we compare the serve
163
174
  | Deno | Web Workers | Filesystem | BroadcastChannel |
164
175
  | Browser | Web Workers | IndexedDB | BroadcastChannel |
165
176
 
166
- The [browser example](examples/browser/README.md) shows an embedded engine with IndexedDB persistence, while the [server app example](examples/server-app/README.md) shows the same interface over the client SDK and the React hooks.
177
+ The [browser example](https://github.com/assetcorp/narsil/blob/main/packages/ts/examples/browser/README.md) shows an embedded engine with IndexedDB persistence, while the [server app example](https://github.com/assetcorp/narsil/blob/main/packages/ts/examples/server-app/README.md) shows the same interface over the client SDK and the React hooks.
167
178
 
168
179
  ## License
169
180
 
@@ -1,4 +1,4 @@
1
- import { NarsilError, ErrorCodes } from '../chunk-FRQVMFY7.mjs';
1
+ import { NarsilError, ErrorCodes } from '../chunk-USKFU7J5.mjs';
2
2
 
3
3
  // src/persistence/filesystem.ts
4
4
  var cachedFs = null;
@@ -1,4 +1,4 @@
1
- import { NarsilError, ErrorCodes } from '../chunk-FRQVMFY7.mjs';
1
+ import { NarsilError, ErrorCodes } from '../chunk-USKFU7J5.mjs';
2
2
 
3
3
  // src/persistence/constants.ts
4
4
  var DEFAULT_INDEXEDDB_NAME = "narsil";
@@ -1,6 +1,6 @@
1
1
  import { E as EmbeddingAdapter } from './adapters-79OIa4xm.js';
2
- import { C as ClusterNamespace } from './config-CNas6wJH.js';
3
- import { h as TaskStore } from './tasks-Df73Tjgq.js';
2
+ import { C as ClusterNamespace } from './config-BhldZgaK.js';
3
+ import { h as TaskStore } from './tasks-CH2Dod8w.js';
4
4
 
5
5
  /**
6
6
  * Context handed to {@link OnRequestHook} for every inbound request, captured