@delali/narsil 0.2.4-next.55 → 0.2.4-next.59
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +39 -28
- package/dist/adapters/filesystem.mjs +1 -1
- package/dist/adapters/indexeddb.mjs +1 -1
- package/dist/{capabilities-DlYw0Ov0.d.ts → capabilities-DSsOGz6F.d.ts} +2 -2
- package/dist/{chunk-BH5SYB5W.mjs → chunk-24RXHMEL.mjs} +604 -150
- package/dist/{chunk-D2S642IV.mjs → chunk-3LHGNRYE.mjs} +1 -1
- package/dist/{chunk-TVT5V4E7.mjs → chunk-B2TFROFX.mjs} +299 -168
- package/dist/{chunk-L225SHNB.mjs → chunk-L6TNPLAR.mjs} +19 -5
- package/dist/{chunk-U2AL3SON.mjs → chunk-THYRHHO2.mjs} +3 -3
- package/dist/{chunk-NTZYA3SJ.mjs → chunk-U7KYW4BA.mjs} +3 -3
- package/dist/{chunk-FRQVMFY7.mjs → chunk-USKFU7J5.mjs} +1 -0
- package/dist/{chunk-4NVPQ6S5.mjs → chunk-UXNWQZST.mjs} +12 -3
- package/dist/{chunk-RGS7FMCB.mjs → chunk-VMRR3DQU.mjs} +74 -16
- package/dist/{chunk-2GWVQWGA.mjs → chunk-YMZCMH5M.mjs} +45 -1
- package/dist/{chunk-UTOFOJOR.mjs → chunk-YNVIUGF5.mjs} +6 -2
- package/dist/{chunk-5T6AOR37.mjs → chunk-ZNZBXMP5.mjs} +601 -60
- package/dist/{chunk-XNQAIO74.mjs → chunk-ZZT3EAUC.mjs} +177 -309
- package/dist/{client-CQUqt9Ir.d.ts → client-BQK6Tf6A.d.ts} +3 -3
- package/dist/client.d.ts +8 -8
- package/dist/client.mjs +1 -0
- package/dist/{config-CNas6wJH.d.ts → config-BhldZgaK.d.ts} +2 -2
- package/dist/{config-bXxnJrQc.d.ts → config-DmXULPa-.d.ts} +15 -7
- package/dist/direct-executor-ALAVE7FC.mjs +12 -0
- package/dist/distribution/coordinator/etcd.d.ts +2 -2
- package/dist/distribution/coordinator/etcd.mjs +2 -2
- package/dist/distribution/coordinator/in-memory.d.ts +2 -2
- package/dist/distribution/coordinator/in-memory.mjs +1 -1
- package/dist/distribution/transport/grpc.mjs +1 -1
- package/dist/distribution.d.ts +8 -8
- package/dist/distribution.mjs +382 -162
- package/dist/embeddings/openai.mjs +2 -2
- package/dist/{engine-Dj4I91Jv.d.ts → engine-CIOyFozK.d.ts} +15 -12
- package/dist/index.browser.mjs +1565 -528
- package/dist/index.d.ts +7 -7
- package/dist/index.mjs +11 -11
- package/dist/invalidation/filesystem.mjs +1 -1
- package/dist/{memory-xrXOZMbw.d.ts → memory-yFaVn2xZ.d.ts} +199 -107
- package/dist/persistence/durability/checkpoint-worker.d.ts +1 -1
- package/dist/persistence/durability/checkpoint-worker.mjs +11 -11
- package/dist/react.d.ts +6 -6
- package/dist/react.mjs +1 -0
- package/dist/{schema-mLpXZBEh.d.ts → schema-d5Afu03A.d.ts} +7 -2
- package/dist/server.d.ts +10 -10
- package/dist/server.mjs +10 -10
- package/dist/{tasks-Df73Tjgq.d.ts → tasks-CH2Dod8w.d.ts} +1 -1
- package/dist/{thread-5UP6MGED.mjs → thread-DYHEQRVY.mjs} +14 -10
- package/dist/{types-BTIVO8DI.d.ts → types-DAK7mnUZ.d.ts} +1 -1
- package/dist/vector/search-worker.d.ts +1 -1
- package/dist/vector/search-worker.mjs +4 -4
- package/dist/workers/entry.mjs +4 -4
- package/package.json +9 -9
- package/dist/direct-executor-RBA5O7EN.mjs +0 -12
package/README.md
CHANGED
|
@@ -25,6 +25,7 @@ Try it in your browser at [narsil.sondelali.com/demo](https://narsil.sondelali.c
|
|
|
25
25
|
## Contents
|
|
26
26
|
|
|
27
27
|
- [Install](#install)
|
|
28
|
+
- [Add Narsil with a coding agent](#add-narsil-with-a-coding-agent)
|
|
28
29
|
- [Quick start](#quick-start)
|
|
29
30
|
- [Features](#features)
|
|
30
31
|
- [Documentation](#documentation)
|
|
@@ -42,7 +43,17 @@ pnpm add @delali/narsil
|
|
|
42
43
|
|
|
43
44
|
Narsil works in Node.js 22 or newer, and in Bun, Deno, and browsers. The [Runtime support](#runtime-support) table covers each runtime.
|
|
44
45
|
|
|
45
|
-
On an arm64 or x64 machine under macOS, Linux, or Windows, the package manager also installs Narsil's native search core as an optional dependency,
|
|
46
|
+
On an arm64 or x64 machine under macOS, Linux, or Windows, the package manager also installs Narsil's native search core as an optional dependency, so the engine searches vector graphs through it. If you omit optional dependencies, Narsil searches those graphs through WebAssembly, with the same results, as it always does in a browser.
|
|
47
|
+
|
|
48
|
+
## Add Narsil with a coding agent
|
|
49
|
+
|
|
50
|
+
The [Narsil skill](https://github.com/assetcorp/narsil/tree/main/skills/narsil) tells a coding agent whether to embed Narsil or serve it, which packages to install alongside it, and which server settings keep a public address safe. Install the skill into the coding agents on your machine with the skills CLI:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
npx skills add assetcorp/narsil
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
The skill has the agent read the Narsil version that your project installs, and it sends the agent to that version's types and documentation. On the [agents page](https://narsil.sondelali.com/agents), you can copy a prompt that sets a coding agent to work on one of six search tasks.
|
|
46
57
|
|
|
47
58
|
## Quick start
|
|
48
59
|
|
|
@@ -85,7 +96,7 @@ const results = await narsil.query('products', {
|
|
|
85
96
|
})
|
|
86
97
|
```
|
|
87
98
|
|
|
88
|
-
Every hit holds the document, its id, and its BM25 score. Beside the hits, `results.count` holds the total number of matching documents, `results.elapsed` holds the query time in milliseconds, and `results.coverage` counts the partitions
|
|
99
|
+
Every hit holds the document, its id, and its BM25 score. Beside the hits, `results.count` holds the total number of matching documents, `results.elapsed` holds the query time in milliseconds, and `results.coverage` counts the partitions behind the answer, so you can tell when a cluster answers from part of its data.
|
|
89
100
|
|
|
90
101
|
## Features
|
|
91
102
|
|
|
@@ -99,43 +110,43 @@ Every hit holds the document, its id, and its BM25 score. Beside the hits, `resu
|
|
|
99
110
|
|
|
100
111
|
**Scale.** The engine routes each document to a [partition](https://github.com/assetcorp/narsil/blob/main/docs/partitions-and-workers.md) by a deterministic hash. When you call `rebalance()`, the engine reshapes the partitions online and buffers incoming writes in a write-ahead queue. Once an index holds 1,000 documents, the engine serves keyword queries from worker copies on half its worker threads, while the HTTP server receives requests on the threads that hold those copies.
|
|
101
112
|
|
|
102
|
-
**Operations.** The [HTTP server](https://github.com/assetcorp/narsil/blob/main/docs/http-server.md) subpath wraps an engine in a REST API with health probes, bulk NDJSON import, snapshot and restore endpoints, and task-based long operations. The [client](https://github.com/assetcorp/narsil/blob/main/docs/client.md) subpath calls every one of those routes from a browser or from Node under the engine's own method names. The [React](https://github.com/assetcorp/narsil/blob/main/docs/react.md) subpath exposes those methods to components as hooks. For observability, you
|
|
113
|
+
**Operations.** The [HTTP server](https://github.com/assetcorp/narsil/blob/main/docs/http-server.md) subpath wraps an engine in a REST API with health probes, bulk NDJSON import, snapshot and restore endpoints, and task-based long operations. The [client](https://github.com/assetcorp/narsil/blob/main/docs/client.md) subpath calls every one of those routes from a browser or from Node under the engine's own method names. The [React](https://github.com/assetcorp/narsil/blob/main/docs/react.md) subpath exposes those methods to components as hooks. For observability, you have [events, plugins, and memory reporting](https://github.com/assetcorp/narsil/blob/main/docs/observability.md). [Language modules](https://github.com/assetcorp/narsil/blob/main/docs/language-support.md) cover 107 languages as separate entry points, including 20 African languages.
|
|
103
114
|
|
|
104
115
|
## Documentation
|
|
105
116
|
|
|
106
117
|
| Guide | What it covers |
|
|
107
118
|
| --- | --- |
|
|
108
|
-
| [Configuration](
|
|
109
|
-
| [Indexes and documents](
|
|
110
|
-
| [Full-text search](
|
|
111
|
-
| [Filters, facets, and pagination](
|
|
112
|
-
| [Vector search](
|
|
113
|
-
| [Hybrid search](
|
|
114
|
-
| [Geosearch](
|
|
115
|
-
| [Embedding adapters](
|
|
116
|
-
| [Persistence and durability](
|
|
117
|
-
| [Partitions and workers](
|
|
118
|
-
| [Language support](
|
|
119
|
-
| [HTTP server](
|
|
120
|
-
| [Cluster mode](
|
|
121
|
-
| [Client](
|
|
122
|
-
| [React](
|
|
123
|
-
| [Observability](
|
|
124
|
-
| [Errors](
|
|
125
|
-
|
|
126
|
-
The [specification](
|
|
119
|
+
| [Configuration](https://github.com/assetcorp/narsil/blob/main/docs/configuration.md) | Every `createNarsil` option, worker tuning, analysis rebuilds, and the tokenizer cache |
|
|
120
|
+
| [Indexes and documents](https://github.com/assetcorp/narsil/blob/main/docs/indexes-and-documents.md) | Schemas, index management, inserts, reads, updates, removals, and batch operations |
|
|
121
|
+
| [Full-text search](https://github.com/assetcorp/narsil/blob/main/docs/full-text-search.md) | Term queries, fuzzy matching, prefix completion, thresholds, highlighting, scoring modes, and suggestions |
|
|
122
|
+
| [Filters, facets, and pagination](https://github.com/assetcorp/narsil/blob/main/docs/filters-facets-and-pagination.md) | Field, array, presence, and geo filters, facet counts, sorting, grouping, cursors, and pinning |
|
|
123
|
+
| [Vector search](https://github.com/assetcorp/narsil/blob/main/docs/vector-search.md) | Vector fields, distance metrics, HNSW promotion, quantization, the native search core, and graph maintenance |
|
|
124
|
+
| [Hybrid search](https://github.com/assetcorp/narsil/blob/main/docs/hybrid-search.md) | Reciprocal rank fusion and linear blending of text and vector rankings |
|
|
125
|
+
| [Geosearch](https://github.com/assetcorp/narsil/blob/main/docs/geosearch.md) | Radius and polygon filters, and the two distance formulas |
|
|
126
|
+
| [Embedding adapters](https://github.com/assetcorp/narsil/blob/main/docs/embedding-adapters.md) | Automatic embedding on insert and query, named adapters, the bundled ones, and custom ones |
|
|
127
|
+
| [Persistence and durability](https://github.com/assetcorp/narsil/blob/main/docs/persistence-and-durability.md) | Storage backends, the write-ahead log, checkpoints, recovery, and snapshots |
|
|
128
|
+
| [Partitions and workers](https://github.com/assetcorp/narsil/blob/main/docs/partitions-and-workers.md) | Partition routing, online rebalancing, worker copies, and multi-instance invalidation |
|
|
129
|
+
| [Language support](https://github.com/assetcorp/narsil/blob/main/docs/language-support.md) | The 107 language modules, analysis revisions and rebuilds, and named tokenizers and stop words |
|
|
130
|
+
| [HTTP server](https://github.com/assetcorp/narsil/blob/main/docs/http-server.md) | Wrapping an engine in a REST API, every route it serves, and long-running tasks |
|
|
131
|
+
| [Cluster mode](https://github.com/assetcorp/narsil/blob/main/docs/cluster.md) | Multi-node indexes: nodes and roles, replication, routed writes, distributed searches and reads, and the calls that fail in a cluster |
|
|
132
|
+
| [Client](https://github.com/assetcorp/narsil/blob/main/docs/client.md) | Reaching a server from a browser or Node, following a task, and the codes it raises |
|
|
133
|
+
| [React](https://github.com/assetcorp/narsil/blob/main/docs/react.md) | The hooks over the client, one shared request per key, and loading a corpus from a component |
|
|
134
|
+
| [Observability](https://github.com/assetcorp/narsil/blob/main/docs/observability.md) | Plugin hooks, engine events, and memory reporting |
|
|
135
|
+
| [Errors](https://github.com/assetcorp/narsil/blob/main/docs/errors.md) | The codes you handle most often, and the three sets that `NarsilError.code` covers |
|
|
136
|
+
|
|
137
|
+
The [specification](https://github.com/assetcorp/narsil/tree/main/packages/spec) defines the `.nrsl` format, the analysis pipeline, and the replication invariants that every implementation must uphold.
|
|
127
138
|
|
|
128
139
|
## Examples
|
|
129
140
|
|
|
130
141
|
| Example | What it shows |
|
|
131
142
|
| --- | --- |
|
|
132
|
-
| [HTTP server](examples/http-server/README.md) | The launcher serves the engine as a REST service with durability, API-key auth, and Docker packaging,
|
|
133
|
-
| [Browser](examples/browser/README.md) | The app embeds the engine in a browser with IndexedDB persistence and Web Worker search. |
|
|
134
|
-
| [Server app](examples/server-app/README.md) | The app calls the HTTP server through the client SDK and the React hooks to load corpora as import tasks. In its Ask view, you can ask questions about those corpora. |
|
|
143
|
+
| [HTTP server](https://github.com/assetcorp/narsil/blob/main/packages/ts/examples/http-server/README.md) | The launcher serves the engine as a REST service with durability, API-key auth, and Docker packaging, while every setting comes from the environment. |
|
|
144
|
+
| [Browser](https://github.com/assetcorp/narsil/blob/main/packages/ts/examples/browser/README.md) | The app embeds the engine in a browser with IndexedDB persistence and Web Worker search. |
|
|
145
|
+
| [Server app](https://github.com/assetcorp/narsil/blob/main/packages/ts/examples/server-app/README.md) | The app calls the HTTP server through the client SDK and the React hooks to load corpora as import tasks. In its Ask view, you can ask questions about those corpora. |
|
|
135
146
|
|
|
136
147
|
## Distribution
|
|
137
148
|
|
|
138
|
-
`@delali/narsil/distribution` holds Narsil's multi-node cluster mode: nodes and roles, replication, coordinator adapters for etcd and in-process testing, TCP and gRPC transports with mutual TLS, and distributed query routing. A cluster node can create, drop, clear, write, update, search, list, count, and suggest across every partition. The [cluster dashboard example](examples/cluster-dashboard) starts three nodes against etcd and shows the state of each partition while you cut the network links. Because the layer is experimental and we may change its APIs without notice, pin an exact version before you depend on it. The [cluster guide](
|
|
149
|
+
`@delali/narsil/distribution` holds Narsil's multi-node cluster mode: nodes and roles, replication, coordinator adapters for etcd and in-process testing, TCP and gRPC transports with mutual TLS, and distributed query routing. A cluster node can create, drop, clear, write, update, search, list, count, and suggest across every partition. The [cluster dashboard example](https://github.com/assetcorp/narsil/tree/main/packages/ts/examples/cluster-dashboard) starts three nodes against etcd and shows the state of each partition while you cut the network links. Because the layer is experimental and we may change its APIs without notice, pin an exact version before you depend on it. The [cluster guide](https://github.com/assetcorp/narsil/blob/main/docs/cluster.md) covers the API, including the packages that the etcd coordinator and the gRPC transport each need. For the contract that every implementation must uphold, read [`packages/spec/distribution`](https://github.com/assetcorp/narsil/tree/main/packages/spec/distribution).
|
|
139
150
|
|
|
140
151
|
## Search quality
|
|
141
152
|
|
|
@@ -148,7 +159,7 @@ We measure ranking quality on the [BEIR](https://github.com/beir-cellar/beir) Sc
|
|
|
148
159
|
- **MAP** tracks precision at every rank where a relevant document appears. A higher MAP means that relevant documents cluster near the top of the ranking.
|
|
149
160
|
- **MRR** measures how soon the first relevant result appears. A higher MRR means that the first relevant document appears nearer the top.
|
|
150
161
|
|
|
151
|
-
Continuous integration executes a separate [SciFact regression test](src/__tests__/relevance/scifact.test.ts) on the same corpus. That test fails the build when ranking quality drops below calibrated thresholds.
|
|
162
|
+
Continuous integration executes a separate [SciFact regression test](https://github.com/assetcorp/narsil/blob/main/packages/ts/src/__tests__/relevance/scifact.test.ts) on the same corpus. That test fails the build when ranking quality drops below calibrated thresholds.
|
|
152
163
|
|
|
153
164
|
Reproduce these scores with `pnpm --filter benchmarks bench -- --tiers relevance`. [BENCHMARKS.md](https://github.com/assetcorp/narsil/blob/main/BENCHMARKS.md) holds the full quality, throughput, and latency tables for all three engines.
|
|
154
165
|
|
|
@@ -163,7 +174,7 @@ Narsil also works as a search server. On the BEIR datasets, we compare the serve
|
|
|
163
174
|
| Deno | Web Workers | Filesystem | BroadcastChannel |
|
|
164
175
|
| Browser | Web Workers | IndexedDB | BroadcastChannel |
|
|
165
176
|
|
|
166
|
-
The [browser example](examples/browser/README.md) shows an embedded engine with IndexedDB persistence, while the [server app example](examples/server-app/README.md) shows the same interface over the client SDK and the React hooks.
|
|
177
|
+
The [browser example](https://github.com/assetcorp/narsil/blob/main/packages/ts/examples/browser/README.md) shows an embedded engine with IndexedDB persistence, while the [server app example](https://github.com/assetcorp/narsil/blob/main/packages/ts/examples/server-app/README.md) shows the same interface over the client SDK and the React hooks.
|
|
167
178
|
|
|
168
179
|
## License
|
|
169
180
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import { E as EmbeddingAdapter } from './adapters-79OIa4xm.js';
|
|
2
|
-
import { C as ClusterNamespace } from './config-
|
|
3
|
-
import { h as TaskStore } from './tasks-
|
|
2
|
+
import { C as ClusterNamespace } from './config-BhldZgaK.js';
|
|
3
|
+
import { h as TaskStore } from './tasks-CH2Dod8w.js';
|
|
4
4
|
|
|
5
5
|
/**
|
|
6
6
|
* Context handed to {@link OnRequestHook} for every inbound request, captured
|