@statewalker/indexer-mem 0.1.2 → 0.1.3

package/README.md

@@ -1,37 +1,57 @@
 # @statewalker/indexer-mem
 
-In-memory vector sub-index (`MemVectorIndex`) implementing the `EmbeddingIndex` contract from `@statewalker/indexer-api`. Consumed by the FTS-backed mem indexers (`indexer-mem-flexsearch`, `indexer-mem-minisearch`), which combine it with a full-text sub-index via `@statewalker/indexer-core`'s composite-index factory.
+## What it is
 
-## Usage
+An in-memory vector sub-index, `MemVectorIndex`, and `memVectorProvider`, the `VectorProvider` that creates it. Search is a brute-force cosine scan over all stored embeddings. The index can save itself as one Apache Arrow IPC stream.
 
-This package is a piece of scaffolding — consumers should reach for the concrete indexer packages:
+## Why it exists
+
+The in-memory indexers (`@statewalker/indexer-mem-flexsearch`, `@statewalker/indexer-mem-minisearch`) differ only in their full-text engine. Both use this package for the vector side, so the vector code and its file format exist once.
+
+## How to use
+
+```sh
+pnpm add @statewalker/indexer-mem
+```
+
+One entry point, `@statewalker/indexer-mem`: `MemVectorIndex`, `memVectorProvider`. Runs anywhere (browser, Node, workers).
+
+Most applications use `createFlexSearchIndexer` or `createMiniSearchIndexer`, which already include `memVectorProvider`. Use this package directly when you build your own indexer from providers.
+
+## Examples
+
+Combine with a full-text provider in a custom indexer:
 
 ```ts
-import { createFlexSearchIndexer } from "@statewalker/indexer-mem-flexsearch";
-// or
-import { createMiniSearchIndexer } from "@statewalker/indexer-mem-minisearch";
-
-const indexer = createFlexSearchIndexer();
-const index = await indexer.createIndex({
-  name: "docs",
-  fulltext: { language: "en" },
-  vector: { dimensionality: 384, model: "all-MiniLM-L6-v2" },
+import { createPersistenceBackedIndexer } from "@statewalker/indexer-core";
+import { memVectorProvider } from "@statewalker/indexer-mem";
+import { flexSearchFullTextProvider } from "@statewalker/indexer-mem-flexsearch";
+
+const indexer = createPersistenceBackedIndexer({
+  providers: [flexSearchFullTextProvider, memVectorProvider],
 });
+```
+
+Use a `MemVectorIndex` on its own:
 
-await index.addDocument([
-  { path: "/docs/a", blockId: "1", content: "hello world", embedding: queryEmbedding },
-]);
+```ts
+import { MemVectorIndex } from "@statewalker/indexer-mem";
 
-for await (const hit of index.search({ queries: ["hello"], embeddings: [queryEmbedding], topK: 10 })) {
-  console.log(hit);
+const vectors = new MemVectorIndex({ dimensionality: 3, model: "test" });
+await vectors.addDocument([{ path: "/a", blockId: "1", embedding: new Float32Array([1, 0, 0]) }]);
+for await (const hit of vectors.search({ embeddings: [new Float32Array([1, 0, 0])], topK: 5 })) {
+  console.log(hit.path, hit.blockId, hit.score); // cosine similarity
 }
 ```
 
-## Exports
+## Internals
 
-- `MemVectorIndex` — in-memory vector sub-index with brute-force cosine search and Arrow IPC serialization (used as the vector side of both mem-flexsearch and mem-minisearch).
+- **Brute force, on purpose.** Every search computes cosine similarity against every stored vector. There is no ANN index, so cost grows linearly with the number of blocks. For large collections use `@statewalker/indexer-duckdb` or `@statewalker/indexer-pglite` (HNSW).
+- **Persistence.** `serialise()` yields one entry named `arrow`: an Arrow IPC stream with columns `path`, `blockId`, `embedding` (`FixedSizeList<Float32>[dimensionality]`) and `metadata` (JSON text). The bytes are cached until the next change, so repeated `flush()` calls without writes do not re-encode. `loadFrom` ignores entries with other names.
+- **Several query embeddings**: each block keeps its best similarity across them.
+- **Dimensionality is checked** on `addDocument` and `search`: `Expected dimensionality 384, got 768`.
+- Dependencies: `@uwdata/flechette` (Arrow encoding without the full Apache Arrow library), `@statewalker/indexer-api`, `@statewalker/indexer-vector`, `@statewalker/indexer-core`.
 
-## Related
+## License
 
-- `@statewalker/indexer-api` — public types and contract
-- `@statewalker/indexer-mem-flexsearch` / `@statewalker/indexer-mem-minisearch` — end-user factories combining `MemVectorIndex` with a full-text backend
+MIT

package/package.json

@@ -1,10 +1,13 @@
 {
   "name": "@statewalker/indexer-mem",
-  "version": "0.1.2",
+  "version": "0.1.3",
   "private": false,
   "type": "module",
-  "description": "In-memory base implementation of @statewalker/indexer-api. Reusable scaffold for full-text and vector in-memory indexers.",
-  "homepage": "https://github.com/statewalker/statewalker-indexer",
+  "description": "In-memory vector sub-index (MemVectorIndex, brute-force cosine, Arrow persistence) and its VectorProvider.",
+  "bugs": {
+    "url": "https://github.com/statewalker/statewalker-search/issues"
+  },
+  "homepage": "https://github.com/statewalker/statewalker-search/tree/main/packages/indexer-mem#readme",
   "author": {
     "name": "Mikhail Kotelnikov",
     "email": "mikhail.kotelnikov@gmail.com"
@@ -12,7 +15,8 @@
   "license": "MIT",
   "repository": {
     "type": "git",
-    "url": "git+ssh://git@github.com/statewalker/statewalker-indexer.git"
+    "url": "git+https://github.com/statewalker/statewalker-search.git",
+    "directory": "packages/indexer-mem"
   },
   "exports": {
     ".": {
@@ -28,9 +32,9 @@
   ],
   "dependencies": {
     "@uwdata/flechette": "^2.5.0",
-    "@statewalker/indexer-api": "^0.1.1",
-    "@statewalker/indexer-vector": "^0.1.1",
-    "@statewalker/indexer-core": "^0.1.1"
+    "@statewalker/indexer-api": "^0.1.2",
+    "@statewalker/indexer-core": "^0.1.2",
+    "@statewalker/indexer-vector": "^0.1.2"
   },
   "devDependencies": {
     "rimraf": "^6.1.3",