@pattern-js/mod-vectors 0.4.0-main.c8eb671

package/docs/index.md

@@ -0,0 +1,94 @@
+# Vectors
+
+`@pattern-js/mod-vectors` gives your app **embedding collections**: named
+indexes of text (or raw vectors) you can search semantically, by keyword, or
+both at once — the retrieval half of RAG, as ordinary ops on the canvas.
+
+```jsonc
+{ "mods": ["@pattern-js/mod-ai", "@pattern-js/mod-vectors"] }
+```
+
+## Collections declare their model
+
+The classic silent RAG bug is indexing with one embedding model and querying
+with another — every score turns to noise and nothing errors. Pattern designs
+it out: a collection **declares its embedding alias** at creation
+(`vectors.collection.ensure`), every embed — index side and query side — goes
+through that alias, and the vector dimensionality **locks on the first
+write**. Re-point the alias at a model with different dims and the next write
+fails with an error that names the alias and the fix (re-index into a fresh
+collection), instead of silently degrading your search.
+
+```
+vectors.collection.ensure  { name: "kb", alias: "embeddings", filterables: ["product", "lang"] }
+```
+
+Aliases come from mod-ai's Settings page — `@pattern-js/mod-ai` must be
+installed for text embedding (raw-vector upsert/query works without it).
+
+## Ingesting: chunk → embed → upsert
+
+`vectors.index` is the one-node ingestion path: give it
+`docs: [{ id, text, meta? }]`, it splits each doc (recursive character
+splitter — `maxChars` 1200, `overlap` 150, paragraph/sentence separators),
+embeds through the collection's alias, and upserts chunks with ids
+`${docId}#${i}`, each carrying its doc's meta. `vectors.chunk` and
+`vectors.upsert` are the same steps as separate nodes when you want the
+pipeline visible.
+
+Writes are **content-hashed**: re-running ingestion over unchanged documents
+embeds nothing and writes nothing — only the diff pays. That's what makes
+"re-index everything on boot" a cheap, idempotent habit.
+
+## Querying: vector, keyword, hybrid
+
+`vectors.query` takes `text` (embedded via the collection's alias) or a raw
+`vector`, and ranks in one of three modes:
+
+- **`vector`** (default) — cosine similarity.
+- **`keyword`** — exact-word ranking: FTS5 when your Node's sqlite build has
+  it (probed at startup), a zero-dependency token-overlap scorer otherwise.
+- **`hybrid`** — both rankings fused with reciprocal-rank fusion (RRF). No
+  score calibration, no tuning knob; exact identifiers and paraphrased
+  questions both surface.
+
+## Filterable metadata
+
+Real corpora carry structure — products, languages, tenants, doc types.
+Declare those meta fields as `filterables` on the collection and they are
+extracted into an indexed side table at write time; `vectors.query`'s
+`filter: { field: value | values[] }` (AND of equality/any-of) then **prunes
+before any scoring touches a vector**. Filtering on an undeclared field is a
+located error naming the field and the fix — never a silently empty result.
+
+The pattern for complex structures: keep the canonical record in
+**mod-store** (the truth), index its text here with `docId` + taxonomy fields
+in meta (the index), and compose `vectors.query → store.doc.get` on the
+canvas.
+
+## The local engine, durability, and offload
+
+The default engine is honest brute force over Float32 vectors in
+`.pattern-data/vectors.db` (WAL sqlite) — no native extensions, no services,
+fine to roughly a hundred thousand vectors. Because the index is a file, it
+survives restarts AND stays visible to **offloaded** workflows: a worker's
+own service instance opens the same database, where an in-memory index would
+be invisible.
+
+Past that scale, the **driver SPI** is the seam: an engine registers
+`{ id, ensureCollection, upsert, query, delete }` in its mod's `ready` (the
+mod-email driver pattern) and receives `{ filter, mode }` to push down.
+`sqlite-vec` is the natural first driver — note it is a NATIVE sqlite
+extension (needs `allowExtension` + platform binaries), which is exactly why
+it ships as a separate driver package rather than in this zero-dependency
+default. pgvector follows the same shape.
+
+## Admin
+
+**Data → Vectors** is the whole RAG loop on one page: the collections table
+(alias, dims, filterables, row count, active engine), **Ingest text** (paste
+anything — it's chunked, embedded through the collection's alias and indexed;
+the collection is created on first use; re-pasting the same text is a no-op
+thanks to content-hash dedupe), and **Search** (hybrid by default; scored
+matches with their text and meta). Paste a document, search for a phrase from
+it, read the scores — no curl required.

package/docs/ops/vectors.chunk.md

@@ -0,0 +1,8 @@
+Split text for indexing without a tokenizer dependency: a recursive character
+splitter that prefers paragraph breaks, then line breaks, then sentence ends,
+merges neighbors up to `maxChars` (default 1200) and carries `overlap`
+characters (default 150) across chunk boundaries so answers spanning a cut
+survive retrieval. Wire `text` (one string) or `docs`
+(`[{ id?, text, meta? }]`); chunks come back as `[{ id, text, meta? }]` with
+ids `${docId}#${i}` and the doc's meta on every chunk. Usually you want
+`vectors.index`, which chunks and writes in one node.

package/docs/ops/vectors.collection.ensure.md

@@ -0,0 +1,7 @@
+Declare an embedding collection — idempotent, so it belongs at the head of
+ingestion workflows. The declaration IS the safety: `alias` names the one
+embedding model this collection indexes and queries with, `dims` locks on the
+first write (a later mismatch errors naming the alias), and `filterables`
+lists the meta fields extracted into the indexed side table — the only fields
+`vectors.query` may filter on. Re-declaring updates alias/filterables; dims
+never silently change.

package/docs/ops/vectors.collections.ingest.md

@@ -0,0 +1,10 @@
+The admin's paste-to-RAG op (behind Data → Vectors → **Ingest text**): chunk a
+pasted text, embed it through the collection's declared alias, and upsert —
+creating the collection (with the given `alias`, default `embeddings`) when it
+doesn't exist yet. An EXISTING collection keeps its declared alias; the input
+is ignored on purpose (re-pointing a live collection's embedding model would
+corrupt its space). Without a `docId` the id derives from the text content, so
+re-pasting the same document is a no-op (content-hash dedupe) and pasting an
+edited version re-indexes only the changed chunks. `meta` (a JSON object) is
+stamped on every chunk — filterable when the collection declares the field.
+Privileged: gate the trigger with the `admin` scope.

package/docs/ops/vectors.collections.list.md

@@ -0,0 +1,3 @@
+List every vector collection with its embedding alias, locked dims, declared
+filterables, row count and the active engine — the backing op for the admin's
+Data → Vectors page. Privileged: gate the trigger with the `admin` scope.

package/docs/ops/vectors.collections.search.md

@@ -0,0 +1,7 @@
+The admin's try-a-query op (behind Data → Vectors → **Search**): top-k scored
+matches from a collection, hybrid by default (vector + keyword, RRF-fused),
+with each match's text (truncated) and meta. This is the smoke test for what
+an ingest actually indexed — paste in one section, search in the next, read
+the scores. For canvas/workflow retrieval use `vectors.query` (config-driven,
+filterable); this op exists so the admin form can pass the collection as an
+input. Privileged: gate the trigger with the `admin` scope.

package/docs/ops/vectors.delete.md

@@ -0,0 +1,5 @@
+Delete rows by id from a collection (vectors, meta index and keyword index
+together). Returns `{ count }`. Chunked docs delete by their chunk ids
+(`${docId}#${i}`) — list them from a `vectors.query` filtered on the doc's
+meta, or re-index the doc instead: unchanged chunks are skipped, changed ones
+overwrite in place.

package/docs/ops/vectors.index.md

@@ -0,0 +1,7 @@
+The one-node RAG ingestion path: chunk → embed → upsert. Wire
+`docs: [{ id, text, meta? }]`; each doc is split (same knobs as
+`vectors.chunk`), embedded through the collection's declared alias, and
+written as chunks `${docId}#${i}` carrying the doc's meta (so declared
+filterables apply to every chunk). Content-hashed like `vectors.upsert`:
+unchanged chunks cost nothing on re-runs. Returns `{ count, chunks }` (rows
+actually written / chunks produced).

package/docs/ops/vectors.query.md

@@ -0,0 +1,13 @@
+Search a collection: wire `text` (embedded via the collection's declared
+alias) or a raw `vector`, get `{ matches: [{ id, score, text, meta }] }`.
+
+`mode` picks the ranking: `vector` (cosine, the default), `keyword`
+(FTS5 when available, token-overlap otherwise — best for exact identifiers),
+or `hybrid` (both, fused with reciprocal-rank fusion — the safe default when
+queries mix names and natural language).
+
+`filter: { field: value | values[] }` is an AND of equality/any-of over the
+collection's DECLARED filterables, pruned through the indexed side table
+BEFORE any scoring — filtering on an undeclared field is a located error, not
+an empty result. Compose `vectors.query → store.doc.get` when the canonical
+record lives in mod-store.

package/docs/ops/vectors.upsert.md

@@ -0,0 +1,8 @@
+Write items into a collection. Each item is `{ id?, text, meta? }` (the text
+embeds through the collection's declared alias — mod-ai must be installed) or
+`{ id, vector, meta? }` (raw vectors skip embedding entirely). Declared
+filterable meta fields are extracted into the indexed side table at write
+time. Writes are content-hashed: items whose id + content match what's stored
+are skipped — no embedding call, no write — so re-running ingestion is cheap
+and idempotent. Returns `{ count, embedded }` (rows written / texts actually
+embedded).

package/package.json

@@ -0,0 +1,53 @@
+{
+  "name": "@pattern-js/mod-vectors",
+  "version": "0.4.0-main.c8eb671",
+  "description": "Vector search for Pattern — embedding collections that DECLARE their alias and dimensions (query-vs-index model mismatch becomes unrepresentable), filterable metadata, hybrid vector+keyword retrieval with RRF fusion, text chunking, and a zero-dependency sqlite-backed local engine with a driver SPI for sqlite-vec/pgvector later.",
+  "keywords": [
+    "pattern",
+    "mod",
+    "vectors",
+    "vector-search",
+    "embeddings",
+    "rag",
+    "hybrid-search"
+  ],
+  "type": "module",
+  "license": "MIT",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js"
+    }
+  },
+  "main": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "files": [
+    "dist",
+    "docs",
+    "README.md"
+  ],
+  "dependencies": {
+    "zod": "^4.0.0",
+    "@pattern-js/core": "0.4.0-main.c8eb671",
+    "@pattern-js/runtime-node": "0.4.0-main.c8eb671"
+  },
+  "devDependencies": {
+    "@types/node": "^22.0.0",
+    "typescript": "^5.6.0"
+  },
+  "engines": {
+    "node": ">=22.5"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/makkr-studio/pattern.git",
+    "directory": "packages/mod-vectors"
+  },
+  "homepage": "https://github.com/makkr-studio/pattern#readme",
+  "bugs": "https://github.com/makkr-studio/pattern/issues",
+  "author": "Makkr Studio (https://github.com/makkr-studio)",
+  "scripts": {
+    "build": "tsc -p tsconfig.json",
+    "typecheck": "tsc -p tsconfig.json --noEmit"
+  }
+}