@dudousxd/nestjs-agent-rag 0.4.0 → 0.5.0

package/README.md

@@ -1,8 +1,9 @@
 # @dudousxd/nestjs-agent-rag
 
 Retrieval-Augmented Generation for [`@dudousxd/nestjs-agent`](../nestjs). Chunking, ingestion, an
-embedding-backed `Retriever`, and two vector stores — in-memory and pgvector. Framework-agnostic:
-it depends only on `-core`, and you bring your own embedding model and (for pgvector) Postgres driver.
+embedding-backed `Retriever`, and three vector stores — in-memory, pgvector and RediSearch.
+Framework-agnostic: it depends only on `-core`, and you bring your own embedding model and (for the
+pgvector / RediSearch stores) database driver.
 
 ## Install
 
@@ -69,10 +70,128 @@ const store = new PgVectorStore(
 await store.ensureSchema(); // CREATE EXTENSION vector + table + HNSW cosine index
 ```
 
+## RediSearch store
+
+`RedisVectorStore` takes anything that can send a raw command — `node-redis`, or `ioredis` wrapped —
+so this package pulls in no Redis driver either. Needs the RediSearch module (Redis Stack / Redis 8+):
+
+```ts
+import { createClient } from 'redis';
+import { RedisVectorStore } from '@dudousxd/nestjs-agent-rag';
+
+const client = createClient({ url: process.env.REDIS_URL });
+await client.connect();
+
+const store = new RedisVectorStore(
+  { sendCommand: (args) => client.sendCommand(args) },
+  { dimensions: 1536, filterableFields: ['audience'] },
+);
+await store.ensureSchema(); // FT.CREATE: TEXT + TAG + HNSW cosine vector field
+```
+
+`filterableFields` declares which metadata keys become filterable `TAG` fields — RediSearch needs them
+up front, so only those keys can be used in a `filter`. This store also does lexical search; see below.
+
+`ensureSchema()` belongs at boot. It creates the index if missing — and if the index already exists,
+it compares it against your configuration rather than assuming they agree. A `filterableFields` entry
+added after the index was created is repaired in place with `FT.ALTER … SCHEMA ADD` (chunks written
+before the repair only become filterable on that key once re-ingested). A changed `dimensions` — a
+model swap — can't be repaired without a full reindex, so it throws `RedisVectorSchemaMismatchError`
+instead of silently leaving the index on the old width.
+
+## Hybrid search (dense + lexical)
+
+`HybridRetriever` fuses several retrievers with Reciprocal Rank Fusion. RRF is **rank**-based, so the
+two halves need no common score scale — cosine similarity and BM25 fuse as they are.
+
+There are two ways to get the lexical half, and which one you want is decided by your deployment, not
+your taste.
+
+**Store-backed (the default for anything multi-process).** `RedisVectorStore` already declares the
+chunk text as a RediSearch `TEXT` field, so the lexical index is *already there* — `LexicalRetriever`
+just queries it with BM25:
+
+```ts
+import { EmbeddingRetriever, HybridRetriever, LexicalRetriever } from '@dudousxd/nestjs-agent-rag';
+
+const retriever = new HybridRetriever([
+  new EmbeddingRetriever(embedder, store),
+  new LexicalRetriever(store), // ← same store, no second index
+]);
+```
+
+Nothing else changes: no chunks to hand the retriever, no state in the process, no deletes to mirror.
+This is the only option that works when **ingestion and search run in different processes** — a
+worker that ingests and an API pod that searches. The API pod never sees the chunks, so an in-process
+index there would be permanently empty (or kept warm by scanning the corpus into JS heap on every
+pod, which is the workaround this exists to delete). It also means a document is lexically findable
+the moment it is upserted, with no refresh window.
+
+`searchText` applies `filter` with exactly the same semantics as `search` — array = OR, **empty array
+= deny** — so a lexical hit can never escape a filter a vector hit would have respected. That matters
+when the filter is carrying capability tokens rather than a hint.
+
+A query is reduced to letter/digit/`_` terms before it is sent, so RediSearch query syntax in user
+input (`*`, `@field:{…}`, `|`, `-`, `=>[KNN …]`) is inert rather than escaped-but-live. A query with
+no searchable terms left — empty, whitespace, or pure punctuation — returns `[]` rather than matching
+the corpus.
+
+**In-process (`KeywordRetriever`).** Still the right answer for `MemoryVectorStore`, for a store with
+no lexical capability, and for single-process deployments where you control both halves. It is a full
+BM25 implementation over chunks you feed it, so chunk once and give the SAME chunks to both halves —
+their ids line up and `HybridRetriever` can fuse the rankings:
+
+```ts
+const chunks = chunkDocuments(docs);
+await ingestChunks(chunks, { embedder, store });
+keyword.add(chunks); // its own copy of the corpus, in this process only
+
+const retriever = new HybridRetriever([new EmbeddingRetriever(embedder, store), keyword]);
+```
+
+Because that is a second index, **deletes have to hit both**, and only this process can see it.
+`KeywordRetriever` keeps its own copy of each chunk's text, so a document dropped from the vector
+store alone stays fully retrievable — `retrieve` will keep returning the removed passage:
+
+```ts
+await store.remove(documentId);
+keyword.remove(documentId); // ← the other half. Same document id; chunk ids collapse for you.
+```
+
+`KeywordRetriever` also exposes `clear()` and `size` for rebuilding the index wholesale. None of this
+bookkeeping exists on the store-backed path above, which is the point of it.
+
+Ask the store which it supports rather than hard-coding:
+
+```ts
+import { isLexicalVectorStore } from '@dudousxd/nestjs-agent-rag';
+
+const lexical = isLexicalVectorStore(store) ? new LexicalRetriever(store) : keywordRetriever;
+```
+
+### Which stores implement `LexicalVectorStore`
+
+| Store | Lexical | Why |
+| --- | --- | --- |
+| `RedisVectorStore` | ✅ | RediSearch indexes the chunk text as `TEXT`; BM25 is already there |
+| `PgVectorStore` | ❌ | the chunks table has no `tsvector`/GIN index, and adding one is a migration |
+| `MemoryVectorStore` | ❌ | single-process by definition — use `KeywordRetriever` |
+
+Postgres is a deliberate omission. Lexical search there needs new DDL (a GIN index), and `CREATE
+INDEX` in `ensureSchema` would take a write lock on an already-populated chunks table at boot; it also
+forces a text-search-configuration choice that must match between index and query or Postgres silently
+falls back to a sequential scan. That wants a migration you run and watch, not a side effect of a
+library upgrade. The interface is open, so it can be added later without changing anything else.
+
 ## API
 
 - `chunkText(text, { chunkSize?, overlap? })` — overlapping, boundary-aware chunks.
 - `ingestDocuments(docs, { embedder, store, chunkSize?, overlap? })` — chunk → embed → upsert.
 - `EmbeddingRetriever(embedder, store)` — the core `Retriever` from an embedder + vector store.
-- `MemoryVectorStore` / `PgVectorStore` — `VectorStore` adapters.
+- `LexicalRetriever(store)` — the core `Retriever` from a store's own full-text index (BM25).
+- `KeywordRetriever` — in-process BM25 over chunks you feed it; the single-process lexical half:
+  `add(chunks)`, `remove(documentId)`, `clear()`, `size`.
+- `HybridRetriever(retrievers, { k?, fetchTopK?, weights? })` — RRF fusion of the two.
+- `MemoryVectorStore` / `PgVectorStore` / `RedisVectorStore` — `VectorStore` adapters.
+- `isLexicalVectorStore(store)` — type guard for the optional `LexicalVectorStore` capability.
 - `createRetrievalTool(retriever, { name?, description?, topK? })` — the agentic-retrieval tool.

package/dist/index.cjs

@@ -25,8 +25,10 @@ __export(index_exports, {
   FilteredRetriever: () => FilteredRetriever,
   HybridRetriever: () => HybridRetriever,
   KeywordRetriever: () => KeywordRetriever,
+  LexicalRetriever: () => LexicalRetriever,
   MemoryVectorStore: () => MemoryVectorStore,
   PgVectorStore: () => PgVectorStore,
+  RedisVectorSchemaMismatchError: () => RedisVectorSchemaMismatchError,
   RedisVectorStore: () => RedisVectorStore,
   RerankingRetriever: () => RerankingRetriever,
   chunkDocuments: () => chunkDocuments,
@@ -34,7 +36,8 @@ __export(index_exports, {
   createRetrievalTool: () => createRetrievalTool,
   documentIdOf: () => documentIdOf,
   ingestChunks: () => ingestChunks,
-  ingestDocuments: () => ingestDocuments
+  ingestDocuments: () => ingestDocuments,
+  isLexicalVectorStore: () => isLexicalVectorStore
 });
 module.exports = __toCommonJS(index_exports);
 
@@ -77,6 +80,10 @@ function chunkText(text, options = {}) {
 __name(chunkText, "chunkText");
 
 // src/vector-store.ts
+function isLexicalVectorStore(store) {
+  return typeof store.searchText === "function";
+}
+__name(isLexicalVectorStore, "isLexicalVectorStore");
 function documentIdOf(chunkId) {
   return chunkId.replace(/#\d+$/, "");
 }
@@ -306,6 +313,19 @@ function toVectorLiteral(embedding) {
 __name(toVectorLiteral, "toVectorLiteral");
 
 // src/redis-vector-store.ts
+var RedisVectorSchemaMismatchError = class extends Error {
+  static {
+    __name(this, "RedisVectorSchemaMismatchError");
+  }
+  index;
+  field;
+  expected;
+  actual;
+  constructor(index, field, expected, actual, remedy) {
+    super(`RediSearch index "${index}" disagrees with this RedisVectorStore's configuration: field "${field}" is ${actual}, expected ${expected}. ${remedy}`), this.index = index, this.field = field, this.expected = expected, this.actual = actual;
+    this.name = "RedisVectorSchemaMismatchError";
+  }
+};
 var RedisVectorStore = class {
   static {
     __name(this, "RedisVectorStore");
@@ -315,23 +335,83 @@ var RedisVectorStore = class {
   prefix;
   dimensions;
   filterableFields;
+  lexicalScorer;
   constructor(client, options = {}) {
     this.client = client;
     this.index = options.index ?? "agent_rag_idx";
     this.prefix = options.prefix ?? "agent_rag:";
     this.dimensions = options.dimensions ?? 1536;
     this.filterableFields = options.filterableFields ?? [];
+    this.lexicalScorer = options.lexicalScorer ?? "BM25";
   }
-  /** Idempotent: create the HNSW/cosine index if `FT.INFO` says it doesn't exist yet. */
+  /**
+  * Idempotent, and drift-aware: create the HNSW/cosine index if `FT.INFO` says it doesn't exist —
+  * and if it DOES exist, compare what's there against this store's configuration instead of
+  * assuming they agree. An index that outlives a config change is the normal case (schemas live in
+  * Redis, config lives in a redeployed process), and the two failure modes it produces are both
+  * invisible at runtime: a `filterableFields` entry added later has no `meta_*` TAG, so every search
+  * filtering on it matches *nothing*; a changed `dimensions` leaves the index on the old width, so
+  * writes are rejected or KNN ranks garbage. Neither raises anything on its own.
+  *
+  * The two are repaired differently, because only one of them CAN be repaired:
+  *
+  * - **A missing filterable TAG is additive** → `FT.ALTER … SCHEMA ADD`. Nothing that already works
+  *   changes, so doing it automatically is safe. Note the chunks written *before* the alter carry no
+  *   `meta_*` hash field at all (`upsert` only writes the fields configured at the time), so they
+  *   only become filterable on that key once they're re-ingested — the schema is repaired here, the
+  *   backfill is the host's call.
+  * - **A dimension (or field-type) change is not repairable** → {@link RedisVectorSchemaMismatchError}.
+  *   It needs a drop + full reindex, which would destroy the host's corpus; failing loudly at boot is
+  *   the only honest option.
+  *
+  * If `FT.INFO` comes back in a shape this parser doesn't recognise, no drift is inferred — an
+  * unreadable reply must not be turned into a false alarm that blocks a boot.
+  */
   async ensureSchema() {
+    let info;
     try {
-      await this.client.sendCommand([
+      info = await this.client.sendCommand([
         "FT.INFO",
         this.index
       ]);
-      return;
     } catch {
+      await this.createIndex();
+      return;
+    }
+    await this.reconcileSchema(info);
+  }
+  /** Repair what can be repaired in place; throw on what can't. */
+  async reconcileSchema(info) {
+    const attributes = parseIndexAttributes(info);
+    if (attributes.size === 0) {
+      return;
+    }
+    const vector = attributes.get("embedding");
+    if (vector?.dimensions !== void 0 && vector.dimensions !== this.dimensions) {
+      throw new RedisVectorSchemaMismatchError(this.index, "embedding", `DIM ${this.dimensions}`, `DIM ${vector.dimensions}`, `A vector field's width is fixed at FT.CREATE and every stored embedding has the old one, so this cannot be altered in place: drop the index (FT.DROPINDEX ${this.index} DD) and re-ingest with the new embedding model.`);
+    }
+    const missing = [];
+    for (const field of this.filterableFields) {
+      const name = `meta_${field}`;
+      const existing = attributes.get(name);
+      if (existing === void 0) {
+        missing.push(name);
+      } else if (existing.type !== "TAG") {
+        throw new RedisVectorSchemaMismatchError(this.index, name, "TAG", existing.type, `A field's type cannot be changed in place: drop the index (FT.DROPINDEX ${this.index} DD) and re-ingest, or rename the metadata key.`);
+      }
+    }
+    for (const name of missing) {
+      await this.client.sendCommand([
+        "FT.ALTER",
+        this.index,
+        "SCHEMA",
+        "ADD",
+        name,
+        "TAG"
+      ]);
     }
+  }
+  async createIndex() {
     const schema = [
       "text",
       "TEXT",
@@ -484,7 +564,100 @@ var RedisVectorStore = class {
     ]);
     return parseSearchReply(reply, this.prefix);
   }
+  /**
+  * Lexical (BM25) search over the same index `search` uses — the {@link LexicalVectorStore} half of
+  * hybrid retrieval, served by RediSearch's own keyword index rather than an in-process copy of the
+  * corpus. `options.filter` is ANDed onto the text clause with the identical TAG semantics as
+  * `search` (array = OR, empty array = deny), so a lexical hit can never escape a filter a vector
+  * hit would have respected.
+  *
+  * A query with no searchable terms — empty, whitespace, or pure punctuation — returns `[]`. That is
+  * deliberate: RediSearch's `*` would match the entire corpus, which is the opposite of what a
+  * relevance query with nothing to be relevant to should do.
+  */
+  async searchText(query, options) {
+    if (filterMatchesNothing(options.filter)) {
+      return [];
+    }
+    const terms = tokenizeQuery(query);
+    if (terms.length === 0) {
+      return [];
+    }
+    const filterQuery = buildFilter(options.filter);
+    const textQuery = `@text:(${terms.join("|")})`;
+    const reply = await this.client.sendCommand([
+      "FT.SEARCH",
+      this.index,
+      filterQuery === "*" ? textQuery : `(${filterQuery} ${textQuery})`,
+      "SCORER",
+      this.lexicalScorer,
+      "WITHSCORES",
+      "RETURN",
+      "3",
+      "text",
+      "source",
+      "metadata_json",
+      "DIALECT",
+      "2",
+      "LIMIT",
+      "0",
+      String(options.topK)
+    ]);
+    return parseSearchReply(reply, this.prefix, "lexical");
+  }
 };
+function parseIndexAttributes(reply) {
+  const attributes = /* @__PURE__ */ new Map();
+  for (const entry of readInfoField(reply, "attributes")) {
+    const fields = readKeyValues(entry);
+    const name = fields.attribute ?? fields.identifier;
+    if (name === void 0) {
+      continue;
+    }
+    const dim = Number(fields.dim);
+    attributes.set(name, {
+      type: (fields.type ?? "").toUpperCase(),
+      ...fields.dim !== void 0 && Number.isFinite(dim) ? {
+        dimensions: dim
+      } : {}
+    });
+  }
+  return attributes;
+}
+__name(parseIndexAttributes, "parseIndexAttributes");
+function readInfoField(reply, key) {
+  if (Array.isArray(reply)) {
+    for (let index = 0; index + 1 < reply.length; index += 2) {
+      if (toStr(reply[index]) === key) {
+        const value = reply[index + 1];
+        return Array.isArray(value) ? value : [];
+      }
+    }
+    return [];
+  }
+  if (typeof reply === "object" && reply !== null && key in reply) {
+    const value = reply[key];
+    return Array.isArray(value) ? value : [];
+  }
+  return [];
+}
+__name(readInfoField, "readInfoField");
+function readKeyValues(entry) {
+  const fields = {};
+  if (Array.isArray(entry)) {
+    for (let index = 0; index + 1 < entry.length; index += 2) {
+      fields[toStr(entry[index]).toLowerCase()] = toStr(entry[index + 1]);
+    }
+    return fields;
+  }
+  if (typeof entry === "object" && entry !== null) {
+    for (const [key, value] of Object.entries(entry)) {
+      fields[key.toLowerCase()] = toStr(value);
+    }
+  }
+  return fields;
+}
+__name(readKeyValues, "readKeyValues");
 function encodeVector(embedding) {
   return Buffer.from(new Float32Array(embedding).buffer);
 }
@@ -510,6 +683,10 @@ function filterMatchesNothing(filter) {
   return Object.values(filter).some((value) => Array.isArray(value) && value.length === 0);
 }
 __name(filterMatchesNothing, "filterMatchesNothing");
+function tokenizeQuery(query) {
+  return query.match(/[\p{L}\p{N}_]+/gu) ?? [];
+}
+__name(tokenizeQuery, "tokenizeQuery");
 function escapeTag(value) {
   return value.replace(/[^a-zA-Z0-9_]/g, (char) => `\\${char}`);
 }
@@ -546,17 +723,19 @@ function toStr(value) {
   return Buffer.isBuffer(value) ? value.toString("utf8") : String(value);
 }
 __name(toStr, "toStr");
-function parseSearchReply(reply, prefix) {
+function parseSearchReply(reply, prefix, scoreSource = "vector") {
+  const stride = scoreSource === "lexical" ? 3 : 2;
   if (Array.isArray(reply)) {
     const passages = [];
-    for (let index = 1; index + 1 < reply.length; index += 2) {
-      const rawFields = reply[index + 1];
+    for (let index = 1; index + stride - 1 < reply.length; index += stride) {
+      const rawFields = reply[index + stride - 1];
       if (Array.isArray(rawFields)) {
         const attrs = {};
         for (let field = 0; field + 1 < rawFields.length; field += 2) {
           attrs[toStr(rawFields[field])] = toStr(rawFields[field + 1]);
         }
-        passages.push(toPassage(toStr(reply[index]), attrs, prefix));
+        const inlineScore = scoreSource === "lexical" ? toStr(reply[index + 1]) : void 0;
+        passages.push(toPassage(toStr(reply[index]), attrs, prefix, scoreSource, inlineScore));
       }
     }
     return passages;
@@ -564,12 +743,19 @@ function parseSearchReply(reply, prefix) {
   if (typeof reply === "object" && reply !== null && "results" in reply) {
     const results = reply.results;
     if (Array.isArray(results)) {
-      return results.map((result) => toPassage(readId(result), readAttributes(result), prefix));
+      return results.map((result) => toPassage(readId(result), readAttributes(result), prefix, scoreSource, scoreSource === "lexical" ? readScore(result) : void 0));
     }
   }
   return [];
 }
 __name(parseSearchReply, "parseSearchReply");
+function readScore(result) {
+  if (typeof result === "object" && result !== null && "score" in result) {
+    return toStr(result.score);
+  }
+  return void 0;
+}
+__name(readScore, "readScore");
 function readId(result) {
   if (typeof result === "object" && result !== null && "id" in result) {
     return toStr(result.id);
@@ -590,13 +776,13 @@ function readAttributes(result) {
   return attrs;
 }
 __name(readAttributes, "readAttributes");
-function toPassage(rawId, attrs, prefix) {
-  const distance = Number(attrs.vector_score ?? "1");
+function toPassage(rawId, attrs, prefix, scoreSource = "vector", inlineScore) {
+  const score = scoreSource === "lexical" ? Number(inlineScore ?? "0") : 1 - Number(attrs.vector_score ?? "1");
   const source = attrs.source;
   return {
     id: rawId.startsWith(prefix) ? rawId.slice(prefix.length) : rawId,
     text: attrs.text ?? "",
-    score: 1 - distance,
+    score: Number.isFinite(score) ? score : 0,
     ...source !== void 0 ? {
       source
     } : {},
@@ -662,10 +848,14 @@ var KeywordRetriever = class {
     this.k1 = options.k1 ?? 1.5;
     this.b = options.b ?? 0.75;
   }
+  /** How many chunks are currently indexed. */
+  get size() {
+    return this.docs.size;
+  }
   /** Index (or re-index) records. Re-adding an existing id replaces its posting and updates stats. */
   add(records) {
     for (const record of records) {
-      this.remove(record.id);
+      this.removeChunk(record.id);
       const tokens = tokenize(record.text);
       const frequencies = /* @__PURE__ */ new Map();
       for (const token of tokens) {
@@ -682,7 +872,30 @@ var KeywordRetriever = class {
       this.totalLength += tokens.length;
     }
   }
-  remove(id) {
+  /**
+  * Forget a source document — every chunk whose id is `${documentId}` or `${documentId}#<n>`, the
+  * same id scheme (and the same {@link documentIdOf} collapse) the vector stores use for their
+  * `remove`. The keyword half of a hybrid index keeps its own copy of the chunk text, so a document
+  * dropped from the vector store stays fully retrievable here until this is called. Unknown ids are
+  * a no-op.
+  */
+  remove(documentId) {
+    for (const id of [
+      ...this.docs.keys()
+    ]) {
+      if (id === documentId || documentIdOf(id) === documentId) {
+        this.removeChunk(id);
+      }
+    }
+  }
+  /** Drop every indexed chunk, resetting the corpus statistics with it. */
+  clear() {
+    this.docs.clear();
+    this.documentFrequency.clear();
+    this.totalLength = 0;
+  }
+  /** Drop ONE chunk by its exact id, decrementing the corpus statistics it contributed. */
+  removeChunk(id) {
     const existing = this.docs.get(id);
     if (existing === void 0) {
       return;
@@ -743,6 +956,25 @@ function tokenize(text) {
 }
 __name(tokenize, "tokenize");
 
+// src/lexical-retriever.ts
+var LexicalRetriever = class {
+  static {
+    __name(this, "LexicalRetriever");
+  }
+  store;
+  constructor(store) {
+    this.store = store;
+  }
+  async retrieve(query, options = {}) {
+    return this.store.searchText(query, {
+      topK: options.topK ?? 5,
+      ...options.filter !== void 0 ? {
+        filter: options.filter
+      } : {}
+    });
+  }
+};
+
 // src/hybrid-retriever.ts
 var HybridRetriever = class {
   static {
@@ -930,8 +1162,10 @@ __name(createRetrievalTool, "createRetrievalTool");
   FilteredRetriever,
   HybridRetriever,
   KeywordRetriever,
+  LexicalRetriever,
   MemoryVectorStore,
   PgVectorStore,
+  RedisVectorSchemaMismatchError,
   RedisVectorStore,
   RerankingRetriever,
   chunkDocuments,
@@ -939,6 +1173,7 @@ __name(createRetrievalTool, "createRetrievalTool");
   createRetrievalTool,
   documentIdOf,
   ingestChunks,
-  ingestDocuments
+  ingestDocuments,
+  isLexicalVectorStore
 });
 //# sourceMappingURL=index.cjs.map