@dudousxd/nestjs-agent-rag 0.4.1 → 0.5.0

package/README.md

@@ -1,8 +1,9 @@
 # @dudousxd/nestjs-agent-rag
 
 Retrieval-Augmented Generation for [`@dudousxd/nestjs-agent`](../nestjs). Chunking, ingestion, an
-embedding-backed `Retriever`, and two vector stores — in-memory and pgvector. Framework-agnostic:
-it depends only on `-core`, and you bring your own embedding model and (for pgvector) Postgres driver.
+embedding-backed `Retriever`, and three vector stores — in-memory, pgvector and RediSearch.
+Framework-agnostic: it depends only on `-core`, and you bring your own embedding model and (for the
+pgvector / RediSearch stores) database driver.
 
 ## Install
 
@@ -69,45 +70,128 @@ const store = new PgVectorStore(
 await store.ensureSchema(); // CREATE EXTENSION vector + table + HNSW cosine index
 ```
 
-## Hybrid search — and deleting from it
+## RediSearch store
+
+`RedisVectorStore` takes anything that can send a raw command — `node-redis`, or `ioredis` wrapped —
+so this package pulls in no Redis driver either. Needs the RediSearch module (Redis Stack / Redis 8+):
+
+```ts
+import { createClient } from 'redis';
+import { RedisVectorStore } from '@dudousxd/nestjs-agent-rag';
+
+const client = createClient({ url: process.env.REDIS_URL });
+await client.connect();
+
+const store = new RedisVectorStore(
+  { sendCommand: (args) => client.sendCommand(args) },
+  { dimensions: 1536, filterableFields: ['audience'] },
+);
+await store.ensureSchema(); // FT.CREATE: TEXT + TAG + HNSW cosine vector field
+```
+
+`filterableFields` declares which metadata keys become filterable `TAG` fields — RediSearch needs them
+up front, so only those keys can be used in a `filter`. This store also does lexical search; see below.
+
+`ensureSchema()` belongs at boot. It creates the index if missing — and if the index already exists,
+it compares it against your configuration rather than assuming they agree. A `filterableFields` entry
+added after the index was created is repaired in place with `FT.ALTER … SCHEMA ADD` (chunks written
+before the repair only become filterable on that key once re-ingested). A changed `dimensions` — a
+model swap — can't be repaired without a full reindex, so it throws `RedisVectorSchemaMismatchError`
+instead of silently leaving the index on the old width.
+
+## Hybrid search (dense + lexical)
+
+`HybridRetriever` fuses several retrievers with Reciprocal Rank Fusion. RRF is **rank**-based, so the
+two halves need no common score scale — cosine similarity and BM25 fuse as they are.
+
+There are two ways to get the lexical half, and which one you want is decided by your deployment, not
+your taste.
+
+**Store-backed (the default for anything multi-process).** `RedisVectorStore` already declares the
+chunk text as a RediSearch `TEXT` field, so the lexical index is *already there* — `LexicalRetriever`
+just queries it with BM25:
 
-Hybrid search means chunking once and feeding the SAME chunks to both halves, so their ids line up
-and `HybridRetriever` can fuse the rankings:
+```ts
+import { EmbeddingRetriever, HybridRetriever, LexicalRetriever } from '@dudousxd/nestjs-agent-rag';
+
+const retriever = new HybridRetriever([
+  new EmbeddingRetriever(embedder, store),
+  new LexicalRetriever(store), // ← same store, no second index
+]);
+```
+
+Nothing else changes: no chunks to hand the retriever, no state in the process, no deletes to mirror.
+This is the only option that works when **ingestion and search run in different processes** — a
+worker that ingests and an API pod that searches. The API pod never sees the chunks, so an in-process
+index there would be permanently empty (or kept warm by scanning the corpus into JS heap on every
+pod, which is the workaround this exists to delete). It also means a document is lexically findable
+the moment it is upserted, with no refresh window.
+
+`searchText` applies `filter` with exactly the same semantics as `search` — array = OR, **empty array
+= deny** — so a lexical hit can never escape a filter a vector hit would have respected. That matters
+when the filter is carrying capability tokens rather than a hint.
+
+A query is reduced to letter/digit/`_` terms before it is sent, so RediSearch query syntax in user
+input (`*`, `@field:{…}`, `|`, `-`, `=>[KNN …]`) is inert rather than escaped-but-live. A query with
+no searchable terms left — empty, whitespace, or pure punctuation — returns `[]` rather than matching
+the corpus.
+
+**In-process (`KeywordRetriever`).** Still the right answer for `MemoryVectorStore`, for a store with
+no lexical capability, and for single-process deployments where you control both halves. It is a full
+BM25 implementation over chunks you feed it, so chunk once and give the SAME chunks to both halves —
+their ids line up and `HybridRetriever` can fuse the rankings:
 
 ```ts
 const chunks = chunkDocuments(docs);
 await ingestChunks(chunks, { embedder, store });
-keyword.add(chunks);
+keyword.add(chunks); // its own copy of the corpus, in this process only
 
 const retriever = new HybridRetriever([new EmbeddingRetriever(embedder, store), keyword]);
 ```
 
-Because there are two indexes, **deletes have to hit both**. `KeywordRetriever` keeps its own copy of
-each chunk's text, so a document dropped from the vector store alone stays fully retrievable —
-`retrieve` will keep returning the removed passage:
+Because that is a second index, **deletes have to hit both**, and only this process can see it.
+`KeywordRetriever` keeps its own copy of each chunk's text, so a document dropped from the vector
+store alone stays fully retrievable — `retrieve` will keep returning the removed passage:
 
 ```ts
 await store.remove(documentId);
 keyword.remove(documentId); // ← the other half. Same document id; chunk ids collapse for you.
 ```
 
-`KeywordRetriever` also exposes `clear()` and `size` for rebuilding the index wholesale.
+`KeywordRetriever` also exposes `clear()` and `size` for rebuilding the index wholesale. None of this
+bookkeeping exists on the store-backed path above, which is the point of it.
 
-## RediSearch store
+Ask the store which it supports rather than hard-coding:
+
+```ts
+import { isLexicalVectorStore } from '@dudousxd/nestjs-agent-rag';
+
+const lexical = isLexicalVectorStore(store) ? new LexicalRetriever(store) : keywordRetriever;
+```
+
+### Which stores implement `LexicalVectorStore`
+
+| Store | Lexical | Why |
+| --- | --- | --- |
+| `RedisVectorStore` | ✅ | RediSearch indexes the chunk text as `TEXT`; BM25 is already there |
+| `PgVectorStore` | ❌ | the chunks table has no `tsvector`/GIN index, and adding one is a migration |
+| `MemoryVectorStore` | ❌ | single-process by definition — use `KeywordRetriever` |
 
-`RedisVectorStore` needs `ensureSchema()` at boot. It creates the index if missing — and if the index
-already exists, it compares it against your configuration rather than assuming they agree. A
-`filterableFields` entry added after the index was created is repaired in place with
-`FT.ALTER … SCHEMA ADD` (chunks written before the repair only become filterable on that key once
-re-ingested). A changed `dimensions` — a model swap — can't be repaired without a full reindex, so it
-throws `RedisVectorSchemaMismatchError` instead of silently leaving the index on the old width.
+Postgres is a deliberate omission. Lexical search there needs new DDL (a GIN index), and `CREATE
+INDEX` in `ensureSchema` would take a write lock on an already-populated chunks table at boot; it also
+forces a text-search-configuration choice that must match between index and query or Postgres silently
+falls back to a sequential scan. That wants a migration you run and watch, not a side effect of a
+library upgrade. The interface is open, so it can be added later without changing anything else.
 
 ## API
 
 - `chunkText(text, { chunkSize?, overlap? })` — overlapping, boundary-aware chunks.
 - `ingestDocuments(docs, { embedder, store, chunkSize?, overlap? })` — chunk → embed → upsert.
 - `EmbeddingRetriever(embedder, store)` — the core `Retriever` from an embedder + vector store.
-- `KeywordRetriever` — BM25 lexical half of hybrid search: `add(chunks)`, `remove(documentId)`,
-  `clear()`, `size`.
+- `LexicalRetriever(store)` — the core `Retriever` from a store's own full-text index (BM25).
+- `KeywordRetriever` — in-process BM25 over chunks you feed it; the single-process lexical half:
+  `add(chunks)`, `remove(documentId)`, `clear()`, `size`.
+- `HybridRetriever(retrievers, { k?, fetchTopK?, weights? })` — RRF fusion of the two.
 - `MemoryVectorStore` / `PgVectorStore` / `RedisVectorStore` — `VectorStore` adapters.
+- `isLexicalVectorStore(store)` — type guard for the optional `LexicalVectorStore` capability.
 - `createRetrievalTool(retriever, { name?, description?, topK? })` — the agentic-retrieval tool.

package/dist/index.cjs

@@ -25,6 +25,7 @@ __export(index_exports, {
   FilteredRetriever: () => FilteredRetriever,
   HybridRetriever: () => HybridRetriever,
   KeywordRetriever: () => KeywordRetriever,
+  LexicalRetriever: () => LexicalRetriever,
   MemoryVectorStore: () => MemoryVectorStore,
   PgVectorStore: () => PgVectorStore,
   RedisVectorSchemaMismatchError: () => RedisVectorSchemaMismatchError,
@@ -35,7 +36,8 @@ __export(index_exports, {
   createRetrievalTool: () => createRetrievalTool,
   documentIdOf: () => documentIdOf,
   ingestChunks: () => ingestChunks,
-  ingestDocuments: () => ingestDocuments
+  ingestDocuments: () => ingestDocuments,
+  isLexicalVectorStore: () => isLexicalVectorStore
 });
 module.exports = __toCommonJS(index_exports);
 
@@ -78,6 +80,10 @@ function chunkText(text, options = {}) {
 __name(chunkText, "chunkText");
 
 // src/vector-store.ts
+function isLexicalVectorStore(store) {
+  return typeof store.searchText === "function";
+}
+__name(isLexicalVectorStore, "isLexicalVectorStore");
 function documentIdOf(chunkId) {
   return chunkId.replace(/#\d+$/, "");
 }
@@ -329,12 +335,14 @@ var RedisVectorStore = class {
   prefix;
   dimensions;
   filterableFields;
+  lexicalScorer;
   constructor(client, options = {}) {
     this.client = client;
     this.index = options.index ?? "agent_rag_idx";
     this.prefix = options.prefix ?? "agent_rag:";
     this.dimensions = options.dimensions ?? 1536;
     this.filterableFields = options.filterableFields ?? [];
+    this.lexicalScorer = options.lexicalScorer ?? "BM25";
   }
   /**
   * Idempotent, and drift-aware: create the HNSW/cosine index if `FT.INFO` says it doesn't exist —
@@ -556,6 +564,47 @@ var RedisVectorStore = class {
     ]);
     return parseSearchReply(reply, this.prefix);
   }
+  /**
+  * Lexical (BM25) search over the same index `search` uses — the {@link LexicalVectorStore} half of
+  * hybrid retrieval, served by RediSearch's own keyword index rather than an in-process copy of the
+  * corpus. `options.filter` is ANDed onto the text clause with the identical TAG semantics as
+  * `search` (array = OR, empty array = deny), so a lexical hit can never escape a filter a vector
+  * hit would have respected.
+  *
+  * A query with no searchable terms — empty, whitespace, or pure punctuation — returns `[]`. That is
+  * deliberate: RediSearch's `*` would match the entire corpus, which is the opposite of what a
+  * relevance query with nothing to be relevant to should do.
+  */
+  async searchText(query, options) {
+    if (filterMatchesNothing(options.filter)) {
+      return [];
+    }
+    const terms = tokenizeQuery(query);
+    if (terms.length === 0) {
+      return [];
+    }
+    const filterQuery = buildFilter(options.filter);
+    const textQuery = `@text:(${terms.join("|")})`;
+    const reply = await this.client.sendCommand([
+      "FT.SEARCH",
+      this.index,
+      filterQuery === "*" ? textQuery : `(${filterQuery} ${textQuery})`,
+      "SCORER",
+      this.lexicalScorer,
+      "WITHSCORES",
+      "RETURN",
+      "3",
+      "text",
+      "source",
+      "metadata_json",
+      "DIALECT",
+      "2",
+      "LIMIT",
+      "0",
+      String(options.topK)
+    ]);
+    return parseSearchReply(reply, this.prefix, "lexical");
+  }
 };
 function parseIndexAttributes(reply) {
   const attributes = /* @__PURE__ */ new Map();
@@ -634,6 +683,10 @@ function filterMatchesNothing(filter) {
   return Object.values(filter).some((value) => Array.isArray(value) && value.length === 0);
 }
 __name(filterMatchesNothing, "filterMatchesNothing");
+function tokenizeQuery(query) {
+  return query.match(/[\p{L}\p{N}_]+/gu) ?? [];
+}
+__name(tokenizeQuery, "tokenizeQuery");
 function escapeTag(value) {
   return value.replace(/[^a-zA-Z0-9_]/g, (char) => `\\${char}`);
 }
@@ -670,17 +723,19 @@ function toStr(value) {
   return Buffer.isBuffer(value) ? value.toString("utf8") : String(value);
 }
 __name(toStr, "toStr");
-function parseSearchReply(reply, prefix) {
+function parseSearchReply(reply, prefix, scoreSource = "vector") {
+  const stride = scoreSource === "lexical" ? 3 : 2;
   if (Array.isArray(reply)) {
     const passages = [];
-    for (let index = 1; index + 1 < reply.length; index += 2) {
-      const rawFields = reply[index + 1];
+    for (let index = 1; index + stride - 1 < reply.length; index += stride) {
+      const rawFields = reply[index + stride - 1];
       if (Array.isArray(rawFields)) {
         const attrs = {};
         for (let field = 0; field + 1 < rawFields.length; field += 2) {
           attrs[toStr(rawFields[field])] = toStr(rawFields[field + 1]);
         }
-        passages.push(toPassage(toStr(reply[index]), attrs, prefix));
+        const inlineScore = scoreSource === "lexical" ? toStr(reply[index + 1]) : void 0;
+        passages.push(toPassage(toStr(reply[index]), attrs, prefix, scoreSource, inlineScore));
       }
     }
     return passages;
@@ -688,12 +743,19 @@ function parseSearchReply(reply, prefix) {
   if (typeof reply === "object" && reply !== null && "results" in reply) {
     const results = reply.results;
     if (Array.isArray(results)) {
-      return results.map((result) => toPassage(readId(result), readAttributes(result), prefix));
+      return results.map((result) => toPassage(readId(result), readAttributes(result), prefix, scoreSource, scoreSource === "lexical" ? readScore(result) : void 0));
     }
   }
   return [];
 }
 __name(parseSearchReply, "parseSearchReply");
+function readScore(result) {
+  if (typeof result === "object" && result !== null && "score" in result) {
+    return toStr(result.score);
+  }
+  return void 0;
+}
+__name(readScore, "readScore");
 function readId(result) {
   if (typeof result === "object" && result !== null && "id" in result) {
     return toStr(result.id);
@@ -714,13 +776,13 @@ function readAttributes(result) {
   return attrs;
 }
 __name(readAttributes, "readAttributes");
-function toPassage(rawId, attrs, prefix) {
-  const distance = Number(attrs.vector_score ?? "1");
+function toPassage(rawId, attrs, prefix, scoreSource = "vector", inlineScore) {
+  const score = scoreSource === "lexical" ? Number(inlineScore ?? "0") : 1 - Number(attrs.vector_score ?? "1");
   const source = attrs.source;
   return {
     id: rawId.startsWith(prefix) ? rawId.slice(prefix.length) : rawId,
     text: attrs.text ?? "",
-    score: 1 - distance,
+    score: Number.isFinite(score) ? score : 0,
     ...source !== void 0 ? {
       source
     } : {},
@@ -894,6 +956,25 @@ function tokenize(text) {
 }
 __name(tokenize, "tokenize");
 
+// src/lexical-retriever.ts
+var LexicalRetriever = class {
+  static {
+    __name(this, "LexicalRetriever");
+  }
+  store;
+  constructor(store) {
+    this.store = store;
+  }
+  async retrieve(query, options = {}) {
+    return this.store.searchText(query, {
+      topK: options.topK ?? 5,
+      ...options.filter !== void 0 ? {
+        filter: options.filter
+      } : {}
+    });
+  }
+};
+
 // src/hybrid-retriever.ts
 var HybridRetriever = class {
   static {
@@ -1081,6 +1162,7 @@ __name(createRetrievalTool, "createRetrievalTool");
   FilteredRetriever,
   HybridRetriever,
   KeywordRetriever,
+  LexicalRetriever,
   MemoryVectorStore,
   PgVectorStore,
   RedisVectorSchemaMismatchError,
@@ -1091,6 +1173,7 @@ __name(createRetrievalTool, "createRetrievalTool");
   createRetrievalTool,
   documentIdOf,
   ingestChunks,
-  ingestDocuments
+  ingestDocuments,
+  isLexicalVectorStore
 });
 //# sourceMappingURL=index.cjs.map