@dudousxd/nestjs-agent-rag 0.6.0 → 0.8.0

package/README.md

@@ -29,6 +29,48 @@ await ingestDocuments(
 const retriever = new EmbeddingRetriever(embedder, store);
 ```
 
+## Chunking record-shaped text
+
+The default chunker is structure-blind: it fills a `chunkSize` window, breaks at the latest
+paragraph → sentence → word boundary inside it, and carries `overlap` characters across the seam.
+That is the right rule for prose — the two halves are still sentences, and the overlap rescues the
+one that got cut.
+
+It is the wrong rule for text whose boundaries mean something. A spreadsheet flattened to one
+field-labelled record per row has real boundaries the chunker does not know exist, so a cut lands
+mid-record routinely — and the half holding the row identifier and the half holding the value end up
+in different chunks, where **neither can answer a question about that row**. Tell it where the
+records are:
+
+```ts
+await ingestDocuments([{ id: 'mvr-2026', text: rows.join('\n') }], {
+  embedder,
+  store,
+  chunkSize: 800,
+  separator: '\n', // one record per line; a record is never split across chunks
+});
+```
+
+Records are packed greedily up to `chunkSize` and never split — except one longer than `chunkSize`,
+which has nowhere to go and is character-split as usual. That fallback is confined to the single
+record that could not fit, rather than applied to the whole document.
+
+**`overlap` is ignored when `separator` is set**, deliberately. Overlap exists to rescue a sentence a
+boundary cut in half; a boundary that never falls inside a record has nothing to rescue, and carrying
+the previous chunk's tail in would duplicate whole records into their neighbours — paying for the
+same rows twice at embedding time and letting one row match from two places.
+
+Measured on a 200-row, 15-column sheet ingested twice — same bytes, same embedder, same questions,
+only the cut positions differing — blind chunking left 27% of records split, and on questions
+targeting a field on the far side of such a cut, BM25 answered 54/66 (MRR 0.818) against 66/66
+(MRR 1.000) with records kept whole. The dense leg scored 7/66 either way: 200 near-identical rows
+produce near-identical vectors, so that leg cannot do row lookup at all and chunking does not change
+it. The win is the lexical leg's, and it only shows up on questions whose answer sits far from the
+row's rare identifying token — ask about a field next to the vehicle id and both arms score 1.000,
+because BM25 finds that token whatever the chunking did around it.
+
+Leave `separator` unset for prose. Nothing about the default path changed.
+
 ## Two ways to use it with the agent
 
 **Agentic (default) — the model decides when to search:**
@@ -171,6 +213,38 @@ const retriever = new HybridRetriever([
 ]);
 ```
 
+### The fused score is not a similarity — do not threshold it
+
+Rank-based fusion is what makes RRF robust across incompatible scales, and the price is that its
+output is a function of **rank alone**. It never sees a similarity. Two things follow:
+
+- The score on a fused passage is **not comparable** to the score on one from `EmbeddingRetriever`,
+  even though it arrives in the same field, of the same type, through the same `Retriever` interface.
+  There is no discriminator — a consumer holding a `Passage[]` cannot tell which kind of number it
+  has. If you show scores to a user or a model, know which retriever produced them.
+- Its range is narrow, fixed and quality-blind: every possible value lies in
+  `[minWeight / (k + fetchTopK), Σ weights / (k + 1)]`, which with the defaults is
+  `[0.0125, 0.0328]`. A perfect match and the least-bad member of a set of terrible matches both land
+  around `0.016`.
+
+What a fused score measures is how strongly the legs *agree*, and agreement tracks nothing about
+correctness: two legs returning the same nearest nothing reinforce it to the same ceiling a unanimous
+correct answer gets. Measured on one corpus, the dense leg alone separated cleanly (weakest real
+answer `0.244`, loudest absent-topic answer `0.143`) while those same queries after fusion scored
+`0.032` and `0.033` — inverted.
+
+So a relevance floor goes on the legs, never on the fusion:
+
+```ts
+const retriever = new HybridRetriever([
+  new EmbeddingRetriever(embedder, store, { minScore: 0.2 }), // ← floor here
+  new LexicalRetriever(store),
+]);
+```
+
+`HybridRetriever` has no `minScore` of its own on purpose: the option would be an invitation to
+apply one to a number that cannot support it.
+
 Nothing else changes: no chunks to hand the retriever, no state in the process, no deletes to mirror.
 This is the only option that works when **ingestion and search run in different processes** — a
 worker that ingests and an API pod that searches. The API pod never sees the chunks, so an in-process
@@ -236,9 +310,13 @@ library upgrade. The interface is open, so it can be added later without changin
 
 ## API
 
-- `chunkText(text, { chunkSize?, overlap? })` — overlapping, boundary-aware chunks.
-- `ingestDocuments(docs, { embedder, store, chunkSize?, overlap? })` — chunk → embed → upsert.
-- `EmbeddingRetriever(embedder, store)` — the core `Retriever` from an embedder + vector store.
+- `chunkText(text, { chunkSize?, overlap?, separator? })` — overlapping, boundary-aware chunks;
+  `separator` packs whole records instead and never splits one (`overlap` is ignored in that mode).
+- `ingestDocuments(docs, { embedder, store, chunkSize?, overlap?, separator? })` — chunk → embed →
+  upsert.
+- `EmbeddingRetriever(embedder, store, { minScore? })` — the core `Retriever` from an embedder +
+  vector store. `minScore` drops passages below a similarity floor, so an unanswerable query can
+  return nothing.
 - `LexicalRetriever(store)` — the core `Retriever` from a store's own full-text index (BM25).
 - `KeywordRetriever` — in-process BM25 over chunks you feed it; the single-process lexical half:
   `add(chunks)`, `remove(documentId)`, `clear()`, `size`.

package/dist/index.cjs

@@ -51,6 +51,9 @@ function chunkText(text, options = {}) {
   if (normalized.length === 0) {
     return [];
   }
+  if (options.separator !== void 0 && options.separator.length > 0) {
+    return chunkByRecord(normalized, chunkSize, options.separator);
+  }
   if (normalized.length <= chunkSize) {
     return [
       normalized
@@ -80,6 +83,40 @@ function chunkText(text, options = {}) {
   return chunks;
 }
 __name(chunkText, "chunkText");
+function chunkByRecord(text, chunkSize, separator) {
+  const chunks = [];
+  let current = "";
+  const flush = /* @__PURE__ */ __name(() => {
+    if (current.length > 0) {
+      chunks.push(current);
+      current = "";
+    }
+  }, "flush");
+  for (const record of text.split(separator)) {
+    const trimmed = record.trim();
+    if (trimmed.length === 0) {
+      continue;
+    }
+    if (trimmed.length > chunkSize) {
+      flush();
+      chunks.push(...chunkText(trimmed, {
+        chunkSize,
+        overlap: 0
+      }));
+      continue;
+    }
+    const candidate = current.length > 0 ? `${current}${separator}${trimmed}` : trimmed;
+    if (candidate.length > chunkSize) {
+      flush();
+      current = trimmed;
+    } else {
+      current = candidate;
+    }
+  }
+  flush();
+  return chunks;
+}
+__name(chunkByRecord, "chunkByRecord");
 
 // src/vector-store.ts
 function isLexicalVectorStore(store) {
@@ -1337,9 +1374,11 @@ var EmbeddingRetriever = class {
   }
   embedder;
   store;
-  constructor(embedder, store) {
+  options;
+  constructor(embedder, store, options = {}) {
     this.embedder = embedder;
     this.store = store;
+    this.options = options;
   }
   async retrieve(query, options = {}) {
     const [embedding] = await this.embedder.embed([
@@ -1348,12 +1387,14 @@ var EmbeddingRetriever = class {
     if (embedding === void 0) {
       return [];
     }
-    return this.store.search(embedding, {
+    const passages = await this.store.search(embedding, {
       topK: options.topK ?? 5,
       ...options.filter !== void 0 ? {
         filter: options.filter
       } : {}
     });
+    const { minScore } = this.options;
+    return minScore === void 0 ? passages : passages.filter((passage) => passage.score >= minScore);
   }
 };