@dudousxd/nestjs-agent-rag 0.5.0 → 0.7.0

package/README.md

@@ -99,6 +99,57 @@ before the repair only become filterable on that key once re-ingested). A change
 model swap — can't be repaired without a full reindex, so it throws `RedisVectorSchemaMismatchError`
 instead of silently leaving the index on the old width.
 
+## Changing metadata without re-embedding
+
+Some metadata is derived from the document's content and only changes when the content does (its
+size, its mime type, its ingestion fingerprint). Some isn't: who may see it, which bases or teams it
+is relevant to, what it was last classified as. That second kind changes on its own schedule, and
+without a way to say so you are left choosing between re-embedding a whole document to change a
+string — paying the model bill, and the ingestion budget, for a reclassification — or refusing to
+stamp the dimension onto chunks at all and resolving it at query time instead, which turns a filter
+the index could have applied into a join you have to.
+
+`updateMetadata` is that way to say so. It rewrites the metadata of **every chunk** of a document and
+touches neither the text nor the embeddings:
+
+```ts
+// this document now covers bases B and C, and no longer A. No embedding is recomputed.
+const chunks = await store.updateMetadata(documentId, { bases: ['B', 'C'] });
+```
+
+The patch is applied as a shallow [JSON Merge Patch](https://www.rfc-editor.org/rfc/rfc7386):
+
+| in the patch | effect |
+| --- | --- |
+| key absent | the stored value is kept — this is a **partial** update, not a replacement |
+| `key: value` | the stored value is replaced |
+| `key: [a, b]` | replaced **wholesale** — an array is a set-valued dimension, not something to append to |
+| `key: null` | the key is **removed**. This is the only way to remove one |
+| `key: undefined` | **ignored**, as if the patch had not mentioned the key |
+
+`undefined` is ignored rather than treated as a removal on purpose. It does not survive
+`JSON.stringify`, and a patch routinely crosses a JSON boundary (an HTTP body, a workflow step
+payload) on its way in, so honouring it would make `{ audience: undefined }` mean *delete* on one
+side of that boundary and *no-op* on the other. It is also what `{ ...doc, audience: doc.audience }`
+produces by accident — and an accidental deletion of an access-control dimension is the worst thing
+this API could do. The cost is that a literal `null` cannot be stored as a value, which is no loss:
+"absent" and "null" filter identically anyway.
+
+The return value is the number of chunks written. **An unknown document id updates nothing and
+returns `0` rather than throwing** — like `remove`, this is a reconciliation-shaped call, made by a
+loop diffing your source of truth against the index, and such a loop races with ingestion and
+deletion by construction; an exception would make the ordinary case an error you had to catch and
+swallow. (A patch that writes nothing also returns `0`: the count is chunks *written*, not chunks
+*matched*.)
+
+Both representations move together. `RedisVectorStore` stores metadata twice — as `meta_<field>`
+TAGs, which is what RediSearch filters on, and as a `metadata_json` blob, which is what comes back on
+a `Passage` — and `updateMetadata` rewrites both from the same merged object, dropping the TAG of a
+removed key rather than leaving it matching a value the document no longer carries. A patched chunk
+is indistinguishable from a re-ingested one: filtering on the new value finds it, filtering on the
+old value does not, and the metadata read back off a retrieved passage agrees with both. (A plain
+`HSET` is all RediSearch needs to re-index — proven against a real Redis Stack, not assumed.)
+
 ## Hybrid search (dense + lexical)
 
 `HybridRetriever` fuses several retrievers with Reciprocal Rank Fusion. RRF is **rank**-based, so the
@@ -120,6 +171,38 @@ const retriever = new HybridRetriever([
 ]);
 ```
 
+### The fused score is not a similarity — do not threshold it
+
+Rank-based fusion is what makes RRF robust across incompatible scales, and the price is that its
+output is a function of **rank alone**. It never sees a similarity. Two things follow:
+
+- The score on a fused passage is **not comparable** to the score on one from `EmbeddingRetriever`,
+  even though it arrives in the same field, of the same type, through the same `Retriever` interface.
+  There is no discriminator — a consumer holding a `Passage[]` cannot tell which kind of number it
+  has. If you show scores to a user or a model, know which retriever produced them.
+- Its range is narrow, fixed and quality-blind: every possible value lies in
+  `[minWeight / (k + fetchTopK), Σ weights / (k + 1)]`, which with the defaults is
+  `[0.0125, 0.0328]`. A perfect match and the least-bad member of a set of terrible matches both land
+  around `0.016`.
+
+What a fused score measures is how strongly the legs *agree*, and agreement tracks nothing about
+correctness: two legs returning the same nearest nothing reinforce it to the same ceiling a unanimous
+correct answer gets. Measured on one corpus, the dense leg alone separated cleanly (weakest real
+answer `0.244`, loudest absent-topic answer `0.143`) while those same queries after fusion scored
+`0.032` and `0.033` — inverted.
+
+So a relevance floor goes on the legs, never on the fusion:
+
+```ts
+const retriever = new HybridRetriever([
+  new EmbeddingRetriever(embedder, store, { minScore: 0.2 }), // ← floor here
+  new LexicalRetriever(store),
+]);
+```
+
+`HybridRetriever` has no `minScore` of its own on purpose: the option would be an invitation to
+apply one to a number that cannot support it.
+
 Nothing else changes: no chunks to hand the retriever, no state in the process, no deletes to mirror.
 This is the only option that works when **ingestion and search run in different processes** — a
 worker that ingests and an API pod that searches. The API pod never sees the chunks, so an in-process
@@ -187,11 +270,66 @@ library upgrade. The interface is open, so it can be added later without changin
 
 - `chunkText(text, { chunkSize?, overlap? })` — overlapping, boundary-aware chunks.
 - `ingestDocuments(docs, { embedder, store, chunkSize?, overlap? })` — chunk → embed → upsert.
-- `EmbeddingRetriever(embedder, store)` — the core `Retriever` from an embedder + vector store.
+- `EmbeddingRetriever(embedder, store, { minScore? })` — the core `Retriever` from an embedder +
+  vector store. `minScore` drops passages below a similarity floor, so an unanswerable query can
+  return nothing.
 - `LexicalRetriever(store)` — the core `Retriever` from a store's own full-text index (BM25).
 - `KeywordRetriever` — in-process BM25 over chunks you feed it; the single-process lexical half:
   `add(chunks)`, `remove(documentId)`, `clear()`, `size`.
 - `HybridRetriever(retrievers, { k?, fetchTopK?, weights? })` — RRF fusion of the two.
-- `MemoryVectorStore` / `PgVectorStore` / `RedisVectorStore` — `VectorStore` adapters.
+- `MemoryVectorStore` / `PgVectorStore` / `RedisVectorStore` — `VectorStore` adapters:
+  `upsert(records)`, `search(embedding, opts)`, `remove(documentId)`,
+  `updateMetadata(documentId, patch)`, `listDocuments(filter?)`.
+- `applyMetadataPatch(metadata, patch)` — the `MetadataPatch` merge semantics, exported so your own
+  `VectorStore` can implement `updateMetadata` with exactly the same ones.
 - `isLexicalVectorStore(store)` — type guard for the optional `LexicalVectorStore` capability.
 - `createRetrievalTool(retriever, { name?, description?, topK? })` — the agentic-retrieval tool.
+
+## Collection maintenance
+
+`remove` and `listDocuments` can only enumerate documents *with* their metadata and delete them *one
+at a time*, which makes routine maintenance quadratic-ish: dropping a 2 000-document collection that
+way is one `listDocuments()` that parses a metadata blob per chunk, then 2 000 `remove()` calls — and
+on Redis each of those is a `SCAN` over the whole keyspace, queues and app cache included.
+
+So `VectorStore` **requires** four more methods, each doing the same work as one filtered query. They
+are not an optional capability and there is nothing to narrow to — every store has them:
+
+```ts
+await store.countChunks({ collection: 'handbook' });     // FT.SEARCH LIMIT 0 0 / count(*)
+await store.listDocumentIds({ collection: 'handbook' }); // ids only — no metadata round-trip
+await store.removeWhere({ collection: 'handbook' });     // one query, returns chunks removed
+await store.removeMany(['doc-a', 'doc-b']);              // N documents, one pass
+```
+
+`removeWhere` is the destructive one, so it has two rules worth knowing before you call it:
+
+| filter | result |
+|---|---|
+| `{ collection: 'handbook' }` | deletes that scope, returns the chunk count |
+| `{ audience: [] }` (or any empty array) | **deletes nothing**, returns `0` — the empty array is this package's deny primitive, and a deny must never be read as "no filter" |
+| `{}` | throws `UnsafeRemovalError` — an empty filter is far more often a filter built wrong than a deliberate request to wipe the store |
+| `{ notFilterable: 'x' }` (Redis) | throws `UnsafeRemovalError` — the key has no `meta_*` TAG to match on |
+
+Deliberate mass deletion is still available; it just has to be spelled out:
+`await store.removeMany(await store.listDocumentIds())`.
+
+`VectorStore.remove` deliberately stays a `SCAN` on the Redis adapter. Turning it into an `FT.SEARCH`
+would mean stamping the document id as an indexed field at write time, and chunks written before that
+change carry no such field — the search would find none of them and `remove` would silently stop
+deleting. The methods above need no new field, so they have no migration.
+
+### Why these are required and `searchText` is optional
+
+A method is **required on `VectorStore`** when every backend can implement it from what it already
+stores. Enumeration and bulk deletion are CRUD over records the store already holds — the same
+footing as `remove` and `listDocuments` — so making them optional would only cost every consumer a
+permanent `if (supported)` and an unreachable "and if it isn't?" branch.
+
+A method goes in an **optional capability interface** when some backend genuinely *cannot* provide it
+without infrastructure the consumer has to adopt. `LexicalVectorStore.searchText` is the only one
+today, and it earns it: RediSearch has BM25 natively, Postgres needs a `tsvector` column, a GIN index
+and a migration you run and watch, and `MemoryVectorStore` would have to implement BM25 itself.
+
+The test is whether a backend can do it at all — not whether adding it would inconvenience an
+implementer who has already shipped.