@dudousxd/nestjs-agent-rag 0.11.0 → 0.13.0

package/NOTICE

@@ -0,0 +1,31 @@
+@dudousxd/nestjs-agent-rag
+Copyright (c) Davide Carvalho. Licensed under the MIT License.
+
+The tree navigation retrieval in src/tree/ (buildDocumentTree, TreeNavigationRetriever,
+TwoStageRetriever, navigate_document) is an independent TypeScript implementation of the method of
+PageIndex ("vectorless", reasoning-based RAG: an LLM-navigated table-of-contents tree per document).
+Its tree-search and document-selection prompts are adapted from PageIndex's.
+
+PageIndex — https://github.com/VectifyAI/PageIndex
+
+MIT License
+
+Copyright (c) 2025 Vectify AI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -230,9 +230,13 @@ const store = new PgLexicalVectorStore(db, {
 });
 ```
 
-Ranking is `ts_rank_cd`; the query goes through `websearch_to_tsquery` (every word must appear), and
-when that finds nothing it retries with any word (`anyTermFallback`, default on) so a long question
-still matches. Without `column`, `ensureSchema`/`schemaStatements` add a GIN index over
+A question is searched by its meaningful terms: its stop words are dropped (English, Portuguese and
+Spanish lists by default, only the language(s) the question is written in; `stopWords` to change or
+`false` to keep every word), and rows holding any remaining term are ranked by the sum of the
+matched terms' IDF (BM25 without term frequencies), `ts_rank_cd` breaking ties, so three rare terms
+beat a row full of "the/of/which". Quoted phrases and `-word` go through `websearch_to_tsquery` as
+written first; `anyTermFallback: false` keeps only that (every word must appear). `keywordTerms()`
+is exported for other stores. Without `column`, `ensureSchema`/`schemaStatements` add a GIN index over
 `to_tsvector(config, text)` — the exact expression the query uses. On a populated table create it
 `CONCURRENTLY` in a migration rather than at boot (see below).
 
@@ -435,6 +439,90 @@ Had `PgVectorStore` simply grown `searchText`, every `isLexicalVectorStore(store
 started using it against an unindexed table on upgrade. Constructing `PgLexicalVectorStore` is the
 opt-in; the index is a migration you run and watch (`schemaStatements()` prints it).
 
+## Tree navigation retrieval (PageIndex-style)
+
+Chunk search ranks passages; on a **long structured document** (a regulation, an RFP, a contract, a
+10-K, a manual) it usually finds the right document and then loses the right page among hundreds of
+look-alike clause or financial-statement chunks. Tree navigation ("vectorless", reasoning-based, the
+method of [PageIndex](https://github.com/VectifyAI/PageIndex)) builds a table of contents per long
+document at ingestion — sections with page ranges and summaries — and at query time an LLM reads that
+outline and walks to the sections that answer, returning their text with the path it took.
+
+Measured on Flippy (149 questions, same answering model): on long GovCon regulations it beat hybrid
+dense+lexical search by **+24 points**, on FinanceBench 10-Ks by **+30**, and gave nothing on short
+wiki notes — at seconds of latency and LLM spend per query instead of 0.3 s. Hence the recommended
+shape: **hybrid first, tree navigation only inside the top long documents**.
+
+```ts
+import {
+  PgDocumentTreeStore, TreeNavigationRetriever, TwoStageRetriever, HybridRetriever,
+  createNavigateDocumentTool, createRetrievalTool, indexDocumentTree, openAiChatTreeLlm,
+} from '@dudousxd/nestjs-agent-rag';
+
+const trees = new PgDocumentTreeStore(db); // pg Pool, Drizzle db, postgres.js sql — like PgVectorStore
+await trees.ensureSchema();                // or copy trees.schemaStatements() into a migration
+const llm = openAiChatTreeLlm({ model: 'gpt-4.1-mini', apiKey }); // or treeLlmFromModelProvider(provider)
+
+// At upload, next to chunk + embed: only long documents get a tree (default ≥ 20 pages or ≥ 60k chars).
+await indexDocumentTree(
+  { pages, title: file.name },              // or { sections: [{ title, level, text, page }] }, + outline
+  { store: trees, documentId: file.id, llm, metadata: { tenant }, source: file.name },
+);
+
+const navigator = new TreeNavigationRetriever({ store: trees, llm });
+// Hybrid picks documents; long ones among the top 3 get navigated (at most 2), short ones keep chunks.
+const retriever = new TwoStageRetriever(hybrid, navigator, { navigateDocuments: 2 });
+// …or let the agent decide: search first, then open a long document.
+providers: [
+  provideAgentTool(createRetrievalTool(hybrid)),
+  provideAgentTool(createNavigateDocumentTool(navigator, { filter: (ctx) => ({ tenant: ctx.actor.tenantRef }) })),
+];
+```
+
+**Building a tree** (`buildDocumentTree` / `indexDocumentTree`) takes the cheapest structure that works
+and never loops:
+
+1. the document's own `outline` (PDF bookmarks), when you pass one;
+2. section titles/levels (`sections` input) or headings found in the text — markdown headers and the
+   keyword/numbered headings of regulations and filings (`PART 52`, `Subpart 19.8`, `Item 7.`,
+   `52.236–5 Material and Workmanship.`), with running page headers and table-of-contents listings
+   dropped;
+3. only when there is no usable structure, the LLM proposes the sections window by window — and only
+   if the whole document fits `maxStructureCalls`;
+4. otherwise, or on any failure, page groups (`Pages 21–25: <first line>`), grouped again when wide.
+
+Summaries go top-down (top levels matter most for navigation) until the **hard per-document budget**
+runs out (`budget: { maxCalls, maxInputTokens, maxOutputTokens, timeoutMs, callTimeoutMs }`, default 60
+calls / 400k input tokens / 5 min); the rest summarize themselves with their opening text. Every call
+is checked against the budget before it is made and nothing is retried, so a document the LLM cannot
+structure costs a bounded amount and still gets a usable tree (`tree.stats` records calls, tokens and
+`fallbackReason`). Builds are deterministic for a deterministic model and fingerprinted: re-indexing an
+unchanged document costs one store read and no LLM call, a changed one re-summarizes only the sections
+whose text changed (`reusedSummaries`), and `cachedTreeLlm(llm, cache)` makes repeated prompts free.
+
+**Navigating** (`TreeNavigationRetriever`): a tree whose outline fits `singlePassMaxChars` (default
+40k chars) is navigated in one call that sees the whole outline; a larger one is walked level by level
+(a beam of `beamWidth` nodes, at most `maxSteps` levels), the model choosing to `expand` or `read` each
+candidate. Each navigation has its own hard budget (default 6 calls / 60k input tokens / 60 s); running
+out returns what was chosen so far. `navigate()` / `search()` return the audit trail with the passages
+— every step's candidates, choices and stated reasoning, and each node's path — and every passage
+carries `documentId`, `nodeId`, `title`, `pageStart`/`pageEnd` and `path` in its metadata. As a plain
+`Retriever` it navigates every tree passing `filter` (the LLM picks `maxDocuments` of them from their
+descriptions when there are more).
+
+**Access control.** `DocumentTreeStore` filters (`get`, `getMany`, `list`) have exactly the vector
+stores' semantics — scalar equality, array match-any, empty-array deny — and the navigator, the
+two-stage retriever and the tool pass the caller's filter to every tree read.
+
+**LLM.** `TreeLlm` is one prompt in, one text reply out: `openAiChatTreeLlm` (any OpenAI-compatible
+`/chat/completions`; pass `body: { enable_thinking: false }` or similar to keep a reasoning model from
+spending the reply budget on thoughts), `treeLlmFromModelProvider` (the agent's `ModelProvider`), or
+your own function. Each request also carries its `task` as data, which is what `keywordTreeLlm()` — a
+deterministic, model-free stand-in for tests — answers from.
+
+PageIndex (© 2025 Vectify AI, MIT) is the method and the prompts' origin; this is an independent
+TypeScript implementation. See [NOTICE](./NOTICE).
+
 ## API
 
 - `chunkText(text, { chunkSize?, overlap?, separator? })` — overlapping, boundary-aware chunks;
@@ -460,7 +548,24 @@ opt-in; the index is a migration you run and watch (`schemaStatements()` prints
 - `instrumentRetriever(retriever, { describe?, collection? })` — wrap any `Retriever` so each
   retrieval publishes on `aviary:rag:retrieval` (`RAG_RETRIEVAL_CHANNEL`).
 - `openAiEmbeddings({ model, baseUrl?, apiKey?, dimensions?, batchSize?, maxInputChars?, headers?,
-  timeoutMs?, onUsage?, fetch? })` — an `EmbeddingProvider` over any OpenAI-compatible `/embeddings`.
+  timeoutMs?, onUsage?, onWarn?, fetch? })` — an `EmbeddingProvider` over any OpenAI-compatible
+  `/embeddings`. A batch the server refuses as too large (413, or TEI's `batch size N > maximum
+  allowed batch size M`; see `isBatchTooLarge`) is split in halves and retried, and later requests to
+  that server and model use the size that worked; each split is reported through `onWarn`.
+- `buildDocumentTree(input, { documentId, llm?, maxDepth?, budget?, summaries?, describe?,
+  detectHeadings?, llmStructure?, maxStructureCalls?, groupSize?, maxLeafUnits?, previous?, metadata?,
+  source? })` — a document's navigation tree (`{ tree, units, unchanged }`). `indexDocumentTree(input,
+  { store, minUnits?, minChars?, …build options })` builds, stores and skips short/unchanged documents.
+- `DocumentTreeStore` (SPI), `MemoryDocumentTreeStore`, `PgDocumentTreeStore(client, { table?,
+  unitsTable? })` with `schemaStatements()` / `ensureSchema()`.
+- `TreeNavigationRetriever({ store, llm, singlePassMaxChars?, beamWidth?, maxSteps?, maxNodes?, budget?,
+  maxPassageChars?, maxContextChars?, maxDocuments?, concurrency? })` — `retrieve`, `search` (with the
+  trail), `navigate(query, documentIds, { filter?, budget? })`.
+- `TwoStageRetriever(firstStage, navigator, { fetchTopK?, considerDocuments?, navigateDocuments?,
+  budget?, keepFirstStageChunks?, documentIdOf?, onNavigation? })`.
+- `createNavigateDocumentTool(navigator, { name?, description?, filter?, budget? })` — the
+  `navigate_document({ documentId, question })` tool.
+- `openAiChatTreeLlm`, `treeLlmFromModelProvider`, `cachedTreeLlm`, `keywordTreeLlm` — `TreeLlm`s.
 - `HttpReranker({ url, model?, apiKey?, format?, headers?, timeoutMs?, fetch? })` — a `Reranker` over
   a Cohere/Jina/Voyage/TEI-style `/rerank` endpoint.