@absolutejs/rag 0.8.0-beta.0 → 0.9.0

package/CHANGELOG.md

@@ -0,0 +1,24 @@
+# Changelog
+
+All notable changes to `@absolutejs/rag`.
+
+This file is generated by `absolute-changelog` from the entries in
+`changelog/`. Edit an entry, not this file — and add new ones under
+`changelog/unreleased/`.
+
+## 0.9.0 — 2026-09-16
+
+### Breaking
+
+- **Reject unsupported lexical and hybrid retrieval instead of silently omitting keyword results; preserve requested channels for scoped heuristic queries** (`createRAGCollection`, `createHeuristicRAGRetrievalStrategy`)
+  _Migration:_ Use a store implementing queryLexical for lexical/hybrid retrieval, or explicitly select vector mode on vector-only stores. Scoped heuristic queries now retain the requested channels; update trace consumers from Scoped direct route to Scoped retrieval route.
+
+### Added
+
+- **Allow servers to request a smaller original-source search result set while preserving candidate ranking, authorization, and exact token budgeting** (`createRAGOriginalTextTools`)
+- **Add verbatim versioned text ingestion and authorized hybrid source tools with caller-supplied model token budgets; preserve multilingual keyword matches** (`chunkRAGOriginalText`, `readRAGOriginalText`, `createRAGOriginalTextTools`)
+
+### Changed
+
+- **Forward AI context policy through RAG WebSocket and SSE chat; accept the next AI 0.1 release while retaining compatibility with the prior AI peer range** (`ragChat`, `createRAGWorkflow`, `useRAG`, `useRAGWorkflow`, `useRAGStream`, `createRAG`, `RAGWorkflowService`, `RAGStreamService`)
+- **Count a complete authorized source-result envelope once when it fits, avoiding serial tokenizer round trips while retaining exact budget validation and oversized-result packing** (`createRAGOriginalTextTools`)

package/README.md

@@ -41,6 +41,19 @@ const results = await searchDocuments(collection, {
 
 The built-in memory store supports development and tests. Published adapters provide PostgreSQL with pgvector, SQLite with optional vec0 acceleration, and Pinecone behind the same `RAGVectorStore` contract. Lexical and vector results can be fused, transformed, and reranked with provider or heuristic rerankers.
 
+### Retrieval channel requirements
+
+Lexical and hybrid retrieval require a store implementing `queryLexical`.
+A backend without that capability raises an actionable error before vector
+embedding/search instead of silently returning vector-only or empty results.
+Explicit vector retrieval remains supported on vector-only backends. Configure
+a lexical-capable adapter before requesting hybrid retrieval; a mode flag does
+not add a missing backend capability.
+
+The built-in heuristic strategy preserves the requested retrieval mode when a
+query is scoped by source or document ID. Scope narrows the searchable corpus;
+it does not remove the need for exact keyword matches within that corpus.
+
 ## Ingestion and source sync
 
 The ingestion pipeline handles files, directories, uploads, URLs, PDFs, office documents, archives, images, and media transcripts. Scheduled connectors can keep collections synchronized from email, GitHub, sitemaps, feeds, directories, and S3-compatible storage.
@@ -57,3 +70,43 @@ The ingestion pipeline handles files, directories, uploads, URLs, PDFs, office d
 - `@absolutejs/rag/ui` exposes presentation-neutral UI contracts.
 
 Pair the retrieval runtime with `@absolutejs/ai` when retrieved context should feed a model or streaming assistant.
+
+### Verbatim original text evidence
+
+Use `chunkRAGOriginalText({ sourceId, version, text }, options)` when citations
+must resolve against an immutable text original. This opt-in path preserves
+whitespace and Unicode instead of normalizing or extracting document formats.
+Each chunk includes `metadata.sourceLocator` with the source ID, immutable
+version and UTF-16 `start`/`end` offsets. `readRAGOriginalText` validates identity,
+version and range before returning the exact original slice. Store and authorize
+the original separately; a locator is not an access grant.
+
+`createRAGOriginalTextTools({ collection, filter, loadSource, budget })` provides
+`search_text_source` and `read_text_source` AI tools. It requests real hybrid
+retrieval with diversity and verifies evidence against originals. `filter` is a
+server-owned scope; `loadSource(id, version)` must reauthorize every read and
+return `null` for inaccessible/deleted versions. The tools require
+`budget: { maxTokens, countTokens }` using the model tokenizer and a budget
+reserved by the AI context policy. Whole passages are selected within that
+budget; omitted passages are flagged rather than silently truncated. Add the
+final tools/instructions before budgeting the model request. Search traces are
+available through `onTrace`; they contain retrieval metadata and should not be
+copied wholesale into public logs. Servers may set `searchTopK` to an integer
+from 1 to 48 (default 12) for a smaller initial evidence lookup. Candidate ranking
+still considers up to 48 matches; authorization and token-budget checks are
+unchanged. A small initial result set does not establish that other facts are
+absent: retain broader search/read tools for missing or ambiguous evidence.
+
+Keyword matching uses Unicode word segmentation and canonical normalization.
+English suffix rules only apply to ASCII words. This improves multilingual exact
+matches; it does not replace evaluation of the selected embedding model.
+
+#### AI context policy compatibility
+
+With the AI 0.1 context-policy release, RAG chat validates the final assembled
+retrieval context before every model request. Its `contextPolicy` config is
+forwarded to both WebSocket and SSE generation. Use a working token target or a
+saved-source recovery callback when appropriate. Providers without capacity
+support require an explicit `contextPolicy: false` raw opt-out. Older supported
+AI peers retain their previous behavior; upgrading RAG alone does not add the AI
+0.1 capacity policy. No model-capacity numbers are defined in RAG.

package/changelog.json

@@ -0,0 +1,61 @@
+{
+	"contract": 1,
+	"name": "@absolutejs/rag",
+	"releases": [
+		{
+			"changes": [
+				{
+					"kind": "changed",
+					"summary": "Forward AI context policy through RAG WebSocket and SSE chat; accept the next AI 0.1 release while retaining compatibility with the prior AI peer range",
+					"symbols": [
+						"ragChat",
+						"createRAGWorkflow",
+						"useRAG",
+						"useRAGWorkflow",
+						"useRAGStream",
+						"createRAG",
+						"RAGWorkflowService",
+						"RAGStreamService"
+					]
+				},
+				{
+					"kind": "added",
+					"summary": "Allow servers to request a smaller original-source search result set while preserving candidate ranking, authorization, and exact token budgeting",
+					"symbols": [
+						"createRAGOriginalTextTools"
+					]
+				},
+				{
+					"kind": "changed",
+					"summary": "Count a complete authorized source-result envelope once when it fits, avoiding serial tokenizer round trips while retaining exact budget validation and oversized-result packing",
+					"symbols": [
+						"createRAGOriginalTextTools"
+					]
+				},
+				{
+					"kind": "added",
+					"summary": "Add verbatim versioned text ingestion and authorized hybrid source tools with caller-supplied model token budgets; preserve multilingual keyword matches",
+					"symbols": [
+						"chunkRAGOriginalText",
+						"readRAGOriginalText",
+						"createRAGOriginalTextTools"
+					]
+				},
+				{
+					"kind": "breaking",
+					"migration": {
+						"instruction": "Use a store implementing queryLexical for lexical/hybrid retrieval, or explicitly select vector mode on vector-only stores. Scoped heuristic queries now retain the requested channels; update trace consumers from Scoped direct route to Scoped retrieval route.",
+						"manual": true
+					},
+					"summary": "Reject unsupported lexical and hybrid retrieval instead of silently omitting keyword results; preserve requested channels for scoped heuristic queries",
+					"symbols": [
+						"createRAGCollection",
+						"createHeuristicRAGRetrievalStrategy"
+					]
+				}
+			],
+			"date": "2026-09-16",
+			"version": "0.9.0"
+		}
+	]
+}

package/dist/adapter-kit/index.js

@@ -11437,7 +11437,8 @@ var STOP_WORDS = new Set([
   "which",
   "why"
 ]);
-var tokenize = (value) => value.toLowerCase().split(/[^a-z0-9]+/i).map((token) => token.trim()).filter((token) => !STOP_WORDS.has(token)).map((token) => token.endsWith("ies") && token.length > 3 ? `${token.slice(0, -3)}y` : token.endsWith("ing") && token.length > 5 ? token.slice(0, -3) : token.endsWith("ed") && token.length > 4 ? token.slice(0, -2) : token.endsWith("es") && token.length > 4 ? token.slice(0, -2) : token.endsWith("s") && token.length > 3 ? token.slice(0, -1) : token).filter((token) => token.length > 1);
+var wordSegmenter = new Intl.Segmenter("und", { granularity: "word" });
+var tokenize = (value) => value.normalize("NFC").toLowerCase().split(/[^\p{L}\p{M}\p{N}]+/u).flatMap((part) => /^[a-z0-9]+$/.test(part) ? [part] : [...wordSegmenter.segment(part)].filter((segment) => segment.isWordLike).map((segment) => segment.segment)).map((token) => token.trim()).filter((token) => !STOP_WORDS.has(token)).map((token) => !/^[a-z]+$/.test(token) ? token : token.endsWith("ies") && token.length > 3 ? `${token.slice(0, -3)}y` : token.endsWith("ing") && token.length > 5 ? token.slice(0, -3) : token.endsWith("ed") && token.length > 4 ? token.slice(0, -2) : token.endsWith("es") && token.length > 4 ? token.slice(0, -2) : token.endsWith("s") && token.length > 3 ? token.slice(0, -1) : token).filter((token) => token.length > 1 || /[^a-z]/.test(token));
 var BM25_K1 = 1.2;
 var BM25_B = 0.75;
 var collectMetadataStrings = (value) => {
@@ -11454,7 +11455,7 @@ var collectMetadataStrings = (value) => {
 };
 var normalizeSourceForLexical = (source) => source.replace(/[#/_.-]+/g, " ").replace(/\bmd\b/g, "markdown").replace(/\bpptx\b/g, "presentation").replace(/\bxlsx\b/g, "spreadsheet workbook sheet").replace(/\bmp3\b/g, "audio transcript media").replace(/\bmp4\b/g, "video transcript media").replace(/\bzip\b/g, "archive bundle");
 var toFieldText = (value) => collectMetadataStrings(value).filter(Boolean).join(" ");
-var normalizeLooseText = (value) => value.toLowerCase().replace(/[^a-z0-9]+/g, " ").trim().replace(/\s+/g, " ");
+var normalizeLooseText = (value) => value.normalize("NFC").toLowerCase().replace(/[^\p{L}\p{M}\p{N}]+/gu, " ").trim().replace(/\s+/g, " ");
 var scoreLoosePhraseMatch = (query, text) => {
   const normalizedQuery = normalizeLooseText(query);
   const normalizedText = normalizeLooseText(text ?? "");
@@ -20788,7 +20789,13 @@ var annotateRetrievalQueryOrigin = (input) => {
   }));
 };
 var shouldRunVectorRetrieval = (mode) => mode === "vector" || mode === "hybrid";
-var shouldRunLexicalRetrieval = (mode, store) => mode === "lexical" || mode === "hybrid" && Boolean(store.queryLexical);
+var shouldRunLexicalRetrieval = (mode, store) => {
+  if (mode === "vector")
+    return false;
+  if (!store.queryLexical)
+    throw new Error(`Retrieval mode "${mode}" requires a store with queryLexical. Configure a lexical-capable adapter or explicitly request vector retrieval.`);
+  return true;
+};
 var resolveRAGRetrievalStrategy = (retrievalStrategy) => {
   if (!retrievalStrategy) {
     return null;
@@ -23115,6 +23122,7 @@ var ragChat = (config) => {
       handleRAGRetrieved(ws, conversationId, assistantMessageId, sources, retrievalStartedAt, retrievedAt, retrievalDurationMs, trace);
       await streamAI(ws, conversationId, assistantMessageId, {
         completeMeta: includeCompleteSources ? { sources } : undefined,
+        contextPolicy: config.contextPolicy,
         maxTurns: config.maxTurns,
         messages: [
           ...history,
@@ -29891,6 +29899,7 @@ var ragChat = (config) => {
       const messageWithContext = buildUserMessage(content, lastMessage.attachments, ragContext);
       const sseStream = streamAIToSSE(conversationId, assistantMessageId, {
         completeMeta: includeCompleteSources ? { sources } : undefined,
+        contextPolicy: config.contextPolicy,
         maxTurns: config.maxTurns,
         messages: [...userHistory, messageWithContext],
         model,
@@ -31389,21 +31398,21 @@ var RAG_NATIVE_QUERY_CANDIDATE_LIMIT = 1000;
 var RAG_VECTOR_DIMENSIONS_DEFAULT = 24;
 var TWO_THIRDS = 2 / 3;
 export {
-  summarizeSQLiteCandidateCoverage,
-  resolveAdaptiveNativeCandidateLimit,
-  rankRAGLexicalMatches,
-  ragChat as ragPlugin,
-  ragChat,
-  querySimilarity,
-  planNativeCandidateSearchK,
-  planNativeCandidateSearchBackfillK,
-  normalizeVector,
-  matchesMetadataFilterRecord,
-  createRAGVector,
-  createRAGCollection,
+  RAG_NATIVE_QUERY_CANDIDATE_LIMIT,
   RAG_VECTOR_DIMENSIONS_DEFAULT,
-  RAG_NATIVE_QUERY_CANDIDATE_LIMIT
+  createRAGCollection,
+  createRAGVector,
+  matchesMetadataFilterRecord,
+  normalizeVector,
+  planNativeCandidateSearchBackfillK,
+  planNativeCandidateSearchK,
+  querySimilarity,
+  ragChat,
+  ragChat as ragPlugin,
+  rankRAGLexicalMatches,
+  resolveAdaptiveNativeCandidateLimit,
+  summarizeSQLiteCandidateCoverage
 };
 
-//# debugId=D25B3C804028B59464756E2164756E21
+//# debugId=5D1CC4781115DD5864756E2164756E21
 //# sourceMappingURL=index.js.map