nexus-ai-pro 1.21.0 → 1.22.0

package/API_STABILITY.md

@@ -456,6 +456,22 @@ with nexus-ai-pro and declares the core versions it works with as a peer depende
 - The access rules are a guarantee: loopback binding by default, a token on every request, the token
   in a header for every change, and loopback hosts only unless others are allowed.
 
+## Retrieval breadth stage (1.22.0)
+
+The loaders (`nexus-ai-pro/loaders/*`), the Redis, Pinecone, Weaviate, Chroma, and SQLite vector
+stores, the retrievers (`nexus-ai-pro/rag/retrievers`), and the MCP registry
+(`nexus-ai-pro/mcp/registry`) are experimental in production readiness. Their contracts follow the
+1.x rules.
+
+- Every vector store keeps the `VectorStore` contract and its contract test, which checks that a
+  metadata filter matches type as well as value.
+- A loader's document ids are stable: the path relative to a directory or repository, the final URL
+  of a page, and the file name plus key or position of a row or record. Chunk ids built from them do
+  not change in a minor release, so a reload keeps replacing the same chunks.
+- The REST stores' payload layout — `content`, `source`, `metadata` as JSON, and `meta_` filter fields
+  — is stable across 1.x, so an index written by one release is read by the next.
+- The registry reads `servers` and `mcpServers`; new configuration fields are optional.
+
 ## Durable execution stage (1.21.0)
 
 Functional workflows (`nexus-ai-pro/graph/functional`) and the SQLite adapters (`nexus-ai-pro/sqlite`)

package/CHANGELOG.md

@@ -4,6 +4,39 @@ Notable changes to this project are recorded here. The format follows [Keep a Ch
 
 ## [Unreleased]
 
+## [1.22.0] - 2026-09-29
+
+Retrieval and integration breadth. Getting documents in, finding the right ones, and reaching the
+tools a team already runs, without a catalogue of dependencies.
+
+### Added
+
+- **Document loaders** on `nexus-ai-pro/loaders/*`, one entry point per format: text files and
+  directories, Markdown with front matter, HTML, CSV, JSON and JSON Lines, PDF through a parser you
+  inject, web pages and sitemaps through the SSRF-safe fetch, and Git repositories. Each streams
+  documents, and `loadIntoStore()` splits them into chunks and adds them to one or more stores in
+  batches.
+- **Four more vector stores** behind the `VectorStore` contract, each through REST or an injected
+  client: Redis with RediSearch (`nexus-ai-pro/rag/redis`), Pinecone, Weaviate, and Chroma
+  (`nexus-ai-pro/rag/pinecone`, `/weaviate`, `/chroma`), plus `SqliteVectorStore` on
+  `nexus-ai-pro/sqlite/vectors`, ranked in JavaScript or by sqlite-vec. All pass the same contract
+  test, which now also checks that a filter matches a value's type.
+- **Composable retrievers** on `nexus-ai-pro/rag/retrievers`: BM25 keyword search (`KeywordIndex`),
+  hybrid search fused by reciprocal rank, reranking through any scorer or a chat model, maximal
+  marginal relevance, parent-document retrieval, and multi-query retrieval.
+- **Retrieval evaluators**: `recallAtK()` and `reciprocalRank()` on `nexus-ai-pro/evaluate`.
+- **An MCP registry** on `nexus-ai-pro/mcp/registry`: many servers from one configuration file —
+  including the file desktop MCP clients use — with per-server allow and deny lists, credentials from
+  the environment by placeholder, health checks, and named tool bundles.
+
+### Changed
+
+- `SqliteValue` includes bytes, which the drivers bind as blobs.
+
+### Fixed
+
+- `ingestDocuments()` honours `overlap: 0`. It used the default overlap of 150 characters instead.
+
 ## [1.21.0] - 2026-09-29
 
 Durable execution everywhere. Durability no longer requires drawing a graph, and a crash costs a
@@ -1085,7 +1118,8 @@ shared policy vocabulary is in place for a future release that revisits this.
 - URL fetching rejects unsafe private-network targets and limits response reads.
 - CLI and audit findings no longer disclose detected secret values.
 
-[Unreleased]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.21.0...HEAD
+[Unreleased]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.22.0...HEAD
+[1.22.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.21.0...v1.22.0
 [1.21.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.20.0...v1.21.0
 [1.20.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.19.0...v1.20.0
 [1.19.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.18.0...v1.19.0

package/README.md

@@ -109,7 +109,7 @@ from the doc comments. The guides live in the repository, so these links go to G
 | [Graphs](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/graphs.md) | Parallel branches, interrupts, checkpoints, time travel, subgraphs, caching, diagrams |
 | [Agents and tools](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/agents.md) | Graph-based agents with approvals and middleware, and the simple tool loop |
 | [Long-term memory](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/memory.md) | Namespaced memory with semantic search, in memory or Redis |
-| [MCP](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/mcp.md) | Borrowing tools from MCP servers and serving your own |
+| [MCP](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/mcp.md) | Borrowing tools from MCP servers, a registry of many, and serving your own |
 | [Traces](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/tracing.md) | Run trees, queries, feedback, comparison, and alerts |
 | [Evaluation](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/evaluation.md) | Datasets, evaluators, experiments, comparisons with a verdict, review queues, LLM judges |
 | [Prompts](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/prompts.md) | Typed templates, content versions, gated promotion, rollback, A/B splits, serving through outages |
@@ -120,6 +120,8 @@ from the doc comments. The guides live in the repository, so these links go to G
 | [Batch tiers](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/batch.md) | The providers' discounted batch APIs, resumable from any process |
 | [Caching](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/caching.md) | Exact and semantic response caches, with memory, Redis, and SQLite adapters |
 | [Grounding](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/grounding.md) | Retrieval, citations, verification, self-consistency, and knowledge graphs |
+| [Loaders](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/loaders.md) | Files, Markdown, HTML, CSV, JSON, PDF, web pages, sitemaps, and Git repositories into retrieval |
+| [Retrieval](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/retrieval.md) | Hybrid search, reranking, MMR, parent documents, and Redis, Pinecone, Weaviate, and Chroma stores |
 | [Embeddings and retrieval](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/embeddings.md) | Embeddings as a routed operation family, and RAG helpers |
 | [Images (experimental)](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/images.md) | Generation and masked edits over three backends, input safety, moderation, asset stores |
 | [Voice](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/voice.md) | Transcription, speech, voice turns, and voice sessions |

package/dist/evaluate/evaluators.d.ts

@@ -80,3 +80,21 @@ export declare function pairwise(options: {
 export declare function passRate(key?: string): SummaryEvaluator;
 /** Total cost of the experiment, so a quality gain that tripled the bill is visible. */
 export declare function totalCost(key?: string): SummaryEvaluator;
+/** Options for the retrieval evaluators. */
+export interface RetrievalEvaluatorOptions {
+    /** Results considered, from the top. Defaults to every result. */
+    k?: number;
+    /** The metric's name. */
+    key?: string;
+}
+/**
+ * The share of an example's relevant ids found in the output's top `k`. The output is a ranked list
+ * of ids or of chunks with an `id`, such as a retriever's results; the example's `expected` is the
+ * relevant id, or a list of them.
+ */
+export declare function recallAtK(options?: RetrievalEvaluatorOptions): Evaluator;
+/**
+ * One over the rank of the first relevant id in the output's top `k`, or 0 when none is there. Its
+ * mean over a dataset is the mean reciprocal rank, which rewards putting the right chunk first.
+ */
+export declare function reciprocalRank(options?: RetrievalEvaluatorOptions): Evaluator;

package/dist/evaluate/evaluators.js

@@ -135,6 +135,41 @@ export function totalCost(key = 'total-cost') {
         { key, score: results.reduce((total, result) => total + (result.cost ?? 0), 0) },
     ];
 }
+export function recallAtK(options = {}) {
+    return (context) => {
+        const relevant = relevantIds(context.example.expected);
+        const found = rankedIds(context.output).slice(0, options.k);
+        const hits = relevant.filter((id) => found.includes(id));
+        const score = relevant.length === 0 ? 1 : hits.length / relevant.length;
+        return {
+            key: options.key ?? (options.k ? `recall@${options.k}` : 'recall'),
+            score,
+            passed: hits.length === relevant.length,
+            ...(hits.length === relevant.length
+                ? {}
+                : { comment: `missing: ${relevant.filter((id) => !hits.includes(id)).join(', ')}` }),
+        };
+    };
+}
+export function reciprocalRank(options = {}) {
+    return (context) => {
+        const relevant = new Set(relevantIds(context.example.expected));
+        const rank = rankedIds(context.output)
+            .slice(0, options.k)
+            .findIndex((id) => relevant.has(id));
+        return { key: options.key ?? 'reciprocal-rank', score: rank < 0 ? 0 : 1 / (rank + 1), passed: rank === 0 };
+    };
+}
+function relevantIds(expected) {
+    if (Array.isArray(expected))
+        return expected.map(String);
+    return expected === undefined || expected === null ? [] : [String(expected)];
+}
+function rankedIds(output) {
+    if (!Array.isArray(output))
+        return [];
+    return output.map((item) => item !== null && typeof item === 'object' && 'id' in item ? String(item.id) : String(item));
+}
 function asText(value) {
     if (value === undefined || value === null)
         return '';

package/dist/evaluate/index.d.ts

@@ -1,6 +1,6 @@
 export { type CompareOptions, compareExperiments, type ExampleComparison, type ExperimentComparison, formatComparison, type MetricComparison, scoreMap, } from './compare.js';
 export { contentVersion, createDataset, type CreateDatasetOptions, datasetFromTraces, FileDatasetStore, FileExperimentStore, type FromTracesOptions, MemoryDatasetStore, MemoryExperimentStore, readExperiment, splitOf, } from './datasets.js';
-export { completed, contains, embeddingSimilarity, exactMatch, mustNotMatch, pairwise, passRate, totalCost, trajectory, type TrajectoryOptions, underCost, underLatency, } from './evaluators.js';
+export { completed, contains, embeddingSimilarity, exactMatch, mustNotMatch, pairwise, passRate, recallAtK, reciprocalRank, type RetrievalEvaluatorOptions, totalCost, trajectory, type TrajectoryOptions, underCost, underLatency, } from './evaluators.js';
 export { AnnotationQueue, type AnnotationQueueOptions, evaluateOnline, type OnlineEvaluationOptions, type OnlineEvaluationReport, type ReviewAnswer, type ReviewItem, type ReviewQuestion, } from './review.js';
 export { evaluate, type EvaluateOptions, type EvaluationTarget, stats, summarize } from './run.js';
 export type { Dataset, DatasetExample, DatasetStore, EvaluationContext, EvaluationScore, Evaluator, ExampleResult, Experiment, ExperimentStore, MetricSummary, SummaryEvaluator, } from '../types/evaluate.js';

package/dist/evaluate/index.js

@@ -1,5 +1,5 @@
 export { compareExperiments, formatComparison, scoreMap, } from './compare.js';
 export { contentVersion, createDataset, datasetFromTraces, FileDatasetStore, FileExperimentStore, MemoryDatasetStore, MemoryExperimentStore, readExperiment, splitOf, } from './datasets.js';
-export { completed, contains, embeddingSimilarity, exactMatch, mustNotMatch, pairwise, passRate, totalCost, trajectory, underCost, underLatency, } from './evaluators.js';
+export { completed, contains, embeddingSimilarity, exactMatch, mustNotMatch, pairwise, passRate, recallAtK, reciprocalRank, totalCost, trajectory, underCost, underLatency, } from './evaluators.js';
 export { AnnotationQueue, evaluateOnline, } from './review.js';
 export { evaluate, stats, summarize } from './run.js';

package/dist/loaders/csv.d.ts

@@ -0,0 +1,29 @@
+import type { DocumentSource, FileInput } from './index.js';
+import type { FileLoaderOptions } from './text.js';
+/** Options for `parseCsv()`. */
+export interface CsvParseOptions {
+    /** The field separator. Defaults to a comma; use `'\t'` for TSV. */
+    delimiter?: string;
+}
+/**
+ * Parses CSV into rows of fields, following RFC 4180: quoted fields may hold the delimiter, line
+ * breaks, and doubled quotes; lines may end in CRLF or LF; and a trailing line break adds no row.
+ */
+export declare function parseCsv(text: string, options?: CsvParseOptions): string[][];
+/** Options for `loadCsv()`. */
+export interface CsvLoaderOptions extends FileLoaderOptions, CsvParseOptions {
+    /**
+     * Columns that make up a row's text, written as `column: value` lines. Defaults to every column not
+     * named in `idColumn` or `metadataColumns`.
+     */
+    textColumns?: readonly string[];
+    /** The column whose value becomes the row's id, after the file name. Defaults to the row number. */
+    idColumn?: string;
+    /** Columns copied into the metadata, where a filter can select on them. */
+    metadataColumns?: readonly string[];
+}
+/**
+ * CSV files, one document per row. The first row names the columns. Each row's text lists its columns
+ * by name, so a model reading a chunk knows what each value is.
+ */
+export declare function loadCsv(inputs: FileInput | Iterable<FileInput>, options?: CsvLoaderOptions): AsyncGenerator<DocumentSource>;

package/dist/loaders/csv.js

@@ -0,0 +1,87 @@
+import { inputsOf, readText } from './input.js';
+export function parseCsv(text, options = {}) {
+    const delimiter = options.delimiter ?? ',';
+    if (delimiter.length !== 1 || delimiter === '"' || delimiter === '\n' || delimiter === '\r') {
+        throw new RangeError('delimiter must be one character other than a quote or a line break');
+    }
+    const rows = [];
+    let row = [];
+    let field = '';
+    let quoted = false;
+    let index = text.charCodeAt(0) === 0xfeff ? 1 : 0;
+    for (; index < text.length; index++) {
+        const char = text[index];
+        if (quoted) {
+            if (char !== '"')
+                field += char;
+            else if (text[index + 1] === '"') {
+                field += '"';
+                index++;
+            }
+            else
+                quoted = false;
+        }
+        else if (char === '"' && field === '')
+            quoted = true;
+        else if (char === delimiter) {
+            row.push(field);
+            field = '';
+        }
+        else if (char === '\n' || char === '\r') {
+            if (char === '\r' && text[index + 1] === '\n')
+                index++;
+            row.push(field);
+            rows.push(row);
+            row = [];
+            field = '';
+        }
+        else
+            field += char;
+    }
+    if (quoted)
+        throw new SyntaxError('The CSV ends inside a quoted field');
+    if (field !== '' || row.length > 0) {
+        row.push(field);
+        rows.push(row);
+    }
+    return rows;
+}
+export async function* loadCsv(inputs, options = {}) {
+    for (const input of inputsOf(inputs)) {
+        const { source, text } = await readText(input);
+        const [header, ...rows] = parseCsv(text, options);
+        if (!header)
+            continue;
+        const column = (name) => {
+            const index = header.indexOf(name);
+            if (index < 0)
+                throw new RangeError(`${source} has no column "${name}"`);
+            return index;
+        };
+        const idIndex = options.idColumn === undefined ? -1 : column(options.idColumn);
+        const metadataIndexes = (options.metadataColumns ?? []).map((name) => [name, column(name)]);
+        const textIndexes = options.textColumns
+            ? options.textColumns.map(column)
+            : header
+                .map((_, index) => index)
+                .filter((index) => index !== idIndex && !metadataIndexes.some(([, metadata]) => metadata === index));
+        for (const [rowIndex, fields] of rows.entries()) {
+            if (fields.length === 1 && fields[0] === '')
+                continue;
+            const key = idIndex >= 0 ? fields[idIndex] : String(rowIndex + 1);
+            yield {
+                id: `${source}#${key}`,
+                text: textIndexes
+                    .map((index) => `${header[index]}: ${fields[index] ?? ''}`)
+                    .filter((line) => !line.endsWith(': '))
+                    .join('\n'),
+                source,
+                metadata: {
+                    ...options.metadata,
+                    ...Object.fromEntries(metadataIndexes.map(([name, index]) => [name, fields[index] ?? ''])),
+                    row: rowIndex + 1,
+                },
+            };
+        }
+    }
+}

package/dist/loaders/git.d.ts

@@ -0,0 +1,40 @@
+import type { DocumentSource } from './index.js';
+import type { FileParser } from './text.js';
+/** Options for `loadGitRepository()`. */
+export interface GitLoaderOptions {
+    /** A branch or tag to read. Cloned at that ref; a local repository without it is read as checked out. */
+    ref?: string;
+    /** File extensions read, lower case with the dot, such as `['.md', '.ts']`. Defaults to every text file. */
+    extensions?: readonly string[];
+    /** Skips a path, relative to the repository root with `/` separators. */
+    ignore?: (path: string) => boolean;
+    /** A parser per extension, as for `loadDirectory()`. Other files load as plain text. */
+    parsers?: Record<string, FileParser>;
+    /** Files larger than this are skipped, in bytes. Defaults to 1 MB. */
+    maxFileBytes?: number;
+    /** The `git` executable. Defaults to `git` on the path. */
+    git?: string;
+    /** Time allowed for the clone, in milliseconds. Defaults to 2 minutes. */
+    timeoutMs?: number;
+    /** Application data added to every document. */
+    metadata?: Record<string, unknown>;
+}
+/** Raised when `git` fails, with what it printed. */
+export declare class GitLoaderError extends Error {
+    /** What `git` wrote to stderr. */
+    readonly stderr: string;
+    constructor(message: string, 
+    /** What `git` wrote to stderr. */
+    stderr: string);
+}
+/**
+ * The files a Git repository tracks, one document each. A remote URL — `https://`, `ssh://`,
+ * `git://`, `file://`, or `user@host:path` — is shallow-cloned into a temporary directory that is
+ * removed afterwards; a local path is read in place. Only tracked files are read, binary files and
+ * files over `maxFileBytes` are skipped, and each document carries the repository and commit in its
+ * metadata, so a citation can link to the exact revision.
+ *
+ * Runs the `git` executable, without a shell, with credential prompts and the `ext::` transport
+ * disabled. A clone keeps the committed line endings, so a commit loads the same on every platform.
+ */
+export declare function loadGitRepository(repository: string, options?: GitLoaderOptions): AsyncGenerator<DocumentSource>;

package/dist/loaders/git.js

@@ -0,0 +1,83 @@
+export class GitLoaderError extends Error {
+    stderr;
+    constructor(message, stderr) {
+        super(message);
+        this.stderr = stderr;
+        this.name = 'GitLoaderError';
+    }
+}
+const REMOTE = /^(?:https?|ssh|git|file):\/\/|^[\w.-]+@[\w.-]+:/;
+export async function* loadGitRepository(repository, options = {}) {
+    if (repository.startsWith('-'))
+        throw new RangeError('A repository must not start with "-"');
+    const { mkdtemp, readFile, rm } = await import('node:fs/promises');
+    const { tmpdir } = await import('node:os');
+    const path = await import('node:path');
+    const remote = REMOTE.test(repository);
+    const clone = remote || options.ref !== undefined;
+    const directory = clone ? await mkdtemp(path.join(tmpdir(), 'nexus-git-')) : repository;
+    try {
+        if (clone) {
+            await git([
+                'clone',
+                ...(remote ? ['--depth', '1'] : []),
+                '--no-tags',
+                ...(options.ref ? ['--branch', options.ref] : []),
+                '--',
+                remote ? repository : path.resolve(repository),
+                directory,
+            ], undefined, options);
+        }
+        const commit = (await git(['rev-parse', 'HEAD'], directory, options)).trim();
+        const files = (await git(['ls-files', '-z'], directory, options)).split('\0').filter(Boolean).sort();
+        const extensions = options.extensions && new Set(options.extensions.map((extension) => extension.toLowerCase()));
+        const maxFileBytes = options.maxFileBytes ?? 1_000_000;
+        for (const file of files) {
+            const extension = path.extname(file).toLowerCase();
+            if (extensions && !extensions.has(extension))
+                continue;
+            if (options.ignore?.(file))
+                continue;
+            let bytes;
+            try {
+                bytes = new Uint8Array(await readFile(path.join(directory, file)));
+            }
+            catch {
+                continue;
+            }
+            if (bytes.length > maxFileBytes || bytes.subarray(0, 8000).includes(0))
+                continue;
+            const metadata = { ...options.metadata, repository, commit, path: file };
+            const parse = options.parsers?.[extension];
+            if (parse) {
+                for await (const document of parse({ source: file, content: bytes })) {
+                    yield { ...document, metadata: { ...metadata, ...document.metadata } };
+                }
+            }
+            else {
+                yield { id: file, text: new TextDecoder().decode(bytes), source: file, metadata };
+            }
+        }
+    }
+    finally {
+        if (clone)
+            await rm(directory, { recursive: true, force: true });
+    }
+}
+async function git(args, cwd, options) {
+    const { execFile } = await import('node:child_process');
+    return new Promise((resolve, reject) => {
+        execFile(options.git ?? 'git', ['-c', 'protocol.ext.allow=never', '-c', 'core.quotePath=false', '-c', 'core.autocrlf=false', ...args], {
+            cwd,
+            timeout: options.timeoutMs ?? 120_000,
+            maxBuffer: 64 * 1024 * 1024,
+            env: { ...process.env, GIT_TERMINAL_PROMPT: '0', GIT_ASKPASS: '', SSH_ASKPASS: '' },
+            windowsHide: true,
+        }, (error, stdout, stderr) => {
+            if (error)
+                reject(new GitLoaderError(`git ${args[0]} failed: ${stderr.trim() || error.message}`, stderr));
+            else
+                resolve(stdout);
+        });
+    });
+}

package/dist/loaders/html.d.ts

@@ -0,0 +1,35 @@
+import type { DocumentSource, FileInput } from './index.js';
+import type { FileLoaderOptions } from './text.js';
+/** An HTML page as text, with what a loader needs from its head and its links. */
+export interface HtmlText {
+    /** The readable text. Headings become Markdown `#` lines and list items `- ` lines. */
+    text: string;
+    /** The `<title>`. */
+    title?: string;
+    /** The `<meta name="description">`. */
+    description?: string;
+    /** Every `href` on an `<a>`, resolved against `baseUrl` when one is given. */
+    links: string[];
+}
+/** Options for `htmlToText()`. */
+export interface HtmlToTextOptions {
+    /**
+     * Keeps only `<main>`, or else `<article>`, when the page has one, dropping navigation, headers,
+     * and footers. On by default.
+     */
+    mainContent?: boolean;
+    /** Resolves relative links. */
+    baseUrl?: string | URL;
+}
+/**
+ * Turns HTML into readable text without a DOM: scripts, styles, and the head are dropped, block
+ * elements become line breaks, headings become Markdown headings so `splitOnMarkdownHeadings` splits
+ * a page at its sections, and entities are decoded. A heuristic, not a browser: it is built for pages
+ * whose content is in the HTML, not rendered by script.
+ */
+export declare function htmlToText(html: string, options?: HtmlToTextOptions): HtmlText;
+/** Options for `loadHtml()`. */
+export interface HtmlLoaderOptions extends FileLoaderOptions, HtmlToTextOptions {
+}
+/** HTML files, one document each, with the page title and description in the metadata. */
+export declare function loadHtml(inputs: FileInput | Iterable<FileInput>, options?: HtmlLoaderOptions): AsyncGenerator<DocumentSource>;

package/dist/loaders/html.js

@@ -0,0 +1,96 @@
+import { inputsOf, readText } from './input.js';
+const DROPPED = /<(script|style|noscript|template|svg|iframe|canvas|head)\b[^>]*>[\s\S]*?<\/\1\s*>/gi;
+const BLOCK = /<\/?(p|div|section|article|main|header|footer|nav|aside|blockquote|pre|table|thead|tbody|tfoot|tr|ul|ol|dl|dt|dd|figure|figcaption|form|fieldset|address|hr)\b[^>]*>/gi;
+const ENTITIES = {
+    amp: '&',
+    lt: '<',
+    gt: '>',
+    quot: '"',
+    apos: "'",
+    nbsp: ' ',
+    mdash: '—',
+    ndash: '–',
+    hellip: '…',
+    copy: '©',
+    reg: '®',
+    trade: '™',
+    lsquo: '‘',
+    rsquo: '’',
+    ldquo: '“',
+    rdquo: '”',
+    laquo: '«',
+    raquo: '»',
+    middot: '·',
+    bull: '•',
+    euro: '€',
+    pound: '£',
+    deg: '°',
+};
+export function htmlToText(html, options = {}) {
+    const title = /<title\b[^>]*>([\s\S]*?)<\/title\s*>/i.exec(html)?.[1];
+    const description = /<meta\b[^>]*name\s*=\s*["']description["'][^>]*content\s*=\s*["']([^"']*)["']/i.exec(html)?.[1] ??
+        /<meta\b[^>]*content\s*=\s*["']([^"']*)["'][^>]*name\s*=\s*["']description["']/i.exec(html)?.[1];
+    let body = html.replace(/<!--[\s\S]*?-->/g, '').replace(DROPPED, '');
+    if (options.mainContent ?? true) {
+        const region = /<main\b[^>]*>([\s\S]*?)<\/main\s*>/i.exec(body) ?? /<article\b[^>]*>([\s\S]*?)<\/article\s*>/i.exec(body);
+        if (region)
+            body = region[1];
+    }
+    const links = [];
+    for (const match of body.matchAll(/<a\b[^>]*href\s*=\s*["']([^"'#][^"']*)["']/gi)) {
+        const href = decodeEntities(match[1]);
+        try {
+            links.push(options.baseUrl ? new URL(href, options.baseUrl).href : href);
+        }
+        catch {
+        }
+    }
+    const text = body
+        .replace(/<h([1-6])\b[^>]*>([\s\S]*?)<\/h\1\s*>/gi, (_, level, inner) => {
+        return `\n\n${'#'.repeat(Number(level))} ${inner.replace(/<[^>]+>/g, '').trim()}\n\n`;
+    })
+        .replace(/<li\b[^>]*>/gi, '\n- ')
+        .replace(/<br\s*\/?>/gi, '\n')
+        .replace(/<\/(td|th)\s*>/gi, '\t')
+        .replace(BLOCK, '\n')
+        .replace(/<[^>]+>/g, '');
+    return {
+        text: tidy(decodeEntities(text)),
+        ...(title ? { title: tidy(decodeEntities(title)) } : {}),
+        ...(description ? { description: decodeEntities(description).trim() } : {}),
+        links: [...new Set(links)],
+    };
+}
+export async function* loadHtml(inputs, options = {}) {
+    for (const input of inputsOf(inputs)) {
+        const { source, text } = await readText(input);
+        const page = htmlToText(text, options);
+        yield {
+            id: source,
+            text: page.text,
+            source,
+            metadata: {
+                ...options.metadata,
+                ...(page.title ? { title: page.title } : {}),
+                ...(page.description ? { description: page.description } : {}),
+            },
+        };
+    }
+}
+function decodeEntities(text) {
+    return text.replace(/&(#x[0-9a-f]+|#\d+|[a-z]+);/gi, (entity, name) => {
+        if (name[0] === '#') {
+            const code = name[1] === 'x' || name[1] === 'X' ? Number.parseInt(name.slice(2), 16) : Number(name.slice(1));
+            return Number.isFinite(code) && code > 0 && code <= 0x10ffff ? String.fromCodePoint(code) : entity;
+        }
+        return ENTITIES[name.toLowerCase()] ?? entity;
+    });
+}
+function tidy(text) {
+    return text
+        .split('\n')
+        .map((line) => line.replace(/[ \t\f\v ]+/g, ' ').trim())
+        .join('\n')
+        .replace(/\n{3,}/g, '\n\n')
+        .trim();
+}

package/dist/loaders/index.d.ts

@@ -0,0 +1,52 @@
+import type { RagChunk } from '../hallucination/rag.js';
+import { type DocumentSource, type IngestionOptions } from '../rag/ingestion.js';
+export type { DocumentSource } from '../rag/ingestion.js';
+/**
+ * Documents produced one at a time. Every loader returns one, so a corpus larger than memory streams
+ * through ingestion instead of being read whole, and an array of documents works as well.
+ */
+export type DocumentLoader = AsyncIterable<DocumentSource> | Iterable<DocumentSource>;
+/**
+ * A file to load: a path, a `file:` URL, or content already in memory with a name for it. In-memory
+ * content is how a loader reads an upload, a database blob, or a file on a runtime without a disk.
+ */
+export type FileInput = string | URL | {
+    source: string;
+    content: string | Uint8Array;
+};
+/** Reads every document from one or more loaders, in order, into an array. */
+export declare function collectDocuments(...loaders: DocumentLoader[]): Promise<DocumentSource[]>;
+/**
+ * Where loaded chunks go: a vector store, a `KeywordIndex`, or anything else with an `add()` for
+ * chunks. Loading into several at once fills a vector store and the keyword index hybrid search reads
+ * in one pass over the corpus.
+ */
+export interface ChunkSink {
+    /** Adds chunks, replacing those whose id is already held. */
+    add(chunks: RagChunk[]): unknown;
+}
+/** Options for `loadIntoStore()`: how documents are split, and how many are embedded at once. */
+export interface LoadIntoStoreOptions extends IngestionOptions {
+    /** Documents split and added to the store together. Defaults to 32. */
+    batchSize?: number;
+    /** Stops loading between batches. */
+    signal?: AbortSignal;
+    /** Called after each batch is stored, for progress reporting. */
+    onBatch?: (progress: LoadIntoStoreResult) => void;
+}
+/** What `loadIntoStore()` stored. */
+export interface LoadIntoStoreResult {
+    /** Documents read from the loaders. */
+    documents: number;
+    /** Chunks added to the store. */
+    chunks: number;
+    /** Characters across every document. */
+    totalCharacters: number;
+}
+/**
+ * Streams documents from loaders into one or more stores: each batch is split into chunks and added to
+ * every store at once, so memory holds one batch at a time however large the corpus is. Chunk ids
+ * derive from document ids, so loading the same corpus again replaces its chunks rather than
+ * duplicating them. A document with no id is numbered in load order.
+ */
+export declare function loadIntoStore(stores: ChunkSink | readonly ChunkSink[], loaders: DocumentLoader | readonly DocumentLoader[], options?: LoadIntoStoreOptions): Promise<LoadIntoStoreResult>;

package/dist/loaders/index.js

@@ -0,0 +1,46 @@
+import { ingestDocuments } from '../rag/ingestion.js';
+export async function collectDocuments(...loaders) {
+    const documents = [];
+    for (const loader of loaders) {
+        for await (const document of loader)
+            documents.push(document);
+    }
+    return documents;
+}
+export async function loadIntoStore(stores, loaders, options = {}) {
+    const batchSize = options.batchSize ?? 32;
+    if (!(Number.isInteger(batchSize) && batchSize > 0))
+        throw new RangeError('batchSize must be a positive integer');
+    const result = { documents: 0, chunks: 0, totalCharacters: 0 };
+    let batch = [];
+    const flush = async () => {
+        if (batch.length === 0)
+            return;
+        const ingested = ingestDocuments(batch, options);
+        const targets = 'add' in stores ? [stores] : stores;
+        await Promise.all(targets.map((store) => store.add(ingested.chunks)));
+        result.chunks += ingested.chunks.length;
+        result.totalCharacters += ingested.totalCharacters;
+        batch = [];
+        options.onBatch?.({ ...result });
+    };
+    const sources = isLoaderList(loaders) ? loaders : [loaders];
+    for (const loader of sources) {
+        for await (const document of loader) {
+            options.signal?.throwIfAborted();
+            result.documents++;
+            batch.push(document.id ? document : { ...document, id: `doc-${result.documents}` });
+            if (batch.length >= batchSize)
+                await flush();
+        }
+    }
+    options.signal?.throwIfAborted();
+    await flush();
+    return result;
+}
+function isLoaderList(value) {
+    if (!Array.isArray(value))
+        return false;
+    const first = value[0];
+    return first !== undefined && typeof first === 'object' && first !== null && !('text' in first);
+}