agentfootprint 8.9.0 → 8.10.0

package/AGENTS.md

@@ -152,6 +152,8 @@ agent.rag(docs);
 await agent.run({ message: 'How long do refunds take?' });
 ```
 
+**Building the index** (8.10.0) is `agentfootprint/rag`: `indexFolder('./docs', { to: store, embedder })`, or the `loadDocuments` → `splitDocuments` → `indexCorpus` pieces, or `npx agentfootprint-index ./docs --to ./corpus.db`. Loaders for text/Markdown/HTML (zero-dep) and PDF (lazy `unpdf`, per-page text so citations can name a page). Re-running embeds only what changed. `defineRAG` stays on the MAIN barrel — it is run-time wiring; that door is index time.
+
 `defineRAG` runs on `defineMemory({ type: SEMANTIC, strategy: TOP_K })`. Same machinery, three deliberate differences: a corpus is **read-only** (it never stores the conversation), it reads under its **own namespace** rather than the run's identity, and its chunks render as **citable `<source>` blocks**. For conversation memory alongside a corpus, register both — `.rag(defineRAG(...))` and `.memory(defineMemory(...))`, each with its own store.
 
 **Why did the agent read this passage?** `agentfootprint.memory.retrieved` carries every candidate with its score — including the ones that were rejected and why. `agentfootprint.memory.attached` fires per chunk that reached the prompt. `agentfootprint.context.injected` reports `source: 'rag'` with that chunk's `retrievalScore` / `rankPosition` / `threshold`. The whole record is on root state as `retrievalEvidence_<id>`, where a backward slice can reach it.

package/CLAUDE.md

@@ -16,6 +16,7 @@ Entry points (package.json exports): `.` core API · `/observe` ALL observabilit
 | adapters/ | hexagonal ports (types.ts = ALL port interfaces) + vendor impls (llm/, memory/, identity/, observability/). memory/sqliteVector.ts (8.9.0) = the only FULL MemoryStore we ship with `search` besides InMemoryStore — exact cosine over a resident Float32Array matrix, hydrated per namespace on first search and dropped on any write to it |
 | recorders/core/ | bridges footprintjs events → typed EventDispatcher (ContextRecorder, EmitBridge, typedEmit) — auto-attached by Agent.createExecutor; most factories also exported via `/observe` for manual wiring (EmitBridge itself stays internal) |
 | recorders/observability/ | consumer recorders over the typed stream (RunStepRecorder, FlowchartRecorder, Status, Trace replay) + `recordRun` — THE producer of a recording `{snapshot, events, structure}` (the shape lens's `observeRecording` consumes; `structure` = `getSpec().buildTimeStructure`, which no snapshot carries). Anything that saves a run goes through it |
+| rag/ | (8.10.0, door `/rag`) index-TIME: `DocumentLoader` adapters (text/markdown/html zero-dep, pdf via lazy `unpdf`) + `Splitter` factories + `indexCorpus` — a REAL footprintjs chart whose commit log IS the indexing report. `defineRAG` deliberately stays on the MAIN barrel (run-time wiring); this door is the half that runs once, before any agent exists |
 | lib/ | first-party sub-libraries: injection-engine/, context-bisect/ (localizeContextBug, toBacktrackTrace + sliceToBacktrackTrace — the atui board serializers), influence-core/, trace-toolpack/ (selfExplain; 6 tools incl. variable-first `backtrack(variable, element?)`), context-ledger/ (which pieces EARNED their tokens — post-run offers/uses/outcomes bookkeeping + demote-never-starve gates `ledgerToolGate`/`ledgerEntryScorer`/`ledgerGated`; grouped-mode folds sf-llm-call inner logs, unmeterable runs → undefined; /observe), mcp/, rag/, tool-lint/ |
 | memory/ | store/ (MemoryStore port) + pipeline presets + stages + beats/facts + causal/ (dev-only, TOP_K+search()-only) + wire/mountMemoryPipeline + retrieval/ (8.8.0: the `RetrievalStrategy` seam + `RetrievalEvidence`, the record a retrieval leaves — `topK()` is what every earlier release did unnamed) |
 | events/ | EventDispatcher (wildcard subs), registry (EVENT_NAMES, AgentfootprintEventMap), payloads |
@@ -39,6 +40,8 @@ Traps: `src/observability/` holds the finder IMPLEMENTATIONS (canonical home; `d
 - **New typed event (3-step)**: payload interface in events/payloads.ts + entry in `AgentfootprintEventMap` (registry.ts:198) + append to `ALL_EVENT_TYPES` (registry.ts:488, count-asserted by tests). New DOMAIN also needs a bridge attach in Agent.createExecutor or emits never reach the dispatcher — AND a hand-edit to `DomainWildcard` (dispatcher.ts:67-82; already missing validation/credential/reliability).
 - **Strategy (vendor sink)**: shapes in strategies/types.ts (Observability :130, Cost :169, LiveStatus :201, Lens :234); attach via `agent.enable.*` or `registerObservabilityStrategy` (strategies/registry.ts). New vendor = export from observability-providers.ts, NOT a new subpath.
 - **Memory store**: implement `MemoryStore` (memory/store/types.ts:113; `search?` REQUIRED for causal memory); pass to `defineMemory({store})`. Memory TYPE/STRATEGY unions are CLOSED (define.types.ts:57/74 — new one edits defineMemory dispatch + a pipeline builder).
+- **Document loader** (8.10.0): implement `DocumentLoader` (rag/types.ts) — `{name, extensions, load}`. `loadDocuments` routes by extension, caller-supplied loaders FIRST, so overriding a built-in is passing yours ahead of it rather than editing `DEFAULT_LOADERS`. A loader MUST NOT rewrite text after offsets are conceivable: the HTML stripper replaces tags with EQUAL-LENGTH whitespace for exactly this reason.
+- **Splitter** (8.10.0): implement `Splitter` (`{name, split(doc) → SplitPiece[]}`), a factory function like the window/retrieval families. THE invariant — `doc.text.slice(charStart, charEnd) === piece.text` — is VERIFIED by `splitDocuments`, not trusted. All offset arithmetic lives once in `splitters/shared.ts`; a strategy that does its own is how the invariant breaks.
 - **Durable store** (8.9.0): `sqliteVectorStore` follows `hosting/sqliteSessions` line for line — lazy `node:sqlite`, WAL read-back on `journalMode`, STRICT tables, schema-identity + schema-version refusals, `':memory:'` refused. TWO things it adds that have no precedent there: `putMany`/`putIfVersion`/`forget` wrap in a transaction (sqliteSessions has none), and the EMBEDDER FINGERPRINT (`'<id>@<dims>'`, one per namespace in `af_index_meta`) is refused at write AND query. `SqliteUnavailableError` is now ONE class in `lib/sqliteUnavailable.ts` re-exported by both doors — a second class of that name is a duplicate type the build refuses.
 - **Retrieval rule** (8.8.0): implement `RetrievalStrategy` (memory/retrieval/types.ts) — `select(pool) → verdict[]`, one verdict per candidate, order preserved; it never touches the store and never embeds. Pass as `defineRAG({retrieval})`. `topK()` is the only shipped one; rerank/MMR are named-but-deferred adapters behind the same interface. `TopKStrategy` is a UNION whose arms exclude (`{topK,threshold}` vs `{retrieval}`) — refused in the type AND at runtime, because two spellings of one rule can disagree.
 - **Injection/skill**: `Injection = {id, flavor, trigger, inject}` (lib/injection-engine/types.ts:161); trigger is a closed 4-variant union (:30 — new kind edits evaluator.ts:40-74 switch). Factories defineSkill etc.; skill graph via `skillGraph()` (skillGraph.ts:392) with pluggable `EntryScorer` (entryScorer.ts:64). `SkillGraphConfig` is a UNION (flat arm `start`/`steps` vs tree arm) — the contradictions it encodes are ALSO refused at build (`.tree()` + `.entry()`/`.route()`, a non-leaf in `skills[]` under a tree, two skills claiming one id, a second `.skillGraph()` on one agent).
@@ -51,6 +54,7 @@ Traps: `src/observability/` holds the finder IMPLEMENTATIONS (canonical home; `d
 ## Change-impact map
 - **conventions.ts** (STAGE_IDS/SUBFLOW_IDS/INJECTION_KEYS) → chart builders that mount by id, ContextRecorder slot attribution, localizer loop-head detection (lib/context-bisect/trajectory.ts:17-33), `stageRole`/`milestoneFor` (Lens contract), BoundaryRecorder. Renaming an id is the whole blast radius.
 - **BoundaryRecorder wiring is THREE connections, all at record time**: `runner.attach` (boundaries), `.subscribe(runner)` (what's inside them), `{getCommitCount}` (where each sits on the commit axis). The third fails SILENTLY — every event stamps `commitIdxBefore: 0`, `boundaryIndex` stays empty by design, and an offline step strip has nothing to place. Unrecoverable after the run (the commit log never records WHEN a boundary was crossed). Wired by `attachFlowchart` (which `enable.flowchart`/`enable.localObservability` both go through) and by `recordRun`; a new entry point must pass all three.
+- **indexCorpus fan-out** (8.10.0) → `maxBranches` on `addParallelForEach` TRUNCATES surplus items rather than queueing them, so the chart fans out over a WINDOW (`take-window` → `embed` → `tally-window` → `more-batches-decider` `{loopTo: 'take-window'}`) that can never exceed the ceiling. A single fan-out over all batches would silently index only the first `maxConcurrentBatches` — pinned by the 12-batches-through-a-window-of-2 test. `embedded` is summed from each branch's `written`, never from the plan's queue, and the fan-out is `failFast: true` because a half-indexed corpus keeps answering.
 - **Embedder fingerprint** (8.9.0) → `Embedder.id` (optional; every shipped embedder sets one, and NONE include dims — the store appends `@<dims>` itself, so an id carrying its own size double-stamps) + `indexDocuments` defaulting `embedderId` to it + `SqliteVectorStore.reconcileFingerprint` (the only comparison site). Rule: dimensions ALWAYS decide, model ids decide only when BOTH sides named themselves — refusing on an absent name would block the majority of callers who never pass `embedderId`.
 - **Retrieval record** (8.8.0) → FOUR stages write one object in sequence: `loadRelevant` (candidates+scores+threshold verdicts) → `pickByBudget` (re-marks admitted→over-budget/over-max-entries) → `formatDefault` (`promptFragment` + `promptPosition`) → the read mount's outputMapper lifts it to root as `retrievalEvidence_<id>`. `memoryRecallInjections` then splits ONE recall into one ActiveInjection PER CHUNK — guarded by a byte-equality check (`fragments.join('\n\n') === systemContent`) that falls back to the single injection rather than change the prompt. `rank` (score order) and `promptPosition` (picker order) are DIFFERENT and both load-bearing: joining fragments in rank order reproduces the right bytes in a sequence the model never saw.
 - **AgentState** → all 8 stages/ files, both builders' mappers, memory-wire STRING-TYPED keys ('runIdentity'/'turnNumber'/… buildAgentChart.ts:177-180 — not refactor-safe), finalizeResult's `reliabilityFail*`/`policyHalt*` reads (rename silently kills the typed errors).

package/bin/agentfootprint-index.mjs

@@ -0,0 +1,192 @@
+#!/usr/bin/env node
+/**
+ * agentfootprint-index — build a corpus index from the command line.
+ *
+ * The third bin in this package, after `agentfootprint-setup` and
+ * `agentfootprint-lint-tools`. It exists because indexing is a BOOT-TIME job:
+ * a cron entry, a deploy step, a thing you run once after adding documents.
+ * Making that a script every consumer writes for themselves — argument
+ * parsing, embedder selection, a progress line — is the kind of small friction
+ * that gets in the way of the first ten minutes.
+ *
+ *   npx agentfootprint-index ./docs --to ./corpus.db
+ *   npx agentfootprint-index ./docs --to ./corpus.db --embedder local --split heading
+ *   npx agentfootprint-index ./docs --to ./corpus.db --dry-run
+ *
+ * It prints the same `IndexReport` the API returns, because the report IS the
+ * output: a second run over unchanged documents prints `embedded 0`, which is
+ * the whole point of an incremental index and the fastest way to see it work.
+ */
+
+import { parseArgs } from 'node:util';
+import { existsSync, statSync } from 'node:fs';
+import { resolve } from 'node:path';
+
+const USAGE = `
+agentfootprint-index — build a corpus index
+
+USAGE
+  agentfootprint-index <dir> --to <file.db> [options]
+
+REQUIRED
+  <dir>                  Directory of documents to index
+  --to <file.db>         SQLite index file (created if missing)
+
+OPTIONS
+  --embedder <name>      static (default) | local | openai | mock
+                         static: bundled weights, no key, no network
+                         local:  on-device sentence-transformer, no key
+                         openai: hosted, needs OPENAI_API_KEY
+                         mock:   letter frequency — plumbing only, NOT semantic
+  --split <name>         heading (default) | paragraph | fixed | whole
+  --chars <n>            Target chunk size. Default 1000
+  --overlap <n>          Overlap between chunks. Default 150
+  --include <.ext,...>   Extensions to index. Default: every format we read
+  --corpus <name>        Namespace to index into. Default '_global'
+  --no-recursive         Do not descend into subdirectories
+  --no-remove            Keep chunks whose document disappeared
+  --dry-run              Report what WOULD happen; write nothing
+  --json                 Print the report as JSON
+  -h, --help             This
+
+EXAMPLES
+  agentfootprint-index ./docs --to ./corpus.db
+  agentfootprint-index ./docs --to ./corpus.db --embedder local --chars 800
+  agentfootprint-index ./docs --to ./corpus.db --dry-run --json
+`;
+
+function fail(message) {
+  process.stderr.write(`agentfootprint-index: ${message}\n`);
+  process.stderr.write('Run with --help for usage.\n');
+  process.exit(1);
+}
+
+const { values, positionals } = (() => {
+  try {
+    return parseArgs({
+      allowPositionals: true,
+      options: {
+        to: { type: 'string' },
+        embedder: { type: 'string' },
+        split: { type: 'string' },
+        chars: { type: 'string' },
+        overlap: { type: 'string' },
+        include: { type: 'string' },
+        corpus: { type: 'string' },
+        recursive: { type: 'boolean', default: true },
+        remove: { type: 'boolean', default: true },
+        'dry-run': { type: 'boolean', default: false },
+        json: { type: 'boolean', default: false },
+        help: { type: 'boolean', short: 'h', default: false },
+      },
+    });
+  } catch (err) {
+    fail(err instanceof Error ? err.message : String(err));
+  }
+})();
+
+if (values.help || positionals.length === 0) {
+  process.stdout.write(USAGE);
+  process.exit(values.help ? 0 : 1);
+}
+
+const dir = resolve(positionals[0]);
+if (!existsSync(dir) || !statSync(dir).isDirectory()) {
+  fail(`'${positionals[0]}' is not a directory.`);
+}
+if (!values.to && !values['dry-run']) {
+  fail('--to <file.db> is required (or use --dry-run to report without writing).');
+}
+
+// Imported through the package's own entry points, so the CLI exercises the
+// same doors a consumer does — a break in the export map fails here too.
+const { indexCorpus } = await import('../dist/esm/doors/rag.js');
+const { byHeading, byParagraph, fixedWithOverlap, wholeDocument } = await import(
+  '../dist/esm/doors/rag.js'
+);
+const { sqliteVectorStore, InMemoryStore, mockEmbedder } = await import(
+  '../dist/esm/doors/memory.js'
+);
+const providers = await import('../dist/esm/doors/providers.js');
+
+const chars = values.chars === undefined ? undefined : Number(values.chars);
+const overlap = values.overlap === undefined ? undefined : Number(values.overlap);
+if (chars !== undefined && !Number.isFinite(chars)) fail('--chars must be a number.');
+if (overlap !== undefined && !Number.isFinite(overlap)) fail('--overlap must be a number.');
+
+const splitters = {
+  heading: () => byHeading({ ...(chars && { maxChars: chars }), ...(overlap !== undefined && { overlapChars: overlap }) }),
+  paragraph: () => byParagraph({ ...(chars && { maxChars: chars }), ...(overlap !== undefined && { overlapChars: overlap }) }),
+  fixed: () => fixedWithOverlap({ ...(chars && { chars }), ...(overlap !== undefined && { overlapChars: overlap }) }),
+  whole: () => wholeDocument(),
+};
+const splitName = values.split ?? 'heading';
+if (!(splitName in splitters)) {
+  fail(`unknown --split '${splitName}'. One of: ${Object.keys(splitters).join(', ')}.`);
+}
+
+const embedders = {
+  static: () => providers.staticEmbedder(),
+  local: () => providers.localEmbedder(),
+  openai: () => providers.openaiEmbedder(),
+  mock: () => mockEmbedder(),
+};
+const embedderName = values.embedder ?? 'static';
+if (!(embedderName in embedders)) {
+  fail(`unknown --embedder '${embedderName}'. One of: ${Object.keys(embedders).join(', ')}.`);
+}
+
+let embedder;
+try {
+  embedder = embedders[embedderName]();
+} catch (err) {
+  fail(err instanceof Error ? err.message : String(err));
+}
+
+// --dry-run writes to a throwaway in-memory index, so it reports real chunk
+// counts and real splitting without touching the file. It still EMBEDS —
+// the alternative is reporting a count that the real run might not match.
+const store = values['dry-run'] ? new InMemoryStore() : sqliteVectorStore({ file: resolve(values.to) });
+
+try {
+  const report = await indexCorpus({
+    source: {
+      dir,
+      recursive: values.recursive,
+      ...(values.include && { include: values.include.split(',').map((e) => e.trim()) }),
+    },
+    store,
+    embedder,
+    splitter: splitters[splitName](),
+    removeMissing: values.remove,
+    ...(values.corpus && { corpus: { conversationId: values.corpus } }),
+  });
+
+  if (values.json) {
+    process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
+  } else {
+    const parts = [
+      `discovered ${report.discovered}`,
+      `loaded ${report.loaded}`,
+      `chunks ${report.chunks}`,
+      `embedded ${report.embedded}`,
+      `skipped ${report.skipped}`,
+      `removed ${report.removed}`,
+    ];
+    process.stdout.write(
+      `${values['dry-run'] ? '[dry run] ' : ''}${parts.join(' · ')}   (${report.elapsedMs}ms, ${report.splitter}, ${report.embedderFingerprint})\n`,
+    );
+    for (const failure of report.failed) {
+      process.stdout.write(`  failed: ${failure.uri} — ${failure.reason}\n`);
+    }
+    if (report.truncated.length > 0) {
+      process.stdout.write(
+        `  ${report.truncated.length} chunk(s) longer than the embedder reads — they were clipped. Lower --chars.\n`,
+      );
+    }
+  }
+  if (typeof store.close === 'function') store.close();
+} catch (err) {
+  process.stderr.write(`agentfootprint-index: ${err instanceof Error ? err.message : String(err)}\n`);
+  process.exit(1);
+}

package/dist/doors/rag.js

@@ -0,0 +1,66 @@
+"use strict";
+/**
+ * agentfootprint/rag — turning a folder of documents into something an agent
+ * can retrieve from.
+ *
+ * Loaders (text, Markdown, HTML, PDF), splitters, and the indexing chart that
+ * runs them: everything between "I have some documents" and "the vectors are
+ * in a store".
+ *
+ * ── Why this is its own door ────────────────────────────────────────────────
+ * Index time is a different PROCESS from run time. It happens at boot, on a
+ * cron, or from a CLI; it touches the filesystem; and for PDFs it reaches for
+ * an optional peer dependency. None of that belongs in the bundle of an agent
+ * that only wants to answer a question — a browser or edge runtime importing
+ * `agentfootprint` should never resolve `node:fs`. A door is the unit at which
+ * a bundler can cut, and this is the cut.
+ *
+ * ── `defineRAG` is NOT here, deliberately ───────────────────────────────────
+ * The retriever stays on the MAIN barrel, with `indexDocuments`, because it is
+ * run-time wiring: it is registered on an agent, it runs on every turn, and it
+ * belongs beside `defineTool` and the rest of the wiring surface. This door is
+ * the other half of the story — the half that runs once, before any agent
+ * exists.
+ *
+ *   import { defineRAG, indexDocuments } from 'agentfootprint';        // wiring
+ *   import { sqliteVectorStore } from 'agentfootprint/memory';         // where it lives
+ *   import { indexFolder, byHeading } from 'agentfootprint/rag';       // building it
+ *
+ * ── Deferred, with destinations named ───────────────────────────────────────
+ * **Re-ranking and MMR** are retrieval-side concerns and belong behind
+ * `RetrievalStrategy` (`agentfootprint/memory`), not here — a re-ranker is a
+ * different rule for choosing among candidates, not a different way of
+ * building an index. **File watching** is deliberately absent: `indexCorpus`
+ * is an explicit call because magic that re-indexes behind you is magic that
+ * re-bills you. Run it at boot, from a cron, or from the CLI.
+ *
+ * @example
+ * ```ts
+ * import { indexFolder } from 'agentfootprint/rag';
+ * import { sqliteVectorStore } from 'agentfootprint/memory';
+ * import { staticEmbedder } from 'agentfootprint/providers';
+ *
+ * const report = await indexFolder('./docs', {
+ *   to: sqliteVectorStore({ file: './corpus.db' }),
+ *   embedder: staticEmbedder(),
+ * });
+ * // { discovered: 3, loaded: 3, chunks: 14, embedded: 14, skipped: 0, removed: 0, … }
+ * ```
+ */
+var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    var desc = Object.getOwnPropertyDescriptor(m, k);
+    if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
+      desc = { enumerable: true, get: function() { return m[k]; } };
+    }
+    Object.defineProperty(o, k2, desc);
+}) : (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    o[k2] = m[k];
+}));
+var __exportStar = (this && this.__exportStar) || function(m, exports) {
+    for (var p in m) if (p !== "default" && !Object.prototype.hasOwnProperty.call(exports, p)) __createBinding(exports, m, p);
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+__exportStar(require("../rag/index.js"), exports);
+//# sourceMappingURL=rag.js.map

package/dist/doors/rag.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"rag.js","sourceRoot":"","sources":["../../src/doors/rag.ts"],"names":[],"mappings":";AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA+CG;;;;;;;;;;;;;;;;AAEH,kDAAgC"}

package/dist/esm/doors/rag.d.ts

@@ -0,0 +1,49 @@
+/**
+ * agentfootprint/rag — turning a folder of documents into something an agent
+ * can retrieve from.
+ *
+ * Loaders (text, Markdown, HTML, PDF), splitters, and the indexing chart that
+ * runs them: everything between "I have some documents" and "the vectors are
+ * in a store".
+ *
+ * ── Why this is its own door ────────────────────────────────────────────────
+ * Index time is a different PROCESS from run time. It happens at boot, on a
+ * cron, or from a CLI; it touches the filesystem; and for PDFs it reaches for
+ * an optional peer dependency. None of that belongs in the bundle of an agent
+ * that only wants to answer a question — a browser or edge runtime importing
+ * `agentfootprint` should never resolve `node:fs`. A door is the unit at which
+ * a bundler can cut, and this is the cut.
+ *
+ * ── `defineRAG` is NOT here, deliberately ───────────────────────────────────
+ * The retriever stays on the MAIN barrel, with `indexDocuments`, because it is
+ * run-time wiring: it is registered on an agent, it runs on every turn, and it
+ * belongs beside `defineTool` and the rest of the wiring surface. This door is
+ * the other half of the story — the half that runs once, before any agent
+ * exists.
+ *
+ *   import { defineRAG, indexDocuments } from 'agentfootprint';        // wiring
+ *   import { sqliteVectorStore } from 'agentfootprint/memory';         // where it lives
+ *   import { indexFolder, byHeading } from 'agentfootprint/rag';       // building it
+ *
+ * ── Deferred, with destinations named ───────────────────────────────────────
+ * **Re-ranking and MMR** are retrieval-side concerns and belong behind
+ * `RetrievalStrategy` (`agentfootprint/memory`), not here — a re-ranker is a
+ * different rule for choosing among candidates, not a different way of
+ * building an index. **File watching** is deliberately absent: `indexCorpus`
+ * is an explicit call because magic that re-indexes behind you is magic that
+ * re-bills you. Run it at boot, from a cron, or from the CLI.
+ *
+ * @example
+ * ```ts
+ * import { indexFolder } from 'agentfootprint/rag';
+ * import { sqliteVectorStore } from 'agentfootprint/memory';
+ * import { staticEmbedder } from 'agentfootprint/providers';
+ *
+ * const report = await indexFolder('./docs', {
+ *   to: sqliteVectorStore({ file: './corpus.db' }),
+ *   embedder: staticEmbedder(),
+ * });
+ * // { discovered: 3, loaded: 3, chunks: 14, embedded: 14, skipped: 0, removed: 0, … }
+ * ```
+ */
+export * from '../rag/index.js';

package/dist/esm/doors/rag.js

@@ -0,0 +1,50 @@
+/**
+ * agentfootprint/rag — turning a folder of documents into something an agent
+ * can retrieve from.
+ *
+ * Loaders (text, Markdown, HTML, PDF), splitters, and the indexing chart that
+ * runs them: everything between "I have some documents" and "the vectors are
+ * in a store".
+ *
+ * ── Why this is its own door ────────────────────────────────────────────────
+ * Index time is a different PROCESS from run time. It happens at boot, on a
+ * cron, or from a CLI; it touches the filesystem; and for PDFs it reaches for
+ * an optional peer dependency. None of that belongs in the bundle of an agent
+ * that only wants to answer a question — a browser or edge runtime importing
+ * `agentfootprint` should never resolve `node:fs`. A door is the unit at which
+ * a bundler can cut, and this is the cut.
+ *
+ * ── `defineRAG` is NOT here, deliberately ───────────────────────────────────
+ * The retriever stays on the MAIN barrel, with `indexDocuments`, because it is
+ * run-time wiring: it is registered on an agent, it runs on every turn, and it
+ * belongs beside `defineTool` and the rest of the wiring surface. This door is
+ * the other half of the story — the half that runs once, before any agent
+ * exists.
+ *
+ *   import { defineRAG, indexDocuments } from 'agentfootprint';        // wiring
+ *   import { sqliteVectorStore } from 'agentfootprint/memory';         // where it lives
+ *   import { indexFolder, byHeading } from 'agentfootprint/rag';       // building it
+ *
+ * ── Deferred, with destinations named ───────────────────────────────────────
+ * **Re-ranking and MMR** are retrieval-side concerns and belong behind
+ * `RetrievalStrategy` (`agentfootprint/memory`), not here — a re-ranker is a
+ * different rule for choosing among candidates, not a different way of
+ * building an index. **File watching** is deliberately absent: `indexCorpus`
+ * is an explicit call because magic that re-indexes behind you is magic that
+ * re-bills you. Run it at boot, from a cron, or from the CLI.
+ *
+ * @example
+ * ```ts
+ * import { indexFolder } from 'agentfootprint/rag';
+ * import { sqliteVectorStore } from 'agentfootprint/memory';
+ * import { staticEmbedder } from 'agentfootprint/providers';
+ *
+ * const report = await indexFolder('./docs', {
+ *   to: sqliteVectorStore({ file: './corpus.db' }),
+ *   embedder: staticEmbedder(),
+ * });
+ * // { discovered: 3, loaded: 3, chunks: 14, embedded: 14, skipped: 0, removed: 0, … }
+ * ```
+ */
+export * from '../rag/index.js';
+//# sourceMappingURL=rag.js.map

package/dist/esm/doors/rag.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"rag.js","sourceRoot":"","sources":["../../../src/doors/rag.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA+CG;AAEH,cAAc,iBAAiB,CAAC"}

package/dist/esm/rag/hash.d.ts

@@ -0,0 +1,2 @@
+/** Hex sha-256 of the bytes or text given. */
+export declare function sha256(input: Uint8Array | string): string;

package/dist/esm/rag/hash.js

@@ -0,0 +1,22 @@
+/**
+ * sha256 — the content hash incremental re-indexing skips on.
+ *
+ * Pattern: leaf utility over `node:crypto`.
+ * Role:    rag/ layer. Distinct from `lib/fnv1a` on purpose: FNV-1a identifies
+ *          content for correlation inside one recording, where a collision
+ *          costs a confusing trace. This one decides whether a document is
+ *          RE-EMBEDDED, where a collision costs a stale answer that looks
+ *          exactly like a fresh one — and it is compared across runs, machines
+ *          and months rather than within a single process.
+ * Emits:   N/A.
+ */
+import { lazyRequire } from '../lib/lazyRequire.js';
+/** Hex sha-256 of the bytes or text given. */
+export function sha256(input) {
+    const crypto = lazyRequire('node:crypto');
+    return crypto
+        .createHash('sha256')
+        .update(typeof input === 'string' ? Buffer.from(input, 'utf8') : Buffer.from(input))
+        .digest('hex');
+}
+//# sourceMappingURL=hash.js.map

package/dist/esm/rag/hash.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"hash.js","sourceRoot":"","sources":["../../../src/rag/hash.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AACH,OAAO,EAAE,WAAW,EAAE,MAAM,uBAAuB,CAAC;AAEpD,8CAA8C;AAC9C,MAAM,UAAU,MAAM,CAAC,KAA0B;IAC/C,MAAM,MAAM,GAAG,WAAW,CAA+B,aAAa,CAAC,CAAC;IACxE,OAAO,MAAM;SACV,UAAU,CAAC,QAAQ,CAAC;SACpB,MAAM,CAAC,OAAO,KAAK,KAAK,QAAQ,CAAC,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,KAAK,EAAE,MAAM,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;SACnF,MAAM,CAAC,KAAK,CAAC,CAAC;AACnB,CAAC"}

package/dist/esm/rag/index.d.ts

@@ -0,0 +1,17 @@
+/**
+ * rag/ — a folder of documents becomes an answering agent.
+ *
+ * @see ./types.ts          the ports and the value objects
+ * @see ./loadDocuments.ts  step 1 — bytes to text
+ * @see ./splitDocuments.ts step 2 — text to chunks that know where they came from
+ * @see ./indexCorpus.ts    step 3 — the chart that embeds and stores them
+ */
+export { loadDocuments, type LoadDocumentsOptions, type LoadDocumentsResult, } from './loadDocuments.js';
+export { splitDocuments, type SplitDocumentsOptions } from './splitDocuments.js';
+export { indexCorpus, buildIndexChart, type IndexCorpusConfig } from './indexCorpus.js';
+export { indexFolder, type IndexFolderOptions } from './indexFolder.js';
+export { sha256 } from './hash.js';
+export { DEFAULT_LOADERS, textLoader, markdownLoader, htmlLoader, pdfLoader, mockLoader, MissingPdfSupportError, type PdfLoaderOptions, type MockLoaderOptions, } from './loaders/index.js';
+export { stripTags } from './loaders/html.js';
+export { byParagraph, byHeading, fixedWithOverlap, wholeDocument, DEFAULT_MAX_CHARS, DEFAULT_OVERLAP_CHARS, type ByParagraphOptions, type ByHeadingOptions, type FixedWithOverlapOptions, } from './splitters/index.js';
+export type { Chunk, DocumentInput, DocumentLoader, DocumentSource, FailedDocument, IndexReport, LoadedDocument, LoadedDocumentDraft, Splitter, SplitPiece, TruncatedChunk, } from './types.js';

package/dist/esm/rag/index.js

@@ -0,0 +1,17 @@
+/**
+ * rag/ — a folder of documents becomes an answering agent.
+ *
+ * @see ./types.ts          the ports and the value objects
+ * @see ./loadDocuments.ts  step 1 — bytes to text
+ * @see ./splitDocuments.ts step 2 — text to chunks that know where they came from
+ * @see ./indexCorpus.ts    step 3 — the chart that embeds and stores them
+ */
+export { loadDocuments, } from './loadDocuments.js';
+export { splitDocuments } from './splitDocuments.js';
+export { indexCorpus, buildIndexChart } from './indexCorpus.js';
+export { indexFolder } from './indexFolder.js';
+export { sha256 } from './hash.js';
+export { DEFAULT_LOADERS, textLoader, markdownLoader, htmlLoader, pdfLoader, mockLoader, MissingPdfSupportError, } from './loaders/index.js';
+export { stripTags } from './loaders/html.js';
+export { byParagraph, byHeading, fixedWithOverlap, wholeDocument, DEFAULT_MAX_CHARS, DEFAULT_OVERLAP_CHARS, } from './splitters/index.js';
+//# sourceMappingURL=index.js.map

package/dist/esm/rag/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../../../src/rag/index.ts"],"names":[],"mappings":"AAAA;;;;;;;GAOG;AACH,OAAO,EACL,aAAa,GAGd,MAAM,oBAAoB,CAAC;AAC5B,OAAO,EAAE,cAAc,EAA8B,MAAM,qBAAqB,CAAC;AACjF,OAAO,EAAE,WAAW,EAAE,eAAe,EAA0B,MAAM,kBAAkB,CAAC;AACxF,OAAO,EAAE,WAAW,EAA2B,MAAM,kBAAkB,CAAC;AACxE,OAAO,EAAE,MAAM,EAAE,MAAM,WAAW,CAAC;AAEnC,OAAO,EACL,eAAe,EACf,UAAU,EACV,cAAc,EACd,UAAU,EACV,SAAS,EACT,UAAU,EACV,sBAAsB,GAGvB,MAAM,oBAAoB,CAAC;AAC5B,OAAO,EAAE,SAAS,EAAE,MAAM,mBAAmB,CAAC;AAE9C,OAAO,EACL,WAAW,EACX,SAAS,EACT,gBAAgB,EAChB,aAAa,EACb,iBAAiB,EACjB,qBAAqB,GAItB,MAAM,sBAAsB,CAAC"}

package/dist/esm/rag/indexCorpus.d.ts

@@ -0,0 +1,102 @@
+import type { TypedScope } from 'footprintjs';
+import type { Embedder } from '../memory/embedding/index.js';
+import type { MemoryStore } from '../memory/store/index.js';
+import type { MemoryIdentity } from '../memory/identity/index.js';
+import type { Chunk, DocumentLoader, DocumentSource, FailedDocument, IndexReport, LoadedDocument, Splitter, TruncatedChunk } from './types.js';
+export interface IndexCorpusConfig {
+    /** Where the documents come from. */
+    readonly source: DocumentSource;
+    /** Where the vectors go. Any `MemoryStore`; `sqliteVectorStore` to keep them. */
+    readonly store: MemoryStore;
+    /** What turns chunk text into vectors. */
+    readonly embedder: Embedder;
+    /**
+     * The namespace to index into. Defaults to the same
+     * `{ conversationId: '_global' }` that `defineRAG` reads from, so index with
+     * no options and retrieve with no options and the two meet.
+     */
+    readonly corpus?: MemoryIdentity;
+    /**
+     * How to cut the documents. Default `byHeading()` — the one strategy that
+     * reads the document's own structure instead of inferring it.
+     */
+    readonly splitter?: Splitter;
+    /** Loaders consulted before the built-ins. */
+    readonly loaders?: readonly DocumentLoader[];
+    /**
+     * The embedder's id, stored per vector and refused on when it changes.
+     * Defaults to `embedder.id`.
+     */
+    readonly embedderId?: string;
+    /** Chunks per embed batch. Default 64. */
+    readonly batchSize?: number;
+    /**
+     * How many batches embed at once. Default 4.
+     *
+     * A hard ceiling on PARALLELISM, not on total work: batches beyond it wait
+     * for the next window rather than being dropped.
+     */
+    readonly maxConcurrentBatches?: number;
+    /** Attempts per batch, including the first. Default 3. */
+    readonly attempts?: number;
+    /**
+     * Delete chunks whose document is no longer in the source. Default true —
+     * an index that answers from a document you deleted is worse than one that
+     * does not answer.
+     */
+    readonly removeMissing?: boolean;
+    /**
+     * The embedder's input ceiling in characters. Chunks longer than this are
+     * embedded anyway (the embedder clips them) and RECORDED in
+     * `report.truncated` — so silent half-embedding becomes a number you can see.
+     * Default 2000, the measured `localEmbedder` cliff.
+     */
+    readonly maxChunkChars?: number;
+}
+/** The chart's scope. Every field here is in the commit log. */
+interface IndexState {
+    discoveredCount: number;
+    documents: readonly LoadedDocument[];
+    failed: readonly FailedDocument[];
+    chunks: readonly Chunk[];
+    toEmbed: readonly Chunk[];
+    toSkip: readonly string[];
+    toRemove: readonly string[];
+    /** The queue of batches still to embed. Drained one window at a time. */
+    pendingBatches: readonly (readonly Chunk[])[];
+    /** The window being fanned out over — never longer than `maxConcurrentBatches`. */
+    window: readonly (readonly Chunk[])[];
+    batchResults: readonly ({
+        written?: number;
+    } | undefined)[];
+    embeddedCount: number;
+    removedCount: number;
+    truncated: readonly TruncatedChunk[];
+    report?: IndexReport;
+    [key: string]: unknown;
+}
+/**
+ * Build and run the indexing chart.
+ *
+ * @returns the report — which is also committed to the run's own log, so it is
+ *   evidence rather than only a return value.
+ *
+ * @example
+ * ```ts
+ * const report = await indexCorpus({
+ *   source: { dir: './docs' },
+ *   store: sqliteVectorStore({ file: './corpus.db' }),
+ *   embedder: staticEmbedder(),
+ * });
+ * // { discovered: 3, loaded: 3, chunks: 14, embedded: 14, skipped: 0, removed: 0, … }
+ * ```
+ */
+export declare function indexCorpus(config: IndexCorpusConfig): Promise<IndexReport>;
+/**
+ * The chart itself, exposed so a caller can mount it, attach recorders to it,
+ * or run it under their own executor — the same freedom every other chart in
+ * this library gives.
+ */
+export declare function buildIndexChart(config: IndexCorpusConfig): ReturnType<typeof compile>;
+declare function compile(config: IndexCorpusConfig): import("footprintjs").RunnableFlowChart<any, TypedScope<IndexState>>;
+export {};