agentfootprint 8.8.0 → 8.10.0

package/AGENTS.md

@@ -152,6 +152,8 @@ agent.rag(docs);
 await agent.run({ message: 'How long do refunds take?' });
 ```
 
+**Building the index** (8.10.0) is `agentfootprint/rag`: `indexFolder('./docs', { to: store, embedder })`, or the `loadDocuments` → `splitDocuments` → `indexCorpus` pieces, or `npx agentfootprint-index ./docs --to ./corpus.db`. Loaders for text/Markdown/HTML (zero-dep) and PDF (lazy `unpdf`, per-page text so citations can name a page). Re-running embeds only what changed. `defineRAG` stays on the MAIN barrel — it is run-time wiring; that door is index time.
+
 `defineRAG` runs on `defineMemory({ type: SEMANTIC, strategy: TOP_K })`. Same machinery, three deliberate differences: a corpus is **read-only** (it never stores the conversation), it reads under its **own namespace** rather than the run's identity, and its chunks render as **citable `<source>` blocks**. For conversation memory alongside a corpus, register both — `.rag(defineRAG(...))` and `.memory(defineMemory(...))`, each with its own store.
 
 **Why did the agent read this passage?** `agentfootprint.memory.retrieved` carries every candidate with its score — including the ones that were rejected and why. `agentfootprint.memory.attached` fires per chunk that reached the prompt. `agentfootprint.context.injected` reports `source: 'rag'` with that chunk's `retrievalScore` / `rankPosition` / `threshold`. The whole record is on root state as `retrievalEvidence_<id>`, where a backward slice can reach it.

package/CLAUDE.md

@@ -13,9 +13,10 @@ Entry points (package.json exports): `.` core API · `/observe` ALL observabilit
 | core/slots/ | the 3 context-slot subflow builders + thinking subflow — intentionally NOT exported |
 | core-flow/ | Sequence/Parallel/Conditional/Loop — RunnerBase subclasses with own charts |
 | patterns/ | Debate/MapReduce/Reflection/SelfConsistency/Swarm/ToT — pure composition of runners, no new control flow |
-| adapters/ | hexagonal ports (types.ts = ALL port interfaces) + vendor impls (llm/, memory/, identity/, observability/) |
+| adapters/ | hexagonal ports (types.ts = ALL port interfaces) + vendor impls (llm/, memory/, identity/, observability/). memory/sqliteVector.ts (8.9.0) = the only FULL MemoryStore we ship with `search` besides InMemoryStore — exact cosine over a resident Float32Array matrix, hydrated per namespace on first search and dropped on any write to it |
 | recorders/core/ | bridges footprintjs events → typed EventDispatcher (ContextRecorder, EmitBridge, typedEmit) — auto-attached by Agent.createExecutor; most factories also exported via `/observe` for manual wiring (EmitBridge itself stays internal) |
 | recorders/observability/ | consumer recorders over the typed stream (RunStepRecorder, FlowchartRecorder, Status, Trace replay) + `recordRun` — THE producer of a recording `{snapshot, events, structure}` (the shape lens's `observeRecording` consumes; `structure` = `getSpec().buildTimeStructure`, which no snapshot carries). Anything that saves a run goes through it |
+| rag/ | (8.10.0, door `/rag`) index-TIME: `DocumentLoader` adapters (text/markdown/html zero-dep, pdf via lazy `unpdf`) + `Splitter` factories + `indexCorpus` — a REAL footprintjs chart whose commit log IS the indexing report. `defineRAG` deliberately stays on the MAIN barrel (run-time wiring); this door is the half that runs once, before any agent exists |
 | lib/ | first-party sub-libraries: injection-engine/, context-bisect/ (localizeContextBug, toBacktrackTrace + sliceToBacktrackTrace — the atui board serializers), influence-core/, trace-toolpack/ (selfExplain; 6 tools incl. variable-first `backtrack(variable, element?)`), context-ledger/ (which pieces EARNED their tokens — post-run offers/uses/outcomes bookkeeping + demote-never-starve gates `ledgerToolGate`/`ledgerEntryScorer`/`ledgerGated`; grouped-mode folds sf-llm-call inner logs, unmeterable runs → undefined; /observe), mcp/, rag/, tool-lint/ |
 | memory/ | store/ (MemoryStore port) + pipeline presets + stages + beats/facts + causal/ (dev-only, TOP_K+search()-only) + wire/mountMemoryPipeline + retrieval/ (8.8.0: the `RetrievalStrategy` seam + `RetrievalEvidence`, the record a retrieval leaves — `topK()` is what every earlier release did unnamed) |
 | events/ | EventDispatcher (wildcard subs), registry (EVENT_NAMES, AgentfootprintEventMap), payloads |
@@ -39,6 +40,9 @@ Traps: `src/observability/` holds the finder IMPLEMENTATIONS (canonical home; `d
 - **New typed event (3-step)**: payload interface in events/payloads.ts + entry in `AgentfootprintEventMap` (registry.ts:198) + append to `ALL_EVENT_TYPES` (registry.ts:488, count-asserted by tests). New DOMAIN also needs a bridge attach in Agent.createExecutor or emits never reach the dispatcher — AND a hand-edit to `DomainWildcard` (dispatcher.ts:67-82; already missing validation/credential/reliability).
 - **Strategy (vendor sink)**: shapes in strategies/types.ts (Observability :130, Cost :169, LiveStatus :201, Lens :234); attach via `agent.enable.*` or `registerObservabilityStrategy` (strategies/registry.ts). New vendor = export from observability-providers.ts, NOT a new subpath.
 - **Memory store**: implement `MemoryStore` (memory/store/types.ts:113; `search?` REQUIRED for causal memory); pass to `defineMemory({store})`. Memory TYPE/STRATEGY unions are CLOSED (define.types.ts:57/74 — new one edits defineMemory dispatch + a pipeline builder).
+- **Document loader** (8.10.0): implement `DocumentLoader` (rag/types.ts) — `{name, extensions, load}`. `loadDocuments` routes by extension, caller-supplied loaders FIRST, so overriding a built-in is passing yours ahead of it rather than editing `DEFAULT_LOADERS`. A loader MUST NOT rewrite text after offsets are conceivable: the HTML stripper replaces tags with EQUAL-LENGTH whitespace for exactly this reason.
+- **Splitter** (8.10.0): implement `Splitter` (`{name, split(doc) → SplitPiece[]}`), a factory function like the window/retrieval families. THE invariant — `doc.text.slice(charStart, charEnd) === piece.text` — is VERIFIED by `splitDocuments`, not trusted. All offset arithmetic lives once in `splitters/shared.ts`; a strategy that does its own is how the invariant breaks.
+- **Durable store** (8.9.0): `sqliteVectorStore` follows `hosting/sqliteSessions` line for line — lazy `node:sqlite`, WAL read-back on `journalMode`, STRICT tables, schema-identity + schema-version refusals, `':memory:'` refused. TWO things it adds that have no precedent there: `putMany`/`putIfVersion`/`forget` wrap in a transaction (sqliteSessions has none), and the EMBEDDER FINGERPRINT (`'<id>@<dims>'`, one per namespace in `af_index_meta`) is refused at write AND query. `SqliteUnavailableError` is now ONE class in `lib/sqliteUnavailable.ts` re-exported by both doors — a second class of that name is a duplicate type the build refuses.
 - **Retrieval rule** (8.8.0): implement `RetrievalStrategy` (memory/retrieval/types.ts) — `select(pool) → verdict[]`, one verdict per candidate, order preserved; it never touches the store and never embeds. Pass as `defineRAG({retrieval})`. `topK()` is the only shipped one; rerank/MMR are named-but-deferred adapters behind the same interface. `TopKStrategy` is a UNION whose arms exclude (`{topK,threshold}` vs `{retrieval}`) — refused in the type AND at runtime, because two spellings of one rule can disagree.
 - **Injection/skill**: `Injection = {id, flavor, trigger, inject}` (lib/injection-engine/types.ts:161); trigger is a closed 4-variant union (:30 — new kind edits evaluator.ts:40-74 switch). Factories defineSkill etc.; skill graph via `skillGraph()` (skillGraph.ts:392) with pluggable `EntryScorer` (entryScorer.ts:64). `SkillGraphConfig` is a UNION (flat arm `start`/`steps` vs tree arm) — the contradictions it encodes are ALSO refused at build (`.tree()` + `.entry()`/`.route()`, a non-leaf in `skills[]` under a tree, two skills claiming one id, a second `.skillGraph()` on one agent).
 - **Ports table** (wired via AgentOptions): PermissionChecker (adapters/types.ts:403), PricingTable (:412), CredentialProvider (identity/types.ts:89), CacheStrategy (cache/types.ts:151, registerCacheStrategy), ThinkingHandler (thinking/types.ts:114 — auto-wire scans HARDCODED SHIPPED_THINKING_HANDLERS, registry.ts:26), ReliabilityConfig (reliability/types.ts:183), OutputSchemaParser (core/outputSchema.ts:62, duck-typed).
@@ -50,6 +54,8 @@ Traps: `src/observability/` holds the finder IMPLEMENTATIONS (canonical home; `d
 ## Change-impact map
 - **conventions.ts** (STAGE_IDS/SUBFLOW_IDS/INJECTION_KEYS) → chart builders that mount by id, ContextRecorder slot attribution, localizer loop-head detection (lib/context-bisect/trajectory.ts:17-33), `stageRole`/`milestoneFor` (Lens contract), BoundaryRecorder. Renaming an id is the whole blast radius.
 - **BoundaryRecorder wiring is THREE connections, all at record time**: `runner.attach` (boundaries), `.subscribe(runner)` (what's inside them), `{getCommitCount}` (where each sits on the commit axis). The third fails SILENTLY — every event stamps `commitIdxBefore: 0`, `boundaryIndex` stays empty by design, and an offline step strip has nothing to place. Unrecoverable after the run (the commit log never records WHEN a boundary was crossed). Wired by `attachFlowchart` (which `enable.flowchart`/`enable.localObservability` both go through) and by `recordRun`; a new entry point must pass all three.
+- **indexCorpus fan-out** (8.10.0) → `maxBranches` on `addParallelForEach` TRUNCATES surplus items rather than queueing them, so the chart fans out over a WINDOW (`take-window` → `embed` → `tally-window` → `more-batches-decider` `{loopTo: 'take-window'}`) that can never exceed the ceiling. A single fan-out over all batches would silently index only the first `maxConcurrentBatches` — pinned by the 12-batches-through-a-window-of-2 test. `embedded` is summed from each branch's `written`, never from the plan's queue, and the fan-out is `failFast: true` because a half-indexed corpus keeps answering.
+- **Embedder fingerprint** (8.9.0) → `Embedder.id` (optional; every shipped embedder sets one, and NONE include dims — the store appends `@<dims>` itself, so an id carrying its own size double-stamps) + `indexDocuments` defaulting `embedderId` to it + `SqliteVectorStore.reconcileFingerprint` (the only comparison site). Rule: dimensions ALWAYS decide, model ids decide only when BOTH sides named themselves — refusing on an absent name would block the majority of callers who never pass `embedderId`.
 - **Retrieval record** (8.8.0) → FOUR stages write one object in sequence: `loadRelevant` (candidates+scores+threshold verdicts) → `pickByBudget` (re-marks admitted→over-budget/over-max-entries) → `formatDefault` (`promptFragment` + `promptPosition`) → the read mount's outputMapper lifts it to root as `retrievalEvidence_<id>`. `memoryRecallInjections` then splits ONE recall into one ActiveInjection PER CHUNK — guarded by a byte-equality check (`fragments.join('\n\n') === systemContent`) that falls back to the single injection rather than change the prompt. `rank` (score order) and `promptPosition` (picker order) are DIFFERENT and both load-bearing: joining fragments in rank order reproduces the right bytes in a sequence the model never saw.
 - **AgentState** → all 8 stages/ files, both builders' mappers, memory-wire STRING-TYPED keys ('runIdentity'/'turnNumber'/… buildAgentChart.ts:177-180 — not refactor-safe), finalizeResult's `reliabilityFail*`/`policyHalt*` reads (rename silently kills the typed errors).
 - **events/** → 72 typed events across 20 domains (counts anti-drift-tested against this file — update BOTH when adding events): ALL_EVENT_TYPES exhaustiveness tests, DomainWildcard hand-list, ~42 importers (recorders, strategies, stream, commentary).

package/bin/agentfootprint-index.mjs

@@ -0,0 +1,192 @@
+#!/usr/bin/env node
+/**
+ * agentfootprint-index — build a corpus index from the command line.
+ *
+ * The third bin in this package, after `agentfootprint-setup` and
+ * `agentfootprint-lint-tools`. It exists because indexing is a BOOT-TIME job:
+ * a cron entry, a deploy step, a thing you run once after adding documents.
+ * Making that a script every consumer writes for themselves — argument
+ * parsing, embedder selection, a progress line — is the kind of small friction
+ * that gets in the way of the first ten minutes.
+ *
+ *   npx agentfootprint-index ./docs --to ./corpus.db
+ *   npx agentfootprint-index ./docs --to ./corpus.db --embedder local --split heading
+ *   npx agentfootprint-index ./docs --to ./corpus.db --dry-run
+ *
+ * It prints the same `IndexReport` the API returns, because the report IS the
+ * output: a second run over unchanged documents prints `embedded 0`, which is
+ * the whole point of an incremental index and the fastest way to see it work.
+ */
+
+import { parseArgs } from 'node:util';
+import { existsSync, statSync } from 'node:fs';
+import { resolve } from 'node:path';
+
+const USAGE = `
+agentfootprint-index — build a corpus index
+
+USAGE
+  agentfootprint-index <dir> --to <file.db> [options]
+
+REQUIRED
+  <dir>                  Directory of documents to index
+  --to <file.db>         SQLite index file (created if missing)
+
+OPTIONS
+  --embedder <name>      static (default) | local | openai | mock
+                         static: bundled weights, no key, no network
+                         local:  on-device sentence-transformer, no key
+                         openai: hosted, needs OPENAI_API_KEY
+                         mock:   letter frequency — plumbing only, NOT semantic
+  --split <name>         heading (default) | paragraph | fixed | whole
+  --chars <n>            Target chunk size. Default 1000
+  --overlap <n>          Overlap between chunks. Default 150
+  --include <.ext,...>   Extensions to index. Default: every format we read
+  --corpus <name>        Namespace to index into. Default '_global'
+  --no-recursive         Do not descend into subdirectories
+  --no-remove            Keep chunks whose document disappeared
+  --dry-run              Report what WOULD happen; write nothing
+  --json                 Print the report as JSON
+  -h, --help             This
+
+EXAMPLES
+  agentfootprint-index ./docs --to ./corpus.db
+  agentfootprint-index ./docs --to ./corpus.db --embedder local --chars 800
+  agentfootprint-index ./docs --to ./corpus.db --dry-run --json
+`;
+
+function fail(message) {
+  process.stderr.write(`agentfootprint-index: ${message}\n`);
+  process.stderr.write('Run with --help for usage.\n');
+  process.exit(1);
+}
+
+const { values, positionals } = (() => {
+  try {
+    return parseArgs({
+      allowPositionals: true,
+      options: {
+        to: { type: 'string' },
+        embedder: { type: 'string' },
+        split: { type: 'string' },
+        chars: { type: 'string' },
+        overlap: { type: 'string' },
+        include: { type: 'string' },
+        corpus: { type: 'string' },
+        recursive: { type: 'boolean', default: true },
+        remove: { type: 'boolean', default: true },
+        'dry-run': { type: 'boolean', default: false },
+        json: { type: 'boolean', default: false },
+        help: { type: 'boolean', short: 'h', default: false },
+      },
+    });
+  } catch (err) {
+    fail(err instanceof Error ? err.message : String(err));
+  }
+})();
+
+if (values.help || positionals.length === 0) {
+  process.stdout.write(USAGE);
+  process.exit(values.help ? 0 : 1);
+}
+
+const dir = resolve(positionals[0]);
+if (!existsSync(dir) || !statSync(dir).isDirectory()) {
+  fail(`'${positionals[0]}' is not a directory.`);
+}
+if (!values.to && !values['dry-run']) {
+  fail('--to <file.db> is required (or use --dry-run to report without writing).');
+}
+
+// Imported through the package's own entry points, so the CLI exercises the
+// same doors a consumer does — a break in the export map fails here too.
+const { indexCorpus } = await import('../dist/esm/doors/rag.js');
+const { byHeading, byParagraph, fixedWithOverlap, wholeDocument } = await import(
+  '../dist/esm/doors/rag.js'
+);
+const { sqliteVectorStore, InMemoryStore, mockEmbedder } = await import(
+  '../dist/esm/doors/memory.js'
+);
+const providers = await import('../dist/esm/doors/providers.js');
+
+const chars = values.chars === undefined ? undefined : Number(values.chars);
+const overlap = values.overlap === undefined ? undefined : Number(values.overlap);
+if (chars !== undefined && !Number.isFinite(chars)) fail('--chars must be a number.');
+if (overlap !== undefined && !Number.isFinite(overlap)) fail('--overlap must be a number.');
+
+const splitters = {
+  heading: () => byHeading({ ...(chars && { maxChars: chars }), ...(overlap !== undefined && { overlapChars: overlap }) }),
+  paragraph: () => byParagraph({ ...(chars && { maxChars: chars }), ...(overlap !== undefined && { overlapChars: overlap }) }),
+  fixed: () => fixedWithOverlap({ ...(chars && { chars }), ...(overlap !== undefined && { overlapChars: overlap }) }),
+  whole: () => wholeDocument(),
+};
+const splitName = values.split ?? 'heading';
+if (!(splitName in splitters)) {
+  fail(`unknown --split '${splitName}'. One of: ${Object.keys(splitters).join(', ')}.`);
+}
+
+const embedders = {
+  static: () => providers.staticEmbedder(),
+  local: () => providers.localEmbedder(),
+  openai: () => providers.openaiEmbedder(),
+  mock: () => mockEmbedder(),
+};
+const embedderName = values.embedder ?? 'static';
+if (!(embedderName in embedders)) {
+  fail(`unknown --embedder '${embedderName}'. One of: ${Object.keys(embedders).join(', ')}.`);
+}
+
+let embedder;
+try {
+  embedder = embedders[embedderName]();
+} catch (err) {
+  fail(err instanceof Error ? err.message : String(err));
+}
+
+// --dry-run writes to a throwaway in-memory index, so it reports real chunk
+// counts and real splitting without touching the file. It still EMBEDS —
+// the alternative is reporting a count that the real run might not match.
+const store = values['dry-run'] ? new InMemoryStore() : sqliteVectorStore({ file: resolve(values.to) });
+
+try {
+  const report = await indexCorpus({
+    source: {
+      dir,
+      recursive: values.recursive,
+      ...(values.include && { include: values.include.split(',').map((e) => e.trim()) }),
+    },
+    store,
+    embedder,
+    splitter: splitters[splitName](),
+    removeMissing: values.remove,
+    ...(values.corpus && { corpus: { conversationId: values.corpus } }),
+  });
+
+  if (values.json) {
+    process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
+  } else {
+    const parts = [
+      `discovered ${report.discovered}`,
+      `loaded ${report.loaded}`,
+      `chunks ${report.chunks}`,
+      `embedded ${report.embedded}`,
+      `skipped ${report.skipped}`,
+      `removed ${report.removed}`,
+    ];
+    process.stdout.write(
+      `${values['dry-run'] ? '[dry run] ' : ''}${parts.join(' · ')}   (${report.elapsedMs}ms, ${report.splitter}, ${report.embedderFingerprint})\n`,
+    );
+    for (const failure of report.failed) {
+      process.stdout.write(`  failed: ${failure.uri} — ${failure.reason}\n`);
+    }
+    if (report.truncated.length > 0) {
+      process.stdout.write(
+        `  ${report.truncated.length} chunk(s) longer than the embedder reads — they were clipped. Lower --chars.\n`,
+      );
+    }
+  }
+  if (typeof store.close === 'function') store.close();
+} catch (err) {
+  process.stderr.write(`agentfootprint-index: ${err instanceof Error ? err.message : String(err)}\n`);
+  process.exit(1);
+}