agentfootprint 8.7.0 → 8.9.0

package/AGENTS.md

@@ -137,16 +137,24 @@ const docs = defineRAG({
   id: 'product-docs',
   store, embedder,
   topK: 3,
-  threshold: 0.7,        // STRICT — no fallback when nothing matches
+  threshold: 0.7,        // STRICT — nothing is injected when nothing matches
 });
-// Retrieved chunks land in the SYSTEM-PROMPT slot, as one system message.
+// Retrieved chunks land in the SYSTEM-PROMPT slot, as one system message
+// of citable `<source id=… doc=… score=…>` blocks.
 // `asRole` was removed in 7.20.0 — it was never read, and passing it throws.
 
 // Wire to agent — `.rag()` is an alias for `.memory()`, same plumbing
 agent.rag(docs);
+
+// No identity argument anywhere: a corpus lives in its own namespace
+// (`corpus`, default `{ conversationId: '_global' }`) — the same one
+// indexDocuments writes to.
+await agent.run({ message: 'How long do refunds take?' });
 ```
 
-`defineRAG` is sugar over `defineMemory({ type: SEMANTIC, strategy: TOP_K })`. Same plumbing, different intent: RAG = document corpus retrieval; `defineMemory` = conversation/run-state memory.
+`defineRAG` runs on `defineMemory({ type: SEMANTIC, strategy: TOP_K })`. Same machinery, three deliberate differences: a corpus is **read-only** (it never stores the conversation), it reads under its **own namespace** rather than the run's identity, and its chunks render as **citable `<source>` blocks**. For conversation memory alongside a corpus, register both — `.rag(defineRAG(...))` and `.memory(defineMemory(...))`, each with its own store.
+
+**Why did the agent read this passage?** `agentfootprint.memory.retrieved` carries every candidate with its score — including the ones that were rejected and why. `agentfootprint.memory.attached` fires per chunk that reached the prompt. `agentfootprint.context.injected` reports `source: 'rag'` with that chunk's `retrievalScore` / `rankPosition` / `threshold`. The whole record is on root state as `retrievalEvidence_<id>`, where a backward slice can reach it.
 
 ### Agent (ReAct primitive)
 
@@ -464,7 +472,7 @@ Resilience decorators live on the `agentfootprint/resilience` subpath
 (not the main barrel). Each preserves the `LLMProvider` interface and
 stacks freely.
 
-### Observability — 71 typed events across 20 domains
+### Observability — 72 typed events across 20 domains
 
 ```typescript
 agent.on('agentfootprint.context.injected', (e) =>

package/CLAUDE.md

@@ -13,11 +13,11 @@ Entry points (package.json exports): `.` core API · `/observe` ALL observabilit
 | core/slots/ | the 3 context-slot subflow builders + thinking subflow — intentionally NOT exported |
 | core-flow/ | Sequence/Parallel/Conditional/Loop — RunnerBase subclasses with own charts |
 | patterns/ | Debate/MapReduce/Reflection/SelfConsistency/Swarm/ToT — pure composition of runners, no new control flow |
-| adapters/ | hexagonal ports (types.ts = ALL port interfaces) + vendor impls (llm/, memory/, identity/, observability/) |
+| adapters/ | hexagonal ports (types.ts = ALL port interfaces) + vendor impls (llm/, memory/, identity/, observability/). memory/sqliteVector.ts (8.9.0) = the only FULL MemoryStore we ship with `search` besides InMemoryStore — exact cosine over a resident Float32Array matrix, hydrated per namespace on first search and dropped on any write to it |
 | recorders/core/ | bridges footprintjs events → typed EventDispatcher (ContextRecorder, EmitBridge, typedEmit) — auto-attached by Agent.createExecutor; most factories also exported via `/observe` for manual wiring (EmitBridge itself stays internal) |
 | recorders/observability/ | consumer recorders over the typed stream (RunStepRecorder, FlowchartRecorder, Status, Trace replay) + `recordRun` — THE producer of a recording `{snapshot, events, structure}` (the shape lens's `observeRecording` consumes; `structure` = `getSpec().buildTimeStructure`, which no snapshot carries). Anything that saves a run goes through it |
 | lib/ | first-party sub-libraries: injection-engine/, context-bisect/ (localizeContextBug, toBacktrackTrace + sliceToBacktrackTrace — the atui board serializers), influence-core/, trace-toolpack/ (selfExplain; 6 tools incl. variable-first `backtrack(variable, element?)`), context-ledger/ (which pieces EARNED their tokens — post-run offers/uses/outcomes bookkeeping + demote-never-starve gates `ledgerToolGate`/`ledgerEntryScorer`/`ledgerGated`; grouped-mode folds sf-llm-call inner logs, unmeterable runs → undefined; /observe), mcp/, rag/, tool-lint/ |
-| memory/ | store/ (MemoryStore port) + pipeline presets + stages + beats/facts + causal/ (dev-only, TOP_K+search()-only) + wire/mountMemoryPipeline |
+| memory/ | store/ (MemoryStore port) + pipeline presets + stages + beats/facts + causal/ (dev-only, TOP_K+search()-only) + wire/mountMemoryPipeline + retrieval/ (8.8.0: the `RetrievalStrategy` seam + `RetrievalEvidence`, the record a retrieval leaves — `topK()` is what every earlier release did unnamed) |
 | events/ | EventDispatcher (wildcard subs), registry (EVENT_NAMES, AgentfootprintEventMap), payloads |
 | conventions.ts | THE builder↔recorder protocol: SUBFLOW_IDS/STAGE_IDS (internal), INJECTION_KEYS/stageRole/milestoneFor (exported, Lens-facing) |
 
@@ -39,6 +39,8 @@ Traps: `src/observability/` holds the finder IMPLEMENTATIONS (canonical home; `d
 - **New typed event (3-step)**: payload interface in events/payloads.ts + entry in `AgentfootprintEventMap` (registry.ts:198) + append to `ALL_EVENT_TYPES` (registry.ts:488, count-asserted by tests). New DOMAIN also needs a bridge attach in Agent.createExecutor or emits never reach the dispatcher — AND a hand-edit to `DomainWildcard` (dispatcher.ts:67-82; already missing validation/credential/reliability).
 - **Strategy (vendor sink)**: shapes in strategies/types.ts (Observability :130, Cost :169, LiveStatus :201, Lens :234); attach via `agent.enable.*` or `registerObservabilityStrategy` (strategies/registry.ts). New vendor = export from observability-providers.ts, NOT a new subpath.
 - **Memory store**: implement `MemoryStore` (memory/store/types.ts:113; `search?` REQUIRED for causal memory); pass to `defineMemory({store})`. Memory TYPE/STRATEGY unions are CLOSED (define.types.ts:57/74 — new one edits defineMemory dispatch + a pipeline builder).
+- **Durable store** (8.9.0): `sqliteVectorStore` follows `hosting/sqliteSessions` line for line — lazy `node:sqlite`, WAL read-back on `journalMode`, STRICT tables, schema-identity + schema-version refusals, `':memory:'` refused. TWO things it adds that have no precedent there: `putMany`/`putIfVersion`/`forget` wrap in a transaction (sqliteSessions has none), and the EMBEDDER FINGERPRINT (`'<id>@<dims>'`, one per namespace in `af_index_meta`) is refused at write AND query. `SqliteUnavailableError` is now ONE class in `lib/sqliteUnavailable.ts` re-exported by both doors — a second class of that name is a duplicate type the build refuses.
+- **Retrieval rule** (8.8.0): implement `RetrievalStrategy` (memory/retrieval/types.ts) — `select(pool) → verdict[]`, one verdict per candidate, order preserved; it never touches the store and never embeds. Pass as `defineRAG({retrieval})`. `topK()` is the only shipped one; rerank/MMR are named-but-deferred adapters behind the same interface. `TopKStrategy` is a UNION whose arms exclude (`{topK,threshold}` vs `{retrieval}`) — refused in the type AND at runtime, because two spellings of one rule can disagree.
 - **Injection/skill**: `Injection = {id, flavor, trigger, inject}` (lib/injection-engine/types.ts:161); trigger is a closed 4-variant union (:30 — new kind edits evaluator.ts:40-74 switch). Factories defineSkill etc.; skill graph via `skillGraph()` (skillGraph.ts:392) with pluggable `EntryScorer` (entryScorer.ts:64). `SkillGraphConfig` is a UNION (flat arm `start`/`steps` vs tree arm) — the contradictions it encodes are ALSO refused at build (`.tree()` + `.entry()`/`.route()`, a non-leaf in `skills[]` under a tree, two skills claiming one id, a second `.skillGraph()` on one agent).
 - **Ports table** (wired via AgentOptions): PermissionChecker (adapters/types.ts:403), PricingTable (:412), CredentialProvider (identity/types.ts:89), CacheStrategy (cache/types.ts:151, registerCacheStrategy), ThinkingHandler (thinking/types.ts:114 — auto-wire scans HARDCODED SHIPPED_THINKING_HANDLERS, registry.ts:26), ReliabilityConfig (reliability/types.ts:183), OutputSchemaParser (core/outputSchema.ts:62, duck-typed).
 - **Output contract** (7.26): `.outputSchema(parser, opts)` — `retries>0` mounts the THIRD Route branch (`STAGE_IDS.OUTPUT_RETRY`, same `{loopTo}` as tool-calls) + swaps the decider for `buildEnforcingDecider` (stages/route.ts); `strategy:'tool-forced'` builds the synthetic tool (core/agent/outputEnforcement.ts) that callLLM appends at REQUEST assembly and normalizes back into `content` inside `singleProviderCall` — it never touches the tools slot, the registry or the dispatcher. TWO build refusals (agent has tools; no derivable jsonSchema) + ONE run-start refusal (`provider.carriesForcedToolChoice`; ABSENCE = NO, the opposite of carriesInMessages). A new wire capability = adapters/types.ts + 6 adapters + 3 resilience wrappers (withFallback publishes the AND).
@@ -49,8 +51,10 @@ Traps: `src/observability/` holds the finder IMPLEMENTATIONS (canonical home; `d
 ## Change-impact map
 - **conventions.ts** (STAGE_IDS/SUBFLOW_IDS/INJECTION_KEYS) → chart builders that mount by id, ContextRecorder slot attribution, localizer loop-head detection (lib/context-bisect/trajectory.ts:17-33), `stageRole`/`milestoneFor` (Lens contract), BoundaryRecorder. Renaming an id is the whole blast radius.
 - **BoundaryRecorder wiring is THREE connections, all at record time**: `runner.attach` (boundaries), `.subscribe(runner)` (what's inside them), `{getCommitCount}` (where each sits on the commit axis). The third fails SILENTLY — every event stamps `commitIdxBefore: 0`, `boundaryIndex` stays empty by design, and an offline step strip has nothing to place. Unrecoverable after the run (the commit log never records WHEN a boundary was crossed). Wired by `attachFlowchart` (which `enable.flowchart`/`enable.localObservability` both go through) and by `recordRun`; a new entry point must pass all three.
+- **Embedder fingerprint** (8.9.0) → `Embedder.id` (optional; every shipped embedder sets one, and NONE include dims — the store appends `@<dims>` itself, so an id carrying its own size double-stamps) + `indexDocuments` defaulting `embedderId` to it + `SqliteVectorStore.reconcileFingerprint` (the only comparison site). Rule: dimensions ALWAYS decide, model ids decide only when BOTH sides named themselves — refusing on an absent name would block the majority of callers who never pass `embedderId`.
+- **Retrieval record** (8.8.0) → FOUR stages write one object in sequence: `loadRelevant` (candidates+scores+threshold verdicts) → `pickByBudget` (re-marks admitted→over-budget/over-max-entries) → `formatDefault` (`promptFragment` + `promptPosition`) → the read mount's outputMapper lifts it to root as `retrievalEvidence_<id>`. `memoryRecallInjections` then splits ONE recall into one ActiveInjection PER CHUNK — guarded by a byte-equality check (`fragments.join('\n\n') === systemContent`) that falls back to the single injection rather than change the prompt. `rank` (score order) and `promptPosition` (picker order) are DIFFERENT and both load-bearing: joining fragments in rank order reproduces the right bytes in a sequence the model never saw.
 - **AgentState** → all 8 stages/ files, both builders' mappers, memory-wire STRING-TYPED keys ('runIdentity'/'turnNumber'/… buildAgentChart.ts:177-180 — not refactor-safe), finalizeResult's `reliabilityFail*`/`policyHalt*` reads (rename silently kills the typed errors).
-- **events/** → 71 typed events across 20 domains (counts anti-drift-tested against this file — update BOTH when adding events): ALL_EVENT_TYPES exhaustiveness tests, DomainWildcard hand-list, ~42 importers (recorders, strategies, stream, commentary).
+- **events/** → 72 typed events across 20 domains (counts anti-drift-tested against this file — update BOTH when adding events): ALL_EVENT_TYPES exhaustiveness tests, DomainWildcard hand-list, ~42 importers (recorders, strategies, stream, commentary).
 - **adapters/types.ts LLMMessage/LLMRequest** → 62 importers: tool_use round-trip (toolCalls.ts:115-135), wire assembly (callLLM.ts:150-160), providers, cache strategies, security/extractSequence, reliability loop.
 - **Cache** → strategy registration is a MODULE SIDE EFFECT (src/index.ts:15-17); an entry point skipping that import silently falls back to NoOp. Resolved once per Agent at construction (Agent.ts:347).
 - **Injection engine eval semantics** → Evaluate stage cursor keystone (buildInjectionEngineSubflow.ts:214-222), Route stage MIRRORS slot filters (keep in sync with buildSystemPromptSlot), read_skill gate (toolCalls.ts:747). The gate's allowed set is TWO kinds of id (8.4.0): a HOP (`deps.allowedSkillIds` = `graph.reachableSkills(cursor)` — activates AND moves the cursor) ∪ an OPEN skill (`deps.openSkillIds` = `Agent.openSkillIds()`, Agent.ts:1052 — a registered skill whose trigger is `llm-activated` AND that the graph declares no incoming edge to; activates, never moves the cursor). Both clauses are load-bearing: the trigger test is "read_skill can really activate this", the edge test keeps a bare model edge `.route(a,m)` from-gated. **8.5.0** extends the SAME trigger test three ways: a decision `tree()` returns `[]` from `reachableSkills` (leaves compile to `rule` triggers, so a pick could only be accepted then dropped — the gate refuses with a tree-specific message, `deps.skillGraphIsTree`, derived in AgentBuilder from `graph.nodes.some(kind==='predicate')`); `surfaceMode:'tool-only'` is refused at agent build for any skill whose trigger is not `llm-activated` (skillBodyDelivery.ts — the read_skill tool result is the only channel that mode has, and the graph never calls read_skill); and `read_skill`'s DESCRIPTION is rebuilt per iteration from `reachableSkills(cursor) ∪ open` (Agent.readSkillOfferFor → buildToolsSlot's `readSkillFor`). **The read_skill ENUM must stay the full catalog** — `toolArgValidation` defaults to `'enforce'` and runs at toolCalls.ts:570, BEFORE the gate at :765, and sets `error = true` which the gate skips; narrowing the enum silently retires the teaching refusal, `skill.rejected`, routeRecorder's rejection hops and the rejected-cap governor.

package/ai-instructions/claude-code/SKILL.md

@@ -379,7 +379,7 @@ if (isPaused(result)) {
 }
 ```
 
-## Observability — 71 typed events × 20 domains
+## Observability — 72 typed events × 20 domains
 
 ```typescript
 agent.on('agentfootprint.context.injected', (e) =>