@yemi33/minions 0.1.2401 → 0.1.2403

package/docs/team-memory.md

@@ -1,164 +1,482 @@
-# Team Memory
-
-Minions agents share knowledge through a layered Markdown system plus a structured SQLite retrieval index. Before an agent does any independent research — web searches, broad codebase exploration, external doc reading — it is expected to check what the team already knows.
-
-This document describes the four-tier read order the engine enforces in dispatch prompts, the rationale, and the constraint that `knowledge/` is **sweep-write-only**.
-
-## Why team memory exists
-
-Multiple agents work the same repository in parallel and across days. Without shared memory:
-
-- Agents re-research the same questions (cost, time, context churn).
-- Agents re-litigate decisions the team already made (drift, regressions).
-- Human-flagged warnings (e.g., "do not self-approve PRs") get ignored because they live outside the prompt.
-
-Team memory closes the loop. The engine always injects pinned context. By default, structured retrieval replaces broad team and personal notes with a bounded task-relevant memory pack. Playbooks still instruct agents to consult `knowledge/` and `notes/inbox/` on disk before going wider.
-
-## Structured memory and retrieval
-
-Migration `017-agent-memory.js` backfills the existing Markdown sources into `memory_records` and an FTS5 index without deleting or rewriting those files. Records carry a type (`semantic`, `episodic`, or `procedural`), scope, trust level, source reference, confidence, importance, validity timestamps, and lifecycle status (`active`, `superseded`, `retracted`, or `expired`).
-
-`engine/memory-retrieval.js` searches active records in global, current-project, and assigned-agent scopes. It reranks lexical matches by scope, trust, path match, recency, importance, and confidence, then packs the best records into a strict byte budget. Agent- or external-authored procedural records are never recalled as instructions. Every injected pack remains inside an `<UNTRUSTED-INPUT>` fence.
-
-Structured retrieval is controlled by the default-on `memoryRetrieval` feature flag:
-
-- **Flag off:** legacy prompt behavior.
-- **Flag on + shadow mode off (default):** inject `Relevant Memory` when retrieval finds evidence; otherwise fall back to legacy notes and personal memory.
-- **Flag on + `memoryRetrievalShadowMode: true`:** compute retrieval and telemetry, but keep legacy prompts.
-
-Settings expose the retrieval bounds directly:
-
-| `engine.*` key | Default | Settings range | Purpose |
-|----------------|---------|----------------|---------|
-| `memoryRetrievalTopK` | `8` | 1–30 records | Maximum records injected into one dispatch |
-| `memoryRetrievalMaxBytes` | `8192` | 1024–65536 bytes | Hard byte budget for the `Relevant Memory` appendix |
-| `memoryRetrievalCandidateLimit` | `50` | 5–200 records | FTS5 candidate pool before deterministic reranking |
-
-The same pane controls shadow mode and episodic capture. The Inbox page's **Memory search** control shows provenance and status and supports explicit pin, retract, and restore actions.
-
-## The four-tier read order
-
-Every agent prompt (built by `engine/playbook.js`) is rendered with the rule: **check team memory first, then look outside.** The canonical read order (from `playbooks/shared-rules.md`) is:
-
-1. **`pinned.md`** — critical context flagged by the human teammate. **READ FIRST.**
-2. **`knowledge/`** — categorized KB entries written by the consolidation sweep.
-3. **`notes.md`** — consolidated team knowledge, decisions, and rolling context.
-4. **`notes/inbox/`** — recent agent findings not yet consolidated.
-
-Agents are also encouraged, after the four primary tiers, to skim `agents/*/live-output.log` for related tasks and the `resultSummary` of prior completed work items on the same topic. Only after exhausting team memory should an agent fall back to web search, broad codebase exploration, or external docs.
-
-### 1. `pinned.md` — human-flagged critical context
-
-`pinned.md` lives at the Minions root (`MINIONS_DIR/pinned.md`). It holds notes the human teammate has explicitly pinned through the dashboard — short, high-signal rules and warnings that **must** be visible to every agent on every dispatch.
-
-The engine injects the file verbatim into the prompt under a "Pinned Context (CRITICAL — READ FIRST)" heading and caps it at 4 KB to protect the prompt budget.
-
-Typical contents include credential constraints (e.g., shared `gh` auth means PR self-approval is blocked), workflow policies (e.g., merging policy, TDD requirement), and active setup state. If pinned context contradicts a playbook instruction, pinned context wins for the duration of the dispatch.
-
-### 2. `knowledge/` — categorized KB entries (sweep-write-only)
-
-`knowledge/` is the long-form team knowledge base. Files are grouped by category:
-
-- `knowledge/architecture/` — design decisions, system overviews
-- `knowledge/conventions/` — coding patterns, repo norms, standards
-- `knowledge/project-notes/` — per-project context and observations
-- `knowledge/build-reports/` — CI/build investigation results
-- `knowledge/reviews/` — PR review findings and rationale
-- `knowledge/agents/<agentId>.md` — per-agent personal memory (curated by the sweep)
-
-Filenames follow the `YYYY-MM-DD-<agent>-<slug>.md` convention so chronological sorting just works.
-
-Agents read `knowledge/` directly with grep/glob/view. They **do not** write to it. See [Sweep-write-only constraint](#sweep-write-only-constraint) below.
-
-**Full-body copies vs. condensed stubs.** When `engine/consolidation.js#classifyToKnowledgeBase` promotes an inbox note into `knowledge/`, it only copies the full body when the note is routed to `conventions/` or its content carries a reusable-knowledge section header (`## Patterns`, `## Conventions`, `## Gotchas`, `## Dependencies`, `## Best Practices`, etc.). Narrowly-scoped, one-off notes (a single PR review, a task status update) instead get a deterministic, LLM-free condensed stub — title + first-paragraph abstract + a link back to the archived inbox note — to avoid duplicating disk usage and prompt-token cost on every future dispatch. Every promoted entry's frontmatter gains a `reusable: true|false` field so this can be audited (issue #604).
-
-### 3. `notes.md` — consolidated team notes
-
-`notes.md` is the rolling team notebook. The consolidation sweep periodically merges high-signal findings from the inbox into dated `### YYYY-MM-DD` sections at the top of `notes.md`, with links back to the underlying KB entries.
-
-In legacy and shadow modes, the engine injects `notes.md` under a "Team Notes (MUST READ)" heading. When active retrieval returns evidence, the bounded `Relevant Memory` pack replaces this broad appendix. When the file exceeds `ENGINE_DEFAULTS.maxNotesPromptBytes`, the legacy path keeps the most recent 10 sections, truncates the body, and appends a footer noting how many older sections live on disk.
-
-### 4. `notes/inbox/` — fresh agent findings
-
-`notes/inbox/` holds the freshest, un-consolidated learnings. Every agent writes exactly one inbox file at the end of a successful task to a path the playbook supplies (`notes/inbox/<agent>-<work-item-id>-<date>-<time>.md`). The file always begins with a YAML frontmatter block (`id`, `agent`, `date`) so the consolidation sweep can track it.
-
-Inbox notes are the canonical place to record:
-
-- New patterns or conventions discovered while implementing.
-- Gotchas, warnings, or repro recipes for future agents.
-- File-and-line citations supporting any claims.
-
-The consolidation sweep is what eventually promotes inbox notes into `knowledge/` and `notes.md`. Until then, agents should grep the inbox themselves for very recent findings on adjacent work.
-
-#### Failure-path exception
-
-Inbox writes are **success artifacts only**. If a task fails, is blocked, is cancelled, or ends partial, the agent must **not** create an inbox note. When `memoryEpisodicCapture` is enabled, lifecycle records a compact structured outcome for successful, partial, and failed dispatches; it never stores transcripts or chain-of-thought and skips nonce-mismatched or injection-flagged reports.
+# Minions Memory System
+
+Minions long-term memory is a hybrid system:
+
+- Markdown files preserve operator-readable context, the inbox workflow, and the
+  legacy prompt path.
+- SQLite records provide scoped lifecycle state, FTS5 search, deterministic
+  task-aware retrieval, and retrieval telemetry.
+
+Neither layer replaces the other. Active retrieval reads structured records from
+SQLite; pinned context and the legacy fallback still read Markdown directly.
+This document covers knowledge memory, not the engine/dashboard process-memory
+diagnostics described in [diagnostics-memory.md](diagnostics-memory.md).
+
+## End-to-end data flow
+
+```text
+Human Quick Note / agent finding / engine note
+                       |
+                       v
+               notes/inbox/*.md
+                       |
+       engine tick: consolidateInbox() (threshold-gated)
+                       |
+          +------------+-------------+----------------+
+          |                          |                |
+          v                          v                v
+      notes.md            knowledge/<category>/   knowledge/agents/
+   team digest             readable KB entry       <agent>.md
+          |                          |                |
+          +--------------------------+----------------+
+                                     |
+                                     v
+                         memory_records + FTS5
+                         (semantic SQL records)
+
+Human pinned note ------------------------------> pinned.md
+Human direct KB authoring ---------> knowledge/ + memory_records
+Dispatch completion (opt-in) -------------------> memory_records
+                                                  (episodic record)
+
+At the next dispatch:
+
+  pinned.md
+      +
+  active retrieval: task query -> scoped FTS5 -> rerank -> bounded Relevant Memory
+      OR
+  legacy fallback: bounded notes.md + bounded knowledge/agents/<agent>.md
+```
 
-## Rationale
+The playbook also tells agents to inspect `knowledge/` and `notes/inbox/` on
+disk. Those directories are not injected wholesale into every prompt.
+
+## Storage and authority
+
+There is no single store that is authoritative for every kind of memory.
+Authority is divided by responsibility:
+
+| Store | Authoritative for | Notes |
+|-------|-------------------|-------|
+| `engine/state.db` tables `memory_records` and `memory_retrieval_runs` | Structured record bodies and metadata used by active retrieval; record lifecycle (`active`, `superseded`, `retracted`, `expired`); retrieval telemetry | `memory_records_fts` is a derived external-content FTS5 index maintained by SQLite triggers. |
+| `pinned.md` | Operator-authored pinned context | Read directly for every rendered agent prompt. It is not a structured-memory pin. |
+| `notes/inbox/*.md` | Pending human, agent, and engine-authored findings awaiting consolidation | Successful agent tasks write one note to the assigned path; failures do not. |
+| `notes.md` | Operator-readable team digest and legacy team-notes prompt input | Consolidation appends here under a file lock. |
+| `knowledge/<category>/*.md` | Curated, browsable knowledge-base content | Written by engine-managed consolidation/sweep paths and explicit human dashboard actions. Agents must not edit it directly. |
+| `knowledge/agents/<agentId>.md` | Legacy per-agent notebook | Written only by the consolidation pipeline; never by the agent itself. |
+| `notes/archive/` and `knowledge/_swept/` | Recovery copies and retired source artifacts | Retention differs by subsystem; see [kb-sweep.md](kb-sweep.md). |
+| `config.json` | Feature and engine settings | Intentionally file-backed configuration, not memory content. |
+
+SQLite is the sole authority for runtime state generally; see
+[design-state-storage.md](design-state-storage.md). The Markdown paths above
+remain intentionally file-backed documents and coordination artifacts.
+
+### Synchronization is directional, not a two-way mirror
+
+The normal ingestion paths often write both layers, but they are not one
+cross-store transaction:
+
+- Migration 017 performed a one-time backfill from existing knowledge files,
+  team-note sections, and per-agent sections into SQL.
+- Inbox consolidation writes knowledge and per-agent files and then upserts
+  corresponding semantic records.
+- `POST /api/knowledge` writes both the knowledge file and a human-trusted SQL
+  record.
+- Episodic capture writes SQL only.
+- Pinned context writes `pinned.md` only.
+
+Direct filesystem edits, `POST /api/notes-save`, manual inbox **Persist** or
+**Promote to KB**, and KB sweep rewrites/moves do not perform a general SQL
+reindex. File pruning also does not expire an already-created structured
+record. Consequently, Markdown and SQL can diverge; active retrieval follows
+SQL, while the legacy path follows the files. There is currently no full
+reindex API or CLI command. The SQL layer also has no scheduled record expiry;
+the `expired` status exists, but the KB sweep's file TTL does not apply it.
+
+## Inputs
+
+### Human inputs
+
+Humans can add memory through the dashboard or its localhost API:
+
+- **Quick Notes** calls `POST /api/notes`, creating a Markdown file in
+  `notes/inbox/` for normal consolidation.
+- **Pinned Notes** calls `POST /api/pinned` (plus `/remove` and `/update`) and
+  edits `pinned.md`.
+- **New Knowledge Base Entry** calls `POST /api/knowledge`, writing a category
+  file and a global, human-trusted semantic record.
+- Inbox actions can persist an item into `notes.md` or promote it directly to a
+  knowledge category.
+- Feedback actions and several engine/operator workflows also create inbox
+  notes, so not every inbox file originates from the Quick Notes form.
+
+### Agent inputs
+
+An agent records durable findings in exactly one assigned
+`notes/inbox/<agent>-<work-item>-<timestamp>.md` file after a successful task.
+The required YAML `agent:` field is the preferred author identity; the filename
+prefix is a fallback. An optional `project:` frontmatter value makes the
+knowledge SQL record project-scoped.
+
+Agents read but never delete, move, or overwrite `knowledge/`, including their
+own `knowledge/agents/<agentId>.md`. Corrections go through a new inbox note so
+the pipeline can preserve provenance and reconcile the old material.
+
+### Completion outcomes
+
+When `engine.memoryEpisodicCapture` is enabled, post-completion lifecycle writes
+a compact SQL episode for successful, partial, and failed dispatches. This path
+is independent of the agent's success-only inbox note.
+
+## Inbox consolidation
+
+`engine.js` calls `consolidateInbox(config)` on every tick. Consolidation starts
+when the number of eligible Markdown files reaches
+`engine.inboxConsolidateThreshold` (default `5`). It excludes inbox items pinned
+against consolidation and files already in the in-flight set.
+
+Before the normal pipeline, a content-hash circuit breaker checks each note's
+first 200 characters plus its full length. If one hash represents more than 80%
+of the eligible batch, consolidation skips the LLM and all `notes.md`, KB/SQL,
+and personal-memory writes, then archives the entire batch, including any
+minority notes. This means a note from a duplicate-heavy batch may exist only
+under `notes/archive/`.
+
+The normal pipeline is:
+
+1. Read the eligible inbox files and the current `notes.md`.
+2. Ask the configured direct LLM (Haiku, one turn) for a concise deduplicated
+   digest. Each inbox body is fenced as untrusted input in the consolidation
+   prompt.
+3. If the runtime is unavailable, output is malformed, the call fails, or the
+   three-minute deadline is exceeded, use the deterministic regex fallback.
+4. Append the digest to `notes.md` under its file lock. If the existing file is
+   present but unreadable, defer the whole batch rather than risk clobbering it.
+5. Classify every source note into a knowledge category. Reusable notes retain
+   their full body; narrow one-off notes become a short stub linked to the
+   archived original.
+6. Upsert a global or project-scoped semantic record for the KB representation.
+7. Route eligible notes into the configured author's personal file and upsert an
+   agent-scoped semantic record.
+8. Archive processed inbox files. If an eligible personal-memory write fails,
+   leave that source note in the inbox so a later tick can retry it.
+
+`notes.md` is kept near 50,000 JavaScript code units by trimming at section
+boundaries and appending removed sections to
+`notes/archive/notes-overflow.md`. If the overflow archive cannot be written,
+the engine leaves the oversized file intact rather than lose content.
+
+### Per-agent memory
+
+Personal files use dated `### YYYY-MM-DD: <title>` sections. Routing requires:
+
+- a valid agent id from frontmatter or the filename;
+- a matching key in `config.agents`;
+- a non-empty body; and
+- a non-`temp-*` agent.
+
+Each appended inbox body is fenced on disk with source provenance. Canonical
+sectioned files are pruned oldest-first to at most
+`engine.agentMemoryMaxEntries` (default `300`) and a 25,000-byte UTF-8 budget.
+The newest section is retained even if it alone exceeds the byte target.
+
+For correction-shaped notes, the conditional reconciliation pass can ask Haiku to
+rewrite stale facts before appending. Failures fall back to a plain append. A
+separate default-off summary pass can fold older sections into a fenced
+`Earlier learnings summary`; it rechecks the file after the LLM call so a
+concurrent append cannot be overwritten.
+
+The heavier KB sweep is a separate process. `engine.autoConsolidateMemory`
+controls its four-hour automatic cadence; it does not control inbox-to-notes
+consolidation. See [kb-sweep.md](kb-sweep.md) for deduplication, TTL, rewrite,
+and archive behavior.
+
+## Structured records and FTS5 retrieval
+
+Migration `017-agent-memory.js` created:
+
+- `memory_records`, including content, type, scope, trust, confidence,
+  importance, provenance, validity, and supersession fields;
+- the external-content `memory_records_fts` table over title, body, tags, and
+  source path, with insert/update/delete triggers; and
+- `memory_retrieval_runs`, capped by the store to the newest 5,000 runs.
+
+Supported record values are:
+
+| Dimension | Values |
+|-----------|--------|
+| Type | `semantic`, `episodic`, `procedural` |
+| Scope | `global`, `project:<key>`, `agent:<key>` |
+| Trust | `human`, `system`, `agent`, `external` |
+| Status | `active`, `superseded`, `retracted`, `expired` |
+
+Built-in file ingestion creates semantic records. Completion capture creates
+episodic records. The schema supports procedural records, but recall excludes
+procedural content unless its trust is `human` or `system`.
+
+Record ids are deterministic from source type, source reference, and content
+hash. Re-upserting identical content is idempotent. New content for the same
+source type/reference automatically supersedes the previous active row unless
+the caller supplies an explicit supersession.
+
+### Query construction and selection
+
+`engine/memory-retrieval.js` builds a task query from the work-item title,
+description, work type, failure class, source plan, references, PR title, and PR
+branch. The SQL store:
+
+1. tokenizes at most 32 unique non-stop-word terms;
+2. issues an OR query against FTS5;
+3. limits dispatch recall to active records in global, current-project, and
+   assigned-agent scopes; and
+4. rejects agent/external procedural records.
+
+FTS5 lexical rank supplies the initial order. A deterministic reranker then
+boosts matching project/agent scope, human/system trust, task-mentioned source
+paths, recency, importance, and confidence. Results are deduplicated by content
+hash, then packed in rank order.
+
+Every packed record includes:
+
+- record id;
+- type and scope;
+- effective date; and
+- `sourcePath` or `sourceRef`.
+
+That provenance is visible both in the `Relevant Memory` appendix and the
+dashboard's structured-memory search.
+
+## Prompt assembly and fallback
+
+`engine/playbook.js` renders the requested playbook, adds shared rules, then
+appends memory context in this order:
+
+1. **Pinned Context**, when `pinned.md` is non-empty.
+2. Either **Relevant Memory** or the legacy pair:
+   - **Team Notes** from `notes.md`;
+   - **Personal Memory** from `knowledge/agents/<agentId>.md`.
+3. Later non-memory appendices such as propagated project instructions.
+
+Pinned context is independent of retrieval and remains present in every mode.
 
-The read order is deliberately ordered by **trust × recency**:
+| Mode | Prompt behavior |
+|------|-----------------|
+| `memoryRetrieval` off | Pinned context plus legacy team/personal memory. No retrieval telemetry. |
+| Flag on, `memoryRetrievalShadowMode: true` | Compute retrieval and store telemetry, but inject the legacy team/personal memory. |
+| Flag on, shadow off, non-empty result | Inject pinned context plus `Relevant Memory`; suppress broad team/personal appendices. |
+| Flag on, shadow off, empty query/no matches | Fall back to legacy team/personal memory. |
+| Retrieval or telemetry write throws | Log a warning and fall back to legacy team/personal memory. |
 
-- `pinned.md` is human-curated and authoritative — it overrides everything else.
-- `knowledge/` is curated by the sweep from validated agent findings — high-signal but slightly delayed.
-- `notes.md` mirrors `knowledge/` in summarized form, optimized for prompt injection.
-- `notes/inbox/` is raw, recent, and unverified — useful for the freshest context but treat critically.
+Missing legacy files are skipped without failing dispatch.
+
+### Bounds
+
+| Content | Default bound | Enforcement |
+|---------|---------------|-------------|
+| Pinned input | 4,096 JavaScript UTF-16 code units, then a truncation notice | `String.length`/`slice`, despite the historical "4 KB" comment. |
+| Retrieved pack | 8,192 UTF-8 bytes (`memoryRetrievalMaxBytes`) | Clamped to 1,024-65,536 bytes; includes record blocks and separators, but not the outer heading/fence. |
+| Selected records | 8 (`memoryRetrievalTopK`) | Clamped to 1-30. |
+| FTS candidate pool | 50 (`memoryRetrievalCandidateLimit`) | Dashboard accepts 5-200; the store hard-caps at 200. |
+| Legacy team notes | 8,192 UTF-8 bytes (`ENGINE_DEFAULTS.maxNotesPromptBytes`) | Prefer the newest ten sections, then UTF-8-safe truncation and an older-entry footer. |
+| Legacy personal memory | 8,192 UTF-8 bytes | Prefer the newest ten sections, then UTF-8-safe truncation. |
+| Any untrusted fence body | 65,536 UTF-8 bytes (`ENGINE_DEFAULTS.untrustedFenceMaxBytes`) | UTF-8-safe truncation inside the fence with a remaining-byte marker. |
+
+The configured retrieval budget controls selected record text, not all Markdown
+or XML-like wrapper overhead added to the final prompt.
 
-Following this order achieves two goals:
+## Security and trust boundaries
 
-1. **Avoid re-research.** If another agent already investigated the same module, fix, or design question, the answer is somewhere in this stack. Reading saves a costly round-trip.
-2. **Respect team decisions.** Pinned policies and consolidated conventions encode prior agreement (often with the human). Skipping team memory and acting on first principles is how regressions and drift happen.
+Memory content is evidence, not a new instruction channel. Before prompt
+insertion, pinned context, retrieved records, team notes, and personal memory
+are wrapped with `engine/untrusted-fence.js`:
 
-## Sweep-write-only constraint
+```text
+<UNTRUSTED-INPUT source="kind:provenance">...</UNTRUSTED-INPUT>
+```
 
-**Agents must never delete, move, or overwrite files in `knowledge/`.** Only the consolidation sweep (`engine/consolidation.js`) writes to `knowledge/`. This rule is restated in every dispatch prompt under a "Knowledge Base Rules" heading.
+The wrapper sanitizes the source attribute, escapes attempted closing tags, and
+truncates by UTF-8 bytes. The consolidation LLM receives separately fenced
+inbox bodies plus an explicit instruction to summarize rather than execute
+their contents.
+
+Additional defenses are:
+
+- agent- or external-trusted procedural records are ineligible for recall;
+- active retrieval includes record provenance and tells the agent to verify
+  claims against live code;
+- a completion report flagged with
+  `failure_class: "injection-flagged"` is not captured as an episode; and
+- nonce-mismatched completion reports never reach episodic capture.
+
+`task_description` is deliberately not fenced because it is the instruction
+being executed, not recalled evidence. For the broader threat model and the
+required policy for new splice sites, see [security.md](security.md).
+
+## Episodic capture
+
+`captureTaskMemoryEpisode()` runs after completion status and nonce validation.
+When enabled, it records:
+
+- outcome, work item/type, agent, and project;
+- PR reference and failure class;
+- bounded affected-file and test lists; and
+- a bounded result summary.
+
+Secret-shaped values are redacted, and lists and strings are capped before
+storage. The preferred summary comes from parsed runtime output or the structured
+completion report. If neither supplies one and stdout has no `"type":` marker,
+lifecycle falls back to raw stdout; that value is redacted and truncated to
+4,000 characters, but transcript- or tool-log-like text is not filtered out.
+Project scope is preferred, then agent scope, then global scope. Records use
+system trust, with higher importance for failed and partial outcomes.
+
+Capture is best-effort: a write failure is logged but does not change the task's
+completion result.
+
+## Configuration and feature flags
+
+| Key | Default | Purpose |
+|-----|---------|---------|
+| `features.memoryRetrieval` | `true` | Enables structured retrieval. Feature resolution is environment override, then `config.features`, then registry default. |
+| `engine.memoryRetrievalShadowMode` | `false` | Measure retrieval without replacing legacy prompt memory. |
+| `engine.memoryRetrievalTopK` | `8` | Maximum selected records. |
+| `engine.memoryRetrievalMaxBytes` | `8192` | UTF-8 byte budget for selected record blocks. |
+| `engine.memoryRetrievalCandidateLimit` | `50` | FTS5 candidates before reranking. |
+| `engine.memoryEpisodicCapture` | `false` | Store compact completion episodes. |
+| `engine.inboxConsolidateThreshold` | `5` | Eligible inbox files required before consolidation starts. |
+| `engine.autoConsolidateMemory` | `false` | Automatically run the separate KB sweep every four hours. |
+| `engine.agentMemoryMaxEntries` | `300` | Non-summary section window in a personal file. |
+| `engine.agentMemorySummaryEnabled` | `false` | Enable LLM compression of older personal sections. |
+| `engine.agentMemorySummaryThreshold` | `30` | Oldest-section batch size for a summary. |
+| `engine.agentMemorySummaryDays` | `30` | Age trigger for summary eligibility. |
+
+The feature flag is exposed by the generic Dashboard **Settings > Features**
+surface. Settings also exposes shadow mode, episodic capture, retrieval bounds,
+the inbox threshold, and automatic KB sweep. The personal-file window and
+summary keys are configuration-only today.
+
+The environment override for the feature is
+`MINIONS_FEATURE_MEMORYRETRIEVAL=1|0`.
+
+## Dashboard and API surfaces
+
+The classic Dashboard **Inbox** page combines Quick Notes, Team Notes,
+Knowledge Base, the manual KB sweep, and **Memory search**; pinned context is
+managed from the Home page. Structured search operates across all scopes and can
+inspect each lifecycle status.
+
+| Method | Path | Role |
+|--------|------|------|
+| `POST` | `/api/notes` | Create an inbox note for consolidation. |
+| `GET` / `POST` | `/api/pinned` | Read or add pinned context. |
+| `POST` | `/api/pinned/update` | Replace a pinned entry. |
+| `POST` | `/api/pinned/remove` | Remove a pinned entry. |
+| `GET` | `/api/knowledge` | List readable knowledge entries. |
+| `POST` | `/api/knowledge` | Create a human-authored KB file and SQL record. |
+| `GET` | `/api/knowledge/<category>/<file>` | Read one KB file. |
+| `POST` | `/api/knowledge/sweep` | Start the asynchronous KB sweep. |
+| `GET` | `/api/knowledge/sweep/status` | Poll sweep status. |
+| `GET` | `/api/memory-records/search?q=...&status=active&limit=20` | Search structured records with provenance. |
+| `POST` | `/api/memory-records/<id>/action` | Apply `pin`, `retract`, or `restore`. |
+| `POST` | `/api/inbox/persist` | Copy an inbox excerpt into `notes.md`, then archive the source. |
+| `POST` | `/api/inbox/promote-kb` | Move an inbox item into a selected KB category. |
+
+A structured-record `pin` is not the same as a pinned note. It changes that SQL
+row to human trust, importance `1`, confidence of at least `0.9`, and active
+status. It does not write `pinned.md`, bypass lexical matching, or guarantee
+selection within the top-K/budget.
+
+## Operations and troubleshooting
+
+### Inspect the active surfaces
+
+```bash
+curl http://localhost:7331/api/pinned
+curl http://localhost:7331/api/knowledge
+curl "http://localhost:7331/api/memory-records/search?q=worktree&status=active&limit=20"
+```
 
-Why:
+Use `GET /api/routes` to confirm the live route catalog.
 
-- The sweep classifies inbox notes into the right category, generates the canonical filename, and updates the `notes.md` summary in lockstep with the `knowledge/` write. Manual edits skip those side effects and desync the system.
-- Agents work in parallel. Letting any agent rewrite `knowledge/` entries makes consolidation non-deterministic and makes blame attribution impossible.
-- Knowledge base entries are referenced by date-stamped path. Rewriting an entry breaks every existing link.
+### Expected legacy fallback
 
-If an agent thinks a `knowledge/` file is wrong, the correct response is to **note the disagreement in the agent's own inbox file**. The sweep will pick up the note on the next consolidation cycle and either supersede or correct the existing entry.
+Seeing **Team Notes** and **Personal Memory** instead of **Relevant Memory** is
+expected when the feature is off, shadow mode is on, the task query has no
+indexable terms, FTS5 finds no eligible rows, or retrieval/telemetry throws.
+Check engine logs for `Memory retrieval failed ...; using legacy memory`.
 
-The same constraint applies to `knowledge/agents/<agentId>.md` — those are curated by the sweep and should not be hand-edited.
+### A note remains in the inbox
 
-## Session State vs. Persistent Memory
+Check:
 
-The PRD that introduced sliding-window memory (W-mq07b8do000nc86a) referenced two distinct write paths — `update_session_state()` and `update_memory()` — borrowed from agent frameworks that model agents as long-lived in-process objects. Minions has neither method because it doesn't model agents that way; understanding the mapping prevents fruitless searches for non-existent APIs.
+1. The eligible file count has reached `engine.inboxConsolidateThreshold`.
+2. The note is not pinned against consolidation.
+3. A prior consolidation is not still in flight.
+4. `notes.md` and its lock are writable.
+5. For a configured agent, the personal-memory write did not fail and defer
+   archiving.
 
-**Session state** = the dispatch's worktree + child process. Each Minions agent runs as a fresh OS process spawned by `engine.js → engine/spawn-agent.js` inside a per-work-item git worktree (`work/<wi-id>` by default; see `shared.deriveWorkItemBranchName`). When the dispatch ends, the engine deletes the worktree and the child exits. There is no persistent "session" object to update — ephemeral state lives in process memory and disk paths under the worktree, both of which are reclaimed automatically. No code is needed to "clear" session state; it never persists in the first place.
+An LLM failure alone should not strand the batch; the regex fallback handles it.
 
-**Persistent memory** = `knowledge/agents/<agentId>.md`, the per-agent file appended to by `engine/consolidation.js` during the inbox sweep. This is the analog of `update_memory()` in PRD terms. It is written only by the consolidation sweep (the [sweep-write-only constraint](#sweep-write-only-constraint) applies), is injected into every subsequent dispatch's prompt for that same agent ID via `engine/playbook.js`, and is bounded by two complementary cuts plus an optional summary pass:
+### Personal memory is missing
 
-| Tunable (under `engine.*` in `config.json`)             | Default | Behavior                                                                                                          |
-|---------------------------------------------------------|---------|-------------------------------------------------------------------------------------------------------------------|
-| `agentMemoryMaxEntries`                                 | `300`   | Sliding-window entry-count cap. Older non-summary sections evicted oldest-first when exceeded.                    |
-| (built-in) `AGENT_MEMORY_BUDGET_BYTES`                  | `25000` | Hard byte ceiling for prompt-injection safety. Always wins when both caps bind. Sticky summary sections obey it.  |
-| `agentMemorySummaryEnabled`                             | `false` | Master switch for the LLM-driven compression pass. Off by default to avoid surprise Haiku spend.                  |
-| `agentMemorySummaryThreshold`                           | `30`    | When the summary pass fires, fold this many oldest entries into one summary section.                              |
-| `agentMemorySummaryDays`                                | `30`    | Age trigger: if the oldest entry is older than this, fold even when under the entry cap.                          |
+Confirm that `agent:` exactly matches a configured `config.agents` key, or that
+the filename starts with that id. `temp-*`, unknown, invalid, and empty authors
+are intentionally skipped. Also remember that successful active retrieval
+suppresses the broad personal-file appendix; the relevant agent-scoped record
+may instead appear inside **Relevant Memory**.
 
-The summary pass runs fire-and-forget after every successful `appendToAgentMemory` write inside `classifyToKnowledgeBase`. It re-reads outside the lock, calls Haiku, then re-acquires the lock and verifies the same oldest sections are still in place (stale-candidate guard) before swapping. Any failure — disabled, no trigger, LLM unavailable, race detected — is a silent no-op; the consolidation pipeline is never blocked on the LLM.
+### Search misses known Markdown
 
-The compressed summary is wrapped in an `<UNTRUSTED-INPUT source="agent-memory-summary:agent=...">` fence on disk. The source material was the inbox bodies of evicted entries, which are themselves untrusted; without the fence, any imperative laundered through summarization could later be executed by an agent reading its own memory.
+Structured search is lexical FTS5, not semantic embeddings. It ignores common
+task stop words, searches only the requested lifecycle status, and dispatch
+retrieval uses exact project/agent scope keys. More importantly, out-of-band
+file edits and file-only mutation paths are not automatically reindexed. Compare
+the record's `sourcePath`/`sourceRef` with the current file before trusting it.
 
-Summary sections are **sticky** under the entry-count cap — they represent compressed knowledge that should outlive ordinary inbox entries. They are detected by their title prefix `Earlier learnings summary` and only the byte budget can evict them.
+Do not repair divergence by editing `engine/state.db` directly. Use the supported
+dashboard/API ingestion paths or fix the originating writer so provenance and
+events remain intact.
 
-**Default-off rationale.** `agentMemorySummaryEnabled` defaults to `false` (intentional deviation from PRD wording that implies "always on"). Enabling it commits operators to per-agent Haiku spend on every consolidation cycle; the entry-count cap on its own already prevents unbounded growth. Operators who have weighed the cost set `engine.agentMemorySummaryEnabled: true` in `config.json` to opt in.
+## Agent read and write rules
 
-## Quick reference for agents
+The shared playbook's on-disk lookup order remains:
 
-```
-1. pinned.md           ← critical, human-flagged. Read first.
-2. knowledge/          ← categorized KB. Read, never write.
-3. notes.md            ← consolidated team notes. Read.
-4. notes/inbox/        ← fresh, unconsolidated findings. Read; write one on success.
+```text
+1. pinned.md          - critical operator context; read first
+2. knowledge/         - categorized KB; read, never write
+3. notes.md           - consolidated team digest
+4. notes/inbox/       - newest unconsolidated findings
+5. related live output and prior work-item summaries
+6. external research
 ```
 
-After the four tiers, optionally consult `agents/*/live-output.log` and prior `resultSummary` fields. Only then go outside.
+This lookup discipline complements prompt retrieval; it does not mean all four
+Markdown tiers are automatically copied into the prompt.
+
+## Implementation map
+
+- [`engine/consolidation.js`](../engine/consolidation.js) - inbox digest,
+  classification, personal routing, pruning, reconciliation, and summary.
+- [`engine/db/migrations/017-agent-memory.js`](../engine/db/migrations/017-agent-memory.js)
+  - schema and one-time Markdown backfill.
+- [`engine/memory-store.js`](../engine/memory-store.js) - record validation,
+  lifecycle, FTS5 query, and telemetry persistence.
+- [`engine/memory-retrieval.js`](../engine/memory-retrieval.js) - task query,
+  reranking, deduplication, provenance formatting, and byte packing.
+- [`engine/playbook.js`](../engine/playbook.js) - pinned/retrieved/legacy prompt
+  assembly and fallback.
+- [`engine/lifecycle.js`](../engine/lifecycle.js) - episodic completion capture.
+- [`engine/untrusted-fence.js`](../engine/untrusted-fence.js) - provenance fence
+  and UTF-8-safe truncation.
+- [`dashboard.js`](../dashboard.js) and
+  [`dashboard/js/memory-search.js`](../dashboard/js/memory-search.js) -
+  operator APIs and UI.
 
 ## See also
 
-- `playbooks/shared-rules.md` — the canonical instruction injected into every dispatch.
-- `engine/playbook.js` — where `pinned.md`, `notes.md`, and per-agent memory get spliced into prompts.
-- `engine/consolidation.js` — the sweep that promotes inbox notes into `knowledge/` and `notes.md`.
+- [kb-sweep.md](kb-sweep.md) - curation and retention of readable KB files.
+- [named-agents.md](named-agents.md) - agent configuration and routing.
+- [security.md](security.md) - trust boundaries and prompt-injection policy.
+- [self-improvement.md](self-improvement.md) - memory within the broader
+  feedback loop.