@yandy0725/pi-memory 1.4.0 → 2.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (23) hide show
  1. package/README.md +245 -140
  2. package/README.zh.md +264 -157
  3. package/index.ts +460 -117
  4. package/package.json +3 -3
  5. package/src/agent-runner.ts +25 -9
  6. package/src/config.ts +81 -6
  7. package/src/dream.ts +99 -59
  8. package/src/entry-file.ts +81 -0
  9. package/src/entry-index.ts +112 -0
  10. package/src/extract.ts +378 -60
  11. package/src/filename.ts +48 -0
  12. package/src/fs-lock.ts +251 -0
  13. package/src/index-source.ts +186 -0
  14. package/src/inject.ts +129 -77
  15. package/src/memory-store.ts +496 -0
  16. package/src/memory-tool.ts +263 -328
  17. package/src/model-resolver.ts +1 -1
  18. package/src/paths.ts +1 -14
  19. package/src/process-lock.ts +122 -0
  20. package/src/sanitize.ts +36 -0
  21. package/src/snapshot.ts +68 -0
  22. package/src/index-file.ts +0 -91
  23. package/src/topic-file.ts +0 -119
package/README.md CHANGED
@@ -1,24 +1,39 @@
1
1
  # pi-memory
2
2
 
3
- File-system driven persistent memory layer for pi coding agent. Stores project knowledge across sessions — facts, preferences, debugging history — in plain Markdown files under `~/.pi/memory/<git|local>/<project>/`.
4
-
5
- Aligned with Claude Code's auto memory mechanism: per-topic MEMORY.md index, automatic topic file surfacing based on relevance, per-turn memory extraction, and typed memory categories.
3
+ File-system driven persistent memory layer for pi coding agent. Stores project knowledge across sessions — facts, preferences, debugging history — as plain Markdown files under `~/.pi/memory/<git|local>/<project>/`.
4
+
5
+ Aligned with Claude Code's auto memory mechanism: **one memory = one file**, a `MEMORY.md` index with exactly one line per memory, relevance-based auto-surfacing, per-turn memory extraction, and typed memory categories.
6
+
7
+ > ## ⚠️ Breaking changes
8
+ >
9
+ > **In 2.1.0:**
10
+ >
11
+ > - **Models must be configured explicitly.** There is no shipped default and no parent-model fallback: `defaults.model` (or a per-task `model`) must exist and be resolvable, or `session_start` reports a config error and initialises **nothing**. See [Model configuration](#model-configuration).
12
+ > - **`/memory on` and `/memory off` are gone.** `enabled` is a `memory.json` switch read once at session start — changing it needs a session restart.
13
+ > - **Automatic 1.x → 2.0 migration has been removed.** Legacy topic files stay on disk untouched but are **invisible** to the memory system (they fail `parseEntryFile`'s five-field v2 frontmatter check). See [1.x data](#1x-data).
14
+ >
15
+ > **In 2.0.0:**
16
+ >
17
+ > - The index is now **one line per memory** (1.x had one line per *topic file*, with many `## entries` inside it).
18
+ > - Each memory lives in **its own file** with five frontmatter fields: `name`, `description`, `type`, `created`, `modified` (1.x used `updated`).
19
+ > - The index is injected as a **system-prompt section** (`memory_index`) that is frozen for the whole session, instead of being appended to the system prompt string.
6
20
 
7
21
  ## Features
8
22
 
9
- - **One `memory` tool**, three actions: `add` (append entry), `remove` (delete entry by title), `search` (query memory or session history)
10
- - **Topic-based file organization**: each `memory add` writes a `## entry` block to a named `.md` file under the project's memory directory
11
- - **`MEMORY.md` index** — one compact line per topic file with a relevance hook: `- [Name](file.md) — summary`
12
- - **Memory types**: four categories — `user`, `feedback` (default), `project`, `reference` — stored in topic file frontmatter
13
- - **Auto-surfacing** ⭐: on every user message, a side-query LLM selects up to N relevant topic files and injects their content into the agent's context. No manual `read` needed — use the built-in `read` tool to inspect memory files. Session-level deduplication prevents re-injecting the same topic.
14
- - **Extract memories** ⭐: after each agent run, an async subagent analyzes the conversation and automatically writes learnings to memory — preferences, conventions, debugging insights
15
- - **Snapshot injection**: on every new session, the MEMORY.md index is appended to the system prompt, keeping the agent aware of past work
16
- - **`/dream` command**: launches a headless agent with a four-phase consolidation (Orient → Gather → Consolidate → Prune) to deduplicate, merge, and rebuild all memory files
17
- - **Dream nudge**: after N sessions or N hours, a gentle notification suggests running `/dream`
18
- - **`/memory` command**: show status, toggle on/off, inspect index and topic files
19
- - **Session search**: the `memory search scope=sessions` action queries past conversation history
20
- - **Readable, clone-safe layout**: memory lives under `~/.pi/memory/git/<host__owner__repo>/` for git repos with an http(s)/ssh/git remote (including scp-style and `git+ssh`/`git+https`), and `~/.pi/memory/local/<absolute-path>/` otherwise — clones and worktrees of the same repo share memory (a fork has its own remote, so it gets its own directory)
21
- - **Path traversal protection**: topic files are validated against escaping the memory directory
23
+ - **One `memory` tool, five actions** for the main agent: `add`, `replace`, `remove`, `list`, `search`. Two more — `rename` and `rebuild_index` — exist **only inside `/dream`'s own headless session** and never appear in the main agent's or the extractor's schema.
24
+ - **One memory = one file** — no more multi-entry `## section` blocks; `name` is the lookup key, adding an existing `name` overwrites it (idempotent).
25
+ - **`MEMORY.md` index** — exactly one line per memory: `- [Name](file.md) — description`. The separator is an **em dash** (`—`, U+2014) with one space on each side.
26
+ - **`memory_index` prompt section, frozen per session** ⭐ — the index goes into `event.systemPromptOptions.sections["memory_index"]` and its value **does not change for the rest of the session**; only compaction re-reads it from disk. Because pi diffs sections and appends nothing when they are unchanged, the system prompt stays byte-identical turn after turn and the provider's prefix cache keeps hitting. `resume` / `fork` / `reload` replay the **recorded** value from the transcript instead of reading disk, so restoring a session does not rewrite its head.
27
+ - **Injection sanitising** — everything injected (index lines, surfaced entry bodies and names) has invisible/bidi characters stripped and `<` `>` escaped, so a memory can never forge `</relevant_memories>`, `<system>`, `<project_instructions>`, `<active_agent …>` or `<memory_index>`. Sanitising happens **at injection time only**: your files on disk are never rewritten (they stay readable and hand-editable).
28
+ - **Auto-surfacing** ⭐ — on every user turn a lightweight side query selects up to `maxFiles` **entries** (selected from `description` alone) and injects their bodies inside `<relevant_memories>`. Already-injected files are deduplicated per session; the manifest is served from an in-process `mtime` cache, so a turn costs one `readdir` plus one `stat` per file. Disabled inside subagents.
29
+ - **Extract memories** ⭐ — after each run an async headless agent receives a **structured rendering of the whole conversation** (every user message in full, assistant text and tool calls, tool results with error flags), not just two messages. It writes through the same `memory` primitives, under a whole-round logical lock it never waits for: if a dream is running, that turn is simply skipped.
30
+ - **`/dream`** — a headless consolidation agent (Orient → Gather Signal → Consolidate → Prune & Index) that merges duplicates, resolves contradictions, renames entries and rebuilds the index. It has **no raw file access**: it only gets the seven `memory` actions, holds the logical lock for the whole round, and snapshots the entire directory on entry.
31
+ - **Dream nudge** — after N sessions or N hours a notification suggests `/dream`.
32
+ - **`/memory`** — full status (switch, directory, index capacity, entry count, last dream, lock state including the holder), plus `unlock`.
33
+ - **Two-level locking** — an in-process logical lock carries the *logical* scope (one primitive call, or a whole dream round); the cross-process `.lock` file is held for **milliseconds only** and is **never reclaimed automatically**. There is no TTL, no heartbeat and no takeover, so mutual exclusion is a hard guarantee; the price is that a lock left behind by a crashed process must be removed by a human (`/memory unlock`).
34
+ - **Snapshots** — every write leaves a rollback point under `.backups/<ts>-<label>/`, keeping the last `lock.snapshotKeep` (directories named `migrate-*` — whole-directory snapshots from an earlier 1.x migration, whose `originals/` subdirectory holds the pre-2.0 topic files — are never pruned). `/dream` is the exception: it snapshots the whole directory **once on entry**, and the primitives inside that round skip their per-file snapshots (one round, one rollback point).
35
+ - **Session search** — `memory search scope=sessions` queries past conversation history.
36
+ - **Readable, clone-safe layout** — memory lives under `~/.pi/memory/git/<host__owner__repo>/` for git repos with an http(s)/ssh/git remote (including scp-style and `git+ssh`/`git+https`), and `~/.pi/memory/local/<absolute-path>/` otherwise — clones and worktrees of the same repo share memory (a fork has its own remote, so it gets its own directory).
22
37
 
23
38
  ## Install
24
39
 
@@ -34,9 +49,70 @@ Or add to `~/.pi/agent/settings.json`:
34
49
  }
35
50
  ```
36
51
 
52
+ ## Storage layout
53
+
54
+ ```
55
+ ~/.pi/memory/git/github.com__owner__repo/
56
+ MEMORY.md — the index: one line per memory (em dash separator)
57
+ SSH-port-on-staging.md
58
+ Test-command.md — one file per memory
59
+ .lock — cross-process write lock (held for milliseconds, never auto-reclaimed)
60
+ .backups/ — rollback points: <ISO-ts>-<label>/ (plus migrate-<ts>/ whole-directory snapshots from earlier 1.x migrations)
61
+ .dream-meta.json — last dream timestamp + session count (drives the nudge)
62
+ sessions/ — persisted headless sessions, only when sessionPersistence is enabled
63
+ ```
64
+
65
+ ### Entry file
66
+
67
+ ```yaml
68
+ ---
69
+ name: SSH port on staging
70
+ description: staging SSH listens on 2222, not 22; key at ~/.ssh/staging
71
+ type: project
72
+ created: 2026-07-13
73
+ modified: 2026-10-02T08:14:03.120Z
74
+ ---
75
+
76
+ staging 的 SSH 用 2222 端口，密钥在 ~/.ssh/staging。
77
+ ```
78
+
79
+ - `name` — unique, human-readable title; the lookup key for `replace` / `remove` / `rename`. Adding the same `name` again overwrites that memory instead of creating a second one.
80
+ - `description` — **one self-contained line**. It is the only text a future session sees when the side query decides relevance, so it must make sense without the body. Bad: `Debugging tips`. Good: `staging SSH listens on 2222, not 22`.
81
+ - `type` — `user` | `feedback` (default) | `project` | `reference`.
82
+ - `created` — `YYYY-MM-DD`, written once and preserved across overwrites.
83
+ - `modified` — ISO 8601, **always written by the store**; callers cannot pass it in.
84
+
85
+ File names are derived from `name` (unsafe characters replaced, 100-byte cap, `-2` / `-3` … only when a *file name* collides). `MEMORY.md` can never be targeted by an entry.
86
+
87
+ ### MEMORY.md index
88
+
89
+ ```
90
+ # Memory Index
91
+
92
+ - [SSH port on staging](SSH-port-on-staging.md) — staging SSH listens on 2222, not 22
93
+ - [Test command](Test-command.md) — run npm test, not npm run test
94
+ ```
95
+
96
+ Writes are **surgical**: only the target line changes, hand-written headings, groups and comments are preserved byte-for-byte, and line order is stable. The one exception is line endings: CRLF (or lone CR) is normalised to LF before parsing, so the first write to a CRLF file rewrites it with LF.
97
+
98
+ ### Memory types
99
+
100
+ | Type | Meaning | Example |
101
+ |------|---------|---------|
102
+ | `user` | User role, preferences, knowledge | "User is a data scientist focused on observability" |
103
+ | `feedback` | Lessons, corrections, confirmations (default) | "Use real DB not mocks — burned last quarter" |
104
+ | `project` | Project state, deadlines, incidents | "Merge freeze starts 2026-03-05 for mobile release" |
105
+ | `reference` | Pointers to external systems | "Bug tracker = Linear INGEST project" |
106
+
107
+ ### Capacity: 200 index lines ≈ 199 memories
108
+
109
+ The index holds at most `memIndexMaxLines` (200) non-empty lines and `memIndexMaxBytes` (25600) bytes. Those 200 lines are **index lines, not memories**: `rebuildIndex` guarantees at least one header line — an existing hand-written header is kept verbatim (trailing blank lines before the first entry are dropped), otherwise it writes `# Memory Index` — and hand-written headings, groups and comments count too. A rebuilt index therefore holds at most about **199 memories per project directory** (fewer if you keep hand-written headings). Exceeding the limit does **not** fail the write: the write succeeds and the tool returns an actionable warning telling the model to merge or drop entries (everything past the limit is invisible on the next load).
110
+
111
+ This is why `/dream` is no longer optional housekeeping — it is **capacity management**. Run it (or accept the nudge) before you approach 199 memories.
112
+
37
113
  ## Configuration
38
114
 
39
- Create `memory.json` in the agent directory (`~/.pi/agent/memory.json`) or the project `.pi/` directory (if trusted):
115
+ Create `memory.json` in the agent directory (`~/.pi/agent/memory.json`) or the project `.pi/` directory (only when the project is trusted):
40
116
 
41
117
  ```json
42
118
  {
@@ -44,189 +120,213 @@ Create `memory.json` in the agent directory (`~/.pi/agent/memory.json`) or the p
44
120
  "memoryDir": "~/.pi/memory",
45
121
  "memIndexMaxLines": 200,
46
122
  "memIndexMaxBytes": 25600,
47
- "defaults": {
48
- "model": "deepseek/deepseek-v4-flash",
49
- "sessionPersistence": { "enabled": false }
50
- },
51
- "dream": {
52
- "nudgeAfterSessions": 5,
53
- "nudgeAfterHours": 24,
54
- "model": "auto",
55
- "thinkLevel": "high",
56
- "sessionPersistence": { "enabled": false }
57
- },
58
- "sessionSearch": {
59
- "maxSessions": 10,
60
- "maxMatches": 5
61
- },
123
+ "memIndexInjectMaxLines": 200,
124
+ "memIndexInjectMaxBytes": 25600,
125
+ "lock": { "timeoutMs": 5000, "snapshotKeep": 5 },
126
+ "defaults": { "model": "provider/model-id", "sessionPersistence": { "enabled": false } },
127
+ "dream": { "nudgeAfterSessions": 5, "nudgeAfterHours": 24, "thinkLevel": "high" },
128
+ "sessionSearch": { "maxSessions": 10, "maxMatches": 5 },
62
129
  "autoSurfacing": {
63
130
  "enabled": true,
64
- "model": "auto",
65
131
  "thinkLevel": "off",
66
- "maxFiles": 5,
67
- "maxTopicBytes": 4096,
68
- "maxInjectionBytes": 20480,
69
- "sessionPersistence": { "enabled": false }
132
+ "maxFiles": 3,
133
+ "maxEntryBytes": 3072,
134
+ "maxInjectionBytes": 10240
70
135
  },
71
136
  "extractMemories": {
72
137
  "enabled": true,
73
- "model": "auto",
74
138
  "thinkLevel": "high",
75
139
  "maxContextTokens": 2000,
76
- "sessionPersistence": { "enabled": false }
140
+ "maxToolResultChars": 500,
141
+ "maxAssistantChars": 2000
77
142
  }
78
143
  }
79
144
  ```
80
145
 
146
+ > Every `model` value must resolve in your registry — there is no default. A missing or unresolvable model makes `session_start` report a config error and initialise nothing. See [Model configuration](#model-configuration).
147
+
81
148
  | Key | Default | Description |
82
149
  |-----|---------|-------------|
83
- | `enabled` | `true` | Toggle the entire memory system on/off |
150
+ | `enabled` | `true` | Toggle the entire memory system on/off. Read once at session start — changing it requires restarting the session |
84
151
  | `memoryDir` | `~/.pi/memory` | Root directory for all memory data |
85
- | `memIndexMaxLines` | `200` | Max lines in `MEMORY.md` before capacity errors |
86
- | `memIndexMaxBytes` | `25600` | Max bytes in `MEMORY.md` before capacity errors |
87
- | `defaults.model` | — | Shared model fallback for all sub-tasks. Per-task `model` overrides |
88
- | `defaults.sessionPersistence.enabled` | `false` | Shared session persistence fallback (default: in-memory). Per-task overrides |
89
- | `defaults.sessionPersistence.sessionDir` | `<project memory dir>/sessions/` | Custom session directory for persisted headless agent sessions |
90
- | `dream.nudgeAfterSessions` | `5` | Sessions since last dream before nudge is shown |
91
- | `dream.nudgeAfterHours` | `24` | Hours since last dream before nudge is shown |
92
- | `dream.model` | — | Model for dream consolidation (`"provider/id"`). Falls back to `defaults.model` → parent model |
93
- | `dream.thinkLevel` | `"high"` | Thinking effort for dream subagent: `"off"`, `"minimal"`, `"low"`, `"medium"`, `"high"`, `"xhigh"` |
94
- | `dream.sessionPersistence.enabled` | `false` | Persist dream agent sessions to disk (debug/audit). Falls back to `defaults.sessionPersistence.enabled` |
95
- | `dream.sessionPersistence.sessionDir` | `<project memory dir>/sessions/` | Custom session directory for dream sessions |
96
- | `sessionSearch.maxSessions` | `10` | Max sessions to scan when searching history |
152
+ | `memIndexMaxLines` | `200` | Write capacity: max non-empty lines in `MEMORY.md` (the `# Memory Index` header and hand-written headings count too, so this is not exactly the memory count) |
153
+ | `memIndexMaxBytes` | `25600` | Write capacity: max bytes of `MEMORY.md` |
154
+ | `memIndexInjectMaxLines` | `200` | Injection budget: max lines of the index put into the `memory_index` section. Same scale as the write capacity on purpose — a smaller budget would hide memories that were written successfully |
155
+ | `memIndexInjectMaxBytes` | `25600` | Injection budget: max bytes of the index section (truncated with a `[truncated: …]` marker) |
156
+ | `lock.timeoutMs` | `5000` | How long a write waits for the logical lock (single primitive) or the cross-process `.lock`. Also the upper bound `session_shutdown` waits for in-flight writes |
157
+ | `lock.snapshotKeep` | `5` | Rollback points kept in `.backups/` (directories named `migrate-*` — whole-directory snapshots from an earlier 1.x migration, whose `originals/` subdirectory holds the pre-2.0 topic files — are never pruned) |
158
+ | `defaults.model` | `— (required)` | Shared model for dream / extract / side query. **No default**: every task that will run must resolve a model, otherwise `session_start` fails (see [Model configuration](#model-configuration)). A per-task `model` overrides it |
159
+ | `defaults.sessionPersistence.enabled` | `false` | Shared fallback: headless sub-sessions (extract / dream / side query) stay in memory by default |
160
+ | `defaults.sessionPersistence.sessionDir` | `<project memory dir>/sessions/` | Custom directory for persisted headless sessions |
161
+ | `dream.nudgeAfterSessions` | `5` | Sessions since the last dream before the nudge is shown |
162
+ | `dream.nudgeAfterHours` | `24` | Hours since the last dream before the nudge is shown |
163
+ | `dream.model` | — | Model for dream consolidation (`"provider/id"`). Falls back to `defaults.model`; required unless `defaults.model` is set (must be resolvable, no parent-model fallback) |
164
+ | `dream.thinkLevel` | `"high"` | Thinking effort for the dream agent: `off` / `minimal` / `low` / `medium` / `high` / `xhigh` |
165
+ | `dream.sessionPersistence.*` | inherits `defaults` | Persist dream sessions to disk (debug/audit) |
166
+ | `sessionSearch.maxSessions` | `10` | Max sessions to scan for `search scope=sessions` |
97
167
  | `sessionSearch.maxMatches` | `5` | Max matches to return from history search |
98
- | `autoSurfacing.enabled` | `true` | ⭐ Enable per-turn topic file auto-injection |
99
- | `autoSurfacing.model` | — | ⭐ Model for side-query relevance selection. Falls back to `defaults.model` → parent model |
100
- | `autoSurfacing.thinkLevel` | `"off"` | ⭐ Thinking effort for side-query (recommended: `"off"` for lightweight selection) |
101
- | `autoSurfacing.sessionPersistence.enabled` | `false` | Persist side-query agent sessions to disk. Falls back to `defaults.sessionPersistence.enabled` |
102
- | `autoSurfacing.maxFiles` | `5` | ⭐ Max topic files to inject per turn |
103
- | `autoSurfacing.maxTopicBytes` | `4096` | ⭐ Max bytes per injected topic file (truncated) |
104
- | `autoSurfacing.maxInjectionBytes` | `20480` | ⭐ Max total bytes of injected content per turn |
168
+ | `autoSurfacing.enabled` | `true` | ⭐ Enable per-turn entry auto-injection |
169
+ | `autoSurfacing.model` | — | ⭐ Model for the relevance side query. Falls back to `defaults.model`; required unless `defaults.model` is set (must be resolvable, no parent-model fallback) |
170
+ | `autoSurfacing.thinkLevel` | `"off"` | ⭐ Thinking effort for the side query (`"off"` keeps it cheap) |
171
+ | `autoSurfacing.maxFiles` | `3` | ⭐ Max entries to inject per turn |
172
+ | `autoSurfacing.maxEntryBytes` | `3072` | ⭐ Max bytes of a single injected entry body (truncated). Replaces 1.x's `maxTopicBytes`, which is ignored |
173
+ | `autoSurfacing.maxInjectionBytes` | `10240` | ⭐ Max total bytes of injected content per turn |
174
+ | `autoSurfacing.sessionPersistence.*` | inherits `defaults` | Persist side-query sessions to disk |
105
175
  | `extractMemories.enabled` | `true` | ⭐ Enable per-turn memory extraction |
106
- | `extractMemories.model` | — | ⭐ Model for the extraction subagent. Falls back to `defaults.model` → parent model |
107
- | `extractMemories.thinkLevel` | `"high"` | ⭐ Thinking effort for extraction: `"off"` / `"minimal"` / `"low"` / `"medium"` / `"high"` / `"xhigh"` |
108
- | `extractMemories.sessionPersistence.enabled` | `false` | Persist extract agent sessions to disk. Falls back to `defaults.sessionPersistence.enabled` |
109
- | `extractMemories.maxContextTokens` | `2000` | ⭐ Max tokens of conversation to analyze |
176
+ | `extractMemories.model` | — | ⭐ Model for the extraction agent. Falls back to `defaults.model`; required unless `defaults.model` is set (must be resolvable, no parent-model fallback) |
177
+ | `extractMemories.thinkLevel` | `"high"` | ⭐ Thinking effort for extraction |
178
+ | `extractMemories.maxContextTokens` | `2000` | ⭐ Budget for the rendered conversation (`× 4` characters; the middle is trimmed first, head and tail are kept, user messages are dropped last) |
179
+ | `extractMemories.maxToolResultChars` | `500` | ⭐ Per-message cap for a rendered `tool_result` |
180
+ | `extractMemories.maxAssistantChars` | `2000` | ⭐ Per-message cap for rendered assistant text (user messages are never truncated) |
181
+ | `extractMemories.sessionPersistence.*` | inherits `defaults` | Persist extract sessions to disk |
110
182
 
111
- Persisted headless agent sessions default to `<project memory dir>/sessions/` — inside the project's memory directory, not inside your working copy. For example `~/.pi/memory/git/github.com__owner__repo/sessions/`.
183
+ Persisted headless sessions default to `<project memory dir>/sessions/` — inside the project's memory directory, not inside your working copy.
112
184
 
113
- Project-level config (`.pi/memory.json`) is only loaded when the project is trusted.
185
+ ## Model configuration
114
186
 
115
- ## How it works
187
+ Every task that will run must resolve a model — **there is no shipped default and no parent-model fallback**. `defaults.model` satisfies all of them; a per-task `model` (`dream.model`, `extractMemories.model`, `autoSurfacing.model`) overrides it.
116
188
 
117
- ### MEMORY.md index
189
+ | Task | Required when |
190
+ |------|---------------|
191
+ | `dream` | the memory system is enabled (`enabled: true`) — always required |
192
+ | `extractMemories` | `extractMemories.enabled` is true |
193
+ | `autoSurfacing` | `autoSurfacing.enabled` is true |
118
194
 
119
- MEMORY.md is a compact **pointer index** — one line per topic file, not per entry:
195
+ With `enabled: false` nothing runs — not even `/dream` or the nudge — so no model is required. At `session_start` pi-memory resolves every required model against the model registry. If one is missing or cannot be resolved, it initialises **nothing**: it shows an error notification `pi-memory config error:` followed by one `- <error>` line per problem, and `/memory` reports `Memory: misconfigured` and `Dir: not initialized`, followed by the same lines. The two possible messages are:
120
196
 
121
- ```
122
- - [Debugging](debugging.md) — SSH uses port 2222; MySQL 30s timeout on staging
123
- - [API Conventions](api.md) — REST handlers in src/api/handlers/; standard error format
124
- ```
197
+ - `no model for <task> — set "<task>.model" or "defaults.model" in memory.json`
198
+ - `model "<value>" for <task> is not resolvable (unknown id or missing credentials)`
125
199
 
126
- Only the index is injected into the system prompt on every session (first 200 lines / 25KB). Topic file content is **not** loaded at session start — it's surfaced on demand via auto-surfacing or the built-in `read` tool.
200
+ Fix `memory.json` and restart the session — the config is read once at session start. In headless/print sessions a config error is silent (no notification is shown), so check `/memory` in an interactive session.
127
201
 
128
- ### Topic file format
202
+ ## How it works
129
203
 
130
- Each topic file uses YAML frontmatter with four fields:
204
+ ### Session lifecycle
131
205
 
132
- ```yaml
133
- ---
134
- name: Debugging Tips
135
- description: Common debugging patterns, SSH ports, MySQL timeout configs
136
- type: feedback
137
- updated: 2026-07-13
138
- ---
206
+ | Event | What pi-memory does |
207
+ |---|---|
208
+ | `session_start` | Load config → validate the required models (a failure means nothing is initialised) → resolve the memory directory → **pick the index value and freeze it** (disk for `startup`/`new`; the recorded transcript value for `resume`/`fork`/`reload`) → register the `memory` tool (once, five actions) → rebuild the manifest cache → dream nudge |
209
+ | `before_agent_start` | Write the frozen value into `sections["memory_index"]` (**unconditionally, every turn**), then auto-surfacing (main session, not a subagent) |
210
+ | `agent_end` | Fire the async extractor; notify `Extracted N memories.` when it wrote something, or `Extract failed: …` once per session |
211
+ | `session_compact` | Clear the injected-file set **and re-read the index from disk** — the only in-session refresh point |
212
+ | `session_shutdown` | Wait for in-flight writes to finish (bounded by `lock.timeoutMs`) so a quit does not leave a stale `.lock` |
139
213
 
140
- ## SSH Gotcha
141
- staging uses port 2222, key at ~/.ssh/staging
214
+ ### Why the index is frozen
142
215
 
143
- ## MySQL Timeout
144
- connection timeout after 30s on staging
145
- ```
216
+ pi builds the system prompt from ordered sections and appends a *patch* message only for sections whose value changed; when nothing changed it appends nothing at all. A custom section is inserted **last**, so `memory_index` is the final part of the system prompt with the entire conversation after it. If its value changed mid-session, the folded head would be rewritten and everything after it would lose its prefix cache. Hence: one value per session, refreshed only at compaction (where the middle of the conversation is being rewritten anyway).
146
217
 
147
- The `description` field is critical — the auto-surfacing side-query uses it to determine relevance. Make it specific.
218
+ The accepted trade-off: memories written during this session are **not** visible in this session's index. The tool result already confirms the write, and the next session sees it.
148
219
 
149
- ### Memory types
150
-
151
- | Type | Meaning | Example |
152
- |------|---------|---------|
153
- | `user` | User role, preferences, knowledge | "User is a data scientist focused on observability" |
154
- | `feedback` | Lessons, corrections, confirmations (default) | "Use real DB not mocks — burned last quarter" |
155
- | `project` | Project state, deadlines, incidents | "Merge freeze starts 2026-03-05 for mobile release" |
156
- | `reference` | Pointers to external systems | "Bug tracker = Linear INGEST project" |
220
+ If the host pi is older than the sections API, pi-memory falls back to appending the index to the system prompt string — same functionality, worse caching.
157
221
 
158
222
  ### Auto-surfacing
159
223
 
160
- On every user message (`before_agent_start` hook):
161
- 1. Scan all topic files, extract their frontmatter metadata
162
- 2. A side-query LLM selects up to `maxFiles` relevant topic files based on the user's query
163
- 3. Already-injected topics are skipped (session-level dedup)
164
- 4. Selected topic file content is injected as context — the agent sees relevant memories automatically
224
+ 1. Build the manifest from the store's `mtime` cache: `[type] file.md — description`, newest first, capped at 200 entries and 4000 characters (the description budget is split evenly, never below 80 characters each).
225
+ 2. A side query (`maxTurns: 1`, no tools) returns up to `maxFiles` file names from that manifest; anything not in the manifest is dropped.
226
+ 3. The selected entries' **bodies** (frontmatter excluded) are truncated to `maxEntryBytes`, sanitised, and injected as one `display: false` custom message wrapped in `<relevant_memories>`; the total stops at `maxInjectionBytes`.
227
+ 4. Injected file names are remembered for the session, and the set is cleared on compaction. You get a `Recalled: N entries` notification.
165
228
 
166
229
  ### Extract memories
167
230
 
168
- After each agent run (`agent_end` hook):
169
- 1. An async headless subagent is spawned with the conversation transcript
170
- 2. It uses `ls` and `read` to check existing topic files and MEMORY.md, then `memory_add` to persist learnings
171
- 3. If it finds learnings worth persisting, it writes them — preferences, conventions, debugging insights
172
- 4. The subagent runs independently; its results benefit future sessions
231
+ The extractor receives a structured rendering instead of a lossy two-message summary:
232
+
233
+ ```
234
+ === Conversation ===
235
+ [1] user: <full text>
236
+ [2] assistant: <text> | tool_call: memory({"action":"list"})
237
+ [3] tool_result: <summary, capped at maxToolResultChars>
238
+ [4] user: <correction>
239
+ ```
240
+
241
+ It runs with the five main-agent actions (never `rename` / `rebuild_index`), no file tools, `maxTurns: 5`, and a 120s timeout. If the logical lock is busy (a dream owns the round) it **skips the turn** rather than queueing — the next `agent_end` will come.
173
242
 
174
- Memory extraction is selective: it ignores one-time tasks, code snippets derivable from the project, and anything already in CLAUDE.md.
243
+ ### Locking
244
+
245
+ | Level | Scope | Behaviour |
246
+ |---|---|---|
247
+ | In-process logical lock (per memory dir) | one primitive call; or a whole dream round | Waits up to `lock.timeoutMs`, then throws a readable error naming the directory. `extract` uses the non-waiting form and skips the turn |
248
+ | Cross-process `.lock` | milliseconds, around the physical write | Acquired with `link` (atomic), **never reclaimed automatically**: no TTL, no heartbeat, no takeover |
249
+
250
+ A crash inside a write can therefore leave a `.lock` behind, and nothing will ever delete it for you — that is the deliberate price of a hard mutual-exclusion guarantee. The error names the pid, op and start time; `/memory unlock` is the one sanctioned way to clear it.
175
251
 
176
252
  ## Tool reference
177
253
 
178
254
  ```
179
- memory(action: "add" | "remove" | "search",
180
- content?, topic?, title?, type?,
181
- entry?, query?, scope?)
255
+ memory(action: "add" | "replace" | "remove" | "list" | "search",
256
+ name?, description?, content?, type?, query?, scope?)
182
257
  ```
183
258
 
259
+ `description` is the only relevance signal a future session gets — always pass a self-contained one with `add` and `replace`.
260
+
184
261
  ### `add`
185
262
 
186
- Appends a `## entry` block to a topic file. If the topic already exists in the index, the MEMORY.md hook is updated to summarize all entries. For new topics, a new index line is created.
263
+ Creates a memory, or **overwrites the one whose `name` matches exactly** (idempotent; `created` is preserved).
264
+
265
+ - `name` (required) — unique, human-readable title
266
+ - `content` (required) — the memory body; it becomes the whole entry file
267
+ - `description` (optional) — one self-contained line; defaults to the first sentence of `content`
268
+ - `type` (optional) — `user` / `feedback` (default) / `project` / `reference`
187
269
 
188
- - **`content`** (required) — knowledge text to persist
189
- - **`topic`** (required) — target filename, e.g. `"debugging.md"`. Auto-created if new
190
- - **`title`** (required) — descriptive, self-contained title for the entry. Only the MEMORY.md index line is injected into future prompts (topic file content is NOT), so the title alone must convey what was learned
191
- - **`type`** (optional) — memory category: `"user"`, `"feedback"` (default), `"project"`, `"reference"`
270
+ ### `replace`
271
+
272
+ Rewrites an existing memory's `content` / `description` / `type`. Looks the entry up by `name`. Renaming is `rename`, which is dream-only.
192
273
 
193
274
  ### `remove`
194
275
 
195
- Deletes an entry by title. Searches across all topic files for the matching `##` block. Updates the MEMORY.md hook for the affected topic. When the last entry in a topic is removed, the topic file and its index line are deleted.
276
+ Deletes the entry matched by `name` — file, index line, and the memory itself. Fails loudly if the entry cannot be found or the file cannot be removed (no silent "deleted").
277
+
278
+ ### `list`
196
279
 
197
- - **`entry`** (required) — exact entry title to remove
280
+ One line per memory: `- name (type, modified …) — description [file]`.
198
281
 
199
282
  ### `search`
200
283
 
201
- Queries either memory files or session history. Memory search returns the full entry block (entire `##` section) for each match.
284
+ - `query` (required)
285
+ - `scope` (optional) — `memory` (default: name, description and body of every entry) or `sessions` (past conversation history)
202
286
 
203
- - **`query`** (required) — search keyword
204
- - **`scope`** (optional) — `"memory"` (default, scans topic files) or `"sessions"` (scans session history)
287
+ ### Dream-only actions
288
+
289
+ `rename` (`name` + `new_name`; the file name and the index line follow, keeping the line's position) and `rebuild_index` (rebuild from disk, preserving a hand-written header). They are registered **only** inside `/dream`'s headless session, so neither the main agent nor the extractor can call them.
205
290
 
206
291
  ## Commands
207
292
 
208
293
  ### `/memory`
209
294
 
210
- Show memory status (enabled/disabled, directory, index line count, topic files, last dream timestamp).
295
+ ```
296
+ /memory — status
297
+ /memory unlock — remove a left-behind .lock (asks for confirmation first)
298
+ ```
299
+
300
+ Status output:
211
301
 
212
302
  ```
213
- /memory — show status
214
- /memory on — enable memory
215
- /memory off — disable memory
303
+ Memory: enabled
304
+ Dir: /home/you/.pi/memory/git/github.com__owner__repo
305
+ Index: 38/200 lines, 2841/25600 bytes, 1 unrecognized lines
306
+ Entries: 37
307
+ Last dream: 2026-10-01T22:10:04.882Z
308
+ Lock: free
216
309
  ```
217
310
 
311
+ - `Index` uses the **write** capacity (`memIndexMax*`) and reports how many non-empty lines could not be parsed as index lines (the `# Memory Index` header and hand-written headings count). CRLF (or lone CR) line endings are normalised to LF before parsing, and the next write emits LF too, so a `MEMORY.md` re-saved by a Windows editor does **not** raise this count.
312
+ - `Lock` is `free`, `held by <op> (pid N on <hostname>, started <ISO>)`, or `unreadable — run /memory unlock`. `/memory unlock` shows the same holder line in its confirmation prompt.
313
+ - In a session started with `enabled: false`, nothing is initialized at boot: `/memory` reports `Memory: disabled` plus `Dir: not initialized — set "enabled": true in memory.json and restart`, there is no way to enable it mid-session, and `/memory unlock` still works without a store.
314
+ - If a required model is missing or cannot be resolved, nothing is initialized and `/memory` reports `Memory: misconfigured` and `Dir: not initialized`, followed by one `- <error>` line per problem. The same errors are shown as an error notification at session start.
315
+
218
316
  ### `/dream`
219
317
 
220
- Launch a headless agent that reads all memory files and consolidates them in four phases:
318
+ Asks for confirmation, snapshots the whole directory, then runs a headless agent through four phases:
319
+
320
+ 1. **Orient** — `list`, read the relevant entries
321
+ 2. **Gather Signal** — find duplicates (several entries for one fact), contradictions, stale items
322
+ 3. **Consolidate** — merge with `replace` + `remove`, rename with `rename`
323
+ 4. **Prune & Index** — `rebuild_index` as a backstop
221
324
 
222
- 1. **Orient** — list files, read MEMORY.md, skim topic files
223
- 2. **Gather Signal** — find duplicates, contradictions, outdated entries
224
- 3. **Consolidate** — merge duplicates, resolve contradictions, update dates
225
- 4. **Prune & Index** — rebuild frontmatter, generate hooks, rebuild MEMORY.md
325
+ It cannot touch files directly: it only has the seven `memory` actions. A summary notification arrives when it finishes, `Dream failed: …` when it does not. The model is configurable via `dream.model`.
226
326
 
227
- The model used can be configured via `dream.model` in `memory.json`.
327
+ ## 1.x data
228
328
 
229
- A confirmation dialog is shown before the consolidation begins. The result summary is shown as a notification when done.
329
+ Automatic 1.x → 2.0 migration has been removed. Legacy topic files (frontmatter with `updated` and without `created`/`modified`, so they fail the five-field v2 frontmatter check) stay on disk untouched and are **invisible to the memory system** — `parseEntryFile` requires the five v2 frontmatter fields, so such files never appear in the index, injections, `list`/`read`/`search`, and `/dream` cannot see them either (dream only has the `memory` tool). To recover their content by hand, split each `## ` section into its own file with v2 frontmatter (`name`, `description`, `type`, `created`, `modified`). Directories created by an earlier migration are still never pruned: `.backups/migrate-*/originals/` holds the pre-2.0 topic files, and `.backups/migrate-*/MEMORY.md` the index as it was then.
230
330
 
231
331
  ## File layout
232
332
 
@@ -234,11 +334,10 @@ A confirmation dialog is shown before the consolidation begins. The result summa
234
334
  ~/.pi/memory/
235
335
  git/
236
336
  github.com__yandy__pi-packages/ ← https://github.com/yandy/pi-packages.git
237
- MEMORY.md — compact index: one line per topic file
238
- .dream-meta.json — last dream timestamp + session count
239
- debugging.md — topic files with frontmatter + ## entries
240
- preferences.md
241
- ...
337
+ MEMORY.md — the index: one line per memory
338
+ SSH-port-on-staging.md
339
+ Test-command.md — one file per memory
340
+ .lock .backups/ .dream-meta.json
242
341
  local/
243
342
  home__yandy__workspace__scratch/ ← non-git directory /home/yandy/workspace/scratch
244
343
  ```
@@ -255,10 +354,16 @@ Directory names are derived as follows:
255
354
 
256
355
  The mapping is not injective: underscores are kept as-is, so `/home/a__b` and `/home/a/b` both map to `home__a__b` (and share one memory directory). Changing or renaming a remote, adding a remote that sorts before the one currently in use, or moving a local directory changes the memory directory, orphaning the old one.
257
356
 
258
- **Legacy layout:** earlier versions stored memory under `~/.pi/memory/<12-char-sha256>/`; those directories are no longer read or written. To migrate a project manually, compute the old hash with `printf '%s' "$(git rev-parse --show-toplevel)" | sha256sum | cut -c1-12` (use `$PWD` outside a git repo), then `mv` that directory to the new location (run `/memory` inside the project to see the new path).
357
+ **Older legacy layout:** versions before 1.x stored memory under `~/.pi/memory/<12-char-sha256>/`; those directories are no longer read or written. To migrate a project manually, compute the old hash with `printf '%s' "$(git rev-parse --show-toplevel)" | sha256sum | cut -c1-12` (use `$PWD` outside a git repo), then `mv` that directory to the new location (run `/memory` inside the project to see the new path) and split its topic files by hand (see [1.x data](#1x-data)).
259
358
 
260
- ## Snapshot semantics
359
+ ## Notifications
261
360
 
262
- On every `session_start`, the `MEMORY.md` index is read and appended to the system prompt via `before_agent_start`. If the index exceeds `memIndexMaxLines` or `memIndexMaxBytes`, it is truncated with a `[truncated]` marker — the agent still gets the most relevant portion. This snapshot is a static copy at the start of the session.
361
+ | When | Notification |
362
+ |---|---|
363
+ | `memory add` succeeds (interactive session) | `Saved: <name>` |
364
+ | Auto-surfacing injected entries | `Recalled: <N> entries` |
365
+ | The extractor wrote memories | `Extracted <N> memory.` / `Extracted <N> memories.` |
366
+ | The extractor failed | `Extract failed: <message>` — at most once per session |
367
+ | `/dream` finished / failed | the headless agent's summary / `Dream failed: <message>` |
263
368
 
264
- Topic file content is surfaced separately via **auto-surfacing** (automatic, per-turn, relevance-based) or the built-in `read` tool.
369
+ Headless sessions (`hasUI === false`) never notify.