@yandy0725/pi-memory 1.4.0 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (23) hide show
  1. package/README.md +248 -140
  2. package/README.zh.md +266 -156
  3. package/index.ts +393 -95
  4. package/package.json +1 -1
  5. package/src/agent-runner.ts +17 -2
  6. package/src/config.ts +31 -6
  7. package/src/dream.ts +98 -56
  8. package/src/entry-file.ts +81 -0
  9. package/src/entry-index.ts +112 -0
  10. package/src/extract.ts +338 -57
  11. package/src/filename.ts +48 -0
  12. package/src/fs-lock.ts +251 -0
  13. package/src/index-source.ts +140 -0
  14. package/src/inject.ts +121 -73
  15. package/src/memory-store.ts +498 -0
  16. package/src/memory-tool.ts +244 -326
  17. package/src/migrate.ts +303 -0
  18. package/src/paths.ts +1 -14
  19. package/src/process-lock.ts +122 -0
  20. package/src/sanitize.ts +36 -0
  21. package/src/snapshot.ts +68 -0
  22. package/src/index-file.ts +0 -91
  23. package/src/topic-file.ts +0 -119
package/README.md CHANGED
@@ -1,24 +1,33 @@
1
1
  # pi-memory
2
2
 
3
- File-system driven persistent memory layer for pi coding agent. Stores project knowledge across sessions — facts, preferences, debugging history — in plain Markdown files under `~/.pi/memory/<git|local>/<project>/`.
3
+ File-system driven persistent memory layer for pi coding agent. Stores project knowledge across sessions — facts, preferences, debugging history — as plain Markdown files under `~/.pi/memory/<git|local>/<project>/`.
4
4
 
5
- Aligned with Claude Code's auto memory mechanism: per-topic MEMORY.md index, automatic topic file surfacing based on relevance, per-turn memory extraction, and typed memory categories.
5
+ Aligned with Claude Code's auto memory mechanism: **one memory = one file**, a `MEMORY.md` index with exactly one line per memory, relevance-based auto-surfacing, per-turn memory extraction, and typed memory categories.
6
+
7
+ > ## ⚠️ 2.0.0 is a breaking storage change
8
+ >
9
+ > - The index is now **one line per memory** (1.x had one line per *topic file*, with many `## entries` inside it).
10
+ > - Each memory lives in **its own file** with five frontmatter fields: `name`, `description`, `type`, `created`, `modified` (1.x used `updated`).
11
+ > - The index is injected as a **system-prompt section** (`memory_index`) that is frozen for the whole session, instead of being appended to the system prompt string.
12
+ >
13
+ > The first `session_start` after the upgrade **migrates an existing 1.x directory automatically**: it snapshots the whole directory into `.backups/migrate-<ts>/`, copies the original topic files into `originals/`, splits every `## entry` into its own file, rebuilds the index, and only then writes the `.migrated` marker. If any step fails, the marker is not written, the backup stays, and the next session retries. See [Migration from 1.x](#migration-from-1x).
6
14
 
7
15
  ## Features
8
16
 
9
- - **One `memory` tool**, three actions: `add` (append entry), `remove` (delete entry by title), `search` (query memory or session history)
10
- - **Topic-based file organization**: each `memory add` writes a `## entry` block to a named `.md` file under the project's memory directory
11
- - **`MEMORY.md` index** — one compact line per topic file with a relevance hook: `- [Name](file.md) — summary`
12
- - **Memory types**: four categories — `user`, `feedback` (default), `project`, `reference` — stored in topic file frontmatter
13
- - **Auto-surfacing** ⭐: on every user message, a side-query LLM selects up to N relevant topic files and injects their content into the agent's context. No manual `read` needed — use the built-in `read` tool to inspect memory files. Session-level deduplication prevents re-injecting the same topic.
14
- - **Extract memories** ⭐: after each agent run, an async subagent analyzes the conversation and automatically writes learnings to memory — preferences, conventions, debugging insights
15
- - **Snapshot injection**: on every new session, the MEMORY.md index is appended to the system prompt, keeping the agent aware of past work
16
- - **`/dream` command**: launches a headless agent with a four-phase consolidation (Orient → Gather → Consolidate → Prune) to deduplicate, merge, and rebuild all memory files
17
- - **Dream nudge**: after N sessions or N hours, a gentle notification suggests running `/dream`
18
- - **`/memory` command**: show status, toggle on/off, inspect index and topic files
19
- - **Session search**: the `memory search scope=sessions` action queries past conversation history
20
- - **Readable, clone-safe layout**: memory lives under `~/.pi/memory/git/<host__owner__repo>/` for git repos with an http(s)/ssh/git remote (including scp-style and `git+ssh`/`git+https`), and `~/.pi/memory/local/<absolute-path>/` otherwise — clones and worktrees of the same repo share memory (a fork has its own remote, so it gets its own directory)
21
- - **Path traversal protection**: topic files are validated against escaping the memory directory
17
+ - **One `memory` tool, five actions** for the main agent: `add`, `replace`, `remove`, `list`, `search`. Two more — `rename` and `rebuild_index` — exist **only inside `/dream`'s own headless session** and never appear in the main agent's or the extractor's schema.
18
+ - **One memory = one file** — no more multi-entry `## section` blocks; `name` is the lookup key, adding an existing `name` overwrites it (idempotent).
19
+ - **`MEMORY.md` index** — exactly one line per memory: `- [Name](file.md) — description`. The separator is an **em dash** (`—`, U+2014) with one space on each side.
20
+ - **`memory_index` prompt section, frozen per session** ⭐ — the index goes into `event.systemPromptOptions.sections["memory_index"]` and its value **does not change for the rest of the session**; only compaction re-reads it from disk. Because pi diffs sections and appends nothing when they are unchanged, the system prompt stays byte-identical turn after turn and the provider's prefix cache keeps hitting. `resume` / `fork` / `reload` replay the **recorded** value from the transcript instead of reading disk, so restoring a session does not rewrite its head.
21
+ - **Injection sanitising** — everything injected (index lines, surfaced entry bodies and names) has invisible/bidi characters stripped and `<` `>` escaped, so a memory can never forge `</relevant_memories>`, `<system>`, `<project_instructions>`, `<active_agent …>` or `<memory_index>`. Sanitising happens **at injection time only**: your files on disk are never rewritten (they stay readable and hand-editable).
22
+ - **Auto-surfacing** ⭐ — on every user turn a lightweight side query selects up to `maxFiles` **entries** (selected from `description` alone) and injects their bodies inside `<relevant_memories>`. Already-injected files are deduplicated per session; the manifest is served from an in-process `mtime` cache, so a turn costs one `readdir` plus one `stat` per file. Disabled inside subagents.
23
+ - **Extract memories** ⭐ — after each run an async headless agent receives a **structured rendering of the whole conversation** (every user message in full, assistant text and tool calls, tool results with error flags), not just two messages. It writes through the same `memory` primitives, under a whole-round logical lock it never waits for: if a dream or migration is running, that turn is simply skipped.
24
+ - **`/dream`** — a headless consolidation agent (Orient → Gather Signal → Consolidate → Prune & Index) that merges duplicates, resolves contradictions, renames entries and rebuilds the index. It has **no raw file access**: it only gets the seven `memory` actions, holds the logical lock for the whole round, and snapshots the entire directory on entry.
25
+ - **Dream nudge** — after N sessions or N hours a notification suggests `/dream`.
26
+ - **`/memory`** — full status (switch, directory, index capacity, entry count, last dream, migration state, lock state including the holder), plus `on` / `off` / `unlock`.
27
+ - **Two-level locking** — an in-process logical lock carries the *logical* scope (one primitive call, or a whole dream/migration round); the cross-process `.lock` file is held for **milliseconds only** and is **never reclaimed automatically**. There is no TTL, no heartbeat and no takeover, so mutual exclusion is a hard guarantee; the price is that a lock left behind by a crashed process must be removed by a human (`/memory unlock`).
28
+ - **Snapshots** — every write leaves a rollback point under `.backups/<ts>-<label>/`, keeping the last `lock.snapshotKeep` (migration backups use a `migrate-` prefix and are never pruned). `/dream` is the exception: it snapshots the whole directory **once on entry**, and the primitives inside that round skip their per-file snapshots (one round, one rollback point).
29
+ - **Session search** — `memory search scope=sessions` queries past conversation history.
30
+ - **Readable, clone-safe layout** — memory lives under `~/.pi/memory/git/<host__owner__repo>/` for git repos with an http(s)/ssh/git remote (including scp-style and `git+ssh`/`git+https`), and `~/.pi/memory/local/<absolute-path>/` otherwise — clones and worktrees of the same repo share memory (a fork has its own remote, so it gets its own directory).
22
31
 
23
32
  ## Install
24
33
 
@@ -34,9 +43,71 @@ Or add to `~/.pi/agent/settings.json`:
34
43
  }
35
44
  ```
36
45
 
46
+ ## Storage layout
47
+
48
+ ```
49
+ ~/.pi/memory/git/github.com__owner__repo/
50
+ MEMORY.md — the index: one line per memory (em dash separator)
51
+ SSH-port-on-staging.md
52
+ Test-command.md — one file per memory
53
+ .lock — cross-process write lock (held for milliseconds, never auto-reclaimed)
54
+ .backups/ — rollback points: <ISO-ts>-<label>/, plus migrate-<ts>/ for the upgrade
55
+ .migrated — marker written as the last step of the 1.x → 2.x migration
56
+ .dream-meta.json — last dream timestamp + session count (drives the nudge)
57
+ sessions/ — persisted headless sessions, only when sessionPersistence is enabled
58
+ ```
59
+
60
+ ### Entry file
61
+
62
+ ```yaml
63
+ ---
64
+ name: SSH port on staging
65
+ description: staging SSH listens on 2222, not 22; key at ~/.ssh/staging
66
+ type: project
67
+ created: 2026-07-13
68
+ modified: 2026-10-02T08:14:03.120Z
69
+ ---
70
+
71
+ staging 的 SSH 用 2222 端口，密钥在 ~/.ssh/staging。
72
+ ```
73
+
74
+ - `name` — unique, human-readable title; the lookup key for `replace` / `remove` / `rename`. Adding the same `name` again overwrites that memory instead of creating a second one.
75
+ - `description` — **one self-contained line**. It is the only text a future session sees when the side query decides relevance, so it must make sense without the body. Bad: `Debugging tips`. Good: `staging SSH listens on 2222, not 22`.
76
+ - `type` — `user` | `feedback` (default) | `project` | `reference`.
77
+ - `created` — `YYYY-MM-DD`, written once and preserved across overwrites.
78
+ - `modified` — ISO 8601, **always written by the store**; callers cannot pass it in.
79
+
80
+ File names are derived from `name` (unsafe characters replaced, 100-byte cap, `-2` / `-3` … only when a *file name* collides). `MEMORY.md` can never be targeted by an entry.
81
+
82
+ ### MEMORY.md index
83
+
84
+ ```
85
+ # Memory Index
86
+
87
+ - [SSH port on staging](SSH-port-on-staging.md) — staging SSH listens on 2222, not 22
88
+ - [Test command](Test-command.md) — run npm test, not npm run test
89
+ ```
90
+
91
+ Writes are **surgical**: only the target line changes, hand-written headings, groups and comments are preserved byte-for-byte, and line order is stable. The one exception is line endings: CRLF (or lone CR) is normalised to LF before parsing, so the first write to a CRLF file rewrites it with LF.
92
+
93
+ ### Memory types
94
+
95
+ | Type | Meaning | Example |
96
+ |------|---------|---------|
97
+ | `user` | User role, preferences, knowledge | "User is a data scientist focused on observability" |
98
+ | `feedback` | Lessons, corrections, confirmations (default) | "Use real DB not mocks — burned last quarter" |
99
+ | `project` | Project state, deadlines, incidents | "Merge freeze starts 2026-03-05 for mobile release" |
100
+ | `reference` | Pointers to external systems | "Bug tracker = Linear INGEST project" |
101
+
102
+ ### Capacity: 200 index lines ≈ 199 memories
103
+
104
+ The index holds at most `memIndexMaxLines` (200) non-empty lines and `memIndexMaxBytes` (25600) bytes. Those 200 lines are **index lines, not memories**: `rebuildIndex` always writes a `# Memory Index` header line, and hand-written headings, groups and comments count too. A rebuilt index therefore holds at most about **199 memories per project directory** (fewer if you keep hand-written headings). Exceeding the limit does **not** fail the write: the write succeeds and the tool returns an actionable warning telling the model to merge or drop entries (everything past the limit is invisible on the next load).
105
+
106
+ This is why `/dream` is no longer optional housekeeping — it is **capacity management**. Run it (or accept the nudge) before you approach 199 memories.
107
+
37
108
  ## Configuration
38
109
 
39
- Create `memory.json` in the agent directory (`~/.pi/agent/memory.json`) or the project `.pi/` directory (if trusted):
110
+ Create `memory.json` in the agent directory (`~/.pi/agent/memory.json`) or the project `.pi/` directory (only when the project is trusted):
40
111
 
41
112
  ```json
42
113
  {
@@ -44,36 +115,25 @@ Create `memory.json` in the agent directory (`~/.pi/agent/memory.json`) or the p
44
115
  "memoryDir": "~/.pi/memory",
45
116
  "memIndexMaxLines": 200,
46
117
  "memIndexMaxBytes": 25600,
47
- "defaults": {
48
- "model": "deepseek/deepseek-v4-flash",
49
- "sessionPersistence": { "enabled": false }
50
- },
51
- "dream": {
52
- "nudgeAfterSessions": 5,
53
- "nudgeAfterHours": 24,
54
- "model": "auto",
55
- "thinkLevel": "high",
56
- "sessionPersistence": { "enabled": false }
57
- },
58
- "sessionSearch": {
59
- "maxSessions": 10,
60
- "maxMatches": 5
61
- },
118
+ "memIndexInjectMaxLines": 200,
119
+ "memIndexInjectMaxBytes": 25600,
120
+ "lock": { "timeoutMs": 5000, "snapshotKeep": 5 },
121
+ "defaults": { "sessionPersistence": { "enabled": false } },
122
+ "dream": { "nudgeAfterSessions": 5, "nudgeAfterHours": 24, "thinkLevel": "high" },
123
+ "sessionSearch": { "maxSessions": 10, "maxMatches": 5 },
62
124
  "autoSurfacing": {
63
125
  "enabled": true,
64
- "model": "auto",
65
126
  "thinkLevel": "off",
66
- "maxFiles": 5,
67
- "maxTopicBytes": 4096,
68
- "maxInjectionBytes": 20480,
69
- "sessionPersistence": { "enabled": false }
127
+ "maxFiles": 3,
128
+ "maxEntryBytes": 3072,
129
+ "maxInjectionBytes": 10240
70
130
  },
71
131
  "extractMemories": {
72
132
  "enabled": true,
73
- "model": "auto",
74
133
  "thinkLevel": "high",
75
134
  "maxContextTokens": 2000,
76
- "sessionPersistence": { "enabled": false }
135
+ "maxToolResultChars": 500,
136
+ "maxAssistantChars": 2000
77
137
  }
78
138
  }
79
139
  ```
@@ -82,151 +142,192 @@ Create `memory.json` in the agent directory (`~/.pi/agent/memory.json`) or the p
82
142
  |-----|---------|-------------|
83
143
  | `enabled` | `true` | Toggle the entire memory system on/off |
84
144
  | `memoryDir` | `~/.pi/memory` | Root directory for all memory data |
85
- | `memIndexMaxLines` | `200` | Max lines in `MEMORY.md` before capacity errors |
86
- | `memIndexMaxBytes` | `25600` | Max bytes in `MEMORY.md` before capacity errors |
87
- | `defaults.model` | — | Shared model fallback for all sub-tasks. Per-task `model` overrides |
88
- | `defaults.sessionPersistence.enabled` | `false` | Shared session persistence fallback (default: in-memory). Per-task overrides |
89
- | `defaults.sessionPersistence.sessionDir` | `<project memory dir>/sessions/` | Custom session directory for persisted headless agent sessions |
90
- | `dream.nudgeAfterSessions` | `5` | Sessions since last dream before nudge is shown |
91
- | `dream.nudgeAfterHours` | `24` | Hours since last dream before nudge is shown |
145
+ | `memIndexMaxLines` | `200` | Write capacity: max non-empty lines in `MEMORY.md` (the `# Memory Index` header and hand-written headings count too, so this is not exactly the memory count) |
146
+ | `memIndexMaxBytes` | `25600` | Write capacity: max bytes of `MEMORY.md` |
147
+ | `memIndexInjectMaxLines` | `200` | Injection budget: max lines of the index put into the `memory_index` section. Same scale as the write capacity on purpose — a smaller budget would hide memories that were written successfully |
148
+ | `memIndexInjectMaxBytes` | `25600` | Injection budget: max bytes of the index section (truncated with a `[truncated: …]` marker) |
149
+ | `lock.timeoutMs` | `5000` | How long a write waits for the logical lock (single primitive) or the cross-process `.lock`. Migration uses a fixed 30s because it rewrites the whole directory inside the lock. Also the upper bound `session_shutdown` waits for in-flight writes |
150
+ | `lock.snapshotKeep` | `5` | Rollback points kept in `.backups/` (directories named `migrate-*` are never pruned) |
151
+ | `defaults.model` | — | Shared model fallback for all sub-tasks; a per-task `model` overrides it, and an unset model falls back to the parent session's model |
152
+ | `defaults.sessionPersistence.enabled` | `false` | Shared fallback: headless sub-sessions (extract / dream / side query) stay in memory by default |
153
+ | `defaults.sessionPersistence.sessionDir` | `<project memory dir>/sessions/` | Custom directory for persisted headless sessions |
154
+ | `dream.nudgeAfterSessions` | `5` | Sessions since the last dream before the nudge is shown |
155
+ | `dream.nudgeAfterHours` | `24` | Hours since the last dream before the nudge is shown |
92
156
  | `dream.model` | — | Model for dream consolidation (`"provider/id"`). Falls back to `defaults.model` → parent model |
93
- | `dream.thinkLevel` | `"high"` | Thinking effort for dream subagent: `"off"`, `"minimal"`, `"low"`, `"medium"`, `"high"`, `"xhigh"` |
94
- | `dream.sessionPersistence.enabled` | `false` | Persist dream agent sessions to disk (debug/audit). Falls back to `defaults.sessionPersistence.enabled` |
95
- | `dream.sessionPersistence.sessionDir` | `<project memory dir>/sessions/` | Custom session directory for dream sessions |
96
- | `sessionSearch.maxSessions` | `10` | Max sessions to scan when searching history |
157
+ | `dream.thinkLevel` | `"high"` | Thinking effort for the dream agent: `off` / `minimal` / `low` / `medium` / `high` / `xhigh` |
158
+ | `dream.sessionPersistence.*` | inherits `defaults` | Persist dream sessions to disk (debug/audit) |
159
+ | `sessionSearch.maxSessions` | `10` | Max sessions to scan for `search scope=sessions` |
97
160
  | `sessionSearch.maxMatches` | `5` | Max matches to return from history search |
98
- | `autoSurfacing.enabled` | `true` | ⭐ Enable per-turn topic file auto-injection |
99
- | `autoSurfacing.model` | — | ⭐ Model for side-query relevance selection. Falls back to `defaults.model` → parent model |
100
- | `autoSurfacing.thinkLevel` | `"off"` | ⭐ Thinking effort for side-query (recommended: `"off"` for lightweight selection) |
101
- | `autoSurfacing.sessionPersistence.enabled` | `false` | Persist side-query agent sessions to disk. Falls back to `defaults.sessionPersistence.enabled` |
102
- | `autoSurfacing.maxFiles` | `5` | ⭐ Max topic files to inject per turn |
103
- | `autoSurfacing.maxTopicBytes` | `4096` | ⭐ Max bytes per injected topic file (truncated) |
104
- | `autoSurfacing.maxInjectionBytes` | `20480` | ⭐ Max total bytes of injected content per turn |
161
+ | `autoSurfacing.enabled` | `true` | ⭐ Enable per-turn entry auto-injection |
162
+ | `autoSurfacing.model` | — | ⭐ Model for the relevance side query. Falls back to `defaults.model` → parent model |
163
+ | `autoSurfacing.thinkLevel` | `"off"` | ⭐ Thinking effort for the side query (`"off"` keeps it cheap) |
164
+ | `autoSurfacing.maxFiles` | `3` | ⭐ Max entries to inject per turn |
165
+ | `autoSurfacing.maxEntryBytes` | `3072` | ⭐ Max bytes of a single injected entry body (truncated). Replaces 1.x's `maxTopicBytes`, which is ignored |
166
+ | `autoSurfacing.maxInjectionBytes` | `10240` | ⭐ Max total bytes of injected content per turn |
167
+ | `autoSurfacing.sessionPersistence.*` | inherits `defaults` | Persist side-query sessions to disk |
105
168
  | `extractMemories.enabled` | `true` | ⭐ Enable per-turn memory extraction |
106
- | `extractMemories.model` | — | ⭐ Model for the extraction subagent. Falls back to `defaults.model` → parent model |
107
- | `extractMemories.thinkLevel` | `"high"` | ⭐ Thinking effort for extraction: `"off"` / `"minimal"` / `"low"` / `"medium"` / `"high"` / `"xhigh"` |
108
- | `extractMemories.sessionPersistence.enabled` | `false` | Persist extract agent sessions to disk. Falls back to `defaults.sessionPersistence.enabled` |
109
- | `extractMemories.maxContextTokens` | `2000` | ⭐ Max tokens of conversation to analyze |
169
+ | `extractMemories.model` | — | ⭐ Model for the extraction agent. Falls back to `defaults.model` → parent model |
170
+ | `extractMemories.thinkLevel` | `"high"` | ⭐ Thinking effort for extraction |
171
+ | `extractMemories.maxContextTokens` | `2000` | ⭐ Budget for the rendered conversation (`× 4` characters; the middle is trimmed first, head and tail are kept, user messages are dropped last) |
172
+ | `extractMemories.maxToolResultChars` | `500` | ⭐ Per-message cap for a rendered `tool_result` |
173
+ | `extractMemories.maxAssistantChars` | `2000` | ⭐ Per-message cap for rendered assistant text (user messages are never truncated) |
174
+ | `extractMemories.sessionPersistence.*` | inherits `defaults` | Persist extract sessions to disk |
110
175
 
111
- Persisted headless agent sessions default to `<project memory dir>/sessions/` — inside the project's memory directory, not inside your working copy. For example `~/.pi/memory/git/github.com__owner__repo/sessions/`.
112
-
113
- Project-level config (`.pi/memory.json`) is only loaded when the project is trusted.
176
+ Persisted headless sessions default to `<project memory dir>/sessions/` — inside the project's memory directory, not inside your working copy.
114
177
 
115
178
  ## How it works
116
179
 
117
- ### MEMORY.md index
180
+ ### Session lifecycle
118
181
 
119
- MEMORY.md is a compact **pointer index** — one line per topic file, not per entry:
182
+ | Event | What pi-memory does |
183
+ |---|---|
184
+ | `session_start` | Load config → resolve the memory directory → run the 1.x migration if needed → **pick the index value and freeze it** (disk for `startup`/`new`; the recorded transcript value for `resume`/`fork`/`reload`) → register the `memory` tool (once, five actions) → rebuild the manifest cache → dream nudge |
185
+ | `before_agent_start` | Write the frozen value into `sections["memory_index"]` (**unconditionally, every turn**), then auto-surfacing (main session, not a subagent) |
186
+ | `agent_end` | Fire the async extractor; notify `Extracted N memories.` when it wrote something, or `Extract failed: …` once per session |
187
+ | `session_compact` | Clear the injected-file set **and re-read the index from disk** — the only in-session refresh point |
188
+ | `session_shutdown` | Wait for in-flight writes to finish (bounded by `lock.timeoutMs`) so a quit does not leave a stale `.lock` |
120
189
 
121
- ```
122
- - [Debugging](debugging.md) — SSH uses port 2222; MySQL 30s timeout on staging
123
- - [API Conventions](api.md) — REST handlers in src/api/handlers/; standard error format
124
- ```
125
-
126
- Only the index is injected into the system prompt on every session (first 200 lines / 25KB). Topic file content is **not** loaded at session start — it's surfaced on demand via auto-surfacing or the built-in `read` tool.
190
+ ### Why the index is frozen
127
191
 
128
- ### Topic file format
192
+ pi builds the system prompt from ordered sections and appends a *patch* message only for sections whose value changed; when nothing changed it appends nothing at all. A custom section is inserted **last**, so `memory_index` is the final part of the system prompt with the entire conversation after it. If its value changed mid-session, the folded head would be rewritten and everything after it would lose its prefix cache. Hence: one value per session, refreshed only at compaction (where the middle of the conversation is being rewritten anyway).
129
193
 
130
- Each topic file uses YAML frontmatter with four fields:
131
-
132
- ```yaml
133
- ---
134
- name: Debugging Tips
135
- description: Common debugging patterns, SSH ports, MySQL timeout configs
136
- type: feedback
137
- updated: 2026-07-13
138
- ---
194
+ The accepted trade-off: memories written during this session are **not** visible in this session's index. The tool result already confirms the write, and the next session sees it.
139
195
 
140
- ## SSH Gotcha
141
- staging uses port 2222, key at ~/.ssh/staging
196
+ If the host pi is older than the sections API, pi-memory falls back to appending the index to the system prompt string — same functionality, worse caching.
142
197
 
143
- ## MySQL Timeout
144
- connection timeout after 30s on staging
145
- ```
198
+ ### Auto-surfacing
146
199
 
147
- The `description` field is critical — the auto-surfacing side-query uses it to determine relevance. Make it specific.
200
+ 1. Build the manifest from the store's `mtime` cache: `[type] file.md — description`, newest first, capped at 200 entries and 4000 characters (the description budget is split evenly, never below 80 characters each).
201
+ 2. A side query (`maxTurns: 1`, no tools) returns up to `maxFiles` file names from that manifest; anything not in the manifest is dropped.
202
+ 3. The selected entries' **bodies** (frontmatter excluded) are truncated to `maxEntryBytes`, sanitised, and injected as one `display: false` custom message wrapped in `<relevant_memories>`; the total stops at `maxInjectionBytes`.
203
+ 4. Injected file names are remembered for the session, and the set is cleared on compaction. You get a `Recalled: N entries` notification.
148
204
 
149
- ### Memory types
205
+ ### Extract memories
150
206
 
151
- | Type | Meaning | Example |
152
- |------|---------|---------|
153
- | `user` | User role, preferences, knowledge | "User is a data scientist focused on observability" |
154
- | `feedback` | Lessons, corrections, confirmations (default) | "Use real DB not mocks — burned last quarter" |
155
- | `project` | Project state, deadlines, incidents | "Merge freeze starts 2026-03-05 for mobile release" |
156
- | `reference` | Pointers to external systems | "Bug tracker = Linear INGEST project" |
207
+ The extractor receives a structured rendering instead of a lossy two-message summary:
157
208
 
158
- ### Auto-surfacing
209
+ ```
210
+ === Conversation ===
211
+ [1] user: <full text>
212
+ [2] assistant: <text> | tool_call: memory({"action":"list"})
213
+ [3] tool_result: <summary, capped at maxToolResultChars>
214
+ [4] user: <correction>
215
+ ```
159
216
 
160
- On every user message (`before_agent_start` hook):
161
- 1. Scan all topic files, extract their frontmatter metadata
162
- 2. A side-query LLM selects up to `maxFiles` relevant topic files based on the user's query
163
- 3. Already-injected topics are skipped (session-level dedup)
164
- 4. Selected topic file content is injected as context — the agent sees relevant memories automatically
217
+ It runs with the five main-agent actions (never `rename` / `rebuild_index`), no file tools, `maxTurns: 5`, and a 120s timeout. If the logical lock is busy (a dream or a migration owns the round) it **skips the turn** rather than queueing — the next `agent_end` will come.
165
218
 
166
- ### Extract memories
219
+ ### Locking
167
220
 
168
- After each agent run (`agent_end` hook):
169
- 1. An async headless subagent is spawned with the conversation transcript
170
- 2. It uses `ls` and `read` to check existing topic files and MEMORY.md, then `memory_add` to persist learnings
171
- 3. If it finds learnings worth persisting, it writes them — preferences, conventions, debugging insights
172
- 4. The subagent runs independently; its results benefit future sessions
221
+ | Level | Scope | Behaviour |
222
+ |---|---|---|
223
+ | In-process logical lock (per memory dir) | one primitive call; or a whole dream / migration round | Waits up to `lock.timeoutMs` (migration: 30s), then throws a readable error naming the directory. `extract` uses the non-waiting form and skips the turn |
224
+ | Cross-process `.lock` | milliseconds, around the physical write | Acquired with `link` (atomic), **never reclaimed automatically**: no TTL, no heartbeat, no takeover |
173
225
 
174
- Memory extraction is selective: it ignores one-time tasks, code snippets derivable from the project, and anything already in CLAUDE.md.
226
+ A crash inside a write can therefore leave a `.lock` behind, and nothing will ever delete it for you — that is the deliberate price of a hard mutual-exclusion guarantee. The error names the pid, op and start time; `/memory unlock` is the one sanctioned way to clear it.
175
227
 
176
228
  ## Tool reference
177
229
 
178
230
  ```
179
- memory(action: "add" | "remove" | "search",
180
- content?, topic?, title?, type?,
181
- entry?, query?, scope?)
231
+ memory(action: "add" | "replace" | "remove" | "list" | "search",
232
+ name?, description?, content?, type?, query?, scope?)
182
233
  ```
183
234
 
235
+ `description` is the only relevance signal a future session gets — always pass a self-contained one with `add` and `replace`.
236
+
184
237
  ### `add`
185
238
 
186
- Appends a `## entry` block to a topic file. If the topic already exists in the index, the MEMORY.md hook is updated to summarize all entries. For new topics, a new index line is created.
239
+ Creates a memory, or **overwrites the one whose `name` matches exactly** (idempotent; `created` is preserved).
240
+
241
+ - `name` (required) — unique, human-readable title
242
+ - `content` (required) — the memory body; it becomes the whole entry file
243
+ - `description` (optional) — one self-contained line; defaults to the first sentence of `content`
244
+ - `type` (optional) — `user` / `feedback` (default) / `project` / `reference`
245
+
246
+ ### `replace`
187
247
 
188
- - **`content`** (required) — knowledge text to persist
189
- - **`topic`** (required) — target filename, e.g. `"debugging.md"`. Auto-created if new
190
- - **`title`** (required) — descriptive, self-contained title for the entry. Only the MEMORY.md index line is injected into future prompts (topic file content is NOT), so the title alone must convey what was learned
191
- - **`type`** (optional) — memory category: `"user"`, `"feedback"` (default), `"project"`, `"reference"`
248
+ Rewrites an existing memory's `content` / `description` / `type`. Looks the entry up by `name`. Renaming is `rename`, which is dream-only.
192
249
 
193
250
  ### `remove`
194
251
 
195
- Deletes an entry by title. Searches across all topic files for the matching `##` block. Updates the MEMORY.md hook for the affected topic. When the last entry in a topic is removed, the topic file and its index line are deleted.
252
+ Deletes the entry matched by `name` — file, index line, and the memory itself. Fails loudly if the entry cannot be found or the file cannot be removed (no silent "deleted").
196
253
 
197
- - **`entry`** (required) — exact entry title to remove
254
+ ### `list`
255
+
256
+ One line per memory: `- name (type, modified …) — description [file]`.
198
257
 
199
258
  ### `search`
200
259
 
201
- Queries either memory files or session history. Memory search returns the full entry block (entire `##` section) for each match.
260
+ - `query` (required)
261
+ - `scope` (optional) — `memory` (default: name, description and body of every entry) or `sessions` (past conversation history)
262
+
263
+ ### Dream-only actions
202
264
 
203
- - **`query`** (required) — search keyword
204
- - **`scope`** (optional) — `"memory"` (default, scans topic files) or `"sessions"` (scans session history)
265
+ `rename` (`name` + `new_name`; the file name and the index line follow, keeping the line's position) and `rebuild_index` (rebuild from disk, preserving a hand-written header). They are registered **only** inside `/dream`'s headless session, so neither the main agent nor the extractor can call them.
205
266
 
206
267
  ## Commands
207
268
 
208
269
  ### `/memory`
209
270
 
210
- Show memory status (enabled/disabled, directory, index line count, topic files, last dream timestamp).
271
+ ```
272
+ /memory — status
273
+ /memory on — enable
274
+ /memory off — disable
275
+ /memory unlock — remove a left-behind .lock (asks for confirmation first)
276
+ ```
277
+
278
+ Status output:
211
279
 
212
280
  ```
213
- /memory — show status
214
- /memory on — enable memory
215
- /memory off — disable memory
281
+ Memory: enabled
282
+ Dir: /home/you/.pi/memory/git/github.com__owner__repo
283
+ Index: 38/200 lines, 2841/25600 bytes, 1 unrecognized lines
284
+ Entries: 37
285
+ Last dream: 2026-10-01T22:10:04.882Z
286
+ Migration: migrated at 2026-09-30T09:12:44.120Z (18 entries from 4 files)
287
+ Lock: free
216
288
  ```
217
289
 
290
+ - `Index` uses the **write** capacity (`memIndexMax*`) and reports how many non-empty lines could not be parsed as index lines (the `# Memory Index` header and hand-written headings count). CRLF (or lone CR) line endings are normalised to LF before parsing, and the next write emits LF too, so a `MEMORY.md` re-saved by a Windows editor does **not** raise this count.
291
+ - `Migration` is `migrated at …`, `not needed` (the marker says nothing had to be moved) or `pending` (no marker / unreadable marker → the next `session_start` retries).
292
+ - `Lock` is `free`, `held by <op> (pid N, started <ISO>)`, or `unreadable — run /memory unlock`.
293
+ - In a session started with `enabled: false`, the memory store is never initialized, so `/memory on` answers `Memory not initialized.` instead of switching anything on — and `/memory unlock` is unreachable in that session. Start a new session (or restart pi) to use either of them.
294
+
218
295
  ### `/dream`
219
296
 
220
- Launch a headless agent that reads all memory files and consolidates them in four phases:
297
+ Asks for confirmation, snapshots the whole directory, then runs a headless agent through four phases:
298
+
299
+ 1. **Orient** — `list`, read the relevant entries
300
+ 2. **Gather Signal** — find duplicates (several entries for one fact), contradictions, stale items
301
+ 3. **Consolidate** — merge with `replace` + `remove`, rename with `rename`
302
+ 4. **Prune & Index** — `rebuild_index` as a backstop
303
+
304
+ It cannot touch files directly: it only has the seven `memory` actions. A summary notification arrives when it finishes, `Dream failed: …` when it does not. The model is configurable via `dream.model`.
305
+
306
+ ## Migration from 1.x
307
+
308
+ Automatic, on the first `session_start` after the upgrade:
221
309
 
222
- 1. **Orient** — list files, read MEMORY.md, skim topic files
223
- 2. **Gather Signal** — find duplicates, contradictions, outdated entries
224
- 3. **Consolidate** — merge duplicates, resolve contradictions, update dates
225
- 4. **Prune & Index** — rebuild frontmatter, generate hooks, rebuild MEMORY.md
310
+ 1. take the logical lock for the whole round (30s);
311
+ 2. snapshot the directory into `.backups/migrate-<ts>/` and copy the original topic files into `.backups/migrate-<ts>/originals/`;
312
+ 3. for every legacy file (one with ≥ 2 `## ` sections, or with an `updated` frontmatter field): split each `## entry` into its own file, carrying over `type` and turning `updated` into `created` / `modified`; names that collide across files get a ` (2)`, ` (3)` suffix;
313
+ 4. `rebuildIndex()`;
314
+ 5. delete the original topic files (they stay in `originals/`);
315
+ 6. write `.migrated`;
316
+ 7. notify `Migrated N memories from M topic files. Backup at <path>`.
226
317
 
227
- The model used can be configured via `dream.model` in `memory.json`.
318
+ A file whose frontmatter already has a `modified` field is **never** treated as a legacy topic file, even when its body contains several `## ` headings — that guard is what keeps a normal v2 entry from being split apart.
228
319
 
229
- A confirmation dialog is shown before the consolidation begins. The result summary is shown as a notification when done.
320
+ A re-run is safe: the marker is only written at the very end, and an entry whose name **and** body already exist is reused instead of being duplicated. If a step fails, nothing is marked, the backup is kept, and the error is reported — the next session retries.
321
+
322
+ **Manual rollback:**
323
+
324
+ ```bash
325
+ cd ~/.pi/memory/git/github.com__owner__repo # the directory /memory prints
326
+ ls .backups/migrate-*/originals/ # pick the run you want to undo
327
+ cp .backups/migrate-<ts>/originals/*.md . # restore the 1.x topic files
328
+ rm .migrated # let the migration run again
329
+ rm <generated entry files> # the ones listed by /memory (Entries) and not in originals/
330
+ ```
230
331
 
231
332
  ## File layout
232
333
 
@@ -234,11 +335,10 @@ A confirmation dialog is shown before the consolidation begins. The result summa
234
335
  ~/.pi/memory/
235
336
  git/
236
337
  github.com__yandy__pi-packages/ ← https://github.com/yandy/pi-packages.git
237
- MEMORY.md — compact index: one line per topic file
238
- .dream-meta.json — last dream timestamp + session count
239
- debugging.md — topic files with frontmatter + ## entries
240
- preferences.md
241
- ...
338
+ MEMORY.md — the index: one line per memory
339
+ SSH-port-on-staging.md
340
+ Test-command.md — one file per memory
341
+ .lock .backups/ .migrated .dream-meta.json
242
342
  local/
243
343
  home__yandy__workspace__scratch/ ← non-git directory /home/yandy/workspace/scratch
244
344
  ```
@@ -255,10 +355,18 @@ Directory names are derived as follows:
255
355
 
256
356
  The mapping is not injective: underscores are kept as-is, so `/home/a__b` and `/home/a/b` both map to `home__a__b` (and share one memory directory). Changing or renaming a remote, adding a remote that sorts before the one currently in use, or moving a local directory changes the memory directory, orphaning the old one.
257
357
 
258
- **Legacy layout:** earlier versions stored memory under `~/.pi/memory/<12-char-sha256>/`; those directories are no longer read or written. To migrate a project manually, compute the old hash with `printf '%s' "$(git rev-parse --show-toplevel)" | sha256sum | cut -c1-12` (use `$PWD` outside a git repo), then `mv` that directory to the new location (run `/memory` inside the project to see the new path).
358
+ **Older legacy layout:** versions before 1.x stored memory under `~/.pi/memory/<12-char-sha256>/`; those directories are no longer read or written. To migrate a project manually, compute the old hash with `printf '%s' "$(git rev-parse --show-toplevel)" | sha256sum | cut -c1-12` (use `$PWD` outside a git repo), then `mv` that directory to the new location (run `/memory` inside the project to see the new path) and let the 1.x → 2.x migration split its topic files.
259
359
 
260
- ## Snapshot semantics
360
+ ## Notifications
261
361
 
262
- On every `session_start`, the `MEMORY.md` index is read and appended to the system prompt via `before_agent_start`. If the index exceeds `memIndexMaxLines` or `memIndexMaxBytes`, it is truncated with a `[truncated]` marker — the agent still gets the most relevant portion. This snapshot is a static copy at the start of the session.
362
+ | When | Notification |
363
+ |---|---|
364
+ | `memory add` succeeds (interactive session) | `Saved: <name>` |
365
+ | Auto-surfacing injected entries | `Recalled: <N> entries` |
366
+ | The extractor wrote memories | `Extracted <N> memory.` / `Extracted <N> memories.` |
367
+ | The extractor failed | `Extract failed: <message>` — at most once per session |
368
+ | The 1.x migration ran | `Migrated <N> memories from <M> topic files. Backup at <path>` |
369
+ | The migration failed | `Memory migration failed: <message>` |
370
+ | `/dream` finished / failed | the headless agent's summary / `Dream failed: <message>` |
263
371
 
264
- Topic file content is surfaced separately via **auto-surfacing** (automatic, per-turn, relevance-based) or the built-in `read` tool.
372
+ Headless sessions (`hasUI === false`) never notify.