@bd7pil/opencode-deep-memory 0.6.0 → 0.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (4) hide show
  1. package/README.md +114 -148
  2. package/dist/index.js +130 -38
  3. package/dist/index.js.map +1 -1
  4. package/package.json +1 -1
package/README.md CHANGED
@@ -7,11 +7,11 @@
7
7
  OpenCode sessions are stateless. Every restart is a cold start. Native compaction
8
8
  destroys conversation content. **deep-memory** adds three layers:
9
9
 
10
- | Layer | Hook | Purpose |
11
- |-------|------|---------|
12
- | **Remember** | `memory_search`, `memory_store`, `memory_forget`, `memory_expand` | Decisions, constraints, gotchas survive across sessions via BM25 + CJK search. Storage at `.deep-memory/` in your project root — visible, version-controllable. |
13
- | **Recover** | `session.created`, `experimental.session.compacting` | Checkpoint captures conversation before compaction destroys it. Resume injection recalls everything on a new session (3000 token first-turn budget). |
14
- | **Compress** | `experimental.chat.messages.transform` | Old reasoning, metadata, system injections, and thinking tags stripped deterministically — no LLM calls. Cache-stable sentinel replacements preserve prompt cache. |
10
+ | Layer | What survives | How |
11
+ |-------|--------------|-----|
12
+ | **Remember** | Decisions, constraints, gotchas | `memory_search` / `memory_store` — BM25 + CJK search across sessions |
13
+ | **Recover** | Full conversation context | Checkpoint captures before compaction; resume injection on new session |
14
+ | **Compress** | Token budget | Deterministic stripping + pressure-triggered deep compression — no LLM calls |
15
15
 
16
16
  ## Quick start
17
17
 
@@ -30,125 +30,116 @@ OpenCode auto-installs on startup. Memory appears at `.deep-memory/` in your pro
30
30
  ## How it works
31
31
 
32
32
  ```
33
- ┌─────────────────────────────┐
34
- │ system.transform │
35
- │ m[0] stable (cache hit) │
36
- │ m[1] volatile (per-turn) │
37
- │ repo map (code symbols) │
38
- └─────────────────────────────┘
39
- ▲
40
- ┌──────────────┐ ┌──────────────┐ │ ┌───────────────────────────┐
41
- │ chat.message │ │ chat.params │ │ │ messages.transform │
42
- │ keyword→notes│ │ agent→budget │ │ │ ① Layer 1: strip reason. │
43
- │ "记住"/"rem" │ │ main 800t │ │ │ ② Layer 2: deep compress │
44
- │ │ │ oracle 400t │ │ │ dedup / error purge / │
45
- └──────────────┘ └──────────────┘ │ │ tool compress / JSON / │
46
- │ │ message prune / CCR │
47
- ┌──────────────┘ └───────────────────────────┘
48
- │
49
- ┌────────────────────┴────────────────────────┐
50
- │ event │
51
- │ session.created → resume + dream schedule │
52
- │ session.idle → enrichment + notify │
53
- │ session.compacted → checkpoint │
54
- └─────────────────────────────────────────────┘
33
+ ┌─────────────────────────────────────────────────────────────────┐
34
+ │ messages.transform (every turn) │
35
+ │ ├─ Strip reasoning/thinking parts (physical removal) │
36
+ │ ├─ Remove system-injected messages (physical removal) │
37
+ │ ├─ Truncate old tool errors │
38
+ │ └─ Deep compress: dedup / tool output / JSON / assistant text │
39
+ └─────────────────────────────────────────────────────────────────┘
40
+
41
+ ┌─────────────────────────────────────────────────────────────────┐
42
+ │ system.transform (every turn) │
43
+ │ ├─ Inject stable: MEMORY.md constraints + tool hint (cache hit)│
44
+ │ └─ Inject volatile: BM25 search results + repo map symbols │
45
+ └─────────────────────────────────────────────────────────────────┘
46
+
47
+ ┌─────────────────────────────────────────────────────────────────┐
48
+ │ compacting (before OpenCode destroys messages) │
49
+ │ ├─ Capture raw messages → checkpoint.raw.json │
50
+ │ ├─ Extract knowledge → checkpoint.md │
51
+ │ └─ Inject structured handoff prompt for LLM │
52
+ └─────────────────────────────────────────────────────────────────┘
53
+
54
+ ┌─────────────────────────────────────────────────────────────────┐
55
+ │ events │
56
+ │ ├─ session.created → resume + dream schedule │
57
+ │ ├─ session.idle → enrichment │
58
+ │ └─ session.compacted → pressure calibration │
59
+ └─────────────────────────────────────────────────────────────────┘
55
60
  ```
56
61
 
57
62
  ## Context compression
58
63
 
59
- Two compression layers run automatically, no LLM calls required.
64
+ Two layers, fully automatic, no LLM calls.
60
65
 
61
- ### Layer 1: Deterministic stripping
66
+ ### Layer 1: Deterministic stripping (always active)
62
67
 
63
- Always active, strips disposable content from old messages:
68
+ | Target | Action |
69
+ |--------|--------|
70
+ | Old reasoning/thinking parts | Physical removal |
71
+ | System injections (`<system-reminder>`, etc.) | Physical removal |
72
+ | Tool errors >100 chars (older than 4 turns) | Truncate |
73
+ | Inline `<thinking>` tags | Regex strip |
64
74
 
65
- | What gets stripped | How | Why safe |
66
- |--------------------|-----|----------|
67
- | `reasoning_details` metadata | Delete the JSON blob | Billing metadata, never reaches model |
68
- | Old reasoning text | Replace with `[cleared]` | Conclusions are in assistant text |
69
- | System injections | Replace with `[stripped]` | `<system-reminder>` stale after one turn |
70
- | Tool errors >100 chars | Truncate | An old error only needs "it failed" |
71
- | Inline `<thinking>` tags | Regex strip | Process, not product |
75
+ No marker pollution — old content is physically removed, not replaced with `[cleared]` or `[stripped]`. This prevents [context confusion](https://www.philschmid.de/context-engineering-part-2).
72
76
 
73
77
  ### Layer 2: Deep compression (pressure-triggered)
74
78
 
75
- Activates when context pressure exceeds thresholds. Inspired by
76
- [DCP](https://github.com/Opencode-DCP/opencode-dynamic-context-pruning),
77
- [Headroom](https://github.com/chopratejas/headroom), and
78
- [Edgee](https://github.com/edgee-ai/edgee).
79
-
80
79
  | Pressure | Threshold | Actions |
81
80
  |----------|-----------|---------|
82
- | **always** | every turn | tool dedup + error purge + tool output compress + JSON crush (all reversible via CCR) |
83
- | **medium** | ≥ 30% context | + old message text truncation (lossy, extracts key info) |
84
- | **high** | ≥ 50% context | + nudge (alerts model to save important findings)
81
+ | **always** | every turn | tool dedup + error purge + tool output compress + JSON crush + assistant text compress |
82
+ | **medium** | ≥ 50K tokens | + memory nudge (prompts LLM to use `memory_store`) |
83
+ | **high** | ≥ 150K tokens | + pressure nudge (prompts LLM to summarize old tasks) |
85
84
 
86
- What gets compressed at medium+:
85
+ Thresholds are absolute, not percentage-based — they work consistently across 200K and 1M+ context windows. Based on [Focus Agent](https://arxiv.org/html/2601.07190v1) research.
87
86
 
88
87
  | Target | Strategy | Source |
89
88
  |--------|----------|--------|
90
- | Duplicate tool calls | Signature matching (`toolName::sortedParams`) | DCP |
91
- | Old error inputs | Purge inputs after 4 turns | DCP |
92
- | File reads | Keep first 50 + key lines + last 20 | Edgee |
93
- | Command outputs | Keep errors + last 30 lines | Edgee |
94
- | Search results | Keep top-20, group by file | Edgee |
95
- | JSON arrays | Keep first 30% + last 15% + dedup middle | Headroom SmartCrusher |
96
- | Old assistant text | Extract key info (headings, code, errors) | DCP |
89
+ | Duplicate tool calls | Signature matching | [DCP][] |
90
+ | Old error inputs | Purge after 4 turns | [DCP][] |
91
+ | File reads | Keep head + key lines + tail | [Edgee][] |
92
+ | Command outputs | Keep errors + tail | [Edgee][] |
93
+ | Search results | Keep top-20, group by file | [Edgee][] |
94
+ | JSON arrays | Head + dedup middle + tail | [Headroom][] |
95
+ | Old assistant text | Preserve structure, compress prose | [LLMLingua][] |
97
96
 
98
- All compressed content is **reversible** via CCR (Compress-Cache-Retrieve):
99
- originals are cached with SHA-256 hash and 5-minute TTL.
100
- Models can retrieve them via the `deep_expand` tool.
97
+ All compressed content is **reversible** via CCR (Compress-Cache-Retrieve) — originals cached with SHA-256 hash, retrievable via `deep_expand` tool.
101
98
 
102
- **Never touched**: user messages, recent 8 messages, protected tools
103
- (question, edit, write, todowrite, memory_store/search/forget).
99
+ **Never touched**: user messages, recent 4K tokens, protected tools (question, edit, write, todowrite, memory_*).
104
100
 
105
- ## Toast notifications
101
+ ## Memory nudge
106
102
 
107
- After each LLM turn, deep-memory shows a toast notification (bottom-right corner) summarizing
108
- what was compressed and injected. The notification level is chosen automatically:
103
+ Detects decisions, constraints, and fixes in conversation — nudges the LLM to persist them.
109
104
 
110
- | Scenario | Level | Content |
111
- |----------|-------|---------|
112
- | Injection only (no compression) | minimal | One-line summary: `-8.5K stripped` |
113
- | Compression (short session) | detailed | Progress bar + per-category breakdown |
114
- | Compression + rich context (repo-map, memory, checkpoint) | extended | Full panel with budget usage |
105
+ | Pattern | Example | Nudge |
106
+ |---------|---------|-------|
107
+ | Decision | "我决定用 PostgreSQL" / "I'll use PostgreSQL" | `memory_store(type="decision")` |
108
+ | Constraint | "不能用 eval()" / "must not use eval()" | `memory_store(type="constraint")` |
109
+ | Error fix | "修复了权限问题" / "fixed the permission error" | `memory_store(type="gotcha")` |
115
110
 
116
- Example toast (detailed level):
111
+ English + Chinese. Pressure nudge and memory nudge have independent cooldowns.
117
112
 
118
- ```
119
- deep-memory | compressed
120
- ─ Compression ─────────────────────────────
121
- │████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░│
122
- reasoning -6.2K | metadata -2.1K | tool_err -0.8K
123
- ─ Injection ───────────────────────────────
124
- m[0] stable 1055B ✓ m[1] volatile 574B
125
- tier=main | mode=normal
126
- repo-map: 12 symbols | memory: 8 entries
127
- ```
113
+ ## Tools
114
+
115
+ | Tool | Purpose |
116
+ |------|---------|
117
+ | `memory_search` | Search persistent memory (BM25 + CJK bigram) |
118
+ | `memory_store` | Store decisions, constraints, gotchas, facts, notes |
119
+ | `memory_forget` | Remove stale memory entries |
120
+ | `memory_expand` | Retrieve original content of a compressed message |
121
+ | `deep_expand` | Retrieve original content via CCR hash |
128
122
 
129
- ## Cache-stable injection
123
+ ## Compaction
130
124
 
131
- Each turn pushes two system prompt fragments:
125
+ When OpenCode compacts a session:
132
126
 
133
- - **Stable** (`<deep-memory-stable>`): constraints, rules, and the tool hint.
134
- These change only when MEMORY.md is updated — typically across sessions, not turns.
135
- Because they're byte-identical turn after turn, the provider's prompt cache hits on this prefix.
127
+ 1. **Capture** raw messages to `checkpoint.raw.json`
128
+ 2. **Extract** knowledge via 5 heuristic extractors
129
+ 3. **Write** structured `checkpoint.md`
130
+ 4. **Inject** Hermes-8 structured prompt + Codex-style handoff prefix
136
131
 
137
- - **Volatile** (`<deep-memory-volatile>`): context-aware search results from the user's
138
- current query, tier-allocated by importance, plus repo map symbols for recently-read files.
139
- This is the only part that changes per turn.
132
+ The LLM produces: Task Overview → Progress → Key Decisions → Constraints → Files Modified → Errors → Next Steps → Critical Context
140
133
 
141
- The injection budget adapts to the agent: main orchestrator gets 800 tokens per turn
142
- (3000 on session resume), deep-reasoning agents get 400, and tool subagents get 80.
134
+ ## Memory consolidation
143
135
 
144
- ## Memory search (BM25 + CJK bigram)
136
+ | Cycle | Trigger | Action |
137
+ |-------|---------|--------|
138
+ | **Auto-dream** | 7 days or notes.md >20 lines | Consolidate notes + checkpoints → MEMORY.md |
139
+ | **Auto-distill** | 30 days | Package recurring workflows → skill candidates |
140
+ | **Enrichment** | Session idle after compaction | LLM enriches checkpoint with cross-references |
145
141
 
146
- Instead of SQLite FTS5, we use a pure-JS BM25 engine with a CJK-aware tokenizer.
147
- Chinese runs are split into sliding 2-character bigrams (`"权限死锁"` →
148
- `["权","权限","限死","死锁","锁"]`), making multi-character CJK phrases searchable
149
- without an embedding model. Latin text uses standard whitespace/punctuation splitting.
150
- The index is rebuilt from Markdown files on startup (<250ms for 2000 entries) and
151
- updated incrementally on writes.
142
+ New projects: MEMORY.md auto-bootstraps from notes.md. Both agents have `memory_forget` enabled.
152
143
 
153
144
  ## Configuration
154
145
 
@@ -161,48 +152,28 @@ updated incrementally on writes.
161
152
  ## Storage
162
153
 
163
154
  ```
164
- <project>/.deep-memory/ ← version-controllable
155
+ <project>/.deep-memory/
165
156
  ├── MEMORY.md persistent decisions/constraints/gotchas
166
157
  ├── notes.md keyword captures
167
158
  ├── checkpoint.md last compaction extraction
159
+ ├── checkpoint.raw.json raw messages dump
168
160
  ├── .schedule.json dream/distill state
169
- └── sessions/<sid>/ per-session archive
161
+ ├── .compaction-log.jsonl compaction audit trail
162
+ └── sessions/<sid>/ per-session archive
170
163
  ```
171
164
 
172
- ## Tools
173
-
174
- | Tool | Purpose |
175
- |------|---------|
176
- | `memory_search` | Search persistent memory across sessions (BM25 + CJK) |
177
- | `memory_store` | Store decisions, constraints, gotchas, facts, notes |
178
- | `memory_forget` | Remove memory entries matching a query |
179
- | `memory_expand` | Decompress a sentinel reference to its original content |
180
- | `deep_expand` | Retrieve original content compressed by CCR (use `[ccr:HASH]` marker) |
181
- | `deep_expand` | Retrieve original content compressed by CCR (use `[ccr:HASH]` marker) |
182
-
183
165
  ## Commands
184
166
 
185
- Copy `.opencode/command/*.md` to your project:
186
-
187
167
  - `/checkpoint` — manually capture session state
188
168
  - `/dream` — consolidate notes into persistent memory
189
169
  - `/distill` — package recurring workflows into skills
190
170
 
191
- ## Design
192
-
193
- **Memory entries** carry a type (`decision`, `constraint`, `gotcha`, `fact`, `note`) and
194
- an importance score. Importance is heuristically derived from entry type, recency,
195
- frequency across sessions, and keyword-match relevance to the current query —
196
- no LLM calls required.
197
-
198
- Entries are stored as Markdown sections (e.g. `## Decisions`, `## Constraints`) in
199
- `MEMORY.md`, with `[date]` timestamps for time-based decay. The BM25 index is rebuilt
200
- from these files on startup and updated incrementally on write.
171
+ ## Development
201
172
 
202
- Background consolidation runs on a 7-day cycle (auto-dream) plus an accumulation trigger
203
- (when `notes.md` exceeds 20 lines). A separate 30-day cycle (auto-distill) packages
204
- recurring workflows into skill candidates. Both use background sessions to avoid
205
- consuming the main session's context budget.
173
+ ```bash
174
+ npm install
175
+ npm run verify # typecheck + test (363) + build + smoke (49)
176
+ ```
206
177
 
207
178
  ## Acknowledgments
208
179
 
@@ -236,6 +207,21 @@ Our per-tool compression strategies (read, bash, grep, glob) are inspired by Edg
236
207
  **[Contextomizer][]** — ultra-fast deterministic library for transforming bloated tool outputs.
237
208
  Our content type detection pipeline is inspired by Contextomizer's approach.
238
209
 
210
+ **[Focus Agent][]** — autonomous memory management for coding agents.
211
+ Our absolute token thresholds and assistant text compression strategy are based on Focus Agent's research.
212
+
213
+ **[LLMLingua][]** — prompt compression for LLMs.
214
+ Our selective assistant text compression (preserve structure, compress prose) is inspired by LLMLingua's approach.
215
+
216
+ **[Codex CLI][]** — OpenAI's coding agent.
217
+ Our handoff prefix pattern (telling the LLM it's resuming a prior task) is based on Codex CLI's compaction protocol.
218
+
219
+ **[Google ADK][]** — Agent Development Kit with append-only event compaction.
220
+ Our structured compaction prompt (Hermes-8 sections) is inspired by ADK's compaction architecture.
221
+
222
+ **[Hermes][]** — production-grade compaction prompt design.
223
+ Our 8-section checkpoint template follows Hermes's structured summary format.
224
+
239
225
  [MiMo-Code]: https://github.com/XiaomiMiMo/MiMo-Code
240
226
  [Magic Context]: https://github.com/cortexkit/magic-context
241
227
  [Aider]: https://github.com/Aider-AI/aider
@@ -247,31 +233,11 @@ Our content type detection pipeline is inspired by Contextomizer's approach.
247
233
  [Headroom]: https://github.com/chopratejas/headroom
248
234
  [Edgee]: https://github.com/edgee-ai/edgee
249
235
  [Contextomizer]: https://github.com/GandalFran/contextomizer
250
-
251
- ## Development
252
-
253
- ```bash
254
- npm install
255
- npm run verify # typecheck + test (363) + build + smoke (49)
256
- ```
257
-
258
- Stats: 54 source files, 27 test files (363 tests), 10 compress modules, 49 smoke checks.
259
-
260
- ## CI/CD (npm Trusted Publishing)
261
-
262
- Releases use npm OIDC Trusted Publishing — no token needed. To set up for a fork:
263
-
264
- 1. **npmjs.com** → Package Settings → Trusted Publishers → Add:
265
- - Owner: your GitHub username
266
- - Repository: your fork name
267
- - Workflow filename: `publish.yml`
268
- 2. **package.json** → update `repository.url` to match your fork
269
- 3. **Push a tag** → GitHub Actions auto-publishes:
270
- ```bash
271
- git tag v1.0.0 && git push origin v1.0.0
272
- ```
273
-
274
- Requirements: npm CLI ≥ 11.5.1, Node.js ≥ 22, `id-token: write` permission, public repository.
236
+ [Focus Agent]: https://arxiv.org/html/2601.07190v1
237
+ [LLMLingua]: https://github.com/microsoft/LLMLingua
238
+ [Codex CLI]: https://github.com/openai/codex
239
+ [Google ADK]: https://github.com/google/adk-python
240
+ [Hermes]: https://github.com/NousResearch/hermes-agent
275
241
 
276
242
  ## License
277
243