@jeffober/thatch 0.1.26 → 0.1.28

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (72) hide show
  1. package/README.md +7 -7
  2. package/artifacts/skills/common.md +1 -1
  3. package/artifacts/skills/thatch-change-walkthrough.md +2 -2
  4. package/artifacts/skills/thatch-code-archaeology.md +252 -0
  5. package/artifacts/skills/thatch-code-review.md +4 -4
  6. package/artifacts/skills/thatch-code-walkthrough.md +2 -2
  7. package/artifacts/skills/thatch-coding-workflow.md +79 -0
  8. package/artifacts/skills/thatch-fact-extractor.md +2 -2
  9. package/artifacts/skills/{pr-description.md → thatch-pr-description.md} +1 -1
  10. package/artifacts/skills/thatch-review-context.md +9 -0
  11. package/artifacts/skills/thatch-review-followup.md +29 -1
  12. package/artifacts/skills/thatch-review-pedantic.md +16 -1
  13. package/artifacts/skills/thatch-review-response.md +365 -0
  14. package/artifacts/skills/thatch-review-synthesizer.md +92 -5
  15. package/artifacts/skills/{split-overlarge-pr.md → thatch-split-overlarge-pr.md} +1 -1
  16. package/artifacts/skills/{ticket-description.md → thatch-ticket-description.md} +3 -3
  17. package/bin/release +3 -3
  18. package/bin/thatch +11 -10
  19. package/docs/dev/README.md +7 -6
  20. package/docs/dev/mcp-parity.md +1 -1
  21. package/docs/dev/skills.md +29 -9
  22. package/docs/plans/README.md +9 -6
  23. package/docs/plans/intuition-drives.md +147 -0
  24. package/docs/plans/nomenclater.md +89 -0
  25. package/docs/plans/prediction-consolidation.md +361 -0
  26. package/docs/qa/README.md +20 -3
  27. package/docs/user/README.md +19 -5
  28. package/package.json +1 -1
  29. package/src/db.ts +6 -6
  30. package/src/embeddings.ts +2 -2
  31. package/src/extract-queue.ts +4 -4
  32. package/src/extraction.ts +4 -4
  33. package/src/git.ts +1 -1
  34. package/src/hygiene.ts +1 -1
  35. package/src/index.ts +40 -34
  36. package/src/mcp.ts +34 -6
  37. package/src/prompts.ts +96 -85
  38. package/src/setup.ts +19 -19
  39. package/src/sideband.ts +3 -3
  40. package/src/skills.ts +104 -6
  41. package/src/tool-defs.ts +93 -14
  42. package/src/tools.ts +12 -2
  43. package/artifacts/skills/thatch-workflow-research.md +0 -91
  44. package/docs/in-progress/prediction-engine.md +0 -404
  45. package/docs/plans/001-memory-store.md +0 -106
  46. package/docs/plans/002-agent-driven-pipelines.md +0 -59
  47. package/docs/plans/003-memory-hygiene.md +0 -61
  48. package/docs/qa/use-cases/README.md +0 -51
  49. package/docs/qa/use-cases/UC-001-memory-roundtrip.md +0 -20
  50. package/docs/qa/use-cases/UC-002-dedup-cycle.md +0 -23
  51. package/docs/qa/use-cases/UC-003-extraction-nudge.md +0 -56
  52. package/docs/qa/use-cases/UC-004-cli-inspection.md +0 -22
  53. package/docs/qa/use-cases/UC-005-setup-install.md +0 -34
  54. package/docs/qa/use-cases/UC-006-cursor-hook-lifecycle.md +0 -32
  55. package/docs/qa/use-cases/UC-007-recall-nudge.md +0 -31
  56. package/docs/qa/use-cases/UC-008-hygiene-heartbeat.md +0 -26
  57. package/docs/qa/use-cases/UC-009-flush-tools-tiers.md +0 -33
  58. package/docs/qa/use-cases/UC-010-prime.md +0 -25
  59. package/docs/qa/use-cases/UC-011-write-time-similarity-warning.md +0 -24
  60. package/docs/qa/use-cases/UC-012-model-migration.md +0 -28
  61. package/docs/qa/use-cases/UC-013-compaction-context.md +0 -25
  62. package/docs/qa/use-cases/UC-014-skill-install-drift.md +0 -28
  63. package/docs/qa/use-cases/UC-015-env-override-matrix.md +0 -28
  64. package/docs/qa/use-cases/UC-016-concurrent-session-isolation.md +0 -37
  65. package/docs/qa/use-cases/UC-017-buffer-tool-vs-buffer-batch.md +0 -27
  66. package/docs/qa/use-cases/UC-018-mcp-startup-setup-detection.md +0 -38
  67. package/docs/qa/use-cases/UC-019-archived-memory-lifecycle.md +0 -36
  68. package/docs/qa/use-cases/UC-020-extraction-escalation.md +0 -52
  69. package/docs/qa/use-cases/UC-021-prediction-autofire.md +0 -21
  70. package/docs/qa/use-cases/UC-022-prediction-dedup.md +0 -19
  71. package/docs/qa/use-cases/UC-023-prediction-confidence-model.md +0 -23
  72. package/docs/qa/use-cases/UC-024-prediction-delete.md +0 -20
@@ -1,404 +0,0 @@
1
- # Prediction engine (in progress)
2
-
3
- > **Status: implementation complete across both host paths.** The
4
- > schema, tools, auto-fire (opencode), sideband prediction fire (MCP),
5
- > prompt instructions, and tests are shipped and passing the quality
6
- > gate. The feature is ready for field testing. This doc is the live
7
- > ledger; when the work closes, graduate durable parts into a permanent
8
- > `docs/dev/prediction-engine.md` and retire this file.
9
-
10
- ## Purpose
11
-
12
- A **user decision model** that learns the user's preferences for
13
- judgment calls the coding agent faces every session: scope (how much to
14
- do), appropriateness (is this the right thing), methodology (the right
15
- way). The agent gets a "guess what the user would want in this
16
- situation" system -- inspired by nak's samskara predictive model, but
17
- different mechanism and domain.
18
-
19
- The delta over flat thatch memory is three things flat memory cannot do:
20
-
21
- 1. **Graded confidence** -- memory is binary (stored or not).
22
- Predictions have a Bayesian posterior that is reinforced or weakened
23
- by user feedback. This enables ambiguity-surfacing: when confidence
24
- is thin or predictions conflict, the agent asks the user instead of
25
- guessing.
26
- 2. **Context-disambiguation** -- matchers separate "when" from "what."
27
- The same topic (tech debt in a PR) can produce opposite predictions
28
- in different situations (flaky test blocking CI = fix it; unrelated
29
- to change = skip it). Flat memory encodes this in prose the LLM must
30
- re-reason about every time; the prediction engine encodes it as
31
- structure.
32
- 3. **Agent-driven feedback loop** -- the LLM itself decides to
33
- strengthen or weaken a prediction based on user feedback, creating a
34
- loop that flat memory lacks.
35
-
36
- ## Design decisions (settled)
37
-
38
- These came out of design discussion with Jeff, July 2026. They are
39
- constraints, not open questions.
40
-
41
- 1. **Auto-fire at chat.message.** The hook already embeds the user's
42
- text for the recall nudge. The prediction fire shares that embedding
43
- and does a second cosine search against the matchers table. When
44
- matchers clear a relevance floor, scored predictions are injected as
45
- a synthetic nudge part. No matchers fire = nothing injected (zero
46
- tokens, zero latency). No "if appropriate" hedging -- the block's
47
- presence is the appropriateness signal.
48
-
49
- 2. **Tool-based update, agent-driven.** No local LLM classifier. Prompt
50
- instructions tell the agent: when the user corrects you, answers a
51
- question, or provides a clear signal, use prediction_update to
52
- create/reinforce/weaken a prediction. The LLM decides with
53
- appropriately nuanced prompt instructions; the tool is the trusted
54
- boundary that handles embedding, dedup, and confidence math.
55
-
56
- 3. **Statistical model query, not AI.** The query (embed + cosine +
57
- weighted edges) is mechanical -- same as thatch_memory_recall but
58
- against different tables. No model call. The agent can also call
59
- prediction_query explicitly when it wants to check the model before a
60
- judgment call.
61
-
62
- 4. **No local LLM.** Considered and declined. The host LLM is already
63
- reading the message and classifying "was this a correction?" as part
64
- of understanding it. Adding a second classifier pays twice for the
65
- same answer. The cost of not having a classifier is wasted nudge text
66
- on turns with no signal -- not wasted model calls.
67
-
68
- 5. **Descriptive context, not directive.** The injected block reads
69
- "[0.72 conf, 4 tests] When X: you tend to Y" -- same shape as
70
- samskara's compound summary. The system prompt instructions govern
71
- how to act on it (follow strong silently, surface weak/competing,
72
- update on user response). No "if appropriate" or "consider" hedging
73
- in the injection itself.
74
-
75
- 6. **Confidence mirrors samskara health.** A Bayesian posterior with
76
- population-prior shrinkage: `confidence = (confirm + k*p0) /
77
- (confirm + disconfirm + k)`. No wall-clock decay -- relevance-gated,
78
- like samskara. Being tested (user gives feedback) is what moves
79
- confidence. Untested predictions stay at p0.
80
-
81
- 7. **LLM is the conflict detector.** The auto-fire returns all
82
- predictions from matching matchers, not just the top one. The LLM
83
- reads competing predictions and decides whether they conflict. No
84
- mechanical conflict heuristic -- the LLM is better at semantic
85
- conflict detection than any rule.
86
-
87
- ## The normative asymmetry (carried from samskara)
88
-
89
- Samskara's intents doc identifies a normative asymmetry: the descriptive
90
- layer (samskara predictions) is ethically free to surface, but the
91
- normative layer (intents -- "what the model is working toward") carries
92
- weight because it shapes behavior. The prediction engine is descriptive
93
- ("you tend to prefer X"), not normative ("I'm working toward X with
94
- you"). It reads user behavior, it does not pursue an agenda. The
95
- firewall from samskara's efficacy model carries over: the model that
96
- uses predictions cannot grade its own predictions -- the user's
97
- feedback is the only signal that moves confidence.
98
-
99
- ## Data model
100
-
101
- Four new tables in the same SQLite DB, same conventions as the existing
102
- `entries`/`dedup_pairs` tables (idempotent CREATE TABLE IF NOT EXISTS,
103
- BLOB embeddings, store column, crypto.randomUUID PKs).
104
-
105
- ### `prediction_matchers`
106
-
107
- Context patterns describing situations. The "when."
108
-
109
- - `id TEXT PRIMARY KEY` (crypto.randomUUID).
110
- - `store TEXT NOT NULL`.
111
- - `description TEXT NOT NULL` -- "reviewing a PR with pre-existing tech
112
- debt."
113
- - `embedding BLOB` -- 384-dim Float32Array, passageEmbed (no query
114
- prefix; matchers are passages, not queries).
115
- - `model TEXT`.
116
- - `created_at`, `updated_at` -- ISO timestamps.
117
-
118
- ### `predictions`
119
-
120
- User preference statements. The "what."
121
-
122
- - `id TEXT PRIMARY KEY`.
123
- - `store TEXT NOT NULL`.
124
- - `statement TEXT NOT NULL` -- "skip it unless it's blocking this
125
- change."
126
- - `rationale TEXT` -- why this prediction exists (inspector context).
127
- - `embedding BLOB` -- for dedup at creation time.
128
- - `model TEXT`.
129
- - `confidence REAL NOT NULL DEFAULT 0.5` -- Bayesian posterior [0,1].
130
- - `confirm_count REAL NOT NULL DEFAULT 0` -- fractional (soft signals
131
- are sub-unit).
132
- - `disconfirm_count REAL NOT NULL DEFAULT 0` -- same.
133
- - `created_at`, `updated_at`.
134
-
135
- ### `prediction_edges`
136
-
137
- Weighted matcher-to-prediction links. The "context-dependence."
138
-
139
- - `matcher_id TEXT NOT NULL` (FK to prediction_matchers, CASCADE).
140
- - `prediction_id TEXT NOT NULL` (FK to predictions, CASCADE).
141
- - `weight REAL NOT NULL DEFAULT 1.0`.
142
- - `created_at`.
143
- - `PRIMARY KEY (matcher_id, prediction_id)`.
144
-
145
- ### `prediction_provenance`
146
-
147
- Audit trail. Gives the inspector context and the agent "plenty of
148
- context" for nuance.
149
-
150
- - `id TEXT PRIMARY KEY`.
151
- - `prediction_id TEXT NOT NULL` (FK to predictions, CASCADE).
152
- - `signal TEXT NOT NULL` -- 'confirm' | 'disconfirm' | 'soft' |
153
- 'create'.
154
- - `detail TEXT` -- what the user said or did.
155
- - `created_at`.
156
-
157
- ## Confidence model
158
-
159
- Mirrors samskara's health posterior with population-prior shrinkage.
160
-
161
- ```
162
- confidence = (confirm_count + k * p0) / (confirm_count + disconfirm_count + k)
163
-
164
- k = 5 (prior strength, pseudo-count)
165
- p0 = 0.5 (population baseline; make population-derived later)
166
- W_SOFT = 0.25 (soft miss weight, same as samskara)
167
- ```
168
-
169
- - `confirm` signal: confirm_count += 1.
170
- - `disconfirm` signal: disconfirm_count += 1.
171
- - `soft` signal: disconfirm_count += W_SOFT (0.25).
172
- - `create` signal: no confidence adjustment, just seeds at p0.
173
- - No wall-clock decay. Relevance-gated: being tested (user gives
174
- feedback) is what moves confidence. Untested = stays at p0.
175
- - confirm_count / disconfirm_count are REAL (fractional). An integer
176
- column would truncate soft signals to 0 (samskara's bug).
177
- - p0 starts at 0.5 for v1. Later: `populationP0(store)` = sum(confirm) /
178
- sum(confirm + disconfirm), with a 0.5 fallback under 20 evidence.
179
-
180
- ## Scoring at query time
181
-
182
- ```
183
- score = cosine(user_text, matcher.embedding) * edge.weight * prediction.confidence
184
- ```
185
-
186
- The auto-fire returns all predictions from matching matchers (not just
187
- the top scoring one), so the LLM can read conflicts. The score is for
188
- ranking and display, not for a mechanical "follow if above X" gate --
189
- the LLM decides whether to follow, surface, or ignore, informed by the
190
- confidence and evidence count shown alongside.
191
-
192
- ## Ambiguity-surfacing (the killer feature)
193
-
194
- - **Strong prediction** (high confidence, sufficient evidence) -> agent
195
- follows silently, does not interrupt.
196
- - **No prediction** -> agent asks the user (same as today).
197
- - **Ambiguous or competing predictions** -> agent surfaces naturally:
198
- "I think you usually prefer X here, but I'm not sure -- what do you
199
- want?"
200
- - **User responds to a surfaced prediction** -> agent calls
201
- prediction_update to reinforce or weaken.
202
-
203
- Value = fewer interruptions, not more. High-confidence predictions
204
- suppress questions the agent would otherwise ask.
205
-
206
- ## Tool surface
207
-
208
- Four tools, following the existing ToolDef pattern in tool-defs.ts.
209
-
210
- ### `prediction_query`
211
-
212
- Mechanical search. Returns scored predictions with matcher
213
- description, confidence, and evidence count.
214
-
215
- Args: `context` (string), `store` (optional).
216
-
217
- ### `prediction_update`
218
-
219
- The trusted boundary. The agent says "here's the situation, here's the
220
- preference, here's what happened." The tool handles: embed matcher,
221
- find/create by cosine dedup, embed prediction, find/create by cosine
222
- dedup, adjust confidence, record provenance.
223
-
224
- Args: `matcher` (string), `prediction` (string), `signal` (confirm |
225
- disconfirm | soft | create), `rationale` (string), `store` (optional).
226
-
227
- ### `prediction_list`
228
-
229
- Inspector. Lists all predictions with confidence, evidence, matchers,
230
- and rationale.
231
-
232
- Args: `store` (optional).
233
-
234
- ## Auto-fire in chat.message hook
235
-
236
- Slots into the existing recall-nudge path in index.ts, sharing the
237
- embedding call. After the recall nudge block, before the catch:
238
-
239
- ```
240
- embedding = await model.queryEmbed(promptText) // already computed
241
-
242
- // recall nudge (existing)
243
- results = db.search([repo, "global"], embedding, { limit: 5 })
244
- if matches >= RECALL_THRESHOLD -> inject recallNudge
245
-
246
- // prediction fire (new)
247
- predItems = db.scorePredictionNudge([repo, "global"], embedding, PREDICTION_THRESHOLD)
248
- if predItems.length > 0:
249
- output.parts.push(predictionNudge(predItems)) // synthetic: true
250
- ```
251
-
252
- One more cosine scan against a separate table. Same brute-force
253
- pattern, same cost profile. No extra embedding call.
254
-
255
- ## System prompt instructions
256
-
257
- New section in all three prompt variants (systemPrompt,
258
- claudeInstructions, cursorInstructions), after "Before Responding."
259
-
260
- New tools registered in the tool list line:
261
- `thatch_prediction_query`, `thatch_prediction_update`,
262
- `thatch_prediction_list` (opencode); bare names for MCP.
263
-
264
- The injected block format:
265
-
266
- ```
267
- ## User decision model
268
- - [0.72 conf, 4 tests] When reviewing a PR with pre-existing tech debt: you tend to skip it unless it's blocking the change
269
- - [0.61 conf, 2 tests] When the tech debt is a flaky test: you've preferred fixing it
270
- ```
271
-
272
- Prompt instructions:
273
- - Follow strong predictions silently.
274
- - Surface weak or competing predictions to the user naturally.
275
- - When the user responds to a surfaced prediction, use
276
- prediction_update to reinforce or weaken.
277
- - When the user corrects you, answers a question, or provides a clear
278
- signal, use prediction_query then prediction_update.
279
-
280
- ## Constants
281
-
282
- ```
283
- PREDICTION_THRESHOLD = 0.45 // lower than RECALL_THRESHOLD (0.55)
284
- MATCHER_DEDUP_COSINE = 0.85 // near-duplicate check at creation
285
- PREDICTION_DEDUP_COSINE = 0.85
286
- P0 = 0.5 // prior (population-derived later)
287
- K = 5 // prior strength
288
- W_SOFT = 0.25 // soft miss weight
289
- ```
290
-
291
- ## Hook integration
292
-
293
- The `tool.execute.after` hook (index.ts) already skips
294
- `thatch_*`-prefixed tools from the extraction buffer.
295
- `thatch_prediction_*` is caught by that check -- no special handling
296
- needed. MCP path same: `mcp__thatch__prediction_*` caught by the
297
- `mcp__thatch__*` filter in extract-queue.ts.
298
-
299
- Prediction calls do NOT drain the extraction buffer -- they are a
300
- separate system. The agent may do both in the same turn (save a memory
301
- AND update a prediction).
302
-
303
- ## Build status
304
-
305
- Landed so far:
306
-
307
- 1. DB schema + methods (db.ts) -- four new tables (prediction_matchers,
308
- predictions, prediction_edges, prediction_provenance) in #initSchema,
309
- plus findMatchers, scorePredictions, findNearestMatcher,
310
- findNearestPrediction, createMatcher, createPrediction, createEdge,
311
- adjustConfidence, addProvenance, listPredictions, populationP0,
312
- getPrediction, getProvenance, deletePrediction, scorePredictionNudge.
313
- Confidence uses Bayesian posterior with k=5, p0=0.5, W_SOFT=0.25.
314
- 2. Tool definitions (tool-defs.ts) -- prediction_query,
315
- prediction_update, prediction_list, prediction_delete. Added to
316
- TOOL_DEFS array (9 -> 13).
317
- 3. Auto-fire (index.ts) -- prediction fire in chat.message hook,
318
- sharing embedding with recall nudge. Prediction blocks injected
319
- when matchers clear PREDICTION_THRESHOLD (0.45). predictionNudge
320
- formatter in prompts.ts.
321
- 4. System prompt (prompts.ts) -- "User Decision Model" section in all
322
- three variants (systemPrompt, claudeInstructions,
323
- cursorInstructions). New tools in all three tool lists.
324
- 5. Tools wiring (tools.ts + mcp.ts) -- automatic via TOOL_DEFS
325
- iteration. No manual changes needed.
326
- 6. Tests -- tests/prediction.test.ts (schema, confidence math, dedup,
327
- scoring, provenance, listPredictions, populationP0). Updated
328
- tool-defs.test.ts and plugin.test.ts tool count assertions (9 -> 13).
329
- 7. MCP path (sideband.ts + bin/thatch) -- the sideband server now
330
- handles `method: "predictions"` alongside `method: "match"`, so
331
- the warm MCP server can query matchers and score predictions for
332
- one-shot hook processes. The `flush-tools` CLI subcommand (already
333
- wired into UserPromptSubmit hooks for Claude Code and Cursor) now
334
- fires predictions alongside the recall nudge in its tier 2 block,
335
- sharing one sideband round-trip via `Promise.all`. A standalone
336
- `flush-predictions` subcommand is also available for testing.
337
- No new hook installation needed -- the existing `flush-tools` hook
338
- covers it.
339
-
340
- Future enhancements:
341
-
342
- 1. **p0 population derivation** -- v1 uses flat 0.5. Once enough
343
- predictions accumulate, switch to population-derived p0 per store.
344
- 2. **Auto-fire block separation** -- separate synthetic part from the
345
- recall nudge, or combined? Starting with separate (simpler).
346
- 3. **Inspector UI** -- `thatch prediction list` CLI subcommand for
347
- terminal inspection. No Svelte UI (thatch has no UI).
348
- 4. **Score formula tuning** -- v1 is `cosine * weight * confidence`.
349
- If confidence dominates (high confidence on weak cosine swamps low
350
- confidence on strong cosine), add samskara's `cosine^1.3` power.
351
- Start simple; let data drive.
352
- 5. **Edge weight tuning** -- v1 is flat 1.0 for all edges. The weight
353
- encodes context-dependence but we have no signal to set it yet.
354
- Could later set it from the matcher's cosine at creation time
355
- (stronger match = higher weight) or from co-occurrence statistics.
356
-
357
- ## Interactions
358
-
359
- - **Memory** (existing) -- distinct system. Memory is facts the
360
- agent/user committed; predictions are learned preferences with graded
361
- confidence. No data flows between them. The agent may write a memory
362
- about a preference AND update a prediction about the same preference;
363
- the memory is the "what" (durable fact), the prediction is the "how
364
- confident" (graded, updated by feedback).
365
- - **Extraction pipeline** (existing) -- prediction tools are excluded
366
- from the extraction buffer by the thatch_* prefix filter. No
367
- feedback loop.
368
- - **Recall nudge** (existing) -- the prediction fire shares the
369
- embedding call and runs in the same chat.message hook path (opencode)
370
- or the same sideband round-trip (MCP). The two nudges are separate
371
- synthetic parts (opencode) or combined in one output block (MCP);
372
- either can be absent.
373
- - **Sideband** (existing, MCP path) -- the sideband server now handles
374
- a `predictions` method alongside `match`. The `flush-tools` CLI
375
- subcommand queries both in parallel via `Promise.all`.
376
- - **Compaction guard** (existing) -- the compacting set check at the
377
- top of chat.message suppresses ALL nudges, including prediction fire.
378
- No change needed.
379
-
380
- ## Gotchas
381
-
382
- - **Matcher embeddings use passageEmbed, not queryEmbed.** Matchers
383
- are passages (stored content matched against), not queries. The query
384
- prefix applies to the user's text at fire time. Same pattern as
385
- remember() which uses passageEmbed for content.
386
- - **confirm_count / disconfirm_count MUST be REAL.** An integer column
387
- truncates soft signals (W_SOFT = 0.25) to 0 and freezes confidence at
388
- p0. This is the same bug samskara had.
389
- - **Cold start is free.** Empty matchers table, findMatchers returns
390
- [], nothing injected. The feature is inert until the first
391
- prediction_update call. No migration, no config.
392
- - **Prediction calls do not drain the extraction buffer.** They are a
393
- separate system. The agent may do both in the same turn.
394
-
395
- ## Reference
396
-
397
- - nak samskara: `docs/dev/samskara.md` in the nak repo (the predictive
398
- model of the user this feature is inspired by).
399
- - nak intents: `docs/dev/in-progress/intents.md` in the nak repo (the
400
- normative layer atop samskara; the C efficacy model and firewall
401
- between employment and efficacy informed the confidence model here).
402
- - thatch memory: this doc's "Interactions" section.
403
- - thatch extraction pipeline: `docs/dev/README.md` + `docs/dev/mcp-parity.md`.
404
- - thatch hook surface: `docs/dev/setup-and-hooks.md`, `docs/dev/gotchas.md`.
@@ -1,106 +0,0 @@
1
- # Plan 001 — Memory Store
2
-
3
- ## Synopsis
4
-
5
- Thatch is an opencode plugin providing persistent memory via local embeddings and
6
- SQLite. Stores are containers keyed by git repo identity (`owner/repo`) with a
7
- shared `global` store for cross-project knowledge.
8
-
9
- ## Decisions
10
-
11
- ### Scope
12
-
13
- - **OpenCode plugin only** — no standalone CLI, no MCP mode. Runs entirely inside
14
- opencode's Bun runtime.
15
- - Distributed as an npm package. Users add `"thatch"` to their `opencode.json`
16
- `plugin` array.
17
-
18
- ### Store naming
19
-
20
- - `<owner>/<repo>` parsed from `git remote get-url origin`.
21
- - Worktree-safe: `git rev-parse --git-common-dir` finds canonical repo root.
22
- - Fallback chain: git remote → git common dir basename → CWD basename.
23
- - Stores are created implicitly on first write (no explicit create tool).
24
-
25
- ### Tool surface
26
-
27
- **`thatch_store_*`** — container management
28
-
29
- | Tool | Behavior |
30
- |------|----------|
31
- | `thatch_store_list` | All stores in the database |
32
-
33
- **`thatch_memory_*`** — CRUD on memories within a store
34
-
35
- | Tool | Behavior |
36
- |------|----------|
37
- | `thatch_memory_remember` | Embed content, upsert into store. Rejects duplicate labels unless `overwrite: true`. |
38
- | `thatch_memory_recall` | Embed query, cosine-similarity across `[repo-store]` + `global`, return top-N. |
39
- | `thatch_memory_list` | Labels + metadata in a store. |
40
- | `thatch_memory_show` | Full content by exact label. |
41
- | `thatch_memory_forget` | Delete by label. |
42
-
43
- Default search scope for `recall` is always repo store + global. Any tool
44
- accepts an explicit `store` param to override the default.
45
-
46
- ### Tech stack
47
-
48
- | Layer | Choice |
49
- |-------|--------|
50
- | SQLite | `bun:sqlite` (built-in, zero dependency) |
51
- | Embeddings | `@huggingface/transformers` v4, `Xenova/bge-small-en-v1.5` (384-dim, 34 MB) |
52
- | Vector search | Hand-rolled cosine similarity on Float32Array (5 lines, no dependency) |
53
- | Plugin API | V1 (`server` export) — custom tools via `tool()` helper |
54
- | Validation | Zod via `tool.schema` from `@opencode-ai/plugin` |
55
-
56
- ### Database
57
-
58
- - **Path**: `~/.config/thatch/thatch.db` (single SQLite file, WAL mode)
59
- - **Schema**: `stores(name TEXT PK)`, `entries(slug TEXT, store TEXT, label TEXT, content TEXT, embedding BLOB, model TEXT, branch TEXT, confidence INTEGER, created_at TEXT, updated_at TEXT, PRIMARY KEY (slug, store))`
60
- - Embeddings stored as raw Float32Array bytes in BLOB columns.
61
-
62
- ### BGE model handling
63
-
64
- - Query embedding: prefixed with `"Represent this sentence for searching relevant passages: "`
65
- - Passage (memory) embedding: no prefix
66
- - Model lazy-loaded on first `remember` or `recall` call.
67
- - Model files cached by Hugging Face Hub in `~/.cache/huggingface/`.
68
-
69
- ### Test principles
70
-
71
- - Tests never reach outside the sandbox or make network calls.
72
- - SQLite tests use `:memory:` databases.
73
- - Embedding model is mocked in tests.
74
- - Git detection tests use temp directories with stub repos.
75
- - `HOME` is overridden to a temp directory for path resolution.
76
-
77
- ## Architecture
78
-
79
- ```
80
- src/index.ts V1 plugin entry — exports `server` function
81
- src/git.ts Repo name detection
82
- src/db.ts SQLite schema, CRUD, brute-force search
83
- src/embeddings.ts BGE model lazy-load, embed API
84
- src/tools.ts Tool definitions (thatch_memory_*, thatch_store_*)
85
- ```
86
-
87
- Each module accepts its dependencies explicitly (DB path, repo name, model
88
- handle) so tests can inject mocks. The plugin entry wires them together with
89
- real defaults.
90
-
91
- ## Dependencies
92
-
93
- ### Runtime
94
-
95
- - `@huggingface/transformers` ^4 — `package.json` dependency
96
-
97
- ### Built-in (no install needed)
98
-
99
- - `bun:sqlite` — Bun built-in
100
- - `@opencode-ai/plugin` — provided by opencode runtime (devDependency for types)
101
-
102
- ### Dev
103
-
104
- - `@opencode-ai/plugin` ^1 — types
105
- - `@types/bun` — Bun type definitions
106
- - TypeScript
@@ -1,59 +0,0 @@
1
- # Plan 002 — Agent-Driven Pipelines
2
-
3
- ## Synopsis
4
-
5
- Fact extraction and deduplication run through the agent, not through plugin
6
- background machinery. The plugin observes, buffers, and nudges; the agent does
7
- all memory writes via the ordinary tools. Supersedes plan 001's "no standalone
8
- CLI" (a CLI now ships in `bin/thatch`) and codifies decisions made during the
9
- 2026-07 repair of the dead extraction/dedup wiring.
10
-
11
- ## Background
12
-
13
- The original implementation contained half-wired background machinery:
14
- `applyClassification`/`applyActions` pipelines that were never called, an
15
- advisory-locks table for coordinating background work across sessions, and
16
- event handlers listening for events that don't exist. Rather than finish that
17
- design, it was amputated in favor of the agent-driven flow the tool + skill
18
- surface already implied.
19
-
20
- ## Decisions
21
-
22
- | Decision | Rationale |
23
- |----------|-----------|
24
- | Plugin never writes memories autonomously | The agent has context the plugin lacks (what's durable vs ephemeral); background writes would need their own LLM calls, cost controls, and conflict handling. Nudge-the-agent gets classification for free. |
25
- | No locks table | Only agents write, through single-shot tool calls; SQLite WAL + busy_timeout covers concurrent tool execution. Advisory locks existed to serialize background jobs that no longer exist. |
26
- | Dedup verdicts in `dedup_pairs`, recorded via `thatch_dedup_mark_checked` | `find_duplicates` must converge: reviewed-and-kept pairs (supplement/contradiction/unrelated) would otherwise re-report forever. |
27
- | Overwrite/forget invalidates verdicts for that slug | A verdict describes content that no longer exists; stale verdicts would suppress genuinely new duplicates. |
28
- | Embedding spaces discriminated by vector dimension, not model tag | Zero-migration safety for `THATCH_MODEL` changes: mismatched entries are skipped, never NaN-ranked. Model tags are unreliable (historical rows carry a short tag, new rows the full HF name). |
29
- | Extraction buffers are per-session, thatch tools excluded | Concurrent sessions must not claim each other's interactions; buffering thatch's own writes would echo the store back into itself. |
30
- | Nudge carries the serialized payload inline | The skill contract is "you will be given a JSON payload"; a nudge that flushes the buffer without delivering it destroys the data it advertises. |
31
- | Skills installed to `$XDG_CONFIG_HOME/opencode/skills`, plugin-owned | Installing into the worktree dirties user repos. Overwrite-on-drift lets skill updates ship with plugin versions; the cost (clobbering local edits) is accepted. |
32
- | Hook failures logged, never swallowed | Two integration points were dead for weeks behind bare `catch {}`. |
33
-
34
- ## Hook wiring (hard-won specifics)
35
-
36
- - `tool.execute.after` is a **top-level plugin hook**; it does not exist on
37
- the event bus. Registering it inside the `event` handler compiles and never
38
- fires.
39
- - `session.created`'s payload is `{ properties: { info: Session } }` — the id
40
- is `properties.info.id`, not `properties.id`.
41
- - Repo detection must use the `worktree` the plugin receives; `process.cwd()`
42
- is the opencode server's cwd and misfiles every store when the server isn't
43
- started from the project directory.
44
-
45
- ## Architecture
46
-
47
- ```
48
- observe: tool.execute.after → ExtractionPipeline (per-session ring buffer, 20)
49
- nudge: chat.message → flush buffer → synthetic text part with JSON payload
50
- act: agent + thatch-fact-extractor skill → thatch_memory_remember
51
-
52
- surface: thatch_find_duplicates → candidates minus checked pairs
53
- act: agent + thatch-dedup-classifier skill → remember/forget
54
- settle: thatch_dedup_mark_checked → dedup_pairs verdict
55
- ```
56
-
57
- ## Dependencies
58
-
59
- Unchanged from plan 001. The amputation removed code, not dependencies.
@@ -1,61 +0,0 @@
1
- # Plan 003 — Memory Hygiene
2
-
3
- ## Synopsis
4
-
5
- Prevents stale and fragmented memory from accumulating: a write-time
6
- similarity warning, recall telemetry, a session-start hygiene heartbeat, and
7
- cluster-level dedup presentation. Extends plan 002's invariant — the plugin
8
- observes and nudges, the agent judges and writes — to store maintenance.
9
-
10
- ## Background
11
-
12
- Before this, the only hygiene mechanism was the manual dedup loop
13
- (`thatch_find_duplicates` → classifier skill → `thatch_dedup_mark_checked`),
14
- which nothing ever triggered, caught only pairwise similarity, and had no
15
- concept of staleness or usage. Memories that stopped being true, fragmented
16
- topics below the pair threshold, and branch-scoped leftovers accumulated
17
- unbounded.
18
-
19
- ## Decisions
20
-
21
- | Decision | Rationale |
22
- |----------|-----------|
23
- | Write-time collision **warns**, never blocks | The embedding is already computed at save time, so the store scan is nearly free. The save proceeds and the warning rides the tool response — the agent decides whether and how to reconcile. Blocking would add a special case to the remember contract and stall workflows on a heuristic. |
24
- | `findSimilar` is separate from `recall` and records no telemetry | The collision check is the plugin looking, not the agent using; counting it as usage would make every write refresh its own neighbors. |
25
- | Recall telemetry (`recall_count`, `last_recalled_at`) stamped inside `recall()` | Retrieval is the "used recently" signal. Stale = neither written nor recalled since the cutoff, so actively-used old memories never nag. |
26
- | Heartbeat rides the session-start reminder | The reminder already fires once per session via `session.created`; hygiene counts piggyback with zero new machinery. Only non-zero signals are reported — a healthy store stays silent. |
27
- | Staleness window hardcoded at 90 days | Zero-config posture. Revisit if real usage disagrees. |
28
- | Orphaned-branch detection skips non-git worktrees | `listBranches` returning `[]` is "unknown", not "no branches"; treating it as empty would mark every scoped memory orphaned. |
29
- | Branch cleanup is agent-driven, not automatic GC | Deleting user memories automatically violates plan 002's core invariant. The heartbeat names the orphaned branches; the agent (or user) decides. |
30
- | Clusters are presentation-only | `findDuplicates` still returns pairs and verdicts stay pairwise (`dedup_pairs` unchanged) — connected components are computed in the tool layer for display, so no schema or verdict-model churn. |
31
- | Column migration reintroduced | Existing databases predate the telemetry columns; `CREATE TABLE IF NOT EXISTS` alone can't add them. `PRAGMA table_info` + `ALTER TABLE ADD COLUMN`, idempotent. |
32
-
33
- ## Architecture
34
-
35
- ```
36
- write: remember tool → passageEmbed → findSimilar(store, emb) ─┐
37
- db.remember(...) → "[saved] ..." │
38
- + "⚠ similar to: ..." warning when hits ≥ 0.85 ◄────────┘
39
-
40
- read: recall() → top-N → UPDATE recall_count+1, last_recalled_at
41
-
42
- session: session.created → hygieneReport(db, repo, worktree)
43
- • findDuplicates count (pending pairs)
44
- • staleEntryCount (90d, max(updated_at, last_recalled_at))
45
- • branchesInStore − listBranches(worktree) → orphan count
46
- → appended to sessionStartReminder as "[thatch hygiene] ..."
47
-
48
- dedup: findDuplicates pairs → union-find in tool layer → clusters
49
- cluster of 3+ → skill instructs consolidate-into-one
50
- ```
51
-
52
- ## Costs
53
-
54
- - `hygieneReport` runs `findDuplicates` (O(n²) cosine) once per session
55
- start; ~100ms at a few thousand entries. Acceptable; revisit with an
56
- index/cache if stores grow past that.
57
- - `recall()` now writes on read (one UPDATE per recall). WAL absorbs this.
58
-
59
- ## Dependencies
60
-
61
- Unchanged.