@jeffober/thatch 0.1.27 → 0.1.29

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (62) hide show
  1. package/README.md +24 -10
  2. package/artifacts/skills/thatch-code-review.md +9 -7
  3. package/artifacts/skills/thatch-fact-extractor.md +2 -2
  4. package/artifacts/skills/thatch-review-economy.md +97 -0
  5. package/artifacts/skills/thatch-review-followup.md +1 -1
  6. package/artifacts/skills/thatch-review-synthesizer.md +15 -5
  7. package/bin/qa-run +38 -0
  8. package/bin/thatch +13 -7
  9. package/docs/dev/README.md +76 -22
  10. package/docs/dev/features/README.md +33 -0
  11. package/docs/dev/features/behavior-engine.md +271 -0
  12. package/docs/dev/features/cicd.md +89 -0
  13. package/docs/dev/features/cli.md +81 -0
  14. package/docs/dev/features/compaction-recovery.md +48 -0
  15. package/docs/dev/features/database.md +244 -0
  16. package/docs/dev/features/deduplication.md +102 -0
  17. package/docs/dev/features/extraction.md +253 -0
  18. package/docs/dev/features/hygiene.md +81 -0
  19. package/docs/dev/features/memory-store.md +154 -0
  20. package/docs/dev/features/multi-host.md +228 -0
  21. package/docs/dev/features/nudge-pipeline.md +282 -0
  22. package/docs/dev/features/prediction-engine.md +253 -0
  23. package/docs/dev/features/qa-system.md +205 -0
  24. package/docs/dev/features/session-lifecycle.md +110 -0
  25. package/docs/dev/features/setup.md +199 -0
  26. package/docs/dev/features/sideband.md +104 -0
  27. package/docs/dev/gotchas.md +3 -3
  28. package/docs/dev/mcp-parity.md +7 -5
  29. package/docs/dev/setup-and-hooks.md +4 -4
  30. package/docs/dev/skills.md +10 -8
  31. package/docs/plans/README.md +2 -2
  32. package/docs/user/README.md +58 -20
  33. package/docs/user/behavior-engine.md +85 -0
  34. package/docs/user/cli.md +98 -0
  35. package/docs/user/code-review.md +73 -0
  36. package/docs/user/deduplication.md +70 -0
  37. package/docs/user/default-behaviors.md +75 -0
  38. package/docs/user/extraction.md +74 -0
  39. package/docs/user/hygiene.md +76 -0
  40. package/docs/user/memory.md +81 -0
  41. package/docs/user/prediction-engine.md +87 -0
  42. package/docs/user/setup.md +110 -0
  43. package/docs/user/skills.md +97 -0
  44. package/package.json +2 -2
  45. package/src/behavior.ts +120 -0
  46. package/src/db.ts +160 -417
  47. package/src/embeddings.ts +0 -41
  48. package/src/extraction.ts +2 -2
  49. package/src/git.ts +3 -1
  50. package/src/index.ts +66 -14
  51. package/src/mcp.ts +59 -2
  52. package/src/prediction.ts +127 -0
  53. package/src/prompts.ts +155 -200
  54. package/src/scoring-engine.ts +374 -0
  55. package/src/seed-behaviors.ts +339 -0
  56. package/src/setup.ts +2 -6
  57. package/src/sideband.ts +36 -8
  58. package/src/skills.ts +1 -0
  59. package/src/tool-defs.ts +235 -9
  60. package/src/tools.ts +11 -1
  61. package/src/vector-math.ts +30 -0
  62. package/docs/qa/README.md +0 -175
@@ -0,0 +1,253 @@
1
+ # Extraction Pipeline
2
+
3
+ Automated capture of durable facts from tool interactions. The agent does not
4
+ need to remember to save — the system buffers tool calls and nudges the agent
5
+ to extract memories from them.
6
+
7
+ The facts most worth remembering overwhelmingly surface in tool call results
8
+ (file contents, command output, API responses, git state) rather than in
9
+ conversation prose. So tool calls serve as both the **trigger** and the
10
+ **queue**. A fact that exists only in conversation never enters the buffer;
11
+ the system prompt instructs the agent to call `thatch_memory_remember`
12
+ directly for conversation-derived knowledge.
13
+
14
+ ## What it does
15
+
16
+ - Buffers every non-`thatch_*`, non-`skill`, non-`task` tool call for later
17
+ extraction
18
+ - Two paths: **direct extraction** via a child session (opencode, primary) and
19
+ **nudge-based extraction** (all hosts, fallback for opencode, primary for
20
+ MCP)
21
+ - The extraction nudge escalates: polite (0–1 missed), insistent (2),
22
+ all-caps shouting (3+)
23
+ - `thatch_get_extraction_payload` fetches queued interactions as JSON,
24
+ keeping the full payload out of the main session's context window
25
+ - `thatch_extraction_done` acknowledges and quiets the nudge
26
+ - AMQP-style buffer lifecycle for opencode: pending → accepted → completed,
27
+ with requeue on failure
28
+ - File-backed JSONL queue for MCP hosts (no cross-call state)
29
+ - Parent-child session drain: the child drains the parent's snapshot,
30
+ preserving interleaved-turn entries added after the snapshot was taken
31
+
32
+ ## How it works
33
+
34
+ ### Tool interaction buffering
35
+
36
+ After every tool execution, non-`thatch_*`, non-`skill`, non-`task` tool
37
+ calls are buffered for later extraction.
38
+
39
+ **opencode** — `tool.execute.after` hook in `src/index.ts`:
40
+
41
+ - In-memory ring buffer per session (`ExtractionPipeline` in
42
+ `src/extraction.ts`)
43
+ - Max 20 interactions per session
44
+ - For child sessions, also tracks new/updated/deleted metrics via
45
+ `childMetrics`
46
+ - `tool.execute.after` is a **plugin hook, not a bus event**. Moving it into
47
+ the `event` handler silently never fires it — the event bus has no such
48
+ event
49
+ - Filtering rationale: `thatch_*` tools would echo the store back into
50
+ itself; `skill`/`task` meta-tools would create a feedback loop (extraction
51
+ triggers a skill load, which gets buffered, which triggers another
52
+ extraction)
53
+
54
+ **MCP hosts** — `bin/thatch`, `src/extract-queue.ts`:
55
+
56
+ - Claude Code: `PostToolBatch` hook → `thatch buffer-batch` (reads JSON from
57
+ stdin: `{ session_id, tool_calls }`)
58
+ - Cursor: `postToolUse` hook → `thatch buffer-tool` (reads JSON from stdin,
59
+ single tool, uses `conversation_id`)
60
+ - File-backed JSONL queue under
61
+ `$XDG_CACHE_HOME/thatch/queue/<session>.jsonl` (max 20, oldest dropped)
62
+ - Silent on success (no stdout) so the agent loop is not delayed
63
+ - Filters the same tools as opencode (`mcp__thatch__memory_remember`,
64
+ `mcp__thatch__extraction_done`, `mcp__thatch__*`, `skill`, `task`, `agent`)
65
+
66
+ ### Direct extraction (opencode, primary path)
67
+
68
+ When a parent session goes idle (`session.status` idle event) with pending
69
+ buffer interactions:
70
+
71
+ 1. `triggerExtraction` adds the parent ID to the `extracting` set — this
72
+ suppresses the nudge in `chat.message`
73
+ 2. Peeks the buffer to count pending interactions
74
+ 3. Creates a child session via
75
+ `client.session.create({ parentID, title: "thatch-extraction" })`
76
+ 4. The `session.created` event fires, populating `childToParent` and
77
+ `parentSnapshots` (a snapshot of the full pending buffer at dispatch time)
78
+ 5. Adds the child ID to the `extractionChildren` set
79
+ 6. Prompts the child with `extractionDirectPrompt(count, sessionID)`
80
+ 7. If background sub-agents are enabled
81
+ (`OPENCODE_EXPERIMENTAL_BACKGROUND_SUBAGENTS`): `promptAsync`. Otherwise:
82
+ fire-and-forget `prompt`
83
+ 8. On prompt failure: `cleanupChild` removes the child from all maps and
84
+ deletes the child session
85
+
86
+ The child session then:
87
+
88
+ - Calls `thatch_get_extraction_payload` to fetch the queued interactions
89
+ - Runs the [thatch-fact-extractor](../skills.md) skill
90
+ - Writes memories via `thatch_memory_remember`
91
+ - Goes idle
92
+
93
+ On child idle (`session.status` idle event, extraction child):
94
+
95
+ 1. Drains the parent's snapshot from the pending buffer via
96
+ `consumeSnapshot` — removes only entries captured at dispatch time by
97
+ reference identity, preserving interleaved-turn entries
98
+ 2. Fires a toast with extraction metrics (only if memories were actually
99
+ written)
100
+ 3. `completeAccepted(parentID)`, resets `missedNudges`
101
+ 4. Cleans up all maps
102
+ 5. `consume(childID)` — drains the child's own buffer
103
+ 6. Deletes the child session
104
+
105
+ ### Nudge-based extraction (fallback for opencode, primary for MCP)
106
+
107
+ If direct extraction was never triggered or threw an error (session not in
108
+ the `extracting` set) **and** the buffer has pending interactions:
109
+
110
+ - The next user message (`chat.message` for opencode,
111
+ `UserPromptSubmit`/`beforeSubmitPrompt` for MCP) gets an extraction nudge
112
+ - The nudge carries the session ID and fetch tool name — not the full
113
+ payload — so the sub-agent calls `thatch_get_extraction_payload` to
114
+ retrieve the interactions as a tool response
115
+
116
+ **Nudge escalation** (via the `missedNudges` counter):
117
+
118
+ | Missed count | Tone |
119
+ |--------------|------|
120
+ | 0–1 | Polite |
121
+ | 2 | Insistent |
122
+ | 3+ | All-caps shouting |
123
+
124
+ The buffer persists until the agent writes a memory or calls
125
+ `thatch_extraction_done`. Ignored nudges repeat and escalate.
126
+
127
+ - **opencode**: in-memory `missedNudges` map
128
+ - **MCP hosts**: file-backed `.count` file per session
129
+
130
+ ### Buffer lifecycle (opencode, AMQP-style)
131
+
132
+ Buffered interactions move through four states in `ExtractionPipeline`
133
+ (`src/extraction.ts`):
134
+
135
+ - **pending** — interactions in the ring buffer. The nudge fires on
136
+ `chat.message`.
137
+ - **accepted** — moved from pending by `accept()`. The nudge quiets, but
138
+ entries are not dropped yet.
139
+ - **completed** — dropped by `completeAccepted()`. Triggered by a
140
+ `memory_remember` or `extraction_done` call in the child, or the child
141
+ going idle.
142
+ - **requeued** — moved back to pending by `requeueAccepted()`. Triggered by
143
+ the child session erroring or being deleted before completing.
144
+
145
+ Key methods:
146
+
147
+ | Method | Action |
148
+ |--------|--------|
149
+ | `push()` | Add interaction to pending buffer (capped at 20) |
150
+ | `peek()` | Read without clearing |
151
+ | `consume()` | Delete the session's pending buffer (called on memory write) |
152
+ | `accept()` | Move pending to accepted — quiet the nudge, hold entries |
153
+ | `completeAccepted()` | Drop accepted entries — extractor finished |
154
+ | `requeueAccepted()` | Move accepted back to pending — extractor died |
155
+ | `consumeSnapshot()` | Remove only entries that were in the snapshot (by reference identity). Used for child-parent drain. |
156
+
157
+ ### Parent-child session drain
158
+
159
+ When a child session writes a memory via `thatch_memory_remember`:
160
+
161
+ 1. `tool.execute.after` detects the memory write
162
+ 2. If in a child session (`childToParent.has(sessionID)`): track metrics,
163
+ `completeAccepted(parentID)`, drain the parent's snapshot via
164
+ `consumeSnapshot`, reset the parent's `missedNudges`
165
+ 3. `consume(sessionID)` — drains the child's own buffer
166
+
167
+ `consumeSnapshot` is **snapshot-aware**: it removes only entries that were
168
+ in the parent's buffer at dispatch time (by reference identity).
169
+ Interleaved-turn entries — added after the snapshot was taken, while the
170
+ child was extracting — survive. This prevents data loss when the parent
171
+ continues working while the child extracts.
172
+
173
+ ### MCP file-backed queue drain
174
+
175
+ - `drainExtractionQueue(sessionID)` calls `resetMissedCount` +
176
+ `consumeQueue` (deletes the JSONL file)
177
+ - Triggered by: `thatch_extraction_done` called with the parent's
178
+ `session_id`, or `thatch_memory_remember` called
179
+ - `appendBatch` in `extract-queue.ts` self-detects
180
+ `memory_remember`/`extraction_done` calls and resets the counter +
181
+ consumes the queue inline
182
+
183
+ ### The `extraction_done` tool
184
+
185
+ - No-op confirmation (`[acknowledged]`) unless `session_id` is passed and
186
+ `drainExtractionQueue` is wired
187
+ - On the MCP path with `session_id`: resets the missed-nudge count +
188
+ consumes (deletes) the file-backed queue
189
+ - The real state transitions happen in the host's post-tool hook
190
+ (`tool.execute.after` for opencode, `PostToolBatch`/`appendBatch` for MCP)
191
+ - The tool exists primarily so the model has a recognizable name to key on
192
+
193
+ ### The `get_extraction_payload` tool
194
+
195
+ - Fetches the queued tool interactions for extraction as serialized JSON
196
+ (`interactions`, `projectStore`, `globalStore`)
197
+ - **opencode**: peeks accepted + pending interactions, builds JSON via
198
+ `buildExtractionPayload`
199
+ - **MCP**: peeks the file-backed queue, builds the same JSON payload
200
+ - Returns `null` when no interactions are queued
201
+ - Read-only — it peeks the queue; it does not consume it. Consumption is
202
+ `extraction_done`'s job
203
+
204
+ ### Background sub-agent support (opencode)
205
+
206
+ - Experimental flag: `OPENCODE_EXPERIMENTAL_BACKGROUND_SUBAGENTS` (set in
207
+ `mise.toml`)
208
+ - When enabled: `promptAsync` for child session creation (async extraction)
209
+ - When disabled: fire-and-forget `prompt` (synchronous but unawaited)
210
+
211
+ ## Interactions with other features
212
+
213
+ - [Memory store](memory-store.md) — extraction writes memories via
214
+ `thatch_memory_remember`
215
+ - [Nudge pipeline](nudge-pipeline.md) — the extraction nudge is tier 1
216
+ (highest priority, returns early)
217
+ - [Session lifecycle](session-lifecycle.md) — direct extraction is triggered
218
+ by `session.status` idle; child lifecycle is managed by the event handler
219
+ - [Skills](../skills.md) — the thatch-fact-extractor skill is dispatched to
220
+ child sessions
221
+ - [Multi-host](multi-host.md) — in-memory ring buffer for opencode,
222
+ file-backed queue for MCP hosts
223
+ - [Compaction recovery](compaction-recovery.md) — extraction nudge is
224
+ suppressed during compaction (tools are blocked)
225
+
226
+ ## Source files
227
+
228
+ | File | Responsibility |
229
+ |------|----------------|
230
+ | `src/extraction.ts` | In-memory ring buffer (`ExtractionPipeline`), shared payload builders (`buildExtractionPayload`, `deriveTitle`, `summarizeArgs`) |
231
+ | `src/extract-queue.ts` | File-backed JSONL queue for MCP hosts |
232
+ | `src/index.ts` | opencode hooks: `tool.execute.after`, `session.status` idle, `session.created`, `session.error`, `session.deleted`, `chat.message` (nudge tier 1) |
233
+ | `bin/thatch` | `buffer-batch`, `buffer-tool`, `flush-tools` subcommands |
234
+ | `src/prompts.ts` | `extractionNudge` (with escalation), `extractionDirectPrompt` |
235
+
236
+ ## Key invariants
237
+
238
+ 1. **Tool filtering is absolute.** `thatch_*`, `skill`, and `task` tools are
239
+ never buffered. Buffering them would echo the store into itself or create
240
+ a feedback loop.
241
+ 2. **The nudge peeks, never flushes.** The buffer persists until a memory
242
+ write or `extraction_done`. Ignored nudges repeat and escalate.
243
+ 3. **`consumeSnapshot` is snapshot-aware.** It removes only entries captured
244
+ at dispatch time (by reference identity), preserving interleaved-turn
245
+ entries added while the child was extracting.
246
+ 4. **`tool.execute.after` is a plugin hook, not a bus event.** The event bus
247
+ has no such event. Moving the buffering logic into the `event` handler
248
+ silently never fires it.
249
+ 5. **MCP host hooks must be silent.** `PostToolBatch`/`postToolUse` produce
250
+ no stdout — only `flush-tools` prints. Any stdout delays the agent loop.
251
+ 6. **The no-save drain runs unconditionally.** The child-idle handler drains
252
+ the remaining snapshot regardless of whether the child wrote memories,
253
+ covering no-save extraction runs.
@@ -0,0 +1,81 @@
1
+ # Hygiene System
2
+
3
+ Store maintenance signals surfaced at session start. The plugin never deletes
4
+ memories itself — it surfaces problems and lets the agent tend the store.
5
+
6
+ ## What it does
7
+
8
+ - Three signals: duplicate candidates, stale entries, orphaned branch memories
9
+ - Runs at session start on all three hosts
10
+ - Only non-zero signals are reported (healthy stores stay silent)
11
+ - Standalone `thatch hygiene` CLI command for manual inspection
12
+
13
+ ## How it works
14
+
15
+ ### Three signals
16
+
17
+ 1. **Duplicate candidates** — `db.findDuplicates(repo)`: near-duplicate memory
18
+ pairs via cosine >= 0.85 pending review. These are the same pairs that
19
+ `find_duplicates` surfaces, but the hygiene report is a count-only summary,
20
+ not the full pair list.
21
+
22
+ 2. **Stale entries** — `db.staleEntryCount(repo, cutoff)`: memories neither
23
+ updated nor recalled in 90+ days (`STALE_DAYS = 90`). Uses `recall_count`
24
+ and `last_recalled_at` columns. Archived memories are excluded
25
+ (`WHERE archived = 0`).
26
+
27
+ 3. **Orphaned branch memories** — `db.branchesInStore(repo)` cross-references
28
+ against `git.listBranches(worktree)`. Memories scoped to branches that no
29
+ longer exist. Skipped when the worktree is not a git repo
30
+ (`listBranches` returns `[]`) to avoid false positives — without a git
31
+ repo, every branch-scoped memory would look orphaned.
32
+
33
+ ### When it runs
34
+
35
+ - **opencode**: `session.created` event for top-level sessions (no `parentID`).
36
+ The hygiene report is appended to the session-start reminder sent via
37
+ `client.session.prompt`.
38
+ - **Claude Code**: `SessionStart` hook → `thatch reminder` (runs
39
+ `hygieneReport`, appends to reminder text, prints to stdout).
40
+ - **Cursor**: `sessionStart` hook → `thatch reminder --json` (same, wrapped as
41
+ `additional_context` JSON).
42
+ - **Standalone**: `thatch hygiene` CLI command (prints report or
43
+ "Store is healthy." to stdout).
44
+
45
+ ### Report format
46
+
47
+ Only non-zero signals are reported. A healthy store produces no output
48
+ (silent). The report is appended to the session-start reminder with
49
+ instructions for the agent to tend the store when convenient.
50
+
51
+ ### Shared implementation
52
+
53
+ `hygieneReport(db, repo, worktree)` in `src/hygiene.ts` is shared by:
54
+
55
+ - The opencode plugin's session-start hook (`src/index.ts`)
56
+ - The CLI's `thatch reminder` and `thatch hygiene` subcommands (`bin/thatch`)
57
+
58
+ ## Interactions with other features
59
+
60
+ - [Deduplication](deduplication.md): hygiene reports the count of pending dedup
61
+ pairs; the agent uses `find_duplicates` to see the actual pairs.
62
+ - [Memory store](memory-store.md): staleness uses `recall_count` and
63
+ `last_recalled_at`; orphan check uses the `branch` column.
64
+ - Session lifecycle: hygiene runs at `session.created` for top-level sessions.
65
+ - Nudge pipeline: hygiene is part of the session-start reminder, not the
66
+ per-message nudge pipeline.
67
+
68
+ ## Source files
69
+
70
+ - `src/hygiene.ts` — `hygieneReport` function
71
+ - `src/db.ts` — `findDuplicates`, `staleEntryCount`, `branchesInStore`
72
+ - `src/git.ts` — `listBranches`
73
+ - `bin/thatch` — `hygiene` and `reminder` subcommands
74
+
75
+ ## Key invariants
76
+
77
+ - The plugin never deletes memories. It surfaces problems; the agent tends the
78
+ store.
79
+ - Only non-zero signals are reported.
80
+ - Orphan check is skipped outside a git repo.
81
+ - Archived memories are excluded from staleness and duplicate checks.
@@ -0,0 +1,154 @@
1
+ # Memory Store
2
+
3
+ The memory store is thatch's core feature: a SQLite-backed store of memories with semantic search via local embeddings. Everything else thatch does — extraction, hygiene, nudges, predictions, behaviors — reads from or writes to this store.
4
+
5
+ ## What it does
6
+
7
+ The store persists memories keyed by label, scoped to a project store or the shared "global" store. Each memory is a labeled chunk of text an agent writes for a future session of itself to read.
8
+
9
+ - Stores memories keyed by label within a store. The store name is auto-detected from the git remote — the `owner/repo` slug — or falls back to `"global"` for cross-project knowledge.
10
+ - Semantic search via cosine similarity over BGE-small-en-v1.5 embeddings (384 dimensions, ~34 MB). The model runs locally and is lazy-loaded.
11
+ - Six tools: `memory_remember`, `memory_recall`, `memory_list`, `memory_show`, `memory_forget`, and `store_list`. All are defined in `src/tool-defs.ts`.
12
+ - Branch scoping. Memories can be scoped to a git branch for work-in-progress context, or left unscoped for project-wide knowledge. Branch-scoped memories are recalled alongside project-wide ones when the branch matches.
13
+ - Confidence grading on a 1–10 scale, stored per memory.
14
+ - Archived flag. Archived memories are long-term records excluded from search and hygiene by default. Pass `includeArchived: true` to surface them.
15
+ - Write-time duplicate detection. When a new memory is similar to existing ones (cosine ≥ 0.85), the save returns a warning listing the near-duplicates. The save always proceeds — the warning asks the agent to reconcile, never blocks.
16
+ - Asymmetric search. Query text gets a BGE-specific prefix for retrieval; stored passages get no prefix. This follows the BGE training convention for asymmetric search.
17
+ - Multi-model dimension check. Entries embedded by a model with a different vector dimension are silently skipped during search, not ranked. The `model` column is informational — dimension is the discriminator.
18
+ - Recall tracking. Each memory carries `recall_count` and `last_recalled_at`, stamped when an agent-initiated search returns it. The hygiene system uses these to detect stale memories.
19
+
20
+ ## How it works
21
+
22
+ ### Store detection
23
+
24
+ `src/git.ts` parses `owner/repo` from the git remote URL. The resolution chain is worktree-safe:
25
+
26
+ 1. Parse `owner/repo` from `git remote get-url origin`.
27
+ 2. Fall back to the basename of the git common dir (worktree-safe).
28
+ 3. Fall back to the directory basename.
29
+ 4. Falls back to `"unknown"` if none of the above yield a result.
30
+
31
+ The "global" store always exists — it is inserted at schema init (`INSERT OR IGNORE INTO stores (name) VALUES ('global')`). Store creation is implicit: the first `memory_remember` to a new store calls `ensureStore`, which creates it via `INSERT OR IGNORE`.
32
+
33
+ ### Embedding model
34
+
35
+ `src/embeddings.ts` defines the `BgeEmbeddingModel` class.
36
+
37
+ - Model: `Xenova/bge-small-en-v1.5` — 384 dimensions, ~34 MB.
38
+ - Lazy-loaded via the `@huggingface/transformers` `pipeline("feature-extraction")` API. The first load downloads from Hugging Face Hub; subsequent loads use the cached model.
39
+ - Query prefix: `"Represent this sentence for searching relevant passages: "` — the BGE asymmetric search convention. Queries get the prefix; passages do not.
40
+ - Mean pooling and L2 normalization are applied via pipeline options.
41
+ - Output is a `Float32Array`.
42
+ - `MockEmbeddingModel` is available for tests.
43
+ - The model can be overridden via the `THATCH_MODEL` environment variable.
44
+
45
+ ### Write path (memory_remember)
46
+
47
+ ```text
48
+ agent calls memory_remember(label, content, opts)
49
+ → model.passageEmbed("# label\n\ncontent") → Float32Array
50
+ → db.findSimilar(store, embedding, { excludeSlug }) → write-time collision check
51
+ → db.remember(store, label, content, embedding, model.name, opts)
52
+ overwrite:false → atomic INSERT (PK constraint rejects duplicates)
53
+ overwrite:true → upsert + clear stale dedup verdicts for that slug
54
+ → return confirmation + warning if similar memories found
55
+ ```
56
+
57
+ The tool prepends `# ${label}\n\n` to the content before embedding. The stored `content` column holds this full formatted text — it is the exact input that was embedded, so a future re-embed requires no reconstruction.
58
+
59
+ The `findSimilar` call is a write-time collision check with no telemetry. It excludes the slug being written (self-exclusion on overwrite). If it finds entries with cosine ≥ 0.85, the return value includes a warning listing them.
60
+
61
+ The save always proceeds. The warning asks the agent to reconcile — merge into one entry (overwrite + forget the other) or record that they are genuinely distinct via `dedup_mark_checked`.
62
+
63
+ On overwrite, the upsert clears stale dedup verdicts: any `dedup_pairs` rows referencing that slug are deleted, since the content change may resolve or create duplicates.
64
+
65
+ ### Read path (memory_recall)
66
+
67
+ ```text
68
+ agent calls memory_recall(query, opts)
69
+ → model.queryEmbed(query) → Float32Array
70
+ → db.recall([repo, "global"], queryEmbedding, { branch?, limit })
71
+ → db.search() → cosine scoring, sort desc, top-N
72
+ → stamp recall_count + last_recalled_at on returned rows
73
+ → return formatted results with scores
74
+ ```
75
+
76
+ The tool layer hardcodes `[repo, "global"]` as the default search scope. Omitting the `store` parameter searches both. Passing a specific store searches only that store.
77
+
78
+ Entries with a mismatched embedding dimension are skipped during search — the `emb.length !== queryEmbedding.length` guard in `db.search()`. This handles the case where a different embedding model was used for older entries.
79
+
80
+ ### Search vs. recall
81
+
82
+ Two methods, one scoring engine, different telemetry policy:
83
+
84
+ | Method | Telemetry | Used by |
85
+ |--------|-----------|---------|
86
+ | `db.search()` | None | Prompt-aware recall nudge (see [nudge-pipeline.md](nudge-pipeline.md)) |
87
+ | `db.recall()` | Stamps `recall_count` / `last_recalled_at` | `thatch_memory_recall` tool, CLI `thatch search` |
88
+
89
+ The distinction is deliberate. The recall nudge checks whether memories relate to a prompt — the plugin is looking, not the agent. Stamping telemetry on a nudge would inflate the "used recently" signal that hygiene reporting depends on. Only explicit agent-initiated recall stamps telemetry.
90
+
91
+ ### Default recall scope
92
+
93
+ The tool layer hardcodes `[repo, "global"]` as the default search scope. This means a `memory_recall` call with no `store` argument searches both the project store and the global store. There is no way to search only the project store without the global store via the tool — pass an explicit `store` argument to scope to one.
94
+
95
+ ### Archived memories
96
+
97
+ The `archived` column is an integer flag: `0` (default) is live, `1` is archived.
98
+
99
+ - `search()`, `findDuplicates()`, and the stale-entry count in hygiene all filter `WHERE archived = 0`.
100
+ - To search archived memories: pass `includeArchived: true` to `memory_recall`.
101
+ - To archive: write with `archived: true`. To unarchive: write with `archived: false`.
102
+ - Updating an already-archived memory requires an explicit `archived` parameter. The guard prevents accidental unarchival — the caller must consciously pass `archived: true` to keep it archived or `archived: false` to unarchive it.
103
+
104
+ ### Slug generation
105
+
106
+ `slugify()` converts a label to a slug: lowercase, whitespace to hyphen, Unicode letters and digits preserved. All-symbol labels fall back to a hash so no label ever maps to an empty slug. ASCII labels produce the same slugs as earlier releases.
107
+
108
+ The slug plus store name form the composite primary key: `PRIMARY KEY (slug, store)`.
109
+
110
+ ### CLI access
111
+
112
+ The `bin/thatch` CLI provides read-only access to the store:
113
+
114
+ | Command | Description |
115
+ |---------|-------------|
116
+ | `thatch stores` | List all stores |
117
+ | `thatch list [store]` | List memory labels in a store |
118
+ | `thatch show <label> [store]` | Display one memory |
119
+ | `thatch forget <label> [store]` | Delete one memory |
120
+ | `thatch search <query> [store]` | Semantic cosine search (limit 10; `"all"` searches project + global) |
121
+
122
+ ## Interactions with other features
123
+
124
+ - **Extraction pipeline** ([extraction.md](extraction.md)): the extraction sub-agent writes memories via `memory_remember`. The extraction buffer drains on any `memory_remember` call.
125
+ - **Deduplication** ([deduplication.md](deduplication.md)): `find_duplicates` scans stored memories for cosine-similar pairs. `dedup_mark_checked` records verdicts in the `dedup_pairs` table. Overwriting or forgetting a memory clears its dedup verdicts.
126
+ - **Hygiene** ([hygiene.md](hygiene.md)): staleness checks use `recall_count` and `last_recalled_at` to find memories nobody has read recently. Duplicate checks use cosine search. Orphaned-branch detection finds branch-scoped memories whose branch no longer exists.
127
+ - **Nudge pipeline** ([nudge-pipeline.md](nudge-pipeline.md)): the recall nudge uses `db.search()` — not `db.recall()` — to avoid inflating telemetry. The nudge fires when search results exceed the similarity threshold (default 0.55).
128
+ - **Prediction engine** ([prediction-engine.md](prediction-engine.md)) and **behavior engine** ([behavior-engine.md](behavior-engine.md)): these share the same embedding model but store vectors in separate tables (`prediction_matchers`, `prediction_predictions`, `behavior_matchers`, `behavior_behaviors`). They do not read from or write to the `entries` table.
129
+
130
+ ## Source files
131
+
132
+ | File | Responsibility |
133
+ |------|---------------|
134
+ | `src/db.ts` | SQLite schema, CRUD for entries and stores, brute-force cosine search (`search` vs. `recall`), dedup verdict tracking, prediction and behavior tables |
135
+ | `src/embeddings.ts` | `BgeEmbeddingModel` (lazy-load, query/passage embed, mean pooling, L2 normalization) and `MockEmbeddingModel` for tests |
136
+ | `src/git.ts` | Detect repo identity (store name) from git remote; worktree-safe fallback chain |
137
+ | `src/tool-defs.ts` | Tool definitions — single source of truth for all tools, including the six memory tools |
138
+ | `bin/thatch` | CLI subcommands: `stores`, `list`, `show`, `forget`, `search` |
139
+
140
+ ## Key invariants
141
+
142
+ 1. **No global mutable state.** Every module accepts dependencies explicitly. `ThatchDB` takes a path; `BgeEmbeddingModel` takes an optional pipeline factory; tools receive a `ctx` with `db` and `model`.
143
+
144
+ 2. **Embedding is a separate concern.** `db.ts` stores and retrieves BLOBs and compares vectors handed to it. It knows nothing about embedding models — it never calls `passageEmbed` or `queryEmbed`. The tool layer is the bridge: it embeds, then hands the vector to the DB.
145
+
146
+ 3. **Embedding spaces are discriminated by vector dimension, not model tag.** The `model` column is written on every save and returned on every read, but no code path filters or branches on it. Dimension mismatch is the only guard. This means a same-dimension model switch produces silently wrong cosine scores, while a different-dimension switch produces silently skipped entries.
147
+
148
+ 4. **Store creation is implicit.** The first `memory_remember` to a new store creates it. There is no explicit "create store" operation.
149
+
150
+ 5. **Default recall scope is repo + global.** The tool layer hardcodes `[ctx.defaultStore, "global"]` when no `store` argument is provided. This is intentional — cross-project knowledge in the global store is always searched alongside project-specific memories.
151
+
152
+ 6. **The write-time similarity warning never blocks the save.** `findSimilar` returns a warning, `remember` proceeds regardless. The warning asks the agent to reconcile after the fact.
153
+
154
+ 7. **`search()` records no usage. `recall()` stamps telemetry. This is deliberate.** The recall nudge (plugin checking relevance) must not inflate the "used recently" signal. Only agent-initiated recall counts as actual usage.