@jeffober/thatch 0.1.27 → 0.1.29

package/docs/dev/features/cli.md

@@ -0,0 +1,81 @@
+# CLI (bin/thatch)
+
+The thatch CLI is a Bun script at `bin/thatch`. It provides memory inspection, MCP server startup, hook commands for MCP hosts, and the setup installer.
+
+## Subcommands
+
+15 subcommands:
+
+| Command | Args | Flags | Stdin | Purpose |
+|---------|------|-------|-------|---------|
+| `stores` | none | none | none | List all store names |
+| `list [store]` | optional store | none | none | List memory labels in a store |
+| `show <label> [store]` | label, optional store | none | none | Display one memory by label |
+| `forget <label> [store]` | label, optional store | none | none | Delete one memory by label |
+| `search <query> [store]` | query, optional store | none | none | Semantic cosine search (limit 10; "all" searches project + global) |
+| `mcp` | none | none | none | Start the stdio MCP server (for Claude Code, Cursor) |
+| `reminder [--json]` | none | `--json` | none | Print session-start reminder + hygiene report |
+| `hygiene` | none | none | none | Print the hygiene report standalone |
+| `prime` | none | none | none | Run thatch-project-primer skill via opencode/agent/claude CLI |
+| `buffer-batch` | none | none | JSON | Append PostToolBatch payload to queue (Claude Code hook) |
+| `buffer-tool` | none | none | JSON | Append single postToolUse interaction to queue (Cursor hook) |
+| `flush-tools [--json]` | none | `--json` | JSON | Peek queue + extraction/recall/prediction/behavior/write nudge |
+| `flush-predictions [--json]` | none | `--json` | JSON | Standalone prediction-only nudge |
+| `setup --claude [--cursor] [--global]` | none | `--claude`, `--cursor`, `--global` | none | Install config + instructions + hooks + skills |
+| (unknown) | none | none | none | Print usage, exit 1 |
+
+## Global behavior
+
+- No `--version`, `--help`, `-h`, or short flags. No subcommand aliases.
+- Unknown command or missing required arg calls `usage()` and exits 1.
+- Args are positional, parsed by hand from `process.argv.slice(2)`. No arg-parsing library.
+- DB opened once at startup, closed at end (except `mcp` which closes early).
+- Stores default to git remote detected by `detectRepo()`; "unknown" on failure. "global" is the shared store. "all" (search only) means project + global.
+
+## prime
+
+Runs the thatch-project-primer skill via an external CLI. Searches `PATH` in order: `opencode`, `agent` (Cursor CLI), `claude`. Uses the first found:
+- opencode: `opencode run "<primer prompt>"`
+- agent: `agent -p "<primer prompt>" --approve-mcps`
+- claude: `claude "<primer prompt>"`
+
+Inherits stdio, exits with the child's exit code. Errors and exits 1 if none found.
+
+## Environment variables
+
+| Variable | Purpose | Default |
+|----------|---------|---------|
+| `XDG_CONFIG_HOME` | Config home base | `~/.config` |
+| `THATCH_DB_PATH` | SQLite DB file location | `$XDG_CONFIG_HOME/thatch/thatch.db` |
+| `THATCH_MODEL` | Embedding model override | `Xenova/bge-small-en-v1.5` |
+| `THATCH_RECALL_THRESHOLD` | Cosine cutoff for recall nudge | `0.55` |
+| `THATCH_PREDICTION_THRESHOLD` | Cutoff for prediction auto-fire | `0.60` |
+| `THATCH_BEHAVIOR_THRESHOLD` | Cutoff for behavior auto-fire | `0.60` |
+| `CLAUDE_PROJECT_DIR` | Claude Code project dir | `process.cwd()` |
+| `CURSOR_PROJECT_DIR` | Cursor project dir | falls through |
+| `CLAUDE_CONFIG_DIR` | Claude config dir | `~/.claude` |
+| `CURSOR_CONFIG_DIR` | Cursor config dir | `~/.cursor` |
+| `OPENCODE_EXPERIMENTAL_BACKGROUND_SUBAGENTS` | Enable async extraction | unset |
+
+## Source files
+
+- `bin/thatch` — the CLI (Bun script, 486 lines)
+- `bin/release` — release helper (bash, separate from the main CLI)
+
+## Interactions with other features
+
+- Memory store ([memory-store.md](memory-store.md)): `stores`, `list`, `show`, `forget`, `search` subcommands
+- Setup ([setup.md](setup.md)): `setup` subcommand
+- Nudge pipeline ([nudge-pipeline.md](nudge-pipeline.md)): `flush-tools`, `flush-predictions`, `reminder` subcommands
+- Extraction ([extraction.md](extraction.md)): `buffer-batch`, `buffer-tool`, `flush-tools` subcommands
+- Hygiene ([hygiene.md](hygiene.md)): `hygiene`, `reminder` subcommands
+- Multi-host ([multi-host.md](multi-host.md)): `mcp` subcommand starts the MCP server for Claude Code and Cursor
+
+## Key invariants
+
+- No arg-parsing library. Args are positional, parsed by hand from `process.argv.slice(2)`.
+- No `--version`, `--help`, `-h`, or short flags. No subcommand aliases.
+- DB opened once at startup, closed at end (except `mcp` which closes early).
+- Store name auto-detected from git remote. "global" is the shared store. "all" (search only) means project + global.
+- Unknown command or missing required arg calls `usage()` and exits 1.
+- `prime` searches PATH in order: `opencode`, `agent`, `claude`. Uses the first found.

package/docs/dev/features/compaction-recovery.md

@@ -0,0 +1,48 @@
+# Compaction Recovery (opencode)
+
+opencode compacts long sessions by generating a summary and replacing the conversation history. Without intervention, a compacted session would lose awareness of thatch -- the agent would stop using memory tools, extraction would stop, and nudges would fire during summary generation where tools are blocked.
+
+## What it does
+
+- Suppresses nudges during compaction (tools are blocked during summary generation)
+- Injects compaction context so the agent retains awareness of thatch after compaction
+- Clears the compacting flag on success (three mechanisms, belt-and-suspenders)
+- Handles compaction failure (clears stale flag if autocontinue never fired)
+
+## How it works
+
+### Four hooks/events
+
+1. **experimental.session.compacting** (hook) -- fires when opencode starts compacting. Adds sessionID to the `compacting` set. Pushes `compactionContext(repo)` into `output.context`. The context tells the agent to preserve what's been learned and decided this session so there's continuity after compaction.
+
+2. **experimental.compaction.autocontinue** (hook) -- fires after compaction succeeds and the auto-continue turn fires. Removes sessionID from the `compacting` set so nudges resume.
+
+3. **session.compacted** (event) -- fires on successful compaction. Clears the `compacting` flag. Belt-and-suspenders: if autocontinue didn't fire or wasn't installed, the event still clears the flag.
+
+4. **Compaction failure fallback** (chat.message hook) -- if a non-compaction chat.message arrives while the session is in the `compacting` set, compaction failed (autocontinue never fired). The hook clears the stale `compacting` flag and proceeds normally. Without this, the session would be stuck with nudges off forever.
+
+### Why nudges are suppressed during compaction
+
+During summary generation, tools are blocked. A nudge that instructs the agent to use `thatch_memory_recall` or `thatch_memory_remember` would cause a blocked-tool error. The `compacting` set is checked at the top of the chat.message hook -- if the session is compacting, all nudges are skipped.
+
+### Compaction context
+
+`compactionContext(repo)` in `src/prompts.ts` generates text that tells the agent: you are using thatch, here's what you've learned and decided this session, preserve this context after compaction. This is pushed into `output.context` (not `output.system`) so it's part of the compaction output, not the permanent system prompt.
+
+## Interactions with other features
+
+- Nudge pipeline ([nudge-pipeline.md](nudge-pipeline.md)): compaction guard is the first check in the chat.message hook; suppresses all 4 tiers
+- Session lifecycle ([session-lifecycle.md](session-lifecycle.md)): `session.compacted` event is handled in the event handler
+- Multi-host ([multi-host.md](multi-host.md)): compaction recovery is opencode-only. Claude Code and Cursor do not have this feature. Mitigation: `CLAUDE.md`/`AGENTS.md` persists through compaction, and SessionStart hooks fire after compaction.
+
+## Source files
+
+- `src/index.ts` -- all four hooks/events (experimental.session.compacting, experimental.compaction.autocontinue, session.compacted event, chat.message compaction guard)
+- `src/prompts.ts` -- compactionContext function
+
+## Key invariants
+
+- The `compacting` set is checked at the top of chat.message. If set, all nudges are skipped.
+- Three mechanisms clear the flag (autocontinue, session.compacted event, chat.message fallback). Belt-and-suspenders.
+- Compaction failure fallback: a non-compaction message during compacting = compaction failed. Clear and proceed.
+- opencode-only feature. MCP hosts rely on static instructions persisting through compaction.

package/docs/dev/features/database.md

@@ -0,0 +1,244 @@
+# Database
+
+Thatch stores all data in a single SQLite file. The schema has 11 tables organized around three concerns: memory entries, the prediction engine, and the behavior engine.
+
+## Configuration
+
+- Default path: `$XDG_CONFIG_HOME/thatch/thatch.db` (override via `THATCH_DB_PATH`)
+- PRAGMAs: `journal_mode=WAL`, `busy_timeout=5000`, `foreign_keys=ON`
+- Single file, no external services
+
+## Schema overview
+
+11 tables in three groups:
+
+| Group | Tables | Purpose |
+|-------|--------|---------|
+| Memory | `stores`, `entries`, `dedup_pairs` | Core memory CRUD + dedup verdicts |
+| Prediction engine | `prediction_matchers`, `predictions`, `prediction_edges`, `prediction_provenance` | User decision model |
+| Behavior engine | `behavior_matchers`, `behaviors`, `behavior_edges`, `behavior_provenance` | LLM self-discipline rules |
+
+The prediction and behavior engines share the same four-table shape (matchers, items, edges, provenance) with different table names. They are separate because the semantics differ: predictions model what the user wants; behaviors model what the LLM should do.
+
+## Memory tables
+
+### stores
+
+Registry of all store names. The `global` store is inserted automatically at schema init.
+
+```sql
+stores(name TEXT PRIMARY KEY)
+```
+
+### entries
+
+The core table. Memory entries with embeddings, metadata, and usage telemetry.
+
+```sql
+entries(
+  slug TEXT NOT NULL,
+  store TEXT NOT NULL REFERENCES stores(name),
+  label TEXT NOT NULL,
+  content TEXT NOT NULL,
+  embedding BLOB,
+  model TEXT,
+  branch TEXT,
+  confidence INTEGER,
+  archived INTEGER NOT NULL DEFAULT 0,
+  created_at TEXT NOT NULL,
+  updated_at TEXT NOT NULL,
+  recall_count INTEGER NOT NULL DEFAULT 0,
+  last_recalled_at TEXT,
+  PRIMARY KEY (slug, store)
+)
+```
+
+- `slug`: derived from label via `slugify()`. Unicode letters/digits preserved. All-symbol labels fall back to hash.
+- `embedding`: raw Float32Array bytes. Null if not yet embedded.
+- `model`: embedding model tag (informational; dimension is the real discriminator).
+- `branch`: git branch scope. Null = project-wide.
+- `confidence`: 1-10 scale. Null = unset.
+- `archived`: 0 = live, 1 = archived. Search, dedup, and staleness queries all exclude archived by default.
+- `recall_count` / `last_recalled_at`: usage telemetry, stamped by `db.recall()` (not `db.search()`).
+
+### dedup_pairs
+
+Records which duplicate-candidate pairs have been reviewed.
+
+```sql
+dedup_pairs(
+  store TEXT NOT NULL,
+  slug_a TEXT NOT NULL,
+  slug_b TEXT NOT NULL,
+  status TEXT NOT NULL,
+  checked_at TEXT NOT NULL,
+  PRIMARY KEY (store, slug_a, slug_b)
+)
+```
+
+- Slugs stored sorted (a < b) for canonical matching.
+- `status`: "duplicate", "supplement", "contradiction", or "unrelated".
+- Overwriting or forgetting either memory clears the verdict (DELETE in `remember()` and `forgetEntry()`).
+
+## Prediction engine tables
+
+### prediction_matchers
+
+Situation descriptions that trigger predictions. Embedded, cosine-matched at `chat.message`.
+
+```sql
+prediction_matchers(
+  id TEXT PRIMARY KEY,
+  store TEXT NOT NULL REFERENCES stores(name),
+  description TEXT NOT NULL,
+  embedding BLOB,
+  model TEXT,
+  created_at TEXT NOT NULL,
+  updated_at TEXT NOT NULL
+)
+```
+
+### predictions
+
+User preference predictions with Bayesian confidence.
+
+```sql
+predictions(
+  id TEXT PRIMARY KEY,
+  store TEXT NOT NULL REFERENCES stores(name),
+  statement TEXT NOT NULL,
+  rationale TEXT,
+  embedding BLOB,
+  model TEXT,
+  confidence REAL NOT NULL DEFAULT 0.5,
+  confirm_count REAL NOT NULL DEFAULT 0,
+  disconfirm_count REAL NOT NULL DEFAULT 0,
+  created_at TEXT NOT NULL,
+  updated_at TEXT NOT NULL
+)
+```
+
+- `confidence`: Bayesian posterior. `p = (confirm + K*P0) / (confirm + disconfirm + K)`, K=5, P0=0.5.
+- `confirm_count` / `disconfirm_count`: REAL (fractional for soft signals, weight 0.25).
+
+### prediction_edges
+
+Many-to-many links between matchers and predictions.
+
+```sql
+prediction_edges(
+  matcher_id TEXT NOT NULL REFERENCES prediction_matchers(id) ON DELETE CASCADE,
+  prediction_id TEXT NOT NULL REFERENCES predictions(id) ON DELETE CASCADE,
+  weight REAL NOT NULL DEFAULT 1.0,
+  created_at TEXT NOT NULL,
+  PRIMARY KEY (matcher_id, prediction_id)
+)
+```
+
+### prediction_provenance
+
+Audit trail of every signal applied to a prediction.
+
+```sql
+prediction_provenance(
+  id TEXT PRIMARY KEY,
+  prediction_id TEXT NOT NULL REFERENCES predictions(id) ON DELETE CASCADE,
+  signal TEXT NOT NULL,
+  detail TEXT,
+  created_at TEXT NOT NULL
+)
+```
+
+- `signal`: "confirm", "disconfirm", "soft", or "create".
+
+## Behavior engine tables
+
+Same shape as prediction engine, with behavior-specific semantics.
+
+### behavior_matchers
+
+```sql
+behavior_matchers(
+  id TEXT PRIMARY KEY,
+  store TEXT NOT NULL REFERENCES stores(name),
+  description TEXT NOT NULL,
+  embedding BLOB,
+  model TEXT,
+  created_at TEXT NOT NULL,
+  updated_at TEXT NOT NULL
+)
+```
+
+### behaviors
+
+```sql
+behaviors(
+  id TEXT PRIMARY KEY,
+  store TEXT NOT NULL REFERENCES stores(name),
+  statement TEXT NOT NULL,
+  rationale TEXT,
+  embedding BLOB,
+  model TEXT,
+  confidence REAL NOT NULL DEFAULT 0.5,
+  confirm_count REAL NOT NULL DEFAULT 0,
+  disconfirm_count REAL NOT NULL DEFAULT 0,
+  created_at TEXT NOT NULL,
+  updated_at TEXT NOT NULL
+)
+```
+
+### behavior_edges
+
+```sql
+behavior_edges(
+  matcher_id TEXT NOT NULL REFERENCES behavior_matchers(id) ON DELETE CASCADE,
+  behavior_id TEXT NOT NULL REFERENCES behaviors(id) ON DELETE CASCADE,
+  weight REAL NOT NULL DEFAULT 1.0,
+  created_at TEXT NOT NULL,
+  PRIMARY KEY (matcher_id, behavior_id)
+)
+```
+
+### behavior_provenance
+
+```sql
+behavior_provenance(
+  id TEXT PRIMARY KEY,
+  behavior_id TEXT NOT NULL REFERENCES behaviors(id) ON DELETE CASCADE,
+  signal TEXT NOT NULL,
+  detail TEXT,
+  created_at TEXT NOT NULL
+)
+```
+
+- `signal`: "confirm" (ham), "disconfirm" (spam), or "codify".
+
+## Schema migration
+
+`recall_count`, `last_recalled_at`, and `archived` are added to pre-existing databases by an idempotent column migration at init. The migration uses `PRAGMA table_info` to check for the column's existence, then `ALTER TABLE ADD COLUMN` if missing. This handles databases created before these columns existed.
+
+## Embedding serialization
+
+Embeddings are raw Float32Array bytes stored as BLOBs. Serialization honors `byteOffset`/`byteLength` — transformers.js can return views into larger tensor buffers, and serializing the whole backing buffer corrupts vectors. Always serialize the view's own bytes, not the underlying buffer.
+
+## Key invariants
+
+- Single SQLite file, WAL mode, 5s busy timeout, foreign keys ON.
+- Embedding spaces are discriminated by vector dimension, not model tag.
+- Embedding serialization honors `byteOffset`/`byteLength` (views into larger buffers).
+- Slugs are slugified labels; slug + store = composite PK.
+- Dedup verdicts auto-clear when either memory is overwritten or forgotten.
+- Archived memories excluded from search, dedup, and staleness by default.
+- Column migration is idempotent (`PRAGMA table_info` + `ALTER TABLE`).
+
+## Interactions with other features
+
+- Memory store ([memory-store.md](memory-store.md)): `entries` table is the core; `stores` table is the registry
+- Deduplication ([deduplication.md](deduplication.md)): `dedup_pairs` table tracks reviewed pairs
+- Prediction engine ([prediction-engine.md](prediction-engine.md)): 4 prediction tables
+- Behavior engine ([behavior-engine.md](behavior-engine.md)): 4 behavior tables
+- Hygiene ([hygiene.md](hygiene.md)): `staleEntryCount` uses `recall_count` and `last_recalled_at`; `branchesInStore` uses `branch` column
+
+## Source files
+
+- `src/db.ts` — all schema, CRUD, search, scoring, migrations

package/docs/dev/features/deduplication.md

@@ -0,0 +1,102 @@
+# Deduplication System
+
+Agent-driven detection and resolution of duplicate memories. The system
+surfaces candidates; the agent decides what to do.
+
+## What it does
+
+- `find_duplicates` tool: cosine similarity pairs, grouped into
+  connected-component clusters
+- `dedup_mark_checked` tool: records a pairwise verdict so `find_duplicates`
+  stops re-reporting
+- Automatic verdict clearing: overwriting either memory clears the verdict
+- `thatch-dedup-classifier` skill: classifies the relationship and reconciles
+  pairs
+
+## How it works
+
+### find_duplicates (tool)
+
+- Parameters: `store` (optional, default project store), `threshold`
+  (optional, default 0.85)
+- Finds memory pairs whose embeddings exceed the cosine similarity threshold
+- Groups related pairs into connected-component clusters: a cluster of 3+
+  labels usually means one topic fragmented across entries that should be
+  consolidated
+- Pairs already resolved via `dedup_mark_checked` are skipped
+- Output: clustered rendering — "Cluster of N:" then
+  `[score:N] labelA <-> labelB` lines
+- Read-only
+
+### dedup_mark_checked (tool)
+
+- Parameters: `label_a`, `label_b` (both required), `status` (one of
+  `duplicate`, `supplement`, `contradiction`, `unrelated`), `store` (optional)
+- Records a pairwise verdict so `find_duplicates` stops re-reporting that pair
+- Labels are slugified internally
+- Overwriting either memory later clears the verdict automatically
+  (DB-level: the `remember()` and `forgetEntry()` functions `DELETE` from
+  `dedup_pairs` for the affected slug)
+- Returns `[checked] store :: a <-> b -> status`
+
+### Verdict lifecycle
+
+1. `find_duplicates` surfaces a pair above threshold
+2. Agent loads the `thatch-dedup-classifier` skill
+3. Agent classifies the pair: `duplicate` (merge), `supplement` (keep both),
+   `contradiction` (resolve), `unrelated` (mark and ignore)
+4. For duplicates: agent merges via `memory_remember(overwrite: true)` +
+   `memory_forget` the other entry
+5. Agent calls `dedup_mark_checked` for the surviving pair
+6. Overwriting either memory clears the verdict → the pair can re-flag if the
+   merge created a new similarity
+
+### The dedup_pairs table
+
+- PK: `(store, slug_a, slug_b)` — slugs stored sorted (`a < b`) for canonical
+  matching
+- No FK to entries (slugs are compared by string, not by FK constraint)
+- Overwriting or forgetting an entry clears all verdicts involving that slug
+  (`DELETE` in `remember()` and `forgetEntry()`)
+
+### thatch-dedup-classifier skill
+
+- Shared skill (`artifacts/skills/thatch-dedup-classifier.md`)
+- Classifies the relationship between two similar memory entries
+- Provides a structured process: read both entries, compare content, decide
+  verdict, reconcile
+- Can be dispatched as a sub-agent by the extraction pipeline or loaded
+  directly by the agent
+
+## Interactions with other features
+
+- [Memory store](memory-store.md): `find_duplicates` searches stored memories;
+  merging uses `memory_remember(overwrite)` + `memory_forget`.
+- [Hygiene](hygiene.md): hygiene report counts pending dedup pairs (same
+  underlying `db.findDuplicates`).
+- Skills: `thatch-dedup-classifier` is the skill that guides the
+  reconciliation process.
+- [Extraction](extraction.md): extraction can trigger the dedup cycle when
+  write-time duplicate detection fires.
+
+## Source files
+
+- `src/db.ts` — `findDuplicates`, `markChecked`, `dedup_pairs` table
+- `src/tool-defs.ts` — 2 dedup tools (`find_duplicates`, `dedup_mark_checked`)
+- `artifacts/skills/thatch-dedup-classifier.md` — classifier skill
+
+## Database tables
+
+- `dedup_pairs(store, slug_a, slug_b, status, checked_at, PK(store, slug_a, slug_b))`
+
+## Key invariants
+
+- `find_duplicates` groups pairs into clusters (connected components). A
+  cluster of 3+ should be consolidated into one memory.
+- Verdicts are automatically cleared when either memory is overwritten or
+  forgotten. This is intended — a merge can create a new similarity that
+  should re-flag.
+- The threshold (0.85) is higher than the recall nudge threshold (0.55) because
+  "duplicate" is a stronger claim than "relates to".
+- Slugs in `dedup_pairs` are stored sorted (`a < b`) for canonical matching
+  regardless of argument order.