claude_memory 0.13.0 → 0.13.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 93e553fc87bd36ebe27b8709a28dea9bd206a669491327fc8b6340e6078a8f67
-  data.tar.gz: c19722c5028ed5746f58792bb34e57939600deaa0a59b6ff00a421b3ed829bdc
+  metadata.gz: fef30295649f5cdb709f3650bbdfe7411ddccd566a906eaa09247a1b88f895a5
+  data.tar.gz: 3edd0ef999cb0c4356ef83c43e20646eed8cc8b68b9a2c8c1963191935eaaf98
 SHA512:
-  metadata.gz: 7bd3a98c636c525e2bfc579e9e265ecef3a07311d4ba543d692abc092ddc0822df86832b9e5b53d802597a6dbe7495aa192210d6d4e5961c21383daac22cef09
-  data.tar.gz: b587239c11c342551f35937ecdbb108254269306fa98fbaac6680d23f8e427a32f26cbb1524c8f19a08da1773e292b6d75df92e101cb724d59fd65017f275832
+  metadata.gz: e23ad823124ffec0b84d81d6f309dbc601ef5c475f827d21cfd4f9f9dbb7172a2c4832ded63c6f38bf9f46f060c5c4e04d94fe2bd9e2ed0044f7da58c38c70a8
+  data.tar.gz: d71ff604a9518affc8d153574dba34361b9b54b49fbffac228ff00c076ce05971b698782551a17bbfc072c84ce4f16e44680857671469db9d1d0c08e87cd9f94

data/.claude-plugin/marketplace.json

@@ -7,7 +7,7 @@
   "plugins": [
     {
       "name": "claude-memory",
-      "version": "0.13.0",
+      "version": "0.13.2",
       "source": "./",
       "description": "Long-term memory for Claude Code. Recalls architecture, conventions, and decisions across sessions, plus an episodic observation log of what happened — so Claude explains your codebase without file traversal, follows your patterns, learns from corrections, and never re-asks what it already learned.",
       "repository": "https://github.com/codenamev/claude_memory"

data/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-memory",
-  "version": "0.13.0",
+  "version": "0.13.2",
   "description": "Long-term memory for Claude Code. Recalls architecture, conventions, and decisions across sessions, plus an episodic observation log of what happened — so Claude explains your codebase without file traversal, follows your patterns, learns from corrections, and never re-asks what it already learned.",
   "author": {
     "name": "Valentino Stoll",

data/CHANGELOG.md

@@ -4,6 +4,31 @@ All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
 
+## [0.13.2] - 2026-06-27
+
+Theme: **Robustness from a real bug report (issue #7) + a 28MB→606K gem.** Closes [#7](https://github.com/codenamev/claude_memory/issues/7) — three independent failure modes around embeddings, FTS, and process lifecycle — plus two backlog items surfaced alongside it. No schema changes, no breaking changes.
+
+### Fixed
+
+- **Adopting a non-384-dim embedding model no longer hard-fails on an existing DB (#7 Finding 1).** The `facts_vec` vec0 table's width is immutable once created, and `embedding_dimensions` was only recorded *after* a successful index — so a tfidf/fresh DB silently created a 384 table and the first 768-dim insert raised `Expected 384 ... received 768`. `VectorIndex#recreate!` now drops + rebuilds `facts_vec` at the resolved width, and `IndexCommand#handle_dimension_mismatch` reads the table's actual width from its DDL and rebuilds up front (covering the unset-meta case, not just a recorded mismatch).
+- **Corrupt FTS5 rank index is now detected and self-heals (#7 Finding 2 + #69).** Concurrent writers can leave `content_fts` so that plain `MATCH` works but `ORDER BY rank` raises "database disk image is malformed" — silently breaking recall while `PRAGMA integrity_check` passes. `doctor` now probes the rank path per DB (`Checks::FtsRankCheck`); the sweep (`PreCompact`/`SessionEnd`) auto-rebuilds on detection (`Sweep::Maintenance#repair_fts_rank`) so recall self-repairs without a manual `compact`; and `LexicalFTS#search` raises a `CorruptRankIndexError` with a compact hint instead of an unhandled stacktrace.
+- **`serve-mcp` can no longer orphan (#7 Finding 3).** A hard kill of the client could leave the stdio MCP server blocked forever on `gets` (no stdin EOF), holding a SQLite connection — observed as serve-mcp processes lingering for days. A parent-death watchdog (`Server#orphaned?` + a 30s thread) exits the process once it's reparented away from its original parent.
+- **The embedding provider no longer leaks into the test suite.** `setup-vectors` wrote `CLAUDE_MEMORY_EMBEDDING_PROVIDER` into the tracked `settings.json`, which Claude Code injects into every subprocess env — flipping provider-dependent specs. It now writes to `settings.local.json` (per-machine), and the suite clears `CLAUDE_MEMORY_EMBEDDING_*` for hermetic runs.
+
+### Changed
+
+- **The published gem dropped from ~28MB to 606K (#71).** The gemspec's `git ls-files` manifest shipped the repo's own ~96MB dogfooding `.claude/memory.sqlite3` (and was trending toward RubyGems' 100MB ceiling). `.claude/` is now excluded from the manifest; users init their own DB, and the plugin manifest + commands/skills/output-styles (sourced at the top level) are unaffected. Regression-guarded by a manifest spec.
+
+## [0.13.1] - 2026-06-23
+
+Theme: **The observational layer, audited and repaired.** A critical examination of every observation in a real (dogfooding) project DB found the episodic layer was producing ~no useful observations and injecting noise into sessions — three evidence-backed defects, now fixed (the data is in `docs/improvements.md` #72–#75). No schema changes, no breaking changes.
+
+### Fixed
+
+- **High-precision Layer-1 observation filter (#74).** The high-recall Observer was scraping code, doc, and transcript fragments past `noise_body?` and injecting them into SessionStart (measured: 38 of 117 obvious-noise rows slipped through — spec fixtures, CHANGELOG table rows, benchmark tree output, even the distiller's own source comments). Strengthened the gate to reject code/JSON `key: "value"`, method calls, table pipes, box-drawing glyphs, `(vector)` labels, and raw JSONL fields, and to require a body to begin like a prose sentence. Verified against the real noise corpus: every sampled fragment now rejected, clean prose decisions/conventions kept.
+- **Corroboration can finally accumulate (#73).** Dedup matched on exact normalized strings, so varied wording of the same event never folded — every observation stayed at `corroboration_count = 1` and the promotion gate could never fire. Replaced exact grouping with greedy clustering over an injected similarity matcher; the default `Observe::TokenOverlapMatcher` (lexical Jaccard, deterministic, free) folds near-duplicates so corroboration climbs toward promotion. Pure synonym paraphrases still need real embeddings (measured: tfidf can't separate them from unrelated text on short bodies) — injectable via the `matcher:` seam.
+- **Observation capture elevated to a first-class SessionStart ask (#72).** Authoring observations was a paragraph buried in the optional deep-distill prompt, which fires almost never (`store_extraction` had zero calls in the layer's lifetime; Layer-1 auto-ingest carried ~100:1 of the load). Decoupled it into its own prominent `## Log What Happened` section. Whether the LLM-authored (Layer-2) path now fires is measurable via the `mcp_extraction` content-item source.
+
 ## [0.13.0] - 2026-06-18
 
 Theme: **Episodic memory — a second kind of memory.** ClaudeMemory gains an append-only *observation* layer ("what happened") that complements the semantic fact store ("what is true"), modeled on [Mastra's Observational Memory](docs/influence/mastra-observational-memory.md). Observations accrue automatically, are deduplicated/consolidated by reflection, and are promoted to facts only after corroboration — making repeated sighting an anti-hallucination gate built into the memory model. Schema advances to v20 (additive; no breaking changes to existing facts/queries).

data/docs/improvements.md

@@ -294,6 +294,8 @@ Source: 2026-04-28 1.0 readiness review (`docs/1_0_punchlist.md` #6)
 
 ### 69. Self-Heal the FTS Rank Index After Concurrent Ingest
 
+**✅ Core shipped 2026-06-27** (detection + self-heal). **Proactive detection:** the doctor `Checks::FtsRankCheck` probes `MATCH … ORDER BY rank` per DB and flags corruption integrity_check misses (landed with issue #7 Finding 2, `2522044`). **Self-heal:** `Sweep::Maintenance#repair_fts_rank` probes the rank path each sweep (PreCompact/SessionEnd) and runs `LexicalFTS#rebuild!` on `malformed`, so recall self-repairs within the session that broke it — no manual `compact`. Also graceful: `LexicalFTS#search` now raises `CorruptRankIndexError` with a compact hint instead of a stacktrace. **Deferred:** the contention-reduction pragmas (busy_timeout + `BEGIN IMMEDIATE`) and the larger external-content-FTS5 redesign — they're risk/scope beyond the self-heal and not needed once the rank index auto-repairs.
+
 Source: 2026-06-16 live incident on `claude/observational-layer-design-7662r9` — observed first-hand, not a study.
 
 **Gap.** The contentless FTS5 index (`content_fts`) silently drifts into a broken state under concurrent writers: the ingest hook (`claude-memory hook ingest`) and the MCP server (`store_extraction` → `Index::LexicalFTS#index_content_item`) both write the same WAL DB, and a large ingest produces `"Database busy, retrying"` (`Store::RetryHandler`) followed by an FTS index where **plain `MATCH` works but `... ORDER BY rank` raises `database disk image is malformed`**. `integrity_check` passes and all rows are intact — so `recall`/`recall_index` ranking is silently degraded (the rank query throws or returns nothing) while nothing looks wrong. The only fix today is the user manually running `claude-memory compact` (which `rebuild_fts` + vacuums). Documented in `docs/influence/...` gotchas and surfaced reactively in the dashboard (`lib/claude_memory/dashboard/api.rb:338`), but never repaired automatically. Severe form (btree corruption, plain `MATCH` also failing) was separately seen when **two** memory MCP servers ran concurrently — de-duping to a single server removed that, but the benign rank-artifact still recurs from hook-vs-MCP contention alone.
@@ -504,6 +506,87 @@ Source: `docs/influence/mastra-observational-memory.md` — architecture study o
 
 ---
 
+### 71. Exclude the project DB from the published gem (gem is 28MB, ~96MB of it the dogfooding DB)
+
+**✅ Shipped 2026-06-27.** Added `.claude/` to the gemspec reject filter; the published gem dropped from **28MB → 606K**. The plugin manifest (`.claude-plugin/`) and top-level `commands/`/`skills/`/`output-styles/` are preserved (the plugin sources them there, not from `.claude/`); runtime `.claude/output_styles` references resolve against the *user's* project dir. Regression-guarded by `spec/claude_memory/gemspec_manifest_spec.rb`.
+
+Source: 2026-06-18 live observation while building the 0.13.0 release gem.
+
+**Problem.** `claude_memory.gemspec` builds its file list from `git ls-files` and rejects `bin/ Gemfile .gitignore .rspec spec/ .github/ .standard.yml` — but **not** `.claude/memory.sqlite3`, which is tracked (per the "always commit the project DB" convention). So the published gem *ships the repo's own dogfooding memory database*: the working-tree DB is ~96MB, compressing to a **28MB gem** (v0.6.0 was 280KB; the gem has been silently growing — 0.9.1 was 19MB — as the DB accumulates). Gem users get nothing from it (they init their own empty DB on install), it bloats every download, and it's trending toward RubyGems' 100MB ceiling.
+
+**Fix.** Add `.claude/` (or at least `.claude/memory.sqlite3` + WAL/SHM siblings) to the gemspec reject filter. Verify with `gem build` that the gem drops to <1MB and that nothing in the gem actually requires the file at runtime (it shouldn't — runtime opens the *user's* DB path via `Configuration`). Add a spec asserting `Gem::Specification.load(...).files` excludes `.claude/memory.sqlite3` so it can't regress.
+
+**Why High.** Low effort, high impact: ~28MB → <1MB published gem, and it removes a slow-growing landmine before it actually exceeds the RubyGems size limit and blocks a release. Not introduced by 0.13.0 — pre-existing and compounding.
+
+**Note on the convention.** This does *not* conflict with "always commit `.claude/memory.sqlite3`" — that's about repo reproducibility for collaborators. Shipping it *in the gem* is a separate, unintended consequence of the `git ls-files` manifest.
+
+---
+
+> **Observational-layer audit (2026-06-23).** A critical examination of every observation in this project's DB found the episodic layer is, in practice, producing ~no useful observations and is injecting noise into sessions. Four root causes (#72–#75), each backed by the live data below. Snapshot at audit time: **113 active observations, 0 consolidated, 0 expired, 0 promoted, every `corroboration_count = 1`**; only `decision`/`preference` kinds; every row traces to a `claude_code` transcript on the `observational-layer-*` branches. The count grew 99 → 105 → 113 *during* the audit session — the dogfooding loop is live and compounding. These supersede the optimistic framing of #68; the mechanism is sound, the inputs and the matching are not.
+
+### 72. Layer-2 (Claude-as-observer) produces **zero** observations — the quality source is silent ⭐
+
+Source: 2026-06-23 observational-layer audit.
+
+**🟡 Structural fix shipped 2026-06-23** (option a). Decoupled the observation-capture ask from the buried, rarely-fired deep-distill paragraph into its own prominent SessionStart section (`ContextInjector#format_observation_capture_prompt` — "## Log What Happened"). This maximizes the chance Claude authors observations, but persistence still rides a `store_extraction` tool call, so **effectiveness is not yet proven** — whether Layer-2 now actually fires is measurable via the `mcp_extraction` content-item source and needs real-session validation (and ultimately the #75 eval). Not closing this until that signal turns positive.
+
+**Problem.** The design's quality observations were always meant to come from **Layer-2** (Claude-as-observer): the SessionStart prompt (`ContextInjector#format_distillation_prompt`) asks Claude to populate the `observations` field of its `memory.store_extraction` call. **The vehicle is dormant.** Evidence (sharpened 2026-06-23 with `mcp_tool_calls.called_at` + `activity_events`):
+- `store_extraction` was invoked **4 times ever — all on 2026-04-17 to 2026-04-30**, i.e. *six-plus weeks before the observational layer (and the `observations` parameter) shipped on 2026-06-16/18*. Those calls **could not** have carried observations; the field didn't exist yet.
+- **Since the layer shipped, `store_extraction` has fired ZERO times.** Layer-2 has never run, not once, in the feature's entire life. (Corroborating: `store_extraction` creates a synthetic `source: "mcp_extraction"` content_item; **0 of the 113 observations trace to `mcp_extraction`** — all to `claude_code` ingest.)
+- **Layer-1 auto-ingest dominates ~100:1**: `activity_events` shows **409 `hook_ingest` vs 4 `store_extraction` total**. Content flows in automatically on every Stop/SessionEnd hook without Claude's involvement, so the Layer-2 deep-distill path — gated on a *fresh session* with *undistilled ≥200-char* items that Claude must *choose* to act on — essentially never triggers.
+
+**Why this is the highest-leverage finding.** Layer-1 (regex over raw transcript) *cannot* produce episodic narrative — it only scrapes fragments (see #74; #74 makes it high-precision, not narrative). The design delegated quality to Layer-2, but Layer-2 is structurally dormant: it isn't a prompt-wording problem (the `store_extraction` schema *does* expose `observations` with a good description, and the prompt *does* ask) — it's that **the path the observations ride on doesn't fire** in normal operation. So the episodic log is, and will remain, 100% Layer-1 scrapes until observation authoring is moved onto a path that actually runs.
+
+**Fix (design fork — not a one-shot code change).** Options:
+- **(a) Author observations on the Layer-1 hook path, but with an LLM.** The hook can't call Claude (no API budget), so this means: have the *next* SessionStart context inject the raw undistilled tail and ask Claude to emit observations directly as part of the normal turn (not gated behind a voluntary `store_extraction` deep-distill). Rides the session, no extra cost — same mechanism the fact-injection uses.
+- **(b) Make Layer-2 fire reliably** — lower the fresh-session/≥200-char gate, or make observation emission a first-class, early, non-optional instruction. Risk: effectiveness is unmeasurable without real A/B sessions, and the headless-recall gap (`project_headless_retrieval_gap.md`) says Claude often won't call MCP tools at all.
+- **(c) Derive observations from what Claude already produces** — the `decisions`/`facts` it extracts are higher-signal than regex scrapes; synthesize observations from those deterministically.
+- Regardless: **add telemetry distinguishing Layer-1 vs Layer-2 observation provenance** so "is Layer-2 firing?" is a dashboard number, not a forensic dig.
+
+**Cross-links.** Blocks the value premise of #68; #74/#73 only make the Layer-1 floor less bad and the loop functional — neither makes the log *good*. This is the one that does.
+
+### 73. Observation dedup/corroboration is normalized-**exact**, so the promotion loop can never fire ⭐
+
+**✅ Shipped 2026-06-23.** Replaced exact-string grouping with greedy clustering over an injected similarity matcher (`Reflector#dedupe_scope`). Default matcher is `Observe::TokenOverlapMatcher` — lexical Jaccard token-overlap (deterministic, free, no embedding dependency), threshold 0.5. Folds the common case (one event re-observed with slightly different wording → corroboration now accumulates and can cross the promotion gate) while keeping unrelated statements apart (Jaccard ~0). **Deliberate limitation, data-driven:** measured that tfidf cosine (0.32) can't separate pure synonym paraphrases from unrelated pairs (0.13) on short bodies, so neither the default lexical matcher nor tfidf folds "use SQLite" vs "chose SQLite" — that needs real embeddings, which can be injected via the `matcher:` seam (any object responding to `similar?(a, b)`) when fastembed is configured.
+
+Source: 2026-06-23 observational-layer audit.
+
+**Problem.** `Observe::Reflector#dedupe` folds observations by `group_by { [scope, normalize(body)] }` where `normalize` is just `downcase` + whitespace-collapse + strip. Two observations corroborate **only if their bodies are byte-identical after lowercasing**. Real captures of "the same thing" are never byte-identical — e.g. the four stored variants "PreCompact hook set.", "PreCompact hook set — the design's Mastra-token-threshold analog.", "PreCompact set alongside ingest + sweep." describe one event but never fold. Result, confirmed in the data: **every observation has `corroboration_count = 1`; 0 consolidated; 0 promoted.** The corroboration gate — the layer's headline anti-hallucination feature — is **dead by construction** on any varied text. It can only fire if the *exact same string* recurs, which regex fragments from different transcript chunks essentially never do.
+
+**Fix.** Corroboration/dedup must be **semantic**, not exact: reuse the existing embedding stack (`Embeddings` + sqlite-vec) to fold observations above a similarity threshold, or fold on a normalized *subject+kind* key rather than the full body. Until then, the promotion gate provides no value and the "graduate after 2 sightings" story is unsupported. Add a spec that two paraphrases of one event corroborate.
+
+**Cross-links.** Without this, #72's quality observations still wouldn't promote.
+
+### 74. Layer-1 Observer ingests code/doc/transcript fragments; `noise_body?` lets ~⅓ through
+
+**✅ Shipped 2026-06-23** (commit `d81a684`). Strengthened `NOISE_BODY_SIGNATURE` (code/JSON `key: "value"`, method calls, spaced table pipes, box-drawing glyphs, `(vector)` labels, JSONL fields) and added a prose-start requirement. Verified against the audit corpus: all five real noise samples now rejected, clean prose kept. **Residual (not a regression):** *truncated-prose* fragments with no code signature (e.g. "encompasses how to use fr…") can still slip — that's the greedy `.+` capture, and ultimately the Layer-2 question (#72), not the noise filter.
+
+Source: 2026-06-23 observational-layer audit.
+
+**Problem.** The Layer-1 Observer runs `decided to (.+)` / `we always|never (.+)` over **raw transcript text**, which on this repo (and any repo whose sessions discuss code) is saturated with trigger phrases inside source, specs, docs, and tool output. The `noise_body?` filter (`NOISE_BODY_SIGNATURE = /\bdef\s|\bclass\s|\bmodule\s|=>|::|","|":\s*"|[{}]|\$\(|&&|\|\|/`) is tuned for code-*syntax* and misses prose/table/transcript fragments. Measured against the live 113: the filter catches **39**, but **38 obvious-noise rows slip through** (≈44% look like noise by a conservative heuristic; manual review puts it higher). Concrete slipped examples actually sitting in the injected log:
+- `[89] decided to use SQLite", kind: "decision", priority: 1) expect(id).to be_a(Integer)…` — a **spec fixture line**.
+- `[104] decided to gate promotion on corroboration" | | Changes | Explicitly…` — a **CHANGELOG table row** (`| |` ≠ `||`, so it dodges the filter).
+- `[48]–[55] / first-person `we always|never`)…` — fragments of the **distiller's own source-code comment**.
+- `[99] · (vector) 78 ├─ How frozen_string_literal…` — **benchmark tree output**.
+
+These are priority-1 `decision` rows, so they *are* injected into Block 1 of SessionStart (observed live in this session's own context) — spending context budget on garbage and risking misdirection (e.g. `[46] decided to use Postgres.`, a fixture string, implying a stack the project doesn't use).
+
+**Fix.** Make Layer-1 high-precision-or-silent: reject bodies that look like code/markdown/transcript (leading `-`/`#`/`|`, table pipes, `key: "value"` shapes, tree glyphs `├─└─`, `(vector)`, backtick-dense spans, JSONL artifacts) — invert the default from high-recall to high-precision, since the recall here is ~all noise. Pair with the `ContentSanitizer`/Observer border. (This is the P1 item from the 2026-06-18 quality review, now empirically confirmed and worse than estimated.)
+
+**Cross-links.** Even fully fixed, Layer-1 is a stopgap until #72; together they decide whether the log is signal or noise.
+
+### 75. The episodic layer has no fair test — this repo is a pathological self-pollution case
+
+Source: 2026-06-23 observational-layer audit.
+
+**Problem.** Every observation traces to this project's *own* `claude_code` design transcripts, whose specs literally contain `insert_observation(body: "decided to use SQLite")` and whose docs are full of "decided to…" prose. claude_memory dogfooding on its own repo is the **worst possible self-test** for the Observer — it maximizes trigger-text density and self-ingestion. So the audit above measures *self-pollution*, not the design's ceiling; a normal Rails/Django app would look very different. We currently have **no measurement of the layer's value on a representative project**, and the optimistic compression/promotion story in #68 was never validated.
+
+**Fix.** Stand up the deferred **LongMemEval-style episodic suite** (#67/#68 medium item) and/or capture a real non-claude_memory project trace as a fixture, and report observation precision (signal vs noise), corroboration/promotion rates, and compression on *that*. Treat "episodic layer adds value" as **unproven** in public materials until this exists (the 0.13.0 blog draft already hedges accordingly). Until then, the self-pollution makes the dashboard Observations panel actively misleading on this repo.
+
+**Cross-links.** Gates any future episodic value claim; depends on #72–#74 being fixed first to be worth measuring.
+
+---
+
 ## Medium Priority
 
 ### ~~18. Shell Completion for CLI~~ ✅ Implemented 2026-03-20

data/lib/claude_memory/commands/checks/fts_rank_check.rb

@@ -0,0 +1,60 @@
+# frozen_string_literal: true
+
+module ClaudeMemory
+  module Commands
+    module Checks
+      # Probes the FTS5 BM25 ranking path (`MATCH ... ORDER BY rank`) — exactly
+      # what recall issues. A contentless FTS5 index can corrupt such that plain
+      # MATCH, `SELECT count(*)`, and `PRAGMA integrity_check` all pass but
+      # `ORDER BY rank` raises "database disk image is malformed", silently
+      # breaking recall while every other check reports healthy (issue #7,
+      # Finding 2). Recoverable with `claude-memory compact`.
+      class FtsRankCheck
+        PROBE_TERM = "the" # extremely common; matches any non-trivial English corpus
+
+        def initialize(db_path, label)
+          @db_path = db_path
+          @label = label
+        end
+
+        def call
+          return skip_result unless File.exist?(@db_path)
+
+          store = Store::SQLiteStore.new(@db_path)
+          begin
+            Index::LexicalFTS.new(store).search(PROBE_TERM, limit: 1)
+            ok_result
+          ensure
+            store.close
+          end
+        rescue Index::LexicalFTS::CorruptRankIndexError
+          corrupt_result
+        rescue => e
+          # Anything unrelated (e.g. no FTS table) is not this check's concern.
+          {status: :ok, label: fts_label, message: "#{@label} FTS5 rank probe skipped: #{e.message}", details: {}}
+        end
+
+        private
+
+        def fts_label = "#{@label}_fts"
+
+        def ok_result
+          {status: :ok, label: fts_label, message: "#{@label} FTS5 rank path: healthy", details: {}}
+        end
+
+        def corrupt_result
+          {
+            status: :error,
+            label: fts_label,
+            message: "#{@label} FTS5 rank index is corrupt — recall is broken (integrity_check misses this). Run 'claude-memory compact' to rebuild.",
+            details: {}
+          }
+        end
+
+        def skip_result
+          {status: :ok, label: fts_label, message: "#{@label} FTS5: no database", details: {}}
+        end
+      end
+    end
+  end
+end

data/lib/claude_memory/commands/doctor_command.rb

@@ -22,6 +22,8 @@ module ClaudeMemory
           Checks::DatabaseCheck.new(manager.project_db_path, "project"),
           Checks::DistillCheck.new(manager.global_db_path, "global"),
           Checks::DistillCheck.new(manager.project_db_path, "project"),
+          Checks::FtsRankCheck.new(manager.global_db_path, "global"),
+          Checks::FtsRankCheck.new(manager.project_db_path, "project"),
           Checks::VecCheck.new,
           Checks::EmbeddingsCheck.new,
           Checks::SnapshotCheck.new,

data/lib/claude_memory/commands/index_command.rb

@@ -77,12 +77,30 @@ module ClaudeMemory
         run_indexing(store, facts, generator, tracker, operation_id, checkpoint, opts)
       end
 
+      # Reconcile the vec0 table's width with the resolved provider before
+      # indexing (issue #7, Finding 1). The vec0 width is immutable once the
+      # table is created and was only recorded in meta *after* a successful run,
+      # so an old tfidf/fresh DB silently created a 384 table and the first
+      # non-384 insert hard-failed. We detect the table's actual width directly
+      # (not via the meta, which may be unset) and rebuild when it differs.
       def handle_dimension_mismatch(store, generator, label)
-        check = Embeddings::DimensionCheck.call(store, generator)
-        return unless check.status == :mismatch
+        target = generator.dimensions
+        # Record the resolved dimension up front so a fresh table is created at
+        # the right width on first insert, not left at the 384 default.
+        store.set_meta("embedding_dimensions", target.to_s)
+        store.set_meta("embedding_provider", generator.name)
 
-        stdout.puts "#{label.capitalize}: Embedding dimensions changed (#{check.stored} → #{check.current}), clearing stale embeddings..."
-        clear_stale_embeddings(store)
+        vec_index = store.vector_index
+        return unless vec_index.available?
+
+        actual = vec_index.table_dimensions
+        return if actual == target # table already at the right width
+
+        if actual # genuine change: existing facts must re-embed at the new width
+          stdout.puts "#{label.capitalize}: Embedding dimensions changed (#{actual} → #{target}), rebuilding vector table..."
+          clear_stale_embeddings(store)
+        end
+        vec_index.recreate!(target)
       end
 
       def find_facts_to_index(store, tracker, label, opts)

data/lib/claude_memory/commands/setup_vectors_command.rb

@@ -14,8 +14,8 @@ module ClaudeMemory
     #   1. Verify the chosen provider is loadable. For fastembed, surface
     #      a clear install command if the gem isn't on $LOAD_PATH.
     #   2. Persist CLAUDE_MEMORY_EMBEDDING_PROVIDER (and optional model)
-    #      into the project's .claude/settings.json env block, the same
-    #      mechanism Claude Code uses for OTel env (see OTel::SettingsWriter).
+    #      into the project's .claude/settings.local.json env block (per-machine,
+    #      gitignored — the provider is a developer choice, not a repo default).
     #   3. Re-embed existing facts under the new provider (unless --no-reindex).
     #   4. Report the final state — provider, dimensions, stored alignment.
     class SetupVectorsCommand < BaseCommand
@@ -110,8 +110,14 @@ module ClaudeMemory
         false
       end
 
+      # Write to settings.local.json (per-machine, gitignored), NOT the tracked
+      # settings.json: the embedding provider is a developer choice. Committing
+      # it would force the optional fastembed dependency on collaborators and —
+      # because Claude Code injects settings env into every subprocess — leak a
+      # non-default provider into the test suite, flipping provider-dependent
+      # specs (issue #7 follow-up, 2026-06-25).
       def settings_path
-        File.join(claude_dir, "settings.json")
+        File.join(claude_dir, "settings.local.json")
       end
 
       def claude_dir

data/lib/claude_memory/distill/null_distiller.rb

@@ -49,8 +49,24 @@ module ClaudeMemory
         /\bwe\s+(?:should\s+)?(?:always|never)\s+(.+)/i
       ].freeze
 
-      # Bodies that look like code / JSON / shell rather than a statement.
-      NOISE_BODY_SIGNATURE = /\bdef\s|\bclass\s|\bmodule\s|=>|::|","|":\s*"|[{}]|\$\(|&&|\|\|/
+      # Bodies that look like code / JSON / shell / markup / transcript rather
+      # than a prose statement. High-precision gate: the Layer-1 observer scrapes
+      # raw transcript spans, which on a code-heavy project are dominated by
+      # source, specs, docs, and tool output — none of which are observations.
+      # (2026-06-23 audit, improvements #74: the prior signature let 38/117
+      # obvious-noise rows through — spec fixtures like `kind: "decision"`,
+      # CHANGELOG table rows, benchmark tree output, the distiller's own source
+      # comments — and they were being injected into SessionStart.)
+      NOISE_BODY_SIGNATURE = Regexp.union(
+        /\bdef\s|\bclass\s|\bmodule\s/,                      # Ruby definitions
+        /=>|::|","|":\s*"|[{}]|\$\(|&&|\|\|/,                # code / JSON / shell punctuation
+        /\w+:\s*["\[{\d]/,                                   # code/JSON key: "value" / key: 1 / key: [
+        /\w\(/,                                              # method/function call: expect(, insert_observation(
+        /\s\|\s/,                                            # spaced table pipe (doc / CHANGELOG rows)
+        /[\u{2500}-\u{257f}]/,                               # box-drawing glyphs (tree / benchmark output)
+        /\(vector\)|\(text\)/,                               # benchmark mode labels
+        /parentUuid|isSidechain|toolUseID|hookName|"type":/  # raw JSONL transcript fields
+      )
 
       def distill(text, content_item_id: nil)
         entities = extract_entities(text)
@@ -166,7 +182,13 @@ module ClaudeMemory
         (s[/\A.{0,240}?[.!?](?=\s|\z)/m] || s[0, 240]).to_s.strip
       end
 
+      # A usable observation reads as a prose sentence. Reject anything that
+      # doesn't begin like one (leading /, |, ·, or box-drawing glyphs from a
+      # code comment or tool output) or that carries a code/markup/transcript
+      # signature.
       def noise_body?(body)
+        return true unless body.match?(/\A[A-Za-z]/)
+
         body.match?(NOISE_BODY_SIGNATURE)
       end
 

data/lib/claude_memory/hook/context_injector.rb

@@ -73,7 +73,12 @@ module ClaudeMemory
 
         if fresh_session?
           undistilled = fetch_undistilled(MAX_UNDISTILLED)
-          sections << format_distillation_prompt(undistilled) if undistilled.any?
+          if undistilled.any?
+            sections << format_distillation_prompt(undistilled)
+            # The episodic-capture ask is its own prominent section (#72), not a
+            # buried paragraph inside the deep-distill prompt.
+            sections << format_observation_capture_prompt
+          end
 
           promotion = fetch_promotion_candidates(MAX_PROMOTION_CANDIDATES)
           sections << format_observation_reflection(promotion) if promotion.any?
@@ -247,15 +252,7 @@ module ClaudeMemory
           "in the object (e.g., \"… because …\", \"… so that …\", \"caused by …\",",
           "\"breaks when …\"). A fact with a reason is recoverable once stale; a",
           "bare conclusion is dead weight. Prefer one fact-with-reason over two",
-          "facts-without.",
-          "",
-          "**Also log what happened (episodic layer):** in the same",
-          "`memory.store_extraction` call, populate `observations` — one per",
-          "discrete event (a decision made, a preference stated, a notable action",
-          "or outcome). Each: a concise `body` of what happened, a `kind`",
-          "(decision/preference/event/…), and a reason for decisions/preferences.",
-          "Observations record \"what happened\"; facts record \"what is true\". They",
-          "accumulate, and a corroborated observation can later graduate into a fact."
+          "facts-without."
         ]
 
         items.each do |item|
@@ -269,6 +266,34 @@ module ClaudeMemory
         lines.join("\n")
       end
 
+      # First-class, standalone ask for the episodic layer (#72). Authoring
+      # observations was previously a paragraph buried inside the optional
+      # deep-distill flow above, and that flow fires almost never — so the
+      # episodic log was 100% Layer-1 scrapes. This decouples it: a prominent,
+      # lightweight instruction to log "what happened" directly, the same way
+      # the fact context rides the session. Effectiveness is measurable via the
+      # `mcp_extraction` content-item source (Layer-2) vs `claude_code` (Layer-1).
+      def format_observation_capture_prompt
+        <<~PROMPT.strip
+          ## Log What Happened (episodic memory)
+
+          Record the recent narrative as **observations** — "what happened",
+          complementing the facts above ("what is true"). For each discrete
+          event in the recent work above (a decision made, a preference stated,
+          a notable fix or outcome), call `memory.store_extraction` with an
+          `observations` array — one entry per event:
+
+          - `body`: one concise sentence of what happened (embed a reason for
+            decisions/preferences — "… because …", "… so that …")
+          - `kind`: `decision`, `preference`, or `event`
+          - `priority`: 1 important, 2 maybe, 3 info
+
+          Keep it to genuine events worth remembering — skip routine steps and
+          code output. Observations accumulate and a corroborated one graduates
+          into a fact. Send them with the facts in the same call, or on their own.
+        PROMPT
+      end
+
       def format_section(title, items)
         items = items.compact.uniq
         return nil if items.empty?

data/lib/claude_memory/index/lexical_fts.rb

@@ -3,6 +3,15 @@
 module ClaudeMemory
   module Index
     class LexicalFTS
+      # Raised when the FTS5 BM25 ranking path fails as "malformed" while the
+      # rest of the DB is fine — a corrupt contentless FTS5 index that
+      # PRAGMA integrity_check misses (issue #7, Finding 2). Recoverable with
+      # `claude-memory compact` (which runs #rebuild!).
+      class CorruptRankIndexError < StandardError; end
+
+      RANK_CORRUPTION_HINT = "FTS5 rank index is corrupt — recall is broken even though " \
+        "the database otherwise looks healthy. Run `claude-memory compact` to rebuild it."
+
       def initialize(store)
         @store = store
         @db = store.db
@@ -35,17 +44,19 @@ module ClaudeMemory
         end
 
         escaped_query = escape_fts_query(query)
-        if contentless?
-          @db.fetch(
-            "SELECT rowid AS content_item_id FROM content_fts WHERE text MATCH ? ORDER BY rank LIMIT ?",
-            escaped_query, limit
-          ).map { |row| row[:content_item_id] }
-        else
-          @db[:content_fts]
-            .where(Sequel.lit("text MATCH ?", escaped_query))
-            .order(:rank)
-            .limit(limit)
-            .select_map(:content_item_id)
+        with_rank_index do
+          if contentless?
+            @db.fetch(
+              "SELECT rowid AS content_item_id FROM content_fts WHERE text MATCH ? ORDER BY rank LIMIT ?",
+              escaped_query, limit
+            ).map { |row| row[:content_item_id] }
+          else
+            @db[:content_fts]
+              .where(Sequel.lit("text MATCH ?", escaped_query))
+              .order(:rank)
+              .limit(limit)
+              .select_map(:content_item_id)
+          end
         end
       end
 
@@ -59,18 +70,20 @@ module ClaudeMemory
         return [] if query.strip == "*"
 
         escaped_query = escape_fts_query(query)
-        if contentless?
-          @db.fetch(
-            "SELECT rowid AS content_item_id, rank FROM content_fts WHERE text MATCH ? ORDER BY rank LIMIT ?",
-            escaped_query, limit
-          ).all
-        else
-          @db[:content_fts]
-            .where(Sequel.lit("text MATCH ?", escaped_query))
-            .order(:rank)
-            .limit(limit)
-            .select(Sequel.lit("content_item_id, rank"))
-            .all
+        with_rank_index do
+          if contentless?
+            @db.fetch(
+              "SELECT rowid AS content_item_id, rank FROM content_fts WHERE text MATCH ? ORDER BY rank LIMIT ?",
+              escaped_query, limit
+            ).all
+          else
+            @db[:content_fts]
+              .where(Sequel.lit("text MATCH ?", escaped_query))
+              .order(:rank)
+              .limit(limit)
+              .select(Sequel.lit("content_item_id, rank"))
+              .all
+          end
         end
       end
 
@@ -117,6 +130,17 @@ module ClaudeMemory
 
       private
 
+      # Run a `MATCH ... ORDER BY rank` query, translating the narrow
+      # "malformed"-on-rank failure into an actionable error instead of an
+      # unhandled Extralite stacktrace (issue #7, Finding 2). Any other error
+      # propagates unchanged.
+      def with_rank_index
+        yield
+      rescue => e
+        raise unless e.message.to_s.include?("malformed")
+        raise CorruptRankIndexError, RANK_CORRUPTION_HINT
+      end
+
       def contentless?
         @contentless
       end

data/lib/claude_memory/index/vector_index.rb

@@ -134,6 +134,33 @@ module ClaudeMemory
         true
       end
 
+      # Drop and rebuild facts_vec at `dimensions`. A vec0 column width is
+      # immutable once the table is created, so adopting a model of a different
+      # dimension (or any model on a DB whose table was created at the 384
+      # default) requires a full rebuild — clearing rows isn't enough (issue #7,
+      # Finding 1). Requires the sqlite-vec extension loaded so the vec0
+      # destructor runs on DROP.
+      # @param dimensions [Integer] new embedding width
+      def recreate!(dimensions)
+        return false unless available?
+
+        @dimensions = dimensions
+        @db.run("DROP TABLE IF EXISTS facts_vec")
+        @vec_table_ensured = false
+        ensure_vec_table!
+        true
+      end
+
+      # The width facts_vec was actually created with, parsed from its DDL — or
+      # nil when the table doesn't exist yet. Detects a stale-width table even
+      # when the embedding_dimensions meta was never written (old tfidf DBs),
+      # which is exactly the case that silently left a 384 table in place.
+      # @return [Integer, nil]
+      def table_dimensions
+        ddl = @db[:sqlite_master].where(type: "table", name: "facts_vec").get(:sql)
+        ddl && ddl[/embedding\s+float\[(\d+)\]/, 1]&.to_i
+      end
+
       # Number of entries in the vec0 virtual table
       def count
         return 0 unless available?

data/lib/claude_memory/mcp/server.rb

@@ -14,16 +14,19 @@ module ClaudeMemory
     # and writes JSON responses to output.
     class Server
       PROTOCOL_VERSION = "2024-11-05"
+      ORPHAN_CHECK_INTERVAL_SECONDS = 30
 
       # @param store_or_manager [Store::SQLiteStore, Store::StoreManager] database backend
       # @param input [IO] input stream for JSON-RPC requests (default: $stdin)
       # @param output [IO] output stream for JSON-RPC responses (default: $stdout)
-      def initialize(store_or_manager, input: $stdin, output: $stdout)
+      # @param parent_pid [Integer] pid to watch for orphaning (default: caller's parent)
+      def initialize(store_or_manager, input: $stdin, output: $stdout, parent_pid: Process.ppid)
         @store_or_manager = store_or_manager
         @tools = Tools.new(store_or_manager)
         @telemetry = Telemetry.new(store_or_manager)
         @input = input
         @output = output
+        @parent_pid = parent_pid
         @running = false
       end
 
@@ -31,12 +34,24 @@ module ClaudeMemory
       # @return [void]
       def run
         @running = true
+        watchdog = start_orphan_watchdog
         while @running
           line = @input.gets
           break unless line
 
           handle_message(line.strip)
         end
+      ensure
+        watchdog&.kill
+      end
+
+      # True once our original parent has exited and we've been reparented
+      # (to PID 1 / a subreaper). Claude Code spawns one stdio MCP server per
+      # session; a hard kill of the client can leave the server blocked on
+      # `gets` forever, holding a SQLite connection — the orphan leak in issue
+      # #7, Finding 3. The watchdog uses this to terminate.
+      def orphaned?
+        @parent_pid > 1 && Process.ppid != @parent_pid
       end
 
       # Signal the read loop to exit after the current message.
@@ -47,6 +62,23 @@ module ClaudeMemory
 
       private
 
+      # Background watch that terminates the process when the parent dies, since
+      # the read loop is otherwise blocked indefinitely on `gets`. Returns the
+      # Thread, or nil when there's no meaningful parent to watch (e.g. a test
+      # harness or a manually-launched server). Uses `exit!` because the orphaned
+      # parent is gone — there's nothing reading our output to flush to, and the
+      # OS releases the SQLite connection on process exit.
+      def start_orphan_watchdog
+        return if @parent_pid <= 1
+
+        Thread.new do
+          loop do
+            sleep ORPHAN_CHECK_INTERVAL_SECONDS
+            exit!(0) if orphaned?
+          end
+        end
+      end
+
       # @return [void]
       def handle_message(line)
         return if line.empty?