claude_memory 0.12.0 → 0.13.0

data/docs/GETTING_STARTED.md

@@ -119,6 +119,44 @@ claude-memory promote <fact_id>
 "Remember that I prefer descriptive commit messages - make that a global preference"
 ```
 
+## Two Kinds of Memory: Facts and Observations
+
+ClaudeMemory remembers two complementary things:
+
+- **Facts** answer *"what is true"* — durable, structured truths about your
+  project (`uses_database: sqlite`, conventions, decisions). This is the
+  semantic layer the sections above describe.
+- **Observations** answer *"what happened"* — an append-only narrative log of
+  the moments in your sessions ("decided to add a corroboration gate so
+  fleeting mentions don't harden into facts"). This is the **episodic** layer
+  (0.13.0+).
+
+| | Facts | Observations |
+|---|---|---|
+| Captures | Durable truths | Events / narrative |
+| Changes | Explicitly (supersession, rejection) | Automatically (dedup, consolidation, expiry) |
+| Promotion | — | Promoted to a fact after corroboration (≥2 sightings) |
+
+**Why this matters:** the distiller used to commit a fact the first time it
+saw a claim — so a database named once in a comparison could become a false
+`uses_database`. Observations make repeated sighting the gate: an observation
+graduates to a fact only after it recurs. That's an anti-hallucination defense
+built into the memory model.
+
+Observations are managed for you — deduplicated and consolidated automatically
+on `PreCompact`/`SessionEnd` at no extra API cost. To see or curate them:
+
+```bash
+# Inspect the episodic log (counts, promotion readiness, compression, recent)
+claude-memory observations
+
+# Promote a corroborated observation to a fact
+claude-memory observations promote <id> --predicate uses_database --object sqlite
+```
+
+The dashboard's **Observations** panel shows the same at a glance, and the
+`/reflect` skill runs a guided survey → consolidate → promote pass.
+
 ## Setting Up Your First Project
 
 ### Scenario 1: Fresh Install (New Project)

data/docs/api_stability.md

@@ -47,7 +47,7 @@ When ambiguous, default is **internal** — easier to promote later than demote.
 
 ## 2. Public CLI surface
 
-All commands listed in `Commands::Registry::COMMANDS` are reachable via `claude-memory <subcommand>`. The full registered set (34 commands as of 0.12.0) is canonically stored in `lib/claude_memory/commands/registry.rb`. Stability:
+All commands listed in `Commands::Registry::COMMANDS` are reachable via `claude-memory <subcommand>`. The full registered set (38 commands as of 0.12.1) is canonically stored in `lib/claude_memory/commands/registry.rb`. Stability:
 
 ### Stable commands (covered by semver)
 
@@ -90,6 +90,7 @@ May change in any minor; treat with care.
 | `claude-memory recall --semantic` / `--mode=hybrid` | Semantic-recall flags depend on the embedding backend; `tfidf` is stable, `fastembed`/`api` may change configuration knobs. |
 | `claude-memory embeddings` | Embedding-backend inspection; the JSON shape evolves with provider work. |
 | `claude-memory import-auto-memory [--dry-run]` | Imports Claude Code auto-memory markdown files into the project DB as facts. Introduced 0.12.0 from the 2026-05-21 audit; argument shape and idempotency contract are stable but the heuristic for predicate mapping may evolve. |
+| `claude-memory setup-vectors [--provider=NAME] [--model=NAME] [--no-reindex] [--dry-run] [--status]` | Documented opt-in path for enabling vector recall via fastembed. Introduced 0.12.1. Writes `CLAUDE_MEMORY_EMBEDDING_PROVIDER` (and optional `CLAUDE_MEMORY_EMBEDDING_MODEL`) to `.claude/settings.json` env block, then re-embeds via `IndexCommand`. fastembed remains a dev/test gem dep by design; install via `gem install fastembed` if not present. Argument shape stable; the underlying re-index implementation may evolve. |
 
 ### Internal / not for external automation
 
@@ -113,7 +114,7 @@ Renaming or repurposing a code is a major-version change.
 
 ## 3. Public MCP tool surface
 
-All 23 tools registered via `MCP::ToolDefinitions.all`. Argument schemas, return shapes (both `content` and `structuredContent`), and tool-annotation hints (`readOnlyHint`, `idempotentHint`, `destructiveHint`) are **stable** for the listed tools.
+All 28 tools registered via `MCP::ToolDefinitions.all` — 25 stable + 3 experimental (the observational-layer tools below). Argument schemas, return shapes (both `content` and `structuredContent`), and tool-annotation hints (`readOnlyHint`, `idempotentHint`, `destructiveHint`) are **stable** for the listed stable tools.
 
 ### Stable MCP tools
 
@@ -133,7 +134,7 @@ All 23 tools registered via `MCP::ToolDefinitions.all`. Argument schemas, return
 | `memory.facts_by_context` | Context | Stable. |
 | `memory.promote` | Management | Stable. |
 | `memory.reject_fact` | Management | Stable since 0.10.0. |
-| `memory.store_extraction` | Management | Argument schema (`facts`, `entities`, `decisions`) stable. |
+| `memory.store_extraction` | Management | Argument schema (`facts`, `entities`, `decisions`) stable. The `observations` field (Layer-2 observer) is **experimental** while the observational layer is built out. |
 | `memory.undistilled` | Distillation | Stable since 0.10.0. |
 | `memory.mark_distilled` | Distillation | Stable since 0.10.0. |
 | `memory.status` | Monitoring | Stable. |
@@ -145,6 +146,16 @@ All 23 tools registered via `MCP::ToolDefinitions.all`. Argument schemas, return
 | `memory.check_setup` | Discovery | Stable. |
 | `memory.list_projects` | Discovery | Stable since 0.10.0. |
 
+### Experimental MCP tools
+
+These are registered in `MCP::ToolDefinitions.all` but **not yet covered by the stability guarantees above** — argument schema and return shape may change while the feature is built out.
+
+| Tool | Group | Status |
+|---|---|---|
+| `memory.observations` | Observational layer | Experimental (0.13.0+). Read-only listing of episodic observations by status/kind/priority with corroboration counts. Return shape may change. |
+| `memory.promote_observation` | Observational layer | Experimental (0.13.0+). Promotes a corroborated observation (≥2 sightings) into a fact; refuses uncorroborated or already-promoted ones (anti-hallucination gate). Args/shape may change. |
+| `memory.consolidate_observations` | Observational layer | Experimental (0.13.0+). Merges related observations into one synthesized row (corroboration combines, sources tombstoned via `consolidated_into`). Args/shape may change. |
+
 ### Stability of tool responses
 
 Both response shapes are stable:
@@ -200,7 +211,7 @@ Adding a new field to `detail_json` is a stable-surface addition (non-breaking).
 
 Current covered events (0.11.0):
 
-- `hook_context`: `context_length`, `context_tokens` (since 0.11.0), `top_fact_ids`, `fact_count`.
+- `hook_context`: `context_length`, `context_tokens` (since 0.11.0), `top_fact_ids`, `fact_count`. (`observation_count`, added with the observational layer, is additive and **experimental** — not yet on the smoke-gate manifest.)
 - `roi_nudge`: `n`, `used`, `pct`, `prior_count` (all since 0.11.0).
 
 `hook_ingest`, `hook_sweep`, `hook_publish` event detail fields are currently **internal** (not on the smoke-gate manifest). Promoting them to stable is a 0.12.x or later task.
@@ -254,7 +265,7 @@ If you need a feature from one of the internal classes, **open an issue** so we
 
 ### Schema migrations
 
-Schema is at v18 as of 0.12.0 with 18 migrations under `db/migrations/`. Migrations remain forward-compatible per the round-trip-spec convention (`feedback_round_trip_migration_specs.md`): each release's specs verify that DBs from the prior 3 schema boundaries can be migrated into the current schema without data loss.
+Schema is at v20 (v18 shipped in 0.12.0; v19–v20 add the observational `observations` table in 0.13.0) with 20 migrations under `db/migrations/`. Migrations remain forward-compatible per the round-trip-spec convention (`feedback_round_trip_migration_specs.md`): each release's specs verify that DBs from the prior 3 schema boundaries can be migrated into the current schema without data loss.
 
 **What's stable:**
 
@@ -267,6 +278,7 @@ Schema is at v18 as of 0.12.0 with 18 migrations under `db/migrations/`. Migrati
 
 - The `vec0` virtual-table internals — sqlite-vec evolution may shift representation.
 - `mcp_tool_calls` retention behavior (currently 90 days, configurable); the column set is stable, the retention default is not.
+- The `observations` table (v19–v20, incl. `corroboration_count`/`promoted_at`/`promoted_fact_id`) — episodic layer. Column set may still change while the layer is experimental.
 
 **What's internal:**
 
@@ -309,7 +321,11 @@ Run before tagging a release; wire into CI on the project's own DB to catch in-c
 6. Distillation backlog < 100 (hard fail) / < 25 (warning).
 7. Project active facts ≥ 5 (sanity floor — catches over-aggressive rejection).
 
-Run via `bundle exec rspec spec/benchmarks/health/ --tag benchmark`.
+Run via `bundle exec rspec spec/benchmarks/health/ --tag benchmark`. The
+spec is **local-only** — `.claude/memory.sqlite3` is git-lfs tracked and CI
+checkout doesn't pull LFS objects, so the spec auto-skips on CI when it
+detects the unresolved pointer. Run it locally after `git lfs pull` to
+validate signal contracts before tagging a release.
 
 ---
 
@@ -319,7 +335,7 @@ Listed here for honesty — these surfaces look public but are not.
 
 - **Dashboard JSON HTTP API.** The `claude-memory dashboard` server's endpoints are an internal interface for the bundled UI. Don't build scripts against `GET /api/trust` etc. — endpoints, response shapes, and even URL paths may change without notice.
 - **`activity_events.detail_json` fields not in `spec/smoke/expected_fields.yml`.** Inspecting a missing field during debugging is fine; relying on it in scripts is not.
-- **The exact text of `additionalContext`.** The Markdown sections (`## Decisions`, `## Conventions`, `## Architecture`, `## Pending Knowledge Extraction`, `## Auto-Memory Mirror`) and their order are stable; the per-fact rendering format inside each section is tuned for prompt quality and may change.
+- **The exact text of `additionalContext`.** The Markdown sections (`## Decisions`, `## Conventions`, `## Architecture`, `## Pending Knowledge Extraction`, `## Auto-Memory Mirror`) and their order are stable; the per-fact rendering format inside each section is tuned for prompt quality and may change. The `## Observations` and `## Observation Reflection` sections (observational layer) and the published `.claude/rules/claude_memory.observations.md` snapshot are **experimental** while the layer is built out.
 - **Internal env vars** (anything not listed in `Configuration` instance methods or in this doc). Examples that exist but are internal: `CLAUDE_MEMORY_LOG_LEVEL`, debug flags surfaced during development.
 - **Test/spec/fixture infrastructure.** `spec/benchmarks/`, `spec/evals/`, `spec/support/` are not public APIs.
 - **Plugin-format paths.** `.claude-plugin/`, `scripts/serve-mcp.sh`, etc. are part of the Claude Code plugin format integration; treat them as opaque.

data/docs/architecture.md

@@ -14,20 +14,22 @@ ClaudeMemory is architected using Domain-Driven Design (DDD) principles with cle
                        │
 ┌──────────────────────▼──────────────────────────────────────┐
 │                   Core Domain Layer                          │
-│  Domain Models: Fact, Entity, Provenance, Conflict          │
+│  Domain Models: Fact, Entity, Provenance, Conflict,         │
+│                 Observation (episodic)                       │
 │  Value Objects: SessionId, TranscriptPath, FactId           │
 │  Null Objects: NullFact, NullExplanation                    │
 └──────────────────────┬──────────────────────────────────────┘
                        │
 ┌──────────────────────▼──────────────────────────────────────┐
 │                 Business Logic Layer                         │
-│  Recall → Resolve → Distill → Ingest → Publish             │
-│  Sweep → Embeddings → MCP → Hook                           │
+│  Recall → Resolve (semantic) → Distill → Ingest → Publish  │
+│  Observe → Reflect (episodic) → Sweep → Embeddings →       │
+│  MCP → Hook                                                  │
 └──────────────────────┬──────────────────────────────────────┘
                        │
 ┌──────────────────────▼──────────────────────────────────────┐
 │                 Infrastructure Layer                         │
-│  Store (SQLite v17 + WAL) → FileSystem → Index (FTS5+Vector)│
+│  Store (SQLite v20 + WAL) → FileSystem → Index (FTS5+Vector)│
 │  Templates                                                   │
 └─────────────────────────────────────────────────────────────┘
 ```
@@ -129,10 +131,19 @@ end
 - `DualQueryTemplate`: Query template handling for dual-database queries
 
 #### Resolve (`resolve/`)
-- Truth maintenance and conflict resolution
+- Truth maintenance and conflict resolution (the **semantic** "what is true" layer)
 - **Transaction safety**: Multi-step operations wrapped in DB transactions
 - PredicatePolicy: Controls single vs. multi-value predicates
 - Handles supersession and conflict detection
+- Also persists observations from the extraction inside the same transaction (see Observe & Reflect)
+
+#### Observe & Reflect (`observe/`)
+- The **episodic** "what happened" layer, complementing the semantic fact store
+- **Observer**: `NullDistiller` emits high-precision Layer-1 observations (regex); Claude-as-observer enriches them via the SessionStart context hook (Layer-2, no extra API cost)
+- **Reflector** (`Observe::Reflector`): deterministic dedup + TTL-expiry of info-level observations runs on `PreCompact`/`SessionEnd`; semantic consolidation (Claude-as-reflector) rides the next turn's context hook
+- **Append-only/tombstoning**: superseded observations are linked via `consolidated_into`, never deleted — preserving provenance
+- **Promotion bridge**: observations are promoted to facts only after corroboration (≥2 sightings) — an anti-hallucination gate
+- `ObservationsRenderer` formats the injected log; `Domain::Observation` is the immutable value object
 
 #### Distill (`distill/`)
 - Extracts facts and entities from transcripts
@@ -164,7 +175,7 @@ end
 
 #### MCP (`mcp/`)
 - Model Context Protocol server
-- Exposes 19 tools including: recall, explain, promote, status, decisions, conventions, architecture, semantic search, check_setup, and more
+- Exposes 28 tools including: recall, explain, promote, status, decisions, conventions, architecture, semantic search, check_setup, the observational-layer tools (observations, promote_observation, consolidate_observations), and more
 - `ResponseFormatter`: Consistent MCP response formatting
 - `SetupStatusAnalyzer`: Initialization and version status analysis
 
@@ -212,6 +223,7 @@ end
   - `Reuse`: most-used facts within window
   - `Health`: db / hooks / vec checks with actionable fix strings
   - `Timeline`: 30-day daily rollup
+  - `Observations` (0.13.0+): episodic-layer panel — counts by status/kind/priority, corroboration + promotion readiness, source→observation compression ratio, recent timeline (first-class main-sidebar panel + Advanced tab)
   - `FactPresenter`, `ScopedFactResolver`: shared rendering / scope-aware ID resolution
 - Connections released after every request — no held WAL writer locks across page loads
 - See [docs/dashboard.md](dashboard.md) for the user-facing guide

data/docs/audit_runbook.md

@@ -178,6 +178,73 @@ Exit code is `0` when `ok: true`, `1` otherwise. `--no-exit` always returns `0`.
 3. Clean up: `claude-memory reject` the historical disputed/superseded rows (or accept them as historical record).
 4. Re-audit.
 
+### C011 — Orphaned observations
+
+**Severity:** warn
+
+**Scope:** both the project and global DBs (observations may be scoped either way).
+
+**Triggered when:** an observation has `source_content_item_id` set but no `content_items` row with that id exists.
+
+**Why it matters:** An observation's `source_content_item_id` is its provenance link back to the transcript chunk it was distilled from. A dangling pointer means the source row was pruned (or never existed), so the observation can no longer be explained — breaking the same provenance guarantee facts enjoy. Observations with a `nil` source (e.g. consolidated ones synthesized from several sources) are *not* flagged.
+
+**Remediation:**
+- Inspect with `memory.observations` (or the dashboard Observations tab).
+- The table is append-only — do **not** delete. If the provenance is genuinely unrecoverable, let the Reflector consolidate or expire the row on the next PreCompact/SessionEnd pass.
+
+### C012 — Observation promotion consistency
+
+**Severity:** error
+
+**Scope:** both DBs.
+
+**Triggered when:** any of the following promotion-state invariants is violated —
+- `promoted_at` is set but `promoted_fact_id` is `NULL`;
+- `promoted_fact_id` points at a fact that does not exist;
+- `promoted_fact_id` points at a fact that is **not** active (rejected/superseded);
+- `promoted_fact_id` is set but `promoted_at` is `NULL`.
+
+**Why it matters:** Promotion is meant to be atomic — `mark_observation_promoted` sets both `promoted_at` and `promoted_fact_id` pointing at a freshly-created, active fact. Half-set state means the write ran partially, or the target fact was later rejected/superseded, leaving the observation pointing at nothing usable. The promotion bridge keys off these columns, so an inconsistent row either re-promotes (duplicate facts) or is silently stuck.
+
+**Remediation:**
+1. `claude-memory explain <fact_id>` on the `promoted_fact_id` to see why the fact is missing/inactive.
+2. If the fact was intentionally rejected, re-open the observation for re-promotion via `memory.promote_observation`.
+3. If `mark_observation_promoted` half-ran, re-run promotion so both columns are set together.
+
+### C013 — Observation tombstone-chain validity
+
+**Severity:** error
+
+**Scope:** both DBs.
+
+**Triggered when:** any of the following tombstone invariants is violated —
+- `consolidated_into` points at a non-existent observation;
+- `consolidated_into` is a self-link (`consolidated_into == id`);
+- a row is `status='active'` yet carries a `consolidated_into` target;
+- a row is `status='consolidated'` yet has no `consolidated_into` keeper.
+
+**Why it matters:** Supersession is append-only: a merged-away observation gets `status='consolidated'` and `consolidated_into` pointing at the surviving keeper, preserving lineage instead of hard-deleting (unlike Mastra's lossy drop). A broken chain corrupts that lineage — recall could surface a tombstoned row, or a consolidated row could orphan its history. A self-link or active-but-tombstoned row is a Reflector bug, not user error.
+
+**Remediation:**
+- Inspect with `memory.observations`.
+- Re-running the deterministic Reflector (fires on PreCompact/SessionEnd) re-derives consolidation for dangling links.
+- A self-link or `active` + `consolidated_into` row signals a Reflector defect — file it rather than hand-editing the append-only table.
+
+### C014 — Observation status / corroboration sanity
+
+**Severity:** warn
+
+**Scope:** both DBs.
+
+**Triggered when:** an observation has a `status` outside `active`/`consolidated`/`expired`, or a `corroboration_count` less than 1.
+
+**Why it matters:** Every observation should carry a known lifecycle status and at least one sighting (a fresh insert counts as 1; the migration default is 1). An unknown status means a migration or an external writer bypassed `insert_observation`; a `corroboration_count < 1` means `increment_corroboration` math went negative. Both break downstream behavior — recall filters key off `status`, and the promotion gate keys off `corroboration_count`.
+
+**Remediation:**
+- Inspect with `memory.observations`.
+- For a bad `corroboration_count`, re-derive sighting counts via the Reflector's dedup pass.
+- For an unknown status, find the writer that bypassed `insert_observation` (the only sanctioned insert path).
+
 ## Adding a new check
 
 The audit is extensible by design.

data/docs/dashboard.md

@@ -116,6 +116,29 @@ sqlite-vec coverage. Each surfaces an actionable fix string (e.g.,
 "Run `claude-memory init` to install the standard hook set"). Status
 escalates to the worst individual check (error > warning > healthy).
 
+### Observations (episodic layer, 0.13.0+)
+
+The episodic counterpart to the fact-based panels. Facts answer "what is
+true"; **observations** are an append-only log of "what happened" in your
+sessions. Surfaced both as a first-class sidebar panel (headline numbers)
+and an Advanced → Observations tab (full detail):
+
+- **Counts by status / kind / priority** — active vs. consolidated vs.
+  expired; decision / preference / event; 🔴 important / 🟡 maybe / 🟢 info.
+- **Corroboration + promotion readiness** — how many observations have been
+  seen enough times (≥2, the corroboration gate) to be promotable to facts,
+  and the highest corroboration count seen. Promotion is the
+  anti-hallucination gate: a one-off mention never becomes a fact.
+- **Compression ratio** — source content tokens ÷ observation tokens, the
+  Mastra-style measure of how much the episodic log condenses raw sessions.
+- **Recent timeline** — the latest observations, newest first, with their
+  priority markers.
+
+Promote a corroborated observation to a fact with `memory.promote_observation`
+(or `claude-memory observations promote`), merge related ones with
+`memory.consolidate_observations`, or run the `/reflect` skill for a guided
+survey → consolidate → promote pass.
+
 ### Activity drill-down
 
 Clicking any moment opens a modal with the parsed payload, prettified JSON,
@@ -190,3 +213,8 @@ WAL writer lock open across page loads.
   flags as stale.
 - `claude-memory dedupe-conflicts` / `reclassify-references` — one-shot
   cleanups for what the Conflicts and Knowledge → References panels surface.
+- `claude-memory observations [list|promote|consolidate]` *(0.13.0+)* — the
+  CLI mirror of the Observations panel: list/inspect the episodic log
+  (`--kind`, `--status`, `--scope`, `--json`), promote a corroborated
+  observation to a fact, or consolidate related ones. `claude-memory stats
+  --observations` prints the counts summary.

data/docs/improvements.md

@@ -1,6 +1,6 @@
 # Improvements to Consider
 
-*Updated: 2026-05-23 - Added AI Memory Systems Landscape Analysis (Nakajima/Opus 4.6 Research article, 2026-03-26) — meta-study of 7 benchmarks + ~12 systems. Four High Priority items: graph traversal as third RRF source (#64), temporal-aware retrieval (#65), bi-temporal schema cleanup (#66), LongMemEval integration (#67). One promotion: improvement #57 (provenance-strength ranking) Medium → High, validated as the "soft epistemic separation" pattern. See `docs/influence/ai-memory-systems-2026.md`. Previously: 2026-05-01 - Added Strands Agent SOPs study (article, not repo) — one M-priority item (parameter blocks in skill frontmatter); rest already implemented or deferred. See `docs/influence/strands-agent-sops.md`. Previously: 2026-04-28 (post-0.10.0) - Restructured 1.0 punchlist around milestone versions. **0.11.0 "Trust & Cost"** ships #47 (token budget), #48 (hallucination rate), #51 (claude-memory show), #53 (first-week ROI nudge — moved up from post-1.0), and a 3-scenario prototype of #49 (harm benchmark). **0.12.0 "Release Discipline"** ships #49 full corpus, #50 (CLAUDE.md baseline), #52 (benchmark scoreboard). **1.0.0** lands soak-validated #54/#55/#56 if time + new #59 API stability audit. See `docs/1_0_punchlist.md` for the full plan with calendar targets. Also added 2026-04-28: two ranking-signal gaps surfaced by the Mercury / "Why Karpathy's Second Brain Breaks" article (Zaid, 2026-04-28) — provenance-strength-aware ranking (#57) and reinforcement/decay scoring (#58). Earlier 2026-04-28 updates: opened the 1.0 punchlist track + added cq study. Previously: 2026-03-30 - Re-studied all 7 influencer repos. New recommendations: CLAUDE_CONFIG_DIR support (#26, from episodic-memory), Usage Stats / ROI Tracking (#27, from grepai v0.35.0). New Features to Avoid: AST-Aware Code Chunking (QMD), Custom Instructions via Env Var (lossless-claw v0.5.2), OpenClaw Context Injection (claude-mem v10.6.0). Repos with no changes: kbs (v0.2.1), claude-supermemory (v2.0.1), episodic-memory (v1.0.15). Previously: 14 features implemented through 2026-03-24.*
+*Updated: 2026-06-17 - Added #70 (recall-preserving fact precision on real transcripts — live obs-experiment found Layer-1 fact noise from prose/comparisons/English-word collisions; claim-context gating was measured to crater the distillation benchmark Fact F1 0.958→0.64, so the lever is downstream: wire ReferenceMaterialDetector into the ingest path / Layer-2. Observation extraction was tightened in-branch; facts left at baseline). Earlier 2026-06-16 - Added #69 (self-heal the FTS rank index after concurrent ingest — live incident: hook-vs-MCP write contention leaves `ORDER BY rank` malformed while data stays intact, silently degrading recall until a manual `compact`). Earlier 2026-06-16 - Added Mastra Observational Memory study — one High Priority item (#68, episodic observation layer: Observer + Reflector + observation→fact promotion bridge) and one Medium item (compression/cache telemetry + LongMemEval episodic suite). Key insight: ClaudeMemory has no episodic layer; observations ("what happened") complement facts ("what is true"). See `docs/influence/mastra-observational-memory.md`. Previously: 2026-05-23 - Added AI Memory Systems Landscape Analysis (Nakajima/Opus 4.6 Research article, 2026-03-26) — meta-study of 7 benchmarks + ~12 systems. Four High Priority items: graph traversal as third RRF source (#64), temporal-aware retrieval (#65), bi-temporal schema cleanup (#66), LongMemEval integration (#67). One promotion: improvement #57 (provenance-strength ranking) Medium → High, validated as the "soft epistemic separation" pattern. See `docs/influence/ai-memory-systems-2026.md`. Previously: 2026-05-01 - Added Strands Agent SOPs study (article, not repo) — one M-priority item (parameter blocks in skill frontmatter); rest already implemented or deferred. See `docs/influence/strands-agent-sops.md`. Previously: 2026-04-28 (post-0.10.0) - Restructured 1.0 punchlist around milestone versions. **0.11.0 "Trust & Cost"** ships #47 (token budget), #48 (hallucination rate), #51 (claude-memory show), #53 (first-week ROI nudge — moved up from post-1.0), and a 3-scenario prototype of #49 (harm benchmark). **0.12.0 "Release Discipline"** ships #49 full corpus, #50 (CLAUDE.md baseline), #52 (benchmark scoreboard). **1.0.0** lands soak-validated #54/#55/#56 if time + new #59 API stability audit. See `docs/1_0_punchlist.md` for the full plan with calendar targets. Also added 2026-04-28: two ranking-signal gaps surfaced by the Mercury / "Why Karpathy's Second Brain Breaks" article (Zaid, 2026-04-28) — provenance-strength-aware ranking (#57) and reinforcement/decay scoring (#58). Earlier 2026-04-28 updates: opened the 1.0 punchlist track + added cq study. Previously: 2026-03-30 - Re-studied all 7 influencer repos. New recommendations: CLAUDE_CONFIG_DIR support (#26, from episodic-memory), Usage Stats / ROI Tracking (#27, from grepai v0.35.0). New Features to Avoid: AST-Aware Code Chunking (QMD), Custom Instructions via Env Var (lossless-claw v0.5.2), OpenClaw Context Injection (claude-mem v10.6.0). Repos with no changes: kbs (v0.2.1), claude-supermemory (v2.0.1), episodic-memory (v1.0.15). Previously: 14 features implemented through 2026-03-24.*
 *Sources:*
 - *[thedotmack/claude-mem](https://github.com/thedotmack/claude-mem) - Memory compression system (v10.6.3, re-studied 2026-03-30)*
 - *[obra/episodic-memory](https://github.com/obra/episodic-memory) - Semantic conversation search (v1.0.15, re-studied 2026-03-30 — no changes)*
@@ -9,6 +9,7 @@
 - *[tobi/qmd](https://github.com/tobi/qmd) - On-device hybrid search engine (v2.0.1+unreleased, re-studied 2026-03-30)*
 - *[MadBomber/kbs](https://github.com/MadBomber/kbs) - Knowledge-Based System with RETE inference (v0.2.1, studied 2026-03-30 — no changes)*
 - *[martian-engineering/lossless-claw](https://github.com/martian-engineering/lossless-claw) - DAG-based lossless context management (v0.5.2, re-studied 2026-03-30)*
+- *[Mastra Observational Memory](https://mastra.ai/blog/observational-memory) - Text-based dual-agent episodic memory (studied 2026-06-16)*
 
 This document contains only unimplemented improvements. Completed items are removed.
 
@@ -291,6 +292,61 @@ Source: 2026-04-28 1.0 readiness review (`docs/1_0_punchlist.md` #6)
 
 ---
 
+### 69. Self-Heal the FTS Rank Index After Concurrent Ingest
+
+Source: 2026-06-16 live incident on `claude/observational-layer-design-7662r9` — observed first-hand, not a study.
+
+**Gap.** The contentless FTS5 index (`content_fts`) silently drifts into a broken state under concurrent writers: the ingest hook (`claude-memory hook ingest`) and the MCP server (`store_extraction` → `Index::LexicalFTS#index_content_item`) both write the same WAL DB, and a large ingest produces `"Database busy, retrying"` (`Store::RetryHandler`) followed by an FTS index where **plain `MATCH` works but `... ORDER BY rank` raises `database disk image is malformed`**. `integrity_check` passes and all rows are intact — so `recall`/`recall_index` ranking is silently degraded (the rank query throws or returns nothing) while nothing looks wrong. The only fix today is the user manually running `claude-memory compact` (which `rebuild_fts` + vacuums). Documented in `docs/influence/...` gotchas and surfaced reactively in the dashboard (`lib/claude_memory/dashboard/api.rb:338`), but never repaired automatically. Severe form (btree corruption, plain `MATCH` also failing) was separately seen when **two** memory MCP servers ran concurrently — de-duping to a single server removed that, but the benign rank-artifact still recurs from hook-vs-MCP contention alone.
+
+**Implementation.**
+
+- **Cheap probe + self-heal in the sweep tail.** In `Sweep::Maintenance`, add a `repair_fts_rank` step: run a bounded `SELECT rowid FROM content_fts WHERE content_fts MATCH 'a' ORDER BY rank LIMIT 1`; on `malformed`, call `Index::LexicalFTS#rebuild!` (already contentless-safe) instead of requiring a manual `compact`. Sweep already runs on PreCompact/SessionEnd, so recall ranking self-repairs within the session that broke it. Guard with a time budget so a huge index doesn't blow the hook timeout.
+- **Reduce the contention that triggers it.** Raise the SQLite `busy_timeout` and use `BEGIN IMMEDIATE` for the FTS-writing transactions so the ingest hook and MCP server serialize cleanly instead of racing (the retry-loop WARN is the symptom). Confirm both the hook path and `ManagementHandlers#store_extraction` open connections with the same pragmas via `Store::RetryHandler`.
+- **Proactive detection.** Add a `doctor` check that runs the rank probe and reports "FTS rank index needs rebuild — run `claude-memory compact`" (or auto-heals if the sweep step lands). Optionally a `roi_nudge`-style one-liner.
+- **Option (larger).** Evaluate external-content FTS5 over `content_items` instead of contentless — more robust to `rebuild` and avoids the auxiliary-index drift entirely. Note as a follow-up, not part of the first fix.
+
+**Acceptance.**
+
+- An integration test that interleaves a large `hook ingest` with an MCP `store_extraction` against the same DB leaves `MATCH ... ORDER BY rank` working (or self-healed by the next sweep) — no manual `compact` required.
+- `doctor` flags the rank-artifact when present.
+- `"Database busy, retrying"` WARN frequency drops under the contention test.
+
+**Effort.** Medium (~2 days). Self-heal step + pragmas are small; the integration test reproducing concurrent writers is the bulk.
+
+**Why high priority.** It silently degrades `recall` — a core feature — and the user has no signal except empty/unranked results until they happen to run `compact`. Recurs on normal usage (hit live 2026-06-16). Relates to the WAL/connection-release discipline already noted for the dashboard.
+
+---
+
+### 70. Recall-Preserving Fact Precision on Real Transcripts (downstream, not regex)
+
+Source: 2026-06-17 obs-experiment live session — observed first-hand.
+
+**Gap.** Layer-1 `NullDistiller` fact extraction is noisy on real (large, mixed-content) transcripts. A live session on a tiny **Ruby + SQLite** project produced `uses_database = postgres/mysql`, `uses_framework = rails/express`, `uses_language = go`, `deployment_platform = docker` — all from prose mentions, comparisons, negations, instruction/skill text, and English-word collisions (`go` in "want this to go", `express` in "expressive"). This is the documented #48 hallucination problem, now characterized with live data.
+
+**What was ruled out (with data).** Gating fact emission on a usage/claim verb near the entity (`using X`, `deployed on X`, `written in X`) was implemented and measured against `spec/benchmarks/distillation/extraction_spec.rb`:
+
+| | Fact Precision | Fact Recall | F1 |
+|---|---|---|---|
+| baseline | 0.919 | 1.0 | 0.958 |
+| + claim-context | 0.615 | 0.667 | **0.64** |
+
+Regex can't separate terse legit claims (`MongoDB database`, `on AWS`, `Dockerized`) from terse prose mentions (`Postgres/MySQL buy you…`) — claim-context trades recall ~1:1 and even loses precision on the clean corpus. **Reverted.**
+
+**Why distiller-level tightening is mostly off the table (2026-06-17 finding).** The benchmark *enforces* high-recall: case `ext_ent_negative` — "We looked at MongoDB but **decided against it**" — still **expects** `uses_database: mongodb`. The fact distiller is deliberately "extract every mention, filter downstream." So negation/comparison **exclusion** also regresses recall (it would drop the rejected-MongoDB case). The genuine downstream precision lever is Layer-2 / the observation→fact promotion gate, which already prevents Layer-1 noise from being *committed* as corroborated facts.
+
+**Done (recall-safe slice):**
+- ✅ **English-word collision fix** — `go` is matched case-sensitively (`Go`/`golang`) so the verb "go" / "go-to" no longer fires `uses_language=go`. Benchmark Fact Precision **0.919 → 0.935**, Recall held at **1.0** (it removed a real false positive — "my go-to database"). `react`/`rust`/`express` are the same collision class and can follow the same `(?-i:)` pattern, each verified against the benchmark.
+
+**Deferred / not worth it:** wiring `ReferenceMaterialDetector` into the ingest path is a no-op for current Layer-1 (it produces stack predicates, not `convention` facts the detector targets) — skip until Layer-1 emits conventions.
+
+**Acceptance (revised).** Distiller-level work is bounded to recall-safe English-word-collision fixes (benchmark Fact F1 ≥ baseline). Broader fact precision is owned by Layer-2 + the promotion gate, not regex.
+
+**Effort.** Medium. Wiring the existing detector is small; extending its heuristics + a real-transcript precision fixture is the bulk.
+
+**Why this priority.** Fact noise predates the observational layer (it's #48) and is mitigated downstream (promotion gate), so it's not a blocker — but it's the most-visible remaining quality gap on real sessions. The observational layer's Layer-1 observation extraction was tightened in this branch (high-precision/low-recall); facts were deliberately left at baseline pending this recall-preserving approach.
+
+---
+
 ## cq Study (2026-04-28)
 
 Source: docs/influence/cq.md — usefulness-focused study (not internals)
@@ -411,6 +467,43 @@ Source: `docs/influence/ai-memory-systems-2026.md` — meta-study of the Nakajim
 
 ---
 
+## Mastra Observational Memory Study (2026-06-16)
+
+Source: `docs/influence/mastra-observational-memory.md` — architecture study of Mastra's Observational Memory (OM), a text-based dual-agent (Observer + Reflector) episodic memory that compresses raw messages into an append-only, dated observation log living in the context window. SOTA on LongMemEval (84–95%) at 3–6× compression, cache-stable by design.
+
+**Headline finding.** In OM's taxonomy ClaudeMemory is the thing it positions against: a structured *semantic* store injected *dynamically per query*. The gap OM exposes is not retrieval quality — it's that **ClaudeMemory has no episodic layer at all.** Facts answer "what is true"; observations answer "what happened." OM is purely episodic, we are purely semantic. The two are complementary, and we already own analogues of OM's Observer (distillation pipeline) and Reflector (Resolve + Sweep) — they just emit facts, not a narrative log.
+
+### High Priority Recommendations
+
+- [x] **68. Episodic Observation Layer (Observer + Reflector + promotion bridge)** ⭐ — ✅ **Shipped 2026-06-16/17** (phases 1–4)
+  - Value: Adds the missing episodic half of memory (narrative "what happened" log) and a cache-stable injection mode, on top of the existing semantic fact store. The promotion bridge (observation→fact on corroboration) doubles as an anti-hallucination gate for the documented reject-churn problem (distiller commits `uses_database`/`uses_framework` facts from one-off doc example text).
+  - Evidence: `docs/influence/mastra-observational-memory.md`. Our distill pipeline (`lib/claude_memory/distill/`) is already an Observer that emits facts; `resolve/` + `sweep/` is already a Reflector over facts. No episodic store exists.
+  - Implementation (phased):
+    1. ✅ **Shipped 2026-06-16** (schema v19): `observations` table (`body`, `kind`, `priority` 🔴/🟡/🟢, `scope`, `source_content_item_id`, `consolidated_into` lineage, `token_count`); `Domain::Observation`; NullDistiller emits observation rows; Resolver persists them; `memory.observations` read tool. **Append-only with tombstoning, not lossy drop** — preserves provenance.
+    2. ✅ **Shipped 2026-06-16**: two-block SessionStart injection via `ContextInjector` — Block 1 = observation log (🔴 marked, 🟡/🟢 stripped as Mastra does for the actor) ahead of Block 2 = undistilled tail; `Observe::ObservationsRenderer` shared with the published `.claude/rules/claude_memory.observations.md` snapshot; `observation_count` added to the `hook_context` activity event for token/compression measurement.
+    3. ✅ **Shipped 2026-06-17**: `Observe::Reflector` — deterministic, free (no LLM) GC. Dedupes near-identical active observations into the newest (tombstone via `consolidated_into`) and expires stale info-level (🟢) ones past a TTL (`observation_info_ttl_days`, default 30); 🔴/🟡 never expire. Provenance-preserving (rows tombstoned, never deleted). Wired into `Sweep::Maintenance#reflect_observations` → `Sweeper#run!`, so it runs on the existing `PreCompact`/`SessionEnd` sweep — context-pressure-triggered, the analog of Mastra's ~40k-token threshold. (Semantic "merge related/surface patterns" deferred to phase 4 — needs the LLM.)
+    4. ✅ **Shipped 2026-06-17** (schema v20): the observation→fact **promotion bridge**. Dedup folds duplicates' `corroboration_count` into the keeper; once an observation crosses `Domain::Observation::PROMOTION_THRESHOLD` (2) sightings it becomes a promotion candidate. `memory.promote_observation` creates the fact via the resolver, links provenance, marks the observation promoted, and **refuses uncorroborated observations server-side** — the anti-hallucination gate. `ContextInjector` surfaces candidates in a SessionStart "## Observation Reflection" section instructing Claude to promote inline (automatic semantic reflection, no extra API cost); a manual `/reflect` skill drives deep on-demand passes. (Trigger is SessionStart rather than PreCompact — the already-wired free injection point; a PreCompact context hook is a possible future refinement.)
+    5. ✅ **Shipped 2026-06-17** (branch `claude/observational-layer-complete`): the LLM half. **Layer-2 Claude-as-observer** — the SessionStart extraction prompt asks Claude to emit episodic observations in the `observations` field of `memory.store_extraction` (coerced/validated at the handler border, persisted via the resolver), making the log rich where Layer-1 regex is high-precision/low-recall. **Semantic reflection** — `memory.consolidate_observations` merges related-but-differently-worded observations into one synthesized row with *combined* corroboration (which can tip it over the promotion gate), tombstoning the sources; surfaced in the reflection section + `/reflect`. (Also fixed a latent Liskov bug: `ReferenceMaterialDetector#reclassify` dropped observations when a fact was present.)
+    6. ✅ **Shipped 2026-06-17** (branch `claude/observational-layer-complete`): observability + measurement. `Dashboard::Observations` panel (`/api/observations`, Advanced → Observations tab) — counts by status/kind/priority, corroboration + promotion readiness, recent timeline, and a **compression ratio** (source content tokens ÷ observation tokens, Mastra-style). The compression metric is the measurement half of design rec E; a full LongMemEval-style episodic benchmark remains (overlaps #67).
+    7. ✅ **Shipped 2026-06-18** (branch `claude/observational-layer-complete`): polish. **PreCompact reflection trigger** — `claude-memory hook context` injects only the reflection nudge (`ContextInjector#reflection_context`) on PreCompact (context pressure, the Mastra token-threshold analog), wired into the standard PreCompact hook set; not the full snapshot. **Observation↔fact provenance** — `memory.observations` exposes status/corroboration_count/promoted_fact_id/consolidated_into; `memory.explain(fact_id)` shows `promoted_from_observations` (reverse link via `observations_for_fact`). Full observational layer complete; remaining: LongMemEval episodic benchmark (#67).
+  - Effort: Large, phased. Phase 1 ~2-3 days; full arc ~2 weeks. Reuses distill/resolve/sweep/publish/context-hook machinery and `context_tokens` telemetry.
+  - Trade-off: reflection is automatic on *lifecycle events* (compaction/session boundaries), not a wall clock — Claude Code has no timer/cron hook, and Routines/subagents incur separate token budgets (rejected). Observer/Reflector reuse the existing session (no extra API cost). **Augments dynamic recall, does not replace it (user-confirmed 2026-06-16).** See claude-code-guide consultation in the influence doc.
+
+### Medium Priority
+
+- [ ] **Compression / cache telemetry + LongMemEval episodic suite** (see influence doc rec E)
+  - Value: Report compression ratio and token reduction on Trust/Health panels using existing `context_tokens` events (0.11.0). Add a LongMemEval-style long-session suite to DevMemBench to score the episodic layer. Overlaps with existing item #67 (LongMemEval integration) — coordinate.
+  - Effort: Medium. Depends on #68 phase 1-2.
+
+### Features to Avoid (from this study)
+
+- **Two always-on background LLM agents** — violates the no-separate-API-call convention. Observer = context-hook injection; Reflector = deterministic shell-side GC + `PreCompact`-injected semantic consolidation (rides the existing session).
+- **Claude Code Routines / subagents for recurring reflection** — Routines run as a separate scheduled cloud session; subagents run in their own context (~7× tokens). Both incur extra spend; reserve only for an explicitly opted-in one-off backfill.
+- **Lossy drop on reflection** ("never forgives") — we tombstone via `consolidated_into` and retain raw `content_items`; provenance is non-negotiable.
+- **Replacing dynamic recall with a wholesale-loaded log** — augment, don't replace; keep `memory.recall` for targeted lookups.
+
+---
+
 ## Medium Priority
 
 ### ~~18. Shell Completion for CLI~~ ✅ Implemented 2026-03-20