claude_memory 0.12.1 → 0.13.0

data/docs/dashboard.md

@@ -116,6 +116,29 @@ sqlite-vec coverage. Each surfaces an actionable fix string (e.g.,
 "Run `claude-memory init` to install the standard hook set"). Status
 escalates to the worst individual check (error > warning > healthy).
 
+### Observations (episodic layer, 0.13.0+)
+
+The episodic counterpart to the fact-based panels. Facts answer "what is
+true"; **observations** are an append-only log of "what happened" in your
+sessions. Surfaced both as a first-class sidebar panel (headline numbers)
+and an Advanced → Observations tab (full detail):
+
+- **Counts by status / kind / priority** — active vs. consolidated vs.
+  expired; decision / preference / event; 🔴 important / 🟡 maybe / 🟢 info.
+- **Corroboration + promotion readiness** — how many observations have been
+  seen enough times (≥2, the corroboration gate) to be promotable to facts,
+  and the highest corroboration count seen. Promotion is the
+  anti-hallucination gate: a one-off mention never becomes a fact.
+- **Compression ratio** — source content tokens ÷ observation tokens, the
+  Mastra-style measure of how much the episodic log condenses raw sessions.
+- **Recent timeline** — the latest observations, newest first, with their
+  priority markers.
+
+Promote a corroborated observation to a fact with `memory.promote_observation`
+(or `claude-memory observations promote`), merge related ones with
+`memory.consolidate_observations`, or run the `/reflect` skill for a guided
+survey → consolidate → promote pass.
+
 ### Activity drill-down
 
 Clicking any moment opens a modal with the parsed payload, prettified JSON,
@@ -190,3 +213,8 @@ WAL writer lock open across page loads.
   flags as stale.
 - `claude-memory dedupe-conflicts` / `reclassify-references` — one-shot
   cleanups for what the Conflicts and Knowledge → References panels surface.
+- `claude-memory observations [list|promote|consolidate]` *(0.13.0+)* — the
+  CLI mirror of the Observations panel: list/inspect the episodic log
+  (`--kind`, `--status`, `--scope`, `--json`), promote a corroborated
+  observation to a fact, or consolidate related ones. `claude-memory stats
+  --observations` prints the counts summary.

data/docs/improvements.md

@@ -1,6 +1,6 @@
 # Improvements to Consider
 
-*Updated: 2026-05-23 - Added AI Memory Systems Landscape Analysis (Nakajima/Opus 4.6 Research article, 2026-03-26) — meta-study of 7 benchmarks + ~12 systems. Four High Priority items: graph traversal as third RRF source (#64), temporal-aware retrieval (#65), bi-temporal schema cleanup (#66), LongMemEval integration (#67). One promotion: improvement #57 (provenance-strength ranking) Medium → High, validated as the "soft epistemic separation" pattern. See `docs/influence/ai-memory-systems-2026.md`. Previously: 2026-05-01 - Added Strands Agent SOPs study (article, not repo) — one M-priority item (parameter blocks in skill frontmatter); rest already implemented or deferred. See `docs/influence/strands-agent-sops.md`. Previously: 2026-04-28 (post-0.10.0) - Restructured 1.0 punchlist around milestone versions. **0.11.0 "Trust & Cost"** ships #47 (token budget), #48 (hallucination rate), #51 (claude-memory show), #53 (first-week ROI nudge — moved up from post-1.0), and a 3-scenario prototype of #49 (harm benchmark). **0.12.0 "Release Discipline"** ships #49 full corpus, #50 (CLAUDE.md baseline), #52 (benchmark scoreboard). **1.0.0** lands soak-validated #54/#55/#56 if time + new #59 API stability audit. See `docs/1_0_punchlist.md` for the full plan with calendar targets. Also added 2026-04-28: two ranking-signal gaps surfaced by the Mercury / "Why Karpathy's Second Brain Breaks" article (Zaid, 2026-04-28) — provenance-strength-aware ranking (#57) and reinforcement/decay scoring (#58). Earlier 2026-04-28 updates: opened the 1.0 punchlist track + added cq study. Previously: 2026-03-30 - Re-studied all 7 influencer repos. New recommendations: CLAUDE_CONFIG_DIR support (#26, from episodic-memory), Usage Stats / ROI Tracking (#27, from grepai v0.35.0). New Features to Avoid: AST-Aware Code Chunking (QMD), Custom Instructions via Env Var (lossless-claw v0.5.2), OpenClaw Context Injection (claude-mem v10.6.0). Repos with no changes: kbs (v0.2.1), claude-supermemory (v2.0.1), episodic-memory (v1.0.15). Previously: 14 features implemented through 2026-03-24.*
+*Updated: 2026-06-17 - Added #70 (recall-preserving fact precision on real transcripts — live obs-experiment found Layer-1 fact noise from prose/comparisons/English-word collisions; claim-context gating was measured to crater the distillation benchmark Fact F1 0.958→0.64, so the lever is downstream: wire ReferenceMaterialDetector into the ingest path / Layer-2. Observation extraction was tightened in-branch; facts left at baseline). Earlier 2026-06-16 - Added #69 (self-heal the FTS rank index after concurrent ingest — live incident: hook-vs-MCP write contention leaves `ORDER BY rank` malformed while data stays intact, silently degrading recall until a manual `compact`). Earlier 2026-06-16 - Added Mastra Observational Memory study — one High Priority item (#68, episodic observation layer: Observer + Reflector + observation→fact promotion bridge) and one Medium item (compression/cache telemetry + LongMemEval episodic suite). Key insight: ClaudeMemory has no episodic layer; observations ("what happened") complement facts ("what is true"). See `docs/influence/mastra-observational-memory.md`. Previously: 2026-05-23 - Added AI Memory Systems Landscape Analysis (Nakajima/Opus 4.6 Research article, 2026-03-26) — meta-study of 7 benchmarks + ~12 systems. Four High Priority items: graph traversal as third RRF source (#64), temporal-aware retrieval (#65), bi-temporal schema cleanup (#66), LongMemEval integration (#67). One promotion: improvement #57 (provenance-strength ranking) Medium → High, validated as the "soft epistemic separation" pattern. See `docs/influence/ai-memory-systems-2026.md`. Previously: 2026-05-01 - Added Strands Agent SOPs study (article, not repo) — one M-priority item (parameter blocks in skill frontmatter); rest already implemented or deferred. See `docs/influence/strands-agent-sops.md`. Previously: 2026-04-28 (post-0.10.0) - Restructured 1.0 punchlist around milestone versions. **0.11.0 "Trust & Cost"** ships #47 (token budget), #48 (hallucination rate), #51 (claude-memory show), #53 (first-week ROI nudge — moved up from post-1.0), and a 3-scenario prototype of #49 (harm benchmark). **0.12.0 "Release Discipline"** ships #49 full corpus, #50 (CLAUDE.md baseline), #52 (benchmark scoreboard). **1.0.0** lands soak-validated #54/#55/#56 if time + new #59 API stability audit. See `docs/1_0_punchlist.md` for the full plan with calendar targets. Also added 2026-04-28: two ranking-signal gaps surfaced by the Mercury / "Why Karpathy's Second Brain Breaks" article (Zaid, 2026-04-28) — provenance-strength-aware ranking (#57) and reinforcement/decay scoring (#58). Earlier 2026-04-28 updates: opened the 1.0 punchlist track + added cq study. Previously: 2026-03-30 - Re-studied all 7 influencer repos. New recommendations: CLAUDE_CONFIG_DIR support (#26, from episodic-memory), Usage Stats / ROI Tracking (#27, from grepai v0.35.0). New Features to Avoid: AST-Aware Code Chunking (QMD), Custom Instructions via Env Var (lossless-claw v0.5.2), OpenClaw Context Injection (claude-mem v10.6.0). Repos with no changes: kbs (v0.2.1), claude-supermemory (v2.0.1), episodic-memory (v1.0.15). Previously: 14 features implemented through 2026-03-24.*
 *Sources:*
 - *[thedotmack/claude-mem](https://github.com/thedotmack/claude-mem) - Memory compression system (v10.6.3, re-studied 2026-03-30)*
 - *[obra/episodic-memory](https://github.com/obra/episodic-memory) - Semantic conversation search (v1.0.15, re-studied 2026-03-30 — no changes)*
@@ -9,6 +9,7 @@
 - *[tobi/qmd](https://github.com/tobi/qmd) - On-device hybrid search engine (v2.0.1+unreleased, re-studied 2026-03-30)*
 - *[MadBomber/kbs](https://github.com/MadBomber/kbs) - Knowledge-Based System with RETE inference (v0.2.1, studied 2026-03-30 — no changes)*
 - *[martian-engineering/lossless-claw](https://github.com/martian-engineering/lossless-claw) - DAG-based lossless context management (v0.5.2, re-studied 2026-03-30)*
+- *[Mastra Observational Memory](https://mastra.ai/blog/observational-memory) - Text-based dual-agent episodic memory (studied 2026-06-16)*
 
 This document contains only unimplemented improvements. Completed items are removed.
 
@@ -291,6 +292,61 @@ Source: 2026-04-28 1.0 readiness review (`docs/1_0_punchlist.md` #6)
 
 ---
 
+### 69. Self-Heal the FTS Rank Index After Concurrent Ingest
+
+Source: 2026-06-16 live incident on `claude/observational-layer-design-7662r9` — observed first-hand, not a study.
+
+**Gap.** The contentless FTS5 index (`content_fts`) silently drifts into a broken state under concurrent writers: the ingest hook (`claude-memory hook ingest`) and the MCP server (`store_extraction` → `Index::LexicalFTS#index_content_item`) both write the same WAL DB, and a large ingest produces `"Database busy, retrying"` (`Store::RetryHandler`) followed by an FTS index where **plain `MATCH` works but `... ORDER BY rank` raises `database disk image is malformed`**. `integrity_check` passes and all rows are intact — so `recall`/`recall_index` ranking is silently degraded (the rank query throws or returns nothing) while nothing looks wrong. The only fix today is the user manually running `claude-memory compact` (which `rebuild_fts` + vacuums). Documented in `docs/influence/...` gotchas and surfaced reactively in the dashboard (`lib/claude_memory/dashboard/api.rb:338`), but never repaired automatically. Severe form (btree corruption, plain `MATCH` also failing) was separately seen when **two** memory MCP servers ran concurrently — de-duping to a single server removed that, but the benign rank-artifact still recurs from hook-vs-MCP contention alone.
+
+**Implementation.**
+
+- **Cheap probe + self-heal in the sweep tail.** In `Sweep::Maintenance`, add a `repair_fts_rank` step: run a bounded `SELECT rowid FROM content_fts WHERE content_fts MATCH 'a' ORDER BY rank LIMIT 1`; on `malformed`, call `Index::LexicalFTS#rebuild!` (already contentless-safe) instead of requiring a manual `compact`. Sweep already runs on PreCompact/SessionEnd, so recall ranking self-repairs within the session that broke it. Guard with a time budget so a huge index doesn't blow the hook timeout.
+- **Reduce the contention that triggers it.** Raise the SQLite `busy_timeout` and use `BEGIN IMMEDIATE` for the FTS-writing transactions so the ingest hook and MCP server serialize cleanly instead of racing (the retry-loop WARN is the symptom). Confirm both the hook path and `ManagementHandlers#store_extraction` open connections with the same pragmas via `Store::RetryHandler`.
+- **Proactive detection.** Add a `doctor` check that runs the rank probe and reports "FTS rank index needs rebuild — run `claude-memory compact`" (or auto-heals if the sweep step lands). Optionally a `roi_nudge`-style one-liner.
+- **Option (larger).** Evaluate external-content FTS5 over `content_items` instead of contentless — more robust to `rebuild` and avoids the auxiliary-index drift entirely. Note as a follow-up, not part of the first fix.
+
+**Acceptance.**
+
+- An integration test that interleaves a large `hook ingest` with an MCP `store_extraction` against the same DB leaves `MATCH ... ORDER BY rank` working (or self-healed by the next sweep) — no manual `compact` required.
+- `doctor` flags the rank-artifact when present.
+- `"Database busy, retrying"` WARN frequency drops under the contention test.
+
+**Effort.** Medium (~2 days). Self-heal step + pragmas are small; the integration test reproducing concurrent writers is the bulk.
+
+**Why high priority.** It silently degrades `recall` — a core feature — and the user has no signal except empty/unranked results until they happen to run `compact`. Recurs on normal usage (hit live 2026-06-16). Relates to the WAL/connection-release discipline already noted for the dashboard.
+
+---
+
+### 70. Recall-Preserving Fact Precision on Real Transcripts (downstream, not regex)
+
+Source: 2026-06-17 obs-experiment live session — observed first-hand.
+
+**Gap.** Layer-1 `NullDistiller` fact extraction is noisy on real (large, mixed-content) transcripts. A live session on a tiny **Ruby + SQLite** project produced `uses_database = postgres/mysql`, `uses_framework = rails/express`, `uses_language = go`, `deployment_platform = docker` — all from prose mentions, comparisons, negations, instruction/skill text, and English-word collisions (`go` in "want this to go", `express` in "expressive"). This is the documented #48 hallucination problem, now characterized with live data.
+
+**What was ruled out (with data).** Gating fact emission on a usage/claim verb near the entity (`using X`, `deployed on X`, `written in X`) was implemented and measured against `spec/benchmarks/distillation/extraction_spec.rb`:
+
+| | Fact Precision | Fact Recall | F1 |
+|---|---|---|---|
+| baseline | 0.919 | 1.0 | 0.958 |
+| + claim-context | 0.615 | 0.667 | **0.64** |
+
+Regex can't separate terse legit claims (`MongoDB database`, `on AWS`, `Dockerized`) from terse prose mentions (`Postgres/MySQL buy you…`) — claim-context trades recall ~1:1 and even loses precision on the clean corpus. **Reverted.**
+
+**Why distiller-level tightening is mostly off the table (2026-06-17 finding).** The benchmark *enforces* high-recall: case `ext_ent_negative` — "We looked at MongoDB but **decided against it**" — still **expects** `uses_database: mongodb`. The fact distiller is deliberately "extract every mention, filter downstream." So negation/comparison **exclusion** also regresses recall (it would drop the rejected-MongoDB case). The genuine downstream precision lever is Layer-2 / the observation→fact promotion gate, which already prevents Layer-1 noise from being *committed* as corroborated facts.
+
+**Done (recall-safe slice):**
+- ✅ **English-word collision fix** — `go` is matched case-sensitively (`Go`/`golang`) so the verb "go" / "go-to" no longer fires `uses_language=go`. Benchmark Fact Precision **0.919 → 0.935**, Recall held at **1.0** (it removed a real false positive — "my go-to database"). `react`/`rust`/`express` are the same collision class and can follow the same `(?-i:)` pattern, each verified against the benchmark.
+
+**Deferred / not worth it:** wiring `ReferenceMaterialDetector` into the ingest path is a no-op for current Layer-1 (it produces stack predicates, not `convention` facts the detector targets) — skip until Layer-1 emits conventions.
+
+**Acceptance (revised).** Distiller-level work is bounded to recall-safe English-word-collision fixes (benchmark Fact F1 ≥ baseline). Broader fact precision is owned by Layer-2 + the promotion gate, not regex.
+
+**Effort.** Medium. Wiring the existing detector is small; extending its heuristics + a real-transcript precision fixture is the bulk.
+
+**Why this priority.** Fact noise predates the observational layer (it's #48) and is mitigated downstream (promotion gate), so it's not a blocker — but it's the most-visible remaining quality gap on real sessions. The observational layer's Layer-1 observation extraction was tightened in this branch (high-precision/low-recall); facts were deliberately left at baseline pending this recall-preserving approach.
+
+---
+
 ## cq Study (2026-04-28)
 
 Source: docs/influence/cq.md — usefulness-focused study (not internals)
@@ -411,6 +467,43 @@ Source: `docs/influence/ai-memory-systems-2026.md` — meta-study of the Nakajim
 
 ---
 
+## Mastra Observational Memory Study (2026-06-16)
+
+Source: `docs/influence/mastra-observational-memory.md` — architecture study of Mastra's Observational Memory (OM), a text-based dual-agent (Observer + Reflector) episodic memory that compresses raw messages into an append-only, dated observation log living in the context window. SOTA on LongMemEval (84–95%) at 3–6× compression, cache-stable by design.
+
+**Headline finding.** In OM's taxonomy ClaudeMemory is the thing it positions against: a structured *semantic* store injected *dynamically per query*. The gap OM exposes is not retrieval quality — it's that **ClaudeMemory has no episodic layer at all.** Facts answer "what is true"; observations answer "what happened." OM is purely episodic, we are purely semantic. The two are complementary, and we already own analogues of OM's Observer (distillation pipeline) and Reflector (Resolve + Sweep) — they just emit facts, not a narrative log.
+
+### High Priority Recommendations
+
+- [x] **68. Episodic Observation Layer (Observer + Reflector + promotion bridge)** ⭐ — ✅ **Shipped 2026-06-16/17** (phases 1–4)
+  - Value: Adds the missing episodic half of memory (narrative "what happened" log) and a cache-stable injection mode, on top of the existing semantic fact store. The promotion bridge (observation→fact on corroboration) doubles as an anti-hallucination gate for the documented reject-churn problem (distiller commits `uses_database`/`uses_framework` facts from one-off doc example text).
+  - Evidence: `docs/influence/mastra-observational-memory.md`. Our distill pipeline (`lib/claude_memory/distill/`) is already an Observer that emits facts; `resolve/` + `sweep/` is already a Reflector over facts. No episodic store exists.
+  - Implementation (phased):
+    1. ✅ **Shipped 2026-06-16** (schema v19): `observations` table (`body`, `kind`, `priority` 🔴/🟡/🟢, `scope`, `source_content_item_id`, `consolidated_into` lineage, `token_count`); `Domain::Observation`; NullDistiller emits observation rows; Resolver persists them; `memory.observations` read tool. **Append-only with tombstoning, not lossy drop** — preserves provenance.
+    2. ✅ **Shipped 2026-06-16**: two-block SessionStart injection via `ContextInjector` — Block 1 = observation log (🔴 marked, 🟡/🟢 stripped as Mastra does for the actor) ahead of Block 2 = undistilled tail; `Observe::ObservationsRenderer` shared with the published `.claude/rules/claude_memory.observations.md` snapshot; `observation_count` added to the `hook_context` activity event for token/compression measurement.
+    3. ✅ **Shipped 2026-06-17**: `Observe::Reflector` — deterministic, free (no LLM) GC. Dedupes near-identical active observations into the newest (tombstone via `consolidated_into`) and expires stale info-level (🟢) ones past a TTL (`observation_info_ttl_days`, default 30); 🔴/🟡 never expire. Provenance-preserving (rows tombstoned, never deleted). Wired into `Sweep::Maintenance#reflect_observations` → `Sweeper#run!`, so it runs on the existing `PreCompact`/`SessionEnd` sweep — context-pressure-triggered, the analog of Mastra's ~40k-token threshold. (Semantic "merge related/surface patterns" deferred to phase 4 — needs the LLM.)
+    4. ✅ **Shipped 2026-06-17** (schema v20): the observation→fact **promotion bridge**. Dedup folds duplicates' `corroboration_count` into the keeper; once an observation crosses `Domain::Observation::PROMOTION_THRESHOLD` (2) sightings it becomes a promotion candidate. `memory.promote_observation` creates the fact via the resolver, links provenance, marks the observation promoted, and **refuses uncorroborated observations server-side** — the anti-hallucination gate. `ContextInjector` surfaces candidates in a SessionStart "## Observation Reflection" section instructing Claude to promote inline (automatic semantic reflection, no extra API cost); a manual `/reflect` skill drives deep on-demand passes. (Trigger is SessionStart rather than PreCompact — the already-wired free injection point; a PreCompact context hook is a possible future refinement.)
+    5. ✅ **Shipped 2026-06-17** (branch `claude/observational-layer-complete`): the LLM half. **Layer-2 Claude-as-observer** — the SessionStart extraction prompt asks Claude to emit episodic observations in the `observations` field of `memory.store_extraction` (coerced/validated at the handler border, persisted via the resolver), making the log rich where Layer-1 regex is high-precision/low-recall. **Semantic reflection** — `memory.consolidate_observations` merges related-but-differently-worded observations into one synthesized row with *combined* corroboration (which can tip it over the promotion gate), tombstoning the sources; surfaced in the reflection section + `/reflect`. (Also fixed a latent Liskov bug: `ReferenceMaterialDetector#reclassify` dropped observations when a fact was present.)
+    6. ✅ **Shipped 2026-06-17** (branch `claude/observational-layer-complete`): observability + measurement. `Dashboard::Observations` panel (`/api/observations`, Advanced → Observations tab) — counts by status/kind/priority, corroboration + promotion readiness, recent timeline, and a **compression ratio** (source content tokens ÷ observation tokens, Mastra-style). The compression metric is the measurement half of design rec E; a full LongMemEval-style episodic benchmark remains (overlaps #67).
+    7. ✅ **Shipped 2026-06-18** (branch `claude/observational-layer-complete`): polish. **PreCompact reflection trigger** — `claude-memory hook context` injects only the reflection nudge (`ContextInjector#reflection_context`) on PreCompact (context pressure, the Mastra token-threshold analog), wired into the standard PreCompact hook set; not the full snapshot. **Observation↔fact provenance** — `memory.observations` exposes status/corroboration_count/promoted_fact_id/consolidated_into; `memory.explain(fact_id)` shows `promoted_from_observations` (reverse link via `observations_for_fact`). Full observational layer complete; remaining: LongMemEval episodic benchmark (#67).
+  - Effort: Large, phased. Phase 1 ~2-3 days; full arc ~2 weeks. Reuses distill/resolve/sweep/publish/context-hook machinery and `context_tokens` telemetry.
+  - Trade-off: reflection is automatic on *lifecycle events* (compaction/session boundaries), not a wall clock — Claude Code has no timer/cron hook, and Routines/subagents incur separate token budgets (rejected). Observer/Reflector reuse the existing session (no extra API cost). **Augments dynamic recall, does not replace it (user-confirmed 2026-06-16).** See claude-code-guide consultation in the influence doc.
+
+### Medium Priority
+
+- [ ] **Compression / cache telemetry + LongMemEval episodic suite** (see influence doc rec E)
+  - Value: Report compression ratio and token reduction on Trust/Health panels using existing `context_tokens` events (0.11.0). Add a LongMemEval-style long-session suite to DevMemBench to score the episodic layer. Overlaps with existing item #67 (LongMemEval integration) — coordinate.
+  - Effort: Medium. Depends on #68 phase 1-2.
+
+### Features to Avoid (from this study)
+
+- **Two always-on background LLM agents** — violates the no-separate-API-call convention. Observer = context-hook injection; Reflector = deterministic shell-side GC + `PreCompact`-injected semantic consolidation (rides the existing session).
+- **Claude Code Routines / subagents for recurring reflection** — Routines run as a separate scheduled cloud session; subagents run in their own context (~7× tokens). Both incur extra spend; reserve only for an explicitly opted-in one-off backfill.
+- **Lossy drop on reflection** ("never forgives") — we tombstone via `consolidated_into` and retain raw `content_items`; provenance is non-negotiable.
+- **Replacing dynamic recall with a wholesale-loaded log** — augment, don't replace; keep `memory.recall` for targeted lookups.
+
+---
+
 ## Medium Priority
 
 ### ~~18. Shell Completion for CLI~~ ✅ Implemented 2026-03-20

data/docs/influence/mastra-observational-memory.md

@@ -0,0 +1,198 @@
+# Mastra Observational Memory — Influence Study
+
+*Analysis Date: 2026-06-16*
+*Source: Mastra "Observational Memory" (announcement + docs + research, Feb 2026)*
+*Type: Architecture study (feature/paradigm, not a full repo clone)*
+*Status: Design exploration — no code yet. Branch `claude/observational-layer-design-7662r9`.*
+
+*Sources:*
+- *[Announcing Observational Memory (Mastra blog)](https://mastra.ai/blog/observational-memory)*
+- *[Observational Memory docs](https://mastra.ai/docs/memory/observational-memory)*
+- *[Observational Memory research / LongMemEval](https://mastra.ai/research/observational-memory)*
+- *[VentureBeat: "Observational memory cuts AI agent costs 10x..."](https://venturebeat.com/data/observational-memory-cuts-ai-agent-costs-10x-and-outscores-rag-on-long)*
+- *[The Decoder: traffic-light priority system](https://the-decoder.com/mastras-open-source-ai-memory-uses-traffic-light-emojis-for-more-efficient-compression/)*
+
+---
+
+## Executive Summary
+
+### What this is
+
+Mastra Observational Memory (OM) is a **text-based, dual-agent episodic memory** for long-running agents. It compresses raw message history into a structured, append-only log of dated **observations** that lives entirely in the LLM context window — no vector or graph DB. It reports state-of-the-art LongMemEval scores (84.23% with gpt-4o; 94.87% with gpt-5-mini) at 3–6× token compression.
+
+### Why ClaudeMemory cares
+
+In Mastra's taxonomy, ClaudeMemory is the thing OM positions *against*: a structured **semantic** store (subject-predicate-object facts with scope, validity windows, supersession, provenance) injected **dynamically per query** via `memory.recall` and SessionStart fact injection.
+
+The key realization from this study: **ClaudeMemory has no episodic layer at all.** Facts answer "what is true." Observations answer "what happened." OM is purely episodic; ClaudeMemory is purely semantic. An observational layer is not redundant with distillation — it is the missing half.
+
+We already own two of OM's four moving parts in spirit:
+- The **distillation pipeline** (NullDistiller + Layer-2 Claude-as-distiller) is an Observer that emits *facts* instead of a *narrative log*.
+- **Resolve + Sweep** is a Reflector that operates on *facts* instead of *observations*.
+
+The work is therefore: add a narrative episodic store, point the existing Observer/Reflector machinery at it, add a cache-stable injection mode, and — uniquely to us — bridge observations into facts via corroboration.
+
+---
+
+## How Mastra OM Works
+
+### Two-block context window
+
+1. **Observation block** — a compressed, append-only log of dated observations (decisions, key events, distilled facts from older messages). Reads like a log of decisions and actions, not documentation.
+2. **Raw tail** — recent messages not yet compressed.
+
+### The Observer
+
+Fires when raw message tokens cross ~30k (configurable). A separate background agent compresses messages into new dated observations appended to the observation block. Each observation captures one discrete event: a user statement, an agent action, a tool-call result, or a preference expressed in passing. 3–6× compression.
+
+### The Reflector
+
+Fires when observations cross ~40k tokens (configurable). A separate background agent garbage-collects: combines related items, reflects on overarching patterns, and drops context that no longer matters.
+
+### Traffic-light priority
+
+Observations carry 🔴 (important) / 🟡 (maybe important) / 🟢 (info only). The priority is **internal** to the Observer/Reflector pipeline. When observations are presented to the main "Actor" agent, 🟡 and 🟢 are stripped — only 🔴 survives — because the priority emojis serve the memory pipeline and are visual noise to the actor.
+
+### Prompt-cache stability (the headline win)
+
+Because the observation block is **append-only between reflections**, the prompt prefix stays stable and every turn gets a full cache hit. Cache invalidates only on a reflection, which is infrequent. This is explicitly contrasted with RAG-style memory that re-retrieves and rewrites the prompt every turn, busting the cache and producing a variable cost curve.
+
+### Storage
+
+Plain text in a standard backend (Postgres / LibSQL / MongoDB), loaded directly into the context window — not pulled through embedding search.
+
+---
+
+## Comparative Analysis vs ClaudeMemory
+
+| Dimension | Mastra OM | ClaudeMemory today |
+|-----------|-----------|--------------------|
+| Memory type | Episodic (narrative log) | Semantic (SPO facts) |
+| Storage | Plain text in context window | Normalized SQLite + FTS5 + vec0 |
+| Retrieval | None — log loaded wholesale | Dynamic per-query (FTS + vector RRF) |
+| Compression | Observer (LLM), 3–6× | NullDistiller + Claude-as-distiller → facts |
+| Consolidation | Reflector (LLM), lossy drop | Resolve (supersession) + Sweep (TTL/GC) |
+| Provenance | Weak — compression is lossy | Strong — provenance receipts, lineage |
+| Cache behavior | Stable append-only prefix | Per-query injection (cache-busting) |
+| Cost | Two background LLM agents (extra API $) | Claude-as-distiller, zero extra API $ |
+
+**The two systems are complementary, not competing.** OM's weakness is exactly ClaudeMemory's strength (provenance, truth maintenance) and vice versa (episodic recall, cache-stable injection).
+
+---
+
+## Adoption Opportunities (prioritized)
+
+### High Priority
+
+**A. Episodic observation store + Layer-1 Observer.** New `observations` table (schema v19 — v18 was taken by OTel telemetry); NullDistiller emits observation rows alongside facts; `memory.observations` read tool. Append-only with `consolidated_into` lineage (mirrors `fact_links`) rather than Mastra's lossy drop — preserves our provenance guarantee. Zero behavior change to facts.
+
+**B. Cache-stable injection.** Publish `.claude/rules/claude_memory.observations.md` (append-only, dated, 🔴+plain only — 🟡/🟢 stripped as Mastra does for the actor). SessionStart injects a two-block context: Block 1 = consolidated observations (stable, cache-friendly), Block 2 = recent undistilled tail. Front-loading a stable block reduces the per-turn `memory.recall` churn that busts caching. *Honest limit:* we influence Claude Code's cache via a stable `additionalContext` prefix within a session; we don't control it. Cross-session caching remains Claude Code's domain.
+
+**C. The observation→fact promotion bridge (unique to us).** The Reflector promotes *corroborated* observations into structured facts. An observation is low-commitment; a fact is committed truth. Requiring repeated, corroborated sightings before promotion is a natural confidence gate — and directly mitigates the documented hallucination problem where the distiller commits `uses_database`/`uses_framework` facts from one-off example text in docs (today producing reject churn). Observation-first, fact-on-corroboration makes premature hallucinated facts never commit.
+
+### Medium Priority
+
+**D. Automatic Reflector (free) — confirmed feasible.** A consultation with the claude-code-guide agent (2026-06-16) confirms automatic reflection is achievable with zero extra API cost, in two tiers:
+- **Deterministic tier (fully autonomous, no model):** dedupe near-identical observations, drop stale 🟢 past a TTL, merge by entity/time window — pure Ruby, run shell-side inside the `PreCompact` and `SessionEnd` hooks (and the existing Sweep). This needs no model and fires automatically.
+- **Semantic tier (autonomous-on-next-turn, rides the session):** at `PreCompact`, the hook injects a reflection instruction via `additionalContext` ("consolidate the observation log: combine related items, surface patterns, drop the irrelevant"). Claude Code itself performs the consolidation on its next turn, inside the existing session — no separate paid call.
+
+See the dedicated section below for why `PreCompact` is the right trigger and what the constraints are. This **supersedes** the earlier "manual `/reflect` only" recommendation: `/reflect` remains as a manual on-demand deep pass, but reflection is now primarily automatic.
+
+**E. Compression / cache telemetry.** Reuse the `context_tokens` telemetry on `hook_context` events (0.11.0) and the Trust/Health panels to report compression ratio and token reduction. Add a LongMemEval-style episodic/long-session suite to DevMemBench alongside the existing retrieval and truth-maintenance suites.
+
+### Features to Avoid (from this study)
+
+- **Two always-on background LLM agents.** Violates the standing convention against features requiring separate Anthropic API calls. Our Observer = context-hook injection (Claude-as-distiller); our Reflector = deterministic shell-side GC + `PreCompact`-injected semantic consolidation that rides the existing session (see automatic-reflection section).
+- **Claude Code Routines / subagents for reflection.** Routines run as a separate scheduled cloud session (separate token budget); subagents run in their own context window (~7× token burn). Both incur extra spend — rejected for recurring reflection. Reserve them, if ever, for a one-off heavy backfill the user explicitly opts into.
+- **Lossy drop on reflection.** Mastra truly discards observations ("never forgives"). We tombstone via `consolidated_into` and retain raw `content_items` — provenance is non-negotiable.
+- **Replacing dynamic recall.** Augment, don't replace. Observations become a front-loaded episodic block; `memory.recall` stays for targeted lookups.
+
+---
+
+## Proposed Data Model (sketch)
+
+```
+observations  (schema v19)
+  id, ts (event time), session_id
+  body            -- dense narrative text, the observation itself
+  kind            -- user_statement | agent_action | tool_result | preference | decision | event
+  priority        -- 1=🔴 important, 2=🟡 maybe, 3=🟢 info  (internal pipeline signal)
+  scope, project_path
+  source_content_item_id   -- provenance back to the raw transcript chunk
+  consolidated_into        -- Reflector lineage (mirrors fact_links supersession)
+  token_count              -- for budget / compression math
+  status, created_at, reflected_at
+```
+
+## Proposed Pipeline Integration
+
+```
+Transcripts → Ingest → Index (FTS5)
+                   ↓
+   ┌─────────────── Distill ───────────────┐
+   │                                         │
+ Facts (SPO, semantic)            Observations (narrative, episodic)  ← NEW
+   │                                         │
+ Resolve (truth maint.)          Reflect (consolidate / GC / pattern) ← NEW
+   │                                         │
+ Store (facts)                   Store (observations)                 ← NEW
+   │                                         │
+   └──────────── Promotion bridge ──────────┘
+           (Reflector promotes corroborated observations → facts)
+                   ↓
+   Publish: stable observation block (cache-friendly) + fact snapshot
+```
+
+## Automatic Reflection in Claude Code (consultation findings, 2026-06-16)
+
+Source: claude-code-guide agent consultation. Citations: [Hooks reference](https://code.claude.com/docs/en/hooks.md), [Subagents](https://code.claude.com/docs/en/subagents.md), [Routines / scheduled tasks](https://code.claude.com/docs/en/web-scheduled-tasks).
+
+**What does not exist:** There is no timer-, cron-, or idle-based hook event. Hook events are lifecycle-driven only — `SessionStart`, `SessionEnd`, `UserPromptSubmit`, `Stop`/`StopFailure`, `PreCompact`/`PostCompact`, `PreToolUse`/`PostToolUse(Failure)`, plus async signals (`FileChanged`, etc.). No hook can force a model turn or enqueue a prompt; a hook can only inject `additionalContext` that the model acts on at its *next* invocation.
+
+**What this unlocks anyway:** `PreCompact` is the right reflection trigger because it fires precisely when the context window is filling — i.e. on *context pressure*. That is conceptually the same signal Mastra uses (Reflector fires at a ~40k-token observation threshold). So "reflect when memory gets big" maps cleanly onto "reflect when Claude Code is about to compact."
+
+**The free automatic pattern (recommended):**
+- `PreCompact` + `SessionEnd` hooks run the **deterministic** Reflector shell-side in Ruby (dedupe / TTL-drop 🟢 / merge) — fully autonomous, no model, no cost.
+- `PreCompact` injects an `additionalContext` instruction that makes Claude perform the **semantic** consolidation (pattern-finding, observation→fact promotion) on its next turn, inside the existing session — no separate paid call.
+- `SessionStart` injects the consolidated two-block observation log (already in recommendation B).
+
+**Where extra cost is unavoidable (and therefore rejected):** truly autonomous *between-session* reflection on a wall clock. That requires Claude Code Routines (separate paid cloud session) or a headless `claude -p` call or a subagent (~7× tokens) — all separate spend. We accept the tradeoff: our reflection is automatic on *lifecycle events* (compaction, session boundaries), not on a wall-clock timer. For our single-developer, local-first scale this is sufficient.
+
+## Suggested Phasing
+
+1. Schema + Layer-1 Observer (table, NullDistiller rows, `memory.observations`).
+2. Stable two-block injection; measure token/compression deltas.
+3. **Automatic Reflector**: deterministic GC shell-side in `PreCompact` + `SessionEnd`/Sweep.
+4. **Automatic semantic reflection**: `PreCompact` `additionalContext` consolidation instruction + observation→fact promotion bridge. Keep a manual `/reflect` skill for on-demand deep passes.
+
+Phase 4 is where this stops being "Mastra-on-Ruby" and becomes a hybrid episodic+semantic system stronger than either alone.
+
+## Decisions for ClaudeMemory (memory-convention format)
+
+Per the `/study-repo` memory discipline, the following are decisions about **claude_memory itself** derived from this study — to be stored via `memory.store_extraction` (`subject=claude_memory`, `decision`/`architecture` predicate, reason clause embedded) once the memory MCP server is connected. External facts about Mastra stay in this influence doc, not in memory.
+
+- **Decision:** claude_memory will add an episodic observation layer that *augments* (does not replace) the dynamic-recall semantic fact store — because facts answer "what is true" and observations answer "what happened," and we currently have no episodic half; recall stays for targeted lookups while observations provide a stable front-loaded narrative. (User-confirmed "augment" on 2026-06-16.)
+- **Decision:** observation reflection will be automatic via the `PreCompact` and `SessionEnd` hooks rather than a manual-only skill — because Claude Code exposes no timer/cron hook, but `PreCompact` fires on context pressure (the analog of Mastra's token-threshold trigger) and rides the existing session at no extra API cost.
+- **Decision:** the Reflector's deterministic GC runs shell-side in Ruby and its semantic consolidation runs via `PreCompact` `additionalContext` (Claude-as-reflector inline) — to keep automatic reflection within the no-extra-API-cost convention, explicitly rejecting Claude Code Routines and subagents because each incurs a separate token budget.
+- **Decision:** reflection will tombstone superseded observations via a `consolidated_into` link rather than hard-deleting them (unlike Mastra's lossy drop) — to preserve claude_memory's provenance guarantee while still bounding context size.
+- **Decision:** an observation is promoted to a structured fact only after corroboration across multiple observations — because requiring repeated sightings before commitment doubles as an anti-hallucination gate against the documented reject-churn from one-off doc/example text.
+
+> **Pending memory ingestion (deferred 2026-06-16).** These were NOT written to the project DB in the remote design session: `.claude/memory.sqlite3` is git-LFS-backed and the container had only the pointer (no `git-lfs`, real DB not materialized), so writing would have clobbered it. Store them in a local session via `memory.store_extraction` with this payload:
+>
+> ```json
+> {
+>   "scope": "project",
+>   "facts": [
+>     {"subject": "claude_memory", "predicate": "decision", "object": "Add an episodic observation layer that augments (does not replace) the dynamic-recall semantic fact store, because facts answer 'what is true' and observations answer 'what happened' and we currently lack the episodic half; recall stays for targeted lookups."},
+>     {"subject": "claude_memory", "predicate": "decision", "object": "Make observation reflection automatic via the PreCompact and SessionEnd hooks rather than a manual-only skill, because Claude Code exposes no timer/cron hook but PreCompact fires on context pressure (the analog of Mastra's token threshold) and rides the existing session at no extra API cost."},
+>     {"subject": "claude_memory", "predicate": "decision", "object": "Run the Reflector's deterministic GC shell-side in Ruby and its semantic consolidation via PreCompact additionalContext (Claude-as-reflector inline), to keep automatic reflection within the no-extra-API-cost convention, explicitly rejecting Claude Code Routines and subagents because each incurs a separate token budget."},
+>     {"subject": "claude_memory", "predicate": "decision", "object": "Tombstone superseded observations via a consolidated_into link rather than hard-deleting them (unlike Mastra's lossy drop), to preserve claude_memory's provenance guarantee while still bounding context size."},
+>     {"subject": "claude_memory", "predicate": "decision", "object": "Promote an observation to a structured fact only after corroboration across multiple observations, because requiring repeated sightings before commitment doubles as an anti-hallucination gate against reject-churn from one-off doc/example text."}
+>   ]
+> }
+> ```
+
+## Open Questions
+
+- **Augment vs replace recall?** Resolved: **augment** (user-confirmed 2026-06-16). Observations become a front-loaded episodic block; `memory.recall` stays for targeted lookups.
+- **Automatic vs manual reflection?** Resolved: **automatic** via `PreCompact`/`SessionEnd` (deterministic GC shell-side + semantic consolidation injected for the next turn), with `/reflect` retained for manual deep passes. The only thing we forgo is wall-clock between-session reflection, which would cost extra (Routines/subagents) — deliberately rejected.

data/docs/influence/strands-agent-sops.md

@@ -0,0 +1,163 @@
+# Strands Agent SOPs Analysis
+
+*Analysis Date: 2026-05-01*
+*Source: AWS Open Source Blog — "Introducing Strands Agent SOPs: Natural Language Workflows for AI Agents"*
+*URL: https://aws.amazon.com/blogs/opensource/introducing-strands-agent-sops-natural-language-workflows-for-ai-agents/*
+*Type: Article (not a repo). PyPI package `strands-agents-sops`, GitHub `strands-agents/agent-sop`.*
+
+---
+
+## Executive Summary
+
+**Agent SOPs** are markdown-based "Standard Operating Procedures" that wrap an AI agent's instructions in a parameterized, RFC-2119-keyworded, chain-able format. Amazon teams use thousands of them internally; the open-source release ships four reference SOPs (`codebase-summary`, `pdd`, `code-task-generator`, `code-assist`) and tooling to author/load them via Strands Agents, MCP prompts, Claude Skills, or raw model calls.
+
+**Verdict for ClaudeMemory**: ClaudeMemory already implements most of what SOPs propose, under different names. Skills (`/distill-transcripts`, `/release`, `/study-repo`) *are* SOPs. The hook context injection pipeline already chains stages (ingest → distill → resolve → publish). The current distillation prompt already uses RFC-2119-ish "MUST" language for the reason-clause requirement.
+
+The genuinely novel ideas — and the only ones worth a closer look — are:
+1. **Explicit parameter contracts** at SOP entry (Required/Optional with defaults), versus our skills that take freeform `$ARGUMENTS`.
+2. **Progress checkpoints + resumability** for long-running workflows (`✅ Step 1 complete` style markers the agent emits).
+3. **Self-describing format spec** (`strands-agents-sops rule` command) that lets Claude author new SOPs from a description.
+
+The rest is old news for us. **Recommendation: do not adopt the Strands library or format. Borrow two narrow ideas (resumability + explicit parameters) into `/distill-transcripts` if and only if real distillation runs are large enough to fail mid-batch.**
+
+## What an SOP Actually Is
+
+Per the article, an SOP is markdown with these conventions:
+
+- **RFC 2119 keywords** (MUST / SHOULD / MAY) for behavioral control.
+- **Required/Optional parameters block** with defaults — gathered from the user via natural-language dialogue at invocation time.
+- **Numbered steps** the agent executes sequentially.
+- **Progress annotations** the agent prints as it goes (`✅ Validated codebase path exists`).
+- **Output artifacts** in a conventional `.sop/<name>/` directory, used as handoff between chained SOPs.
+
+Example parameter declaration (the only verbatim format snippet in the article):
+
+```
+Required Parameters:
+• codebase_path: Path to the codebase to analyze
+Optional Parameters:
+• output_dir: Directory where documentation will be stored (default: ".sop/summary")
+```
+
+Invocation surfaces:
+
+- **Strands Agents (Python)**: `Agent(system_prompt=code_assist, tools=[editor, shell])` — SOP becomes the system prompt.
+- **MCP**: SOPs registered as MCP *prompts* (the `prompts/list` + `prompts/get` channel), invoked with `@codebase-summary` in Kiro CLI / `/prompts` listing.
+- **Claude Skills**: A CLI converts SOPs to Anthropic Skill format.
+- **Direct LLM**: paste into a model's message and run.
+
+Composition is **sequential chaining via artifact handoff** — `codebase-summary` writes docs, `pdd` reads them. No nesting/include directive is shown.
+
+## How This Maps to ClaudeMemory Today
+
+| Strands concept | Our equivalent | Status |
+|---|---|---|
+| Markdown SOP | `lib/claude_memory/commands/skills/*.md` (Anthropic Skills) | ✅ Have it |
+| MCP prompts surface | `MCP::QueryGuide` registers `memory_guide` via `prompts/list`+`prompts/get` | ✅ Have it |
+| RFC-2119 "MUST" in instructions | `distill-transcripts.md:38-43` uses MUST for reason-clause embed | ✅ Have it |
+| SessionStart prompt injection | `hook_command.rb:213` writes `hookSpecificOutput.additionalContext` | ✅ Have it |
+| SOP chaining via artifacts | `Ingest → Distill → Resolve → Store → Publish` (CLAUDE.md L72-79) | ✅ Have it (DB rows are the artifacts) |
+| AI-assisted SOP authoring | `/skill-creator` skill | ✅ Have it |
+| Format spec exposable to Claude | `strands-agents-sops rule` CLI | ⚠️ Partial — our distillation prompt is in `distill-transcripts.md`, not exposed as a tool |
+| Required/Optional parameter contract | We pass `$ARGUMENTS` as freeform text | ❌ Missing |
+| Progress checkpoints + resumability | `/distill-transcripts` runs end-to-end; no mid-batch checkpoint | ❌ Missing |
+| Reference SOPs (`codebase-summary` etc.) | N/A — wrong domain | ❌ Not applicable |
+
+The pattern is clear: we independently arrived at the same architecture. The two bullets in the "Missing" rows are the only candidates worth thinking about for adoption.
+
+## Where SOPs Could Improve Distillation
+
+### 1. Resumability for `/distill-transcripts`
+
+**Current state.** `/distill-transcripts --limit 10` calls `memory.undistilled`, processes items one-by-one, calls `memory.mark_distilled` after each. If the run aborts mid-batch (rate limit, context exhaustion, user Ctrl-C), the items processed before the abort are marked, the rest are not. There is no explicit checkpoint file, but the DB itself is the checkpoint.
+
+**SOPs angle.** SOPs add visible progress markers (`✅ Item 4/10 complete`) and a resume contract (`if .sop/distill/state.json exists, skip processed items`).
+
+**Honest verdict.** Our DB-as-checkpoint already handles resumability. The visible-progress angle is a UX win for big runs but not a correctness improvement. **Do this only if we add a `--limit 100`+ workflow that users actually run.** Today nobody runs that.
+
+### 2. Required/Optional Parameter Block in Skills
+
+**Current state.** `/distill-transcripts` accepts `--limit N` parsed implicitly inside the skill body. Other skills accept `$ARGUMENTS` as a freeform blob. Users discover parameters by reading the skill markdown.
+
+**SOPs angle.** Declared parameter blocks let the agent prompt the user before running ("what's the codebase_path?"), and let tooling (an SOP registry, MCP prompt list) introspect the contract.
+
+**Honest verdict.** Anthropic Skills allow YAML frontmatter that already does this (`argument-hint`, parameter docs). We are under-using that frontmatter. **Cheap, safe improvement.** Adding a `Parameters:` block to the top of `distill-transcripts.md`, `release.md`, `study-repo.md` (and friends) costs ~10 minutes per skill and makes them self-documenting to both humans and any agent reading them.
+
+### 3. Format-Spec-As-Tool for Authoring
+
+**Current state.** `/skill-creator` exists. It has the format knowledge in its prompt body.
+
+**SOPs angle.** `strands-agents-sops rule` is a CLI command that prints the SOP format spec to stdout, so any agent can `Bash` it and learn how to author one. This is a small but real ergonomic win — the spec lives in one place, not duplicated into every "make a new skill" prompt.
+
+**Honest verdict.** Marginal. We don't have a sprawl of skill-authoring locations to consolidate. **Defer indefinitely** unless we start writing many more skills.
+
+## What NOT to Adopt
+
+- **The Python package itself.** Strands is a Python agent framework; we're a Ruby gem. No reuse path.
+- **The `.sop/<name>/` artifact directory convention.** We persist via DB rows + `claude_memory.generated.md`. Adding a parallel filesystem artifact tree would just add cleanup burden and an out-of-DB state to reconcile.
+- **The four reference SOPs (`codebase-summary` etc.).** Wrong domain — they're for code-workflow agents, not memory pipelines. Nothing to lift.
+- **Renaming "skills" to "SOPs" in our docs.** Anthropic's term is *Skills*; that's the term Claude Code users know. Adopting Amazon's term creates confusion for zero gain.
+- **Sequential-only chaining as an enforced pattern.** Our pipeline already chains, but we should keep room for parallel work (e.g., NullDistiller layer 1 runs synchronously in the ingest hook regardless of layer 2/3). SOP chaining is sequential by construction.
+
+## Adoption Opportunities
+
+### Medium Priority
+
+#### 1. Parameter blocks in skill frontmatter
+
+- **Value**: Self-documenting skills; Claude can prompt the user for missing parameters instead of guessing from `$ARGUMENTS`. Better intro-spectability for any future skill registry UI.
+- **Evidence**: Article's `Required Parameters / Optional Parameters` block — only structural snippet quoted verbatim; Anthropic Skills format already supports `argument-hint` and similar fields we under-use.
+- **Implementation**: Add `## Parameters` section near the top of `lib/claude_memory/commands/skills/distill-transcripts.md`, `release.md`, `study-repo.md`, `quality-update.md`, `improve.md`. Format: bullet list with `name: description (default: …)`.
+- **Effort**: ~30 minutes total across all skills.
+- **Trade-off**: Tiny doc maintenance burden; no runtime cost.
+- **Recommendation**: ADOPT (low-cost, high-clarity).
+
+### Low Priority
+
+#### 2. Progress markers + explicit checkpoint file in `/distill-transcripts`
+
+- **Value**: Better UX on long runs (users see progress); cleaner resume after mid-batch failure.
+- **Evidence**: Article shows `✅ Validated codebase path exists` style output; SOPs document progress to support resumability.
+- **Implementation**: Have `/distill-transcripts` print `[N/M] item <docid> → K facts` after each `memory.mark_distilled`. Optionally, write `.claude/distill_state.json` with `last_processed_content_id` so a re-run can resume.
+- **Effort**: ~1 hour for stdout markers; ~3 hours including a state file with safe-resume semantics.
+- **Trade-off**: State file adds another moving piece; DB already handles correctness, so this is purely UX. Not worth doing until somebody actually runs `/distill-transcripts --limit 100+` regularly.
+- **Recommendation**: DEFER. Revisit if dashboard/usage data shows multi-hundred-item distillation runs.
+
+#### 3. SOP-style format spec exposed as MCP prompt
+
+- **Value**: Lets a future "make me a new skill" agent fetch our skill format spec via MCP `prompts/get` instead of duplicating it.
+- **Evidence**: Article's `strands-agents-sops rule` command — same idea.
+- **Implementation**: Add a `skill_authoring_guide` prompt to `MCP::QueryGuide` alongside `memory_guide`.
+- **Effort**: ~1 hour.
+- **Trade-off**: Solves a problem we don't yet have. We have one skill-authoring location (`/skill-creator`).
+- **Recommendation**: DEFER until skill sprawl is a real problem.
+
+### Features to Avoid
+
+- **Generic "SOP runtime" abstraction** layered over our skills: pure ceremony. Anthropic Skills already give us the runtime.
+- **`.sop/<name>/` artifact filesystem** parallel to our DB: doubles state, doubles cleanup, halves the value of having a curated SQLite store.
+- **Adopting the term "SOP" anywhere user-facing**: term collision with Skills.
+
+## Implementation Recommendations
+
+**Phase 1 (do this in any 0.12.x release).** Add `## Parameters` blocks to the existing skill markdowns. ~30 minutes. Closes the only meaningful gap from this study.
+
+**Phase 2 (defer).** Progress markers + `.claude/distill_state.json` checkpoint, only after we see real users running large distillation batches.
+
+**Phase 3 (avoid unless triggered).** MCP-prompt-exposed skill format spec, only after we have ≥3 skill-authoring locations to consolidate.
+
+## Architecture Decisions
+
+**Preserve.** Our DB-as-checkpoint substrate, our use of Anthropic Skills as the SOP equivalent, our `additionalContext` injection on SessionStart, our distillation prompt's explicit reason-clause requirement.
+
+**Adopt.** Explicit parameter declarations in skill frontmatter (Phase 1).
+
+**Reject.** Strands Python package, `.sop/` artifact tree, generic SOP runtime abstraction, terminology adoption ("SOP" → user-facing).
+
+## Key Takeaways
+
+1. **We are already doing this.** Strands describes a class of patterns — markdown instructions, MCP prompts, parameterized invocation, sequential chaining, RFC-2119 vocabulary — that ClaudeMemory has independently. The existence of Strands is *validation*, not a roadmap.
+2. **Anthropic Skills ≈ Strands SOPs.** Same idea, different label, different ecosystem. Don't refactor toward Strands; we'd just be renaming Skills.
+3. **One narrow win.** Explicit parameter declarations in skill frontmatter cost ~30 minutes and make our skills self-documenting. Worth doing.
+4. **One narrow defer.** Progress markers + checkpoint files in `/distill-transcripts` are real UX improvements *if* anyone runs distillation at scale; today nobody does. Revisit when the data says to.
+5. **No deep architectural shifts.** Nothing in the article justifies a redesign of our distillation, storage, or prompting pipelines.

data/docs/quality_review.md

@@ -9,6 +9,51 @@
 
 ---
 
+## Observational Layer — Pre-Merge Review (2026-06-18)
+
+**Review Date:** 2026-06-18
+**Previous Review:** 2026-04-28 (51 days ago)
+**Scope:** the observational-layer branch (`claude/observational-layer-design-7662r9`, 22 commits ahead of `origin/main`, ~57 files). Two parallel expert-lens reviews — core/data layer (migrations 019/020, `Domain::Observation`, `SQLiteStore` observation methods, `Resolver`) and pipeline layer (`NullDistiller` extraction, renderer, `Reflector`, `ContextInjector`, MCP handlers, dashboard panel).
+
+**Verdict:** No hard merge-blockers. The append-only/tombstone discipline is consistent, `Domain::Observation` is a clean immutable value object, border validation (`coerce_observation`) is textbook, and test coverage on this surface is above the repo average. The items below are latent correctness edge-cases + cleanups; none break the system as shipped (the layer is experimental and observations are project-scoped only today).
+
+### High — address or consciously accept before merge
+
+- **H1 · `consolidate_observations` read-modify-write race** — `lib/claude_memory/store/sqlite_store.rb:805-826` (Evans/Bernhardt). The source `SELECT` and `combined = sources.sum{…}` run *outside* the `@db.transaction` block, and the tombstone `UPDATE` (822) doesn't re-assert `status: "active"`. Two reflectors firing close together (PreCompact + SessionEnd) could double-count corroboration or re-tombstone an already-consolidated source. SQLite's single-writer lock narrows the window but doesn't close the gap. **Fix:** move the read inside the transaction and re-filter `status: "active"` on the update. Mechanical, ~1h incl. spec.
+- **P1 · `noise_body?` over-broad — drops legit prose** — `lib/claude_memory/distill/null_distiller.rb:53,169` (Grimm/Bernhardt). `NOISE_BODY_SIGNATURE` matches `::`, `{}`, `=>` anywhere, so `"decided to adopt ClaudeMemory::Observation as the model"` is silently dropped — common in Ruby prose. Confirmed at runtime. **This is a precision-tuning *design* change to extraction behavior, not a mechanical fix** — per the project's data-driven-design convention it should be surveyed against real corpus data before retuning, not changed blind. Candidate: narrow to strong structural markers (`def `/`class `/`module `, JSON `","`/`":\s*"`, `$(`, `&&`, `||`), drop bare `::`/`{}`/`=>`, add a false-negative spec corpus.
+- **P2 · cross-scope promote nudge — latent wrong-DB landmine** — `lib/claude_memory/hook/context_injector.rb:159-194` + `lib/claude_memory/mcp/handlers/management_handlers.rb:88` (Evans/Beck). `fetch_promotion_candidates` flat-maps project+global stores; the reflection block emits `[obs #<id>]` (a *per-DB* autoincrement id) with no scope; `promote_observation`/`consolidate_observations` default `scope: "project"`. A global-store candidate would route the promote call to the wrong DB. **Dead today** (nothing writes observations to the global DB), but a genuine landmine if global observations ever appear. **Fix:** restrict reflection candidates to the project store + document, or scope-tag the nudge line (mirror `emitted_facts_by_scope`). ~1-2h.
+- **M1 · `consolidate_observations` has zero test coverage** — the most complex method in `sqlite_store.rb` is the only observation method with no specs (the `< 2 → nil` guard, summed-corroboration-tips-threshold, multi-row tombstone are all load-bearing and untested). Add specs alongside the H1 fix. ~1h.
+
+### Medium
+
+- **M2/P7 · token-estimate `/4.0` heuristic duplicated 3×** — `sqlite_store.rb:714,819` + `dashboard/observations.rb:98` (Metz DRY). The compression-ratio correctness depends on both halves using the same divisor. Extract `Core::TokenEstimate.from_chars`/`.from_bytes`. ~1h.
+- **M3/P10 · `recent_observations` `min_priority` name inverted** — `sqlite_store.rb:731` (Beck revealing-names). Filters `priority <= min_priority`, but priority is inverted (1=important), so a higher "minimum" returns *more* rows. Rename `max_priority_value`/`importance_floor`. ~30m + callers.
+- **M4 · `Resolver#apply` `@return` rdoc stale** — `resolver.rb:29` omits the `:observations_created` and `:fact_ids` keys this PR adds. ~10m.
+- **M5 · `fact_ids` array silently contains `nil`s** — `resolver.rb:45,61` (Grimm meaningful-returns). Comment promises positional alignment with `extraction.facts`, but `:discard` contributes `nil`; the sole consumer already `.compact.first`s. Pick one contract and document it (or compact at source). ~20m.
+- **P3 · `clean_observation_body` is 6 chained gsubs, brittle** — `null_distiller.rb:178` (Bernhardt). Pure text logic buried as a private method; extract to a tested `Observe::BodyCleaner` with an input→output spec table. ~2h.
+- **P4 · `extract_decisions`/`extract_observations` double-scan `DECISION_PATTERNS`** — `null_distiller.rb:105,138` (Metz DRY). Two full regex passes per chunk on the P95<5ms hot path; titles and bodies also diverge, complicating later corroboration. ~2-3h.
+- **P5 · `consolidate_observations` reuses `coerce_observation(args)` on the whole tool-args hash** — `management_handlers.rb:151` (Grimm border). Couples the consolidation tool's param names to the observation schema and pulls in `kind`/`priority` defaults the caller may not intend. Pass a narrowed hash. ~1h.
+- **P6 · dashboard N+1 across stores** — `dashboard/observations.rb:48-99` (Evans, bounded). 8-12 small aggregate queries per load; acceptable at store-count 2 but the `.where(status: "active")` predicate repeats ~6×. One `group_and_count(:status)` per store. ~2h.
+
+### Low (fast-follow cleanups)
+
+- **L1** `persist_observations` reaches into raw hashes (`obs[:body]`…) — coerce through `Domain::Observation` at the border; defaults are triplicated. `resolver.rb:81`. ~1-2h.
+- **L2** `respond_to?(:observations)` guard is dead defensiveness — `Extraction` always defines it. `resolver.rb:82`. ~5m.
+- **L3/P8** status strings (`"active"`/`"consolidated"`/`"expired"`) and `2`/`3` literals scattered — add `STATUSES`/reuse `PROMOTION_THRESHOLD`/`INFO` from `Domain::Observation`. ~30m.
+- **L4** `increment_corroboration` returns void while sibling mutators return `updated > 0` — make symmetric. `sqlite_store.rb:772`. ~10m.
+- **L5** migration index DDL uses raw `CREATE INDEX` rather than Sequel's `index` DSL — idiomatic-only. ~30m, optional.
+- **L6** `consolidate_observations` doesn't thread `session_id` (synthesized rows get NULL) — document the intent or thread it. ~5m.
+
+### What's done well
+
+Append-only/tombstone discipline honored end-to-end with "row preserved, not deleted" specs; `Domain::Observation` immutable/frozen/self-validating with intention-revealing predicates; `corroborated?(threshold)` kept a total function (threshold injected, not hard-coded); resolver change genuinely additive (observations persist *inside* the extraction transaction, "no observations → fact behavior unchanged" tested); pure Sequel datasets throughout (no raw SQL except index DDL); promotion-gate tests pin the anti-hallucination invariant; `coerce_observation` border validation with `filter_map` drops invalids without aborting the batch; the Go-language case-sensitivity fix is clean and well-specced.
+
+### Recommended pre-merge action
+
+Fix the mechanical items — **H1** (read-inside-transaction), **P2** (defuse cross-scope), **M1** (the missing consolidation spec), **M4/M5** (document this PR's own new `apply` surface). Flag **P1** (regex retune) for a data-driven decision — do not change blind. Everything else is tracked here as fast-follow.
+
+---
+
 ## Post-0.11 Investigation: Hallucination Rate Metric Calibration (2026-04-30)
 
 When #48 (hallucination-rate metric) was first run against this project's real DB, it surfaced numbers that *looked* alarming: