claude_memory 0.12.1 → 0.13.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (57) hide show
  1. checksums.yaml +4 -4
  2. data/.claude/memory.sqlite3 +0 -0
  3. data/.claude/rules/claude_memory.generated.md +6 -1
  4. data/.claude/settings.local.json +2 -1
  5. data/.claude-plugin/marketplace.json +2 -2
  6. data/.claude-plugin/plugin.json +2 -2
  7. data/CHANGELOG.md +28 -0
  8. data/CLAUDE.md +11 -6
  9. data/README.md +35 -0
  10. data/db/migrations/019_add_observations.rb +43 -0
  11. data/db/migrations/020_add_observation_promotion.rb +33 -0
  12. data/docs/GETTING_STARTED.md +38 -0
  13. data/docs/api_stability.md +16 -5
  14. data/docs/architecture.md +18 -6
  15. data/docs/audit_runbook.md +67 -0
  16. data/docs/dashboard.md +28 -0
  17. data/docs/improvements.md +94 -1
  18. data/docs/influence/mastra-observational-memory.md +198 -0
  19. data/docs/influence/strands-agent-sops.md +163 -0
  20. data/docs/quality_review.md +45 -0
  21. data/lib/claude_memory/audit/checks.rb +149 -0
  22. data/lib/claude_memory/audit/runner.rb +4 -0
  23. data/lib/claude_memory/commands/census_command.rb +1 -1
  24. data/lib/claude_memory/commands/hook_command.rb +16 -3
  25. data/lib/claude_memory/commands/initializers/hooks_configurator.rb +3 -1
  26. data/lib/claude_memory/commands/install_skill_command.rb +4 -0
  27. data/lib/claude_memory/commands/observations_command.rb +367 -0
  28. data/lib/claude_memory/commands/registry.rb +1 -0
  29. data/lib/claude_memory/commands/skills/reflect.md +68 -0
  30. data/lib/claude_memory/commands/stats_command.rb +60 -1
  31. data/lib/claude_memory/dashboard/api.rb +4 -0
  32. data/lib/claude_memory/dashboard/index.html +154 -2
  33. data/lib/claude_memory/dashboard/observations.rb +115 -0
  34. data/lib/claude_memory/dashboard/server.rb +1 -0
  35. data/lib/claude_memory/distill/extraction.rb +6 -4
  36. data/lib/claude_memory/distill/null_distiller.rb +86 -3
  37. data/lib/claude_memory/distill/reference_material_detector.rb +4 -1
  38. data/lib/claude_memory/domain/observation.rb +118 -0
  39. data/lib/claude_memory/embeddings/generator.rb +1 -1
  40. data/lib/claude_memory/hook/context_injector.rb +100 -2
  41. data/lib/claude_memory/mcp/handlers/management_handlers.rb +113 -2
  42. data/lib/claude_memory/mcp/handlers/query_handlers.rb +48 -1
  43. data/lib/claude_memory/mcp/instructions_builder.rb +1 -0
  44. data/lib/claude_memory/mcp/query_guide.rb +28 -0
  45. data/lib/claude_memory/mcp/tool_definitions.rb +58 -0
  46. data/lib/claude_memory/mcp/tools.rb +3 -0
  47. data/lib/claude_memory/observe/observations_renderer.rb +49 -0
  48. data/lib/claude_memory/observe/reflector.rb +91 -0
  49. data/lib/claude_memory/publish.rb +53 -1
  50. data/lib/claude_memory/resolve/resolver.rb +45 -8
  51. data/lib/claude_memory/store/schema_manager.rb +1 -1
  52. data/lib/claude_memory/store/sqlite_store.rb +181 -0
  53. data/lib/claude_memory/sweep/maintenance.rb +15 -1
  54. data/lib/claude_memory/sweep/sweeper.rb +7 -1
  55. data/lib/claude_memory/version.rb +1 -1
  56. data/lib/claude_memory.rb +5 -0
  57. metadata +11 -1
@@ -0,0 +1,68 @@
1
+ # Reflect
2
+
3
+ Consolidate the episodic observation log and promote corroborated observations into
4
+ durable facts. This is the manual, on-demand counterpart to the automatic Reflector
5
+ that runs during sweep — use it for a deeper pass.
6
+
7
+ ## Usage
8
+
9
+ ```
10
+ /reflect
11
+ /reflect --scope project
12
+ ```
13
+
14
+ ## Instructions
15
+
16
+ You are the Reflector for ClaudeMemory's episodic observation layer. Observations are
17
+ the "what happened" log; facts are the "what is true" store. Your job is to look across
18
+ the recent observations, find what has become a stable truth, and promote it — while
19
+ leaving one-off noise alone.
20
+
21
+ Work in three passes:
22
+
23
+ ### 1. Survey
24
+
25
+ Call `memory.observations` (use `important_only: true` first for the 🔴 entries, then a
26
+ broader pass). Read the log as a narrative of what has happened in this project.
27
+
28
+ ### 2. Consolidate related observations
29
+
30
+ Where several observations describe the **same thing in different words** (regex dedup only
31
+ catches exact matches), merge them with `memory.consolidate_observations(from_ids: […],
32
+ body: "<synthesis>")`. The synthesized observation inherits the **combined** corroboration of
33
+ its sources — which often tips it over the promotion threshold — and the originals are
34
+ tombstoned (preserved and linked, not deleted). Use the `#id` shown for each observation.
35
+
36
+ ### 3. Promote corroborated observations → facts
37
+
38
+ The promotion bridge is gated: an observation must have been **corroborated** (sighted
39
+ repeatedly — `corroboration_count` ≥ the threshold) before it can become a fact. This is
40
+ deliberate: requiring repeated sightings before commitment is an anti-hallucination gate
41
+ against one-off doc/example text.
42
+
43
+ For each observation that represents a **stable, repeated truth**:
44
+
45
+ - Call `memory.promote_observation` with `observation_id`, a `predicate`
46
+ (`decision` / `convention` / `architecture`), and an `object` that **embeds a reason**
47
+ ("… because …", "… so that …", "to avoid …"). A bare conclusion is dead weight.
48
+ - The tool refuses observations that are not yet corroborated — do not try to force them.
49
+ If something genuinely matters but has only been seen once, leave it; it will become
50
+ eligible once it recurs.
51
+
52
+ Skip observations that are:
53
+
54
+ - transient (debugging steps, one-off events),
55
+ - already captured as facts (check `memory.recall` / `memory.decisions` first),
56
+ - example/illustrative text rather than a claim about *this* project.
57
+
58
+ ### 4. Report
59
+
60
+ Summarize what you promoted (observation → fact) and what you intentionally left as
61
+ observations and why. Do not delete or rewrite observations — the deterministic Reflector
62
+ handles dedup/expiry during sweep; your job is the semantic judgment the regex pass can't make.
63
+
64
+ ## Notes
65
+
66
+ - Promotion preserves provenance: the new fact links back to the observation's source.
67
+ - Promoted observations are marked so they are not re-suggested.
68
+ - No extra API cost — this runs inside the existing Claude Code session.
@@ -13,7 +13,7 @@ module ClaudeMemory
13
13
  SCOPE_PROJECT = "project"
14
14
 
15
15
  def call(args)
16
- opts = parse_options(args, {scope: SCOPE_ALL, tools: false, tokens: false, stale: false, since_days: nil, stale_days: nil}) do |o|
16
+ opts = parse_options(args, {scope: SCOPE_ALL, tools: false, tokens: false, stale: false, observations: false, since_days: nil, stale_days: nil}) do |o|
17
17
  OptionParser.new do |parser|
18
18
  parser.banner = "Usage: claude-memory stats [options]"
19
19
  parser.on("--scope SCOPE", ["all", "global", "project"],
@@ -21,6 +21,7 @@ module ClaudeMemory
21
21
  parser.on("--tools", "Show MCP tool-call usage stats") { o[:tools] = true }
22
22
  parser.on("--tokens", "Show SessionStart context-injection token budget") { o[:tokens] = true }
23
23
  parser.on("--stale", "Show facts not recalled in CLAUDE_MEMORY_STALE_DAYS (default 14)") { o[:stale] = true }
24
+ parser.on("--observations", "Show episodic observation counts (status, kind, promotable)") { o[:observations] = true }
24
25
  parser.on("--since DAYS", Integer, "Limit --tools/--tokens to last N days") { |v| o[:since_days] = v }
25
26
  parser.on("--stale-days N", Integer, "Override staleness threshold for --stale") { |v| o[:stale_days] = v }
26
27
  end
@@ -31,6 +32,10 @@ module ClaudeMemory
31
32
  return print_mcp_tool_call_stats(opts[:since_days])
32
33
  end
33
34
 
35
+ if opts[:observations]
36
+ return print_observation_stats
37
+ end
38
+
34
39
  if opts[:tokens]
35
40
  return print_token_budget_stats(opts[:since_days])
36
41
  end
@@ -90,6 +95,60 @@ module ClaudeMemory
90
95
  0
91
96
  end
92
97
 
98
+ def print_observation_stats
99
+ manager = ClaudeMemory::Store::StoreManager.new
100
+ stores = %w[project global]
101
+ .filter_map { |scope| manager.store_if_exists(scope) }
102
+ .select { |store| store.db.table_exists?(:observations) }
103
+
104
+ stdout.puts "Observation Statistics (episodic layer)"
105
+ stdout.puts "=" * 50
106
+
107
+ threshold = ClaudeMemory::Domain::Observation::PROMOTION_THRESHOLD
108
+
109
+ total = stores.sum { |s| s.observations.count }
110
+ if total.zero?
111
+ stdout.puts "No observations recorded yet."
112
+ manager.close
113
+ return 0
114
+ end
115
+
116
+ active = stores.sum { |s| s.observations.where(status: "active").count }
117
+ consolidated = stores.sum { |s| s.observations.where(status: "consolidated").count }
118
+ expired = stores.sum { |s| s.observations.where(status: "expired").count }
119
+ promoted = stores.sum { |s| s.observations.exclude(promoted_at: nil).count }
120
+ promotable = stores.sum do |s|
121
+ s.observations.where(status: "active", promoted_at: nil)
122
+ .where { corroboration_count >= threshold }.count
123
+ end
124
+
125
+ stdout.puts "Active: #{active}"
126
+ stdout.puts "Consolidated: #{consolidated}"
127
+ stdout.puts "Expired: #{expired}"
128
+ stdout.puts "Promoted: #{promoted}"
129
+ stdout.puts "Promotable (>= #{threshold} sightings): #{promotable}"
130
+ stdout.puts
131
+
132
+ kinds = Hash.new(0)
133
+ stores.each do |store|
134
+ store.observations.where(status: "active").group_and_count(:kind).each do |row|
135
+ kinds[row[:kind]] += row[:count]
136
+ end
137
+ end
138
+
139
+ stdout.puts "By kind (active):"
140
+ if kinds.empty?
141
+ stdout.puts " (none)"
142
+ else
143
+ kinds.sort_by { |_k, v| -v }.each do |kind, count|
144
+ stdout.puts " #{count.to_s.rjust(4)} - #{kind}"
145
+ end
146
+ end
147
+
148
+ manager.close
149
+ 0
150
+ end
151
+
93
152
  def open_readonly(db_path)
94
153
  Sequel.connect("extralite://#{db_path}")
95
154
  end
@@ -456,6 +456,10 @@ module ClaudeMemory
456
456
  Efficacy::Reporter.report(events, timeframe: {since: since, session_id: session_id})
457
457
  end
458
458
 
459
+ def observations
460
+ Observations.new(@manager).report
461
+ end
462
+
459
463
  def timeline
460
464
  Timeline.new(@manager).days
461
465
  end
@@ -163,6 +163,39 @@
163
163
  .delta.flat { color: var(--text-dim); }
164
164
  .moments-sub { font-size: 12px; color: var(--text-dim); margin-top: 6px; }
165
165
 
166
+ /* Observations sidebar panel */
167
+ .obs-headline {
168
+ display: flex;
169
+ gap: 14px;
170
+ align-items: baseline;
171
+ margin-bottom: 6px;
172
+ }
173
+ .obs-headline .n {
174
+ font-size: 32px;
175
+ font-weight: 600;
176
+ letter-spacing: -0.02em;
177
+ line-height: 1;
178
+ }
179
+ .obs-headline .obs-stat { display: flex; flex-direction: column; gap: 2px; }
180
+ .obs-headline .obs-stat-n { font-size: 18px; font-weight: 600; color: var(--text); line-height: 1; }
181
+ .obs-headline .obs-stat-label { font-size: 10px; color: var(--text-faint); text-transform: uppercase; letter-spacing: 0.06em; }
182
+ .obs-sub { font-size: 12px; color: var(--text-dim); margin: 8px 0; }
183
+ .obs-breakdowns { display: flex; flex-direction: column; gap: 8px; margin-top: 10px; }
184
+ .obs-breakdown-label { font-size: 10px; text-transform: uppercase; letter-spacing: 0.06em; color: var(--text-faint); margin-bottom: 4px; }
185
+ .obs-badges { display: flex; flex-wrap: wrap; gap: 6px; }
186
+ .obs-badge {
187
+ display: inline-flex;
188
+ align-items: center;
189
+ gap: 4px;
190
+ padding: 2px 8px;
191
+ border-radius: 999px;
192
+ border: 1px solid var(--border);
193
+ background: var(--surface2);
194
+ font-size: 11px;
195
+ color: var(--text-dim);
196
+ }
197
+ .obs-badge .obs-badge-n { font-weight: 600; color: var(--text); font-variant-numeric: tabular-nums; }
198
+
166
199
  /* Knowledge-base sidebar panel */
167
200
  .kb-totals {
168
201
  display: flex;
@@ -1157,6 +1190,13 @@
1157
1190
  <div class="moments-sub" id="moments-sub"></div>
1158
1191
  </div>
1159
1192
 
1193
+ <div class="panel" id="panel-observations">
1194
+ <div class="panel-label">Episodic observations
1195
+ <span class="panel-hint" title="The 'what happened' log that complements facts ('what is true'). Observations accrue as the Observer runs, and become facts once corroborated ≥2×.">ⓘ</span>
1196
+ </div>
1197
+ <div id="obs-panel-body"></div>
1198
+ </div>
1199
+
1160
1200
  <div class="panel" id="panel-review">
1161
1201
  <div class="panel-label">Needs review</div>
1162
1202
  <div class="review-rows" id="review-rows"></div>
@@ -1234,6 +1274,7 @@
1234
1274
  </div>
1235
1275
  <div class="drawer-tabs">
1236
1276
  <div class="drawer-tab active" data-adv="knowledge">Knowledge</div>
1277
+ <div class="drawer-tab" data-adv="observations">Observations</div>
1237
1278
  <div class="drawer-tab" data-adv="overview">Overview</div>
1238
1279
  <div class="drawer-tab" data-adv="facts">Facts</div>
1239
1280
  <div class="drawer-tab" data-adv="explore">Explore</div>
@@ -1253,6 +1294,13 @@
1253
1294
  <div class="knowledge-body" id="knowledge-body"></div>
1254
1295
  </div>
1255
1296
 
1297
+ <div class="drawer-panel" id="adv-observations">
1298
+ <div id="obs-summary" class="adv-card"></div>
1299
+ <div class="adv-card" style="padding: 0;">
1300
+ <div id="obs-recent"></div>
1301
+ </div>
1302
+ </div>
1303
+
1256
1304
  <div class="drawer-panel" id="adv-overview">
1257
1305
  <div id="overview-rows"></div>
1258
1306
  <div class="adv-card">
@@ -1444,7 +1492,77 @@ async function api(path, params = {}) {
1444
1492
 
1445
1493
  // ==================== Load cycle ====================
1446
1494
  async function loadAll() {
1447
- await Promise.all([loadHealth(), loadTrust(), loadMoments(), loadKnowledgePanel(), loadReusePanel()]);
1495
+ await Promise.all([loadHealth(), loadTrust(), loadMoments(), loadKnowledgePanel(), loadReusePanel(), loadObservationsPanel()]);
1496
+ }
1497
+
1498
+ // Sidebar "Episodic observations" panel — headline numbers + breakdowns.
1499
+ // The richer recent timeline lives in the Advanced drawer (loadObservations).
1500
+ async function loadObservationsPanel() {
1501
+ const data = await api('observations');
1502
+ const t = data.totals || {}, c = data.compression || {}, corr = data.corroboration || {};
1503
+ const body = document.getElementById('obs-panel-body');
1504
+ if (!body) return;
1505
+
1506
+ const active = t.active || 0;
1507
+ if (!active) {
1508
+ body.innerHTML = `
1509
+ <div class="feed-empty" style="padding: 24px 16px; text-align: left;">
1510
+ <h3>No observations yet</h3>
1511
+ <p>The episodic log — "what happened" — accrues as the Observer runs over your sessions. Once an observation is corroborated ≥2×, it becomes a fact.</p>
1512
+ </div>`;
1513
+ return;
1514
+ }
1515
+
1516
+ const promotable = corr.promotable || 0;
1517
+ body.innerHTML = `
1518
+ <div class="obs-headline">
1519
+ <div class="obs-stat">
1520
+ <span class="n">${active.toLocaleString()}</span>
1521
+ <span class="obs-stat-label">active</span>
1522
+ </div>
1523
+ <div class="obs-stat">
1524
+ <span class="obs-stat-n">${promotable.toLocaleString()}</span>
1525
+ <span class="obs-stat-label">ready to promote</span>
1526
+ </div>
1527
+ <div class="obs-stat">
1528
+ <span class="obs-stat-n">${c.ratio ? c.ratio + '×' : '—'}</span>
1529
+ <span class="obs-stat-label">compression</span>
1530
+ </div>
1531
+ </div>
1532
+ <div class="obs-sub">${promotable
1533
+ ? `${promotable.toLocaleString()} ready to promote — observations corroborated ≥2× become facts (max seen ×${corr.max || 0}).`
1534
+ : `None ready yet — observations corroborated ≥2× become facts (max seen ×${corr.max || 0}).`}</div>
1535
+ <div class="obs-breakdowns">
1536
+ ${renderObsBreakdown('By priority', priorityBadges(data.by_priority || {}))}
1537
+ ${renderObsBreakdown('By kind', kindBadges(data.by_kind || {}))}
1538
+ </div>`;
1539
+ }
1540
+
1541
+ const PRIORITY_MARKERS = {1: '🔴', 2: '🟡', 3: '🟢'};
1542
+ const PRIORITY_LABELS = {1: 'important', 2: 'maybe', 3: 'info'};
1543
+
1544
+ function priorityBadges(byPriority) {
1545
+ return [1, 2, 3]
1546
+ .filter(p => byPriority[p] || byPriority[String(p)])
1547
+ .map(p => {
1548
+ const n = byPriority[p] || byPriority[String(p)] || 0;
1549
+ return `<span class="obs-badge">${PRIORITY_MARKERS[p]} ${PRIORITY_LABELS[p]} <span class="obs-badge-n">${n}</span></span>`;
1550
+ }).join('');
1551
+ }
1552
+
1553
+ function kindBadges(byKind) {
1554
+ return Object.entries(byKind)
1555
+ .sort((a, b) => b[1] - a[1])
1556
+ .map(([kind, n]) => `<span class="obs-badge">${esc(kind)} <span class="obs-badge-n">${n}</span></span>`)
1557
+ .join('');
1558
+ }
1559
+
1560
+ function renderObsBreakdown(label, badgesHtml) {
1561
+ if (!badgesHtml) return '';
1562
+ return `<div>
1563
+ <div class="obs-breakdown-label">${label}</div>
1564
+ <div class="obs-badges">${badgesHtml}</div>
1565
+ </div>`;
1448
1566
  }
1449
1567
 
1450
1568
  async function loadKnowledgePanel() {
@@ -2180,7 +2298,41 @@ function switchAdvTab(name) {
2180
2298
 
2181
2299
  // ==================== Advanced drawer loaders ====================
2182
2300
  async function loadAdvanced() {
2183
- await Promise.all([loadKnowledge(), loadOverview(), loadFacts(), loadEfficacy(), loadConflicts(), loadActivityLog(), loadTelemetry()]);
2301
+ await Promise.all([loadKnowledge(), loadObservations(), loadOverview(), loadFacts(), loadEfficacy(), loadConflicts(), loadActivityLog(), loadTelemetry()]);
2302
+ }
2303
+
2304
+ async function loadObservations() {
2305
+ const data = await api('observations');
2306
+ const t = data.totals || {}, c = data.compression || {}, corr = data.corroboration || {};
2307
+ const promotable = corr.promotable || 0;
2308
+ const summary = document.getElementById('obs-summary');
2309
+ if (summary) {
2310
+ summary.innerHTML = `
2311
+ <h3>Episodic observations <span style="color: var(--text-faint); font-weight: normal;">— what happened</span></h3>
2312
+ <div style="display: flex; flex-wrap: wrap; gap: 16px; color: var(--text-dim); font-size: 13px;">
2313
+ <span><span class="big">${(t.active || 0).toLocaleString()}</span> active</span>
2314
+ <span>${t.consolidated || 0} consolidated · ${t.expired || 0} expired · ${t.promoted || 0} promoted</span>
2315
+ <span>${promotable} ready to promote (max ×${corr.max || 0}) — observations corroborated ≥2× become facts</span>
2316
+ <span>compression ${c.ratio ? c.ratio + '×' : '—'} <span style="color: var(--text-faint);">(${(c.source_tokens || 0).toLocaleString()} → ${(c.observation_tokens || 0).toLocaleString()} tok)</span></span>
2317
+ </div>
2318
+ <div class="obs-breakdowns" style="margin-top: 12px;">
2319
+ ${renderObsBreakdown('By priority', priorityBadges(data.by_priority || {}))}
2320
+ ${renderObsBreakdown('By kind', kindBadges(data.by_kind || {}))}
2321
+ </div>`;
2322
+ }
2323
+ const recentEl = document.getElementById('obs-recent');
2324
+ const recent = data.recent || [];
2325
+ if (recentEl) {
2326
+ recentEl.innerHTML = recent.length ? recent.map(o => `
2327
+ <div style="padding: 8px 12px; border-bottom: 1px solid var(--border);">
2328
+ <span style="color: var(--text-faint); font-size: 11px;">#${o.id} · ${esc(o.kind)} · p${o.priority}${o.corroboration_count > 1 ? ' · ×' + o.corroboration_count : ''} · ${esc(o.observed_ago || '')}</span>
2329
+ <div style="color: var(--text); font-size: 13px;">${PRIORITY_MARKERS[o.priority] ? PRIORITY_MARKERS[o.priority] + ' ' : ''}${esc(o.body || '')}</div>
2330
+ </div>`).join('') : `
2331
+ <div class="feed-empty" style="text-align: left;">
2332
+ <h3>No observations yet</h3>
2333
+ <p>The episodic log — "what happened" — accrues from your sessions as the Observer runs. Once an observation is corroborated ≥2×, it gets promoted into a fact.</p>
2334
+ </div>`;
2335
+ }
2184
2336
  }
2185
2337
 
2186
2338
  async function loadTelemetry() {
@@ -0,0 +1,115 @@
1
+ # frozen_string_literal: true
2
+
3
+ module ClaudeMemory
4
+ module Dashboard
5
+ # Observability for the episodic observation layer. Surfaces counts by
6
+ # status/kind/priority, corroboration + promotion readiness, a Mastra-style
7
+ # compression ratio (source content tokens ÷ observation tokens), and a
8
+ # recent timeline. Aggregated across the project and global stores.
9
+ #
10
+ # Pulled out of Dashboard::API so the queries live next to the data.
11
+ class Observations
12
+ RECENT_LIMIT = 20
13
+
14
+ def initialize(manager)
15
+ @manager = manager
16
+ end
17
+
18
+ def report
19
+ stores = observation_stores
20
+ return empty_report if stores.empty?
21
+
22
+ {
23
+ totals: totals(stores),
24
+ by_kind: by_field(stores, :kind),
25
+ by_priority: by_field(stores, :priority),
26
+ corroboration: corroboration(stores),
27
+ compression: compression(stores),
28
+ recent: recent(stores)
29
+ }
30
+ end
31
+
32
+ private
33
+
34
+ def observation_stores
35
+ [@manager.project_store, @manager.global_store].compact.select { |s| s.db.table_exists?(:observations) }
36
+ end
37
+
38
+ def empty_report
39
+ {
40
+ totals: {active: 0, consolidated: 0, expired: 0, promoted: 0},
41
+ by_kind: {}, by_priority: {},
42
+ corroboration: {max: 0, promotable: 0},
43
+ compression: {observation_tokens: 0, source_tokens: 0, ratio: nil},
44
+ recent: []
45
+ }
46
+ end
47
+
48
+ def totals(stores)
49
+ {
50
+ active: count_where(stores, status: "active"),
51
+ consolidated: count_where(stores, status: "consolidated"),
52
+ expired: count_where(stores, status: "expired"),
53
+ promoted: stores.sum { |s| s.observations.exclude(promoted_at: nil).count }
54
+ }
55
+ end
56
+
57
+ def count_where(stores, **filter)
58
+ stores.sum { |s| s.observations.where(**filter).count }
59
+ end
60
+
61
+ def by_field(stores, field)
62
+ merged = Hash.new(0)
63
+ stores.each do |store|
64
+ store.observations.where(status: "active").group_and_count(field).each do |row|
65
+ merged[row[field]] += row[:count]
66
+ end
67
+ end
68
+ merged
69
+ end
70
+
71
+ def corroboration(stores)
72
+ threshold = Domain::Observation::PROMOTION_THRESHOLD
73
+ {
74
+ max: stores.map { |s| s.observations.where(status: "active").max(:corroboration_count) || 0 }.max,
75
+ promotable: stores.sum { |s|
76
+ s.observations.where(status: "active", promoted_at: nil).where { corroboration_count >= threshold }.count
77
+ }
78
+ }
79
+ end
80
+
81
+ # Source content tokens vs the tokens the observations distilled them into.
82
+ # ratio > 1 means the episodic log is a compression of its source.
83
+ def compression(stores)
84
+ obs_tokens = stores.sum { |s| s.observations.where(status: "active").sum(:token_count) || 0 }
85
+ source_tokens = stores.sum { |s| source_tokens_for(s) }
86
+ ratio = obs_tokens.zero? ? nil : (source_tokens.to_f / obs_tokens).round(1)
87
+ {observation_tokens: obs_tokens, source_tokens: source_tokens, ratio: ratio}
88
+ end
89
+
90
+ def source_tokens_for(store)
91
+ ids = store.observations
92
+ .where(status: "active").exclude(source_content_item_id: nil)
93
+ .distinct.select(:source_content_item_id)
94
+ .map { |r| r[:source_content_item_id] }
95
+ return 0 if ids.empty?
96
+
97
+ bytes = store.content_items.where(id: ids).sum(:byte_len) || 0
98
+ (bytes / 4.0).round
99
+ end
100
+
101
+ def recent(stores)
102
+ stores
103
+ .flat_map { |s| s.recent_observations(limit: RECENT_LIMIT) }
104
+ .sort_by { |o| o[:observed_at].to_s }.reverse.first(RECENT_LIMIT)
105
+ .map do |o|
106
+ {
107
+ id: o[:id], kind: o[:kind], priority: o[:priority],
108
+ corroboration_count: o[:corroboration_count], body: o[:body],
109
+ observed_ago: Core::RelativeTime.format(o[:observed_at])
110
+ }
111
+ end
112
+ end
113
+ end
114
+ end
115
+ end
@@ -62,6 +62,7 @@ module ClaudeMemory
62
62
  }
63
63
  }
64
64
  @server.mount_proc("/api/timeline") { |_req, res| with_fresh_connections { json_response(res, api.timeline) } }
65
+ @server.mount_proc("/api/observations") { |_req, res| with_fresh_connections { json_response(res, api.observations) } }
65
66
  @server.mount_proc("/api/recall") { |req, res| with_fresh_connections { json_response(res, api.recall(req.query)) } }
66
67
  @server.mount_proc("/api/conflicts") { |req, res| with_fresh_connections { handle_conflicts(api, req, res) } }
67
68
  @server.mount_proc("/api/moments") { |req, res| with_fresh_connections { handle_moments(api, req, res) } }
@@ -3,17 +3,18 @@
3
3
  module ClaudeMemory
4
4
  module Distill
5
5
  class Extraction
6
- attr_reader :entities, :facts, :decisions, :signals
6
+ attr_reader :entities, :facts, :decisions, :signals, :observations
7
7
 
8
- def initialize(entities: [], facts: [], decisions: [], signals: [])
8
+ def initialize(entities: [], facts: [], decisions: [], signals: [], observations: [])
9
9
  @entities = entities
10
10
  @facts = facts
11
11
  @decisions = decisions
12
12
  @signals = signals
13
+ @observations = observations
13
14
  end
14
15
 
15
16
  def empty?
16
- entities.empty? && facts.empty? && decisions.empty? && signals.empty?
17
+ entities.empty? && facts.empty? && decisions.empty? && signals.empty? && observations.empty?
17
18
  end
18
19
 
19
20
  def to_h
@@ -21,7 +22,8 @@ module ClaudeMemory
21
22
  entities: entities,
22
23
  facts: facts,
23
24
  decisions: decisions,
24
- signals: signals
25
+ signals: signals,
26
+ observations: observations
25
27
  }
26
28
  end
27
29
  end
@@ -21,7 +21,10 @@ module ClaudeMemory
21
21
  ENTITY_PATTERNS = {
22
22
  "database" => /\b(postgresql|postgres|mysql|sqlite|mongodb|redis)\b/i,
23
23
  "framework" => /\b(rails|sinatra|django|express|next\.?js|react|vue)\b/i,
24
- "language" => /\b(ruby|python|javascript|typescript|go|rust)\b/i,
24
+ # `Go` is matched case-sensitively (via the inline (?-i:) flag) so the
25
+ # English verb "go" / "go-to" doesn't masquerade as the language; the
26
+ # other languages stay case-insensitive. `golang` normalizes to `go`.
27
+ "language" => /\b(ruby|python|javascript|typescript|rust|(?-i:Go)|golang)\b/i,
25
28
  "platform" => /\b(aws|gcp|azure|heroku|vercel|netlify|docker|kubernetes)\b/i
26
29
  }.freeze
27
30
 
@@ -35,17 +38,33 @@ module ClaudeMemory
35
38
  /\buniversally\b/i
36
39
  ].freeze
37
40
 
41
+ # Observation-specific convention patterns: stricter than the shared
42
+ # CONVENTION_PATTERNS. Bare `always (.+)` / `never (.+)` / `we use (.+)`
43
+ # match code, prose, and instruction text ("never answer from memory",
44
+ # "never nil. def …"), so observations require explicit convention
45
+ # framing or a first-person "we always/never".
46
+ OBSERVATION_CONVENTION_PATTERNS = [
47
+ /\bconvention[:\s]+(.+)/i,
48
+ /\bstandard[:\s]+(.+)/i,
49
+ /\bwe\s+(?:should\s+)?(?:always|never)\s+(.+)/i
50
+ ].freeze
51
+
52
+ # Bodies that look like code / JSON / shell rather than a statement.
53
+ NOISE_BODY_SIGNATURE = /\bdef\s|\bclass\s|\bmodule\s|=>|::|","|":\s*"|[{}]|\$\(|&&|\|\|/
54
+
38
55
  def distill(text, content_item_id: nil)
39
56
  entities = extract_entities(text)
40
57
  facts = extract_facts(text, entities)
41
58
  decisions = extract_decisions(text)
42
59
  signals = extract_signals(text)
60
+ observations = extract_observations(text)
43
61
 
44
62
  Extraction.new(
45
63
  entities: entities,
46
64
  facts: facts,
47
65
  decisions: decisions,
48
- signals: signals
66
+ signals: signals,
67
+ observations: observations
49
68
  )
50
69
  end
51
70
 
@@ -55,7 +74,9 @@ module ClaudeMemory
55
74
  found = []
56
75
  ENTITY_PATTERNS.each do |type, pattern|
57
76
  text.scan(pattern).flatten.uniq.each do |name|
58
- found << {type: type, name: name.downcase, confidence: 0.7}
77
+ normalized = name.downcase
78
+ normalized = "go" if normalized == "golang"
79
+ found << {type: type, name: normalized, confidence: 0.7}
59
80
  end
60
81
  end
61
82
  found.uniq { |e| [e[:type], e[:name]] }
@@ -103,6 +124,68 @@ module ClaudeMemory
103
124
  signals
104
125
  end
105
126
 
127
+ # Layer-1 Observer: emit episodic observations for the same signals the
128
+ # distiller can detect by regex. A decision being made and a convention
129
+ # being stated are both "things that happened" worth logging in the
130
+ # episodic layer, independent of the semantic facts they also produce.
131
+ # Decisions are 🔴 (important), conventions 🟡 (maybe) — the priority is
132
+ # an internal Observer/Reflector signal. Richer observations come from
133
+ # the Layer-2 Claude-as-observer pass (a later phase).
134
+ def extract_observations(text)
135
+ observations = []
136
+ scope_hint = global_scope_signal?(text) ? "global" : "project"
137
+
138
+ DECISION_PATTERNS.each do |pattern|
139
+ text.scan(pattern).flatten.each do |match|
140
+ observations << build_observation("decision", Domain::Observation::IMPORTANT, "decided to #{match.strip}", scope_hint)
141
+ end
142
+ end
143
+
144
+ OBSERVATION_CONVENTION_PATTERNS.each do |pattern|
145
+ text.scan(pattern).flatten.each do |match|
146
+ observations << build_observation("preference", Domain::Observation::MAYBE, match.strip, scope_hint)
147
+ end
148
+ end
149
+
150
+ observations.compact.uniq { |o| [o[:kind], o[:body]] }.first(10)
151
+ end
152
+
153
+ # Returns nil for content that isn't a usable statement: code/JSON noise,
154
+ # or fewer than three words after trimming to the first sentence.
155
+ def build_observation(kind, priority, body, scope_hint)
156
+ cleaned = trim_to_statement(clean_observation_body(body))
157
+ return nil if cleaned.empty? || noise_body?(cleaned) || cleaned.split.size < 3
158
+
159
+ {kind: kind, priority: priority, body: cleaned.slice(0, 500), scope_hint: scope_hint}
160
+ end
161
+
162
+ # Cap a captured span to its first sentence (and a hard length limit) so a
163
+ # greedy `.+` match can't swallow a whole code block or JSON line.
164
+ def trim_to_statement(text)
165
+ s = text.to_s.strip
166
+ (s[/\A.{0,240}?[.!?](?=\s|\z)/m] || s[0, 240]).to_s.strip
167
+ end
168
+
169
+ def noise_body?(body)
170
+ body.match?(NOISE_BODY_SIGNATURE)
171
+ end
172
+
173
+ # The distiller scans raw transcript text, which is JSONL — so a captured
174
+ # body can carry JSON/escaping artifacts (`\n`, `\"`, a trailing `"}`,
175
+ # a leading `= `/`### ` from injected memory/markdown). Normalize them out:
176
+ # cleaner bodies read better in the injected log AND normalize more
177
+ # consistently, which is what the Reflector's dedup/corroboration keys off.
178
+ def clean_observation_body(body)
179
+ body.to_s
180
+ .gsub(/\\+[ntr]/, " ") # literal \n \t \r (even multiply-escaped) -> space
181
+ .gsub(/\\+"/, '"') # escaped quote -> quote
182
+ .gsub(/\\+/, "") # residual backslashes
183
+ .gsub(/\s+/, " ") # collapse whitespace
184
+ .sub(/\A[\s"'`,:=#*>\-\]}]+/, "") # leading JSON/markdown artifacts
185
+ .sub(/[\s"'`,:\-\]}]+\z/, "") # trailing JSON artifacts
186
+ .strip
187
+ end
188
+
106
189
  def global_scope_signal?(text)
107
190
  GLOBAL_SCOPE_PATTERNS.any? { |pattern| text.match?(pattern) }
108
191
  end
@@ -76,11 +76,14 @@ module ClaudeMemory
76
76
  end
77
77
  end
78
78
 
79
+ # Only facts are transformed; every other field must pass through
80
+ # unchanged (an earlier version silently dropped observations).
79
81
  Distill::Extraction.new(
80
82
  entities: extraction.entities,
81
83
  facts: new_facts,
82
84
  decisions: extraction.decisions,
83
- signals: extraction.signals
85
+ signals: extraction.signals,
86
+ observations: extraction.observations
84
87
  )
85
88
  end
86
89