claude_memory 0.2.0 → 0.3.0

data/docs/remaining_improvements.md

@@ -0,0 +1,330 @@
+# Remaining Improvements from Analysis
+
+This document contains the improvements that have NOT yet been implemented from the episodic-memory and claude-mem analysis.
+
+**Note:** The "index" command to generate embeddings for existing facts has been completed (2026-01-23).
+
+---
+
+## 1. Background Processing for Hooks
+
+### What Episodic-Memory Does
+
+**Background Sync**:
+```bash
+# SessionStart hook
+episodic-memory sync --background
+```
+
+Runs in background, user continues working immediately.
+
+**File**: `src/sync.ts`
+
+### What We Should Do
+
+**Priority**: MEDIUM
+
+**Implementation**:
+
+1. **Background processing flag**:
+   ```ruby
+   # lib/claude_memory/commands/hook_command.rb
+   def call(args)
+     opts = parse_options(args, { async: false })
+
+     if opts[:async]
+       # Fork and detach
+       pid = fork do
+         Process.setsid  # Detach from terminal
+         execute_hook(subcommand, payload)
+       end
+       Process.detach(pid)
+
+       stdout.puts "Hook execution started in background (PID: #{pid})"
+       return Hook::ExitCodes::SUCCESS
+     end
+
+     execute_hook(subcommand, payload)
+   end
+   ```
+
+2. **Hook configuration**:
+   ```json
+   {
+     "hooks": {
+       "SessionStart": [{
+         "matcher": "startup|resume",
+         "hooks": [{
+           "type": "command",
+           "command": "claude-memory hook ingest --async",
+           "async": true
+         }]
+       }]
+     }
+   }
+   ```
+
+**Benefits**:
+- Non-blocking hooks (user continues working immediately)
+- Better user experience for long-running operations
+- Doesn't delay session startup
+
+**Trade-offs**:
+- Background process management complexity
+- Need logging for background execution
+- Potential race conditions if multiple sessions start simultaneously
+
+---
+
+## 2. ROI Metrics and Token Economics
+
+### What claude-mem Does
+
+**Discovery Token Tracking**:
+- `discovery_tokens` field on observations table
+- Tracks tokens spent discovering each piece of knowledge
+- Cumulative metrics in session summaries
+- Footer displays ROI: "Access 10k tokens for 2,500t"
+
+**File**: `src/services/sqlite/Database.ts`
+
+```typescript
+observations: {
+  id: INTEGER PRIMARY KEY,
+  title: TEXT,
+  narrative: TEXT,
+  discovery_tokens: INTEGER,  // ← Cost tracking
+  created_at_epoch: INTEGER
+}
+
+session_summaries: {
+  cumulative_discovery_tokens: INTEGER,  // ← Running total
+  observation_count: INTEGER
+}
+```
+
+**Context Footer Example**:
+```markdown
+💡 **Token Economics:**
+- Context shown: 2,500 tokens
+- Research captured: 10,000 tokens
+- ROI: 4x compression
+```
+
+### What We Should Do
+
+**Priority**: MEDIUM
+
+**Implementation**:
+
+1. **Add metrics table**:
+   ```ruby
+   create_table :ingestion_metrics do
+     primary_key :id
+     foreign_key :content_item_id, :content_items
+     Integer :input_tokens
+     Integer :output_tokens
+     Integer :facts_extracted
+     DateTime :created_at
+   end
+   ```
+
+2. **Track during distillation**:
+   ```ruby
+   # lib/claude_memory/distill/distiller.rb
+   def distill(content)
+     response = api_call(content)
+     facts = extract_facts(response)
+
+     store_metrics(
+       input_tokens: response.usage.input_tokens,
+       output_tokens: response.usage.output_tokens,
+       facts_extracted: facts.size
+     )
+
+     facts
+   end
+   ```
+
+3. **Display in CLI**:
+   ```ruby
+   # claude-memory stats
+   def stats_cmd
+     metrics = store.aggregate_metrics
+     puts "Token Economics:"
+     puts "  Input: #{metrics[:input_tokens]} tokens"
+     puts "  Output: #{metrics[:output_tokens]} tokens"
+     puts "  Facts: #{metrics[:facts_extracted]}"
+     puts "  Efficiency: #{metrics[:facts_extracted] / metrics[:input_tokens].to_f} facts/token"
+   end
+   ```
+
+4. **Add to published snapshot**:
+   ```markdown
+   <!-- At bottom of .claude/rules/claude_memory.generated.md -->
+
+   ---
+
+   *Memory stats: 145 facts from 12,500 ingested tokens (86 facts/1k tokens)*
+   ```
+
+**Benefits**:
+- Visibility into memory system efficiency
+- Justifies API costs (shows compression ratio)
+- Helps tune distillation prompts for better extraction
+
+**Trade-offs**:
+- Requires API usage tracking
+- Adds database complexity
+- May not be meaningful for all distiller implementations
+
+---
+
+## 3. Structured Logging
+
+### Implementation
+
+**Priority**: LOW
+
+1. **Add structured logger**:
+   ```ruby
+   # lib/claude_memory/logging/logger.rb
+   module ClaudeMemory
+     module Logging
+       class Logger
+         def initialize(output = $stderr, level: :info)
+           @output = output
+           @level = level
+         end
+
+         def info(message, metadata = {})
+           log(:info, message, metadata)
+         end
+
+         def error(message, exception: nil, metadata = {})
+           log(:error, message, metadata.merge(exception: exception&.message))
+         end
+
+         private
+
+         def log(level, message, metadata)
+           return if should_skip?(level)
+
+           log_entry = {
+             timestamp: Time.now.iso8601,
+             level: level,
+             message: message
+           }.merge(metadata)
+
+           @output.puts JSON.generate(log_entry)
+         end
+       end
+     end
+   end
+   ```
+
+2. **Usage in Ingester**:
+   ```ruby
+   def ingest(...)
+     logger.info("Starting ingestion",
+       session_id: session_id,
+       transcript_path: transcript_path
+     )
+
+     begin
+       # ... ingestion logic ...
+
+       logger.info("Ingestion complete",
+         content_items_created: 1,
+         facts_extracted: facts.size
+       )
+     rescue => e
+       logger.error("Ingestion failed",
+         exception: e,
+         session_id: session_id
+       )
+       raise
+     end
+   end
+   ```
+
+**Benefits**:
+- Better debugging with structured data
+- Easy log parsing and analysis
+- Consistent log format
+
+**Trade-offs**:
+- Additional complexity
+- Log output may be verbose
+
+---
+
+
+## Features to Avoid
+
+### 1. Chroma Vector Database
+
+**Their Approach**: Hybrid SQLite FTS5 + Chroma vector search.
+
+**Our Take**: **Skip it.** Adds significant complexity:
+
+- Python dependency
+- ChromaDB server
+- Embedding generation
+- Sync overhead
+
+**Alternative**: We've implemented lightweight TF-IDF embeddings without external dependencies.
+
+### 2. Claude Agent SDK for Distillation
+
+**Their Approach**: Use `@anthropic-ai/claude-agent-sdk` for observation compression.
+
+**Our Take**: **Skip it.** We already have `Distill::Distiller` interface. SDK adds:
+
+- Node.js dependency
+- Subprocess management
+- Complex event loop
+
+**Alternative**: Direct API calls via `anthropic-rb` gem (if we implement distiller).
+
+### 3. Worker Service Background Process
+
+**Their Approach**: Long-running worker with HTTP API + MCP wrapper.
+
+**Our Take**: **Skip it.** We use MCP server directly:
+
+- No background process to manage
+- No port conflicts
+- No PID files
+- Simpler deployment
+
+**Alternative**: Keep stdio-based MCP server. Add HTTP transport only if needed.
+
+### 4. Web Viewer UI
+
+**Their Approach**: React-based web UI at `http://localhost:37777`.
+
+**Our Take**: **Skip for MVP.** Significant effort for uncertain value:
+
+- React + esbuild
+- SSE implementation
+- State management
+- CSS/theming
+
+**Alternative**: CLI output is sufficient. Add web UI if users request it.
+
+---
+
+## Implementation Priorities
+
+### Medium Priority
+
+1. **Background Processing** - Non-blocking hooks for better UX
+2. **ROI Metrics** - Track token economics for distillation
+
+### Low Priority
+
+3. **Structured Logging** - Better debugging with JSON logs
+4. **Health Monitoring** - Only if we add background worker
+5. **Web Viewer UI** - Only if users request visualization
+6. **Configuration-Driven Context** - Only if users request snapshot customization

data/lefthook.yml

@@ -8,3 +8,16 @@ pre-commit:
       stage_fixed: true
     tests:
       run: bundle exec rspec
+    quality-review:
+      run: |
+        staged_ruby=$(git diff --cached --name-only --diff-filter=ACM | grep '\.rb$' || true)
+        if [ -n "$staged_ruby" ]; then
+          echo "Running quality review on staged changes..."
+          claude -p "/review-commit" \
+            --no-session-persistence \
+            --model haiku \
+            --allowedTools "Bash Read Grep Glob" \
+            --no-chrome
+        else
+          echo "No Ruby files staged, skipping quality review"
+        fi

data/lib/claude_memory/commands/checks/claude_md_check.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+module ClaudeMemory
+  module Commands
+    module Checks
+      # Checks if CLAUDE.md exists and imports snapshot
+      class ClaudeMdCheck
+        CLAUDE_MD_PATH = ".claude/CLAUDE.md"
+
+        def call
+          unless File.exist?(CLAUDE_MD_PATH)
+            return {
+              status: :warning,
+              label: "claude_md",
+              message: "No .claude/CLAUDE.md found",
+              details: {path: CLAUDE_MD_PATH}
+            }
+          end
+
+          content = File.read(CLAUDE_MD_PATH)
+
+          if content.include?("claude_memory.generated.md")
+            {
+              status: :ok,
+              label: "claude_md",
+              message: "CLAUDE.md imports snapshot",
+              details: {path: CLAUDE_MD_PATH}
+            }
+          else
+            {
+              status: :warning,
+              label: "claude_md",
+              message: "CLAUDE.md does not import snapshot",
+              details: {path: CLAUDE_MD_PATH}
+            }
+          end
+        end
+      end
+    end
+  end
+end

data/lib/claude_memory/commands/checks/database_check.rb

@@ -0,0 +1,120 @@
+# frozen_string_literal: true
+
+module ClaudeMemory
+  module Commands
+    module Checks
+      # Checks database existence, schema, and health
+      class DatabaseCheck
+        def initialize(db_path, label)
+          @db_path = db_path
+          @label = label
+        end
+
+        def call
+          unless File.exist?(@db_path)
+            return missing_database_result
+          end
+
+          check_database_health
+        rescue => e
+          {
+            status: :error,
+            label: @label,
+            message: "#{@label} database error: #{e.message}",
+            details: {}
+          }
+        end
+
+        private
+
+        def missing_database_result
+          if @label == "global"
+            {
+              status: :error,
+              label: @label,
+              message: "Global database not found: #{@db_path}",
+              details: {}
+            }
+          else
+            {
+              status: :warning,
+              label: @label,
+              message: "Project database not found: #{@db_path} (run 'claude-memory init')",
+              details: {}
+            }
+          end
+        end
+
+        def check_database_health
+          store = ClaudeMemory::Store::SQLiteStore.new(@db_path)
+
+          details = {
+            path: @db_path,
+            adapter: "extralite",
+            schema_version: store.schema_version,
+            fact_count: store.facts.count,
+            content_count: store.content_items.count,
+            conflict_count: store.conflicts.where(status: "open").count,
+            last_ingest: store.content_items.max(:ingested_at)
+          }
+
+          warnings = []
+          errors = []
+
+          # Check for open conflicts
+          if details[:conflict_count] > 0
+            warnings << "#{details[:conflict_count]} open conflict(s) need resolution"
+          end
+
+          # Check for missing ingests
+          if details[:last_ingest].nil? && @label == "project"
+            warnings << "No content has been ingested yet"
+          end
+
+          # Check for stuck operations
+          tracker = ClaudeMemory::Infrastructure::OperationTracker.new(store)
+          stuck_ops = tracker.stuck_operations
+          if stuck_ops.any?
+            stuck_ops.each do |op|
+              warnings << "Stuck operation '#{op[:operation_type]}' (started #{op[:started_at]}). Run 'claude-memory recover' to reset."
+            end
+          end
+          details[:stuck_operations] = stuck_ops.size
+
+          # Run schema validation
+          validator = ClaudeMemory::Infrastructure::SchemaValidator.new(store)
+          validation = validator.validate
+          details[:schema_valid] = validation[:valid]
+
+          # Collect validation issues
+          if validation[:issues].any?
+            validation[:issues].each do |issue|
+              if issue[:severity] == "error"
+                errors << issue[:message]
+              else
+                warnings << issue[:message]
+              end
+            end
+          end
+
+          store.close
+
+          status = if errors.any?
+            :error
+          else
+            (warnings.any? ? :warning : :ok)
+          end
+
+          {
+            status: status,
+            label: @label,
+            message: "#{@label.capitalize} database exists: #{@db_path}",
+            details: details,
+            warnings: warnings,
+            errors: errors
+          }
+        end
+      end
+    end
+  end
+end

data/lib/claude_memory/commands/checks/hooks_check.rb

@@ -0,0 +1,112 @@
+# frozen_string_literal: true
+
+require "json"
+
+module ClaudeMemory
+  module Commands
+    module Checks
+      # Checks hooks configuration in settings files
+      class HooksCheck
+        SETTINGS_PATHS = [".claude/settings.json", ".claude/settings.local.json"].freeze
+        EXPECTED_HOOKS = %w[Stop SessionStart PreCompact SessionEnd].freeze
+
+        def call
+          hooks_found = false
+          warnings = []
+          paths_checked = []
+
+          SETTINGS_PATHS.each do |path|
+            next unless File.exist?(path)
+
+            paths_checked << path
+            result = check_settings_file(path)
+
+            if result[:has_hooks]
+              hooks_found = true
+              warnings.concat(result[:warnings])
+            end
+          end
+
+          # Check for orphaned hooks
+          if hooks_found && !mcp_configured?
+            warnings << "Orphaned hooks detected: claude-memory hooks are configured but MCP server is not"
+            warnings << "Run 'claude-memory uninstall' to remove hooks, or 'claude-memory init' to reconfigure"
+          end
+
+          if hooks_found
+            {
+              status: warnings.any? ? :warning : :ok,
+              label: "hooks",
+              message: "Hooks configured in #{paths_checked.join(", ")}",
+              details: {
+                paths: paths_checked,
+                fallback_available: false
+              },
+              warnings: warnings
+            }
+          else
+            {
+              status: :warning,
+              label: "hooks",
+              message: "No hooks configured. Run 'claude-memory init' or configure manually.",
+              details: {
+                paths: SETTINGS_PATHS,
+                fallback_available: true,
+                fallback_commands: [
+                  "claude-memory ingest --session-id <id> --transcript-path <path>",
+                  "claude-memory sweep --budget 5",
+                  "claude-memory publish"
+                ]
+              },
+              warnings: []
+            }
+          end
+        end
+
+        private
+
+        def check_settings_file(path)
+          warnings = []
+
+          begin
+            config = JSON.parse(File.read(path))
+
+            unless config["hooks"]&.any?
+              return {has_hooks: false, warnings: []}
+            end
+
+            # Check if any hooks contain claude-memory commands
+            claude_memory_hooks = config["hooks"].values.flatten.any? do |hook_array|
+              next false unless hook_array.is_a?(Hash) && hook_array["hooks"].is_a?(Array)
+              hook_array["hooks"].any? { |h| h["command"]&.include?("claude-memory") }
+            end
+
+            return {has_hooks: false, warnings: []} unless claude_memory_hooks
+
+            # Check for missing recommended hooks
+            missing = EXPECTED_HOOKS - config["hooks"].keys
+            if missing.any?
+              warnings << "Missing recommended hooks in #{path}: #{missing.join(", ")}"
+            end
+
+            {has_hooks: true, warnings: warnings}
+          rescue JSON::ParserError
+            {has_hooks: false, warnings: ["Invalid JSON in #{path}"]}
+          end
+        end
+
+        def mcp_configured?
+          mcp_path = ".claude.json"
+          return false unless File.exist?(mcp_path)
+
+          begin
+            config = JSON.parse(File.read(mcp_path))
+            config["mcpServers"]&.key?("claude-memory")
+          rescue JSON::ParserError
+            false
+          end
+        end
+      end
+    end
+  end
+end