claude-memory-hub 0.5.2 → 0.8.0

package/CHANGELOG.md

@@ -5,6 +5,212 @@ Format follows [Keep a Changelog](https://keepachangelog.com/).
 
 ---
 
+## [0.8.0] - 2026-04-02
+
+Major release: test infrastructure, architectural fixes, hook performance, data portability.
+
+### Phase 1 — Unit Tests (0% → 91 tests)
+
+- **bun:test infrastructure** — 10 test files, 91 tests, 161 assertions, 225ms runtime
+- **In-memory SQLite** — all tests use `:memory:` databases for isolation, zero filesystem side effects
+- **Test coverage modules:** schema, session-store, long-term-store, entity-extractor, observation-extractor, vector-search, working-memory, injection-validator, compact-interceptor, health-monitor
+- **Test helpers** — `createTestDb()`, `seedSession()`, `seedEntity()`, `seedSummary()`, `mockPostToolUseHook()` in `tests/setup.ts`
+
+### Phase 4 — L1 WorkingMemory Redesign
+
+- **Read-through cache** — `WorkingMemory` rewritten from dead in-process Map to read-through cache over `SessionStore`. First call loads entities from SQLite, subsequent calls serve from cache (<1ms)
+- **Previous behavior:** `workingMemory.summarize()` always returned `""` because hook scripts (short-lived) wrote to L2 directly, MCP server never populated L1
+- **New behavior:** MCP server's `memory_session_notes` tool returns real data via L1 cache
+- **API changes:** removed `add()` method (no callers), added `refresh()` and `invalidate()`. Constructor accepts `SessionStore` for DI
+- **Cache TTL:** 5 minutes, invalidated on session end
+
+### Phase 5 — Hook Performance (Batch Queue)
+
+- **Batch queue** — `src/capture/batch-queue.ts` implements write-through batching for PostToolUse. Events appended to `~/.claude-memory-hub/batch/queue.jsonl` (~3ms) instead of direct DB write (~75ms)
+- **Opportunistic flush** — each hook invocation tries to flush batch to DB if lock available
+- **File-based lock** — PID-based with 30s staleness check, prevents dead lock accumulation
+- **Fallback** — if batch dir unavailable or enqueue fails, falls back to direct write
+- **Env var:** `CLAUDE_MEMORY_HUB_BATCH=auto|enabled|disabled`
+
+### Phase 6 — Export/Import CLI
+
+- **`bunx claude-memory-hub export`** — JSONL streaming export to stdout. Options: `--since TIMESTAMP`, `--table TABLE`
+- **`bunx claude-memory-hub import`** — JSONL import from stdin with UPSERT semantics. Option: `--dry-run`
+- **`bunx claude-memory-hub cleanup`** — remove old data beyond retention period. Option: `--days N` (default 90)
+- **BLOB handling** — embedding vectors encoded as `{"$base64": true, "encoded": "..."}` for JSON portability
+- **Schema version header** — first JSONL line declares `__schema_version` for compatibility validation
+- **Idempotent import** — sessions (ON CONFLICT id), summaries (ON CONFLICT session_id), embeddings (ON CONFLICT doc_type+doc_id)
+
+### Bug Fixes
+
+- **Observation regex trailing `\b`** — patterns ending with `:` (decision:, TODO:, performance:) had trailing `\b` that failed to match because `:` followed by space = no word boundary. Removed trailing `\b` from colon-ending patterns
+
+### New/Modified Files
+
+```
+NEW:
+  tests/setup.ts                       — test infrastructure
+  tests/unit/*.test.ts                 — 10 test files (91 tests)
+  src/capture/batch-queue.ts           — write-through batch queue
+  src/export/exporter.ts               — JSONL streaming export
+  src/export/importer.ts               — JSONL streaming import
+
+MODIFIED:
+  src/memory/working-memory.ts         — read-through cache over SessionStore
+  src/hooks-entry/post-tool-use.ts     — batch queue fast path
+  src/capture/observation-extractor.ts — regex trailing \b fix
+  src/cli/main.ts                      — export, import, cleanup commands
+  package.json                         — test scripts, version 0.8.0
+```
+
+### New CLI Commands
+
+```bash
+bunx claude-memory-hub export [--since T] [--table T]  # JSONL export to stdout
+bunx claude-memory-hub import [--dry-run]               # JSONL import from stdin
+bunx claude-memory-hub cleanup [--days N]               # Remove old data
+```
+
+### New Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `CLAUDE_MEMORY_HUB_BATCH` | `auto` | Batch mode: auto, enabled, disabled |
+
+---
+
+## [0.7.0] - 2026-04-01
+
+Hardening release: honest resource analysis, search scaling, improved observation capture, DB auto-cleanup, summarizer reliability.
+
+### Correctness & Honesty
+
+- **Smart Resource Loader rewritten** — v0.4-v0.6 claimed "~24K tokens saved per session" but Claude Code has NO external API to filter resource loading (confirmed by reading Claude Code source). `formatContextAdvice()` now provides **honest, actionable advice**: shows frequently-used resources and overhead awareness instead of misleading "deferred for token efficiency" language
+- **Project CLAUDE.md name validation** — `scanClaudeMd()` now validates relative paths against `SAFE_COMMAND_NAME_RE` before registering, preventing invalid resource names in registry
+
+### Semantic Search Scaling
+
+- **Pre-filter by doc_type** — `semanticSearch()` accepts `docType` option to filter at SQL level, reducing memory usage for targeted queries
+- **Max candidates cap** — new `maxCandidates` option (default 2000) with `ORDER BY created_at DESC` prevents OOM on large datasets (>5000 embeddings)
+- **Configurable threshold** — similarity threshold now configurable via `SemanticSearchOptions` (default 0.2), was hard-coded
+- **Batch embedding reindex** — `reindexAllEmbeddings()` uses `embedBatch()` with chunk size 16 instead of processing 1-by-1. Tries batch API first, falls back to sequential
+
+### Embedding Model
+
+- **True batch processing** — `embedBatch()` processes in configurable chunks (default 8), attempts native `@huggingface/transformers` batch call first, falls back to individual if unsupported
+
+### Observation Extraction
+
+- **8 new patterns** — expanded from 6 to 14 heuristics:
+  - Tool output: DEPRECATED, SECURITY, VULNERABILITY (importance 4), "discovered", "root cause", "switched to" (3), HACK, WORKAROUND, "bottleneck", "OOM" (2)
+  - User prompts: "MUST" (4), "don't", "never", "avoid" (3), "prefer", "always use", "convention is" (2)
+- **Increased value capture** — max observation length 300 → 500 characters for richer context
+
+### Health Monitoring & Auto-Cleanup
+
+- **Embeddings size check** — new `checkEmbeddingsSize()` health check, warns when >5000 embeddings
+- **Disk check includes WAL** — total disk size now sums `memory.db` + `-wal` + `-shm` files. Tiered thresholds: 200MB warn, 500MB error
+- **`cleanupOldData()`** — transaction-safe cleanup with configurable retention (default 90 days). Deletes: sessions, entities, notes, summaries, embeddings, resource_usage, old health checks. Runs WAL checkpoint after large deletions
+
+### LLM Summarizer Reliability
+
+- **Retry logic** — `tryCliSummary()` now retries once (2 attempts total) with 1s pause between attempts before falling back to rule-based
+- **CLI availability TTL** — `isClaudeCliAvailable()` cache expires after 5 minutes when `false`, allowing recovery if `claude` CLI becomes available mid-session (was cached forever)
+
+### Modified Files
+
+```
+src/context/smart-resource-loader.ts  — honest advice, no misleading claims
+src/context/resource-registry.ts      — CLAUDE.md name validation
+src/search/semantic-search.ts         — pre-filter, maxCandidates, batch reindex
+src/search/embedding-model.ts         — true batch processing with chunking
+src/capture/observation-extractor.ts  — 8 new patterns, 500-char cap
+src/health/monitor.ts                 — embeddings check, WAL disk, auto-cleanup
+src/summarizer/cli-summarizer.ts      — retry logic, CLI TTL recovery
+```
+
+---
+
+## [0.6.0] - 2026-04-01
+
+Major release: semantic search, resource intelligence, observation capture, CLAUDE.md tracking, LLM summarization.
+
+### Phase 1 — ResourceRegistry + Entity Coverage
+
+- **ResourceRegistry** — unified scanner for ALL `.claude` locations: skills (58), agents (36), commands (65), workflows (10), CLAUDE.md. Parses agent frontmatter `name:` for correct resolution (e.g., `ios-developer` → `~/.claude/agent_mobile/ios/AGENT.md`). 3-level token estimation: listing (~50-200), full (200-8000), total (all files on disk)
+- **OverheadReport** — `memory_context_budget` MCP tool now shows: fixed token overhead breakdown, unused skill/agent detection, potential savings recommendations
+- **InjectionValidator** — sanitizes context before `UserPromptSubmit` injection. Strips HTML comments, caps at 4500 chars, filters dead resource recommendations via `filterAliveRecommendations()`
+- **Agent/Skill entities** — `Agent` and `Skill` tool calls now produce `entity_type="decision"` entities (importance 3/2), visible in summarization and compact scoring
+- **Expanded resource types** — `resource_usage` table tracks 8 types: skill, agent, command, workflow, claude_md, memory, mcp_tool, hook (was 5)
+- **Real token costs** — `SmartResourceLoader` uses ResourceRegistry for actual file-size-based estimates instead of hardcoded 500 fallback
+
+### Phase 2 — Schema v3 + Observations + CLAUDE.md Tracking
+
+- **Schema migration v3** — entities table rebuilt with `observation` type in CHECK constraint + new `claude_md_registry` table
+- **Observation extractor** — heuristic-based free-form capture from tool output and user prompts. Keywords: IMPORTANT/CRITICAL (importance 4), decision:/NOTE: (3), TODO:/FIXME: (2). Max 1 observation per tool call, capped at 300 chars
+- **CLAUDE.md tracker** — walks from `cwd` to root, finds all CLAUDE.md files, extracts `## sections` + 200-char previews, content-hash change detection (only re-parses on change), injects rule summary into context
+- **Session summarizer** includes top 5 observations in L3 summaries
+- **Vector search** reindexes observation entities alongside decisions and errors
+
+### Phase 3 — LLM Summarization Pipeline
+
+- **3-tier fallback** — Tier 1: PostCompact summary (free, already existed). Tier 2: `claude -p ... --print` subprocess with 30s timeout. Tier 3: Rule-based (always available)
+- **Hook recursion guard** — `CLAUDE_MEMORY_HUB_SKIP_HOOKS=1` env var set on CLI subprocess, checked by all 5 hook entry scripts. Prevents infinite loop when CLI summarizer triggers hooks
+- **Configurable** — `CLAUDE_MEMORY_HUB_LLM=auto|cli-only|rule-based` env var. `CLAUDE_MEMORY_HUB_LLM_TIMEOUT_MS` for custom timeout
+
+### Phase 4 — Semantic Search
+
+- **Embedding model** — `@huggingface/transformers` with `all-MiniLM-L6-v2` (384-dim, 90MB cached, 9ms warm inference). Lazy-loaded: only imports when first embedding requested. Graceful degradation if package not installed
+- **Pure JS cosine similarity** — no native sqlite-vec binary needed. Fast enough for <1000 docs. Embeddings stored as BLOBs in new `embeddings` table (schema v4)
+- **Hybrid search** — `searchIndex()` now merges FTS5 BM25 + TF-IDF + semantic cosine similarity. Deduplicates by id+type, keeps highest score
+- **Auto-indexing** — session-end hook generates embedding for new summaries automatically
+- **Opt-in** — `CLAUDE_MEMORY_HUB_EMBEDDINGS=auto|disabled` env var. `@huggingface/transformers` is `optionalDependencies` — install failure doesn't break anything
+
+### New Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `CLAUDE_MEMORY_HUB_LLM` | `auto` | Summarization mode: auto, cli-only, rule-based |
+| `CLAUDE_MEMORY_HUB_LLM_TIMEOUT_MS` | `30000` | CLI summarizer timeout in ms |
+| `CLAUDE_MEMORY_HUB_EMBEDDINGS` | `auto` | Embedding mode: auto, disabled |
+| `CLAUDE_MEMORY_HUB_SKIP_HOOKS` | — | Set to `1` to suppress hooks (internal use) |
+
+### New/Modified Files
+
+```
+NEW:
+  src/context/resource-registry.ts      — unified resource scanner
+  src/context/injection-validator.ts    — context sanitization
+  src/capture/observation-extractor.ts  — free-form observation capture
+  src/context/claude-md-tracker.ts      — CLAUDE.md scanning + tracking
+  src/summarizer/cli-summarizer.ts      — Tier 2 CLI summarization
+  src/search/embedding-model.ts         — lazy @huggingface/transformers
+  src/search/semantic-search.ts         — cosine similarity search
+
+MODIFIED:
+  src/db/schema.ts                      — migrations v3 + v4
+  src/types/index.ts                    — EntityType += observation
+  src/capture/entity-extractor.ts       — Agent/Skill + observation extraction
+  src/capture/hook-handler.ts           — registry + validator + CLAUDE.md + observations
+  src/context/smart-resource-loader.ts  — uses ResourceRegistry
+  src/context/resource-tracker.ts       — 8 resource types
+  src/mcp/tool-handlers.ts             — overhead report in context_budget
+  src/summarizer/session-summarizer.ts  — 3-tier pipeline
+  src/search/search-workflow.ts         — hybrid FTS5+TF-IDF+semantic
+  src/search/vector-search.ts           — reindex includes observations+embeddings
+  src/db/session-store.ts               — getSessionObservations()
+  src/hooks-entry/*.ts                  — SKIP_HOOKS recursion guard
+```
+
+### Dependencies
+
+```
+KEPT:     @modelcontextprotocol/sdk
+ADDED:    @huggingface/transformers (optional — semantic search)
+```
+
+---
+
 ## [0.5.2] - 2026-04-01
 
 ### Fixed
@@ -111,10 +317,11 @@ Every Claude Code session loads ALL skills, agents, rules, and memory files into
 
 ### Impact
 ```
-BEFORE: 23-51K tokens overhead (all resources loaded)
-AFTER:  Only frequently-used resources recommended
-        Rare resources loaded on demand via SkillTool
-        ~10-30K tokens saved per session on heavy setups
+Tracks resource usage across sessions
+Identifies unused skills/agents with token cost estimates
+Provides recommendations for manual cleanup
+NOTE: Claude Code loads ALL resources regardless — this is
+      an analysis tool, not a filter. See v0.7.0 for details.
 ```
 
 ### Files

package/README.md

@@ -31,22 +31,28 @@ Long session: Claude auto-compacts at 200K tokens
 
 Every session: ALL skills + agents + rules loaded
                → 23-51K tokens consumed before you type anything
-               → Most of them never used
+               → No external tool can prevent this (Claude Code limitation)
+               → But you CAN identify and remove unused resources
 
 Search:        Keyword-only, no semantic ranking
                → Irrelevant results, wasted tokens on full records
 ```
 
-**Four problems. No existing tool solves all of them.**
+**Four problems. memory-hub solves three directly and provides analysis for the fourth.**
 
 | Problem | Claude Code built-in | claude-mem | memory-hub |
 |---------|:-------------------:|:----------:|:----------:|
 | Cross-session memory | -- | Yes | **Yes** |
 | Influence what compact preserves | -- | -- | **Yes** |
 | Save compact output | -- | -- | **Yes** |
-| Token budget optimization | -- | -- | **Yes** |
-| Hybrid search (FTS5 + TF-IDF) | -- | Partial | **Yes** |
+| Token overhead analysis | -- | -- | **Yes** |
+| Semantic search (embeddings) | -- | Chroma (external) | **Yes (offline)** |
+| Hybrid search (FTS5 + TF-IDF + semantic) | -- | Partial | **Yes** |
 | 3-layer progressive search | -- | Yes | **Yes** |
+| Resource overhead analysis | -- | -- | **Yes** |
+| CLAUDE.md rule tracking | -- | -- | **Yes** |
+| Free-form observation capture | -- | Yes | **Yes** |
+| LLM summarization (3-tier) | -- | Yes (API) | **Yes (free)** |
 | Browser UI | -- | Yes | **Yes** |
 | Health monitoring | -- | -- | **Yes** |
 | Migrate from claude-mem | N/A | N/A | **Yes** |
@@ -107,27 +113,27 @@ Session N+1     → UserPromptSubmit hook fires
                 → Claude starts with history, not from zero
 ```
 
-### Layer 4 — Smart Resource Loading
+### Layer 4 — Resource Intelligence & Overhead Analysis
 
 ```
-                 Typical Claude Code session
+ResourceRegistry scans your setup:
+  58 skills, 36 agents, 65 commands, 10 workflows, CLAUDE.md chain
 
-    BEFORE memory-hub          AFTER memory-hub
-    ┌──────────────────┐        ┌──────────────────┐
-    │ System prompt 8K │        │ System prompt 8K │
-    │ ALL skills  10K  │        │ Used skills  3K  │
-    │ ALL agents   5K  │        │ Used agents  1K  │
-    │ ALL rules   15K  │        │ Key rules    5K  │
-    │ ALL memory   5K  │        │ Relevant mem 2K  │
-    ├──────────────────┤        ├──────────────────┤
-    │ OVERHEAD:  ~43K  │        │ OVERHEAD:  ~19K  │
-    │                  │        │ SAVED:     ~24K  │
-    └──────────────────┘        └──────────────────┘
+ResourceTracker records actual usage per session:
+  "skill:mobile-development used 4/5 recent sessions"
+  "agent:veo3-prompt-expert used 0/5 recent sessions"
+
+OverheadReport identifies waste:
+  "42/58 skills never used → ~1500 listing tokens overhead"
+  "CLAUDE.md chain is 8200 tokens → consider consolidating"
+
+UserPromptSubmit injects priority hints:
+  "Frequently-used: skill:debugging, agent:planner, agent:tester"
 ```
 
-memory-hub tracks which skills/agents/tools you **actually use**, then recommends only those for future sessions. Rare resources load on demand via SkillTool.
+> **Transparency note:** Claude Code loads ALL resources into its system prompt — no external tool can prevent this. memory-hub provides **analysis and prioritization**, not filtering. To actually reduce token overhead, remove or relocate unused skills/agents based on the overhead report.
 
-### Layer 5 — 3-Layer Progressive Search (new in v0.5)
+### Layer 5 — 3-Layer Progressive Search + Semantic (new in v0.5/v0.6)
 
 ```
 Traditional search: query → ALL full records → 5000+ tokens wasted
@@ -140,7 +146,35 @@ memory-hub search:  query → Layer 1 (index)    → ~50 tokens/result
                     Token savings: ~80-90% vs. full context
 ```
 
-Hybrid ranking: FTS5 BM25 for keyword matches + TF-IDF cosine similarity for semantic ranking. Zero external dependencies — pure TypeScript implementation.
+Hybrid ranking: FTS5 BM25 (keyword) + TF-IDF (term frequency) + **semantic cosine similarity** (384-dim embeddings, v0.6). "debugging tips" now matches "error fixing" even without shared keywords.
+
+### Layer 6 — Resource Intelligence (new in v0.6)
+
+```
+ResourceRegistry scans ALL .claude locations:
+  ~/.claude/skills/          58 skills → listing + full + total tokens
+  ~/.claude/agents/          36 agents → frontmatter name: resolution
+  ~/.claude/agent_mobile/    ios-developer → agent_mobile/ios/AGENT.md
+  ~/.claude/commands/        65 commands → relative path naming
+  ~/.claude/workflows/       10 workflows
+  ~/.claude/CLAUDE.md        + project CLAUDE.md chain
+
+OverheadReport:
+  "56/64 skills unused in last 10 sessions → ~1033 listing tokens wasted"
+  "CLAUDE.md chain is 3222 tokens"
+```
+
+### Layer 7 — Observation Capture (new in v0.6)
+
+```
+Tool output contains "IMPORTANT: always pool DB connections"
+  → observation entity (importance=4) saved to L2
+  → included in session summary
+  → searchable across sessions
+
+User prompt contains "remember that we use TypeScript strict"
+  → observation entity (importance=3) saved to L2
+```
 
 ---
 
@@ -195,6 +229,9 @@ Hybrid ranking: FTS5 BM25 for keyword matches + TF-IDF cosine similarity for sem
                    │   resource_usage   │
                    │   fts_memories     │
                    │   tfidf_index      │
+                   │   embeddings       │
+                   │   claude_md_       │
+                   │    registry        │
                    │   health_checks    │
                    └────────────────────┘
 ```
@@ -269,6 +306,9 @@ bunx claude-memory-hub migrate     # Import data from claude-mem
 bunx claude-memory-hub viewer      # Open browser UI at localhost:37888
 bunx claude-memory-hub health      # Run health diagnostics
 bunx claude-memory-hub reindex     # Rebuild TF-IDF search index
+bunx claude-memory-hub export      # Export data as JSONL to stdout
+bunx claude-memory-hub import      # Import JSONL from stdin
+bunx claude-memory-hub cleanup     # Remove old data (default: 90 days)
 ```
 
 ### Requirements
@@ -361,6 +401,9 @@ Migration is idempotent — safe to run multiple times with zero duplicates.
 | **v0.3.0** | Removed API key requirement, 1-command install |
 | **v0.4.0** | Smart resource loading, token budget optimization |
 | **v0.5.0** | Production hardening, hybrid search, 3-layer progressive search, browser UI, health monitoring, claude-mem migration |
+| **v0.6.0** | ResourceRegistry (170 resources), semantic search (384-dim embeddings), observation capture, CLAUDE.md tracking, 3-tier LLM summarization, overhead analysis |
+| **v0.7.0** | Honest resource analysis, semantic search scaling, batch embeddings, 14 observation patterns, DB auto-cleanup, summarizer retry |
+| **v0.8.0** | 91 unit tests (was 0%), L1 WorkingMemory read-through cache, PostToolUse batch queue (75ms→3ms), JSONL export/import CLI, data cleanup command |
 
 See [CHANGELOG.md](CHANGELOG.md) for full details.
 
@@ -369,13 +412,22 @@ See [CHANGELOG.md](CHANGELOG.md) for full details.
 ## Dependencies
 
 ```
-@modelcontextprotocol/sdk    MCP stdio server
-bun:sqlite                   Built-in, zero install
+@modelcontextprotocol/sdk          MCP stdio server (required)
+bun:sqlite                         Built-in, zero install
+@huggingface/transformers          Semantic search embeddings (optional)
 ```
 
-That's it. **One npm package.** The other is built into Bun.
+**Two npm packages + one optional.** No Python. No Chroma. No HTTP server. No API key. No Docker.
+
+### Environment Variables
 
-No Python. No Chroma. No HTTP server. No API key. No Docker.
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `CLAUDE_MEMORY_HUB_LLM` | `auto` | Summarization: auto, cli-only, rule-based |
+| `CLAUDE_MEMORY_HUB_LLM_TIMEOUT_MS` | `30000` | CLI summarizer timeout |
+| `CLAUDE_MEMORY_HUB_EMBEDDINGS` | `auto` | Embeddings: auto, disabled |
+| `CLAUDE_MEMORY_HUB_BATCH` | `auto` | PostToolUse batching: auto, enabled, disabled |
+| `CMH_LOG_LEVEL` | `info` | Log level: debug, info, warn, error |
 
 ---