engrm 0.1.0

package/COMPETITIVE.md

@@ -0,0 +1,174 @@
+# Competitive Analysis — Why Engrm Wins
+
+## Market Landscape
+
+The AI agent memory space is early-stage and fragmented. No player has captured the "shared memory for coding agents" category. Here's where everyone stands:
+
+---
+
+## Head-to-Head Comparison
+
+### claude-mem (thedotmack)
+**What it is**: Local-only memory plugin for Claude Code. SQLite + ChromaDB. AGPL-3.0.
+
+| Aspect | claude-mem | Engrm |
+|---|---|---|
+| Storage | Local SQLite + local ChromaDB | Local SQLite + Candengo Vector (remote) |
+| Cross-device | No | Yes (offline-first sync) |
+| Team support | No | Yes (shared namespaces) |
+| Multi-agent | Claude Code only | Claude Code + OpenClaw + MCP agents |
+| Search quality | ChromaDB default embeddings | BGE-M3 hybrid + cross-encoder reranking |
+| Workpacks | No | Yes |
+| Secret scrubbing | No | Yes |
+| Self-hosted backend | N/A (local only) | Yes |
+| License | AGPL-3.0 (viral copyleft) | FSL-1.1-ALv2 (Fair Source, converts to Apache 2.0 after 2yr) |
+| Offline support | Always local | Offline-first with sync |
+
+**Why we win**: claude-mem is a good single-device tool but architecturally can't do cross-device or team without a major rewrite. Engrm is built from scratch around cross-device sync — no fork, no shared code, no AGPL constraints.
+
+---
+
+### mem0
+**What it is**: Cloud memory layer for AI agents. VC-funded. Proprietary SaaS.
+
+| Aspect | mem0 | Engrm |
+|---|---|---|
+| Hosting | SaaS only (their cloud) | Self-hosted or our cloud |
+| Privacy | Code context sent to mem0's servers | Your infrastructure, your data |
+| Cross-device | Yes (cloud) | Yes (self-hosted or cloud) |
+| Team support | Limited | Built-in from day one |
+| Multi-agent | Multiple via API | Multiple via MCP standard |
+| Offline support | No (requires internet) | Yes (offline-first) |
+| Workpacks | No | Yes |
+| Pricing | Per-API-call | Subscription (predictable) |
+| Lock-in | High (proprietary API, their storage) | Low (MCP standard, self-hosted option) |
+
+**Why we win**: Privacy and control. Many developers and enterprises won't send code context to a third-party cloud. Self-hosted Engrm with offline-first sync is the answer they're looking for. mem0's SaaS-only model is their weakness.
+
+---
+
+### Cognee
+**What it is**: Knowledge graph memory for AI agents. Open source. Focus on semantic relationships.
+
+| Aspect | Cognee | Engrm |
+|---|---|---|
+| Architecture | Knowledge graphs (Neo4j / networkx) | Vector search (Qdrant + BGE-M3) |
+| Strength | Relationship reasoning | Fast semantic retrieval |
+| Cross-device | No | Yes |
+| Team support | No | Yes |
+| Setup complexity | High (graph DB, entity extraction) | Low (SQLite local, REST API remote) |
+| Agent support | OpenClaw focus | Claude Code + OpenClaw + MCP |
+| Developer focus | General AI memory | Purpose-built for coding agents |
+
+**Why we win**: Cognee is solving a different problem (knowledge graphs for reasoning). We're solving the practical problem developers have right now: "I need my AI to remember what I did yesterday, on any machine."
+
+---
+
+### OpenClaw Built-in Memory
+**What it is**: Local Markdown files + SQLite with FTS5 and sqlite-vec. Per-agent, per-device.
+
+| Aspect | OpenClaw Memory | Engrm |
+|---|---|---|
+| Storage | Local Markdown + SQLite | Local SQLite + remote vector |
+| Cross-device | No | Yes |
+| Cross-agent | No (OpenClaw only) | Yes (Claude Code + OpenClaw) |
+| Team support | No | Yes |
+| Search | 70/30 vector/BM25 hybrid | BGE-M3 hybrid + reranking |
+| Workpacks | No | Yes |
+| Memory plugin slot | Replaceable (`plugins.slots.memory`) | Can slot in as replacement |
+
+**Why we win**: OpenClaw's memory is good for single-device, single-agent use. We extend it to cross-device, cross-agent, and team scenarios while plugging into their existing memory slot architecture.
+
+---
+
+### Cursor / Windsurf / Continue Memory
+**What it is**: Proprietary, IDE-integrated context. Varies by product.
+
+| Aspect | IDE-Native Memory | Engrm |
+|---|---|---|
+| Portability | Locked to one IDE | Works across agents/IDEs |
+| Cross-device | Via IDE account sync (limited) | Full observation sync |
+| Control | Proprietary, opaque | Open plugin, transparent |
+| Team | Some (via IDE features) | Built-in, flexible |
+
+**Why we win**: These are walled gardens. Switch IDE, lose memory. We're the portable layer that follows you regardless of which agent or IDE you use.
+
+---
+
+## Defensible Moats
+
+### 1. Full Stack Ownership
+We control plugin + backend + workpack ecosystem. No dependency on third-party pricing or availability changes. This is rare — most memory solutions depend on external vector DBs.
+
+### 2. Workpack Ecosystem
+No competitor has anything like workpacks. This is a compounding advantage:
+- More users → more observations → better auto-generated workpacks
+- Better workpacks → more value → more users
+- Premium workpacks = recurring revenue with high margins
+- Community workpacks = distribution and engagement
+
+### 3. Network Effects (Team Memory)
+Each new team member makes the memory more valuable for everyone. This creates organic retention and word-of-mouth growth. Individual tools don't have this dynamic.
+
+### 4. Self-Hosted Trust
+In an era of increasing AI privacy concerns, being self-hostable is a strategic differentiator. Enterprise customers with compliance requirements can only use self-hosted solutions. This market is underserved.
+
+### 5. Cross-Agent Portability
+MCP is becoming the standard protocol for AI agent tools. By supporting MCP natively, we're positioned for every future agent that adopts the protocol. Competitors locked to one agent lose users when developers switch tools.
+
+### 6. Offline-First Architecture
+The only solution that works without internet. This sounds like a niche feature but it's actually critical for:
+- Developers commuting (trains, flights)
+- Corporate environments with restricted internet
+- Regions with unreliable connectivity
+- Privacy-first users who want to control when data syncs
+
+### 7. Free Tier with Cloud Sync
+Every other Claude Code memory plugin is local-only. We're the first to offer free cloud sync (10K observations, 2 devices). This makes cross-device memory the default experience, not a paid upgrade. The free tier is generous enough to be genuinely useful — adoption first, monetisation via natural upgrade when users hit limits or need team features.
+
+### 8. FSL License + Proprietary Sentinel
+Source-available under FSL-1.1-ALv2 (Fair Source). Developers can read, modify, and run the core freely. Competitors cannot fork and offer a competing hosted service. Each version converts to Apache 2.0 after 2 years — a trust signal no competitor matches. Premium features (Sentinel real-time AI audit) are in a separate private repo, delivered to paying customers only. This is the GitLab CE/EE pattern: open core attracts adoption, proprietary premium drives revenue.
+
+---
+
+## Market Timing Advantage
+
+### Why Now
+
+1. **Claude Code** is Anthropic's fastest-growing developer tool. Memory is the #1 community request.
+2. **OpenClaw** hit 100k+ GitHub stars in weeks. Its community is actively building and seeking memory solutions.
+3. **MCP adoption** is accelerating. Every major AI tool is adding MCP support.
+4. **Developer AI spend** is normalising. Companies are budgeting for AI tooling. Memory is a natural add-on.
+5. **No incumbent** has captured the cross-device shared memory category. The window is open.
+
+### First Mover Advantage Opportunity
+The agent memory market will consolidate. The first solution that nails cross-device + team + workpacks becomes the default. Switching costs in memory systems are high (you can't easily migrate years of observations). Early adopters become long-term customers.
+
+---
+
+## Go-to-Market Strategy
+
+### Phase 1: Developer Adoption (Months 1-3)
+- Open-source the plugin (MIT license) → GitHub stars, trust, community
+- Launch on Claude Code plugin marketplace + OpenClaw skill directory
+- Blog posts, demo videos, Twitter/X developer threads
+- Free tier with self-hosted Candengo Vector
+
+### Phase 2: Team Conversion (Months 3-6)
+- Target teams already using Claude Code or OpenClaw
+- Team onboarding flow: one config snippet, instant shared memory
+- Case studies from our own team (dogfooding with Alchemy development)
+
+### Phase 3: Enterprise + Workpacks (Months 6-12)
+- Enterprise self-hosted offering with support SLA
+- Premium workpack marketplace launch
+- Partner with framework communities for co-branded workpacks
+- Conference talks and workshops
+
+### Distribution Channels
+1. **GitHub** — open-source plugin repo, README, examples
+2. **Plugin marketplaces** — Claude Code, OpenClaw skill registry
+3. **Candengo website** — product page, pricing, docs
+4. **Developer communities** — Reddit, HackerNews, Discord servers
+5. **Content marketing** — blog posts, tutorials, comparison guides
+6. **Word of mouth** — team features drive organic growth

package/CONTEXT-OPTIMIZATION.md

@@ -0,0 +1,305 @@
+# Context & Token Optimization Research
+
+## Date: 2026-03-10
+
+## Current State
+
+- 7 observations, 124KB DB, 178 tests passing
+- FTS5 search: 0.01-0.10ms (well under 50ms target)
+- `search()` tool: 11-15ms
+- `session_context()`: 11ms, ~653 tokens for 7 observations
+- Context injection: top 10 by quality, 150-char narrative truncation
+
+## Open Source Landscape
+
+### Key Projects Studied
+
+| Project | Key Technique | Relevance |
+|---------|--------------|-----------|
+| [Mem0](https://github.com/mem0ai/mem0) | Memory compression, graph memory, 90% token reduction | High |
+| [OpenMemory](https://github.com/CaviraOSS/OpenMemory) | Hierarchical Memory Decomposition, temporal validity, multi-sector embeddings | High |
+| [mcp-memory-service](https://github.com/doobidoo/mcp-memory-service) | Dream-inspired consolidation, knowledge graph, decay scoring | Medium |
+| [token-optimizer-mcp](https://github.com/ooples/token-optimizer-mcp) | 95%+ token reduction via caching/compression | Medium |
+| [mcp-memory-keeper](https://github.com/mkreyman/mcp-memory-keeper) | Token budgets with safety buffers | Medium |
+| [Speakeasy Dynamic Toolsets](https://www.speakeasy.com/blog/how-we-reduced-token-usage-by-100x-dynamic-toolsets-v2) | 96% input token reduction via meta-tools and progressive disclosure | High |
+| [MCP Protocol #1576](https://github.com/modelcontextprotocol/modelcontextprotocol/issues/1576) | Schema redundancy reduction | Low |
+
+### Key Research
+
+- [Mem0 Paper (arXiv 2504.19413)](https://arxiv.org/abs/2504.19413): 26% accuracy improvement, 91% lower latency, 90% token savings
+- [Scott Spence: Optimising MCP Context](https://scottspence.com/posts/optimising-mcp-server-context-usage-in-claude-code): 66K tokens consumed before conversation starts with many MCPs
+
+## Identified Improvements
+
+### Tier 1: Quick Wins (XS-S effort)
+
+#### 1. Token Budget (not count limit)
+**Source**: mcp-memory-keeper
+**Current**: `session_context` caps at 10 observations regardless of size
+**Proposed**: Cap at ~800 tokens, fill greedily by relevance score
+**Impact**: Prevents context blowup at scale
+
+#### 2. Facts-First Context
+**Source**: Mem0 compression engine
+**Current**: Context shows narrative snippets (prose, verbose)
+**Proposed**: Show `facts[]` array (bullet points) instead of narrative — 50% denser information per token
+**Impact**: ~50% more information in same token budget
+
+#### 3. Tiered Context Injection
+**Source**: Speakeasy progressive disclosure pattern
+**Current**: All observations get title + 150-char narrative equally
+**Proposed**:
+  - Top 3 by relevance: title + facts/narrative snippet
+  - Remaining: title-only, single line
+  - Footer: "N more observations available via `search`"
+**Impact**: ~40% token reduction on context injection
+
+#### 4. Recency × Quality Blended Scoring
+**Source**: General best practice, Mem0
+**Current**: `ORDER BY quality DESC, created_at_epoch DESC` — quality dominates
+**Proposed**: `score = quality * 0.6 + recency_normalized * 0.4`
+**Impact**: Better relevance = less re-searching by agent
+
+#### 5. Terse Tool Descriptions
+**Source**: MCP Protocol Issue #1576
+**Current**: Verbose `.describe()` strings on each tool parameter
+**Proposed**: Trim to minimal descriptions, remove obvious ones
+**Impact**: ~500 tokens saved per conversation
+
+#### 6. Double-Injection Guard
+**Current**: session-start hook AND `session_context` MCP tool can both fire
+**Proposed**: Track injection per session, skip if already done
+**Impact**: Eliminates 100% duplication when both paths fire
+
+### Tier 2: Medium Effort (M)
+
+#### 7. Knowledge Supersession
+**Source**: OpenMemory temporal graph (`valid_from` / `valid_to`)
+**Current**: Old observations can contradict current reality
+**Proposed**: `superseded_by` field, auto-detect when new observation about same topic is saved
+**Impact**: Prevents stale/contradictory context
+
+#### 8. Concept-Based Linking
+**Source**: Mem0 graph memory, OpenMemory
+**Current**: `concepts` stored but unused for retrieval
+**Proposed**: Auto-link observations sharing concepts/files, cluster in context injection
+**Impact**: Better retrieval, fewer redundant observations shown
+
+#### 9. CWD/File-Aware Boosting
+**Current**: No awareness of what files the developer is working on
+**Proposed**: Boost observations whose `files_modified` overlap with current `git diff --name-only` or directory listing
+**Impact**: More relevant context for current work
+
+### Tier 3: Larger Effort (L)
+
+#### 10. Consolidation Scheduling
+**Source**: mcp-memory-service dream-inspired consolidation
+**Current**: Lifecycle aging planned but not implemented
+**Proposed**: Nightly merge of related low-quality observations, decay scoring
+**Impact**: Fewer, better observations over time
+
+## What NOT to Copy
+
+- Knowledge graphs with D3.js dashboards — over-engineered for our use case
+- Multi-sector embeddings (5 memory types) — too complex for now
+- Cloud sync via Cloudflare — we have Candengo Vector
+- REST API alongside MCP — MCP is our interface
+- Code mode workarounds — we're optimising within MCP
+
+## Performance Baseline (2026-03-10)
+
+```
+Database: 7 observations, 124KB
+FTS5 search: 0.01-0.10ms
+search() tool: 11-15ms
+session_context(): 11ms, ~653 tokens
+Context injection: ~2,612 chars for 7 obs
+Unit tests: 178 passing (600ms)
+```
+
+---
+
+## Implementation Plan
+
+### Decision: What Makes the Cut
+
+| # | Improvement | Verdict | Sprint | Rationale |
+|---|-------------|---------|--------|-----------|
+| 1 | Token budget | **YES** | 1 | Prevents blowup, foundational for tiered context |
+| 2 | Facts-first context | **YES** | 1 | Tightly coupled with token budget changes |
+| 3 | Tiered context | **YES** | 1 | Tightly coupled with token budget changes |
+| 4 | Blended scoring | **YES** | 2 | Independent, touches same file as 1-3 so do after |
+| 5 | Terse descriptions | **YES** | 1 | Isolated, zero-risk, immediate savings |
+| 6 | Double-injection guard | **YES** | 1 | Simple module-level flag, prevents waste |
+| 7 | Knowledge supersession | **YES** | 2 | Requires migration, medium effort, high value |
+| 8 | Concept linking | **DEFER** | 3+ | Needs join table, unclear value until more data |
+| 9 | CWD/file-aware boosting | **DEFER** | 3+ | Adds latency to hook, complexity for unclear gain |
+| 10 | Consolidation scheduling | **DEFER** | 3+ | Needs background process, premature at 7 obs |
+
+### Sprint 1: Core Context Optimizations
+
+Three parallel workstreams, all low-risk:
+
+#### Step A: Token Budget + Facts-First + Tiered Context (Items 1+2+3)
+
+**File**: `src/context/inject.ts`
+
+These three are inseparable — switching to a token budget requires deciding what fills it (facts-first) and how to tier it.
+
+**Changes**:
+1. Add `estimateTokens(text: string): number` — simple `Math.ceil(text.length / 4)` heuristic (standard for English, 10% safety margin built in by using budget of 720 out of 800)
+2. Change `buildSessionContext` signature: replace `maxObservations: number = 10` with options object `{ tokenBudget?: number, maxCount?: number }` for backwards compat
+3. Replace count-based LIMIT with larger fetch (LIMIT 50), apply token-budget filling in TypeScript
+4. Add `facts: string | null` to `ContextObservation` interface, update `toContextObservation`
+5. Rewrite `formatContextForInjection` for tiered output:
+   - **Top tier (first 3)**: title + parsed facts as bullet points (fallback: 100-char narrative)
+   - **Lower tier (rest)**: title-only, one line
+   - **Footer**: "N more observations available via search"
+6. Add `totalActive: number` to `InjectedContext` for footer count
+
+**File**: `hooks/session-start.ts`
+- Update `buildSessionContext` call to use new options: `buildSessionContext(db, event.cwd, { tokenBudget: 800 })`
+
+**Token math**: With 800 budget → ~3 detailed entries (20-40 tokens each) + 7-10 title-only entries (10-15 each) + 30 tokens header/footer
+
+#### Step B: Terse Tool Descriptions (Item 5)
+
+**File**: `src/server.ts`
+
+Trim tool and parameter descriptions:
+
+| Tool | Before | After |
+|------|--------|-------|
+| `save_observation` | "Save a coding observation (discovery, bugfix, decision, pattern, etc.) to memory" | "Save an observation to memory" |
+| `search` | "Search memory for relevant observations, discoveries, and decisions" | "Search memory" |
+| `get_observations` | "Fetch full details for specific observation IDs" | "Get observations by ID" |
+| `timeline` | "Get chronological context around a specific observation" | "Timeline around an observation" |
+| `pin_observation` | "Pin or unpin an observation to prevent lifecycle aging" | "Pin/unpin observation" |
+| `session_context` | "Get relevant project memory for the current session. Call at the start of a session to load prior context." | "Load project memory for this session" |
+
+Parameter descriptions: trim verbose ones, remove `.describe()` where enum/type is self-documenting.
+
+Estimated savings: ~100-150 tokens per conversation.
+
+#### Step C: Double-Injection Guard (Item 6)
+
+**File**: `src/server.ts`
+
+Since hooks and MCP server are separate processes (can't share state directly), use a simple module-level flag:
+
+```typescript
+let contextServed = false;
+```
+
+In `session_context` tool handler:
+- First call: serve full context, set `contextServed = true`
+- Subsequent calls: return "Context already loaded. Use search for specific queries."
+
+The flag resets on process restart, which maps to session boundaries in stdio transport.
+
+**Note**: Hook-to-MCP dedup (via DB flag) deferred to Sprint 2 as it needs a migration.
+
+### Sprint 2: Scoring + Supersession
+
+#### Step D: Recency × Quality Blended Scoring (Item 4)
+
+**File**: `src/context/inject.ts`
+
+Add scoring function:
+```
+computeBlendedScore(quality, createdAtEpoch, nowEpoch):
+  recencyNorm = max(0, 1 - (now - created) / (30 * 86400))  // 30-day linear decay
+  return quality * 0.6 + recencyNorm * 0.4
+```
+
+Fetch 50 candidates from DB (sorted by quality DESC), re-sort in TypeScript by blended score before applying token budget.
+
+Result: 2-day-old q=0.5 beats 25-day-old q=0.7 — correct behavior.
+
+#### Step E: Knowledge Supersession (Item 7)
+
+**Files**: `src/storage/migrations.ts`, `src/storage/sqlite.ts`, `src/context/inject.ts`, `src/server.ts`
+
+1. Migration v2: `ALTER TABLE observations ADD COLUMN superseded_by INTEGER REFERENCES observations(id) ON DELETE SET NULL`
+2. `MemDatabase.supersedeObservation(oldId, newId)` — sets `superseded_by` and archives old
+3. Filter `AND superseded_by IS NULL` in context and search queries
+4. Add `supersedes` param to `save_observation` tool
+
+---
+
+## Test Plan
+
+### New Tests for Sprint 1 (inject.test.ts)
+
+```
+describe("estimateTokens", () => {
+  test("empty string returns 0")
+  test("short string estimates correctly")
+  test("long narrative estimates within expected range")
+})
+
+describe("token budget", () => {
+  test("respects token budget — stops adding when budget exhausted")
+  test("always includes pinned observations even if they exceed budget")
+  test("800 token budget fits more entries than old 10-count limit with narratives")
+  test("single huge observation doesn't crash — graceful overflow")
+  test("default budget is 800 when no options provided")
+})
+
+describe("facts-first formatting", () => {
+  test("top-tier observations show facts as bullet points")
+  test("falls back to narrative snippet when no facts available")
+  test("lower-tier observations show title only")
+  test("footer shows correct remaining count")
+  test("handles malformed facts JSON gracefully")
+})
+
+describe("tiered context", () => {
+  test("first 3 observations get detailed format")
+  test("observations 4+ get title-only format")
+  test("footer says 'N more available via search'")
+  test("with ≤3 observations, all get detailed format, no footer")
+})
+```
+
+### New Tests for Sprint 1 (server.test.ts or inline)
+
+```
+describe("double-injection guard", () => {
+  test("first session_context call returns full context")
+  test("second session_context call returns abbreviated message")
+})
+```
+
+### New Tests for Sprint 2 (inject.test.ts)
+
+```
+describe("blended scoring", () => {
+  test("recent medium-quality beats old high-quality")
+  test("very old observations get near-zero recency boost")
+  test("same-age observations sort by quality")
+  test("blended score is always between 0 and 1")
+})
+```
+
+### New Tests for Sprint 2 (sqlite.test.ts, inject.test.ts)
+
+```
+describe("supersession", () => {
+  test("superseded observation excluded from session context")
+  test("superseded observation deprioritized in search results")
+  test("save with supersedes param archives old observation")
+  test("superseded_by set to NULL on referenced observation delete")
+  test("migration v2 adds superseded_by column correctly")
+})
+```
+
+## Risks & Mitigations
+
+| Risk | Impact | Mitigation |
+|------|--------|------------|
+| Token estimation inaccuracy (code tokens shorter than English) | Over-fill budget | Use 10% safety margin (720 of 800 effective budget) |
+| `buildSessionContext` signature change breaks hook | Hook crash | Use options object with defaults for backwards compat |
+| Module-level guard persists across sessions (if process reused) | Stale guard | Stdio transport = 1 process per session, safe |
+| Facts stored as JSON may be malformed | Formatting crash | Try/catch JSON.parse, fall back to raw string |
+| LIMIT 50 fetch overhead vs LIMIT 10 | Marginal latency | SQLite with indexes handles 50 rows in <1ms |