engrm 0.1.0

package/SWOT.md

@@ -0,0 +1,148 @@
+# SWOT Analysis — Engrm
+
+## Strengths
+
+### S1: Own the Full Stack
+We control both the plugin (Engrm) and the backend (Candengo Vector). No dependency on third-party vector DB pricing, availability, or API changes. We can optimise the entire pipeline end-to-end.
+
+### S2: Production-Grade Vector Search
+Candengo Vector uses BGE-M3 (state-of-the-art multilingual embeddings) with hybrid dense+sparse search and optional cross-encoder reranking. This is measurably superior to ChromaDB's default embeddings. Better search = more relevant memories surfaced = better developer experience.
+
+### S3: Self-Hosted / Privacy-First
+Developers and enterprises increasingly resist sending code context to third-party clouds. Candengo Vector can be self-hosted on-premise. No code leaves the organisation's infrastructure.
+
+### S4: Offline-First Architecture
+SQLite as local source of truth with sync outbox means the system works perfectly without internet. This is critical for developers who work on trains, planes, coffee shops, or behind corporate firewalls.
+
+### S5: Multi-Tenant from Day One
+Candengo Vector's site_id/namespace model was built for multi-tenancy. Team memory, per-project isolation, and cross-project discovery are architectural features, not afterthoughts.
+
+### S6: Workpack Ecosystem
+No competitor has curated knowledge packs. Workpacks transform the product from "memory" to "memory + expertise" — a significantly more valuable proposition.
+
+### S7: Existing Infrastructure
+Candengo Vector is already running in production with ingest, search, query, and admin APIs. We're building on proven infrastructure, not starting from scratch.
+
+### S8: Market Timing
+The AI coding agent market is in explosive growth (Claude Code, OpenClaw 100k+ stars, Cursor, Windsurf). Memory is the #1 requested feature across all agent communities. First mover with a cross-device, cross-agent solution captures the category.
+
+---
+
+## Weaknesses
+
+### W1: No Existing User Base for Memory Products
+Candengo's current customers use it for domain knowledge RAG (hospitality, business intelligence). Developer tooling is a new market for us — different buyer persona, different distribution channels.
+
+### W2: MCP Ecosystem Immaturity
+The Model Context Protocol is young. OpenClaw's MCP support is still Phase 0 (merged Feb 2026). Breaking changes to MCP could require rework. Agent-specific plugin formats may fragment the market.
+
+### W3: Small Team
+Building and maintaining plugins for multiple agents (Claude Code, OpenClaw, future agents) requires ongoing effort across different ecosystems, languages (TypeScript, Python), and release cycles.
+
+### W4: Developer Tooling Distribution
+Reaching developers requires different go-to-market than enterprise SaaS. GitHub presence, developer relations, documentation quality, and community engagement are table stakes. This is a capability we need to build.
+
+### W5: Observation Quality Dependency
+The value of shared memory depends on the quality of captured observations. If the extraction produces noisy, redundant, or inaccurate observations, shared memory amplifies noise rather than signal. Quality filtering is essential but hard to get right.
+
+### W6: Candengo Vector Search Limitations
+Currently no metadata field filtering in the search API (only site_id, namespace, source_type). This limits query flexibility. Needs to be added before launch.
+
+---
+
+## Opportunities
+
+### O1: Cross-Agent Memory Standard
+If we establish Engrm as the de facto shared memory layer, we become the "Twilio of agent memory" — the infrastructure everyone builds on. The MCP standard makes this possible by providing a common interface.
+
+### O2: Workpack Marketplace Revenue
+Curated knowledge packs (security, testing, framework patterns, compliance) are recurring revenue with high margins. Enterprise customers will pay for proprietary workpacks and custom knowledge curation.
+
+### O3: Auto-Generated Workpacks
+With enough observations across users (anonymised, opt-in), we can detect cross-cutting patterns and automatically generate workpacks. "90% of FastAPI projects hit this CORS issue" → auto-curated pattern. This creates a data flywheel that competitors can't replicate.
+
+### O4: Team Analytics Dashboard
+Visibility into team knowledge patterns: "What are the most common issues?" "Which areas have knowledge gaps?" "How is AI-assisted productivity trending?" This is valuable to engineering managers and CTOs.
+
+### O5: Enterprise Compliance
+SOC 2, GDPR, on-premise deployment. Enterprise engineering teams want AI tools but face compliance barriers. Self-hosted Engrm with audit logging and data residency controls addresses this directly.
+
+### O6: Integration with Alchemy Platform
+AIMY (our conversational AI) + Engrm creates a powerful combination: AIMY conducts project interviews, stores context in Candengo Vector, and all AI agents across the platform share that knowledge. End-to-end Alchemy integration is a unique selling point.
+
+### O7: OpenClaw's Massive Community
+OpenClaw has 100k+ GitHub stars and a rapidly growing community that's actively looking for memory solutions. Being an early, high-quality memory plugin for OpenClaw could drive massive adoption.
+
+### O8: CI/CD and DevOps Integration
+Beyond coding agents, the memory layer could capture observations from CI/CD pipelines, deployment logs, incident reports. "Last time this test failed, the fix was..." — extending memory beyond interactive sessions.
+
+### O9: Language and Framework Workpacks
+Partner with framework communities to create official workpacks: "Django Best Practices by Django Core Team", "React Patterns by Meta Engineers". Co-branded content drives distribution and credibility.
+
+---
+
+## Threats
+
+### T1: Anthropic / OpenAI Building Native Memory
+If Claude Code or ChatGPT build first-party cross-device memory, our plugin becomes redundant for those platforms. Mitigation: cross-agent support (no single-platform dependency), self-hosted differentiation, workpack ecosystem as defensible moat.
+
+### T2: claude-mem Going Cross-Device
+claude-mem already has remote Chroma support in its config. If the maintainer adds cross-device sync, our Claude Code differentiation weakens. Mitigation: superior search quality (BGE-M3 vs ChromaDB defaults), team features, and multi-agent support. Our clean-room build means no licensing entanglement regardless.
+
+### T3: mem0 Capturing the Market
+mem0 has funding, brand recognition, and multi-agent support. If they move fast on cross-device and self-hosted options, they could capture the market before us. Mitigation: self-hosted positioning (mem0 is SaaS-only), workpack ecosystem, developer trust through open-source plugin.
+
+### T4: MCP Standard Fragmentation
+If major agents abandon MCP for proprietary plugin formats, our "write once, work everywhere" value proposition breaks. Mitigation: the MCP standard has strong momentum (Anthropic backing, broad adoption), and we can always add agent-specific adapters.
+
+### T5: Privacy Concerns with Shared Memory
+Developers may resist syncing their code observations to any remote server, even self-hosted. Observations inevitably capture code patterns, file paths, and problem descriptions. Mitigation: secret scrubbing, sensitivity classification, clear data policies, and the self-hosted option.
+
+### T6: Backend Substitution
+Someone forks the plugin and swaps Candengo Vector for Pinecone/Weaviate/etc., offering a competing service. Mitigation: FSL-1.1-ALv2 license explicitly prohibits competing uses. Developers can self-host with Candengo Vector (encouraged), but cannot build a competing hosted product. Sentinel (the premium AI audit feature) is in a separate private repo, not published. The deep Candengo Vector integration (provisioning, account routing, hybrid search pipeline) also makes casual swaps non-trivial.
+
+### T7: Quality at Scale
+As team sizes grow, observation volume increases. Without good quality filtering, relevance decay, and deduplication, search results degrade. This is a hard problem that requires ongoing investment.
+
+---
+
+## Strategic Position Summary
+
+```
+              HIGH IMPACT
+                  │
+    ┌─────────────┼─────────────┐
+    │   THREATS   │ OPPORTUNITIES│
+    │             │              │
+    │  T1: Native │ O1: Memory   │
+    │  platform   │ standard     │
+    │  memory     │              │
+    │             │ O2: Workpack │
+    │  T3: mem0   │ marketplace  │
+    │  momentum   │              │
+    │             │ O7: OpenClaw │
+    │             │ community    │
+    ├─────────────┼──────────────┤
+    │  WEAKNESSES │  STRENGTHS   │
+    │             │              │
+    │  W1: New    │ S1: Full     │
+    │  market     │ stack        │
+    │             │              │
+    │  W4: Dev    │ S2: Superior │
+    │  distrib.   │ search       │
+    │             │              │
+    │  W3: Small  │ S4: Offline  │
+    │  team       │ first        │
+    │             │              │
+    └─────────────┼──────────────┘
+                  │
+              LOW IMPACT
+```
+
+## Key Strategic Moves
+
+1. **Lead with open-source plugin** → drives adoption, builds trust, creates community
+2. **Differentiate on self-hosted + offline-first** → mem0 can't match this
+3. **Workpacks as moat** → unique value that compounds over time
+4. **OpenClaw community first** → 100k+ stars, actively seeking memory solutions
+5. **Team features from day one** → enterprise upsell path built in

package/SYNC-ARCHITECTURE.md

@@ -0,0 +1,294 @@
+# Sync Architecture — Engrm
+
+## Date: 2026-03-10
+## Status: Reviewed & Confirmed
+
+---
+
+## Overview
+
+Bidirectional sync between local SQLite instances and Candengo Vector, enabling cross-device and cross-agent team memory. Every device maintains a full local cache — search is always local-first and fast, even offline.
+
+```
+Device A (Claude Code)                     Device B (Codex CLI)
+┌───────────────────┐                     ┌───────────────────┐
+│  SQLite (primary)  │───push──→          │  SQLite (primary)  │───push──→
+│  FTS5 search       │          │         │  FTS5 search       │          │
+│  offline-first     │←──pull───┤         │  offline-first     │←──pull───┤
+└───────────────────┘          │         └───────────────────┘          │
+                                ▼                                        ▼
+                    ┌──────────────────────────┐
+                    │   Candengo Vector         │
+                    │   + Sync Metadata Layer   │
+                    │   BGE-M3 embeddings       │
+                    │   hybrid search           │
+                    └──────────────────────────┘
+```
+
+---
+
+## Confirmed Decisions
+
+### 1. SQLite is Source of Truth Per-Device
+
+**Status**: Confirmed ✓
+
+- Each device has its own SQLite with WAL mode, FTS5 external content
+- Offline-first: all operations work without network
+- Vector downtime never blocks local operations
+- Push failures queue in outbox with exponential backoff
+
+### 2. Candengo Vector is the Sync Hub (Not Queried at Search Time)
+
+**Status**: Confirmed with modification ✓
+
+- Vector stores embeddings + metadata for team-wide semantic search (future Phase 4+)
+- **Change**: Add a thin **sync metadata layer** rather than overloading Vector with sync-specific queries
+- Sync metadata tracks: `(source_id, ingested_at_server, action, payload_json)`
+- This decouples the sync protocol from the vector search engine
+- Vector remains a search index; sync metadata is the replication log
+
+**Rationale**: Vector databases are optimised for similarity search, not for "list all documents modified since X". A separate sync metadata table (in the Candengo web backend) is cleaner and more maintainable.
+
+### 3. Same MCP Server for All Agents
+
+**Status**: Confirmed ✓
+
+Validated working with:
+- Claude Code (`.mcp.json`, stdio) — **tested live**
+- Codex CLI (`config.toml`, stdio) — **tested live, all 6 tools working**
+- Cursor, Windsurf, Cline, VS Code Copilot, Zed (all MCP stdio compatible)
+
+**Change**: Auto-detect agent from MCP `clientInfo` during `initialize` handshake instead of hardcoding `"claude-code"`. The protocol already exchanges this — we just need to read it.
+
+**No agent-specific logic in the MCP server.** Hooks are agent-specific by nature (different agents have different hook systems), but the MCP server stays generic.
+
+### 4. Token-Budgeted Context with Blended Scoring
+
+**Status**: Confirmed with modifications ✓
+
+- Token budget (default 800) — implemented Sprint 1
+- Blended scoring: quality * 0.6 + recency * 0.4 — implemented Sprint 2
+- Tiered output: top 3 detailed (facts-first), rest title-only — implemented Sprint 1
+
+**Changes**:
+- Make `token_budget` and quality/recency weights **configurable** in settings.json
+- Cap pinned observations at **5 detailed** in context injection (prevent pinned observations consuming entire budget)
+- CWD-aware boosting deferred to Phase 4 (dogfood)
+
+### 5. Knowledge Supersession
+
+**Status**: Confirmed with modification ✓
+
+- `superseded_by` column, manual agent-initiated supersession — implemented Sprint 2
+
+**Change**: Allow supersession **chains** (depth 2-3) instead of blocking already-superseded observations. When superseding, resolve to the current chain head automatically:
+```
+Agent says: "supersede #1"
+#1 already superseded by #2
+→ System resolves: supersede #2 (the chain head) instead
+→ All previous versions (#1, #2) remain archived
+```
+
+**Change**: Add supersession events to the **sync outbox** — lifecycle changes (not just new observations) need to propagate to other devices.
+
+**Keep manual**: Automatic supersession (detecting same topic) is too risky — title similarity could accidentally supersede unrelated observations. Revisit in Phase 4.
+
+### 6. Bidirectional Sync Protocol
+
+**Status**: Design confirmed ✓
+
+See detailed design below.
+
+---
+
+## Push Protocol (Local → Vector)
+
+Already designed in PLAN.md Phase 3.2. No changes.
+
+```
+1. Observation saved to local SQLite
+2. Added to sync_outbox (status: pending)
+3. Background timer (every 30s): flush pending outbox items
+4. POST /v1/ingest with observation data + metadata
+5. On success: mark outbox item as synced
+6. On failure: exponential backoff (30s, 60s, 120s, max 5min)
+7. Source ID: {user_id}-{device_id}-obs-{local_id}
+```
+
+**New**: Also push lifecycle changes (supersession, archival) to a sync change feed endpoint, not just new observations.
+
+---
+
+## Pull Protocol (Vector → Local)
+
+### Change Feed Approach
+
+**Do NOT use `/v1/search` for pull.** Search is "find relevant documents for a query." Pull is "give me all changes since cursor X." These are fundamentally different.
+
+**Endpoint**: `GET /v1/sync/changes?cursor=X&namespace=Y&limit=1000`
+
+Returns a stream of change events:
+```json
+{
+  "changes": [
+    {
+      "cursor": "2026-03-10T16:00:00.000Z",
+      "source_id": "david-macbook-obs-42",
+      "action": "create",
+      "payload": { /* full observation data as JSON */ }
+    },
+    {
+      "cursor": "2026-03-10T16:05:00.000Z",
+      "source_id": "david-macbook-obs-42",
+      "action": "supersede",
+      "superseded_by_source_id": "david-macbook-obs-55"
+    }
+  ],
+  "next_cursor": "2026-03-10T16:05:00.000Z",
+  "has_more": false
+}
+```
+
+### Critical: Server-Side Cursor, Not Client Timestamps
+
+**Do NOT use client-side `created_at_epoch` as the high-water-mark.** Clock skew between devices means observations can arrive at Vector with misleading timestamps.
+
+Use **server-assigned ingestion timestamp** (`cursor`) as the ordering key. This is monotonic and eliminates clock skew issues.
+
+### Pull Flow
+
+```
+1. On startup: read last_pull_cursor from sync_state table
+2. GET /v1/sync/changes?cursor=last_pull_cursor&namespace=N&limit=1000
+3. For each change:
+   a. Parse source_id → extract user_id, device_id, local_id
+   b. Skip if source_id matches current device (it's our own data)
+   c. If action=create:
+      - Check remote_observations table for existing source_id
+      - If exists: update (in case of re-push after edit)
+      - If new: insert into observations + FTS5 index
+      - Record mapping in remote_observations table
+   d. If action=supersede:
+      - Look up local_id for superseded source_id
+      - Apply supersession locally (archive, remove from FTS)
+   e. If action=delete:
+      - Look up and remove locally
+4. Update last_pull_cursor in sync_state
+5. If has_more: repeat from step 2 (pagination)
+```
+
+### Pull Frequency
+
+- **On startup**: Paginated backfill (1000 at a time, non-blocking)
+- **Periodic**: Every 60 seconds (less frequent than push at 30s — pull is less latency-sensitive)
+- **On-demand**: Not needed initially
+
+### First Install Backfill
+
+New device joining a team should NOT slam all observations into SQLite at once. Instead:
+- Pull in pages of 1000, oldest first
+- Update cursor after each page
+- First session can start immediately with whatever has been pulled so far
+- Background continues pulling remaining pages
+
+### Pull Scope
+
+Pull **ALL projects** for the user's namespace, not just the current CWD project. Reason: switching directories shouldn't require a re-pull. The data is small and lifecycle management keeps it bounded.
+
+---
+
+## New Schema: Remote Observations Mapping
+
+Migration v3:
+
+```sql
+CREATE TABLE remote_observations (
+    source_id   TEXT PRIMARY KEY,     -- {user_id}-{device_id}-obs-{local_id}
+    local_id    INTEGER NOT NULL,     -- local observations.id
+    pulled_at_epoch INTEGER NOT NULL  -- when we pulled it
+);
+
+CREATE INDEX idx_remote_obs_local ON remote_observations(local_id);
+```
+
+This maps between remote identity (source_id) and local autoincrement ID. Needed for:
+- Dedup on pull (don't re-insert what we already have)
+- Applying remote supersession (look up local_id from source_id)
+- Identifying which observations are "ours" vs "theirs"
+
+---
+
+## Gotchas & Mitigations
+
+### 1. Clock Skew Between Devices
+
+**Risk**: Two machines with different system clocks produce misleading `created_at_epoch` values.
+**Mitigation**: Sync protocol uses server-assigned timestamps for ordering, never client timestamps. Client `created_at_epoch` is preserved for display only.
+
+### 2. SQLite Concurrent Access (Hooks vs MCP Server)
+
+**Risk**: Hook processes open separate `MemDatabase` instances. If the MCP server is simultaneously writing, lock contention can occur.
+**Mitigation**: WAL mode allows concurrent reads. For writes, SQLite's built-in locking handles this — hooks are short-lived and writes are fast. Monitor during dogfood. If contention becomes an issue, have hooks communicate via a local queue file instead of opening the DB directly.
+
+### 3. Storage Growth from Full Pull
+
+**Risk**: 5-person team × 100 obs/day × 3 months = 45K observations cached locally (90-225MB).
+**Mitigation**: Lifecycle management (aging → archival → purge) bounds growth. Compacted observations are much smaller. Consider pulling **metadata-only** for team observations (not your own), with lazy detail fetch via `get_observations` — would cut pull bandwidth and storage by ~80%.
+
+### 4. FTS5 External Content Mode
+
+**Risk**: If observations are updated (not just superseded), FTS5 index must be manually updated.
+**Mitigation**: Current code handles insert and delete. Add update logic when we implement "merge facts into existing observation" (the dedup merge path).
+
+### 5. Schema Evolution vs Vector Metadata
+
+**Risk**: New SQLite columns (e.g., `superseded_by`) may not exist in Vector metadata.
+**Mitigation**: Never rely on Vector metadata for anything except search filtering. The sync metadata layer stores the full observation as JSON — that's the pull source, not Vector's chunked content.
+
+### 6. Secret Observations
+
+**Risk**: `sensitivity: "secret"` observations are never synced but stored in plaintext locally.
+**Mitigation**: Acceptable for Phase 3. Consider encrypting narrative/facts for secret observations in Phase 5 (public launch).
+
+### 7. Disaster Recovery
+
+**Risk**: If Vector loses data, no mechanism to reconstruct.
+**Mitigation**: Add `engrm sync --full` CLI command that re-pushes all local observations. Since each device has its own data, all devices running `--full` reconstructs the complete dataset.
+
+---
+
+## Implementation Order
+
+| Step | Description | Blocks |
+|------|-------------|--------|
+| 1 | Auto-detect agent from MCP `clientInfo` | Nothing |
+| 2 | Make token_budget + scoring weights configurable | Nothing |
+| 3 | Fix supersession chains (resolve to head) | Step 5 |
+| 4 | Build Candengo Vector REST client (`src/sync/client.ts`) | Step 5, 7 |
+| 5 | Implement push engine (outbox flush + lifecycle events) | Step 7 |
+| 6 | Design sync metadata endpoint on Candengo backend | Step 7 |
+| 7 | Implement pull engine (cursor-based, paginated) | Nothing |
+| 8 | Add `remote_observations` mapping table (migration v3) | Step 7 |
+| 9 | Add `engrm sync --full` disaster recovery | Nothing |
+| 10 | Multi-agent `init` CLI (Claude Code, Codex, Cursor) | Nothing |
+
+Steps 1-3 can be done immediately (no server-side dependencies).
+Steps 4-8 are Phase 3 core (require Candengo Vector API prep).
+Steps 9-10 are Phase 3.5 (polish).
+
+---
+
+## Multi-Agent Compatibility
+
+| Agent | Config Format | Config Location | Hook Support | Status |
+|-------|--------------|-----------------|-------------|--------|
+| Claude Code | JSON | `.mcp.json` | PostToolUse, Stop, SessionStart | Live ✓ |
+| Codex CLI | TOML | `~/.codex/config.toml` | TBD (different hook system) | MCP verified ✓ |
+| Cursor | UI Settings | Cursor settings | None (MCP tools only) | Compatible |
+| Windsurf | UI Settings | Windsurf settings | None (MCP tools only) | Compatible |
+| Cline | JSON | VS Code settings | None (MCP tools only) | Compatible |
+| VS Code Copilot | JSON | VS Code settings | None (MCP tools only) | Compatible |
+| Zed | JSON | Zed settings | None (MCP tools only) | Compatible |
+
+**Key insight**: Agents without hook support can still use all MCP tools manually. The agent calls `save_observation` explicitly instead of auto-capturing via hooks. Context injection works via `session_context` tool call (most agents support "tool use on session start" or similar).