engrm 0.1.0

package/PLAN.md

@@ -0,0 +1,278 @@
+# Implementation Plan — Engrm
+
+## Approach
+
+**Internal tooling first.** We're building this so our dev team can share project context across machines and developers. The public product comes later — first it needs to work for us.
+
+Built from scratch. claude-mem is a reference for how to hook into Claude Code (hooks, MCP registration, observation capture patterns) but no code is shared. This avoids AGPL licensing issues and lets us design the architecture around cross-device team memory from the start.
+
+## Component Architecture
+
+```
+engrm/
+├── src/
+│   ├── server.ts           # MCP protocol handler (entry point)
+│   ├── tools/              # MCP tool implementations
+│   │   ├── search.ts       # search() — hybrid local + remote, project-scoped
+│   │   ├── timeline.ts     # timeline() — chronological context
+│   │   ├── get.ts          # get_observations() — fetch by ID
+│   │   ├── save.ts         # save_observation() — manual save with quality scoring
+│   │   └── pin.ts          # pin_observation() — prevent aging
+│   ├── capture/            # Observation extraction
+│   │   ├── extractor.ts    # Extract observations from tool use
+│   │   ├── scrubber.ts     # Secret/PII scrubbing
+│   │   ├── quality.ts      # Quality scoring (0.0-1.0)
+│   │   └── dedup.ts        # Near-duplicate detection (title similarity)
+│   ├── storage/            # Local storage layer
+│   │   ├── sqlite.ts       # SQLite database (source of truth)
+│   │   ├── migrations.ts   # Schema migrations
+│   │   ├── outbox.ts       # Sync outbox queue
+│   │   └── projects.ts     # Project identity (git remote → canonical ID)
+│   ├── lifecycle/           # Observation lifecycle management
+│   │   ├── aging.ts        # Daily: active → aging after 30 days
+│   │   ├── compaction.ts   # Weekly: aging → archived, generate digests
+│   │   └── purge.ts        # Monthly: delete archived > 12 months
+│   ├── sync/               # Remote sync layer
+│   │   ├── client.ts       # Candengo Vector REST client
+│   │   └── engine.ts       # Sync engine (outbox flush, backfill, archival cleanup)
+│   ├── context/            # Context injection
+│   │   └── inject.ts       # Session start context builder
+│   └── config.ts           # Configuration management
+│
+├── hooks/                  # Claude Code hooks
+│   ├── post-tool-use.sh    # Observation capture
+│   └── stop.sh             # Session summary + sync flush
+│
+├── package.json
+├── tsconfig.json
+├── BRIEF.md
+├── SPEC.md
+├── PLAN.md                 # This file
+└── CLAUDE.md
+```
+
+---
+
+## Phase 1: Local MCP Server + Provisioning (Weeks 1-2)
+
+**Goal**: Working MCP server with local SQLite storage, and a self-service provisioning flow so any developer can go from zero to working memory in under 2 minutes.
+
+### 1.1 MCP Server Core
+
+| Task | Description | Effort |
+|---|---|---|
+| Project scaffolding | TypeScript + Bun, MCP SDK, bun:sqlite | S |
+| SQLite schema + migrations | projects, observations, sessions, sync_outbox tables (see SPEC §1-2) | M |
+| Project identity detection | Auto-detect canonical project ID from git remote URL, normalise, store in projects table | M |
+| MCP tool: `save_observation` | Save to local SQLite with project FK, quality score, add to sync outbox | S |
+| MCP tool: `search` | Local SQLite FTS5 search, project-scoped by default, quality-weighted ranking | M |
+| MCP tool: `get_observations` | Fetch by IDs from local SQLite | S |
+| MCP tool: `timeline` | Chronological context around an observation | M |
+| MCP tool: `pin_observation` | Pin/unpin observations to prevent aging | XS |
+| Quality scoring | Score observations at capture time (0.0-1.0) based on type, content signals (see SPEC §2) | M |
+| Secret scrubber | Regex-based scrubbing of API keys, passwords, tokens before storage | M |
+| Relative file paths | Store file paths relative to project root, resolve at capture time | S |
+| Configuration | `~/.engrm/settings.json` — local paths, remote config | S |
+
+### 1.2 Self-Provisioning
+
+| Task | Description | Effort |
+|---|---|---|
+| Engrm landing page | `www.engrm.dev` — product page + signup + install instructions | M |
+| Account provisioning backend | Signup → create mem_accounts row, namespace, provision token | M |
+| Provision API endpoint | `POST /v1/mem/provision` — exchange token for permanent credentials | S |
+| `npx engrm init` | CLI command: redeem token, write settings, register MCP + hooks in Claude Code | M |
+| Team invite flow | Admin creates team → invite URL → member joins with team namespace pre-configured | M |
+| Self-hosted init path | `--url` flag for custom endpoints, `--manual` for air-gapped environments | S |
+
+### 1.3 Provisioning Flow
+
+```
+1. Developer visits www.engrm.dev
+2. Signs up (email or GitHub OAuth)
+3. Backend provisions account + namespace
+4. Page shows personalised install command:
+   npx engrm init --token=cmt_abc123...
+5. Developer runs command in terminal
+6. Plugin exchanges token → gets API key, endpoint, namespace
+7. Plugin writes settings.json, registers MCP server + hooks in Claude Code
+8. Next Claude Code session has memory
+```
+
+For teams: admin creates team at `www.engrm.dev/team`, shares invite link, team members get pre-configured for the shared namespace.
+
+**Deliverable**: A working MCP server that Claude Code can call, with self-service provisioning from candengo.com. Any developer can sign up and be running in under 2 minutes.
+
+---
+
+## Phase 2: Claude Code Hooks (Week 3)
+
+**Goal**: Automatic observation capture from Claude Code sessions.
+
+| Task | Description | Effort |
+|---|---|---|
+| PostToolUse hook | Shell script that extracts observations from tool results | L |
+| Stop hook | Session summary generation, sync flush | M |
+| MCP server registration | `.mcp.json` config for Claude Code | XS |
+| Hooks registration | `hooks.json` for Claude Code | XS |
+| Context injection | Inject relevant history on session start (via MCP tool call) | M |
+| Observation quality filtering | Skip trivial tool uses (ls, cat of small files), focus on meaningful work | M |
+
+**Deliverable**: Claude Code automatically captures observations as you work. Session summaries on exit. Relevant history injected on start.
+
+### Observation Extraction Design
+
+This is the hardest problem. What makes a good observation?
+
+**Capture triggers** (PostToolUse):
+- File edits → what changed and why
+- Command execution with errors → what failed and how it was fixed
+- Multiple file reads in sequence → likely investigating something
+- Test runs → pass/fail context
+
+**Skip** (low signal):
+- Simple file reads (single `cat`)
+- `ls`, `pwd`, `git status` and similar navigation
+- Repeated identical tool calls
+
+**Extraction approach**: The hook sends the tool name + result summary to the MCP server. The server decides whether it's worth capturing based on the tool type and content. Quality score is assigned at capture time. Observations are batched per-session and deduplicated (title similarity > 0.8 against last 24h → merge into existing).
+
+### Observation Lifecycle + Deduplication
+
+| Task | Description | Effort |
+|---|---|---|
+| Deduplication on save | Check title similarity against last 24h for same project, merge if > 0.8 | M |
+| Aging job | Daily: move active observations older than 30 days to aging (0.7x search weight) | S |
+| Archival + compaction | Weekly: observations > 90 days grouped by session, summarised into digest | L |
+| Purge job | Monthly: delete archived observations > 12 months (keep digests + pinned) | S |
+| FTS5 index maintenance | Remove archived observations from FTS5 index during compaction | S |
+| Quota check | Count active+aging observations for free tier enforcement | S |
+
+**Why this matters now**: Without lifecycle management, a developer generating ~100 observations/day hits 10K in ~3 months. Search results degrade as old, irrelevant observations pollute rankings. Compaction turns 25 old observations from a debugging session into one useful digest. Aging reduces the weight of stale knowledge. The free tier stays usable because only active+aging observations count toward the 10K limit — compacted observations are free.
+
+---
+
+## Phase 3: Cross-Device Sync + Team Memory (Weeks 4-6)
+
+**Goal**: Offline-first sync to Candengo Vector with team support from day one. Work on laptop, continue on desktop. Other developers' observations are searchable too.
+
+Team memory isn't a separate phase — it's the reason we're building this. User identity, attribution, and shared namespaces are built into the sync layer from the start.
+
+### 3.1 Candengo Vector API Prep
+
+| Task | Description | Effort |
+|---|---|---|
+| Metadata filtering on search API | `metadata_filters` param on `/v1/search` — filter by `project_canonical`, `user_id`, etc. | S |
+| Document listing by source_type | `GET /v1/documents?source_type=X` with pagination | S |
+| Document deletion by source_id | `DELETE /v1/documents/{source_id}` — needed for archival/compaction cleanup | S |
+| Device/user ID tracking in metadata | Accept `device_id`, `user_id` in metadata | XS |
+
+### 3.2 Sync Engine
+
+| Task | Description | Effort |
+|---|---|---|
+| Candengo Vector REST client | TypeScript HTTP client for `/v1/ingest`, `/v1/search`, `/v1/ingest/batch`, `/v1/documents/{id}` | M |
+| Fire-and-forget sync | On observation save → attempt immediate push | S |
+| Background sync timer | Every 30s → flush pending outbox items (batch of 50) | S |
+| Startup backfill | On boot → sync observations saved while offline (high-water-mark) | M |
+| Connectivity detection | Skip sync when offline, resume when connected | S |
+| Retry with exponential backoff | Failed syncs retry 30s, 60s, 120s, max 5min | S |
+| Observation → Candengo mapping | Map to ingest format with `project_canonical` in metadata, source_id = `{user}-{device}-obs-{id}` | M |
+| Archival sync | When compaction runs: delete archived source_ids from Vector, ingest digest | M |
+
+### 3.3 Team + Hybrid Search
+
+| Task | Description | Effort |
+|---|---|---|
+| Hybrid search orchestrator | Query local FTS5 + Candengo `/v1/search` in parallel, scoped by `project_canonical` | M |
+| Result merging + deduplication | Merge by source_id, weighted scoring (semantic × quality × lifecycle) | M |
+| Graceful degradation | Candengo unreachable → local-only search (transparent) | S |
+| Device ID generation | Auto-generate stable device ID on first run | XS |
+| User identity + attribution | `user_id` in all observations, "david/laptop" in results | S |
+| Source ID namespacing | `{user_id}-{device_id}-obs-{local_id}` prevents all collisions | S |
+| Visibility controls | `shared` / `personal` / `secret` flags | M |
+| Team search scope | Search own + team observations by default, filtered by `project_canonical` | M |
+| Cross-project search | Support `project: "*"` to search across all projects | S |
+
+**Deliverable**: Full cross-device team sync. Observations from any team member appear on any device within 30 seconds. Works offline, syncs when reconnected. Projects are matched across machines by git remote URL. New developer installs, connects to the shared namespace, and their agent has the full team knowledge base.
+
+### Backfill Strategy
+
+Instead of diffing all IDs on every startup (expensive at scale), use a high-water-mark:
+
+```
+1. Store last_synced_epoch locally
+2. On startup: SELECT * FROM observations WHERE created_at_epoch > last_synced_epoch
+3. Batch push missing observations
+4. Update last_synced_epoch
+```
+
+Simple, efficient, scales to any observation count.
+
+---
+
+## Phase 4: Dogfood (Weeks 7-8)
+
+**Goal**: Run it internally on our projects (Candengo, Alchemy, AIMY). Fix what hurts.
+
+| Task | Description | Effort |
+|---|---|---|
+| Team onboarding | Install on all dev machines, shared Candengo Vector namespace | S |
+| Observation quality tuning | Adjust capture filters based on real usage — too noisy? too quiet? | M |
+| Search relevance tuning | Adjust scoring weights based on real queries | M |
+| Bug fixes from dogfooding | Whatever breaks | M |
+| Automated testing | Unit tests, sync integration tests | L |
+| Performance benchmarking | <50ms local search, <200ms remote search | M |
+
+---
+
+## Phase 5: Public Launch (Weeks 9-10)
+
+| Task | Description | Effort |
+|---|---|---|
+| One-line installer | `npx engrm install` or similar | M |
+| CLI tool | `engrm status`, `search`, `sync` commands | M |
+| Documentation | Installation, configuration, usage guide | M |
+| GitHub repo (FSL-1.1-ALv2 license) | README, examples, contributing guide, LICENSE file | M |
+| Free tier limits enforcement | Observation count, device count checks against account tier | M |
+| Upgrade flow | In-plugin nudge when approaching limits, link to engrm.dev/upgrade | S |
+
+**Licensing**: Core client released under FSL-1.1-ALv2 (Functional Source License, Fair Source). Source-available — developers can read, modify, and self-host freely. The restriction: nobody can fork it and offer a competing hosted service. Each version converts to Apache 2.0 after 2 years. Sentinel (real-time AI audit) is proprietary, delivered from a separate private repo to paying customers only.
+
+---
+
+## Effort Key
+
+| Size | Estimated Effort | Description |
+|---|---|---|
+| XS | < 2 hours | Trivial change, config, or wrapper |
+| S | 2-4 hours | Straightforward implementation |
+| M | 4-8 hours | Moderate complexity, some design decisions |
+| L | 1-2 days | Significant feature, requires careful design |
+
+---
+
+## Dependencies & Critical Path
+
+```
+Phase 1 (Local MCP) ──→ Phase 2 (Hooks) ──→ Phase 3 (Sync + Team) ──→ Phase 4 (Dogfood) ──→ Phase 5 (Launch)
+```
+
+**Phase 1+2 are usable standalone** — local-only memory is already valuable.
+**Phase 3 is the whole point** — cross-device team sync is why we're building this.
+**Phase 4 is essential** — dogfooding on our own projects before releasing externally.
+
+---
+
+## Risk Register
+
+| Risk | Impact | Likelihood | Mitigation |
+|---|---|---|---|
+| Observation quality too noisy | High | Medium | Quality scoring (0.0-1.0), skip below 0.1, deduplication on save, compaction at 90 days |
+| Observation volume exceeds quota | Medium | High | Lifecycle management: aging → archival → purge. Compaction summarises old sessions into digests. Only active+aging counts toward quota |
+| Project identity mismatch across machines | High | Medium | Canonical ID from normalised git remote URL. Fallback: `.engrm.json` in project root |
+| Search relevance degrades over time | High | Medium | Quality-weighted ranking, lifecycle scoring (aging=0.7x), project scoping, compaction removes noise |
+| Source ID collisions across devices | Medium | High | Source ID = `{user_id}-{device_id}-obs-{local_id}` — unique across all dimensions |
+| MCP protocol breaking changes | High | Low | Pin MCP SDK version, abstract protocol layer |
+| Secret leakage in observations | Critical | Medium | Multi-layer scrubbing, sensitivity classification, relative file paths only |
+| Sync conflicts | Medium | Low | Source ID namespacing — structurally impossible for two users to overwrite each other |

package/README.md

@@ -0,0 +1,121 @@
+# Engrm
+
+**Cross-device memory for AI coding agents** — powered by [Candengo Vector](https://www.candengo.com).
+
+Engrm captures observations from AI-assisted coding sessions (discoveries, bugfixes, decisions, patterns) and syncs them to Candengo Vector so they're available on any machine, in any future session.
+
+Built from scratch. Inspired by [claude-mem](https://github.com/thedotmack/claude-mem)'s approach to hooking into Claude Code — but designed around cross-device sync from day one, with Candengo Vector as the backend.
+
+---
+
+## The Problem
+
+Our dev team works across multiple machines and projects (Candengo, Alchemy, AIMY). Every Claude Code session starts from zero — no memory of what was done yesterday, on another machine, or by another developer. The agent re-discovers the same patterns, hits the same gotchas, and asks the same questions that were already answered in a different session.
+
+## The Solution
+
+An MCP server + Claude Code hooks that:
+
+1. **Capture observations automatically** from coding sessions via Claude Code hooks (`PostToolUse`, `Stop`)
+2. **Store locally** in SQLite (offline-first, always available)
+3. **Sync to Candengo Vector** for cross-device, cross-developer search (BGE-M3 hybrid dense+sparse, cross-encoder reranking)
+4. **Inject relevant context** on session start — so the agent picks up where you (or a teammate) left off, on any machine
+
+---
+
+## Architecture
+
+```
+Developer's Machine (any device)
++---------------------------------------------+
+|  Claude Code                                 |
+|       | MCP (stdio)                          |
+|  +-----------------------------------+       |
+|  |  Engrm MCP Server                 |       |
+|  |  - Observation capture            |       |
+|  |  - Local SQLite + FTS5            |       |
+|  |  - Sync outbox queue              |       |
+|  |  - Secret scrubbing               |       |
+|  +--------------+--------------------+       |
+|                 | HTTPS (when available)      |
++-----------------+----------------------------+
+                  |
+                  v
++---------------------------------------------+
+|  Candengo Vector (self-hosted or cloud)      |
+|  - BGE-M3 hybrid dense+sparse search        |
+|  - Cross-encoder reranking                   |
+|  - Multi-tenant (site_id / namespace)        |
++---------------------------------------------+
+```
+
+**Key principle**: SQLite is the source of truth. Always available, always fast. Candengo Vector is the sync target that enables cross-device search. If the remote is unreachable, everything still works locally.
+
+---
+
+## Getting Started
+
+Sign up at [engrm.dev](https://engrm.dev), then run the install command shown on the page:
+
+```bash
+npx engrm init --token=cmt_your_token_here
+```
+
+That's it. The command provisions your account, writes config, registers the MCP server and hooks in Claude Code. Your next Claude Code session has memory.
+
+**For teams**: Admin creates a team at [engrm.dev/team](https://engrm.dev/team), shares the invite link. Team members sign up via the link and are pre-configured for the shared namespace.
+
+**Self-hosted**: Point at your own Candengo Vector instance:
+```bash
+npx engrm init --url=https://vector.internal.company.com --token=cmt_...
+```
+
+---
+
+## How It Hooks Into Claude Code
+
+Engrm integrates via two mechanisms:
+
+### MCP Server
+Registered in Claude Code's MCP config, exposes tools the agent can call:
+- `search` — find relevant observations from memory
+- `timeline` — chronological context around an observation
+- `get_observations` — fetch full details by ID
+- `save_observation` — manually save something worth remembering
+
+### Claude Code Hooks
+Shell scripts triggered by Claude Code lifecycle events:
+- `PostToolUse` — extract observations from tool results (the main capture path)
+- `Stop` — generate session summary, flush sync queue
+
+This is similar to how claude-mem hooks in, but our implementation is independent — no shared code, no fork relationship.
+
+---
+
+## Status
+
+**Building.** The product brief, technical spec, and implementation plan are complete. Implementation is starting with Phase 1: local MCP server with SQLite storage, observation capture, and secret scrubbing.
+
+---
+
+## Project Documents
+
+| Document | Contents |
+|---|---|
+| [BRIEF.md](BRIEF.md) | Product brief, target users, revenue model |
+| [SPEC.md](SPEC.md) | Technical specification: schemas, MCP tools, sync engine, search pipeline |
+| [PLAN.md](PLAN.md) | Phased implementation plan with effort estimates |
+| [SWOT.md](SWOT.md) | Strengths, weaknesses, opportunities, threats |
+| [COMPETITIVE.md](COMPETITIVE.md) | Competitive landscape analysis |
+| [MARKET.md](MARKET.md) | Market research, pricing, growth projections |
+| [INFRASTRUCTURE.md](INFRASTRUCTURE.md) | Scaling roadmap, capacity planning |
+
+---
+
+## License
+
+Functional Source License v1.1 (FSL-1.1-ALv2) — source-available, part of the [Fair Source](https://fair.io) movement. Free to use, modify, and self-host for any non-competing purpose. You cannot offer this as a competing hosted service. Each version converts to Apache 2.0 after 2 years. Sentinel (real-time AI audit) is a separate proprietary product. See [LICENSE](LICENSE) for details.
+
+---
+
+Built by [Unimpossible Consultants](https://unimpossible.com) — the team behind Candengo, Alchemy, and AIMY.