engrm 0.1.0

package/SERVER-API-PLAN.md

@@ -0,0 +1,553 @@
+# Server-Side API Plan — Engrm
+
+**Status**: Approved (Devstral review: 2026-03-11)
+**Target**: Candengo Vector server (`/Volumes/Data/devs/candengo-vector/`)
+**Stack**: FastAPI + Python, Qdrant (embedded), SQLite, Stripe
+
+---
+
+## 1. Protocol Fixes (Pre-Blockers)
+
+### 1.1 Bearer Auth Support
+Add `Authorization: Bearer <cvk_key>` support to `app/auth.py` `get_auth_context()`.
+- Extract token from Bearer header alongside existing `X-Api-Key`
+- Validate `cvk_` prefix before DB lookup
+- Keep both headers supported — existing clients use `X-Api-Key`
+
+### 1.2 Client Search Params
+Fix client-side (`engrm/src/sync/client.ts`) to use `query`/`limit` instead of `q`/`top_k`.
+Server schema is already deployed — don't add aliases.
+
+### 1.3 Bulk Delete
+**`POST /v1/documents/delete-batch`**
+- Body: `{ source_ids: string[], namespace?: string }`
+- Response: `{ deleted: int }`
+- Iterates source_ids, calls `vector_store.delete_by_source()` for each
+- Auth: requires `ingest` scope
+
+---
+
+## 2. Sync Engine
+
+### 2.1 Sync Events Table
+New table in `/data/mem.db` (or accounts.db):
+
+```sql
+CREATE TABLE sync_events (
+    id          INTEGER PRIMARY KEY AUTOINCREMENT,
+    namespace   TEXT NOT NULL,
+    source_id   TEXT NOT NULL,
+    action      TEXT NOT NULL,        -- create | update | delete
+    payload     TEXT,                 -- JSON observation on create/update, null on delete
+    device_id   TEXT,                 -- originating device (for echo filtering)
+    ingested_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
+);
+
+CREATE INDEX idx_sync_events_ns_cursor ON sync_events(namespace, id);
+```
+
+Populated on every `POST /v1/ingest` and `POST /v1/documents/delete-batch` for Mem source types.
+
+### 2.2 Change Feed
+**`GET /v1/sync/changes`**
+- Query params: `cursor` (integer, last seen event ID), `namespace`, `limit` (default 50, max 1000)
+- Response: `{ changes: [...], cursor: "new_id", has_more: bool }`
+- Query: `WHERE namespace = ? AND id > ? ORDER BY id LIMIT ?`
+- Namespace scoped via API key auth
+- Server-side device_id echo filter (bandwidth optimisation, not correctness)
+- Auth: requires `search` scope
+
+### 2.3 Client Fix: Pull Loop
+Fix `engrm/src/sync/pull.ts` to loop on `has_more: true` for paginated backfill.
+
+---
+
+## 3. Namespace Model
+
+- **Single namespace per team** — all members share it
+- Solo users get personal namespace at signup (existing behaviour)
+- `sensitivity: "shared"` → synced, visible to all team members
+- `sensitivity: "personal"` → synced, server filters so only owner sees in search
+- `sensitivity: "secret"` → never synced, local SQLite only
+
+### 3.1 Personal Visibility Enforcement (Security Blocker)
+In `/v1/search`: when results come from a team namespace, inject server-side filter:
+`(sensitivity != 'personal' OR metadata.user_id == <authed_user_id>)`
+
+This is enforced in the search router based on the authenticated API key's account,
+not by trusting client-side metadata filters.
+
+---
+
+## 4. Mem Provisioning
+
+### 4.1 Provision Endpoint
+**`POST /v1/mem/provision`**
+- Body: `{ code?: string, token?: string }` — exactly one required
+  - `code` = OAuth callback code (Flow A)
+  - `token` = provisioning token `cmt_...` (Flow C)
+- Response: `{ api_key: "cvk_...", site_id, namespace, endpoint, user_id, teams: [] }`
+- Server: validates code/token, creates or looks up account, creates namespace-scoped API key
+
+### 4.2 Device Code Flow (RFC 8628)
+**`POST /v1/auth/device/code`**
+- Response: `{ device_code, user_code: "XXXX-YYYY", verification_uri, interval: 5 }`
+- Stores in `mem_device_codes` table with 15-minute expiry
+
+**`POST /v1/auth/device/token`**
+- Body: `{ device_code: "..." }`
+- Response: `{ status: "pending" }` or `{ status: "complete", api_key: "cvk_...", ... }`
+- Client polls at `interval` seconds, backs off on `slow_down`
+
+### 4.3 Token Revocation
+**`POST /v1/auth/revoke`**
+- Body: `{ api_key_prefix: "cvk_abc..." }` (first 12 chars for identification)
+- Revokes the matching API key
+- Auth: session cookie or service secret
+
+---
+
+## 5. Team Management
+
+### 5.1 Create Team
+**`POST /v1/mem/teams`**
+- Body: `{ name: "My Team" }`
+- Creates namespace, team record, adds creator as owner
+- Response: `{ id, name, namespace_id, role: "owner" }`
+
+### 5.2 List Teams
+**`GET /v1/mem/teams`**
+- Response: `[{ id, name, namespace_id, role, member_count }]`
+
+### 5.3 Team Detail
+**`GET /v1/mem/teams/{id}`**
+- Response: `{ id, name, namespace_id, members: [{ user_id, email, role, joined_at }] }`
+
+### 5.4 Invite
+**`POST /v1/mem/teams/{id}/invite`**
+- Body: `{ email?: string, role: "member"|"admin" }`
+- Response: `{ invite_code, invite_url, expires_at }`
+- Token-based (table-backed, revocable, single-use by default)
+- Owner/admin only
+
+### 5.5 Join Team
+**`POST /v1/mem/teams/{id}/join`**
+- Body: `{ invite_code: "..." }`
+- Grants existing API key access to team namespace (no new key)
+- Response: `{ team_id, namespace_id }`
+- Validates seat limit against billing plan
+
+### 5.6 Remove Member
+**`DELETE /v1/mem/teams/{id}/members/{user_id}`**
+- Query: `?action=keep|delete` (default: keep)
+- `keep` = revoke namespace access, set `revoked_at`, observations stay
+- `delete` = revoke access + async job to delete observations by `metadata.user_id`
+- Owner/admin only, can't remove last owner
+- Returns 202 for `action=delete` (async), 200 for `action=keep`
+
+### 5.7 Voluntary Leave
+**`DELETE /v1/mem/teams/{id}/members/me`**
+- Same as remove but self-targeted, no admin required
+- Can't leave if last owner
+
+### 5.8 Update Team
+**`PATCH /v1/mem/teams/{id}`**
+- Body: `{ name?: string }`
+- Owner/admin only
+- Low priority
+
+---
+
+## 6. Billing & Access Control
+
+### 6.1 Mem Plan Limits
+
+```python
+MEM_PLAN_LIMITS = {
+    "free":       {"max_observations": 10_000,  "max_devices": 2, "max_users": 1},
+    "solo":       {"max_observations": 50_000,  "max_devices": None, "max_users": 1},
+    "pro":        {"max_observations": None,     "max_devices": None, "max_users": 1},
+    "team":       {"max_observations": None,     "max_devices": None, "max_users": None, "min_seats": 3},
+    "enterprise": {"max_observations": None,     "max_devices": None, "max_users": None},
+}
+```
+
+### 6.2 Billing Endpoints
+
+**`GET /v1/mem/plans`** (public, no auth)
+- Returns plans with limits, pricing, currency, billing_period
+
+**`POST /v1/mem/billing/checkout`**
+- Body: `{ plan: "solo"|"pro"|"team", seats?: int }`
+- Validates: team requires seats >= 3
+- Creates Stripe checkout with `metadata.product = "mem"`
+- Idempotency key: `{account_id}-{plan}-{timestamp_minute}`
+- Response: `{ url }`
+
+**`GET /v1/mem/billing`**
+- Response:
+```json
+{
+  "plan": "free",
+  "usage": { "observations": 8200, "devices": 2, "users": 1 },
+  "limits": { "max_observations": 10000, "max_devices": 2, "max_users": 1 },
+  "usage_pct": { "observations": 82, "devices": 100, "users": 100 },
+  "warn_at_pct": 80,
+  "upgrade_available": true,
+  "suggested_plan": "solo",
+  "upgrade_url": "https://engrm.dev/upgrade",
+  "current_period_end": "2026-04-11T00:00:00Z",
+  "trial_ends_at": null
+}
+```
+
+**`POST /v1/mem/billing/add-seats`**
+- Body: `{ count: int }`
+- Validates subscription is active/trialing
+- Response: `{ total_seats, price_per_seat, new_monthly_total }`
+
+**`POST /v1/mem/billing/remove-seats`**
+- Body: `{ count: int }`
+- Validates: can't go below 3, can't go below active member count
+- 409 Conflict if validation fails
+- Response: `{ total_seats, price_per_seat, new_monthly_total }`
+
+**`GET /v1/mem/billing/portal`**
+- Response: `{ url }` — Stripe customer portal, return_url → Mem dashboard
+
+**`DELETE /v1/mem/billing`**
+- Cancels subscription (cancel_at_period_end = true by default)
+- Query: `?immediate=true` for immediate cancellation
+- Response: `{ cancelled: true, access_until: "2026-04-11" }`
+
+**`POST /v1/mem/billing/reactivate`**
+- Un-cancel before period end
+- Response: `{ reactivated: true }`
+
+**`GET /v1/mem/billing/invoices`**
+- Query: `?limit=10`
+- Response: `[{ id, amount, currency, status, date }]`
+
+### 6.3 Enforcement Points
+
+| Limit | Endpoint | Status | Response |
+|---|---|---|---|
+| Observation count | `POST /v1/ingest` | 402 | `{ error, limit, current, suggested_plan, upgrade_url }` |
+| Device count | `POST /v1/mem/provision` | 402 | `{ error, limit, current }` |
+| User/seat count | `POST /v1/mem/teams/{id}/join` | 402 | `{ error, seats_used, seats_total }` |
+| Seat minimum | `POST /v1/mem/billing/remove-seats` | 409 | `{ error, min_seats: 3 }` |
+
+Server-side observation count is authoritative (Qdrant count by namespace).
+Client-side count is an approximation for UX only.
+
+### 6.4 Downgrade Handling
+- Allow downgrade even if over new limit
+- Block new syncs (402) until under limit
+- Existing observations untouched
+- Lifecycle (aging → archival) naturally reduces count
+- Pinned observations still count toward quota
+
+### 6.5 Cancellation Handling
+- Stripe `customer.subscription.deleted` → downgrade to free
+- Existing data preserved, sync blocked above free limit
+
+---
+
+## 7. Usage & Activity
+
+### 7.1 User Usage
+**`GET /v1/mem/usage`**
+- Response: `{ observations_total, observations_this_month, syncs_this_month, by_project: [...] }`
+- Derived from usage_meter counters (not Qdrant aggregation)
+
+### 7.2 Team Usage
+**`GET /v1/mem/teams/{id}/usage`**
+- Response: `{ total_observations, by_member: [{ user_id, count }], by_project: [{ canonical, count }] }`
+
+### 7.3 Activity Heatmap (GitHub-style graph)
+**`GET /v1/mem/activity`**
+- Query: `?days=365&team_id=optional`
+- Response: `{ days: [{ date, count, by_type: { bugfix: 2, discovery: 3 } }] }`
+- Aggregated from usage_meter counter table (increment on ingest), NOT Qdrant scroll
+
+### 7.4 Usage Counter Table
+
+```sql
+CREATE TABLE mem_activity (
+    id          INTEGER PRIMARY KEY AUTOINCREMENT,
+    namespace   TEXT NOT NULL,
+    user_id     TEXT NOT NULL,
+    date        TEXT NOT NULL,       -- YYYY-MM-DD
+    obs_type    TEXT NOT NULL,       -- bugfix, discovery, etc.
+    count       INTEGER DEFAULT 0,
+    UNIQUE(namespace, user_id, date, obs_type)
+);
+
+CREATE INDEX idx_mem_activity_ns_date ON mem_activity(namespace, date);
+```
+
+Incremented on each `POST /v1/ingest` for Mem source types.
+Heatmap query: `SELECT date, obs_type, SUM(count) FROM mem_activity WHERE namespace=? AND date>=? GROUP BY date, obs_type`
+
+---
+
+## 8. Data Management
+
+### 8.1 Export (Synchronous Streaming)
+**`POST /v1/mem/export`**
+- Body: `{ scope: "user"|"team"|"project", team_id?, project_canonical?, format: "jsonl" }`
+- Response: streamed `application/x-ndjson` (chunked transfer encoding)
+- Synchronous — no job queue needed for v1 data sizes
+
+### 8.2 Purge
+**`POST /v1/mem/purge`**
+- Body: `{ namespace, before_epoch, dry_run: bool, confirm: "DELETE" }`
+- `dry_run: true` → `{ would_delete: int }`
+- `dry_run: false` + `confirm: "DELETE"` → executes, `{ deleted: int }`
+- Admin scope required
+- Rate limited: 1 per namespace per 24 hours
+- Logged to audit table
+
+### 8.3 On-Demand Compaction
+**`POST /v1/mem/lifecycle/compact`**
+- Triggers server-side compaction for caller's namespace
+- Rate limited: once per 24 hours
+- Helps users actively manage quota when near limit
+
+### 8.4 Device Management
+**`POST /v1/mem/devices/{device_id}/revoke`**
+- Marks device as revoked in `mem_devices`, frees slot
+- Auth: account owner
+
+### 8.5 Import (Deferred to post-v1)
+
+---
+
+## 9. Database Schema
+
+All Mem tables in `accounts.db` (same DB for foreign key integrity).
+
+```sql
+-- Mem subscriptions (separate from Vector billing)
+CREATE TABLE mem_subscriptions (
+    account_id              TEXT PRIMARY KEY REFERENCES accounts(id),
+    plan                    TEXT NOT NULL DEFAULT 'free',
+    seats                   INTEGER DEFAULT 1,
+    stripe_subscription_id  TEXT,
+    stripe_customer_id      TEXT,
+    status                  TEXT DEFAULT 'active',
+    trial_ends_at           TEXT,
+    cancel_at_period_end    INTEGER DEFAULT 0,
+    created_at              TEXT NOT NULL,
+    updated_at              TEXT NOT NULL
+);
+
+-- Team management
+CREATE TABLE mem_teams (
+    id              TEXT PRIMARY KEY,
+    name            TEXT NOT NULL,
+    namespace_id    TEXT NOT NULL,
+    created_by      TEXT NOT NULL REFERENCES accounts(id),
+    created_at      TEXT NOT NULL
+);
+
+CREATE TABLE mem_team_members (
+    team_id     TEXT NOT NULL REFERENCES mem_teams(id),
+    account_id  TEXT NOT NULL REFERENCES accounts(id),
+    role        TEXT NOT NULL DEFAULT 'member',   -- owner | admin | member
+    joined_at   TEXT NOT NULL,
+    revoked_at  TEXT,                             -- set on removal with action=keep
+    PRIMARY KEY (team_id, account_id)
+);
+
+CREATE TABLE mem_invites (
+    id          TEXT PRIMARY KEY,
+    team_id     TEXT NOT NULL REFERENCES mem_teams(id),
+    invite_code TEXT UNIQUE NOT NULL,
+    email       TEXT,
+    role        TEXT NOT NULL DEFAULT 'member',
+    max_uses    INTEGER DEFAULT 1,                -- NULL = unlimited
+    use_count   INTEGER DEFAULT 0,
+    created_by  TEXT NOT NULL,
+    created_at  TEXT NOT NULL,
+    expires_at  TEXT NOT NULL,
+    used_by     TEXT,
+    used_at     TEXT
+);
+
+CREATE INDEX idx_invites_code ON mem_invites(invite_code);
+
+-- Device tracking (for limit enforcement)
+CREATE TABLE mem_devices (
+    id              TEXT PRIMARY KEY,
+    account_id      TEXT NOT NULL,
+    device_id       TEXT NOT NULL,
+    device_name     TEXT,
+    status          TEXT DEFAULT 'active',         -- active | revoked
+    registered_at   TEXT NOT NULL,
+    last_seen_at    TEXT NOT NULL,
+    UNIQUE(account_id, device_id)
+);
+
+-- Device code flow (RFC 8628)
+CREATE TABLE mem_device_codes (
+    device_code     TEXT PRIMARY KEY,
+    user_code       TEXT UNIQUE NOT NULL,
+    account_id      TEXT,                          -- NULL until approved
+    status          TEXT NOT NULL DEFAULT 'pending', -- pending | approved | expired
+    poll_interval   INTEGER DEFAULT 5,
+    created_at      TEXT NOT NULL,
+    expires_at      TEXT NOT NULL,
+    approved_at     TEXT
+);
+
+-- Provisioning tokens
+CREATE TABLE mem_provision_tokens (
+    token           TEXT PRIMARY KEY,              -- cmt_...
+    account_id      TEXT NOT NULL REFERENCES accounts(id),
+    created_at      TEXT NOT NULL,
+    expires_at      TEXT NOT NULL,
+    used_at         TEXT
+);
+
+-- Sync change feed
+CREATE TABLE sync_events (
+    id              INTEGER PRIMARY KEY AUTOINCREMENT,
+    namespace       TEXT NOT NULL,
+    source_id       TEXT NOT NULL,
+    action          TEXT NOT NULL,                  -- create | update | delete
+    payload         TEXT,                           -- JSON on create/update, null on delete
+    device_id       TEXT,
+    ingested_at     TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
+);
+
+CREATE INDEX idx_sync_events_ns_cursor ON sync_events(namespace, id);
+
+-- Activity heatmap counters
+CREATE TABLE mem_activity (
+    id              INTEGER PRIMARY KEY AUTOINCREMENT,
+    namespace       TEXT NOT NULL,
+    user_id         TEXT NOT NULL,
+    date            TEXT NOT NULL,                  -- YYYY-MM-DD
+    obs_type        TEXT NOT NULL,
+    count           INTEGER DEFAULT 0,
+    UNIQUE(namespace, user_id, date, obs_type)
+);
+
+CREATE INDEX idx_mem_activity_ns_date ON mem_activity(namespace, date);
+
+-- Stripe webhook idempotency
+CREATE TABLE stripe_events_processed (
+    event_id        TEXT PRIMARY KEY,
+    processed_at    TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
+);
+```
+
+---
+
+## 10. Stripe Integration
+
+### 10.1 Separate Products
+Create distinct Stripe Products for Mem (not reuse Vector products):
+- Engrm Solo ($9/mo)
+- Engrm Pro ($19/mo)
+- Engrm Team ($12/seat/mo)
+
+All with `metadata.product = "mem"`.
+
+### 10.2 Webhook Routing
+Separate endpoint: `POST /webhooks/stripe/mem`
+- Own signing secret
+- Idempotent (check `stripe_events_processed` table)
+- Events: `checkout.session.completed`, `customer.subscription.updated`,
+  `customer.subscription.deleted`, `invoice.payment_succeeded`, `invoice.payment_failed`
+
+### 10.3 Checkout Metadata
+Always include in subscription_data.metadata:
+```python
+{
+    "product": "mem",
+    "plan": plan,
+    "account_id": account_id,
+    "seats": seats
+}
+```
+
+---
+
+## 11. Security Requirements
+
+1. **Personal visibility**: Server-side filter on `/v1/search` — `(sensitivity != 'personal' OR user_id == authed_user)`
+2. **Quota authority**: Server-side Qdrant count is authoritative, not client SQLite count
+3. **Clock manipulation**: Use server `ingested_at` for lifecycle, not client `created_at_epoch`
+4. **Rate limits**: 5 checkout sessions per account per hour
+5. **Idempotency keys**: On all Stripe checkout creation
+6. **Seat integrity**: `remove-seats` validates `remaining >= active_member_count`
+7. **Bearer prefix validation**: Reject non-`cvk_` tokens before DB lookup
+8. **Scrubber patterns**: Add `stripe_customer_id`, `stripe_subscription_id` to secret scrubber
+
+---
+
+## 12. Implementation Priority
+
+### Phase 3.0 — Sync Blockers
+1. Bearer auth support in `get_auth_context()`
+2. Client search param fix (`q`/`top_k` → `query`/`limit`)
+3. `sync_events` table + `GET /v1/sync/changes`
+4. `POST /v1/documents/delete-batch`
+5. `POST /v1/mem/provision`
+6. Personal visibility filter in `/v1/search`
+
+### Phase 3.1 — Auth Flows
+7. `POST /v1/auth/device/code` + `/v1/auth/device/token`
+8. `POST /v1/auth/revoke`
+
+### Phase 3.2 — Team Management
+9. Teams CRUD (create, list, detail)
+10. Invite + join
+11. Remove member + voluntary leave
+
+### Phase 3.3 — Billing
+12. `mem_subscriptions` table + `GET /v1/mem/plans`
+13. Checkout + billing status
+14. Add/remove seats
+15. Stripe webhook handler (`/webhooks/stripe/mem`)
+16. Quota enforcement on ingest + provision
+
+### Phase 3.4 — Observability
+17. `GET /v1/mem/usage` + `GET /v1/mem/teams/{id}/usage`
+18. `GET /v1/mem/activity` (heatmap)
+19. `mem_activity` counter table
+
+### Phase 3.5 — Data Management
+20. `POST /v1/mem/export` (streaming)
+21. `POST /v1/mem/purge` (with dry_run + confirm)
+22. `POST /v1/mem/lifecycle/compact`
+23. `POST /v1/mem/devices/{id}/revoke`
+
+### Deferred (post-v1)
+- Import endpoint
+- Dashboard graph visualization
+- Team rename
+- Annual billing
+- Enterprise SSO
+
+---
+
+## 13. File Structure (candengo-vector)
+
+```
+app/
+├── routers/
+│   ├── mem.py              # /v1/mem/* — teams, usage, activity, export, purge
+│   ├── mem_billing.py      # /v1/mem/billing/* — checkout, seats, portal
+│   ├── mem_provision.py    # /v1/mem/provision, /v1/auth/device/*
+│   └── sync.py             # /v1/sync/changes
+├── services/
+│   ├── mem_teams.py        # Team CRUD, invite, join, remove
+│   ├── mem_billing.py      # Plan limits, subscription management
+│   ├── mem_devices.py      # Device registration, tracking, revocation
+│   ├── mem_activity.py     # Activity counters, heatmap queries
+│   ├── mem_provision.py    # Provisioning token + device code logic
+│   └── sync_events.py      # Sync event log, change feed queries
+```