purecontext-mcp 1.1.1 → 1.1.2

package/docs/dev/PHASE12_TASKS.md

@@ -1,335 +0,0 @@
-# Phase 12 — Task Breakdown
-
-**Goal**: Add rate limiting and multi-tenant authentication for hosted/shared PureContext deployments. This enables running PureContext as a service for teams or organizations.
-
-**Scope rationale**: When PureContext is deployed as a shared service (HTTP/SSE transport), it needs protection against abuse and support for multiple users/organizations with isolated data. This phase transforms PureContext from a single-user CLI tool into a multi-tenant service.
-
-**Approach**: Tasks build from basic rate limiting through API key authentication to full tenant isolation. Each layer is independent and provides value.
-
----
-
-## Task 96: Token Bucket Rate Limiter
-
-Implement a token bucket rate limiter for controlling request throughput.
-
-**Deliverables:**
-
-- `src/server/rate-limiter.ts`
-  - `TokenBucketLimiter` class:
-    ```typescript
-    class TokenBucketLimiter {
-      constructor(options: LimiterOptions);
-      tryConsume(key: string, tokens?: number): LimitResult;
-      getRemainingTokens(key: string): number;
-      reset(key: string): void;
-    }
-
-    interface LimiterOptions {
-      maxTokens: number;      // Bucket capacity (default: 100)
-      refillRate: number;     // Tokens per second (default: 10)
-      refillInterval: number; // Ms between refills (default: 100)
-    }
-
-    interface LimitResult {
-      allowed: boolean;
-      remainingTokens: number;
-      retryAfterMs: number;   // 0 if allowed
-    }
-    ```
-  - Token bucket algorithm:
-    - Each client key has a bucket with capacity `maxTokens`
-    - Tokens refill at `refillRate` per second
-    - Requests consume tokens (1 by default, more for expensive operations)
-    - When bucket empty, requests are rejected with retry time
-
-- `src/server/rate-limit-store.ts`
-  - In-memory store with LRU eviction for bucket state
-  - `LRUCache<string, BucketState>` with 10,000 entry limit
-  - Handles concurrent access with minimal locking
-
-- Integrate into HTTP server:
-  - Apply rate limiter to all MCP tool calls
-  - Key by: IP address (anonymous) or API key (authenticated)
-  - Return `429 Too Many Requests` when limited
-  - Include `Retry-After` header with retry time
-
-- Configuration:
-  - `rateLimit.enabled`: boolean (default: `true` for HTTP, `false` for stdio)
-  - `rateLimit.maxTokens`: number (default: `100`)
-  - `rateLimit.refillRate`: number (default: `10`)
-  - `rateLimit.perToolLimits`: map of tool names to token costs
-    - `index-folder`: 10 tokens (expensive)
-    - `search-symbols`: 1 token
-    - `get-symbol-source`: 1 token
-    - etc.
-
-**Key technical notes:**
-- Token bucket is more flexible than fixed window — allows bursts while maintaining average rate
-- Per-tool costs reflect operation expense (indexing >> retrieval)
-- LRU eviction prevents memory exhaustion from many unique clients
-- IP-based limiting is fallback; API key limiting is preferred
-
-**Verify:** Send 150 requests in rapid succession. Verify first 100 succeed, next 50 get 429. Wait for refill, verify requests succeed again. Verify `Retry-After` header present on 429 responses.
-
-**Tests:** Token consumption: verify count decrements. Refill: verify tokens restore over time. LRU eviction: verify oldest keys evicted. Per-tool costs: indexing consumes more tokens than search. Concurrent access: no race conditions.
-
----
-
-## Task 97: API Key Authentication
-
-Implement API key based authentication for tenant identification.
-
-**Deliverables:**
-
-- `src/server/auth/api-key.ts`
-  - `ApiKeyValidator` class:
-    ```typescript
-    class ApiKeyValidator {
-      validate(apiKey: string): Promise<AuthResult>;
-      generate(tenantId: string, permissions: Permission[]): string;
-      revoke(apiKey: string): void;
-    }
-
-    interface AuthResult {
-      valid: boolean;
-      tenantId?: string;
-      permissions?: Permission[];
-      rateLimitTier?: string;
-    }
-
-    type Permission = 'read' | 'write' | 'admin';
-    ```
-  - API key format: `cl_live_<tenantId>_<random>_<checksum>`
-    - Prefix: `cl_live_` for production, `cl_test_` for test keys
-    - TenantId: 8-char hex (short identifier)
-    - Random: 24-char base62 random string
-    - Checksum: 4-char CRC for validation without DB lookup
-  - Fast validation: checksum verifies format without DB hit
-  - Full validation: DB lookup for permissions and revocation check
-
-- `src/core/db/api-keys.ts`
-  - SQLite table `api_keys`:
-    ```sql
-    CREATE TABLE api_keys (
-      key_hash TEXT PRIMARY KEY,       -- SHA-256 of full key
-      tenant_id TEXT NOT NULL,
-      permissions TEXT NOT NULL,        -- JSON array
-      rate_limit_tier TEXT NOT NULL,
-      created_at TEXT NOT NULL,
-      last_used_at TEXT,
-      revoked_at TEXT,
-      FOREIGN KEY (tenant_id) REFERENCES tenants(id)
-    );
-    CREATE INDEX idx_api_keys_tenant ON api_keys(tenant_id);
-    ```
-  - Never store raw API keys — only hashes
-  - Track last usage for auditing
-
-- HTTP middleware:
-  - Extract API key from `Authorization: Bearer <key>` header
-  - Or from `X-API-Key: <key>` header (alternative)
-  - Validate and attach `AuthResult` to request context
-  - Return `401 Unauthorized` for missing/invalid keys
-  - Return `403 Forbidden` for insufficient permissions
-
-- CLI tool for key management:
-  - `purecontext-mcp keys create --tenant <id> --permissions read,write`
-  - `purecontext-mcp keys list --tenant <id>`
-  - `purecontext-mcp keys revoke <key-prefix>`
-  - Output includes full key only on creation (not stored)
-
-**Key technical notes:**
-- API keys are hashed before storage — raw keys cannot be recovered
-- Checksum allows fast format validation without DB access
-- Permissions are simple RBAC: read (query), write (index), admin (manage)
-- Rate limit tiers allow different limits per subscription level
-
-**Verify:** Generate API key. Use it in request header. Verify authentication succeeds. Revoke key. Verify subsequent requests fail. Verify invalid key returns 401.
-
-**Tests:** Key generation: verify format. Checksum validation: valid key passes, tampered key fails. DB validation: revoked key fails. Permissions: read-only key cannot index. Rate limit tier: different limits applied.
-
----
-
-## Task 98: Multi-Tenant Data Isolation
-
-Implement tenant-level data isolation for shared deployments.
-
-**Deliverables:**
-
-- `src/core/db/tenants.ts`
-  - SQLite table `tenants`:
-    ```sql
-    CREATE TABLE tenants (
-      id TEXT PRIMARY KEY,
-      name TEXT NOT NULL,
-      created_at TEXT NOT NULL,
-      settings TEXT,               -- JSON for tenant-specific config
-      storage_quota_bytes INTEGER, -- Max storage per tenant
-      storage_used_bytes INTEGER   -- Current usage
-    );
-    ```
-  - `TenantStore` class:
-    ```typescript
-    class TenantStore {
-      create(name: string): Tenant;
-      get(id: string): Tenant | null;
-      update(id: string, updates: Partial<Tenant>): void;
-      delete(id: string): void;
-      list(): Tenant[];
-    }
-    ```
-
-- Update all data tables with tenant isolation:
-  - Add `tenant_id` column to `repos`, `symbols`, `files`, `dep_edges`, `embeddings`
-  - Add foreign key constraint to `tenants.id`
-  - Create indexes on `tenant_id` for efficient filtering
-  - Migration script for existing single-tenant data
-
-- Update all data access:
-  - `SymbolStore`: all queries filtered by `tenant_id`
-  - `FileStore`: all queries filtered by `tenant_id`
-  - `DepStore`: all queries filtered by `tenant_id`
-  - `VectorStore`: separate HNSW indexes per tenant
-
-- Tenant context:
-  - `TenantContext` object attached to request
-  - All service methods receive tenant context
-  - Prevents cross-tenant data access
-
-- Storage quotas:
-  - Track storage per tenant (files + embeddings)
-  - Reject indexing when quota exceeded
-  - Return `507 Insufficient Storage` with quota details
-
-**Key technical notes:**
-- Tenant isolation is enforced at the data layer, not just API layer
-- Separate HNSW indexes prevent cross-tenant semantic search leakage
-- Storage quotas prevent runaway usage
-- Existing single-tenant data migrates to a default "local" tenant
-
-**Verify:** Create two tenants. Index different repos under each. Verify searches only return tenant's own symbols. Verify one tenant cannot access another's repos.
-
-**Tests:** Tenant creation/deletion. Cross-tenant query: tenant A cannot see tenant B's symbols. Storage quota: reject indexing when exceeded. Migration: existing data belongs to default tenant.
-
----
-
-## Task 99: Admin API
-
-Add administrative endpoints for managing tenants and monitoring.
-
-**Deliverables:**
-
-- `src/server/admin-api.ts`
-  - Admin routes (require `admin` permission):
-    - `POST /admin/tenants` — create tenant
-    - `GET /admin/tenants` — list tenants
-    - `GET /admin/tenants/:id` — get tenant details
-    - `PATCH /admin/tenants/:id` — update tenant
-    - `DELETE /admin/tenants/:id` — delete tenant (and all data)
-    - `POST /admin/tenants/:id/keys` — create API key for tenant
-    - `GET /admin/tenants/:id/keys` — list API keys
-    - `DELETE /admin/keys/:prefix` — revoke API key
-
-- Monitoring endpoints:
-  - `GET /admin/stats`
-    ```json
-    {
-      "tenants": 15,
-      "total_repos": 142,
-      "total_symbols": 2450000,
-      "storage_used_bytes": 1073741824,
-      "requests_24h": 50000,
-      "rate_limit_hits_24h": 120,
-      "uptime_seconds": 86400
-    }
-    ```
-  - `GET /admin/tenants/:id/stats`
-    ```json
-    {
-      "repos": 12,
-      "symbols": 180000,
-      "storage_used_bytes": 52428800,
-      "requests_24h": 3000,
-      "rate_limit_hits_24h": 5
-    }
-    ```
-
-- Request logging:
-  - Log all requests to SQLite table `request_log`:
-    ```sql
-    CREATE TABLE request_log (
-      id INTEGER PRIMARY KEY,
-      timestamp TEXT NOT NULL,
-      tenant_id TEXT,
-      tool TEXT NOT NULL,
-      duration_ms INTEGER,
-      status TEXT,          -- 'success', 'error', 'rate_limited'
-      error_message TEXT
-    );
-    CREATE INDEX idx_request_log_tenant ON request_log(tenant_id, timestamp);
-    ```
-  - Prune logs older than 30 days (configurable)
-  - Aggregate stats computed from logs
-
-**Key technical notes:**
-- Admin API is separate from MCP tools — REST endpoints only
-- Request logging enables usage analytics and debugging
-- Log pruning prevents unbounded growth
-- Stats are computed from logs, not maintained separately (simpler)
-
-**Verify:** Create tenant via admin API. Generate API key. Use key to index repo. View tenant stats — verify request counted. Delete tenant — verify data deleted.
-
-**Tests:** Tenant CRUD: create, read, update, delete. API key lifecycle: create, list, revoke. Stats aggregation: requests counted correctly. Log pruning: old entries removed. Admin permission: non-admin cannot access.
-
----
-
-## Task 100: Phase 12 Integration Tests
-
-Validate the complete rate limiting and multi-tenant system.
-
-**Deliverables:**
-
-- Integration tests `test/integration/phase12.test.ts`:
-  1. Rate limiting: 100 requests succeed, 101st fails with 429
-  2. Rate limiting: verify `Retry-After` header present
-  3. Rate limiting: verify per-tool costs (indexing uses more tokens)
-  4. Rate limiting: verify refill restores capacity
-  5. API key: valid key authenticates successfully
-  6. API key: invalid key returns 401
-  7. API key: revoked key returns 401
-  8. API key: checksum validation catches tampering
-  9. Tenant isolation: tenant A cannot see tenant B's data
-  10. Tenant isolation: search returns only tenant's symbols
-  11. Storage quota: indexing rejected when quota exceeded
-  12. Admin API: create tenant, generate key, index, query stats
-  13. Admin API: delete tenant removes all data
-  14. Full suite regression: all Phase 1–11 tests still green
-
-- Load testing:
-  - Simulate 100 concurrent clients
-  - Verify rate limiting applies correctly under load
-  - Verify no data leakage under concurrent multi-tenant access
-
-**Verify:** `npm run test` passes. Load tests verify system stability under concurrent access.
-
----
-
-## Order of Execution
-
-```
-Task 96: Token bucket rate limiter         ███░░░░░░░ Rate Limiting
-Task 97: API key authentication            █████░░░░░ Auth
-Task 98: Multi-tenant data isolation       ███████░░░ Isolation
-Task 99: Admin API                         █████████░ Admin
-Task 100: Integration tests                ██████████ Polish
-```
-
-Tasks proceed in order: rate limiting (96) provides basic protection, API keys (97) enable tenant identification, isolation (98) secures data, admin API (99) enables management.
-
----
-
-## Post-Phase 12: What Comes Next
-
-Phase 12 enables PureContext as a hosted service. The final phase adds visualization:
-
-- **Phase 13**: Web UI for exploring the symbol graph visually