@soulcraft/sdk 1.4.6 → 1.4.8

package/docs/ADR-002-transport-protocol.md

@@ -0,0 +1,248 @@
+# ADR-002: Transport Protocol — Wire Format, Auth, and Reconnection
+
+**Status:** Accepted
+**Date:** 2026-03-02
+**Supersedes:** None
+**See also:** `ADR-001-sdk-design.md` (Decision 5)
+
+---
+
+## Context
+
+The SDK must communicate between kit apps (browser) or product backends and a Brainy
+server over a network. Four distinct communication patterns exist across the platform:
+
+1. **Stateless RPC** — kit apps call Brainy methods on demand (no persistent connection needed)
+2. **Real-time bidirectional RPC + push** — Venue kit apps need live change events alongside RPC
+3. **Server-push only** — Workshop workspace event stream (VFS/entity mutations)
+4. **In-process** — server-mode SDK calls Brainy directly with zero serialization overhead
+
+These patterns have meaningfully different requirements for serialization, auth, and
+error handling. This ADR records the decisions made for each.
+
+---
+
+## Decision 1: Four Transports, Fixed Serialization
+
+Four transport implementations are provided. Serialization format is fixed per transport
+and is never configurable:
+
+| Transport | Class | Serialization | Direction | Use case |
+|-----------|-------|--------------|-----------|---------|
+| `local` | `LocalTransport` | None (in-process) | — | Server mode, zero overhead |
+| `http` | `HttpTransport` | JSON | Request/response | Stateless RPC — kit apps, simple clients |
+| `ws` | `WsTransport` | MessagePack binary | Bidirectional | Real-time — RPC + change push events |
+| `sse` | `SseTransport` | text/event-stream | Server → Client | Live updates only — VFS/entity events |
+
+**Rationale for fixed serialization:**
+- HTTP=JSON: universally debuggable with `curl`, browser DevTools, and server logs.
+  The request volume at typical HTTP RPC frequencies makes binary encoding a
+  premature optimization with no measurable benefit.
+- WebSocket=MessagePack: bidirectional real-time traffic has high message frequency
+  and persistent connections where binary encoding is measurably more efficient.
+  MessagePack was already the wire format in `@soulcraft/brainy-client`.
+- Making serialization configurable adds complexity with no practical benefit —
+  consumers never need to mix formats within a single transport.
+
+**Rejected alternatives:**
+- MessagePack over HTTP: non-standard, incompatible with standard proxies and logging tools.
+- gRPC/Protobuf: over-engineered for the current scale and requires a schema compilation step.
+- Configurable serializer per transport: extra complexity for a use case that has never arisen.
+
+---
+
+## Decision 2: HTTP Transport Wire Format
+
+**Endpoint:** `POST {baseUrl}/api/brainy/rpc`
+
+**Request body** (JSON):
+```json
+{ "method": "find", "args": [{ "query": "candle kits", "limit": 10 }] }
+```
+
+**Success response** (JSON):
+```json
+{ "result": [ ... ] }
+```
+
+**Error response** (JSON):
+```json
+{ "error": { "code": "BRAINY_NOT_FOUND", "message": "Entity not found" } }
+```
+
+**Auth:** `Authorization: Bearer <capability-token>` for server-to-server calls;
+`credentials: 'include'` (session cookies) for browser kit apps.
+
+**Timeout:** 30 seconds via `AbortController`. Throws `SDKTimeoutError` on expiry.
+
+**HTTP status mapping:**
+| Status | Error class |
+|--------|------------|
+| 401 | `SDKAuthError` |
+| 403 | `SDKForbiddenError` |
+| network failure | `SDKDisconnectedError` |
+| 200 + `error` body | `SDKRpcError` |
+
+The HTTP transport is stateless — `isAlive()` always returns `true`. Errors surface
+as rejected promises from individual `call()` invocations.
+
+---
+
+## Decision 3: WebSocket Transport Wire Format
+
+**Endpoint:** `wss://{host}/api/brainy/ws`
+
+**Auth:** Capability token sent as `Authorization: Bearer <token>` on the WebSocket
+upgrade request. This is a Bun-specific extension (`@ts-expect-error` in source).
+Standard browser `WebSocket` does not support custom headers on the upgrade request —
+browser kit apps use the HTTP transport or pass the token as a `?token=` query param.
+
+**Scope param:** `?scope=<tenantSlug>` on the connection URL. Venue uses this to
+select the correct per-tenant Brainy instance.
+
+### Connection handshake
+
+After a successful upgrade + auth check the server sends a `ready` message:
+```
+Server → Client: { "type": "ready", "scope": "wicks-and-whiskers" }
+```
+
+`WsTransport.connect()` resolves only after this message. A 15-second timeout
+applies — if no `ready` arrives the connection is aborted with an error.
+
+### RPC messages (binary MessagePack)
+
+**Client → Server:**
+```
+{ id: "42", method: "vfs.readdir", args: ["/workspace/docs"] }
+```
+
+The `id` is a monotonically incrementing integer cast to string, unique per connection.
+
+**Server → Client (RPC response):**
+```
+{ id: "42", result: [ ... ] }
+{ id: "42", error: { code: "VFS_NOT_FOUND", message: "Path not found" } }
+```
+
+Pending RPCs are matched to their response by `id`. A per-call 30-second timeout
+rejects calls that receive no response.
+
+### Change push events (binary MessagePack)
+
+```
+{ type: "change", event: "add"|"update"|"delete"|"relate"|"unrelate",
+  entity?: { ... }, relation?: { ... } }
+```
+
+Push events have no `id` field and are dispatched to all registered `onEvent` handlers.
+
+### Reconnection
+
+On unexpected disconnection (any close code except 4001, 4003, or explicit `close()`):
+
+- All pending RPC calls are rejected with `SDKDisconnectedError`
+- Reconnect is attempted with **exponential backoff**: `min(1000 × 2^attempt, 30_000)` ms
+- Maximum 10 attempts (configurable via constructor)
+- After 10 failed attempts the transport enters `closed` state permanently
+
+**Auth failure codes — no retry:**
+| Close code | Meaning | Behaviour |
+|-----------|---------|----------|
+| 4001 | Unauthorized (bad/expired token) | Reject pending calls with `SDKAuthError`, set `closed` |
+| 4003 | Forbidden (insufficient scope) | Reject pending calls with `SDKForbiddenError`, set `closed` |
+
+### Connection states
+
+```
+disconnected → connecting → connected → disconnected (unexpected) → connecting (reconnect) ...
+                                      → closed (explicit close() or auth failure)
+```
+
+---
+
+## Decision 4: SSE Transport Wire Format
+
+**Endpoint:** `GET {baseUrl}/api/workspace/events[?workspaceId=<id>]`
+
+Uses the browser's native `EventSource` API. `EventSource` handles reconnection
+automatically using the server-sent `retry:` interval. Additionally, the SDK applies
+manual exponential backoff (identical schedule to WsTransport) on `onerror` events
+to prevent thundering-herd reconnects on server restart.
+
+**Event format** (JSON-encoded in the SSE `data:` field):
+```
+data: {"type":"change","event":"update","entity":{"id":"abc","nounType":"Booking",...}}
+```
+
+**Connection confirmation** (internal, not dispatched to listeners):
+```
+data: {"type":"connected","connectionId":"conn-abc123"}
+```
+
+**Receive-only:** `call()` throws `SDKError` with code `SSE_NO_RPC`. For outbound
+operations pair this transport with an `HttpTransport`.
+
+---
+
+## Decision 5: Local Transport (In-Process)
+
+Used exclusively in server mode. Wraps a native `Brainy` instance. No serialization,
+no network, no error class mapping.
+
+**Method resolution:** Dot-separated method paths are resolved by recursive property
+traversal on the Brainy instance. `'vfs.readdir'` → `brain.vfs.readdir(args)`.
+
+**Change events:** `onEvent`/`offEvent` delegate to Brainy's built-in change listeners
+directly on the instance. No intermediary.
+
+**isAlive():** Always `true`. **close():** No-op.
+
+---
+
+## Decision 6: Shared Error Classes
+
+All transports share a common error hierarchy:
+
+```
+SDKError (base, has `code: string`)
+├── SDKAuthError        (401 / close code 4001)
+├── SDKForbiddenError   (403 / close code 4003)
+├── SDKTimeoutError     (30s per-call timeout)
+├── SDKDisconnectedError (transport not connected, or unexpected close)
+└── SDKRpcError         (server returned { error: { code, message } })
+```
+
+Consumer code catches `SDKAuthError` to redirect to login, `SDKDisconnectedError`
+to show a reconnecting state, etc.
+
+---
+
+## Decision 7: Capability Tokens for Server-to-Server Auth
+
+Transport-level auth uses HMAC-SHA256 capability tokens:
+
+```
+<base64url(payload)>.<base64url(signature)>
+```
+
+`payload` is `{ sub, scope, iat, exp }` JSON. Default TTL: 1 hour.
+
+**Timing-safe verification** uses `crypto.timingSafeEqual` to prevent timing attacks.
+
+These tokens are separate from OIDC session tokens. They are used for:
+- Kit apps making RPC calls to the Venue or Workshop backend
+- Server-to-server calls (e.g. Academy → Workshop)
+
+Session cookie auth (browser kit apps in same-origin context) uses `credentials: 'include'`
+on the HTTP transport and does not require a capability token.
+
+---
+
+## Related Documents
+
+- `ADR-001-sdk-design.md` — Overall SDK architecture (Decision 5: transport rationale)
+- `ADR-003-instance-strategies.md` — How server-side Brainy instances are pooled and selected
+- `src/transports/` — Implementation of all four transports
+- `src/modules/brainy/errors.ts` — Shared error class hierarchy
+- `src/modules/brainy/auth.ts` — Capability token creation and verification

package/docs/ADR-003-instance-strategies.md

@@ -0,0 +1,216 @@
+# ADR-003: Brainy Instance Strategies — Pooling, Lifecycle, and Cortex Activation
+
+**Status:** Accepted
+**Date:** 2026-03-02
+**Supersedes:** Workshop's `brainy-singleton.ts`, Venue's inline `tenantCache`
+**See also:** `ADR-001-sdk-design.md` (Decision 4), `ADR-002-transport-protocol.md`
+
+---
+
+## Context
+
+Each product manages Brainy instances differently:
+
+- **Workshop** — one Brainy per user × workspace. Potentially hundreds of instances
+  across all Workshop users. Each user's data is isolated by email hash.
+- **Venue** — one Brainy per tenant (one Brainy for all of "Wicks & Whiskers", another
+  for "The Candle Studio"). Tenant count is bounded but tenants can be busy.
+- **Academy** — one Brainy per course section, or per learner × course. Domain-specific
+  key logic that doesn't fit either Workshop or Venue's pattern.
+
+Before the SDK, each product maintained its own singleton/cache implementation with
+diverging eviction policies, flush-on-shutdown behaviour, and Cortex activation code.
+This caused drift and subtle bugs.
+
+`BrainyInstancePool` centralizes the pattern in one place.
+
+---
+
+## Decision 1: Three Built-In Strategies
+
+| Strategy | Key | Storage path | Product |
+|----------|-----|-------------|---------|
+| `per-user` | `emailHash:workspaceId` | `{dataPath}/{emailHash}/{workspaceId}/` | Workshop |
+| `per-tenant` | `tenantSlug` | `{dataPath}/{tenantSlug}/` | Venue |
+| `per-scope` | caller-supplied `scopeKey(userId, workspaceId)` | caller-supplied factory | Academy, custom |
+
+`per-scope` is a general escape hatch — the caller provides both the key function
+and a `factory: () => Promise<Brainy>` per scope. This avoids a fourth built-in
+strategy that would only serve edge cases.
+
+Default max instances: `200` for `per-user`, `50` for `per-tenant`. Both are
+configurable via `maxInstances`.
+
+---
+
+## Decision 2: LRU Eviction with Non-Blocking Flush
+
+The pool is backed by `lru-cache`. When the cache is full and a new instance is
+needed, the LRU entry is evicted.
+
+**Eviction is synchronous** — `lru-cache`'s `dispose` callback fires synchronously.
+`brain.flush()` is initiated non-blocking from within `dispose` if `flushOnEvict: true`.
+This is intentional: delaying eviction until flush completes would block the incoming
+request that triggered the eviction.
+
+**Consequence:** On abrupt process termination, recently evicted instances may not have
+flushed. Products should call `sdk.shutdown()` (which calls `pool.shutdown()`) in their
+`SIGTERM` handler to ensure all pending flushes complete before exit.
+
+**The `flushOnEvict` default is `true`.** Setting it to `false` is only appropriate for
+ephemeral development data or when the caller guarantees clean shutdown via explicit
+`pool.flush(key)` calls.
+
+---
+
+## Decision 3: Concurrent Init Deduplication
+
+Brainy's cold start takes 25–30 seconds on a dataset of typical size. In a multi-request
+environment it is common for several requests to arrive simultaneously for the same scope
+before the first init completes.
+
+Without deduplication each request would spawn a separate Brainy init, creating multiple
+instances pointing at the same storage path — a correctness violation (concurrent writers
+on the same mmap files corrupt data).
+
+The pool uses a `pending: Map<string, Promise<Brainy>>` to deduplicate. The first request
+for a key starts the init and stores the Promise. Subsequent requests for the same key
+return the existing Promise and await the same init. Only one Brainy is ever created per key.
+
+---
+
+## Decision 4: Storage Backends and Cortex Activation
+
+Two `storage` values are supported:
+
+| Value | When to use | Cortex behavior |
+|-------|-------------|-----------------|
+| `'filesystem'` | Local development | No Cortex plugin loaded |
+| `'mmap-filesystem'` | Production (GCE with persistent disk) | Cortex plugin loaded |
+
+When `storage: 'mmap-filesystem'`, the pool passes `plugins: ['@soulcraft/cortex']` to
+the Brainy constructor. Cortex intercepts Brainy's storage layer at init time and upgrades
+the filesystem provider to native Rust mmap SSTables. From the pool's perspective Brainy
+is initialized the same way in both modes — the storage upgrade is transparent.
+
+**Cortex is a Brainy plugin, not a separate init step.** The pool does not call any
+Cortex activation API directly. License activation is handled by `fromLicense()` which
+writes `.soulcraft.json` before any Brainy instance is created. Cortex reads the JWT
+from `.soulcraft.json` at plugin load time.
+
+### Cortex license flow (from Workshop experience)
+
+This sequence is critical and order-dependent:
+
+```
+1. fromLicense({ product }) — exchanges license key with Portal
+2.   → Portal returns sc_cortex_<jwt> in the config bundle
+3.   → fromLicense() writes { "cortex": "sc_cortex_..." } to .soulcraft.json atomically
+4. Pool._initBrainy() called on first request
+5.   → new Brainy({ plugins: ['@soulcraft/cortex'] })
+6.   → Cortex plugin loads, reads .soulcraft.json, validates JWT offline
+7.   → If valid: upgrades storage to native mmap — logs "Providers: N/10 native"
+8.   → brain.init() completes
+```
+
+**If step 3 is skipped** (`.soulcraft.json` absent or contains a short-code like
+`CX-63QM-UV67` instead of a JWT), Cortex logs a validation error and falls back to
+pure filesystem storage. The SDK continues to function — Brainy works without Cortex —
+but native SIMD acceleration is disabled.
+
+**The CX code is a short-code, not the JWT.** `fromLicense()` exchanges the short-code
+for a JWT. The JWT is what Cortex validates offline. Never write a raw `CX-…` code into
+`.soulcraft.json` and expect Cortex to accept it — it will not.
+
+### Known issue (Cortex ≤ 2.1.3, fixed in 2.1.5)
+
+In Cortex 2.1.3 the public key compiled into the native binary did not match the key
+Portal used to sign JWTs. Every JWT was rejected with `invalid signature` regardless
+of validity. This was fixed in `@soulcraft/cortex@2.1.5` (compile-time key rotation
+by the Cortex team). The minimum required version is now `>=2.1.5` in peer dependencies.
+
+---
+
+## Decision 5: `onInit` Hook for Product-Specific Post-Init Logic
+
+The pool config accepts an optional `onInit(brain, storagePath)` async callback
+called after each new Brainy instance completes `init()`. This is the escape hatch
+for product-specific logic that must run once per instance creation:
+
+- **Workshop:** VFS integrity check, project metadata migration
+- **Venue:** Tenant schema validation, default entity seeding
+- **Academy:** Course-specific index warming
+
+The `onInit` callback runs before the instance is added to the cache. If it throws,
+the init fails, the instance is not cached, and the caller receives the error. A
+subsequent request for the same key will retry the full init sequence.
+
+---
+
+## Decision 6: Graceful Shutdown
+
+`sdk.shutdown()` calls `pool.shutdown()` which:
+1. Calls `brain.flush()` on every live instance in parallel
+2. Clears the cache
+3. Clears the pending-init map
+
+Products must call `sdk.shutdown()` from their `SIGTERM` handler:
+
+```typescript
+process.on('SIGTERM', async () => {
+  await sdk.shutdown()
+  process.exit(0)
+})
+```
+
+The `fromLicense()` factory also calls `pool.shutdown()` as part of its shutdown
+sequence (billing buffer flush → heartbeat stop → pool flush).
+
+---
+
+## Decision 7: Storage Path Conventions
+
+| Strategy | Path |
+|----------|------|
+| `per-user` | `{dataPath}/{emailHash}/{workspaceId}/` |
+| `per-tenant` | `{dataPath}/{tenantSlug}/` |
+| `per-scope` | Determined by caller's `factory` |
+
+`emailHash` is an 8-character SHA-256 hex prefix of the lowercase+trimmed email
+address, computed by `computeEmailHash()`. Using a hash rather than the raw email
+avoids filesystem encoding issues and obscures PII from directory listings.
+
+All paths are created with `fs.mkdir(path, { recursive: true })` at init time — the
+pool does not require pre-existing directories.
+
+---
+
+## Decision 8: `BrainyInitializingError` for Non-Blocking Cold Start
+
+Products that prefer to return HTTP 503 during cold start (rather than blocking the
+client for 25–30 seconds) can pass a non-blocking flag and catch
+`BrainyInitializingError`:
+
+```typescript
+app.onError((err, c) => {
+  if (err instanceof BrainyInitializingError) {
+    return c.json({ error: 'Service starting', retryAfter: err.retryAfter }, 503, {
+      'Retry-After': String(err.retryAfter),
+    })
+  }
+})
+```
+
+This is optional — the default pool behaviour blocks the request until init completes,
+which is correct for Workshop (users expect their workspace to load, even if slowly).
+Venue may prefer the 503 pattern to avoid long-hanging HTTP requests on the first
+tenant request after a cold deploy.
+
+---
+
+## Related Documents
+
+- `ADR-001-sdk-design.md` — Overall SDK architecture (Decision 3: instance strategies)
+- `ADR-002-transport-protocol.md` — How clients connect to the pool's Brainy instances
+- `src/server/instance-pool.ts` — Full implementation
+- `src/server/from-license.ts` — License activation and `.soulcraft.json` write (step 3 above)

package/docs/IMPLEMENTATION-PLAN.md

@@ -1,6 +1,6 @@
 # @soulcraft/sdk — Detailed Implementation Plan
 
-**Status:** Ready to implement
+**Status:** Complete as of v1.4.6
 **Decisions:** See `ADR-001-sdk-design.md` for all architecture decisions and rationale.
 **Research basis:** Full cross-project exploration of Workshop, Venue, Academy, brainy-client,
 and @soulcraft/auth conducted 2026-02-27.
@@ -38,11 +38,6 @@ A new session MUST read the source file(s) before implementing each module.
 
 All items done. Repo created, ADR written, scaffold committed.
 
-Remaining scaffolding items (minor, do at start of Phase 2):
-- [ ] `vitest.config.ts` — test runner config
-- [ ] `.env.example` — document any env vars the SDK needs
-- [ ] Update `src/modules/*/types.ts` placeholder comments as modules are implemented
-
 ---
 
 ## Phase 2 — Core Data Layer
@@ -634,48 +629,35 @@ renderEmailTemplate(template: string, data: Record<string, unknown>): string
 
 ## Phase 6 — npm Publish + Migrate All Products
 
-### 6a. Pre-publish checklist
-- [ ] All tests passing (`bun test`)
-- [ ] TypeScript builds cleanly (`bun run check`)
-- [ ] No stubs, no TODOs, no `as any`
-- [ ] JSDoc on every exported symbol
-- [ ] Module-level `@module` block on every file
-- [ ] `package.json` version set to `1.0.0`
-- [ ] `README.md` written with usage examples for all three entry points
+### 6a. Pre-publish checklist ✅ Complete (v1.4.6)
+- [x] All tests passing (`bun test`) — 330 tests, 21 suites
+- [x] TypeScript builds cleanly (`bun run check`)
+- [x] No stubs, no TODOs, no `as any`
+- [x] JSDoc on every exported symbol
+- [x] Module-level `@module` block on every file
+- [x] Published as `@soulcraft/sdk@1.4.6` (first published as `1.0.0`, now at `1.4.6`)
+- [x] `README.md` written with usage examples for all three entry points
 
 ### 6b. Publish
 ```bash
 npm publish --access restricted
 ```
 
-### 6c. Migrate Workshop
-Files to update in `/media/dpsifr/storage/home/Projects/workshop/`:
-1. `package.json` — add `@soulcraft/sdk`, remove `@soulcraft/brainy-client`, `@soulcraft/auth`
-2. `build.sh` — add `sdk` to bundle loop, remove `brainy-client` and `auth`
-3. `auth/better-auth.ts` — replace with `createProductAuth('workshop', ...)` from SDK
-4. `server-hono.ts` — replace brainy-client handler factories with SDK handlers
-5. `brainy/brainy-singleton.ts` — replace with `sdk.server.instancePool`
-6. `brainy/remote-brainy.ts` — replace with SDK `WsTransport`
-7. `brainy/usage-tracking.ts` — replace with `sdk.license.credits.*` + `sdk.billing.*`
-8. All `import { ... } from '@soulcraft/auth'` → `from '@soulcraft/sdk'`
-9. All `import { createBrainyClient } from '@soulcraft/brainy-client'` → `from '@soulcraft/sdk/client'`
-
-### 6d. Migrate Venue
-Files to update in `/home/dpsifr/Projects/venue/`:
-1. `apps/web/package.json` — same package swap
-2. `apps/web/server.ts` — replace brainy-client WS handler with SDK
-3. `apps/web/src/lib/server/auth.ts` — replace with `createProductAuth('venue', ...)`
-4. `apps/web/src/routes/api/brainy/rpc/+server.ts` — replace with SDK HTTP handler
-5. All notification code → `sdk.notifications.*`
-
-### 6e. Migrate Academy
-Files to update in `/media/dpsifr/storage/home/Projects/academy/`:
-1. `package.json`, `auth/better-auth.ts`, `server.ts` — same patterns as Workshop
+### 6c. Migrate Workshop ✅ Complete
+Workshop migrated to `@soulcraft/sdk@1.4.6`. Uses `fromLicense({ product: 'workshop' })`.
+All brainy-client and auth imports replaced. Bundle loop updated.
+
+### 6d. Migrate Venue ⏳ In progress
+Venue has open actions in the handoff file — `fromLicense()` migration pending.
+See `/home/dpsifr/.strategy/PLATFORM-HANDOFF.md`.
+
+### 6e. Migrate Academy ✅ Complete
+Academy migrated to `@soulcraft/sdk@1.4.6`.
 
 ### 6f. Post-migration
-- Deprecate `@soulcraft/brainy-client` on npmjs.com (`npm deprecate`)
-- Deprecate `@soulcraft/auth` on npmjs.com
-- Update `PLATFORM-HANDOFF.md` thread to RESOLVED
+- `@soulcraft/brainy-client` — deprecated on npmjs.com ✅
+- `@soulcraft/auth` — deprecated on npmjs.com ✅
+- Handoff thread: resolved for Workshop and Academy; Venue still pending
 
 ---