@soulcraft/sdk 1.4.6 → 1.4.7

package/docs/ADR-003-instance-strategies.md

@@ -0,0 +1,216 @@
+# ADR-003: Brainy Instance Strategies — Pooling, Lifecycle, and Cortex Activation
+
+**Status:** Accepted
+**Date:** 2026-03-02
+**Supersedes:** Workshop's `brainy-singleton.ts`, Venue's inline `tenantCache`
+**See also:** `ADR-001-sdk-design.md` (Decision 4), `ADR-002-transport-protocol.md`
+
+---
+
+## Context
+
+Each product manages Brainy instances differently:
+
+- **Workshop** — one Brainy per user × workspace. Potentially hundreds of instances
+  across all Workshop users. Each user's data is isolated by email hash.
+- **Venue** — one Brainy per tenant (one Brainy for all of "Wicks & Whiskers", another
+  for "The Candle Studio"). Tenant count is bounded but tenants can be busy.
+- **Academy** — one Brainy per course section, or per learner × course. Domain-specific
+  key logic that doesn't fit either Workshop or Venue's pattern.
+
+Before the SDK, each product maintained its own singleton/cache implementation with
+diverging eviction policies, flush-on-shutdown behaviour, and Cortex activation code.
+This caused drift and subtle bugs.
+
+`BrainyInstancePool` centralizes the pattern in one place.
+
+---
+
+## Decision 1: Three Built-In Strategies
+
+| Strategy | Key | Storage path | Product |
+|----------|-----|-------------|---------|
+| `per-user` | `emailHash:workspaceId` | `{dataPath}/{emailHash}/{workspaceId}/` | Workshop |
+| `per-tenant` | `tenantSlug` | `{dataPath}/{tenantSlug}/` | Venue |
+| `per-scope` | caller-supplied `scopeKey(userId, workspaceId)` | caller-supplied factory | Academy, custom |
+
+`per-scope` is a general escape hatch — the caller provides both the key function
+and a `factory: () => Promise<Brainy>` per scope. This avoids a fourth built-in
+strategy that would only serve edge cases.
+
+Default max instances: `200` for `per-user`, `50` for `per-tenant`. Both are
+configurable via `maxInstances`.
+
+---
+
+## Decision 2: LRU Eviction with Non-Blocking Flush
+
+The pool is backed by `lru-cache`. When the cache is full and a new instance is
+needed, the LRU entry is evicted.
+
+**Eviction is synchronous** — `lru-cache`'s `dispose` callback fires synchronously.
+`brain.flush()` is initiated non-blocking from within `dispose` if `flushOnEvict: true`.
+This is intentional: delaying eviction until flush completes would block the incoming
+request that triggered the eviction.
+
+**Consequence:** On abrupt process termination, recently evicted instances may not have
+flushed. Products should call `sdk.shutdown()` (which calls `pool.shutdown()`) in their
+`SIGTERM` handler to ensure all pending flushes complete before exit.
+
+**The `flushOnEvict` default is `true`.** Setting it to `false` is only appropriate for
+ephemeral development data or when the caller guarantees clean shutdown via explicit
+`pool.flush(key)` calls.
+
+---
+
+## Decision 3: Concurrent Init Deduplication
+
+Brainy's cold start takes 25–30 seconds on a dataset of typical size. In a multi-request
+environment it is common for several requests to arrive simultaneously for the same scope
+before the first init completes.
+
+Without deduplication each request would spawn a separate Brainy init, creating multiple
+instances pointing at the same storage path — a correctness violation (concurrent writers
+on the same mmap files corrupt data).
+
+The pool uses a `pending: Map<string, Promise<Brainy>>` to deduplicate. The first request
+for a key starts the init and stores the Promise. Subsequent requests for the same key
+return the existing Promise and await the same init. Only one Brainy is ever created per key.
+
+---
+
+## Decision 4: Storage Backends and Cortex Activation
+
+Two `storage` values are supported:
+
+| Value | When to use | Cortex behavior |
+|-------|-------------|-----------------|
+| `'filesystem'` | Local development | No Cortex plugin loaded |
+| `'mmap-filesystem'` | Production (GCE with persistent disk) | Cortex plugin loaded |
+
+When `storage: 'mmap-filesystem'`, the pool passes `plugins: ['@soulcraft/cortex']` to
+the Brainy constructor. Cortex intercepts Brainy's storage layer at init time and upgrades
+the filesystem provider to native Rust mmap SSTables. From the pool's perspective Brainy
+is initialized the same way in both modes — the storage upgrade is transparent.
+
+**Cortex is a Brainy plugin, not a separate init step.** The pool does not call any
+Cortex activation API directly. License activation is handled by `fromLicense()` which
+writes `.soulcraft.json` before any Brainy instance is created. Cortex reads the JWT
+from `.soulcraft.json` at plugin load time.
+
+### Cortex license flow (from Workshop experience)
+
+This sequence is critical and order-dependent:
+
+```
+1. fromLicense({ product }) — exchanges license key with Portal
+2.   → Portal returns sc_cortex_<jwt> in the config bundle
+3.   → fromLicense() writes { "cortex": "sc_cortex_..." } to .soulcraft.json atomically
+4. Pool._initBrainy() called on first request
+5.   → new Brainy({ plugins: ['@soulcraft/cortex'] })
+6.   → Cortex plugin loads, reads .soulcraft.json, validates JWT offline
+7.   → If valid: upgrades storage to native mmap — logs "Providers: N/10 native"
+8.   → brain.init() completes
+```
+
+**If step 3 is skipped** (`.soulcraft.json` absent or contains a short-code like
+`CX-63QM-UV67` instead of a JWT), Cortex logs a validation error and falls back to
+pure filesystem storage. The SDK continues to function — Brainy works without Cortex —
+but native SIMD acceleration is disabled.
+
+**The CX code is a short-code, not the JWT.** `fromLicense()` exchanges the short-code
+for a JWT. The JWT is what Cortex validates offline. Never write a raw `CX-…` code into
+`.soulcraft.json` and expect Cortex to accept it — it will not.
+
+### Known issue (Cortex ≤ 2.1.3, fixed in 2.1.5)
+
+In Cortex 2.1.3 the public key compiled into the native binary did not match the key
+Portal used to sign JWTs. Every JWT was rejected with `invalid signature` regardless
+of validity. This was fixed in `@soulcraft/cortex@2.1.5` (compile-time key rotation
+by the Cortex team). The minimum required version is now `>=2.1.5` in peer dependencies.
+
+---
+
+## Decision 5: `onInit` Hook for Product-Specific Post-Init Logic
+
+The pool config accepts an optional `onInit(brain, storagePath)` async callback
+called after each new Brainy instance completes `init()`. This is the escape hatch
+for product-specific logic that must run once per instance creation:
+
+- **Workshop:** VFS integrity check, project metadata migration
+- **Venue:** Tenant schema validation, default entity seeding
+- **Academy:** Course-specific index warming
+
+The `onInit` callback runs before the instance is added to the cache. If it throws,
+the init fails, the instance is not cached, and the caller receives the error. A
+subsequent request for the same key will retry the full init sequence.
+
+---
+
+## Decision 6: Graceful Shutdown
+
+`sdk.shutdown()` calls `pool.shutdown()` which:
+1. Calls `brain.flush()` on every live instance in parallel
+2. Clears the cache
+3. Clears the pending-init map
+
+Products must call `sdk.shutdown()` from their `SIGTERM` handler:
+
+```typescript
+process.on('SIGTERM', async () => {
+  await sdk.shutdown()
+  process.exit(0)
+})
+```
+
+The `fromLicense()` factory also calls `pool.shutdown()` as part of its shutdown
+sequence (billing buffer flush → heartbeat stop → pool flush).
+
+---
+
+## Decision 7: Storage Path Conventions
+
+| Strategy | Path |
+|----------|------|
+| `per-user` | `{dataPath}/{emailHash}/{workspaceId}/` |
+| `per-tenant` | `{dataPath}/{tenantSlug}/` |
+| `per-scope` | Determined by caller's `factory` |
+
+`emailHash` is an 8-character SHA-256 hex prefix of the lowercase+trimmed email
+address, computed by `computeEmailHash()`. Using a hash rather than the raw email
+avoids filesystem encoding issues and obscures PII from directory listings.
+
+All paths are created with `fs.mkdir(path, { recursive: true })` at init time — the
+pool does not require pre-existing directories.
+
+---
+
+## Decision 8: `BrainyInitializingError` for Non-Blocking Cold Start
+
+Products that prefer to return HTTP 503 during cold start (rather than blocking the
+client for 25–30 seconds) can pass a non-blocking flag and catch
+`BrainyInitializingError`:
+
+```typescript
+app.onError((err, c) => {
+  if (err instanceof BrainyInitializingError) {
+    return c.json({ error: 'Service starting', retryAfter: err.retryAfter }, 503, {
+      'Retry-After': String(err.retryAfter),
+    })
+  }
+})
+```
+
+This is optional — the default pool behaviour blocks the request until init completes,
+which is correct for Workshop (users expect their workspace to load, even if slowly).
+Venue may prefer the 503 pattern to avoid long-hanging HTTP requests on the first
+tenant request after a cold deploy.
+
+---
+
+## Related Documents
+
+- `ADR-001-sdk-design.md` — Overall SDK architecture (Decision 3: instance strategies)
+- `ADR-002-transport-protocol.md` — How clients connect to the pool's Brainy instances
+- `src/server/instance-pool.ts` — Full implementation
+- `src/server/from-license.ts` — License activation and `.soulcraft.json` write (step 3 above)

package/docs/IMPLEMENTATION-PLAN.md

@@ -1,6 +1,6 @@
 # @soulcraft/sdk — Detailed Implementation Plan
 
-**Status:** Ready to implement
+**Status:** Complete as of v1.4.6
 **Decisions:** See `ADR-001-sdk-design.md` for all architecture decisions and rationale.
 **Research basis:** Full cross-project exploration of Workshop, Venue, Academy, brainy-client,
 and @soulcraft/auth conducted 2026-02-27.
@@ -38,11 +38,6 @@ A new session MUST read the source file(s) before implementing each module.
 
 All items done. Repo created, ADR written, scaffold committed.
 
-Remaining scaffolding items (minor, do at start of Phase 2):
-- [ ] `vitest.config.ts` — test runner config
-- [ ] `.env.example` — document any env vars the SDK needs
-- [ ] Update `src/modules/*/types.ts` placeholder comments as modules are implemented
-
 ---
 
 ## Phase 2 — Core Data Layer
@@ -634,48 +629,35 @@ renderEmailTemplate(template: string, data: Record<string, unknown>): string
 
 ## Phase 6 — npm Publish + Migrate All Products
 
-### 6a. Pre-publish checklist
-- [ ] All tests passing (`bun test`)
-- [ ] TypeScript builds cleanly (`bun run check`)
-- [ ] No stubs, no TODOs, no `as any`
-- [ ] JSDoc on every exported symbol
-- [ ] Module-level `@module` block on every file
-- [ ] `package.json` version set to `1.0.0`
-- [ ] `README.md` written with usage examples for all three entry points
+### 6a. Pre-publish checklist ✅ Complete (v1.4.6)
+- [x] All tests passing (`bun test`) — 330 tests, 21 suites
+- [x] TypeScript builds cleanly (`bun run check`)
+- [x] No stubs, no TODOs, no `as any`
+- [x] JSDoc on every exported symbol
+- [x] Module-level `@module` block on every file
+- [x] Published as `@soulcraft/sdk@1.4.6` (first published as `1.0.0`, now at `1.4.6`)
+- [x] `README.md` written with usage examples for all three entry points
 
 ### 6b. Publish
 ```bash
 npm publish --access restricted
 ```
 
-### 6c. Migrate Workshop
-Files to update in `/media/dpsifr/storage/home/Projects/workshop/`:
-1. `package.json` — add `@soulcraft/sdk`, remove `@soulcraft/brainy-client`, `@soulcraft/auth`
-2. `build.sh` — add `sdk` to bundle loop, remove `brainy-client` and `auth`
-3. `auth/better-auth.ts` — replace with `createProductAuth('workshop', ...)` from SDK
-4. `server-hono.ts` — replace brainy-client handler factories with SDK handlers
-5. `brainy/brainy-singleton.ts` — replace with `sdk.server.instancePool`
-6. `brainy/remote-brainy.ts` — replace with SDK `WsTransport`
-7. `brainy/usage-tracking.ts` — replace with `sdk.license.credits.*` + `sdk.billing.*`
-8. All `import { ... } from '@soulcraft/auth'` → `from '@soulcraft/sdk'`
-9. All `import { createBrainyClient } from '@soulcraft/brainy-client'` → `from '@soulcraft/sdk/client'`
-
-### 6d. Migrate Venue
-Files to update in `/home/dpsifr/Projects/venue/`:
-1. `apps/web/package.json` — same package swap
-2. `apps/web/server.ts` — replace brainy-client WS handler with SDK
-3. `apps/web/src/lib/server/auth.ts` — replace with `createProductAuth('venue', ...)`
-4. `apps/web/src/routes/api/brainy/rpc/+server.ts` — replace with SDK HTTP handler
-5. All notification code → `sdk.notifications.*`
-
-### 6e. Migrate Academy
-Files to update in `/media/dpsifr/storage/home/Projects/academy/`:
-1. `package.json`, `auth/better-auth.ts`, `server.ts` — same patterns as Workshop
+### 6c. Migrate Workshop ✅ Complete
+Workshop migrated to `@soulcraft/sdk@1.4.6`. Uses `fromLicense({ product: 'workshop' })`.
+All brainy-client and auth imports replaced. Bundle loop updated.
+
+### 6d. Migrate Venue ⏳ In progress
+Venue has open actions in the handoff file — `fromLicense()` migration pending.
+See `/home/dpsifr/.strategy/PLATFORM-HANDOFF.md`.
+
+### 6e. Migrate Academy ✅ Complete
+Academy migrated to `@soulcraft/sdk@1.4.6`.
 
 ### 6f. Post-migration
-- Deprecate `@soulcraft/brainy-client` on npmjs.com (`npm deprecate`)
-- Deprecate `@soulcraft/auth` on npmjs.com
-- Update `PLATFORM-HANDOFF.md` thread to RESOLVED
+- `@soulcraft/brainy-client` — deprecated on npmjs.com ✅
+- `@soulcraft/auth` — deprecated on npmjs.com ✅
+- Handoff thread: resolved for Workshop and Academy; Venue still pending
 
 ---