@holokai/neuron-sdk 0.1.0

package/README.md

@@ -0,0 +1,628 @@
+# @holokai/neuron-sdk
+
+SDK for building neurons — processes that connect to the BigBrain gateway and execute tasks on behalf of workflows.
+
+A neuron advertises one or more **capabilities** (typed task handlers), opens a persistent connection to the gateway, and receives work via SSE. The gateway routes tasks to the right neuron based on capability type and scope. The built-in `apps/worker` uses this SDK; so will any external neuron (desktop app, on-prem agent, custom integration).
+
+---
+
+## Installation
+
+```sh
+pnpm add @holokai/neuron-sdk
+# or: npm install @holokai/neuron-sdk
+# or: yarn add @holokai/neuron-sdk
+```
+
+Runs in **Node 20+** (uses the global `fetch` and `ReadableStream`) and **modern browsers** (Chromium, Firefox, Safari latest). The package has no `node:*` imports — Holokai Desktop / Electron and the in-browser neuron-tester both consume the same SDK build.
+
+---
+
+## Quick start
+
+```ts
+import { Neuron } from '@holokai/neuron-sdk';
+
+const neuron = new Neuron({
+  gatewayUrl: 'https://api.example.com',
+  neuronId: 'my-worker-001',            // stable per process/device
+  auth: () => process.env.GATEWAY_TOKEN!, // or an async refresh function
+});
+
+neuron.handle(
+  { type: 'core/script.python', scope: 'any', concurrency: 4 },
+  async (input, ctx) => {
+    ctx.log.info({}, 'running python script');
+    const result = await runPython(input as { code: string }, ctx.signal);
+    return { output: result };
+  },
+);
+
+await neuron.start();
+
+process.on('SIGTERM', () => neuron.stop({ drain: true, timeoutMs: 30_000 }));
+```
+
+---
+
+## Core concepts
+
+### Capability
+
+A capability is a named contract a neuron can fulfill. It has:
+
+| Field | Type | Description |
+|---|---|---|
+| `type` | `string` | Namespaced identifier: `{owner}/{package}[/{name}][.{variant}]` |
+| `scope` | `'any' \| { onBehalfOf: userId }` | Who this handler serves |
+| `concurrency` | `number` | Max parallel tasks for this capability |
+
+Examples:
+- `core/script.python` — server-side Python execution, any user
+- `desktop/mcp/drive.read` — user's Google Drive, scoped to that user
+- `org/01HBQ8ZK/custom-tool` — org-specific capability
+
+### Namespace authorization
+
+The gateway enforces which identities may advertise which namespaces:
+
+| Namespace prefix | Permitted advertisers | Required scope |
+|---|---|---|
+| `core/*` | Server service identities | `any` |
+| `desktop/*` | Attested desktop binaries | `{ onBehalfOf: userId }` |
+| `org/{orgId}/*` | Tenant-installed extensions | `any` within org |
+| `user/{userId}/*` | Personal extensions | `{ onBehalfOf: userId }` |
+
+Registration is rejected (HTTP 403) if the JWT is not authorized for the namespace.
+
+### Scope
+
+- **`'any'`** — the neuron handles tasks for any user. Shared queue; no per-user binding.
+- **`{ onBehalfOf: userId }`** — the neuron handles tasks for that specific user only. The JWT's user binding must match; the gateway enforces this at registration time.
+
+One neuron may advertise mixed-scope capabilities:
+
+```ts
+neuron.handle({ type: 'desktop/mcp/drive.read', scope: { onBehalfOf: 'user:abc' }, concurrency: 1 }, driveHandler);
+neuron.handle({ type: 'core/script.python',     scope: 'any',                       concurrency: 4 }, pythonHandler);
+```
+
+### Lease
+
+A lease is how the gateway delivers a task. When the broker dispatches a task to a neuron, the gateway:
+1. Writes a DB row with `leaseId` and expiry.
+2. Sends a `lease` frame over SSE.
+
+The SDK validates input, invokes your handler, and sends either `ack` (success) or `nack` (failure) over HTTPS POST. Until the ack arrives, the message remains unacknowledged in the broker — if the neuron disconnects, the broker redelivers automatically.
+
+**Handlers must be idempotent.** A task that completed a side effect but failed to ack will be retried. Use capability-native idempotency keys (e.g., Google Drive `requestId`, HTTP `Idempotency-Key`).
+
+### Neuron ID
+
+`neuronId` must be **stable across restarts** for the same process/device. The gateway uses it to correlate reconnects. Pick something meaningful and persistent:
+
+```ts
+// Server process
+neuronId: `worker-${process.env.HOSTNAME}-${process.pid}`
+
+// Desktop app — persist a UUID to disk
+neuronId: await readOrCreateStoredId('/var/lib/myapp/neuron-id')
+```
+
+---
+
+## API reference
+
+### `new Neuron(opts: NeuronOptions)`
+
+```ts
+interface NeuronOptions {
+  gatewayUrl: string;           // Base URL, e.g. 'https://api.example.com'
+  neuronId: string;             // Stable per process/device
+  auth: AuthCallback;           // Token provider (see below)
+  logger?: NeuronLogger;        // Plug in pino, console, etc. Default: silent
+  heartbeatIntervalMs?: number; // Default 10_000 (10s)
+  initialReconnectDelayMs?: number; // Default 500ms
+  maxReconnectDelayMs?: number;     // Default 30_000 (30s)
+  fetch?: typeof fetch;         // Override for tests
+}
+```
+
+---
+
+### `neuron.handle(reg, handler)`
+
+Register a handler for a capability. Must be called **before** `neuron.start()`.
+
+```ts
+neuron.handle(reg: CapabilityRegistration, handler: Handler): void
+```
+
+```ts
+interface CapabilityRegistration {
+  type: string;                   // Namespaced capability type
+  scope: 'any' | { onBehalfOf: string };
+  concurrency: number;            // Max parallel executions
+  inputSchema?: JsonSchema;       // Optional; validated before invoking handler
+  outputSchema?: JsonSchema;      // Optional; validated before acking
+}
+
+type Handler<TInput = unknown, TOutput = unknown> = (
+  input: TInput,
+  ctx: HandlerContext,
+) => Promise<TOutput> | TOutput;
+```
+
+If `inputSchema` is provided and the lease input fails validation, the handler is never called — the SDK sends a `schema-mismatch` nack automatically. Same for `outputSchema` on the return value.
+
+---
+
+### `neuron.start()`
+
+Opens the SSE stream, posts `register` with all advertised capabilities, and starts the heartbeat loop.
+
+```ts
+await neuron.start(): Promise<void>
+```
+
+Throws if no handlers have been registered. Subsequent calls are no-ops.
+
+---
+
+### `neuron.stop(opts?)`
+
+Graceful shutdown: refuse new leases, drain in-flight handlers, disconnect.
+
+```ts
+await neuron.stop({ drain?: boolean, timeoutMs?: number }): Promise<void>
+```
+
+| Option | Default | Description |
+|---|---|---|
+| `drain` | `true` | Wait for in-flight handlers to finish |
+| `timeoutMs` | `30_000` | After this, abort remaining in-flight tasks |
+
+Queued leases (received but not yet started due to concurrency limits) are nacked `retryable` on shutdown so the broker redelivers immediately.
+
+---
+
+### `neuron.updateCapability(reg)`
+
+Change concurrency for an already-registered capability at runtime. Useful when available resources change (e.g., desktop plugged into power).
+
+```ts
+await neuron.updateCapability({ type: string, concurrency: number }): Promise<void>
+```
+
+Notifies the gateway via `update-capability` so it can tune AMQP prefetch. Safe to call while tasks are in flight — in-flight tasks finish at the old concurrency; new slots open at the new limit.
+
+---
+
+### `HandlerContext`
+
+The second argument to every handler:
+
+```ts
+interface HandlerContext {
+  task: TaskPayload;    // Full task metadata
+  leaseId: string;      // Gateway lease ID (for logging/debugging)
+  signal: AbortSignal;  // Fires on gateway cancel or SDK shutdown
+  log: NeuronLogger;    // Logger scoped to this lease
+  progress(update: { percent?: number; message?: string; data?: unknown }): void;
+  fail(reason: string, opts?: { code?: string }): never;
+}
+```
+
+#### `ctx.signal`
+
+Pass this to any fetch/HTTP calls so they abort cleanly:
+
+```ts
+const resp = await fetch(url, { signal: ctx.signal });
+```
+
+When the gateway sends a `cancel` frame (workflow was cancelled by the user), `ctx.signal` fires and your handler should throw/return promptly. An `AbortError` from a cancelled signal is treated as `nack(kind='cancelled')` — benign, not counted as a failure.
+
+#### `ctx.progress()`
+
+Report progress for long-running tasks. Non-blocking; safe to call many times.
+
+```ts
+ctx.progress({ percent: 50, message: 'halfway done' });
+ctx.progress({ data: { rowsProcessed: 1200 } });
+```
+
+#### `ctx.fail()`
+
+Terminal failure — nacks the lease with `kind='terminal'` (no retry). Use when the task can never succeed regardless of retries.
+
+```ts
+if (!validApiKey) {
+  ctx.fail('invalid API key', { code: 'INVALID_API_KEY' });
+  // unreachable — fail() throws internally
+}
+```
+
+---
+
+### `AuthCallback`
+
+```ts
+type AuthCallback = () => Promise<string> | string;
+```
+
+Called on every POST and SSE open. The SDK retries once on 401, so implement token refresh inside the callback:
+
+```ts
+let token = await fetchInitialToken();
+
+const auth = async () => {
+  if (isExpired(token)) {
+    token = await refreshToken();
+  }
+  return token.accessToken;
+};
+```
+
+---
+
+### `NeuronLogger`
+
+```ts
+interface NeuronLogger {
+  debug(meta: Record<string, unknown>, msg?: string): void;
+  info(meta: Record<string, unknown>, msg?: string): void;
+  warn(meta: Record<string, unknown>, msg?: string): void;
+  error(meta: Record<string, unknown>, msg?: string): void;
+}
+```
+
+Matches pino's call signature directly. To use pino:
+
+```ts
+import pino from 'pino';
+const logger = pino({ level: 'info' });
+const neuron = new Neuron({ ..., logger });
+```
+
+---
+
+### `defineCapability()`
+
+A typed helper that produces a capability definition from zod schemas, with handler inputs/outputs fully inferred:
+
+```ts
+import { defineCapability, Neuron } from '@holokai/neuron-sdk';
+import { z } from 'zod';
+
+const pythonCap = defineCapability({
+  type: 'core/script.python',
+  description: 'Run a Python script in a sandbox',
+  input: z.object({ code: z.string(), args: z.record(z.unknown()).optional() }),
+  output: z.object({ stdout: z.string(), exitCode: z.number() }),
+  scope: 'any',          // optional default; overridable at handle() time
+  concurrency: 4,        // optional default; overridable at handle() time
+});
+
+// Handler input/output inferred — no manual type annotations:
+neuron.handle(pythonCap, async (input, ctx) => {
+  // input: { code: string; args?: Record<string, unknown> }
+  return { stdout: '...', exitCode: 0 };
+});
+
+// Or override defaults per deployment:
+neuron.handle(pythonCap, { scope: { onBehalfOf: 'user-123' }, concurrency: 8 }, fn);
+```
+
+Validation uses the **original zod schemas** (so `.refine()`, `.transform()`, `.preprocess()`, `.brand()` all run as expected). The JSON Schema produced by `zod-to-json-schema` is shipped on the wire to the gateway and planner but is not used as the runtime validator — that avoids silently dropping zod features that don't translate.
+
+The fall-back `neuron.handle(registration, fn)` shape (raw JSON Schema + capability fields) is still accepted for low-level callers that don't have zod schemas on hand.
+
+---
+
+### Events
+
+`Neuron` extends `EventEmitter` from [`eventemitter3`](https://github.com/primus/eventemitter3) — a browser-clean, type-safe drop-in for Node's `EventEmitter`. Subscribe with `.on()`, `.once()`, or `.off()`:
+
+```ts
+const off = neuron.on('frame', (frame) => {
+  log.append(frame); // see every parsed SSE frame
+});
+off(); // unsubscribe
+
+neuron.once('connection:registered', ({ sessionId }) => {
+  console.log(`registered as ${sessionId}`);
+});
+
+neuron.on('connection:reconnecting', ({ attempt, delayMs, reason }) => {
+  statusBar.show(`reconnecting (attempt ${attempt}, ${delayMs}ms)`, reason);
+});
+```
+
+The full event map:
+
+| Event | Payload | When |
+|---|---|---|
+| `connection:open` | _none_ | SSE handshake succeeded; before `register` is posted |
+| `connection:registered` | `{ sessionId: string; capabilities: number }` | Gateway accepted the register frame |
+| `connection:close` | `reason: string` | SSE stream ended (clean shutdown or transient error) |
+| `connection:reconnecting` | `{ attempt, delayMs, reason }` | Reconnect scheduled, before backoff delay |
+| `error` | `err: Error` | Transport / parse error (reconnect handling is automatic) |
+| `frame` | `frame: ServerToNeuronFrame` | Pass-through of every parsed inbound SSE frame |
+
+The `frame` event is a firehose mirror of the gateway's `/neuron/events` SSE stream. It fires for **every** frame including ones the SDK handles internally (`lease`, `cancel`) — useful for debug UIs, observability, and audit logs. The high-level `neuron.handle()` API is implemented as a `frame` listener filtered to `lease` frames, so `handle()` and direct `frame` subscribers compose without conflict.
+
+Subscribe **before** calling `neuron.start()` if you care about events that fire as part of the connection sequence (`connection:open`, the initial `registered` frame). Late subscribers will still see subsequent events.
+
+The event surface is designed to grow — future entries (e.g. `'workflow:task-completed'` for cross-neuron notifications) slot in without breaking subscribers.
+
+---
+
+## Examples
+
+### Simple any-scoped capability
+
+```ts
+import { Neuron } from '@holokai/neuron-sdk';
+
+const neuron = new Neuron({
+  gatewayUrl: process.env.GATEWAY_URL!,
+  neuronId: `worker-${process.env.HOSTNAME}`,
+  auth: () => process.env.GATEWAY_TOKEN!,
+});
+
+neuron.handle(
+  { type: 'core/http', scope: 'any', concurrency: 8 },
+  async (input, ctx) => {
+    const { url, method = 'GET', headers = {}, body } = input as {
+      url: string;
+      method?: string;
+      headers?: Record<string, string>;
+      body?: string;
+    };
+
+    const resp = await fetch(url, { method, headers, body, signal: ctx.signal });
+    const text = await resp.text();
+
+    return { status: resp.status, body: text };
+  },
+);
+
+await neuron.start();
+```
+
+### User-scoped capability (desktop / MCP)
+
+```ts
+const userId = await getUserIdFromToken(authToken);
+
+neuron.handle(
+  { type: 'desktop/mcp/drive.read', scope: { onBehalfOf: userId }, concurrency: 1 },
+  async (input, ctx) => {
+    const { fileId } = input as { fileId: string };
+    ctx.progress({ message: 'fetching file metadata' });
+
+    const file = await driveClient.files.get({ fileId, signal: ctx.signal });
+    ctx.progress({ percent: 50, message: 'downloading content' });
+
+    const content = await driveClient.files.download({ fileId, signal: ctx.signal });
+    return { name: file.name, mimeType: file.mimeType, content };
+  },
+);
+```
+
+### Long-running task with progress
+
+```ts
+neuron.handle(
+  { type: 'core/data.transform', scope: 'any', concurrency: 2 },
+  async (input, ctx) => {
+    const { rows } = input as { rows: unknown[] };
+    const results: unknown[] = [];
+
+    for (let i = 0; i < rows.length; i++) {
+      if (ctx.signal.aborted) break; // respect cancellation
+
+      results.push(await processRow(rows[i]));
+      ctx.progress({ percent: Math.round(((i + 1) / rows.length) * 100) });
+    }
+
+    return { results };
+  },
+);
+```
+
+### Error handling
+
+```ts
+neuron.handle(
+  { type: 'core/llm', scope: 'any', concurrency: 4 },
+  async (input, ctx) => {
+    const { prompt, model } = input as { prompt: string; model: string };
+
+    let response;
+    try {
+      response = await llmClient.complete({ prompt, model, signal: ctx.signal });
+    } catch (err) {
+      if ((err as Error).name === 'AbortError') {
+        throw err; // let the SDK handle cancellation
+      }
+      if ((err as any).status === 400) {
+        // Bad request — no point retrying
+        ctx.fail(`invalid LLM request: ${(err as Error).message}`, { code: 'INVALID_REQUEST' });
+      }
+      // Anything else: throw to get a retryable nack
+      throw err;
+    }
+
+    return { text: response.text, tokensUsed: response.usage.total };
+  },
+);
+```
+
+### Graceful shutdown
+
+```ts
+await neuron.start();
+
+const shutdown = async (signal: string) => {
+  console.log(`${signal} received — draining...`);
+  await neuron.stop({ drain: true, timeoutMs: 30_000 });
+  process.exit(0);
+};
+
+process.on('SIGTERM', () => shutdown('SIGTERM'));
+process.on('SIGINT',  () => shutdown('SIGINT'));
+```
+
+### Runtime concurrency update
+
+```ts
+import { powerMonitor } from 'electron'; // or any power/resource API
+
+powerMonitor.on('on-ac', async () => {
+  await neuron.updateCapability({ type: 'core/script.python', concurrency: 8 });
+});
+
+powerMonitor.on('on-battery', async () => {
+  await neuron.updateCapability({ type: 'core/script.python', concurrency: 1 });
+});
+```
+
+---
+
+## Configuration reference
+
+| Option | Default | Notes |
+|---|---|---|
+| `gatewayUrl` | — | Base URL, no trailing slash |
+| `neuronId` | — | Stable per device/process; reused on reconnect |
+| `auth` | — | Called on every POST + SSE open |
+| `logger` | silent no-op | Plug in pino or console |
+| `heartbeatIntervalMs` | `10_000` | Sent every 10s; renews lease expiries |
+| `initialReconnectDelayMs` | `500` | First reconnect delay |
+| `maxReconnectDelayMs` | `30_000` | Reconnect delay ceiling (exponential backoff) |
+
+---
+
+## Protocol details
+
+This section covers enough to debug; the wire-level frame schemas are bundled into the SDK and exported alongside the public types.
+
+### Transport
+
+- **Server → Neuron**: SSE stream at `GET /neuron/events?neuronId={id}`
+- **Neuron → Server**: HTTPS POST endpoints (one per frame type)
+
+No WebSockets — SSE + HTTPS POST traverses enterprise proxies cleanly.
+
+### Frame types
+
+**Neuron → Server (POST):**
+
+| Frame | Path | When |
+|---|---|---|
+| `register` | `/neuron/register` | On SSE open |
+| `update-capability` | `/neuron/update-capability` | On concurrency change |
+| `heartbeat` | `/neuron/heartbeat` | Every 10s while connected |
+| `ack` | `/neuron/ack` | Handler succeeded |
+| `nack` | `/neuron/nack` | Handler failed |
+| `progress` | `/neuron/progress` | During handler execution |
+
+**Server → Neuron (SSE):**
+
+| Frame | When |
+|---|---|
+| `registered` | Gateway accepted the register frame; effective capabilities returned |
+| `lease` | Task to execute |
+| `cancel` | Gateway cancels a specific lease (workflow cancelled) |
+| `capability-revoked` | A capability was deauthorized mid-session |
+| `disconnect` | Gateway is closing the connection cleanly |
+
+### Heartbeat
+
+Every ~10s the SDK posts:
+
+```json
+{
+  "type": "heartbeat",
+  "neuronId": "worker-host-1234",
+  "inFlight": [
+    { "leaseId": "01HB...", "taskId": "task-abc", "startedAt": "2024-01-01T12:00:00Z" }
+  ],
+  "sentAt": "2024-01-01T12:00:10Z"
+}
+```
+
+The gateway renews lease expiries for every `leaseId` in `inFlight` in a single DB round-trip. **Only running leases appear in `inFlight`** — queued leases (admitted but waiting for a concurrency slot) are excluded so the reaper doesn't mistakenly extend their TTL.
+
+### Lease lifecycle
+
+```
+Gateway                                   Neuron SDK
+  │                                           │
+  │── lease (SSE) ──────────────────────────▶ │
+  │                                           │ validate input schema
+  │                                           │ wait for pool slot
+  │                                           │ invoke handler
+  │◀── heartbeats (POST every 10s) ────────── │
+  │                                           │ handler returns
+  │                                           │ validate output schema
+  │◀── ack (POST) ────────────────────────── │
+  │
+  │  [or on failure]
+  │◀── nack(kind='retryable') ────────────── │  broker redelivers
+  │◀── nack(kind='terminal') ─────────────── │  workflow fails
+  │◀── nack(kind='schema-mismatch') ──────── │  structured error surfaced
+  │◀── nack(kind='cancelled') ────────────── │  benign; signal fired
+```
+
+### Reconnection
+
+The SDK uses exponential backoff (default: 500ms initial, doubling, 30s ceiling). On reconnect:
+1. SSE stream re-opens.
+2. `register` is posted with the full capability list.
+3. Heartbeat loop resumes.
+
+In-flight leases from before the disconnect were never acked — the broker redelivers them. The reaper flips stale `assigned` DB rows back to `pending`. Your handlers will receive the same task again; make them idempotent.
+
+---
+
+## Troubleshooting
+
+### `Neuron.handle() must be called before start()`
+
+Register all capabilities before calling `start()`. For runtime changes, use `updateCapability()`.
+
+### `Neuron.start() requires at least one handler`
+
+You must call `neuron.handle(...)` at least once before `start()`.
+
+### Registration rejected with HTTP 403
+
+Your JWT is not authorized for the namespace you're trying to advertise. Check:
+- The capability type uses the correct namespace prefix (`core/*` requires a service identity, not a user JWT).
+- For user-scoped capabilities, the `onBehalfOf` value matches the user in your JWT.
+
+### Tasks not being received
+
+1. Check the `registered` log line — it shows the **effective** capabilities after the gateway's policy/auth filters. If a capability is missing, the gateway filtered it.
+2. Check `neuronId` — if it changes across restarts, the gateway may have stale session state.
+3. Check heartbeat logs — if heartbeats are failing, the lease reaper may be reclaiming tasks faster than they're running.
+
+### Handler keeps getting re-delivered
+
+The handler completed but the `ack` POST failed (gateway returned 4xx or 5xx). Look for `ack POST rejected by gateway` in logs. Root causes:
+- Output failed the `outputSchema` check — fix your handler's return shape.
+- Gateway was restarting — the next delivery will succeed once it's back.
+
+### Tasks stuck in `assigned` state
+
+Heartbeats aren't reaching the gateway, so the reaper hasn't reclaimed them. Check:
+- Is the SSE stream still connected? (`transport reconnecting` log entries)
+- Are heartbeat POSTs failing? (`heartbeat post failed` log entries)
+- Is `heartbeatIntervalMs` misconfigured to a very large value?
+
+### Auth 401 on every request
+
+The `auth` callback isn't returning a valid token. If using a refresh flow, ensure the callback fetches a fresh token on each call (the SDK caches nothing — it's your callback's job).