@zyno-io/dk-server-foundation 26.216.430

package/docs/content/mesh-service.md

@@ -0,0 +1,204 @@
+# MeshService
+
+Typed RPC between distributed application instances. Each node gets a unique integer ID and can invoke handlers on any other node (or itself) with full type safety. Uses Redis pub/sub for messaging and sorted sets for node tracking.
+
+## Usage
+
+### Define a message map
+
+```typescript
+import { MeshService, MeshMessageMap } from '@zyno-io/dk-server-foundation';
+
+type MyMessages = {
+    getStatus: { request: { verbose: boolean }; response: { status: string; uptime: number } };
+    invalidateCache: { request: { keys: string[] }; response: { cleared: number } };
+};
+```
+
+### Create and start a node
+
+```typescript
+const mesh = new MeshService<MyMessages>('my-app');
+
+mesh.registerHandler('getStatus', async data => {
+    return { status: 'ok', uptime: process.uptime() };
+});
+
+mesh.registerHandler('invalidateCache', async data => {
+    const cleared = await cache.delete(data.keys);
+    return { cleared };
+});
+
+await mesh.start();
+console.log(`Node started with instance ID: ${mesh.instanceId}`);
+```
+
+### Invoke a handler on another node
+
+```typescript
+// Call a specific node by its instance ID
+const result = await mesh.invoke(targetInstanceId, 'getStatus', { verbose: true });
+console.log(result.status); // fully typed
+
+// Calling your own instance ID routes directly to the local handler (no pub/sub)
+const local = await mesh.invoke(mesh.instanceId, 'getStatus', { verbose: false });
+```
+
+### List nodes in the mesh
+
+```typescript
+const nodes = await mesh.getNodes();
+for (const node of nodes) {
+    console.log(`${node.instanceId} @ ${node.hostname}${node.self ? ' (self)' : ''}`);
+}
+// 1 @ web-server-01
+// 2 @ web-server-02 (self)
+// 3 @ web-server-03
+```
+
+### Track node departures
+
+```typescript
+mesh.setNodeCleanedUpCallback(async instanceId => {
+    console.log(`Node ${instanceId} left the mesh`);
+    // Clean up resources associated with that node
+});
+```
+
+### Graceful shutdown
+
+```typescript
+await mesh.stop();
+```
+
+## API
+
+### `new MeshService<T extends MeshMessageMap>(key: string, options?: MeshServiceOptions)`
+
+Creates a new mesh node.
+
+- **`key`** -- Logical mesh name. All nodes using the same key form one mesh. Different keys are fully independent.
+- **`options`** -- Optional tuning parameters (see below).
+
+### `MeshServiceOptions`
+
+| Option                | Type                   | Default | Description                                                                                                                |
+| --------------------- | ---------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------- |
+| `heartbeatIntervalMs` | `number`               | `5000`  | How often this node refreshes its heartbeat in the registry.                                                               |
+| `nodeTtlMs`           | `number`               | `15000` | How long a node can go without a heartbeat before the leader removes it.                                                   |
+| `requestTimeoutMs`    | `number`               | `10000` | Default timeout for remote invocations. Reset on each heartbeat from the handler, so long-running handlers won't time out. |
+| `leaderOptions`       | `LeaderServiceOptions` | —       | Options passed to the internal `LeaderService` used for cleanup leader election.                                           |
+
+### Properties
+
+| Property     | Type     | Description                                                             |
+| ------------ | -------- | ----------------------------------------------------------------------- |
+| `instanceId` | `number` | Unique integer ID assigned to this node on `start()`. `0` before start. |
+
+### Methods
+
+#### `registerHandler<K>(type: K, handler: (data: T[K]['request']) => T[K]['response'] | Promise<T[K]['response']>): void`
+
+Register a handler for a message type. Handlers can be registered before or after `start()`. Registering a handler for a type that already has one replaces it.
+
+#### `invoke<K>(instanceId: number, type: K, data: T[K]['request']): Promise<T[K]['response']>`
+
+Send a typed request to a specific node and wait for the response.
+
+- If `instanceId` matches the local node, the handler is called directly (no pub/sub).
+- If the target node doesn't exist or doesn't respond, the promise rejects with `MeshRequestTimeoutError`.
+- If the target has no handler for the type, rejects with `MeshNoHandlerError`.
+- If the handler throws, rejects with `MeshHandlerError` containing the error message.
+
+#### `getNodes(): Promise<MeshNode[]>`
+
+Returns all live nodes in the mesh. Each `MeshNode` contains:
+
+```typescript
+interface MeshNode {
+    instanceId: number; // The node's unique integer ID
+    hostname: string; // The OS hostname of the machine running the node
+    self: boolean; // true if this is the calling node
+}
+```
+
+Only nodes with active heartbeats are returned. Stopped or expired nodes are excluded.
+
+#### `setNodeCleanedUpCallback(cb: (instanceId: number) => void | Promise<void>): void`
+
+Register a callback invoked when the leader detects and removes an expired node. Only fires on the current leader instance. Errors in the callback are logged but don't affect cleanup of other nodes.
+
+#### `start(): Promise<void>`
+
+Join the mesh. Acquires a unique instance ID, subscribes to its pub/sub channel, registers in the heartbeat set, and starts leader election. Throws if already running.
+
+#### `stop(): Promise<void>`
+
+Leave the mesh. Stops heartbeats, rejects all pending outbound requests with `Error('MeshService stopped')`, unsubscribes from pub/sub, and removes itself from the heartbeat registry. Safe to call before `start()` or multiple times.
+
+## Error Classes
+
+### `MeshRequestTimeoutError`
+
+The target node did not respond within the timeout period. This can happen if the target has crashed, is unreachable, or the Redis pub/sub connection is disrupted.
+
+### `MeshHandlerError`
+
+The target node's handler threw an error. The `message` property contains the original error message from the remote handler.
+
+### `MeshNoHandlerError`
+
+No handler is registered for the requested message type -- either locally (direct invocation) or on the remote node.
+
+## Architecture
+
+### Node Registry
+
+Nodes are tracked in a Redis sorted set (`{prefix}:mesh:{key}:heartbeats`) where the score is the last heartbeat timestamp (from Redis server time, avoiding clock skew) and the member is the instance ID. Node metadata (hostname) is stored in a Redis hash (`{prefix}:mesh:{key}:nodes`).
+
+Unique instance IDs are assigned via `INCR` on `{prefix}:mesh:{key}:next_id`. Both the heartbeat entry and the nodes hash entry are removed on graceful stop or leader-driven cleanup of expired nodes.
+
+### Messaging
+
+Each node subscribes to its own pub/sub channel: `{prefix}:mesh:{key}:node:{instanceId}`.
+
+Three message types flow over these channels:
+
+| Type          | Direction         | Purpose                                                  |
+| ------------- | ----------------- | -------------------------------------------------------- |
+| **Request**   | Caller -> Handler | `{ requestId, senderInstanceId, type, data, timeoutMs }` |
+| **Response**  | Handler -> Caller | `{ requestId, reply: true, data?, error? }`              |
+| **Heartbeat** | Handler -> Caller | `{ requestId, heartbeat: true }`                         |
+
+### Request Heartbeats
+
+While a handler is executing, the handling node sends periodic heartbeat messages back to the caller (every `requestTimeoutMs * 0.75`). The caller resets its timeout timer on each heartbeat. This allows short absolute timeouts while supporting arbitrarily long handler execution.
+
+The handler uses the **caller's** `requestTimeoutMs` (sent in the request message) for its heartbeat interval, so mixed-timeout configurations work correctly.
+
+### Leader Election
+
+Each mesh uses an internal `LeaderService` (key: `mesh:{key}`) for leader election. The leader is responsible for:
+
+1. Running the cleanup Lua script on each heartbeat cycle
+2. Removing expired nodes from the sorted set
+3. Invoking the `nodeCleanedUpCallback` for each removed node
+
+Only one node (the leader) performs cleanup at any time.
+
+### Local Invocation
+
+When `invoke` is called with the node's own instance ID, the handler is called directly as a function call -- no serialization, no pub/sub, no timeout. Errors propagate as-is.
+
+## Configuration
+
+The Redis connection is configured via environment variables with the `MESH_REDIS_` prefix (falls back to `REDIS_`):
+
+| Variable                   | Description                                                  |
+| -------------------------- | ------------------------------------------------------------ |
+| `MESH_REDIS_HOST`          | Redis host                                                   |
+| `MESH_REDIS_PORT`          | Redis port                                                   |
+| `MESH_REDIS_PREFIX`        | Key prefix (falls back to `REDIS_PREFIX`, then package name) |
+| `MESH_REDIS_SENTINEL_HOST` | Sentinel host (optional)                                     |
+| `MESH_REDIS_SENTINEL_PORT` | Sentinel port (optional)                                     |
+| `MESH_REDIS_SENTINEL_NAME` | Sentinel master name (optional)                              |

package/docs/content/redis.md

@@ -0,0 +1,168 @@
+# Redis
+
+Utilities for Redis-backed caching, distributed locking, and inter-process communication.
+
+## Client Creation
+
+```typescript
+import { createRedis, createRedisOptions } from '@zyno-io/dk-server-foundation';
+
+// Create with default config (REDIS_HOST, REDIS_PORT)
+const { client, prefix } = createRedis();
+
+// Create with specific config prefix
+const { client, prefix } = createRedis('CACHE_REDIS');
+```
+
+Each call creates a separate connection. All clients are tracked and can be disconnected with `disconnectAllRedis()`.
+
+## Cache
+
+Redis-backed cache with TTL support:
+
+```typescript
+import { Cache } from '@zyno-io/dk-server-foundation';
+
+// String values
+await Cache.set('key', 'value', 3600); // TTL in seconds
+const value = await Cache.get('key');
+
+// Object values (auto JSON serialization)
+await Cache.setObj('user:123', { name: 'Alice', role: 'admin' }, 3600);
+const user = await Cache.getObj<{ name: string; role: string }>('user:123');
+```
+
+| Method                        | Description                                    |
+| ----------------------------- | ---------------------------------------------- |
+| `Cache.get(key)`              | Get a string value (returns `null` if missing) |
+| `Cache.set(key, val, ttl)`    | Set a string value with TTL in seconds         |
+| `Cache.getObj<T>(key)`        | Get and deserialize a JSON value               |
+| `Cache.setObj(key, val, ttl)` | Serialize and set a JSON value with TTL        |
+
+Uses `CACHE_REDIS_*` environment variables (falls back to `REDIS_*`).
+
+## Distributed Mutex
+
+Redis-backed distributed locking for coordinating work across multiple instances:
+
+```typescript
+import { withMutex, withMutexes, MutexAcquisitionError } from '@zyno-io/dk-server-foundation';
+
+// Single mutex
+const result = await withMutex({
+    key: ['user', userId],
+    fn: async didWait => {
+        // Critical section - only one instance executes at a time
+        // didWait: true if we had to wait for the lock
+        return await processPayment(userId);
+    },
+    retryCount: 30, // Max retry attempts (default: 30)
+    retryDelay: 1000, // Delay between retries in ms (default: 1000)
+    renewInterval: 1000 // Lock renewal interval in ms (default: 1000)
+});
+
+// Multiple mutexes (acquired in order)
+const result = await withMutexes({
+    keys: [
+        ['wallet', walletA],
+        ['wallet', walletB]
+    ],
+    fn: async didWait => {
+        return await transferFunds(walletA, walletB, amount);
+    }
+});
+```
+
+### Keys
+
+Mutex keys can be primitives or arrays: `['wallet', 123]` becomes `wallet:123`.
+
+### Error Handling
+
+Throws `MutexAcquisitionError` if the lock cannot be acquired within the retry window.
+
+### Options
+
+| Option          | Type                               | Default | Description                            |
+| --------------- | ---------------------------------- | ------- | -------------------------------------- |
+| `key` / `keys`  | `MutexKey`                         | —       | Lock key(s) — string, number, or array |
+| `fn`            | `(didWait: boolean) => Promise<T>` | —       | Function to execute under lock         |
+| `retryCount`    | `number`                           | `30`    | Max retry attempts                     |
+| `retryDelay`    | `number`                           | `1000`  | Delay between retries (ms)             |
+| `renewInterval` | `number`                           | `1000`  | Lock renewal interval (ms)             |
+
+### Modes
+
+Configure via `MUTEX_MODE`:
+
+- **`redis`** (default) — Uses Redis Lua scripts for atomic acquire/release/renew. Suitable for multi-instance deployments.
+- **`local`** — In-process locking with no Redis dependency. Useful for single-instance or testing scenarios.
+
+Uses `MUTEX_REDIS_*` environment variables (falls back to `REDIS_*`).
+
+### DevConsole Integration
+
+Active mutexes and acquisition history are visible in the [DevConsole](./devconsole.md) Mutex view.
+
+## Broadcast Channels
+
+Redis pub/sub for inter-process communication:
+
+```typescript
+import { createBroadcastChannel } from '@zyno-io/dk-server-foundation';
+
+// Create a typed broadcast channel
+const channel = createBroadcastChannel<{ userId: string; action: string }>('user-events');
+
+channel.subscribe(data => {
+    console.log(`${data.userId}: ${data.action}`);
+});
+
+channel.publish({ userId: '123', action: 'login' });
+```
+
+### Distributed Methods
+
+Execute a function locally and broadcast to all instances:
+
+```typescript
+import { createDistributedMethod } from '@zyno-io/dk-server-foundation';
+
+class CacheManager {
+    invalidate = createDistributedMethod<{ key: string }>(
+        {
+            name: 'invalidate',
+            logger: () => this.logger
+        },
+        async data => {
+            this.localCache.delete(data.key);
+        }
+    );
+}
+
+// Calling invalidate() runs locally AND broadcasts to all instances
+await cacheManager.invalidate({ key: 'user:123' });
+```
+
+The `logger` option takes a getter function so it can reference `this.logger` even when used as a class field initializer. If omitted, a default logger scoped to `Distributed:<name>` is used.
+
+Uses `BROADCAST_REDIS_*` environment variables (falls back to `REDIS_*`).
+
+## Configuration
+
+All Redis utilities support independent connection configuration via environment variable prefixes:
+
+| Utility   | Env Prefix          | Fallback  |
+| --------- | ------------------- | --------- |
+| Cache     | `CACHE_REDIS_*`     | `REDIS_*` |
+| Mutex     | `MUTEX_REDIS_*`     | `REDIS_*` |
+| Broadcast | `BROADCAST_REDIS_*` | `REDIS_*` |
+
+Common variables for each prefix:
+
+| Variable       | Description    | Default     |
+| -------------- | -------------- | ----------- |
+| `*_HOST`       | Redis host     | `127.0.0.1` |
+| `*_PORT`       | Redis port     | `6379`      |
+| `*_DB`         | Redis database | `0`         |
+| `*_KEY_PREFIX` | Key prefix     | —           |

package/docs/content/srpc.md

@@ -0,0 +1,261 @@
+# SRPC (Simple RPC)
+
+Bidirectional RPC over WebSocket with HMAC authentication, ts-proto code generation, multiplexed binary streams, and OpenTelemetry tracing.
+
+## Overview
+
+SRPC uses a prefix-based message routing system. Messages are defined in `.proto` files and compiled to TypeScript with `dksf-gen-proto`. Prefixes determine direction:
+
+- `u*` -- Upstream: client -> server requests
+- `d*` -- Downstream: server -> client requests
+
+## Proto Generation
+
+Define messages in a `.proto` file:
+
+```protobuf
+syntax = "proto3";
+
+message ClientMessage {
+    string requestId = 1;
+    bytes reply = 2;
+    string error = 3;
+    bytes trace = 4;
+    bytes pingPong = 5;
+    bytes byteStreamOperation = 6;
+
+    // Upstream (client -> server)
+    bytes uEcho = 100;
+    bytes uGetUser = 101;
+
+    // Downstream (server -> client)
+    bytes dNotify = 200;
+}
+
+message ServerMessage {
+    string requestId = 1;
+    bytes reply = 2;
+    string error = 3;
+    bytes trace = 4;
+    bytes pingPong = 5;
+    bytes byteStreamOperation = 6;
+
+    bytes uEcho = 100;
+    bytes uGetUser = 101;
+    bytes dNotify = 200;
+}
+
+// Request/response types
+message UEchoRequest { string message = 1; }
+message UEchoResponse { string message = 1; }
+message UGetUserRequest { int32 id = 1; }
+message UGetUserResponse { string name = 1; string email = 2; }
+message DNotifyRequest { string event = 1; }
+message DNotifyResponse { bool acknowledged = 1; }
+```
+
+Generate TypeScript:
+
+```bash
+dksf-gen-proto my-service.proto src/generated/proto/my-service
+```
+
+Options:
+
+| Flag             | Description                        |
+| ---------------- | ---------------------------------- |
+| `--use-date`     | Use `Date` instead of `Timestamp`  |
+| `--use-map-type` | Use `Map` instead of plain objects |
+| `--only-types`   | Generate only type definitions     |
+
+## Server
+
+```typescript
+import { SrpcServer } from '@zyno-io/dk-server-foundation';
+import { ClientMessage, ServerMessage } from './generated/proto/my-service';
+
+const server = new SrpcServer({
+    logger: myLogger,
+    clientMessage: ClientMessage,
+    serverMessage: ServerMessage,
+    wsPath: '/rpc'
+});
+
+// Handle new connections
+server.registerConnectionHandler(async stream => {
+    console.log(`Client connected: ${stream.clientId}`);
+    stream.meta = { userId: stream.clientId };
+});
+
+// Handle upstream messages (client -> server)
+server.registerMessageHandler('uEcho', async (stream, data) => {
+    return { message: data.message };
+});
+
+server.registerMessageHandler('uGetUser', async (stream, data) => {
+    const user = await db.query(User).filter({ id: data.id }).findOne();
+    return { name: user.name, email: user.email };
+});
+
+// Handle disconnections
+server.registerDisconnectHandler(async (stream, cause) => {
+    console.log(`Client ${stream.clientId} disconnected: ${cause}`);
+});
+```
+
+### Invoking Client Methods (Server -> Client)
+
+```typescript
+// Send to a specific client
+const stream = server.streamsById.get(streamId);
+const result = await server.invoke(stream, 'dNotify', { event: 'orderUpdated' });
+
+// Create a reusable invoke function
+const invoke = SrpcServer.createInvoke(() => server);
+await invoke(stream, 'dNotify', { event: 'orderUpdated' }, 5000);
+```
+
+### `ISrpcServerOptions`
+
+| Option          | Type             | Description                            |
+| --------------- | ---------------- | -------------------------------------- |
+| `logger`        | `ScopedLogger`   | Logger instance                        |
+| `clientMessage` | `SrpcMessageFns` | ts-proto generated client message type |
+| `serverMessage` | `SrpcMessageFns` | ts-proto generated server message type |
+| `wsPath`        | `string`         | WebSocket path (e.g., `/rpc`)          |
+| `debug`         | `boolean`        | Enable debug logging                   |
+
+### Server Properties
+
+| Property            | Type                      | Description                     |
+| ------------------- | ------------------------- | ------------------------------- |
+| `streamsById`       | `Map<string, SrpcStream>` | All active streams by stream ID |
+| `streamsByClientId` | `Map<string, SrpcStream>` | Active streams by client ID     |
+
+### Authentication
+
+Default authentication uses HMAC signatures with clock drift tolerance:
+
+```typescript
+// Override authorization logic
+server.setClientAuthorizer(async (clientId, signature, timestamp) => {
+    // Custom auth logic
+    return isValid;
+});
+
+// Provide per-client secrets
+server.setClientKeyFetcher(async clientId => {
+    return await getClientSecret(clientId);
+});
+```
+
+Configure via `SRPC_AUTH_SECRET` and `SRPC_AUTH_CLOCK_DRIFT_MS` (default: 30 seconds).
+
+## Client
+
+```typescript
+import { SrpcClient } from '@zyno-io/dk-server-foundation';
+import { ClientMessage, ServerMessage } from './generated/proto/my-service';
+
+const client = new SrpcClient(
+    logger,
+    'wss://api.example.com/rpc',
+    ClientMessage,
+    ServerMessage,
+    'client-id-123',
+    { role: 'worker' }, // Optional metadata
+    'shared-secret', // Optional auth secret
+    { enableReconnect: true } // Options
+);
+
+client.connect(); // Non-async: initiates connection in background
+
+// Handle downstream messages (server -> client)
+client.registerMessageHandler('dNotify', async data => {
+    console.log(`Event: ${data.event}`);
+    return { acknowledged: true };
+});
+
+// Invoke upstream messages (client -> server)
+const result = await client.invoke('uEcho', { message: 'hello' });
+console.log(result.message); // 'hello'
+
+// Connection lifecycle
+client.registerConnectionHandler(async () => {
+    /* connected */
+});
+client.registerDisconnectHandler(async cause => {
+    /* disconnected */
+});
+
+// Check connection status
+if (client.isConnected) {
+    /* ... */
+}
+
+// Disconnect (non-async: closes connection immediately)
+client.disconnect();
+```
+
+### `SrpcClientOptions`
+
+| Option            | Type      | Default | Description                  |
+| ----------------- | --------- | ------- | ---------------------------- |
+| `enableReconnect` | `boolean` | `true`  | Auto-reconnect on disconnect |
+
+## Binary Streams
+
+`SrpcByteStream` provides multiplexed binary streaming over existing SRPC connections. Streams are `Duplex` node streams with backpressure support.
+
+```typescript
+// Sender side
+const sender = SrpcByteStream.createSender(stream);
+sender.write(buffer);
+sender.end();
+
+// Receiver side (via handler)
+const receiver = SrpcByteStream.createReceiver(stream, streamId);
+receiver.on('data', chunk => {
+    /* process binary data */
+});
+receiver.on('end', () => {
+    /* stream finished */
+});
+```
+
+### Constants
+
+| Constant                     | Value     | Description                                 |
+| ---------------------------- | --------- | ------------------------------------------- |
+| `HIGH_WATER_MARK`            | 256 KB    | WebSocket buffer threshold for backpressure |
+| `PENDING_RECEIVER_MAX_BYTES` | 2 MB      | Max buffer before receiver is created       |
+| `PENDING_RECEIVER_TTL_MS`    | 5 seconds | Timeout for pending receiver data           |
+
+## Disconnect Causes
+
+```typescript
+type SrpcDisconnectCause = 'disconnect' | 'duplicate' | 'timeout' | 'badArg';
+```
+
+| Cause        | Description                                |
+| ------------ | ------------------------------------------ |
+| `disconnect` | Normal disconnection                       |
+| `duplicate`  | Another connection with the same client ID |
+| `timeout`    | Heartbeat timeout                          |
+| `badArg`     | Invalid connection arguments               |
+
+## Error Handling
+
+```typescript
+import { SrpcError } from '@zyno-io/dk-server-foundation';
+
+// Throw user-facing errors in handlers
+throw new SrpcError('Invalid input', true); // isUserError: true
+
+// Non-user errors are logged but return generic message to client
+throw new SrpcError('Internal failure', false);
+```
+
+## OpenTelemetry Integration
+
+SRPC automatically propagates trace context between client and server. Spans are created for each RPC call with the message prefix as the span name.