@zyno-io/ts-server-foundation 26.712.36

package/docs/content/mesh-client.md

@@ -0,0 +1,333 @@
+# Mesh Client Tracking
+
+Track clients connected across multiple backend nodes and invoke operations on any client regardless of which node it's connected to. Built on top of [MeshService](./mesh-service.md).
+
+Three layers, each building on the previous:
+
+1. **MeshClientRegistry** — tracks which clients are connected where, with metadata
+2. **MeshClientService** — adds transparent cross-node client invocation
+3. **MeshSrpcServer** — extends SrpcServer with auto-registration, lifecycle callbacks, and distributed invoke
+
+## MeshClientRegistry
+
+Track which clients are connected to which node, with arbitrary metadata.
+
+```typescript
+import { MeshClientRegistry, MeshClientRedisRegistry } from '@zyno-io/ts-server-foundation';
+
+interface ClientMeta {
+    userId: string;
+    role: string;
+}
+
+// Usually you don't construct this manually — MeshClientService and MeshSrpcServer create it for you.
+// But if you need standalone tracking:
+const backend = new MeshClientRedisRegistry<ClientMeta>('my-app');
+const registry = new MeshClientRegistry<ClientMeta>(mesh.instanceId, backend);
+
+await registry.register('client-123', { userId: 'user-1', role: 'admin' });
+
+const client = await registry.getClient('client-123');
+// { clientId: 'client-123', nodeId: 1, connectedAt: 1710000000000, metadata: { userId: 'user-1', role: 'admin' } }
+
+const all = await registry.listClients();
+const local = await registry.listClientsForNode(mesh.instanceId);
+
+// Update metadata (ownership-safe: only updates if this node owns the registration)
+const updated = await registry.updateMetadata('client-123', {
+    userId: 'user-1',
+    role: 'superadmin'
+});
+
+// Ownership-safe: only removes if this node owns the registration
+const removed = await registry.unregister('client-123'); // true if removed, false if client moved
+```
+
+The `MeshClientRegistryBackend` interface is pluggable — implement your own for database-backed tracking:
+
+```typescript
+import type { MeshClientRegistryBackend, RegisteredClient, RegisterResult } from '@zyno-io/ts-server-foundation';
+
+class DatabaseClientRegistry<TMeta> implements MeshClientRegistryBackend<TMeta> {
+    async register(clientId: string, nodeId: number, metadata: TMeta, allowSupersede = true): Promise<RegisterResult> {
+        // Atomically create an active registration, return a conflict when another
+        // node owns it and supersession is disabled, or report the superseded node.
+        throw new Error('Implement active registration');
+    }
+    async reserve(clientId: string, nodeId: number, metadata: TMeta, allowSupersede = true): Promise<RegisterResult> {
+        // Pending reservations must remain hidden from lookup/list operations.
+        throw new Error('Implement pending reservation');
+    }
+    async activate(clientId: string, nodeId: number, metadata: TMeta): Promise<boolean> {
+        // Promote this node's pending reservation to active without changing ownership.
+        throw new Error('Implement reservation activation');
+    }
+    async unregister(clientId: string, nodeId: number): Promise<boolean> {
+        throw new Error('Implement ownership-safe removal');
+    }
+    async updateMetadata(clientId: string, nodeId: number, metadata: TMeta): Promise<boolean> {
+        throw new Error('Implement ownership-safe metadata update');
+    }
+    async getClient(clientId: string): Promise<RegisteredClient<TMeta> | undefined> {
+        throw new Error('Implement active lookup');
+    }
+    async listClients(): Promise<RegisteredClient<TMeta>[]> {
+        throw new Error('Implement active listing');
+    }
+    async listClientsForNode(nodeId: number): Promise<RegisteredClient<TMeta>[]> {
+        throw new Error('Implement active listing by node');
+    }
+    async cleanupNode(nodeId: number): Promise<RegisteredClient<TMeta>[]> {
+        throw new Error('Implement ownership-safe node cleanup');
+    }
+}
+```
+
+### API
+
+#### `new MeshClientRegistry<TMeta>(nodeId: number, backend: MeshClientRegistryBackend<TMeta>)`
+
+Creates a registry bound to a specific mesh node ID.
+
+#### `register(clientId, metadata, allowSupersede?)` → `Promise<RegisterResult>`
+
+Register an active client on this node. A `connectedAt` timestamp (epoch ms) is automatically recorded. The result is `{ status: 'ok', supersededNodeId }` when registration succeeds or `{ status: 'conflict', ownerNodeId }` when another node owns the client and supersession is disabled. `ownerNodeId` can be `null` if the owner disappears during the conflict/readback race. The built-in Redis backend allows supersession by default.
+
+#### `reserve(clientId, metadata, allowSupersede?)` → `Promise<RegisterResult>`
+
+Atomically reserve ownership without making the client visible to `getClient()`, list operations, or invocation. The result has the same conflict/supersession shape as `register()`.
+
+#### `activate(clientId, metadata)` → `Promise<boolean>`
+
+Promote a pending reservation owned by this node to an active registration and store its final metadata. It returns `false` if the reservation disappeared or ownership moved to another node.
+
+#### `unregister(clientId)` → `Promise<boolean>`
+
+Remove a client registration. Returns `true` if the client was owned by this node and was removed. Returns `false` if the client had already reconnected to a different node (ownership-safe).
+
+#### `updateMetadata(clientId, metadata)` → `Promise<boolean>`
+
+Update metadata for a registered client. Returns `true` if the client was owned by this node and was updated. Returns `false` if the client is not registered or has moved to a different node (ownership-safe).
+
+#### `getClient(clientId)` → `Promise<RegisteredClient<TMeta> | undefined>`
+
+Look up a client by ID across all nodes. The returned `RegisteredClient` includes `clientId`, `nodeId`, `connectedAt` (epoch ms), and `metadata`.
+
+#### `listClients()` → `Promise<RegisteredClient<TMeta>[]>`
+
+List all registered clients across all nodes.
+
+#### `listClientsForNode(nodeId?)` → `Promise<RegisteredClient<TMeta>[]>`
+
+List clients for a specific node. Defaults to this registry's node.
+
+#### `cleanupNode(nodeId?)` → `Promise<RegisteredClient<TMeta>[]>`
+
+Remove all clients for a node, returning the orphaned clients (with metadata). Only removes clients still owned by that node — clients that reconnected elsewhere are left intact.
+
+---
+
+## MeshClientService
+
+Combines MeshClientRegistry with MeshService for transparent cross-node client invocation. You provide a `clientInvokeFn` for delivery to a client connected to this node; it is called for both local invocations and requests forwarded from another node.
+
+```typescript
+import { MeshClientService } from '@zyno-io/ts-server-foundation';
+
+// Define broadcast types for type-safe broadcasting
+interface MyBroadcasts {
+    configUpdated: { keys: string[] };
+}
+
+const clientService = new MeshClientService<ClientMeta, MyBroadcasts>({
+    key: 'my-app',
+    clientInvokeFn: async (clientId, type, data, timeoutMs) => {
+        // Deliver to a client connected to this node, whether the caller is local or remote.
+        return localDelivery(clientId, type, data);
+    }
+});
+
+await clientService.start();
+
+await clientService.registerClient('client-123', { userId: 'user-1', role: 'admin' });
+
+// Update metadata after registration (ownership-safe)
+await clientService.updateClientMetadata('client-123', { userId: 'user-1', role: 'superadmin' });
+
+// Invoke on any client — routes through mesh if on a different node
+const result = await clientService.invoke('client-123', 'notify', { text: 'hello' });
+
+// Broadcast to all nodes
+clientService.registerBroadcastHandler('configUpdated', (data, senderInstanceId) => {
+    console.log(`Config updated by node ${senderInstanceId}:`, data.keys);
+});
+await clientService.broadcast('configUpdated', { keys: ['feature-flag-x'] });
+
+const clients = await clientService.clientRegistry.listClients();
+
+await clientService.stop();
+```
+
+### API
+
+#### `new MeshClientService<TMeta, TBroadcasts>(options)`
+
+| Option               | Type                                                     | Description                                                                                   |
+| -------------------- | -------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
+| `key`                | `string`                                                 | Mesh key (internally namespaced as `_mc:{key}`)                                               |
+| `meshOptions`        | `MeshServiceOptions`                                     | Optional tuning for the internal mesh node                                                    |
+| `registryBackend`    | `MeshClientRegistryBackend`                              | Optional custom backend (defaults to `MeshClientRedisRegistry`)                               |
+| `clientInvokeFn`     | `(clientId, type, data, timeoutMs?) => Promise<unknown>` | Delivers local clients for both local and forwarded invocations                               |
+| `clientUpdateMetaFn` | `(clientId, metadata) => boolean`                        | Optional owning-node hook that applies a cross-node metadata update to the live local client. |
+
+#### Properties
+
+| Property         | Type                        | Description                   |
+| ---------------- | --------------------------- | ----------------------------- |
+| `instanceId`     | `number`                    | This node's mesh instance ID  |
+| `clientRegistry` | `MeshClientRegistry<TMeta>` | Direct access to the registry |
+
+#### Methods
+
+| Method                                                                     | Description                                                            |
+| -------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
+| `start()`                                                                  | Start the internal mesh and initialize the registry                    |
+| `stop()`                                                                   | Clean up own clients, stop the mesh                                    |
+| `registerClient(clientId, metadata, allowSupersede?)` → `Promise<boolean>` | Register on this node; false means a non-superseded ownership conflict |
+| `reserveClient(clientId, metadata, allowSupersede?)` → `Promise<boolean>`  | Reserve ownership without exposing the client until activation         |
+| `activateClient(clientId, metadata)` → `Promise<boolean>`                  | Promote this node's reservation to active                              |
+| `unregisterClient(clientId)` → `Promise<boolean>`                          | Unregister (returns false if client moved elsewhere)                   |
+| `updateClientMetadata(clientId, metadata)` → `Promise<boolean>`            | Update metadata (returns false if client moved)                        |
+| `invoke(clientId, type, data, timeoutMs?)`                                 | Invoke on any client, routes automatically                             |
+| `registerBroadcastHandler(type, handler)`                                  | Register a handler for a broadcast type                                |
+| `broadcast(type, data, options?)`                                          | Broadcast to all nodes in the mesh                                     |
+
+Before `start()`, registration and reservation are accepted as no-op compatibility calls, activation/update/unregister return `false`, and `invoke()` throws `ClientNotFoundError`. Callers that need discoverable state must await `start()` first. Local invocation calls `clientInvokeFn` directly. Remote invocation forwards to the owning node and preserves `ClientDisconnectedError`; other delivery failures become `ClientInvocationError`. Remote metadata updates run `clientUpdateMetaFn` on the owner and persist the accepted metadata in the registry.
+
+---
+
+## MeshSrpcServer
+
+Extends `SrpcServer` with mesh client tracking. Single class — no need to create an SrpcServer separately.
+
+```typescript
+import { MeshSrpcServer } from '@zyno-io/ts-server-foundation';
+import { ClientMessage, ServerMessage } from './generated/proto';
+
+const server = new MeshSrpcServer({
+    // SrpcServer options
+    logger,
+    clientMessage: ClientMessage,
+    serverMessage: ServerMessage,
+    wsPath: '/srpc',
+
+    // Mesh options
+    meshKey: 'my-app'
+});
+
+// Register SRPC handlers as usual
+server.registerMessageHandler('uEcho', async (stream, data) => {
+    return { message: `Echo: ${data.message}` };
+});
+
+// Lifecycle callbacks
+server.onClientConnected((clientId, metadata) => {
+    console.log(`Client ${clientId} connected`);
+    db.updatePresence(clientId, 'online');
+});
+
+server.onClientDisconnected((clientId, metadata) => {
+    // Only fires if the client actually left — NOT if it reconnected to another node
+    console.log(`Client ${clientId} disconnected`);
+    db.updatePresence(clientId, 'offline');
+});
+
+server.onNodeClientsOrphaned((nodeId, clients) => {
+    // Fires on the leader when a dead node is cleaned up
+    // Only includes clients that didn't reconnect elsewhere
+    console.log(`Node ${nodeId} died, ${clients.length} orphaned clients`);
+    for (const client of clients) {
+        db.updatePresence(client.clientId, 'offline');
+    }
+});
+
+await server.meshStart();
+
+// Update client metadata at any time (ownership-safe)
+await server.updateClientMetadata('client-123', { ...metadata, role: 'superadmin' });
+
+// Type-safe invoke on any client, regardless of which node
+await server.invoke('client-123', 'dNotify', { text: 'hello' });
+
+// Broadcast to all nodes (uses MeshService broadcast under the hood)
+// Add a TBroadcasts generic to the server for type-safe broadcasts:
+//   new MeshSrpcServer<Meta, ClientMsg, ServerMsg, RegistryMeta, MyBroadcasts>(...)
+server.registerBroadcastHandler('configUpdated', (data, senderInstanceId) => {
+    console.log(`Config updated by node ${senderInstanceId}:`, data);
+});
+await server.broadcast('configUpdated', { keys: ['feature-flag-x'] });
+
+// Access the registry
+const allClients = await server.clientRegistry.listClients();
+
+// Shutdown
+await server.meshStop();
+server.close();
+```
+
+### API
+
+#### Constructor
+
+```typescript
+new MeshSrpcServer(options: ISrpcServerOptions & MeshSrpcServerOptions)
+```
+
+`MeshSrpcServerOptions`:
+
+| Option            | Type                        | Description                                    |
+| ----------------- | --------------------------- | ---------------------------------------------- |
+| `meshKey`         | `string`                    | Mesh key                                       |
+| `meshOptions`     | `MeshServiceOptions`        | Optional mesh tuning                           |
+| `registryBackend` | `MeshClientRegistryBackend` | Optional custom backend                        |
+| `extractMetadata` | `(stream) => TRegistryMeta` | Optional metadata extraction from SRPC streams |
+
+#### Properties
+
+| Property         | Type                                | Description                   |
+| ---------------- | ----------------------------------- | ----------------------------- |
+| `meshInstanceId` | `number`                            | This node's mesh instance ID  |
+| `clientRegistry` | `MeshClientRegistry<TRegistryMeta>` | Direct access to the registry |
+
+#### Methods
+
+| Method                                       | Description                                                                               |
+| -------------------------------------------- | ----------------------------------------------------------------------------------------- |
+| `meshStart()`                                | Start mesh client tracking                                                                |
+| `meshStop()`                                 | Stop mesh client tracking (call before `close()`)                                         |
+| `updateClientMetadata(clientId, metadata)`   | Update metadata (returns false if client moved); also updates local cache                 |
+| `invoke(clientId, prefix, data, timeoutMs?)` | Type-safe invoke on any client across any node                                            |
+| `registerBroadcastHandler(type, handler)`    | Register a handler for a broadcast type (see [MeshService broadcasts](./mesh-service.md)) |
+| `broadcast(type, data, options?)`            | Broadcast to all nodes in the mesh                                                        |
+| `onClientConnected(handler)`                 | Fires on the node the client connected to                                                 |
+| `onClientDisconnected(handler)`              | Fires on the node the client disconnected from                                            |
+| `onNodeClientsOrphaned(handler)`             | Fires on the **leader node** when a dead node's clients are cleaned up                    |
+
+Plus all `SrpcServer` methods: `registerMessageHandler`, `registerConnectionHandler`, `registerDisconnectHandler`, `setClientAuthorizer`, etc.
+
+`MeshSrpcServer` reserves client ownership before SRPC activation, so pending connections never appear in active registry lookups. After activation, top-level assignments and deletions on `stream.meta` are batched in a microtask and synchronized to the registry. Nested mutations are not observable through the proxy; replace the top-level value or call `updateClientMetadata()`. Explicit metadata updates currently merge supplied keys into the live `stream.meta` object while replacing the registry metadata value, so callers that need deletion parity should delete top-level stream keys directly. Calls for a remote client route through the mesh to the owning SRPC stream. Connected/disconnected callbacks are serialized per client, and a stale disconnect from a same-node replacement does not unregister the replacement or publish an offline callback.
+
+## Current Limits
+
+- The built-in Redis registry applies a 24-hour safety TTL to its client hash and per-node sets. Registry writes refresh those keys, but mesh heartbeats currently do not; registrations in a completely quiet mesh can therefore expire while clients remain connected.
+
+---
+
+## Error Classes
+
+| Error                     | When                                                                  |
+| ------------------------- | --------------------------------------------------------------------- |
+| `ClientNotFoundError`     | `invoke()` called with a clientId not in the registry                 |
+| `ClientDisconnectedError` | Client was in the registry but no longer connected on the target node |
+| `ClientInvocationError`   | Remote delivery failed (wraps the original error message)             |
+| `MeshRequestTimeoutError` | The remote node didn't respond to the mesh forwarding request         |

package/docs/content/mesh-service.md

@@ -0,0 +1,255 @@
+# MeshService
+
+Typed RPC between distributed application instances. Each node gets a unique integer ID and can invoke handlers on any other node (or itself) with full type safety. Uses Redis pub/sub for messaging and sorted sets for node tracking.
+
+## Usage
+
+### Define a message map
+
+```typescript
+import { MeshService, MeshMessageMap } from '@zyno-io/ts-server-foundation';
+
+type MyMessages = {
+    getStatus: { request: { verbose: boolean }; response: { status: string; uptime: number } };
+    invalidateCache: { request: { keys: string[] }; response: { cleared: number } };
+};
+```
+
+### Create and start a node
+
+```typescript
+const mesh = new MeshService<MyMessages>('my-app');
+
+mesh.registerHandler('getStatus', async data => {
+    return { status: 'ok', uptime: process.uptime() };
+});
+
+mesh.registerHandler('invalidateCache', async data => {
+    const cleared = await cache.delete(data.keys);
+    return { cleared };
+});
+
+await mesh.start();
+console.log(`Node started with instance ID: ${mesh.instanceId}`);
+```
+
+### Invoke a handler on another node
+
+```typescript
+// Call a specific node by its instance ID
+const result = await mesh.invoke(targetInstanceId, 'getStatus', { verbose: true });
+console.log(result.status); // fully typed
+
+// Calling your own instance ID routes directly to the local handler (no pub/sub)
+const local = await mesh.invoke(mesh.instanceId, 'getStatus', { verbose: false });
+
+// Per-request timeout (overrides the service-level default)
+const fast = await mesh.invoke(targetInstanceId, 'getStatus', { verbose: false }, 2000);
+```
+
+### List nodes in the mesh
+
+```typescript
+const nodes = await mesh.getNodes();
+for (const node of nodes) {
+    console.log(`${node.instanceId} @ ${node.hostname}${node.self ? ' (self)' : ''}`);
+}
+// 1 @ web-server-01
+// 2 @ web-server-02 (self)
+// 3 @ web-server-03
+```
+
+### Track node departures
+
+```typescript
+mesh.setNodeCleanedUpCallback(async instanceId => {
+    console.log(`Node ${instanceId} left the mesh`);
+    // Clean up resources associated with that node
+});
+```
+
+### Broadcast to all nodes
+
+Fire-and-forget messages to every node in the mesh. Unlike `invoke`, broadcasts have no response.
+
+```typescript
+import { MeshService, MeshBroadcastMap } from '@zyno-io/ts-server-foundation';
+
+type MyBroadcasts = {
+    configUpdated: { keys: string[] };
+    userLoggedOut: { userId: string };
+};
+
+const mesh = new MeshService<MyMessages, MyBroadcasts>('my-app');
+
+// Register handlers for broadcast types
+mesh.registerBroadcastHandler('configUpdated', (data, senderInstanceId) => {
+    console.log(`Config update from node ${senderInstanceId}:`, data.keys);
+    reloadConfig(data.keys);
+});
+
+await mesh.start();
+
+// Broadcast to all nodes (including self)
+await mesh.broadcast('configUpdated', { keys: ['feature-flags'] });
+
+// Skip self-delivery
+await mesh.broadcast('userLoggedOut', { userId: '123' }, { skipSelf: true });
+```
+
+### Graceful shutdown
+
+```typescript
+await mesh.stop();
+```
+
+## API
+
+### `new MeshService<T extends MeshMessageMap, B extends MeshBroadcastMap>(key: string, options?: MeshServiceOptions)`
+
+Creates a new mesh node.
+
+- **`key`** -- Logical mesh name. All nodes using the same key form one mesh. Different keys are fully independent.
+- **`options`** -- Optional tuning parameters (see below).
+
+### `MeshServiceOptions`
+
+| Option                | Type                   | Default | Description                                                                                                                |
+| --------------------- | ---------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------- |
+| `heartbeatIntervalMs` | `number`               | `5000`  | How often this node refreshes its heartbeat in the registry.                                                               |
+| `nodeTtlMs`           | `number`               | `15000` | How long a node can go without a heartbeat before the leader removes it.                                                   |
+| `requestTimeoutMs`    | `number`               | `10000` | Default timeout for remote invocations. Reset on each heartbeat from the handler, so long-running handlers won't time out. |
+| `leaderOptions`       | `LeaderServiceOptions` | —       | Options passed to the internal `LeaderService` used for cleanup leader election.                                           |
+
+### Properties
+
+| Property     | Type     | Description                                                             |
+| ------------ | -------- | ----------------------------------------------------------------------- |
+| `instanceId` | `number` | Unique integer ID assigned to this node on `start()`. `0` before start. |
+
+### Methods
+
+#### `registerHandler<K>(type: K, handler: (data: T[K]['request']) => T[K]['response'] | Promise<T[K]['response']>): void`
+
+Register a handler for a message type. Handlers can be registered before or after `start()`. Registering a handler for a type that already has one replaces it.
+
+#### `invoke<K>(instanceId: number, type: K, data: T[K]['request'], timeoutMs?: number): Promise<T[K]['response']>`
+
+Send a typed request to a specific node and wait for the response.
+
+- **`timeoutMs`** -- Optional per-request timeout. Falls back to the service-level `requestTimeoutMs` if not provided. The handler-side heartbeat interval automatically adjusts to the caller's timeout.
+- If `instanceId` matches the local node, the handler is called directly (no pub/sub, no timeout).
+- If the target node doesn't exist or doesn't respond, the promise rejects with `MeshRequestTimeoutError`.
+- If the target has no handler for the type, rejects with `MeshNoHandlerError`.
+- If the handler throws, rejects with `MeshHandlerError` containing the error message.
+
+#### `registerBroadcastHandler<K>(type: K, handler: (data: B[K], senderInstanceId: number) => void | Promise<void>): void`
+
+Register a handler for a broadcast message type. Broadcast handlers receive the message data and the sender's instance ID. Errors thrown in handlers are logged but do not propagate.
+
+#### `broadcast<K>(type: K, data: B[K], options?: MeshBroadcastOptions): Promise<void>`
+
+Send a fire-and-forget message to all nodes in the mesh (including self, unless `skipSelf: true`).
+
+| Option     | Type      | Default | Description                         |
+| ---------- | --------- | ------- | ----------------------------------- |
+| `skipSelf` | `boolean` | `false` | Don't deliver the broadcast to self |
+
+#### `getNodes(): Promise<MeshNode[]>`
+
+Returns all live nodes in the mesh. Each `MeshNode` contains:
+
+```typescript
+interface MeshNode {
+    instanceId: number; // The node's unique integer ID
+    hostname: string; // The OS hostname of the machine running the node
+    self: boolean; // true if this is the calling node
+}
+```
+
+The method returns the current heartbeat registry members. Gracefully stopped nodes remove themselves immediately; expired nodes can remain visible until the leader's cleanup pass removes them.
+
+#### `setNodeCleanedUpCallback(cb: (instanceId: number) => void | Promise<void>): void`
+
+Register a callback invoked when the leader detects and removes an expired node. Only fires on the current leader instance. Errors in the callback are logged but don't affect cleanup of other nodes.
+
+#### `start(): Promise<void>`
+
+Join the mesh. Acquires a unique instance ID, subscribes to its pub/sub channel and broadcast channel, registers in the heartbeat set, and starts leader election. Throws if already running.
+
+#### `stop(): Promise<void>`
+
+Leave the mesh. Stops heartbeats, rejects all pending outbound requests with `Error('MeshService stopped')`, unsubscribes from pub/sub, and removes itself from the heartbeat registry. Safe to call before `start()` or multiple times.
+
+## Error Classes
+
+### `MeshRequestTimeoutError`
+
+The target node did not respond within the timeout period. This can happen if the target has crashed, is unreachable, or the Redis pub/sub connection is disrupted.
+
+### `MeshHandlerError`
+
+The target node's handler threw an error. The `message` property contains the original error message from the remote handler.
+
+### `MeshNoHandlerError`
+
+No handler is registered for the requested message type -- either locally (direct invocation) or on the remote node.
+
+## Architecture
+
+### Node Registry
+
+Nodes are tracked in a Redis sorted set (`{prefix}:mesh:{key}:heartbeats`) where the score is the last heartbeat timestamp (from Redis server time, avoiding clock skew) and the member is the instance ID. Node metadata (hostname) is stored in a Redis hash (`{prefix}:mesh:{key}:nodes`).
+
+Unique instance IDs are assigned via `INCR` on `{prefix}:mesh:{key}:next_id`. Both the heartbeat entry and the nodes hash entry are removed on graceful stop or leader-driven cleanup of expired nodes.
+
+### Messaging
+
+Each node subscribes to its own pub/sub channel: `{prefix}:mesh:{key}:node:{instanceId}`.
+
+Three message types flow over these channels:
+
+| Type          | Direction         | Purpose                                                  |
+| ------------- | ----------------- | -------------------------------------------------------- |
+| **Request**   | Caller -> Handler | `{ requestId, senderInstanceId, type, data, timeoutMs }` |
+| **Response**  | Handler -> Caller | `{ requestId, reply: true, data?, error? }`              |
+| **Heartbeat** | Handler -> Caller | `{ requestId, heartbeat: true }`                         |
+
+### Broadcast
+
+All nodes also subscribe to a shared broadcast channel: `{prefix}:mesh:{key}:broadcast`. Broadcasts are fire-and-forget — the sender publishes once and all subscribers receive the message. No response is collected.
+
+Channel routing: the subscriber's `message` handler inspects the channel name to distinguish broadcast messages from point-to-point messages on the instance channel.
+
+### Request Heartbeats
+
+While a handler is executing, the handling node sends periodic heartbeat messages back to the caller (every `requestTimeoutMs * 0.75`). The caller resets its timeout timer on each heartbeat. This allows short absolute timeouts while supporting arbitrarily long handler execution.
+
+The handler uses the **caller's** `requestTimeoutMs` (sent in the request message) for its heartbeat interval, so mixed-timeout configurations work correctly.
+
+### Leader Election
+
+Each mesh uses an internal `LeaderService` (key: `mesh:{key}`) for leader election. The leader is responsible for:
+
+1. Running the cleanup Lua script on each heartbeat cycle
+2. Removing expired nodes from the sorted set
+3. Invoking the `nodeCleanedUpCallback` for each removed node
+
+Only one node (the leader) performs cleanup at any time.
+
+### Local Invocation
+
+When `invoke` is called with the node's own instance ID, the handler is called directly as a function call -- no serialization, no pub/sub, no timeout. Errors propagate as-is.
+
+## Configuration
+
+The Redis connection is configured via environment variables with the `MESH_REDIS_` prefix (falls back to `REDIS_`):
+
+| Variable                   | Description                                                  |
+| -------------------------- | ------------------------------------------------------------ |
+| `MESH_REDIS_HOST`          | Redis host                                                   |
+| `MESH_REDIS_PORT`          | Redis port                                                   |
+| `MESH_REDIS_PREFIX`        | Key prefix (falls back to `REDIS_PREFIX`, then package name) |
+| `MESH_REDIS_SENTINEL_HOST` | Sentinel host (optional)                                     |
+| `MESH_REDIS_SENTINEL_PORT` | Sentinel port (optional)                                     |
+| `MESH_REDIS_SENTINEL_NAME` | Sentinel master name (optional)                              |