@rljson/server 0.0.14 → 0.0.16

package/README.architecture.md

@@ -86,7 +86,9 @@ The `Node` class sits above `Server` and `Client`, bridging `@rljson/network` to
 3. **Manages transport**: Uses injectable factories (`CreateHubTransport`/`CreateClientTransport`) to create the transport layer, keeping the Node class transport-agnostic.
 4. **Agent lifecycle**: An optional `createAgent` factory in `NodeDeps` is called on every `ready` event. The returned `AgentHandle.stop()` is called before the next role transition. This enables application-level wiring (e.g. FsAgent) without circular dependencies.
 5. **Serialized transitions**: Role transitions are queued — a new `role-changed` event waits for the previous transition to complete before starting. This prevents race conditions between teardown and setup.
-6. **Error resilience**: Errors in user-provided code (agent factories, transport factories) are caught and logged. The node continues functioning — a failed transport degrades connectivity but doesn't crash, a failed agent leaves the node's core intact.
+6. **Hub-changed reconnect** (v0.0.14): Subscribes to `NetworkManager`'s `hub-changed` event in addition to `role-changed`. When the hub changes but the node's role stays `client`, the node tears down its connection and reconnects to the new hub. This prevents split-brain scenarios where clients remain attached to a stale hub.
+7. **Clean socket teardown** (v0.0.14): `_tearDownCurrentRole()` explicitly calls `disconnect()` on client sockets before clearing the reference. This prevents orphaned Socket.IO connections from auto-reconnecting to the old hub.
+8. **Error resilience**: Errors in user-provided code (agent factories, transport factories) are caught and logged. The node continues functioning — a failed transport degrades connectivity but doesn't crash, a failed agent leaves the node's core intact.
 
 ```text
 ┌─────────────────────────────────────────┐

package/README.blog.md

@@ -17,3 +17,11 @@ Add posts as Markdown entries in this file (newest last). Keep each post small a
 - Why it matters
 - Links: PRs, docs, demos
 ```
+
+## 2026-03-20 — v0.0.14: Split-brain fix and hub-changed reconnect
+
+- Node class now listens to `hub-changed` events from NetworkManager — clients reconnect when hub changes but role stays `client`
+- `_tearDownCurrentRole()` explicitly disconnects sockets before clearing references — prevents orphaned connections
+- Validated on 4-node Windows test lab: E2E Reports 18 & 19 both score **38/41 passed, 0 failures**
+- Previous Report 17 showed split-brain (two simultaneous hubs, 23/41 passed) — now fully resolved
+- PR: https://github.com/rljson/server/pull/14

package/README.trouble.md

@@ -10,9 +10,42 @@ found in the LICENSE file in the root of this package.
 
 ## Table of contents <!-- omit in toc -->
 
+- [Split-Brain: Clients not reconnecting on hub change (fixed in v0.0.14)](#split-brain-clients-not-reconnecting-on-hub-change-fixed-in-v0014)
 - [Vscode Windows: Debugging is not working](#vscode-windows-debugging-is-not-working)
 - [Test Isolation: Socket.IO event listener accumulation](#test-isolation-socketio-event-listener-accumulation)
 
+## Split-Brain: Clients not reconnecting on hub change (fixed in v0.0.14)
+
+Date: 2026-03-20
+
+**Problem:**
+
+In a 4-node deployment, two nodes simultaneously acted as hub (split-brain). Clients stayed connected to the old hub while a new hub was elected. File sync stopped working because the hub had no real clients.
+
+**Symptoms:**
+
+- E2E Report 17: 23/41 passed, 18 failed
+- Two nodes reporting `role=hub` simultaneously
+- Files written by one hub never appearing on clients
+- File counts diverging between nodes (hub accumulating files, clients stuck)
+
+**Root Cause:**
+
+Two bugs in the `Node` class:
+
+1. **Missing `hub-changed` listener**: Node only subscribed to `role-changed` from NetworkManager. When the hub changed but the node's role stayed `client`, the `role-changed` handler skipped (same role). Clients never reconnected to the new hub.
+
+2. **No socket disconnect on teardown**: `_tearDownCurrentRole()` set `_clientSocket = undefined` without calling `disconnect()`. The orphaned Socket.IO connection kept auto-reconnecting to the old hub (especially with the `socket.connect()` reconnect fix from v0.0.13).
+
+**Solution (v0.0.14):**
+
+1. Added `_onHubChanged` listener that tears down and reconnects when hub changes while role stays `client`
+2. Added explicit `socket.disconnect()` call in `_tearDownCurrentRole()` before clearing the reference
+
+**Validation:**
+
+- E2E Reports 18 & 19: **38/41 passed, 0 failures** on 4-node test lab
+
 ## Vscode Windows: Debugging is not working
 
 Date: 2025-03-08

package/dist/README.architecture.md

@@ -86,7 +86,9 @@ The `Node` class sits above `Server` and `Client`, bridging `@rljson/network` to
 3. **Manages transport**: Uses injectable factories (`CreateHubTransport`/`CreateClientTransport`) to create the transport layer, keeping the Node class transport-agnostic.
 4. **Agent lifecycle**: An optional `createAgent` factory in `NodeDeps` is called on every `ready` event. The returned `AgentHandle.stop()` is called before the next role transition. This enables application-level wiring (e.g. FsAgent) without circular dependencies.
 5. **Serialized transitions**: Role transitions are queued — a new `role-changed` event waits for the previous transition to complete before starting. This prevents race conditions between teardown and setup.
-6. **Error resilience**: Errors in user-provided code (agent factories, transport factories) are caught and logged. The node continues functioning — a failed transport degrades connectivity but doesn't crash, a failed agent leaves the node's core intact.
+6. **Hub-changed reconnect** (v0.0.14): Subscribes to `NetworkManager`'s `hub-changed` event in addition to `role-changed`. When the hub changes but the node's role stays `client`, the node tears down its connection and reconnects to the new hub. This prevents split-brain scenarios where clients remain attached to a stale hub.
+7. **Clean socket teardown** (v0.0.14): `_tearDownCurrentRole()` explicitly calls `disconnect()` on client sockets before clearing the reference. This prevents orphaned Socket.IO connections from auto-reconnecting to the old hub.
+8. **Error resilience**: Errors in user-provided code (agent factories, transport factories) are caught and logged. The node continues functioning — a failed transport degrades connectivity but doesn't crash, a failed agent leaves the node's core intact.
 
 ```text
 ┌─────────────────────────────────────────┐

package/dist/README.blog.md

@@ -17,3 +17,11 @@ Add posts as Markdown entries in this file (newest last). Keep each post small a
 - Why it matters
 - Links: PRs, docs, demos
 ```
+
+## 2026-03-20 — v0.0.14: Split-brain fix and hub-changed reconnect
+
+- Node class now listens to `hub-changed` events from NetworkManager — clients reconnect when hub changes but role stays `client`
+- `_tearDownCurrentRole()` explicitly disconnects sockets before clearing references — prevents orphaned connections
+- Validated on 4-node Windows test lab: E2E Reports 18 & 19 both score **38/41 passed, 0 failures**
+- Previous Report 17 showed split-brain (two simultaneous hubs, 23/41 passed) — now fully resolved
+- PR: https://github.com/rljson/server/pull/14

package/dist/README.trouble.md

@@ -10,9 +10,42 @@ found in the LICENSE file in the root of this package.
 
 ## Table of contents <!-- omit in toc -->
 
+- [Split-Brain: Clients not reconnecting on hub change (fixed in v0.0.14)](#split-brain-clients-not-reconnecting-on-hub-change-fixed-in-v0014)
 - [Vscode Windows: Debugging is not working](#vscode-windows-debugging-is-not-working)
 - [Test Isolation: Socket.IO event listener accumulation](#test-isolation-socketio-event-listener-accumulation)
 
+## Split-Brain: Clients not reconnecting on hub change (fixed in v0.0.14)
+
+Date: 2026-03-20
+
+**Problem:**
+
+In a 4-node deployment, two nodes simultaneously acted as hub (split-brain). Clients stayed connected to the old hub while a new hub was elected. File sync stopped working because the hub had no real clients.
+
+**Symptoms:**
+
+- E2E Report 17: 23/41 passed, 18 failed
+- Two nodes reporting `role=hub` simultaneously
+- Files written by one hub never appearing on clients
+- File counts diverging between nodes (hub accumulating files, clients stuck)
+
+**Root Cause:**
+
+Two bugs in the `Node` class:
+
+1. **Missing `hub-changed` listener**: Node only subscribed to `role-changed` from NetworkManager. When the hub changed but the node's role stayed `client`, the `role-changed` handler skipped (same role). Clients never reconnected to the new hub.
+
+2. **No socket disconnect on teardown**: `_tearDownCurrentRole()` set `_clientSocket = undefined` without calling `disconnect()`. The orphaned Socket.IO connection kept auto-reconnecting to the old hub (especially with the `socket.connect()` reconnect fix from v0.0.13).
+
+**Solution (v0.0.14):**
+
+1. Added `_onHubChanged` listener that tears down and reconnects when hub changes while role stays `client`
+2. Added explicit `socket.disconnect()` call in `_tearDownCurrentRole()` before clearing the reference
+
+**Validation:**
+
+- E2E Reports 18 & 19: **38/41 passed, 0 failures** on 4-node test lab
+
 ## Vscode Windows: Debugging is not working
 
 Date: 2025-03-08

package/dist/server.d.ts

@@ -49,6 +49,19 @@ export interface ServerOptions {
      * Defaults to false (local cache enabled).
      */
     disableLocalCache?: boolean;
+    /**
+     * Interval in milliseconds for application-level health checks.
+     * The server pings each connected client and prunes those that
+     * do not respond within {@link healthCheckTimeoutMs}.
+     * Defaults to 30 000 (30 s). Set to 0 to disable health checks.
+     */
+    healthCheckIntervalMs?: number;
+    /**
+     * Timeout in milliseconds to wait for a health check pong.
+     * Clients that do not respond within this window are pruned.
+     * Defaults to 10 000 (10 s).
+     */
+    healthCheckTimeoutMs?: number;
 }
 export declare class Server extends BaseNode {
     private _route;
@@ -77,6 +90,9 @@ export declare class Server extends BaseNode {
     private _disableLocalCache;
     private _latestRef;
     private _bootstrapHeartbeatTimer?;
+    private _healthCheckIntervalMs;
+    private _healthCheckTimeoutMs;
+    private _healthCheckTimer?;
     private _tornDown;
     constructor(_route: Route, _localIo: Io, _localBs: Bs, options?: ServerOptions);
     /**
@@ -181,6 +197,18 @@ export declare class Server extends BaseNode {
      * Each client's dedup pipeline will filter out refs it already has.
      */
     private _broadcastBootstrapHeartbeat;
+    /**
+     * Starts the periodic health check timer if not already running.
+     * Each cycle sends a ping to every non-broadcast client and waits
+     * for a pong. Clients that do not respond are pruned.
+     */
+    private _startHealthChecks;
+    /**
+     * Sends a health ping to each connected (non-broadcast) client.
+     * If a client does not respond within `_healthCheckTimeoutMs`,
+     * the server force-disconnects and removes it.
+     */
+    private _runHealthCheck;
     /**
      * Starts the periodic bootstrap heartbeat timer if configured
      * and not already running.

package/dist/server.js

@@ -393,9 +393,14 @@ class Client extends BaseNode {
     };
     socket.on("disconnect", disconnectHandler);
     socket.on("connect", reconnectHandler);
+    const healthHandler = (payload) => {
+      sockets.ioUp.emit("__health:pong", { nonce: payload.nonce });
+    };
+    sockets.ioDown.on("__health:ping", healthHandler);
     this._connectionCleanup = () => {
       socket.off("disconnect", disconnectHandler);
       socket.off("connect", reconnectHandler);
+      sockets.ioDown.off("__health:ping", healthHandler);
     };
   }
   /**
@@ -655,6 +660,8 @@ class Server extends BaseNode {
     this._logger = options?.logger ?? noopLogger;
     this._peerInitTimeoutMs = options?.peerInitTimeoutMs ?? 3e4;
     this._disableLocalCache = options?.disableLocalCache ?? false;
+    this._healthCheckIntervalMs = options?.healthCheckIntervalMs ?? 3e4;
+    this._healthCheckTimeoutMs = options?.healthCheckTimeoutMs ?? 1e4;
     this._syncConfig = options?.syncConfig;
     this._refLogSize = options?.refLogSize ?? 1e3;
     this._ackTimeoutMs = options?.ackTimeoutMs ?? options?.syncConfig?.ackTimeoutMs ?? 1e4;
@@ -728,6 +735,10 @@ class Server extends BaseNode {
   // Bootstrap state
   _latestRef;
   _bootstrapHeartbeatTimer;
+  // Health check state
+  _healthCheckIntervalMs;
+  _healthCheckTimeoutMs;
+  _healthCheckTimer;
   _tornDown = false;
   /**
    * Initializes Io and Bs multis on the server.
@@ -786,6 +797,7 @@ class Server extends BaseNode {
       this._registerDisconnectHandler(clientId, ioUp);
       this._sendBootstrap(ioDown);
       this._startBootstrapHeartbeat();
+      this._startHealthChecks();
       this._logger.info("Server", "Client socket added successfully", {
         clientId,
         totalClients: this._clients.size
@@ -836,6 +848,7 @@ class Server extends BaseNode {
     this._registerDisconnectHandler(clientId, ioUp);
     this._sendBootstrap(ioDown);
     this._startBootstrapHeartbeat();
+    this._startHealthChecks();
     this._logger.info("Server", "Broadcast-only socket added", {
       clientId,
       totalClients: this._clients.size
@@ -1034,6 +1047,54 @@ class Server extends BaseNode {
       ioDown.emit(this._events.bootstrap, payload);
     }
   }
+  // ...........................................................................
+  // Health checks
+  // ...........................................................................
+  /**
+   * Starts the periodic health check timer if not already running.
+   * Each cycle sends a ping to every non-broadcast client and waits
+   * for a pong. Clients that do not respond are pruned.
+   */
+  _startHealthChecks() {
+    if (this._healthCheckTimer || this._healthCheckIntervalMs <= 0) return;
+    this._healthCheckTimer = setInterval(() => {
+      this._runHealthCheck();
+    }, this._healthCheckIntervalMs);
+    this._healthCheckTimer.unref();
+  }
+  /**
+   * Sends a health ping to each connected (non-broadcast) client.
+   * If a client does not respond within `_healthCheckTimeoutMs`,
+   * the server force-disconnects and removes it.
+   */
+  _runHealthCheck() {
+    for (const [clientId, { ioUp, ioDown }] of this._clients.entries()) {
+      if (clientId.startsWith("broadcast_")) continue;
+      const nonce = Math.random().toString(36).slice(2);
+      let resolved = false;
+      const handler = (payload) => {
+        if (payload?.nonce !== nonce) return;
+        resolved = true;
+        ioUp.off("__health:pong", handler);
+        clearTimeout(timer);
+      };
+      ioUp.on("__health:pong", handler);
+      const timer = setTimeout(() => {
+        if (resolved) return;
+        ioUp.off("__health:pong", handler);
+        this._logger.warn(
+          "Server.Health",
+          "Client failed health check — pruning",
+          { clientId }
+        );
+        if ("disconnect" in ioUp) {
+          ioUp.disconnect(true);
+        }
+        this.removeSocket(clientId);
+      }, this._healthCheckTimeoutMs);
+      ioDown.emit("__health:ping", { nonce });
+    }
+  }
   /**
    * Starts the periodic bootstrap heartbeat timer if configured
    * and not already running.
@@ -1315,6 +1376,10 @@ class Server extends BaseNode {
       clearInterval(this._bootstrapHeartbeatTimer);
       this._bootstrapHeartbeatTimer = void 0;
     }
+    if (this._healthCheckTimer) {
+      clearInterval(this._healthCheckTimer);
+      this._healthCheckTimer = void 0;
+    }
     this._removeAllListeners();
     for (const cleanup of this._disconnectCleanups.values()) {
       cleanup();
@@ -1569,6 +1634,18 @@ class Node {
         await this._becomeClient();
         break;
     }
+    if (!this._running) return;
+    const networkRole = this._networkManager.getTopology().myRole;
+    if (networkRole !== this._role && networkRole !== "unassigned") {
+      this._logger.info(
+        "Node",
+        `Reconciling stale role: node=${this._role} → network=${networkRole}`
+      );
+      await this._performTransition({
+        previous: this._role,
+        current: networkRole
+      });
+    }
   }
   async _becomeHub() {
     await this._ioMem.init();

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@rljson/server",
-  "version": "0.0.14",
+  "version": "0.0.16",
   "description": "Rljson server description",
   "homepage": "https://github.com/rljson/server",
   "bugs": "https://github.com/rljson/server/issues",