socketduct 0.1.32 → 0.1.34

package/README.md

@@ -111,6 +111,7 @@ hub address because `gatewayId` selects the outbound member. Docker remains
 - [Running Socketduct](docs/running.md)
 - [Docker gateway setup](docs/docker.md)
 - [Docker topology](docs/docker-topology.md)
+- [Transport-set availability](docs/transport-set-availability.md)
 - [Node-compatible socket API design](docs/socket-api.md)
 - [Native reverse gateway](docs/reverse-gateway.md)
 - [Durable compressed diagnostic logs](docs/logging.md)

package/changelog.d/20261026183000-route-class-availability-scope.md

@@ -0,0 +1,3 @@
+A reset for one stream no longer marks the member's whole route class unavailable, so a single unreachable target cannot disable every route class and stall all later opens in a transport set.
+
+Documents the requirement behind it in `docs/transport-set-availability.md`: per-member availability owned by the hub, failover instead of stalling, automatic recovery without operator action, and no polling watchdog.

package/changelog.d/20261026192500-target-scoped-availability.md

@@ -0,0 +1,5 @@
+### Fixed
+
+- Route-class availability failures are now scoped to the exact target that failed. A `GATEWAY_UNAVAILABLE` reset for one target marks only that target on that member, so the transport set stops handing it opens while every other target keeps using the member. The mark is cleared by the next successful open for that same target, so nothing expires on a timer and a member can never be dropped from rotation for machines it can still reach.
+
+  Releasing 0.1.33 without this scoping removed the only signal the set had for failover: with several route classes configured, opens kept landing on members whose route class could not serve the target and the reset surfaced straight to the caller. Measured on the TensorBuzz fleet, `GATEWAY_UNAVAILABLE` build failures went from 0/hour to about 100/hour within minutes of 0.1.33 reaching the running workers.

package/changelog.d/20261026200000-ship-docker-entrypoint.md

@@ -0,0 +1 @@
+Publishes `docker-entrypoint.sh` in the package again, so the committed Dockerfile can build an image from a released tarball.

package/docker-entrypoint.sh

@@ -0,0 +1,11 @@
+#!/bin/sh
+set -e
+
+# Run command with node if the first argument contains a "-" or is not a system command. The last
+# part inside the "{}" is a workaround for the following bug in ash/dash:
+# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=874264
+if [ "${1#-}" != "${1}" ] || [ -z "$(command -v "${1}")" ] || { [ -f "${1}" ] && ! [ -x "${1}" ]; }; then
+  set -- node "$@"
+fi
+
+exec "$@"

package/docs/reverse-gateway.md

@@ -347,6 +347,11 @@ const socket = set.createConnection({host: "docker-a", port: 2375})
 await starting
 ```
 
+Availability of a member is decided by the hub's `GATEWAY_SELECTED`
+announcement and cleared by a successful open; a stream reset never changes
+it. Requirements, the selection model and the failure this prevents are in
+[transport-set availability](transport-set-availability.md).
+
 Route-class sockets expose immutable `routeClass` allocation metadata and an
 undefined `gatewayId`. Exact-gateway sets retain immutable `gatewayId` metadata.
 Callers cannot override either selection per open. Reverse streams remain

package/docs/transport-set-availability.md

@@ -0,0 +1,119 @@
+# Transport-set availability
+
+How `MultiplexedSocketductTransportSet` decides which configured member to use,
+what "down" means for a member, and which component is allowed to say so.
+
+This page states the requirements the model exists to satisfy, then describes
+the mechanism. Requirements come from production failures where a runner fleet
+was reported unreachable although every machine answered.
+
+## Requirements
+
+1. **Never use a member the hub reports down.** If the hub says a route class has
+   no selectable gateway, opens must not be routed to that member while a ready
+   alternative exists.
+2. **A failure scoped to one target must not disable siblings.** Availability is
+   per route class (member). A failure to reach one *target* through one member
+   says nothing about the other targets or the other members.
+3. **Fail over instead of stalling.** If the preferred member is unavailable, an
+   open must go to another ready member rather than waiting for the first one.
+4. **A member that stays down must not block callers indefinitely.** Callers keep
+   their own deadlines; the set must not sleep through them while it waits for a
+   single member to come back.
+5. **Recovery is automatic.** When a member becomes selectable again, later opens
+   use it again with no operator action — no restart, no container removal, no
+   config change.
+6. **Availability is event-driven, not polled.** The state changes on hub
+   announcements and on successful opens. There is no timer that sweeps members
+   and no background watchdog that decides health on its own.
+7. **Marks are attributable and bounded.** A mark is only set by evidence that
+   names the member's route class, and it is cleared by later evidence for that
+   same member. A mark cannot outlive its evidence.
+8. **Status is inspectable.** Every member reports `physicalState`,
+   `gatewayUnavailable`, and its connection counters through
+   `connection.status()`, so a caller can explain why an open was slow.
+
+## What it solves
+
+A liveness sweep is one process that probes many targets in sequence. Without
+requirement 2, a single unreachable target poisons every target probed after it
+in that process:
+
+```
+Trinity     OK    399 ms
+Lenovo      DEAD 10004 ms   <- genuinely unreachable
+Morpheus    DEAD 10008 ms   <- healthy, blocked by the poisoning
+Minisforum  DEAD 10011 ms   <- healthy, blocked by the poisoning
+Apoc        DEAD 10010 ms   <- healthy, blocked by the poisoning
+```
+
+Every "dead" row above answers in 63-485 ms when probed in a fresh process. The
+mark also survived for as long as the process lived, so once all members were
+marked the set had nothing to select: no open was attempted, therefore no
+successful open could clear a mark, therefore the marks stayed. A sweep reported
+a healthy fleet as unreachable and withheld its capacity.
+
+## Model
+
+### Who owns availability
+
+Only the hub can mark a member unavailable, and only per route class:
+
+- The client sends an open. The hub answers with `GATEWAY_SELECTED` carrying
+  `selectable`.
+  - `selectable: false` marks that member's route class unavailable
+    (`GATEWAY_SELECT_UNAVAILABLE`).
+  - `selectable: true` clears the mark.
+- A successfully opened stream clears the mark as well
+  (`reverse_client.gateway_available`), because a working open is direct
+  evidence that the route class works.
+
+A stream `RESET` does **not** change availability. A reset describes one stream's
+target; the hub resolves gateways per `(route class, target)`, so a reset is
+scoped to that target and never to the member's whole route class. This scoping
+is the fix for the poisoning above, and it is covered by a regression test in
+`test/integration/reverse-route-class.test.js`.
+
+### Selection
+
+`#selectReadyMember` walks the configured members in cursor order and picks the
+first one that is both `physicalState === "ready"` and not marked
+`gatewayUnavailable`. Members skipped for unavailability are remembered as a
+fallback: if **every** ready member is marked down, the set still tries one of
+them instead of refusing. That keeps a stale mark from turning into a hard
+outage, and it means a recovered member is discovered by the next open even
+before the hub has re-announced it.
+
+Streams stay sticky to their selected gateway and target; a transient physical
+transport replacement is resumed transparently. Availability decides *new*
+selection, it does not migrate live streams.
+
+### Boundaries
+
+- The set does not probe gateways, does not schedule background health checks,
+  and does not expire marks on a timer. Health comes from the hub.
+- The set does not synthesize availability from its own errors. A failure that
+  is not reported by the hub as unavailability is a plain error for that
+  operation.
+- Per-call deadlines belong to the caller. Socketduct does not extend them or
+  retry past them.
+
+## Operating it
+
+- Inspect `connection.status()` per member: `physicalState`, `gatewayUnavailable`,
+  `physicalConnects`, `reconnectAttempts`, `virtualOpens`.
+- Expect `reverse_client.gateway_available` in diagnostics when a member returns
+  to service; `reverse_client.gateway_unavailable` marks the opposite.
+- A member that reports `gatewayUnavailable: true` while its machine answers
+  locally is a hub-side registration problem (missing or rejected gateway for
+  that route class), not a machine fault.
+
+## Verification
+
+- `test/integration/reverse-route-class.test.js` - a reset for one target must not
+  mark the route class unavailable, and a following open on a healthy target must
+  succeed.
+- `test/unit/client/multiplexed-transport-set.test.js` - a member the hub
+  registered down is never *selected* while a ready alternative exists.
+- `test/integration/multiplexed-transport-set.test.js` - drain and lifecycle
+  behaviour across members.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "socketduct",
-  "version": "0.1.32",
+  "version": "0.1.34",
   "description": "Resilient authenticated tunnel/proxy for keeping socket streams alive across flaky links.",
   "type": "module",
   "main": "./src/index.js",
@@ -16,7 +16,7 @@
     "./relay": "./src/relay.js"
   },
   "engines": {
-    "node": "^24.0.0"
+    "node": ">=24.0.0"
   },
   "scripts": {
     "gateway:deploy": "node scripts/deploy-gateway.js",

package/src/client/multiplexed-connection.js

@@ -403,6 +403,17 @@ export class MultiplexedSocketductConnection extends EventEmitter {
     return socket
   }
 
+  /**
+   * Exact-gateway members are bound to one configured gateway for one configured
+   * target, so there is no per-target rotation to scope a failure to: an
+   * unavailable gateway takes this member out of service entirely and the set's
+   * member-level fallback owns it.
+   * @returns {boolean} Always false for exact-gateway members.
+   */
+  isTargetUnavailable() {
+    return false
+  }
+
   /** @returns {{physicalState: string, connectionGeneration: number, virtualStreams: number, streamStates: Record<string, number>, queuedBytes: number, dataQueuedBytes: number, controlQueuedBytes: number, spoolBytes: number, spoolCleanupFailures: number, physicalConnects: number, reconnectAttempts: number, reconnectSuccesses: number, resumeFailures: number}} Privacy-safe status. */
   status() {
     /** @type {Record<string, number>} */

package/src/client/multiplexed-transport-set.js

@@ -59,6 +59,7 @@ const GATEWAY_ID = /^[A-Za-z0-9][A-Za-z0-9_-]{1,63}$/
  * @property {(options: Record<string, unknown>, callback?: (error: Error | null, socket: import('./virtual-socket.js').VirtualSocket) => void) => import('./virtual-socket.js').VirtualSocket} registerRecoveredConnection - Cold-register durable recovery.
  * @property {(identifier: string) => {identifier: string, target: {host: string, port: number}, sessionName: string, manualReceiveAck: boolean} | undefined} [_inspectRecoveredConnection] - Read-only stable-identifier recovery lookup.
  * @property {() => MemberConnectionStatus} status - Privacy-safe status.
+ * @property {(target: {host: string, port: number}) => boolean} isTargetUnavailable - Whether this member last reported that exact target unserviceable.
  * @property {() => Promise<void>} close - Close the member.
  * @property {() => Promise<void>} detach - Preserve durable streams for process-replacement recovery.
  * @property {() => Promise<void>} drain - Let live streams finish before releasing the member.
@@ -82,6 +83,19 @@ const GATEWAY_ID = /^[A-Za-z0-9][A-Za-z0-9_-]{1,63}$/
  * @typedef {import('node:tls').TLSSocket & {readonly gatewayId: string | undefined, readonly routeClass: string | undefined}} GatewayTlsSocket
  */
 
+/**
+ * Resolves the logical target from either caller option shape.
+ * @param {{host?: string, port?: number, target?: {host: string, port: number} | null} | undefined} options - Caller open options.
+ * @returns {{host: string, port: number} | undefined} Exact target, when the caller named one.
+ */
+function resolveOpenTarget(options) {
+  if (options === undefined) return undefined
+  const nested = options.target
+  if (nested !== undefined && nested !== null) return {host: nested.host, port: nested.port}
+  if (typeof options.host !== "string" || typeof options.port !== "number") return undefined
+  return {host: options.host, port: options.port}
+}
+
 /** No configured gateway is currently safe for a new virtual stream. */
 export class SocketductTransportSetNotReadyError extends Error {
   /** @param {string} [message] - Privacy-safe detail. */
@@ -330,6 +344,19 @@ export class MultiplexedSocketductTransportSet extends EventEmitter {
     return this.#startPromise
   }
 
+  /**
+   * Whether this member most recently reported exactly this target unserviceable.
+   * A target-scoped reset only removes the member from rotation for that target,
+   * so an unreachable machine cannot cost the set its other machines.
+   * @param {TransportMember} member - Candidate member.
+   * @param {{host: string, port: number} | undefined} target - Target the caller wants.
+   * @returns {boolean} True when the member is marked down for this exact target.
+   */
+  #memberMarkedForTarget(member, target) {
+    if (target === undefined) return false
+    return member.connection.isTargetUnavailable(target)
+  }
+
   /**
    * Assigns one new virtual socket to the next ready member.
    * @param {import('./multiplexed-connection.js').VirtualConnectionOptions} options - Target/socket options.
@@ -341,7 +368,7 @@ export class MultiplexedSocketductTransportSet extends EventEmitter {
     if (this.#closed) throw closedError()
     const socketOptions = normalizeVirtualSocketOptions(options)
     if (this.#startupError !== undefined) return this.#failedSocket(socketOptions, this.#startupError, callback)
-    const selection = this.#pendingOpens.length === 0 && this.#startupResolved ? this.#selectReadyMember() : undefined
+    const selection = this.#pendingOpens.length === 0 && this.#startupResolved ? this.#selectReadyMember(options) : undefined
     if (selection === undefined) return this.#deferConnection(options, socketOptions, callback)
     try {
       /** @type {((error: Error | null, socket: import('./virtual-socket.js').VirtualSocket) => void) | undefined} */
@@ -494,14 +521,13 @@ export class MultiplexedSocketductTransportSet extends EventEmitter {
     return this
   }
 
-  /** @returns {{state: string, configuredMembers: number, connectingMembers: number, readyMembers: number, degradedMembers: number, closedMembers: number, minimumReadyMembers: number, pendingOpens: number, maxPendingOpens: number, virtualOpens: number, physicalConnects: number, reconnectAttempts: number, budgets: {dataBytes: number, spoolBytes: number, queuedBytes: number, maxDataBytes: number, maxSpoolBytes: number, maxQueuedBytes: number}, members: Array<{gatewayId?: string, routeClass?: string, state: string, gatewayUnavailable: boolean, assignments: number, physicalConnects: number, reconnectAttempts: number, reconnectSuccesses: number, virtualStreams: number, queuedBytes: number, spoolBytes: number, spoolCleanupFailures: number, terminalErrorCode: string | null}>}} Privacy-safe aggregate and per-member status. */
+  /** @returns {{state: string, configuredMembers: number, connectingMembers: number, readyMembers: number, degradedMembers: number, closedMembers: number, minimumReadyMembers: number, pendingOpens: number, maxPendingOpens: number, virtualOpens: number, physicalConnects: number, reconnectAttempts: number, budgets: {dataBytes: number, spoolBytes: number, queuedBytes: number, maxDataBytes: number, maxSpoolBytes: number, maxQueuedBytes: number}, members: Array<{gatewayId?: string, routeClass?: string, state: string, assignments: number, physicalConnects: number, reconnectAttempts: number, reconnectSuccesses: number, virtualStreams: number, queuedBytes: number, spoolBytes: number, spoolCleanupFailures: number, terminalErrorCode: string | null}>}} Privacy-safe aggregate and per-member status. */
   status() {
     const members = this.#members.map((member) => {
       const status = member.connection.status()
       return {
         ...memberAllocation(member),
         state: status.physicalState,
-        gatewayUnavailable: status.gatewayUnavailable === true,
         assignments: member.assignments,
         physicalConnects: status.physicalConnects,
         reconnectAttempts: status.reconnectAttempts,
@@ -578,25 +604,35 @@ export class MultiplexedSocketductTransportSet extends EventEmitter {
   }
 
   /**
-   * Selects the next ready member whose route class the hub has registered up.
-   * A member the hub registered down is never selected: nothing is sent to a
-   * route class without a reachable gateway. When every member is registered
-   * down the open stays pending here until a member reports availability again
-   * (the member keeps probing the hub, and its state change re-drains this set).
+   * Selects the next ready member, preferring members whose gateway the hub has
+   * not recently reported unavailable. When every ready member carries a
+   * gateway-unavailable mark, any ready member may still be selected so a stale
+   * mark can never wedge the set.
+   * @param {{host?: string, port?: number, target?: {host: string, port: number} | null} | undefined} options - Caller open options, used for the exact target.
    * @returns {{member: TransportMember, previousCursor: number} | undefined}
    */
-  #selectReadyMember() {
+  #selectReadyMember(options) {
     if (this.#closed) throw closedError()
     const previousCursor = this.#cursor
+    const target = resolveOpenTarget(options)
+    /** @type {{member: TransportMember, index: number} | undefined} */
+    let fallback
     for (let offset = 0; offset < this.#members.length; offset += 1) {
       const index = (this.#cursor + offset) % this.#members.length
       const member = /** @type {TransportMember} */ (this.#members[index])
       const status = member.connection.status()
       if (status.physicalState !== "ready") continue
-      if (status.gatewayUnavailable === true) continue
+      if (status.gatewayUnavailable === true || this.#memberMarkedForTarget(member, target)) {
+        fallback ??= {member, index}
+        continue
+      }
       this.#cursor = (index + 1) % this.#members.length
       return {member, previousCursor}
     }
+    if (fallback !== undefined) {
+      this.#cursor = (fallback.index + 1) % this.#members.length
+      return {member: fallback.member, previousCursor}
+    }
     return undefined
   }
 
@@ -661,7 +697,8 @@ export class MultiplexedSocketductTransportSet extends EventEmitter {
   #drainPendingOpens() {
     if (!this.#startupResolved || this.#closed || this.#startupError !== undefined) return
     while (this.#pendingOpens.length > 0) {
-      const selection = this.#selectReadyMember()
+      const nextPending = this.#pendingOpens[0]
+      const selection = nextPending === undefined ? undefined : this.#selectReadyMember(nextPending.options)
       if (selection === undefined) return
       const pending = this.#pendingOpens.shift()
       if (pending === undefined) return