socketduct 0.1.33 → 0.1.35

package/README.md

@@ -111,6 +111,7 @@ hub address because `gatewayId` selects the outbound member. Docker remains
 - [Running Socketduct](docs/running.md)
 - [Docker gateway setup](docs/docker.md)
 - [Docker topology](docs/docker-topology.md)
+- [Transport-set availability](docs/transport-set-availability.md)
 - [Node-compatible socket API design](docs/socket-api.md)
 - [Native reverse gateway](docs/reverse-gateway.md)
 - [Durable compressed diagnostic logs](docs/logging.md)

package/changelog.d/20261026183000-route-class-availability-scope.md

@@ -1 +1,3 @@
 A reset for one stream no longer marks the member's whole route class unavailable, so a single unreachable target cannot disable every route class and stall all later opens in a transport set.
+
+Documents the requirement behind it in `docs/transport-set-availability.md`: per-member availability owned by the hub, failover instead of stalling, automatic recovery without operator action, and no polling watchdog.

package/changelog.d/20261026192500-target-scoped-availability.md

@@ -0,0 +1,5 @@
+### Fixed
+
+- Route-class availability failures are now scoped to the exact target that failed. A `GATEWAY_UNAVAILABLE` reset for one target marks only that target on that member, so the transport set stops handing it opens while every other target keeps using the member. The mark is cleared by the next successful open for that same target, so nothing expires on a timer and a member can never be dropped from rotation for machines it can still reach.
+
+  Releasing 0.1.33 without this scoping removed the only signal the set had for failover: with several route classes configured, opens kept landing on members whose route class could not serve the target and the reset surfaced straight to the caller. Measured on the TensorBuzz fleet, `GATEWAY_UNAVAILABLE` build failures went from 0/hour to about 100/hour within minutes of 0.1.33 reaching the running workers.

package/changelog.d/20261026200000-ship-docker-entrypoint.md

@@ -0,0 +1 @@
+Publishes `docker-entrypoint.sh` in the package again, so the committed Dockerfile can build an image from a released tarball.

package/changelog.d/20261027034500-reverse-open-rejection-diagnostics.md

@@ -0,0 +1 @@
+The reverse hub's periodic status now reports rejected opens broken down by reset code, and clients log each relay reset as a `virtual_stream.reset` diagnostic event, so gateway outages can be told apart from authorization or routing rejections without new instrumentation.

package/docker-entrypoint.sh

@@ -0,0 +1,11 @@
+#!/bin/sh
+set -e
+
+# Run command with node if the first argument contains a "-" or is not a system command. The last
+# part inside the "{}" is a workaround for the following bug in ash/dash:
+# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=874264
+if [ "${1#-}" != "${1}" ] || [ -z "$(command -v "${1}")" ] || { [ -f "${1}" ] && ! [ -x "${1}" ]; }; then
+  set -- node "$@"
+fi
+
+exec "$@"

package/docs/reverse-gateway.md

@@ -347,6 +347,11 @@ const socket = set.createConnection({host: "docker-a", port: 2375})
 await starting
 ```
 
+Availability of a member is decided by the hub's `GATEWAY_SELECTED`
+announcement and cleared by a successful open; a stream reset never changes
+it. Requirements, the selection model and the failure this prevents are in
+[transport-set availability](transport-set-availability.md).
+
 Route-class sockets expose immutable `routeClass` allocation metadata and an
 undefined `gatewayId`. Exact-gateway sets retain immutable `gatewayId` metadata.
 Callers cannot override either selection per open. Reverse streams remain

package/docs/transport-set-availability.md

@@ -0,0 +1,119 @@
+# Transport-set availability
+
+How `MultiplexedSocketductTransportSet` decides which configured member to use,
+what "down" means for a member, and which component is allowed to say so.
+
+This page states the requirements the model exists to satisfy, then describes
+the mechanism. Requirements come from production failures where a runner fleet
+was reported unreachable although every machine answered.
+
+## Requirements
+
+1. **Never use a member the hub reports down.** If the hub says a route class has
+   no selectable gateway, opens must not be routed to that member while a ready
+   alternative exists.
+2. **A failure scoped to one target must not disable siblings.** Availability is
+   per route class (member). A failure to reach one *target* through one member
+   says nothing about the other targets or the other members.
+3. **Fail over instead of stalling.** If the preferred member is unavailable, an
+   open must go to another ready member rather than waiting for the first one.
+4. **A member that stays down must not block callers indefinitely.** Callers keep
+   their own deadlines; the set must not sleep through them while it waits for a
+   single member to come back.
+5. **Recovery is automatic.** When a member becomes selectable again, later opens
+   use it again with no operator action — no restart, no container removal, no
+   config change.
+6. **Availability is event-driven, not polled.** The state changes on hub
+   announcements and on successful opens. There is no timer that sweeps members
+   and no background watchdog that decides health on its own.
+7. **Marks are attributable and bounded.** A mark is only set by evidence that
+   names the member's route class, and it is cleared by later evidence for that
+   same member. A mark cannot outlive its evidence.
+8. **Status is inspectable.** Every member reports `physicalState`,
+   `gatewayUnavailable`, and its connection counters through
+   `connection.status()`, so a caller can explain why an open was slow.
+
+## What it solves
+
+A liveness sweep is one process that probes many targets in sequence. Without
+requirement 2, a single unreachable target poisons every target probed after it
+in that process:
+
+```
+Trinity     OK    399 ms
+Lenovo      DEAD 10004 ms   <- genuinely unreachable
+Morpheus    DEAD 10008 ms   <- healthy, blocked by the poisoning
+Minisforum  DEAD 10011 ms   <- healthy, blocked by the poisoning
+Apoc        DEAD 10010 ms   <- healthy, blocked by the poisoning
+```
+
+Every "dead" row above answers in 63-485 ms when probed in a fresh process. The
+mark also survived for as long as the process lived, so once all members were
+marked the set had nothing to select: no open was attempted, therefore no
+successful open could clear a mark, therefore the marks stayed. A sweep reported
+a healthy fleet as unreachable and withheld its capacity.
+
+## Model
+
+### Who owns availability
+
+Only the hub can mark a member unavailable, and only per route class:
+
+- The client sends an open. The hub answers with `GATEWAY_SELECTED` carrying
+  `selectable`.
+  - `selectable: false` marks that member's route class unavailable
+    (`GATEWAY_SELECT_UNAVAILABLE`).
+  - `selectable: true` clears the mark.
+- A successfully opened stream clears the mark as well
+  (`reverse_client.gateway_available`), because a working open is direct
+  evidence that the route class works.
+
+A stream `RESET` does **not** change availability. A reset describes one stream's
+target; the hub resolves gateways per `(route class, target)`, so a reset is
+scoped to that target and never to the member's whole route class. This scoping
+is the fix for the poisoning above, and it is covered by a regression test in
+`test/integration/reverse-route-class.test.js`.
+
+### Selection
+
+`#selectReadyMember` walks the configured members in cursor order and picks the
+first one that is both `physicalState === "ready"` and not marked
+`gatewayUnavailable`. Members skipped for unavailability are remembered as a
+fallback: if **every** ready member is marked down, the set still tries one of
+them instead of refusing. That keeps a stale mark from turning into a hard
+outage, and it means a recovered member is discovered by the next open even
+before the hub has re-announced it.
+
+Streams stay sticky to their selected gateway and target; a transient physical
+transport replacement is resumed transparently. Availability decides *new*
+selection, it does not migrate live streams.
+
+### Boundaries
+
+- The set does not probe gateways, does not schedule background health checks,
+  and does not expire marks on a timer. Health comes from the hub.
+- The set does not synthesize availability from its own errors. A failure that
+  is not reported by the hub as unavailability is a plain error for that
+  operation.
+- Per-call deadlines belong to the caller. Socketduct does not extend them or
+  retry past them.
+
+## Operating it
+
+- Inspect `connection.status()` per member: `physicalState`, `gatewayUnavailable`,
+  `physicalConnects`, `reconnectAttempts`, `virtualOpens`.
+- Expect `reverse_client.gateway_available` in diagnostics when a member returns
+  to service; `reverse_client.gateway_unavailable` marks the opposite.
+- A member that reports `gatewayUnavailable: true` while its machine answers
+  locally is a hub-side registration problem (missing or rejected gateway for
+  that route class), not a machine fault.
+
+## Verification
+
+- `test/integration/reverse-route-class.test.js` - a reset for one target must not
+  mark the route class unavailable, and a following open on a healthy target must
+  succeed.
+- `test/unit/client/multiplexed-transport-set.test.js` - a member the hub
+  registered down is never *selected* while a ready alternative exists.
+- `test/integration/multiplexed-transport-set.test.js` - drain and lifecycle
+  behaviour across members.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "socketduct",
-  "version": "0.1.33",
+  "version": "0.1.35",
   "description": "Resilient authenticated tunnel/proxy for keeping socket streams alive across flaky links.",
   "type": "module",
   "main": "./src/index.js",

package/src/client/multiplexed-connection.js

@@ -403,6 +403,17 @@ export class MultiplexedSocketductConnection extends EventEmitter {
     return socket
   }
 
+  /**
+   * Exact-gateway members are bound to one configured gateway for one configured
+   * target, so there is no per-target rotation to scope a failure to: an
+   * unavailable gateway takes this member out of service entirely and the set's
+   * member-level fallback owns it.
+   * @returns {boolean} Always false for exact-gateway members.
+   */
+  isTargetUnavailable() {
+    return false
+  }
+
   /** @returns {{physicalState: string, connectionGeneration: number, virtualStreams: number, streamStates: Record<string, number>, queuedBytes: number, dataQueuedBytes: number, controlQueuedBytes: number, spoolBytes: number, spoolCleanupFailures: number, physicalConnects: number, reconnectAttempts: number, reconnectSuccesses: number, resumeFailures: number}} Privacy-safe status. */
   status() {
     /** @type {Record<string, number>} */
@@ -948,6 +959,7 @@ export class MultiplexedSocketductConnection extends EventEmitter {
       const code = typeof detail.code === "string" ? detail.code : "STREAM_RESET"
       stream.remoteClosed = true
       stream.state = "reset"
+      logDiagnostic(this.#options.logger, "virtual_stream.reset", {streamId: stream.id, errorCode: code})
       const lost = code.startsWith("RESUME_") || code === "SESSION_LOST" || code.includes("UNCERTAIN")
       stream.socket._receiveError(lost
         ? new SocketductSessionLostError("Virtual stream could not be resumed", code)

package/src/client/multiplexed-transport-set.js

@@ -59,6 +59,7 @@ const GATEWAY_ID = /^[A-Za-z0-9][A-Za-z0-9_-]{1,63}$/
  * @property {(options: Record<string, unknown>, callback?: (error: Error | null, socket: import('./virtual-socket.js').VirtualSocket) => void) => import('./virtual-socket.js').VirtualSocket} registerRecoveredConnection - Cold-register durable recovery.
  * @property {(identifier: string) => {identifier: string, target: {host: string, port: number}, sessionName: string, manualReceiveAck: boolean} | undefined} [_inspectRecoveredConnection] - Read-only stable-identifier recovery lookup.
  * @property {() => MemberConnectionStatus} status - Privacy-safe status.
+ * @property {(target: {host: string, port: number}) => boolean} isTargetUnavailable - Whether this member last reported that exact target unserviceable.
  * @property {() => Promise<void>} close - Close the member.
  * @property {() => Promise<void>} detach - Preserve durable streams for process-replacement recovery.
  * @property {() => Promise<void>} drain - Let live streams finish before releasing the member.
@@ -82,6 +83,19 @@ const GATEWAY_ID = /^[A-Za-z0-9][A-Za-z0-9_-]{1,63}$/
  * @typedef {import('node:tls').TLSSocket & {readonly gatewayId: string | undefined, readonly routeClass: string | undefined}} GatewayTlsSocket
  */
 
+/**
+ * Resolves the logical target from either caller option shape.
+ * @param {{host?: string, port?: number, target?: {host: string, port: number} | null} | undefined} options - Caller open options.
+ * @returns {{host: string, port: number} | undefined} Exact target, when the caller named one.
+ */
+function resolveOpenTarget(options) {
+  if (options === undefined) return undefined
+  const nested = options.target
+  if (nested !== undefined && nested !== null) return {host: nested.host, port: nested.port}
+  if (typeof options.host !== "string" || typeof options.port !== "number") return undefined
+  return {host: options.host, port: options.port}
+}
+
 /** No configured gateway is currently safe for a new virtual stream. */
 export class SocketductTransportSetNotReadyError extends Error {
   /** @param {string} [message] - Privacy-safe detail. */
@@ -330,6 +344,19 @@ export class MultiplexedSocketductTransportSet extends EventEmitter {
     return this.#startPromise
   }
 
+  /**
+   * Whether this member most recently reported exactly this target unserviceable.
+   * A target-scoped reset only removes the member from rotation for that target,
+   * so an unreachable machine cannot cost the set its other machines.
+   * @param {TransportMember} member - Candidate member.
+   * @param {{host: string, port: number} | undefined} target - Target the caller wants.
+   * @returns {boolean} True when the member is marked down for this exact target.
+   */
+  #memberMarkedForTarget(member, target) {
+    if (target === undefined) return false
+    return member.connection.isTargetUnavailable(target)
+  }
+
   /**
    * Assigns one new virtual socket to the next ready member.
    * @param {import('./multiplexed-connection.js').VirtualConnectionOptions} options - Target/socket options.
@@ -341,7 +368,7 @@ export class MultiplexedSocketductTransportSet extends EventEmitter {
     if (this.#closed) throw closedError()
     const socketOptions = normalizeVirtualSocketOptions(options)
     if (this.#startupError !== undefined) return this.#failedSocket(socketOptions, this.#startupError, callback)
-    const selection = this.#pendingOpens.length === 0 && this.#startupResolved ? this.#selectReadyMember() : undefined
+    const selection = this.#pendingOpens.length === 0 && this.#startupResolved ? this.#selectReadyMember(options) : undefined
     if (selection === undefined) return this.#deferConnection(options, socketOptions, callback)
     try {
       /** @type {((error: Error | null, socket: import('./virtual-socket.js').VirtualSocket) => void) | undefined} */
@@ -581,11 +608,13 @@ export class MultiplexedSocketductTransportSet extends EventEmitter {
    * not recently reported unavailable. When every ready member carries a
    * gateway-unavailable mark, any ready member may still be selected so a stale
    * mark can never wedge the set.
+   * @param {{host?: string, port?: number, target?: {host: string, port: number} | null} | undefined} options - Caller open options, used for the exact target.
    * @returns {{member: TransportMember, previousCursor: number} | undefined}
    */
-  #selectReadyMember() {
+  #selectReadyMember(options) {
     if (this.#closed) throw closedError()
     const previousCursor = this.#cursor
+    const target = resolveOpenTarget(options)
     /** @type {{member: TransportMember, index: number} | undefined} */
     let fallback
     for (let offset = 0; offset < this.#members.length; offset += 1) {
@@ -593,7 +622,7 @@ export class MultiplexedSocketductTransportSet extends EventEmitter {
       const member = /** @type {TransportMember} */ (this.#members[index])
       const status = member.connection.status()
       if (status.physicalState !== "ready") continue
-      if (status.gatewayUnavailable === true) {
+      if (status.gatewayUnavailable === true || this.#memberMarkedForTarget(member, target)) {
         fallback ??= {member, index}
         continue
       }
@@ -668,7 +697,8 @@ export class MultiplexedSocketductTransportSet extends EventEmitter {
   #drainPendingOpens() {
     if (!this.#startupResolved || this.#closed || this.#startupError !== undefined) return
     while (this.#pendingOpens.length > 0) {
-      const selection = this.#selectReadyMember()
+      const nextPending = this.#pendingOpens[0]
+      const selection = nextPending === undefined ? undefined : this.#selectReadyMember(nextPending.options)
       if (selection === undefined) return
       const pending = this.#pendingOpens.shift()
       if (pending === undefined) return

package/src/logging/structured-logger.js

@@ -27,6 +27,7 @@ const COMPRESSION = new Set(["auto", "off", "zstd", "none"])
 const STATES = new Set(["configured", "connecting", "registering", "ready", "reconnecting", "disconnected", "closed", "starting", "degraded", "failed"])
 const SAFE_ERROR_CODES = new Set([
   "AUTH_FAILED",
+  "AMBIGUOUS_ROUTE",
   "AUTH_NEGOTIATION_FAILED",
   "AUTH_OWNERSHIP_MISMATCH",
   "AUTH_REQUIRED",
@@ -51,6 +52,7 @@ const SAFE_ERROR_CODES = new Set([
   "HANDSHAKE_TIMEOUT",
   "HUB_SHUTDOWN",
   "INPUT_LIMIT_EXCEEDED",
+  "INVALID_ROUTE_RESOLUTION",
   "INVALID_STREAM_OPEN",
   "PROTOCOL_NEGOTIATION_FAILED",
   "PROTOCOL_VIOLATION",
@@ -77,6 +79,7 @@ const SAFE_ERROR_CODES = new Set([
   "TARGET_OPEN_TIMEOUT",
   "TARGET_READ_FAILED",
   "TARGET_UNAUTHORIZED",
+  "UNKNOWN_GATEWAY_ID",
   "TLS_ERROR",
   "TRANSPORT_CLOSED",
   "UNKNOWN"
@@ -156,7 +159,7 @@ const NUMBER_FIELDS = new Set([
   "virtualStreams",
   "wireDataBytes"
 ])
-const RUNTIME_STATUS_FIELDS = new Set([...NUMBER_FIELDS, "compression", "physicalState"])
+const RUNTIME_STATUS_FIELDS = new Set([...NUMBER_FIELDS, "compression", "physicalState", "rejectedOpensByCode"])
 const EVENT_FIELDS = new Map([
   ["process.started", new Set(["runtimeMode", "transportSecurity", "compression", "compressionLevel", "frameMaxBytes", "flushIntervalMs", "criticalFlushIntervalMs", "maxSegmentBytes", "retentionDays", "retentionBytes", "maxQueueBytes", "criticalReserveBytes"])],
   ["process.stopped", new Set(["runtimeMode"])],
@@ -174,6 +177,7 @@ const EVENT_FIELDS = new Map([
   ["transport_set.terminal_failure", new Set(["configuredMembers", "closedMembers", "errorCode"])],
   ["virtual_stream.opening", new Set(["streamId", "generation", "gatewayId", "recovered"])],
   ["virtual_stream.opened", new Set(["streamId", "generation", "gatewayId", "recovered"])],
+  ["virtual_stream.reset", new Set(["streamId", "errorCode"])],
   ["virtual_stream.closed", new Set(["streamId", "generation", "gatewayId"])],
   ["virtual_stream.error", new Set(["streamId", "generation", "gatewayId", "routeClass", "errorCode", "disconnectIncidentId", "disconnectReason", "incidentOrigin", "physicalRole", "clientConnectionGeneration", "gatewayConnectionGeneration", "fanOutCount", "resumeSuccessCount", "terminalStreamCount"])],
   ["reverse_gateway.target_opened", new Set(["streamId", "generation", "gatewayId"])],
@@ -249,7 +253,7 @@ export function formatDiagnosticLogLine(options) {
   const base = {timestamp: timestamp.toISOString(), event: options.event, processId, role: options.role, ...envelope}
   let line = formatLogLine(base, safeFields)
   if (Buffer.byteLength(line) > MAX_JSONL_BYTES && safeFields.errorCode !== undefined) {
-    /** @type {Record<string, string | number | boolean>} */
+    /** @type {Record<string, string | number | boolean | Record<string, number>>} */
     const boundedFields = {...safeFields}
     delete boundedFields.errorStack
     if (typeof boundedFields.errorMessage === "string") {
@@ -595,12 +599,12 @@ export function errorDiagnostics(error, fallback = "UNKNOWN") {
 /**
  * @param {string} event - Event name.
  * @param {Record<string, unknown>} fields - Candidate fields.
- * @returns {Record<string, string | number | boolean> | undefined} Allowlisted fields, or undefined for an unknown event.
+ * @returns {Record<string, string | number | boolean | Record<string, number>> | undefined} Allowlisted fields, or undefined for an unknown event.
  */
 function sanitizeFields(event, fields) {
   const allowed = EVENT_FIELDS.get(event)
   if (allowed === undefined || typeof fields !== "object" || fields === null || Array.isArray(fields)) return undefined
-  /** @type {Record<string, string | number | boolean>} */
+  /** @type {Record<string, string | number | boolean | Record<string, number>>} */
   const safe = {}
   for (const field of allowed) {
     const value = fields[field]
@@ -656,6 +660,22 @@ function sanitizeFields(event, fields) {
       const errorCode = normalizeErrorCode(value)
       safe.errorCode = errorCode
       safe.errorClass = classifyError(errorCode)
+    } else if (field === "rejectedOpensByCode" && typeof value === "object" && value !== null && !Array.isArray(value)) {
+      const entries = Object.entries(value)
+      if (entries.length <= 16) {
+        /** @type {Record<string, number>} */
+        const bounded = {}
+        let valid = true
+
+        for (const [code, count] of entries) {
+          if (!SAFE_ERROR_CODES.has(code) || typeof count !== "number" || !Number.isSafeInteger(count) || count < 0) {
+            valid = false
+            break
+          }
+          bounded[code] = count
+        }
+        if (valid) safe.rejectedOpensByCode = bounded
+      }
     }
   }
   const errorDetails = Object.getOwnPropertyDescriptor(fields, ERROR_DETAILS)?.value
@@ -671,7 +691,7 @@ function sanitizeFields(event, fields) {
 
 /**
  * @param {{event: string, processId: number, role: string, timestamp: string}} base - Required JSONL fields.
- * @param {Record<string, string | number | boolean>} fields - Sanitized event fields.
+ * @param {Record<string, string | number | boolean | Record<string, number>>} fields - Sanitized event fields.
  * @returns {string} One JSONL record.
  */
 function formatLogLine(base, fields) {

package/src/reverse/client.js

@@ -32,6 +32,16 @@ import {attachDisconnectAttribution, createDisconnectAttribution, parseDisconnec
 const DEFAULT_STREAM_BUFFER_BYTES = 4 * 1024 * 1024
 const DEFAULT_CONNECTION_BUFFER_BYTES = 32 * 1024 * 1024
 const MANIFEST_PAGE_STREAMS = 64
+const MAX_TARGET_UNAVAILABLE_ENTRIES = 64
+
+/**
+ * Builds the per-target availability key for one logical target.
+ * @param {{host: string, port: number}} target - Logical target.
+ * @returns {string} Stable key.
+ */
+function targetKey(target) {
+  return `${target.host}:${target.port}`
+}
 
 /**
  * @typedef {object} ReverseGatewayConnectionOptions
@@ -169,6 +179,14 @@ export class ReverseGatewayConnection extends EventEmitter {
   /** @type {number} */ #spoolCleanupFailures = 0
   /** @type {Set<Promise<void>>} */ #spoolCleanupTasks = new Set()
   /** @type {number | undefined} - Wall-clock time of the last hub-reported gateway unavailability. */ #gatewayUnavailableSince
+  /**
+   * Targets this member most recently answered with "the hub has no gateway for
+   * that target". Scoped to the exact target so one unreachable machine cannot
+   * take this route class out of rotation for every other target, and cleared by
+   * the next OPENED for the same target instead of expiring on a timer.
+   * @type {Map<string, number>}
+   */
+  #targetUnavailableSince = new Map()
   /** @type {import('./disconnect-attribution.js').DisconnectAttribution | undefined} */ #disconnectAttribution
 
   /** @param {ReverseGatewayConnectionOptions} options - Hub route and safety policy. */
@@ -386,6 +404,34 @@ export class ReverseGatewayConnection extends EventEmitter {
     return this.#gatewayUnavailableSince !== undefined
   }
 
+  /**
+   * Whether this member most recently reported the exact target unserviceable.
+   * Scoped per target: one unreachable machine must not remove this route class
+   * from rotation for every other machine.
+   * @param {{host: string, port: number}} target - Logical target a caller wants to open.
+   * @returns {boolean} True while this member is marked down for that target.
+   */
+  isTargetUnavailable(target) {
+    return this.#targetUnavailableSince.has(targetKey(target))
+  }
+
+  /** Records that this member could not serve a target, bounded so long-lived sets stay small. */
+  #markTargetUnavailable(/** @type {{host: string, port: number}} */ target) {
+    const key = targetKey(target)
+    this.#targetUnavailableSince.delete(key)
+    this.#targetUnavailableSince.set(key, Date.now())
+    while (this.#targetUnavailableSince.size > MAX_TARGET_UNAVAILABLE_ENTRIES) {
+      const oldest = this.#targetUnavailableSince.keys().next()
+      if (oldest.done === true) break
+      this.#targetUnavailableSince.delete(oldest.value)
+    }
+  }
+
+  /** Clears a target mark; a successful open is the only thing that proves it recovered. */
+  #clearTargetUnavailable(/** @type {{host: string, port: number}} */ target) {
+    this.#targetUnavailableSince.delete(targetKey(target))
+  }
+
   status() {
     /** @type {Record<string, number>} */
     const streamStates = {}
@@ -785,6 +831,7 @@ export class ReverseGatewayConnection extends EventEmitter {
       stream.sendWindow = Math.min(stream.sendWindow, detail.initialWindow)
       stream.state = stream.localEnded ? "half-closed" : "open"
       stream.opened = true
+      this.#clearTargetUnavailable(stream.target)
       if (this.#gatewayUnavailableSince !== undefined) {
         this.#gatewayUnavailableSince = undefined
         logDiagnostic(this.#options.logger, "reverse_client.gateway_available", {
@@ -853,6 +900,13 @@ export class ReverseGatewayConnection extends EventEmitter {
       // selected, so no open can succeed, so no mark can ever be cleared.
       // Route-class availability is owned by the hub's GATEWAY_SELECTED response,
       // which is class-scoped and re-read on every probe.
+      //
+      // What the reset does own is this target: the transport set must stop
+      // handing this member opens for a target it just reported unserviceable,
+      // while every other target keeps using it. The mark is keyed by target and
+      // cleared by the next OPENED for that target, so it never expires on a
+      // timer and can never wedge the set.
+      if (code === "GATEWAY_UNAVAILABLE" || code === "GATEWAY_DISCONNECTED") this.#markTargetUnavailable(stream.target)
       const error = attachDisconnectAttribution(new SocketductStreamError("Reverse gateway stream was reset", code), validHubAttribution(detail) ?? (code === "GATEWAY_DISCONNECTED" ? this.#disconnectAttribution : undefined))
       stream.socket._receiveError(error)
       return

package/src/reverse/hub.js

@@ -175,6 +175,8 @@ export async function startReverseGatewayHub(/** @type {ReverseGatewayHubOptions
   let gatewayRegistrations = 0
   let rejectedRegistrations = 0
   let rejectedOpens = 0
+  /** @type {Map<string, number>} */
+  const rejectedOpensByCode = new Map()
   let rawDataBytes = 0
   let wireDataBytes = 0
   let closing = false
@@ -707,6 +709,12 @@ export async function startReverseGatewayHub(/** @type {ReverseGatewayHubOptions
     }
   }
 
+  /** @param {string} code - V3 reset code sent to the rejecting client. */
+  function countRejectedOpen(code) {
+    rejectedOpens += 1
+    rejectedOpensByCode.set(code, (rejectedOpensByCode.get(code) || 0) + 1)
+  }
+
   /** @param {Record<string, unknown>} value - Resume manifest entry. */
   function parseResumeItem(value) {
     const item = /** @type {{streamId?: unknown, identifier?: unknown, clientToGatewaySequence?: unknown, clientToGatewayAck?: unknown, gatewayToClientSequence?: unknown, gatewayToClientAck?: unknown, localEnded?: unknown, remoteEnded?: unknown}} */ (value)
@@ -734,17 +742,17 @@ export async function startReverseGatewayHub(/** @type {ReverseGatewayHubOptions
       }
       const candidates = options.resolveOpenGateways?.({principal: client.principal, routeClass: client.routeClass, target})
       if (!Array.isArray(candidates) || candidates.some((candidate) => typeof candidate !== "string" || !GATEWAY_ID.test(candidate))) {
-        rejectedOpens += 1
+        countRejectedOpen("INVALID_ROUTE_RESOLUTION")
         await sendControl(client, V3FrameType.RESET, frame.streamId, 0, {code: "INVALID_ROUTE_RESOLUTION"})
         return
       }
       if (candidates.length === 0) {
-        rejectedOpens += 1
+        countRejectedOpen("TARGET_UNAUTHORIZED")
         await sendControl(client, V3FrameType.RESET, frame.streamId, 0, {code: "TARGET_UNAUTHORIZED"})
         return
       }
       if (candidates.length !== 1 || new Set(candidates).size !== 1) {
-        rejectedOpens += 1
+        countRejectedOpen("AMBIGUOUS_ROUTE")
         await sendControl(client, V3FrameType.RESET, frame.streamId, 0, {code: "AMBIGUOUS_ROUTE"})
         return
       }
@@ -757,15 +765,15 @@ export async function startReverseGatewayHub(/** @type {ReverseGatewayHubOptions
       gatewayId = request.gatewayId
     }
     if (options.authorizeOpen({principal: client.principal, gatewayId, target}) !== true) {
-      rejectedOpens += 1
+      countRejectedOpen("TARGET_UNAUTHORIZED")
       await sendControl(client, V3FrameType.RESET, frame.streamId, 0, {code: "TARGET_UNAUTHORIZED"})
       return
     }
     const gatewayOwner = gateways.get(gatewayId)
     const gateway = gatewayOwner?.physical
     if (gatewayOwner === undefined || gateway === undefined || gateway.socket.destroyed || gateway.closing || gateway.resumeRequired || gateway.role !== "gateway" || gateway.gatewayId !== gatewayId || gatewayOwner.gatewayId !== gatewayId || gatewayOwner.userId !== gateway.principal?.userId) {
-      rejectedOpens += 1
       const known = gatewayOwner !== undefined || options.knownGatewayIds?.includes(gatewayId) === true
+      countRejectedOpen(known ? "GATEWAY_UNAVAILABLE" : "UNKNOWN_GATEWAY_ID")
       await sendControl(client, V3FrameType.RESET, frame.streamId, 0, {code: known ? "GATEWAY_UNAVAILABLE" : "UNKNOWN_GATEWAY_ID"})
       return
     }
@@ -1222,6 +1230,7 @@ export async function startReverseGatewayHub(/** @type {ReverseGatewayHubOptions
       gatewayRegistrations,
       rejectedRegistrations,
       rejectedOpens,
+      rejectedOpensByCode: Object.fromEntries(rejectedOpensByCode),
       virtualStreams: bridges,
       rawDataBytes,
       wireDataBytes,

package/test/integration/multiplexed-failures.test.js

@@ -80,8 +80,11 @@ describe("multiplexed failures", () => {
   it("authentication and exact-target authorization fail closed", async () => withTestCleanup(async (after) => {
     const relay = await startMultiplexedTcpRelay({tokens: tokens(), authorizeTarget: () => false})
     after(() => relay.close())
+    /** @type {Array<{event: string} & Record<string, unknown>>} */
+    const lifecycle = []
     const unauthorizedConnection = new MultiplexedSocketductConnection({
-      relay: {host: "127.0.0.1", port: relay.port, token: "valid-token"}
+      relay: {host: "127.0.0.1", port: relay.port, token: "valid-token"},
+      logger: {log: (/** @type {string} */ event, /** @type {Record<string, unknown>} */ fields = {}) => lifecycle.push({event, ...fields})}
     })
     after(() => unauthorizedConnection.close())
     const denied = unauthorizedConnection.createConnection({host: "127.0.0.1", port: 9})
@@ -89,6 +92,7 @@ describe("multiplexed failures", () => {
     expect(deniedError instanceof SocketductStreamError).toBeTruthy()
     expect(deniedError.code).toBe("TARGET_UNAUTHORIZED")
     expect(relay.status().activeTargets).toBe(0)
+    expect({value: lifecycle.find((entry) => entry.event === "virtual_stream.reset")?.errorCode, context: "the client logs the relay reset code"}).toMatchObject({value: "TARGET_UNAUTHORIZED"})
 
     const invalidConnection = new MultiplexedSocketductConnection({
       relay: {host: "127.0.0.1", port: relay.port, token: "wrong-token"},

package/test/integration/reverse-route-class.test.js

@@ -533,12 +533,19 @@ describe("reverse gateway route-class", () => {
     const reset = await nextError(unavailableStream)
     expect(reset.code).toBe("GATEWAY_UNAVAILABLE")
     unavailableStream.destroy()
+    expect({value: hub.status().rejectedOpensByCode?.GATEWAY_UNAVAILABLE, context: "the hub status counts the rejected open by its reset code"}).toMatchObject({value: 1})
 
     // The route class is healthy, so the reset must not take it out of service.
     // A member marked unavailable this way is never selected again, which would
     // stall every later open in the set until the caller's own deadline.
     expect({value: client.status().gatewayUnavailable, context: "a target-scoped reset must not mark the route class unavailable"}).toMatchObject({value: false})
 
+    // The member must still be skipped for the target the hub could not serve,
+    // otherwise every later open for it keeps paying a full round trip, while
+    // every other target on this member stays in rotation.
+    expect({value: client.isTargetUnavailable({host: targetB.host, port: targetB.port}), context: "the unserviceable target is marked on this member"}).toMatchObject({value: true})
+    expect({value: client.isTargetUnavailable({host: targetA.host, port: targetA.port}), context: "a different target stays available on the same member"}).toMatchObject({value: false})
+
     // A healthy target on the same route class still opens and exchanges data.
     const healthyStream = client.createConnection({host: targetA.host, port: targetA.port})
     await onceEvent(healthyStream, "connect")

package/test/unit/client/multiplexed-transport-set.test.js

@@ -72,6 +72,7 @@ class FakeMember extends EventEmitter {
     this.generation = 0
     this.physicalConnects = 0
     this.gatewayUnavailable = false
+    /** @type {Set<string>} */ this.unavailableTargets = new Set()
     /** @type {Promise<void>} */
     this.connectPromise = new Promise((resolve, reject) => {
       this.resolveConnect = resolve
@@ -133,6 +134,11 @@ class FakeMember extends EventEmitter {
     return this.recoveryDescriptors.get(identifier)
   }
 
+  /** @param {{host: string, port: number}} target - Logical target. */
+  isTargetUnavailable(target) {
+    return this.unavailableTargets.has(`${target.host}:${target.port}`)
+  }
+
   status() {
     return {
       physicalState: this.state,
@@ -558,6 +564,21 @@ describe("multiplexed transport set", () => {
     await set.close()
   })
 
+  it("a member marked down for one target keeps serving every other target", async () => {
+    const {set, members} = fixture()
+    const started = set.start()
+    members.forEach((member) => member.ready())
+    await started
+    at(members, 0).unavailableTargets.add("broken.internal:443")
+    for (let offset = 0; offset < 6; offset += 1) {
+      const socket = set.createConnection({host: "broken.internal", port: 443})
+      expect(gatewayOf(socket) === "starlink").toBe(false)
+    }
+    const healthy = set.createConnection({host: "healthy.internal", port: 443})
+    expect(gatewayOf(healthy)).toBe("starlink")
+    await set.close()
+  })
+
   it("a stale gateway-unavailable mark cannot wedge the set when every ready member is marked", async () => {
     const {set, members} = fixture()
     const started = set.start()

package/test/unit/logging/structured-logger.test.js

@@ -468,6 +468,17 @@ describe("structured logger", {timeoutMs: 15000}, () => {
     }
   })
 
+  it("sanitizes the rejected-opens-by-code histogram", () => {
+    /** @type {string[]} */
+    const lines = []
+    const logger = createStructuredLogger({role: "relay", writeLine: (line) => lines.push(line)})
+    logger.log("runtime.status", {rejectedOpensByCode: {GATEWAY_UNAVAILABLE: 3, UNKNOWN_GATEWAY_ID: 1}})
+    expect(JSON.parse(lines[0] ?? "{}").rejectedOpensByCode).toEqual({GATEWAY_UNAVAILABLE: 3, UNKNOWN_GATEWAY_ID: 1})
+
+    logger.log("runtime.status", {rejectedOpensByCode: {GATEWAY_UNAVAILABLE: 3, PRIVATE_TOKEN: 2}})
+    expect(JSON.parse(lines[1] ?? "{}").rejectedOpensByCode).toBeUndefined()
+  })
+
   it("oversized sanitized records fall back to a bounded envelope", () => {
     /** @type {string[]} */
     const lines = []