socketduct 0.1.21 → 0.1.23

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,3 @@
1
+ # Outbound reverse gateway spool directory environment variable
2
+
3
+ The standalone outbound reverse gateway runtime now accepts `SOCKETDUCT_SPOOL_DIRECTORY`, which points the gateway's durable target-output spool at an operator-chosen persistent path. The default remains the process temporary directory, so containerized gateways can move the bounded spool behind exactly-once resume replay onto a host volume instead of the container's ephemeral, size-bounded `/tmp`. The variable is accepted only in `reverse-gateway` mode and is rejected in every other mode.
@@ -0,0 +1,3 @@
1
+ # Bound the compressed-log critical reserve test to a coverable charge
2
+
3
+ The compressed-log queue-pressure test asserted that a critical disconnect record is accepted while a 4096-byte queue holds a 1024-byte critical reserve, but the record's conservative charge (line bytes × 3 + 128) exceeds the reserve, so the writer correctly dropped it and the test failed on every run. Accepted log sequences are append-only and cannot be evicted without tearing the durable journal, so the test now reserves 2048 bytes — covering the critical record's charge — and the logging docs state the reserve sizing bound explicitly.
@@ -0,0 +1,3 @@
1
+ # Preserve terminal outcomes and coordinate paired resume deadlines for retained reverse streams
2
+
3
+ The reverse hub now retains the exact terminal code and incident attribution for a stream that ends while its peer owner is still resumable, so a reconnecting peer that presents the stream in its resume manifest receives the causal result instead of `RESUME_SESSION_LOST` (genuinely absent state still fails closed with `RESUME_SESSION_LOST`). While a final resume manifest waits on a temporarily detached counterpart, the hub extends the pair's original resume deadline by at most one additional resume window, once per retention episode, so reconnect backoff equal to the window no longer loses resumable streams.
@@ -0,0 +1,3 @@
1
+ # Bound retained reverse terminal outcomes and retain them only when delivery may have been missed
2
+
3
+ The reverse hub now retains a stream's exact terminal outcome only for an owner that may genuinely have missed the terminal delivery (physically absent, closing, or in paired-resume/pending-manifest state) — a healthy attached owner that receives the frame retains nothing — and hard-caps each owner's retained outcomes at the configured per-owner stream bound, evicting the oldest first. This prevents unbounded hub memory growth from authenticated stream churn while evicted streams still fail closed with `RESUME_SESSION_LOST` (never a replay or a target reopen).
package/docs/logging.md CHANGED
@@ -28,7 +28,11 @@ SOCKETDUCT_LOG_COMPRESSION_LEVEL=19
28
28
  Those values are the defaults. Reaching the uncompressed frame limit starts a
29
29
  flush immediately. Flush deadlines are measured from the oldest queued work and
30
30
  are never extended by later records; critical lifecycle, security, and resume
31
- events use the shorter deadline and may use the reserved queue capacity.
31
+ events use the shorter deadline and may use the reserved queue capacity. A
32
+ critical record is guaranteed acceptance only while its conservative charge
33
+ (line bytes × 3 + 128) fits the reserve, because accepted sequences are
34
+ append-only and can never be evicted from the durable journal; size the
35
+ reserve for the largest critical record you must retain under queue pressure.
32
36
  Segment size, age, and directory quota are bounded independently. Invalid or mutually
33
37
  inconsistent values stop startup.
34
38
  The former `SOCKETDUCT_LOG_MAX_FILE_BYTES` and `SOCKETDUCT_LOG_MAX_FILES`
@@ -98,7 +98,13 @@ replays only durable chunks above the receiver-confirmed high-water position,
98
98
  with its flow-control window reduced by bytes already buffered but not yet
99
99
  consumed at the receiver. A registered gateway completing this paired resume is
100
100
  not healthy or eligible for a new selection or `OPEN`; retained peers alone may
101
- attach to submit their resume manifests.
101
+ attach to submit their resume manifests. A stream that terminates while its
102
+ peer owner is still retained keeps its exact terminal code and incident
103
+ attribution, and a reconnecting peer that presents the stream in its resume
104
+ manifest receives that causal result; genuinely absent state still fails closed
105
+ with `RESUME_SESSION_LOST`. While a final resume manifest waits on a
106
+ temporarily detached counterpart, the hub extends the pair's original resume
107
+ deadline by at most one additional resume window, once per retention episode.
102
108
 
103
109
  The hub owns a separate ordered output queue for every client and gateway
104
110
  physical connection. Each queue defaults to a 32 MiB bound and can be set with
@@ -298,9 +304,17 @@ SOCKETDUCT_REVERSE_TOKEN='<secret>' \
298
304
  SOCKETDUCT_REVERSE_HUB_CA_FILE=/certs/hub-ca.crt \
299
305
  SOCKETDUCT_REVERSE_HUB_SERVERNAME=server3.example.net \
300
306
  SOCKETDUCT_TARGET_ALLOWLIST=docker-a:2375 \
307
+ SOCKETDUCT_SPOOL_DIRECTORY=/var/lib/socketduct/spool \
301
308
  node src/index.js
302
309
  ```
303
310
 
311
+ `SOCKETDUCT_SPOOL_DIRECTORY` is reverse-gateway-only and points the gateway's
312
+ durable target-output spool — the bounded store behind exactly-once resume
313
+ replay — at an operator-chosen path. Without it the spool lives under the
314
+ process temporary directory, which is ephemeral and size-bounded inside a
315
+ container; set it to a persistent host mount so spool capacity and retention
316
+ do not depend on container lifetime. The runtime rejects it in every other mode.
317
+
304
318
  The namespace/default-route configuration remains outside Socketduct. Each
305
319
  gateway registration represents one exact target server and WAN class;
306
320
  Socketduct does not aggregate away either identity. Socketduct needs no inbound
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "socketduct",
3
- "version": "0.1.21",
3
+ "version": "0.1.23",
4
4
  "description": "Resilient authenticated tunnel/proxy for keeping socket streams alive across flaky links.",
5
5
  "type": "module",
6
6
  "main": "./src/index.js",
@@ -54,6 +54,7 @@ const SUPPORTED_PROTOCOL_VERSIONS = new Set([2, 3])
54
54
  * @property {string} relayHost - Relay listen host.
55
55
  * @property {number} relayPort - Relay listen port.
56
56
  * @property {number} resumeWindowMs - Resume window in milliseconds.
57
+ * @property {string} [spoolDirectory] - Outbound gateway durable spool root; reverse-gateway mode only.
57
58
  * @property {number} maxTargetBufferBytes - Per-session byte limit for buffered target data while the tunnel is detached.
58
59
  * @property {number} maxTotalBufferBytes - Process-wide byte limit for buffered target data across all sessions.
59
60
  * @property {number} [initialTargetRetryDelayMs] - Initial delay between target acquisition attempts. Default 250.
@@ -143,6 +144,8 @@ export function loadEnvironmentConfig(env = process.env) {
143
144
  ...(reverseClientCertFile === undefined ? {} : {certFile: reverseClientCertFile, keyFile: /** @type {string} */ (reverseClientKeyFile)}),
144
145
  ...(env.SOCKETDUCT_REVERSE_HUB_SERVERNAME?.trim() ? {servername: env.SOCKETDUCT_REVERSE_HUB_SERVERNAME.trim()} : {})
145
146
  }
147
+ const spoolDirectory = env.SOCKETDUCT_SPOOL_DIRECTORY?.trim() || undefined
148
+ if (spoolDirectory !== undefined && mode !== "reverse-gateway") throw new Error("SOCKETDUCT_SPOOL_DIRECTORY is reverse-gateway-only")
146
149
  const logDirectory = env.SOCKETDUCT_LOG_DIRECTORY?.trim() || undefined
147
150
  if (env.SOCKETDUCT_LOG_MAX_FILE_BYTES !== undefined || env.SOCKETDUCT_LOG_MAX_FILES !== undefined) throw new Error("SOCKETDUCT_LOG_MAX_FILE_BYTES and SOCKETDUCT_LOG_MAX_FILES were replaced by compressed log frame, segment, and retention settings")
148
151
  let fileLogging
@@ -251,7 +254,8 @@ export function loadEnvironmentConfig(env = process.env) {
251
254
  port: parsePort(env.SOCKETDUCT_REVERSE_HUB_PORT, "SOCKETDUCT_REVERSE_HUB_PORT", DEFAULT_RELAY_PORT),
252
255
  token: reverseToken ?? ""
253
256
  },
254
- reverseHubTls
257
+ reverseHubTls,
258
+ ...(spoolDirectory === undefined ? {} : {spoolDirectory})
255
259
  } : {})
256
260
  }
257
261
  }
@@ -32,6 +32,7 @@ import {
32
32
  sanitizePeerControlDetail,
33
33
  validateFingerprintKeyEpoch
34
34
  } from "./disconnect-attribution.js"
35
+ import {insertBoundedTerminalOutcome, ownerMayMissTerminalDelivery} from "./terminal-outcomes.js"
35
36
 
36
37
  const DEFAULT_OUTPUT_BUFFER_BYTES = 32 * 1024 * 1024
37
38
  const DEFAULT_RESUME_MS = 60_000
@@ -84,8 +85,9 @@ const MANIFEST_PAGE_STREAMS = 64
84
85
  * @property {number} resumeNextPage
85
86
  * @property {Array<Record<string, unknown>>} resumeItems
86
87
  * @property {boolean} resumeFinal
87
- * @property {Array<{streamId: number, bridge: Bridge | undefined, code?: string}>} resumeEntries
88
+ * @property {Array<{streamId: number, bridge: Bridge | undefined, code?: string, attribution?: import('./disconnect-attribution.js').DisconnectAttribution}>} resumeEntries
88
89
  * @property {ReturnType<typeof setTimeout> | undefined} resumeTimer
90
+ * @property {boolean} resumeExtended - Whether this physical already used its one bounded paired-resume extension.
89
91
  * @property {(() => void) | undefined} clearHandshake
90
92
  * @property {HubOutputItem[]} outputQueue
91
93
  * @property {number} outputBytes
@@ -97,6 +99,7 @@ const MANIFEST_PAGE_STREAMS = 64
97
99
  * @typedef {object} HubOutputItem
98
100
  * @property {Buffer} bytes
99
101
  * @property {(delivered: boolean) => void} resolve
102
+ * @typedef {{code: string, attribution: import('./disconnect-attribution.js').DisconnectAttribution | undefined}} TerminalOutcome
100
103
  * @typedef {object} GatewayOwner
101
104
  * @property {string} gatewayId
102
105
  * @property {string} connectionId
@@ -104,7 +107,10 @@ const MANIFEST_PAGE_STREAMS = 64
104
107
  * @property {number} generation
105
108
  * @property {HubPhysical | undefined} physical
106
109
  * @property {Map<number, Bridge>} streams
110
+ * @property {Map<string, TerminalOutcome>} terminalOutcomes - Exact terminal outcomes for streams this owner may still present.
107
111
  * @property {ReturnType<typeof setTimeout> | undefined} expiryTimer
112
+ * @property {number | undefined} detachedAt - Epoch start of the current retention episode.
113
+ * @property {boolean} expiryExtended - Whether the current episode already used its one bounded extension.
108
114
  * @typedef {object} ClientOwner
109
115
  * @property {string} connectionId
110
116
  * @property {string} userId
@@ -113,7 +119,10 @@ const MANIFEST_PAGE_STREAMS = 64
113
119
  * @property {Map<number, Bridge>} streams
114
120
  * @property {string | undefined} gatewayId
115
121
  * @property {string | undefined} routeClass
122
+ * @property {Map<string, TerminalOutcome>} terminalOutcomes - Exact terminal outcomes for streams this owner may still present.
116
123
  * @property {ReturnType<typeof setTimeout> | undefined} expiryTimer
124
+ * @property {number | undefined} detachedAt - Epoch start of the current retention episode.
125
+ * @property {boolean} expiryExtended - Whether the current episode already used its one bounded extension.
117
126
  * @typedef {object} Bridge
118
127
  * @property {HubPhysical} client
119
128
  * @property {ClientOwner} clientOwner
@@ -225,6 +234,7 @@ export async function startReverseGatewayHub(/** @type {ReverseGatewayHubOptions
225
234
  resumeFinal: false,
226
235
  resumeEntries: [],
227
236
  resumeTimer: undefined,
237
+ resumeExtended: false,
228
238
  clearHandshake: undefined,
229
239
  outputQueue: [],
230
240
  outputBytes: 0,
@@ -362,11 +372,16 @@ export async function startReverseGatewayHub(/** @type {ReverseGatewayHubOptions
362
372
  streams: new Map(),
363
373
  gatewayId: undefined,
364
374
  routeClass: undefined,
365
- expiryTimer: undefined
375
+ terminalOutcomes: new Map(),
376
+ expiryTimer: undefined,
377
+ detachedAt: undefined,
378
+ expiryExtended: false
366
379
  }
367
380
  owner.generation = physical.generation
368
381
  owner.physical = physical
369
382
  owner.expiryTimer = undefined
383
+ owner.detachedAt = undefined
384
+ owner.expiryExtended = false
370
385
  physical.streams = owner.streams
371
386
  physical.lastClientStreamId = Math.max(0, ...owner.streams.keys())
372
387
  physical.resumeRequired = existing !== undefined && owner.streams.size > 0
@@ -415,11 +430,16 @@ export async function startReverseGatewayHub(/** @type {ReverseGatewayHubOptions
415
430
  generation: physical.generation,
416
431
  physical: undefined,
417
432
  streams: new Map(),
418
- expiryTimer: undefined
433
+ terminalOutcomes: new Map(),
434
+ expiryTimer: undefined,
435
+ detachedAt: undefined,
436
+ expiryExtended: false
419
437
  }
420
438
  owner.generation = physical.generation
421
439
  owner.physical = physical
422
440
  owner.expiryTimer = undefined
441
+ owner.detachedAt = undefined
442
+ owner.expiryExtended = false
423
443
  physical.streams = owner.streams
424
444
  physical.nextGatewayStreamId = Math.max(0, ...owner.streams.keys()) + 2
425
445
  physical.resumeRequired = existing !== undefined && owner.streams.size > 0
@@ -480,7 +500,15 @@ export async function startReverseGatewayHub(/** @type {ReverseGatewayHubOptions
480
500
  presented.add(item.streamId)
481
501
  const bridge = physical.streams.get(item.streamId)
482
502
  if (bridge === undefined) {
483
- physical.resumeEntries.push({streamId: item.streamId, bridge: undefined, code: "RESUME_SESSION_LOST"})
503
+ const owner = physical.role === "client" ? clients.get(/** @type {string} */ (physical.connectionId)) : gateways.get(/** @type {string} */ (physical.gatewayId))
504
+ const retainedKey = `${item.streamId}\0${item.identifier}`
505
+ const retained = owner === undefined ? undefined : owner.terminalOutcomes.get(retainedKey)
506
+ if (owner !== undefined && retained !== undefined) {
507
+ owner.terminalOutcomes.delete(retainedKey)
508
+ physical.resumeEntries.push({streamId: item.streamId, bridge: undefined, code: retained.code, ...(retained.attribution === undefined ? {} : {attribution: retained.attribution})})
509
+ } else {
510
+ physical.resumeEntries.push({streamId: item.streamId, bridge: undefined, code: "RESUME_SESSION_LOST"})
511
+ }
484
512
  continue
485
513
  }
486
514
  if (bridge.identifier !== item.identifier) {
@@ -506,6 +534,7 @@ export async function startReverseGatewayHub(/** @type {ReverseGatewayHubOptions
506
534
  physical.resumeItems = []
507
535
  physical.resumeFinal = true
508
536
  await tryCompleteResumes()
537
+ if (physical.resumeRequired) coordinateResumeDeadlines(physical)
509
538
  }
510
539
 
511
540
  /** Completes physical resume responses only after both logical endpoints have declared durable positions. */
@@ -538,7 +567,19 @@ export async function startReverseGatewayHub(/** @type {ReverseGatewayHubOptions
538
567
  for (const entry of physical.resumeEntries) {
539
568
  const bridge = entry.bridge
540
569
  if (bridge === undefined || !bridgeStillOwned(bridge)) {
541
- results.push({streamId: entry.streamId, status: "reset", code: entry.code ?? "RESUME_SESSION_LOST"})
570
+ let code = entry.code ?? "RESUME_SESSION_LOST"
571
+ let attribution = entry.attribution
572
+ if (bridge !== undefined) {
573
+ const owner = physical.role === "client" ? clients.get(/** @type {string} */ (physical.connectionId)) : gateways.get(/** @type {string} */ (physical.gatewayId))
574
+ const retainedKey = `${entry.streamId}\0${bridge.identifier}`
575
+ const retained = owner === undefined ? undefined : owner.terminalOutcomes.get(retainedKey)
576
+ if (owner !== undefined && retained !== undefined) {
577
+ owner.terminalOutcomes.delete(retainedKey)
578
+ code = retained.code
579
+ attribution = retained.attribution
580
+ }
581
+ }
582
+ results.push({streamId: entry.streamId, status: "reset", code, ...(attribution === undefined ? {} : attribution)})
542
583
  continue
543
584
  }
544
585
  try {
@@ -594,6 +635,37 @@ export async function startReverseGatewayHub(/** @type {ReverseGatewayHubOptions
594
635
  physical.resumeTimer = undefined
595
636
  }
596
637
 
638
+ /**
639
+ * Coordinates the paired resume deadlines while a final manifest waits on a
640
+ * temporarily detached counterpart: one bounded extension of the original
641
+ * retention deadlines so reconnect backoff equal to the window cannot win.
642
+ */
643
+ function coordinateResumeDeadlines(/** @type {HubPhysical} */ physical) {
644
+ let deadline
645
+ for (const entry of physical.resumeEntries) {
646
+ const bridge = entry.bridge
647
+ if (bridge === undefined || !bridgeStillOwned(bridge)) continue
648
+ const peerOwner = physical.role === "client" ? bridge.gatewayOwner : bridge.clientOwner
649
+ if (peerOwner.physical !== undefined || peerOwner.expiryTimer === undefined || peerOwner.detachedAt === undefined) continue
650
+ if (peerOwner.expiryExtended !== true) {
651
+ peerOwner.expiryExtended = true
652
+ const extendedDeadline = peerOwner.detachedAt + 2 * resumeWindowMs
653
+ clearTimeout(peerOwner.expiryTimer)
654
+ peerOwner.expiryTimer = setTimeout(() => {
655
+ void expireOwner(peerOwner, physical.role === "client" ? "gateway" : "client")
656
+ }, Math.max(0, extendedDeadline - Date.now()))
657
+ peerOwner.expiryTimer.unref?.()
658
+ }
659
+ const peerDeadline = peerOwner.detachedAt + (peerOwner.expiryExtended === true ? 2 : 1) * resumeWindowMs
660
+ deadline = Math.max(deadline ?? peerDeadline, peerDeadline)
661
+ }
662
+ if (deadline === undefined || physical.resumeExtended === true || physical.resumeTimer === undefined || deadline <= Date.now()) return
663
+ physical.resumeExtended = true
664
+ clearTimeout(physical.resumeTimer)
665
+ physical.resumeTimer = setTimeout(() => terminate(physical, "RESUME_WINDOW_EXPIRED"), deadline - Date.now())
666
+ physical.resumeTimer.unref?.()
667
+ }
668
+
597
669
  /** Reconciles one bridge to receiver-committed byte positions for a fresh physical generation. */
598
670
  function reconcileBridge(/** @type {Bridge} */ bridge) {
599
671
  const client = bridge.clientResume
@@ -808,6 +880,7 @@ export async function startReverseGatewayHub(/** @type {ReverseGatewayHubOptions
808
880
  forwardControl(destination, frame.type, destinationStreamId, frame.sequence, detail)
809
881
  if (frame.type === V3FrameType.RESET || fromClient) {
810
882
  removeBridge(bridge)
883
+ if (typeof detail.code === "string") retainTerminalOutcome(bridge, detail.code, attribution)
811
884
  if (attribution !== undefined && bridge.disconnectAttribution !== undefined) {
812
885
  logDiagnostic(options.logger, "reverse_hub.streams_reset", {
813
886
  ...attribution,
@@ -840,6 +913,33 @@ export async function startReverseGatewayHub(/** @type {ReverseGatewayHubOptions
840
913
  forwardOutput(destination, encodeV3Frame({type, streamId, sequence}, encodeV3Control(value)))
841
914
  }
842
915
 
916
+ /**
917
+ * Retains the exact terminal outcome for each still-registered owner that may
918
+ * genuinely have missed the terminal delivery, bounded to that owner's stream
919
+ * limit. A healthy attached owner already received the frame and will not
920
+ * re-present the stream, so it retains nothing.
921
+ * @param {Bridge} bridge - Stream.
922
+ * @param {string} code - Terminal code.
923
+ * @param {import('./disconnect-attribution.js').DisconnectAttribution | undefined} attribution - Incident attribution.
924
+ */
925
+ function retainTerminalOutcome(bridge, code, attribution) {
926
+ retainTerminalOutcomeFor(clients.get(bridge.clientOwner.connectionId), bridge.clientOwner, `${bridge.clientStreamId}\0${bridge.identifier}`, code, attribution)
927
+ retainTerminalOutcomeFor(gateways.get(bridge.gatewayOwner.gatewayId), bridge.gatewayOwner, `${bridge.gatewayStreamId}\0${bridge.identifier}`, code, attribution)
928
+ }
929
+
930
+ /**
931
+ * @param {ClientOwner | GatewayOwner | undefined} owner - Current owner record.
932
+ * @param {ClientOwner | GatewayOwner} bridgeOwner - Owner recorded on the bridge.
933
+ * @param {string} key - Stream identity key.
934
+ * @param {string} code - Terminal code.
935
+ * @param {import('./disconnect-attribution.js').DisconnectAttribution | undefined} attribution - Incident attribution.
936
+ */
937
+ function retainTerminalOutcomeFor(owner, bridgeOwner, key, code, attribution) {
938
+ if (owner === undefined || owner !== bridgeOwner) return
939
+ if (ownerMayMissTerminalDelivery(owner.physical) !== true) return
940
+ insertBoundedTerminalOutcome(owner.terminalOutcomes, key, {code, attribution}, maxStreams)
941
+ }
942
+
843
943
  /** @param {Bridge} bridge - Stream. @param {string} code - Reset code. @param {HubPhysical} [excluded] - Already-dead peer. @returns {Promise<void>} Terminal control delivery. */
844
944
  function closeBridge(bridge, code, excluded) {
845
945
  const owned = bridgeStillOwned(bridge)
@@ -854,6 +954,7 @@ export async function startReverseGatewayHub(/** @type {ReverseGatewayHubOptions
854
954
  if (bridge.client !== excluded && !bridge.client.socket.destroyed) deliveries.push(sendControl(bridge.client, V3FrameType.RESET, bridge.clientStreamId, 0, detail))
855
955
  if (bridge.gateway !== excluded && !bridge.gateway.socket.destroyed) deliveries.push(sendControl(bridge.gateway, V3FrameType.RESET, bridge.gatewayStreamId, 0, detail))
856
956
  removeBridge(bridge)
957
+ retainTerminalOutcome(bridge, code, attribution)
857
958
  if (attribution !== undefined) logDiagnostic(options.logger, "reverse_hub.streams_reset", {
858
959
  ...attribution,
859
960
  gatewayId: bridge.gatewayOwner.gatewayId,
@@ -999,6 +1100,10 @@ export async function startReverseGatewayHub(/** @type {ReverseGatewayHubOptions
999
1100
  /** @param {GatewayOwner | ClientOwner} owner - Detached identity. @param {'gateway' | 'client'} role - Owner role. */
1000
1101
  function scheduleOwnerExpiry(owner, role) {
1001
1102
  if (owner.expiryTimer !== undefined) clearTimeout(owner.expiryTimer)
1103
+ if (owner.detachedAt === undefined) {
1104
+ owner.detachedAt = Date.now()
1105
+ owner.expiryExtended = false
1106
+ }
1002
1107
  owner.expiryTimer = setTimeout(() => {
1003
1108
  void expireOwner(owner, role)
1004
1109
  }, resumeWindowMs)
@@ -1010,6 +1115,7 @@ export async function startReverseGatewayHub(/** @type {ReverseGatewayHubOptions
1010
1115
  owner.expiryTimer = undefined
1011
1116
  if (owner.physical !== undefined) return
1012
1117
  for (const bridge of [...owner.streams.values()]) await closeBridge(bridge, "RESUME_WINDOW_EXPIRED")
1118
+ owner.terminalOutcomes.clear()
1013
1119
  if (role === "gateway") {
1014
1120
  const gatewayOwner = /** @type {GatewayOwner} */ (owner)
1015
1121
  if (gateways.get(gatewayOwner.gatewayId) === gatewayOwner) gateways.delete(gatewayOwner.gatewayId)
@@ -80,6 +80,7 @@ export async function startOutboundReverseGatewayRuntime(config, logger) {
80
80
  maxStreamBufferBytes: config.maxTargetBufferBytes,
81
81
  maxTotalBufferBytes: config.maxTotalBufferBytes,
82
82
  compressionLimits: compressionLimits(config),
83
+ ...(config.spoolDirectory === undefined ? {} : {spoolDirectory: config.spoolDirectory}),
83
84
  ...(logger === undefined ? {} : {logger})
84
85
  })
85
86
  await gateway.start()
@@ -0,0 +1,55 @@
1
+ /**
2
+ * Bounded, eligibility-gated retention of exact terminal outcomes for reverse
3
+ * stream owners.
4
+ *
5
+ * A retained terminal outcome only matters to an owner that may genuinely have
6
+ * missed the terminal frame. A healthy attached physical receives the frame and
7
+ * retires the stream locally, so it never re-presents the stream and must not
8
+ * accumulate retained outcomes — unbounded growth under authenticated stream
9
+ * churn would otherwise exhaust hub memory. Retention is additionally hard-capped
10
+ * to the owner's configured stream bound so that even a detached/resuming owner
11
+ * cannot retain more outcomes than it could ever present; eviction is fail-closed
12
+ * (the evicted stream later resolves as RESUME_SESSION_LOST, never a replay or a
13
+ * target reopen).
14
+ */
15
+
16
+ /**
17
+ * @typedef {object} TerminalOutcome
18
+ * @property {string} code - Exact terminal classification.
19
+ * @property {import('./disconnect-attribution.js').DisconnectAttribution | undefined} attribution - Validated incident, when one was in flight.
20
+ */
21
+
22
+ /**
23
+ * @typedef {object} OwnerPhysicalState
24
+ * @property {boolean} closing - Whether the physical close has begun.
25
+ * @property {boolean} resumeRequired - Whether the physical is still in its paired-resume / pending-manifest state.
26
+ */
27
+
28
+ /**
29
+ * Decides whether an owner may have missed a terminal delivery and therefore
30
+ * needs the exact outcome retained for a later resume manifest.
31
+ * @param {OwnerPhysicalState | undefined} physical - The owner's current physical state.
32
+ * @returns {boolean} Whether the terminal outcome should be retained for the owner.
33
+ */
34
+ export function ownerMayMissTerminalDelivery(/** @type {OwnerPhysicalState | undefined} */ physical) {
35
+ if (physical === undefined) return true
36
+ return physical.closing || physical.resumeRequired
37
+ }
38
+
39
+ /**
40
+ * Inserts one terminal outcome into an owner's bounded retention map, evicting
41
+ * the oldest entry (deterministic Map insertion order) whenever the insertion
42
+ * would exceed the bound. Re-inserting a retained key updates it in place.
43
+ * @param {Map<string, TerminalOutcome>} outcomes - The owner's retained outcomes.
44
+ * @param {string} key - The stream identity key.
45
+ * @param {TerminalOutcome} outcome - The terminal outcome to retain.
46
+ * @param {number} limit - The maximum number of retained outcomes.
47
+ */
48
+ export function insertBoundedTerminalOutcome(/** @type {Map<string, TerminalOutcome>} */ outcomes, /** @type {string} */ key, /** @type {TerminalOutcome} */ outcome, /** @type {number} */ limit) {
49
+ outcomes.set(key, outcome)
50
+ while (outcomes.size > limit) {
51
+ const oldest = outcomes.keys().next()
52
+ if (oldest.done === true) break
53
+ outcomes.delete(/** @type {string} */ (oldest.value))
54
+ }
55
+ }
@@ -1,7 +1,7 @@
1
1
  /* eslint-disable jsdoc/escape-inline-tags, jsdoc/require-param, jsdoc/require-returns */
2
2
  import assert from "node:assert/strict"
3
3
  import {spawn} from "node:child_process"
4
- import {mkdtemp, rm} from "node:fs/promises"
4
+ import {mkdtemp, readFile, readdir, rm} from "node:fs/promises"
5
5
  import {createServer} from "node:net"
6
6
  import {tmpdir} from "node:os"
7
7
  import {join} from "node:path"
@@ -131,6 +131,31 @@ async function withTimeout(promise, label) {
131
131
  }
132
132
  }
133
133
 
134
+ /** @param {string} directory - Directory. @returns {Promise<string[]>} Entries, or an empty list for a missing directory. */
135
+ async function readDirectoryOrEmpty(directory) {
136
+ try {
137
+ return await readdir(directory)
138
+ } catch (error) {
139
+ if (error instanceof Error && /** @type {Error & {code?: string}} */ (error).code === "ENOENT") return []
140
+ throw error
141
+ }
142
+ }
143
+
144
+ /**
145
+ * Polls until the probe resolves truthy, bounded by the process timeout.
146
+ * @param {() => boolean | Promise<boolean>} probe - Condition to test.
147
+ * @param {string} label - Timeout label.
148
+ * @returns {Promise<void>}
149
+ */
150
+ async function waitUntil(probe, label) {
151
+ const deadline = Date.now() + PROCESS_TIMEOUT_MS
152
+ for (;;) {
153
+ if (await probe()) return
154
+ if (Date.now() >= deadline) throw new Error(`Timed out: ${label}`)
155
+ await new Promise((resolve) => setTimeout(resolve, 50))
156
+ }
157
+ }
158
+
134
159
  /** @param {import('node:events').EventEmitter} emitter - Source. @param {string} event - Event. */
135
160
  async function onceEvent(emitter, event) {
136
161
  await new Promise((resolve, reject) => {
@@ -377,3 +402,97 @@ test("one route-class client physical connection interleaves streams through dif
377
402
  streamA.destroy()
378
403
  streamB.destroy()
379
404
  })
405
+
406
+ test("outbound gateway process spools durably under SOCKETDUCT_SPOOL_DIRECTORY when the hub link is lost", async (t) => {
407
+ const root = await mkdtemp(join(tmpdir(), "socketduct-reverse-spool-process-"))
408
+ t.after(() => rm(root, {recursive: true, force: true}))
409
+ const certificates = generateTestCertificates(root)
410
+ const gatewaySpoolDirectory = join(root, "gateway-spool")
411
+ const clientSpoolDirectory = join(root, "client-spool")
412
+ /** @type {import('node:net').Socket | undefined} */
413
+ let targetSocket
414
+ const targetSockets = new Set()
415
+ const target = createServer((socket) => {
416
+ targetSocket = socket
417
+ targetSockets.add(socket)
418
+ socket.once("close", () => targetSockets.delete(socket))
419
+ socket.on("data", (chunk) => socket.write(Buffer.concat([Buffer.from("docker:"), chunk])))
420
+ })
421
+ const targetPort = await listen(target)
422
+ t.after(() => {
423
+ for (const socket of targetSockets) socket.destroy()
424
+ return new Promise((resolve) => target.close(() => resolve(undefined)))
425
+ })
426
+ const {child: hub, port: hubPort} = await startProcessOnReservedPort({
427
+ SOCKETDUCT_MODE: "reverse-hub",
428
+ SOCKETDUCT_RELAY_HOST: "127.0.0.1",
429
+ SOCKETDUCT_TLS_CERT_FILE: certificates.serverCertFile,
430
+ SOCKETDUCT_TLS_KEY_FILE: certificates.serverKeyFile,
431
+ SOCKETDUCT_AUTH_TOKENS: "home-o2:spool-gateway-token,tensorbuzz:spool-client-token",
432
+ SOCKETDUCT_REVERSE_GATEWAY_BINDINGS: "o2:home-o2",
433
+ SOCKETDUCT_REVERSE_ROUTES: `tensorbuzz:o2:127.0.0.1:${targetPort}`,
434
+ SOCKETDUCT_STATUS_INTERVAL_MS: "0"
435
+ }, "reverse-hub.started")
436
+ const gateway = await startProcess({
437
+ SOCKETDUCT_MODE: "reverse-gateway",
438
+ SOCKETDUCT_REVERSE_GATEWAY_ID: "o2",
439
+ SOCKETDUCT_REVERSE_CONNECTION_ID: "spool_process_gateway",
440
+ SOCKETDUCT_REVERSE_HUB_HOST: "127.0.0.1",
441
+ SOCKETDUCT_REVERSE_HUB_PORT: String(hubPort),
442
+ SOCKETDUCT_REVERSE_TOKEN: "spool-gateway-token",
443
+ SOCKETDUCT_REVERSE_HUB_CA_FILE: certificates.caCertFile,
444
+ SOCKETDUCT_REVERSE_HUB_SERVERNAME: "localhost",
445
+ SOCKETDUCT_TARGET_ALLOWLIST: `127.0.0.1:${targetPort}`,
446
+ SOCKETDUCT_STATUS_INTERVAL_MS: "0",
447
+ SOCKETDUCT_SPOOL_DIRECTORY: gatewaySpoolDirectory
448
+ }, "reverse-gateway.started")
449
+ t.after(() => stopProcess(gateway))
450
+ const client = new ReverseGatewayConnection({
451
+ gatewayId: "o2",
452
+ connectionId: "spool_process_client",
453
+ hub: {host: "127.0.0.1", port: hubPort, token: "spool-client-token"},
454
+ tls: {caFile: certificates.caCertFile, servername: "localhost"},
455
+ spoolDirectory: clientSpoolDirectory,
456
+ reconnectDelayMs: 50,
457
+ maxReconnectDelayMs: 100
458
+ })
459
+ t.after(() => client.close())
460
+ await withTimeout(client.connect(), "spool client connect")
461
+ const socket = client.createConnection({host: "127.0.0.1", port: targetPort, sessionName: "docker-spool-process"})
462
+ await withTimeout(onceEvent(socket, "connect"), "spool stream open")
463
+ const initialResponse = new Promise((resolve, reject) => { socket.once("error", reject); socket.once("data", resolve) })
464
+ socket.write("GET /version")
465
+ assert.equal(Buffer.from(/** @type {Uint8Array} */ (await withTimeout(initialResponse, "spool initial response"))).toString(), "docker:GET /version")
466
+
467
+ hub.kill("SIGKILL")
468
+ await withTimeout(new Promise((resolve) => hub.once("exit", () => resolve(undefined))), "hub exit")
469
+ assert(targetSocket !== undefined)
470
+ const spooledMarker = Buffer.alloc(32, 0x61)
471
+ targetSocket.write(spooledMarker)
472
+ const spoolHasPayload = async () => {
473
+ const markerBase64 = spooledMarker.toString("base64")
474
+ for (const session of await readDirectoryOrEmpty(gatewaySpoolDirectory)) {
475
+ const files = await readDirectoryOrEmpty(join(gatewaySpoolDirectory, session))
476
+ if (!files.includes("chunks.jsonl")) continue
477
+ const lines = (await readFile(join(gatewaySpoolDirectory, session, "chunks.jsonl"), "utf8")).split("\n")
478
+ if (lines.some((line) => {
479
+ if (line === "") return false
480
+ try {
481
+ return /** @type {{dataBase64?: unknown}} */ (JSON.parse(line)).dataBase64 === markerBase64
482
+ } catch (error) {
483
+ if (error instanceof SyntaxError) return false
484
+ throw error
485
+ }
486
+ })) return true
487
+ }
488
+ return false
489
+ }
490
+ await waitUntil(spoolHasPayload, "durable spool chunk under SOCKETDUCT_SPOOL_DIRECTORY")
491
+ const defaultGatewaySpoolRoot = join(tmpdir(), `socketduct-${typeof process.getuid === "function" ? process.getuid() : "user"}`, "reverse-gateway")
492
+ const defaultSpooled = (await readDirectoryOrEmpty(defaultGatewaySpoolRoot)).some((entry) => entry.startsWith("o2-spool_process_gateway"))
493
+ assert.equal(defaultSpooled, false, "spool must not fall back to the ephemeral default root")
494
+
495
+ socket.destroy()
496
+ client.close()
497
+ await stopProcess(gateway)
498
+ })
@@ -0,0 +1,333 @@
1
+ /* eslint-disable jsdoc/escape-inline-tags, jsdoc/require-param, jsdoc/require-returns */
2
+ import assert from "node:assert/strict"
3
+ import {mkdtemp, rm} from "node:fs/promises"
4
+ import {connect as connectTcp, createServer} from "node:net"
5
+ import {tmpdir} from "node:os"
6
+ import {join} from "node:path"
7
+ import {Duplex} from "node:stream"
8
+ import test from "node:test"
9
+
10
+ import {StaticTokenStore} from "../../src/auth/static-token-store.js"
11
+ import {OutboundReverseGateway, ReverseGatewayConnection, startReverseGatewayHub} from "../../src/reverse/index.js"
12
+ import {generateTestCertificates} from "../helpers/tls-certificates.js"
13
+
14
+ /** @param {import('node:net').Server} server - Server. */
15
+ async function listen(server) {
16
+ await new Promise((resolve, reject) => {
17
+ server.once("error", reject)
18
+ server.listen(0, "127.0.0.1", () => { server.off("error", reject); resolve(undefined) })
19
+ })
20
+ const address = server.address()
21
+ assert(address !== null && typeof address !== "string")
22
+ return address.port
23
+ }
24
+
25
+ /** @param {import('node:events').EventEmitter} emitter - Emitter. @param {string} event - Event. */
26
+ async function onceEvent(emitter, event) {
27
+ await new Promise((resolve, reject) => {
28
+ emitter.once("error", reject)
29
+ emitter.once(event, () => { emitter.off("error", reject); resolve(undefined) })
30
+ })
31
+ }
32
+
33
+ /** @param {import('node:stream').Duplex} socket - Socket. @param {number} bytes - Expected bytes. */
34
+ async function readBytes(socket, bytes) {
35
+ return await new Promise((resolve, reject) => {
36
+ /** @type {Buffer[]} */
37
+ const chunks = []
38
+ let length = 0
39
+ const cleanup = () => { socket.off("data", onData); socket.off("error", onError) }
40
+ /** @param {Error} error - Stream failure. */
41
+ const onError = (error) => { cleanup(); reject(error) }
42
+ /** @param {Buffer} chunk - Stream bytes. */
43
+ const onData = (chunk) => {
44
+ chunks.push(chunk)
45
+ length += chunk.length
46
+ if (length < bytes) return
47
+ cleanup()
48
+ resolve(Buffer.concat(chunks))
49
+ }
50
+ socket.on("data", onData)
51
+ socket.on("error", onError)
52
+ })
53
+ }
54
+
55
+ /** @param {Promise<unknown>} promise - Work. @param {string} label - Timeout label. */
56
+ async function withTimeout(promise, label) {
57
+ let timer
58
+ try {
59
+ return await Promise.race([
60
+ promise,
61
+ new Promise((_, reject) => { timer = setTimeout(() => reject(new Error(`Timed out: ${label}`)), 2_000) })
62
+ ])
63
+ } finally {
64
+ if (timer !== undefined) clearTimeout(timer)
65
+ }
66
+ }
67
+
68
+ /** @param {() => boolean} predicate - Completion predicate. @param {string} label - Failure label. */
69
+ async function waitUntil(predicate, label) {
70
+ const deadline = Date.now() + 4_000
71
+ while (!predicate()) {
72
+ if (Date.now() >= deadline) throw new Error(`Timed out: ${label}`)
73
+ await new Promise((resolve) => setTimeout(resolve, 10))
74
+ }
75
+ }
76
+
77
+ /** @param {import('node:stream').Duplex} socket - Stream. @returns {Promise<Error & {code?: string}>} Next stream error. */
78
+ async function nextError(socket) {
79
+ return await new Promise((resolve) => socket.once("error", (error) => resolve(/** @type {Error & {code?: string}} */ (error))))
80
+ }
81
+
82
+ /** Target that records received bytes and can flood the gateway buffer on demand while the peer physical is down. */
83
+ class RecordingInterruptibleTarget extends Duplex {
84
+ /** @type {Buffer[]} */
85
+ chunks = []
86
+
87
+ constructor() {
88
+ super()
89
+ setImmediate(() => this.emit("connect"))
90
+ }
91
+
92
+ /** @override */
93
+ _read() {}
94
+
95
+ /** @param {Buffer} chunk - Received bytes. @param {BufferEncoding} _encoding - Ignored encoding. @param {(error?: Error | null) => void} callback - Completion. @override */
96
+ _write(chunk, _encoding, callback) {
97
+ this.chunks.push(Buffer.from(chunk))
98
+ callback()
99
+ }
100
+
101
+ /** @param {Buffer} payload - Target-originated bytes delivered to the gateway. */
102
+ flood(payload) { this.emit("data", payload) }
103
+ }
104
+
105
+ test("a paired resume completes when the first counterpart return lands just past the original resume window", async (t) => {
106
+ const root = await mkdtemp(join(tmpdir(), "socketduct-reverse-resume-extension-"))
107
+ t.after(() => rm(root, {recursive: true, force: true}))
108
+ const certificates = generateTestCertificates(root)
109
+ let targetAccepts = 0
110
+ const targetSockets = new Set()
111
+ const target = createServer((socket) => {
112
+ targetAccepts += 1
113
+ targetSockets.add(socket)
114
+ socket.once("close", () => targetSockets.delete(socket))
115
+ socket.on("data", (chunk) => socket.write(Buffer.concat([Buffer.from("target:"), chunk])))
116
+ })
117
+ const targetPort = await listen(target)
118
+ t.after(() => { for (const socket of targetSockets) socket.destroy(); return new Promise((resolve) => target.close(() => resolve(undefined))) })
119
+ const tokens = new StaticTokenStore([{username: "home", token: "gateway-token"}, {username: "app", token: "client-token"}])
120
+ const hub = await startReverseGatewayHub({tokens, tls: {certFile: certificates.serverCertFile, keyFile: certificates.serverKeyFile}, authorizeGateway: () => true, authorizeClientGateway: () => true, authorizeOpen: () => true, resumeWindowMs: 250})
121
+ t.after(() => hub.close())
122
+ const tls = {caFile: certificates.caCertFile, servername: "localhost"}
123
+ const gateway = new OutboundReverseGateway({gatewayId: "o2", connectionId: "extension_gateway_01", hub: {host: "127.0.0.1", port: hub.port, token: "gateway-token"}, tls, spoolDirectory: join(root, "gateway-spool"), authorizeTarget: () => true, reconnectDelayMs: 350, maxReconnectDelayMs: 350})
124
+ t.after(() => gateway.close())
125
+ await gateway.start()
126
+ const client = new ReverseGatewayConnection({gatewayId: "o2", connectionId: "extension_client_01", hub: {host: "127.0.0.1", port: hub.port, token: "client-token"}, tls, spoolDirectory: join(root, "client-spool"), reconnectDelayMs: 10, maxReconnectDelayMs: 20})
127
+ t.after(() => client.close())
128
+ await client.connect()
129
+ const socket = client.createConnection({identifier: "extension_stream_01", sessionName: "extension-stream", host: "127.0.0.1", port: targetPort})
130
+ await onceEvent(socket, "connect")
131
+ socket.write("before|")
132
+ const preResponse = readBytes(socket, 14)
133
+ assert.equal((await withTimeout(preResponse, "pre-resume target echo")).toString(), "target:before|")
134
+
135
+ await hub.closeActiveGateway("o2")
136
+ await waitUntil(() => gateway.status().physicalState === "reconnecting", "gateway interruption")
137
+ socket.write("after|")
138
+ const postResponse = readBytes(socket, 13)
139
+ // The gateway's first return (~350ms) lands after the original 250ms window;
140
+ // the bounded paired-resume extension must keep both deadlines open for it.
141
+ await waitUntil(() => gateway.status().physicalConnects >= 2 && gateway.status().physicalState === "ready", "gateway return just past the original window")
142
+ assert.equal((await withTimeout(postResponse, "post-resume target echo")).toString(), "target:after|")
143
+ assert.equal(targetAccepts, 1, "resume must keep the original logical target socket")
144
+ assert.equal(hub.status().virtualStreams, 1)
145
+
146
+ const closed = onceEvent(socket, "close")
147
+ socket.end()
148
+ await withTimeout(closed, "post-resume cleanup close")
149
+ await waitUntil(() => hub.status().virtualStreams === 0 && client.status().virtualStreams === 0 && gateway.status().virtualStreams === 0 && targetSockets.size === 0, "post-resume ownership cleanup")
150
+ })
151
+
152
+ test("a retained terminal outcome answers a reconnecting peer with the causal code instead of RESUME_SESSION_LOST", async (t) => {
153
+ const root = await mkdtemp(join(tmpdir(), "socketduct-reverse-retained-outcome-"))
154
+ t.after(() => rm(root, {recursive: true, force: true}))
155
+ const certificates = generateTestCertificates(root)
156
+ let acceptsA = 0
157
+ const targetSocketsA = new Set()
158
+ const targetA = createServer((socket) => {
159
+ acceptsA += 1
160
+ targetSocketsA.add(socket)
161
+ socket.once("close", () => targetSocketsA.delete(socket))
162
+ socket.on("data", (chunk) => socket.write(Buffer.concat([Buffer.from("a:"), chunk])))
163
+ })
164
+ const portA = await listen(targetA)
165
+ t.after(() => { for (const socket of targetSocketsA) socket.destroy(); return new Promise((resolve) => targetA.close(() => resolve(undefined))) })
166
+ const tokens = new StaticTokenStore([{username: "home", token: "gateway-token"}, {username: "app", token: "client-token"}])
167
+ const hub = await startReverseGatewayHub({tokens, tls: {certFile: certificates.serverCertFile, keyFile: certificates.serverKeyFile}, authorizeGateway: () => true, authorizeClientGateway: () => true, authorizeOpen: () => true, resumeWindowMs: 2_000})
168
+ t.after(() => hub.close())
169
+ const tls = {caFile: certificates.caCertFile, servername: "localhost"}
170
+ /** @type {RecordingInterruptibleTarget | undefined} */
171
+ let targetB
172
+ let targetBCreates = 0
173
+ const gateway = new OutboundReverseGateway({
174
+ gatewayId: "o2",
175
+ connectionId: "outcome_gateway_01",
176
+ hub: {host: "127.0.0.1", port: hub.port, token: "gateway-token"},
177
+ tls,
178
+ spoolDirectory: join(root, "gateway-spool"),
179
+ authorizeTarget: () => true,
180
+ reconnectDelayMs: 300,
181
+ maxReconnectDelayMs: 300,
182
+ maxStreamBufferBytes: 32,
183
+ createTargetConnection: (target) => {
184
+ if (target.port === 2376) {
185
+ targetBCreates += 1
186
+ targetB = new RecordingInterruptibleTarget()
187
+ return /** @type {import('node:net').Socket} */ (/** @type {unknown} */ (targetB))
188
+ }
189
+ return connectTcp(target)
190
+ }
191
+ })
192
+ t.after(() => gateway.close())
193
+ await gateway.start()
194
+ const client = new ReverseGatewayConnection({gatewayId: "o2", connectionId: "outcome_client_01", hub: {host: "127.0.0.1", port: hub.port, token: "client-token"}, tls, spoolDirectory: join(root, "client-spool"), reconnectDelayMs: 500, maxReconnectDelayMs: 500})
195
+ t.after(() => client.close())
196
+ await client.connect()
197
+ const socketA = client.createConnection({identifier: "outcome_stream_a", sessionName: "outcome-a", host: "127.0.0.1", port: portA})
198
+ const socketB = client.createConnection({identifier: "outcome_stream_b", sessionName: "outcome-b", host: "target-b", port: 2376})
199
+ await Promise.all([onceEvent(socketA, "connect"), onceEvent(socketB, "connect")])
200
+ socketA.write("one|")
201
+ const preA = readBytes(socketA, 6)
202
+ assert.equal((await withTimeout(preA, "stream A pre-interruption echo")).toString(), "a:one|")
203
+ socketB.write("two|")
204
+ await waitUntil(() => targetB !== undefined && Buffer.concat(targetB.chunks).toString() === "two|", "stream B bytes reached its target")
205
+ const terminalB = nextError(socketB)
206
+
207
+ // The gateway physical dies and the client peer is kicked into a slow (500ms)
208
+ // reconnect. While both are apart, target B overflows the gateway stream
209
+ // buffer: the gateway resets and destroys B in memory, but its RESET cannot
210
+ // reach the hub, so the hub still owns B when the gateway re-registers.
211
+ await hub.closeActiveGateway("o2")
212
+ await waitUntil(() => gateway.status().physicalState === "reconnecting", "gateway interruption")
213
+ await new Promise((resolve) => setTimeout(resolve, 100))
214
+ assert.equal(client.status().physicalState, "reconnecting", "the client peer must still be mid-reconnect when target B overflows")
215
+ targetB?.flood(Buffer.alloc(64, 0x42))
216
+ await waitUntil(() => hub.status().gatewayPhysicalConnections >= 1, "gateway re-registration before the peer returns")
217
+ assert.equal(targetB?.destroyed, true, "the overflowed stream target must be released in memory")
218
+
219
+ const error = /** @type {Error & Record<string, unknown>} */ (await withTimeout(terminalB, "stream B terminal outcome after peer return"))
220
+ assert.equal(error.code, "RESUME_STREAM_OMITTED", "the reconnecting peer must receive the retained causal code")
221
+ assert.match(String(error.disconnectIncidentId ?? ""), /^[A-Za-z0-9_-]{22}$/, "the retained outcome must carry the incident attribution")
222
+
223
+ const postA = readBytes(socketA, 12)
224
+ socketA.write("one-after|")
225
+ assert.equal((await withTimeout(postA, "stream A post-resume echo")).toString(), "a:one-after|")
226
+ assert.equal(acceptsA, 1, "the surviving stream must resume on its original target")
227
+ assert.equal(targetBCreates, 1, "the terminal stream must never reopen its target")
228
+ assert.equal(targetB?.destroyed, true, "the terminal stream target must be released exactly once")
229
+
230
+ const closedA = onceEvent(socketA, "close")
231
+ socketA.end()
232
+ await withTimeout(closedA, "stream A cleanup close")
233
+ await waitUntil(() => hub.status().virtualStreams === 0 && client.status().virtualStreams === 0 && gateway.status().virtualStreams === 0 && targetSocketsA.size === 0, "retained-outcome ownership cleanup")
234
+ })
235
+
236
+ test("a paired resume extension is bounded by one window and genuine expiry still releases state", async (t) => {
237
+ const root = await mkdtemp(join(tmpdir(), "socketduct-reverse-resume-bounded-"))
238
+ t.after(() => rm(root, {recursive: true, force: true}))
239
+ const certificates = generateTestCertificates(root)
240
+ let targetAccepts = 0
241
+ const targetSockets = new Set()
242
+ const target = createServer((socket) => {
243
+ targetAccepts += 1
244
+ targetSockets.add(socket)
245
+ socket.once("close", () => targetSockets.delete(socket))
246
+ })
247
+ const targetPort = await listen(target)
248
+ t.after(() => { for (const socket of targetSockets) socket.destroy(); return new Promise((resolve) => target.close(() => resolve(undefined))) })
249
+ const tokens = new StaticTokenStore([{username: "home", token: "gateway-token"}, {username: "app", token: "client-token"}])
250
+ const hub = await startReverseGatewayHub({tokens, tls: {certFile: certificates.serverCertFile, keyFile: certificates.serverKeyFile}, authorizeGateway: () => true, authorizeClientGateway: () => true, authorizeOpen: () => true, resumeWindowMs: 250})
251
+ t.after(() => hub.close())
252
+ const tls = {caFile: certificates.caCertFile, servername: "localhost"}
253
+ const gateway = new OutboundReverseGateway({gatewayId: "o2", connectionId: "bounded_expiry_gateway", hub: {host: "127.0.0.1", port: hub.port, token: "gateway-token"}, tls, spoolDirectory: join(root, "gateway-spool"), authorizeTarget: () => true, reconnectDelayMs: 1_000, maxReconnectDelayMs: 1_000})
254
+ t.after(() => gateway.close())
255
+ await gateway.start()
256
+ const client = new ReverseGatewayConnection({gatewayId: "o2", connectionId: "bounded_expiry_client", hub: {host: "127.0.0.1", port: hub.port, token: "client-token"}, tls, spoolDirectory: join(root, "client-spool"), reconnectDelayMs: 10, maxReconnectDelayMs: 20})
257
+ t.after(() => client.close())
258
+ await client.connect()
259
+ const socket = client.createConnection({identifier: "bounded_stream_01", sessionName: "bounded-stream", host: "127.0.0.1", port: targetPort})
260
+ const terminal = nextError(socket)
261
+ await onceEvent(socket, "connect")
262
+
263
+ await hub.closeActiveGateway("o2")
264
+ await waitUntil(() => gateway.status().physicalState === "reconnecting", "gateway interruption")
265
+ // The gateway returns at ~1000ms, far beyond the one bounded extension
266
+ // (original 250ms window extended once to 500ms), so the pair must expire.
267
+ const error = /** @type {Error & {code?: string}} */ (await withTimeout(terminal, "bounded expired resume"))
268
+ assert.equal(error.code, "RESUME_WINDOW_EXPIRED")
269
+ await waitUntil(() => hub.status().virtualStreams === 0 && client.status().virtualStreams === 0 && gateway.status().virtualStreams === 0 && targetSockets.size === 0, "bounded expiry releases all retained state")
270
+ assert.equal(targetAccepts, 1, "the expired stream must not reopen its target")
271
+ })
272
+
273
+ test("multiple concurrent streams share one physical interruption and resume together on their original targets", async (t) => {
274
+ const root = await mkdtemp(join(tmpdir(), "socketduct-reverse-multi-stream-resume-"))
275
+ t.after(() => rm(root, {recursive: true, force: true}))
276
+ const certificates = generateTestCertificates(root)
277
+ let targetAccepts = 0
278
+ const targetSockets = new Set()
279
+ const target = createServer((socket) => {
280
+ targetAccepts += 1
281
+ targetSockets.add(socket)
282
+ socket.once("close", () => targetSockets.delete(socket))
283
+ socket.on("data", (chunk) => socket.write(Buffer.concat([Buffer.from("t:"), chunk])))
284
+ })
285
+ const targetPort = await listen(target)
286
+ t.after(() => { for (const socket of targetSockets) socket.destroy(); return new Promise((resolve) => target.close(() => resolve(undefined))) })
287
+ const tokens = new StaticTokenStore([{username: "home", token: "gateway-token"}, {username: "app", token: "client-token"}])
288
+ const hub = await startReverseGatewayHub({tokens, tls: {certFile: certificates.serverCertFile, keyFile: certificates.serverKeyFile}, authorizeGateway: () => true, authorizeClientGateway: () => true, authorizeOpen: () => true, resumeWindowMs: 2_000})
289
+ t.after(() => hub.close())
290
+ const tls = {caFile: certificates.caCertFile, servername: "localhost"}
291
+ const gateway = new OutboundReverseGateway({gatewayId: "o2", connectionId: "multi_resume_gateway", hub: {host: "127.0.0.1", port: hub.port, token: "gateway-token"}, tls, spoolDirectory: join(root, "gateway-spool"), authorizeTarget: () => true, reconnectDelayMs: 30, maxReconnectDelayMs: 30})
292
+ t.after(() => gateway.close())
293
+ await gateway.start()
294
+ const client = new ReverseGatewayConnection({gatewayId: "o2", connectionId: "multi_resume_client", hub: {host: "127.0.0.1", port: hub.port, token: "client-token"}, tls, spoolDirectory: join(root, "client-spool"), reconnectDelayMs: 10, maxReconnectDelayMs: 20})
295
+ t.after(() => client.close())
296
+ await client.connect()
297
+ /** @type {import('node:stream').Duplex[]} */
298
+ const sockets = []
299
+ /** @type {Promise<string>[]} */
300
+ const echoes = []
301
+ for (let index = 1; index <= 3; index += 1) {
302
+ const socket = client.createConnection({identifier: `multi_stream_${index}`, sessionName: `multi-stream-${index}`, host: "127.0.0.1", port: targetPort})
303
+ sockets.push(socket)
304
+ await onceEvent(socket, "connect")
305
+ }
306
+ let echoIndex = 1
307
+ for (const socket of sockets) {
308
+ socket.write(`pre${echoIndex}|`)
309
+ echoes.push(readBytes(socket, 2 + 3 + String(echoIndex).length + 1).then((bytes) => bytes.toString()))
310
+ echoIndex += 1
311
+ }
312
+ const preEchoes = await Promise.all(echoes.map((promise) => withTimeout(promise, "pre-resume echo")))
313
+ assert.deepEqual(preEchoes, ["t:pre1|", "t:pre2|", "t:pre3|"])
314
+ assert.equal(targetAccepts, 3)
315
+
316
+ await hub.closeActiveGateway("o2")
317
+ let postIndex = 1
318
+ for (const socket of sockets) {
319
+ socket.write(`post${postIndex}|`)
320
+ postIndex += 1
321
+ }
322
+ /** @type {Promise<string>[]} */
323
+ const postEchoes = sockets.map((socket, index) => readBytes(socket, 2 + 4 + String(index + 1).length + 1).then((bytes) => bytes.toString()))
324
+ await waitUntil(() => gateway.status().physicalConnects >= 2 && gateway.status().physicalState === "ready", "shared gateway re-registration")
325
+ const postEchoesReceived = await Promise.all(postEchoes.map((promise) => withTimeout(promise, "post-resume echo")))
326
+ assert.deepEqual(postEchoesReceived, ["t:post1|", "t:post2|", "t:post3|"], "every stream must resume on its original target without cross-stream loss")
327
+ assert.equal(targetAccepts, 3, "the shared interruption must not create replacement targets")
328
+ assert.equal(hub.status().virtualStreams, 3)
329
+
330
+ const closes = sockets.map((socket) => { socket.end(); return onceEvent(socket, "close") })
331
+ await Promise.all(closes.map((promise) => withTimeout(promise, "multi-stream cleanup close")))
332
+ await waitUntil(() => hub.status().virtualStreams === 0 && client.status().virtualStreams === 0 && gateway.status().virtualStreams === 0 && targetSockets.size === 0, "multi-stream ownership cleanup")
333
+ })
@@ -337,6 +337,31 @@ test("loadEnvironmentConfig parses outbound reverse gateway verified TLS setting
337
337
  assert.deepEqual(config.reverseHubTls, {caFile: "/certs/ca.crt", certFile: "/certs/client.crt", keyFile: "/certs/client.key", servername: "server3.example.test"})
338
338
  })
339
339
 
340
+ test("loadEnvironmentConfig parses the reverse-gateway spool directory", () => {
341
+ const config = loadEnvironmentConfig({
342
+ SOCKETDUCT_MODE: "reverse-gateway",
343
+ SOCKETDUCT_REVERSE_GATEWAY_ID: "starlink",
344
+ SOCKETDUCT_REVERSE_CONNECTION_ID: "home_starlink_01",
345
+ SOCKETDUCT_REVERSE_HUB_HOST: "server3.example.test",
346
+ SOCKETDUCT_REVERSE_TOKEN: "gateway-token",
347
+ SOCKETDUCT_TARGET_ALLOWLIST: "docker-server:2375",
348
+ SOCKETDUCT_SPOOL_DIRECTORY: " /spool/reverse-gateway "
349
+ })
350
+ assert.equal(config.spoolDirectory, "/spool/reverse-gateway")
351
+ assert.equal("spoolDirectory" in loadEnvironmentConfig({
352
+ SOCKETDUCT_MODE: "reverse-gateway",
353
+ SOCKETDUCT_REVERSE_GATEWAY_ID: "starlink",
354
+ SOCKETDUCT_REVERSE_CONNECTION_ID: "home_starlink_01",
355
+ SOCKETDUCT_REVERSE_HUB_HOST: "server3.example.test",
356
+ SOCKETDUCT_REVERSE_TOKEN: "gateway-token",
357
+ SOCKETDUCT_TARGET_ALLOWLIST: "docker-server:2375",
358
+ SOCKETDUCT_SPOOL_DIRECTORY: " "
359
+ }), false)
360
+ assert.throws(() => loadEnvironmentConfig({SOCKETDUCT_MODE: "relay", SOCKETDUCT_SPOOL_DIRECTORY: "/spool"}), /reverse-gateway-only/)
361
+ assert.throws(() => loadEnvironmentConfig({SOCKETDUCT_MODE: "reverse-hub", SOCKETDUCT_SPOOL_DIRECTORY: "/spool"}), /reverse-gateway-only/)
362
+ assert.throws(() => loadEnvironmentConfig({SOCKETDUCT_SPOOL_DIRECTORY: "/spool"}), /reverse-gateway-only/)
363
+ })
364
+
340
365
  test("loadEnvironmentConfig rejects malformed reverse mode policy and credentials", () => {
341
366
  assert.throws(() => loadEnvironmentConfig({SOCKETDUCT_MODE: "unknown"}), /Unsupported SOCKETDUCT_MODE/)
342
367
  assert.throws(() => loadEnvironmentConfig({SOCKETDUCT_MODE: "reverse-hub", SOCKETDUCT_REVERSE_GATEWAY_BINDINGS: "missing-user"}), /gatewayId:username/)
@@ -61,11 +61,13 @@ test("compressed logger durably acknowledges ordered records and closes asynchro
61
61
  test("compressed logger enforces one shared bounded queue with a critical reserve", async (t) => {
62
62
  const root = await mkdtemp(join(tmpdir(), "socketduct-compressed-pressure-"))
63
63
  const directory = join(root, "logs")
64
+ // The reserve must cover the critical event's conservative charge (line bytes × 3 + 128);
65
+ // an accepted-but-undelivered sequence can never be evicted without tearing the durable journal.
64
66
  const logger = createCompressedJsonlFileLogger({
65
67
  directory,
66
68
  role: "reverse-client",
67
69
  maxQueueBytes: 4096,
68
- criticalReserveBytes: 1024,
70
+ criticalReserveBytes: 2048,
69
71
  frameMaxBytes: 1024,
70
72
  flushIntervalMs: 60_000,
71
73
  criticalFlushIntervalMs: 60_000
@@ -0,0 +1,54 @@
1
+ import assert from "node:assert/strict"
2
+ import test from "node:test"
3
+
4
+ import {insertBoundedTerminalOutcome, ownerMayMissTerminalDelivery} from "../../src/reverse/terminal-outcomes.js"
5
+
6
+ test("a healthy attached physical does not retain a terminal outcome; absent, closing, and pending-resume states do", () => {
7
+ // Healthy attached: the peer receives the terminal frame and retires the
8
+ // stream locally, so it never re-presents it. Retaining here is the
9
+ // unbounded-growth bug: it must be a no-eligibility (no retention).
10
+ assert.equal(ownerMayMissTerminalDelivery({closing: false, resumeRequired: false}), false)
11
+ // Detached (no physical): the terminal frame cannot be delivered.
12
+ assert.equal(ownerMayMissTerminalDelivery(undefined), true)
13
+ // Closing: the frame may not be flushed.
14
+ assert.equal(ownerMayMissTerminalDelivery({closing: true, resumeRequired: false}), true)
15
+ // Paired-resume / pending-manifest state: the frame is answered through the
16
+ // RESUME manifest instead of a live relay.
17
+ assert.equal(ownerMayMissTerminalDelivery({closing: false, resumeRequired: true}), true)
18
+ })
19
+
20
+ test("retention is capped at the owner stream bound, evicting the oldest first in deterministic insertion order", () => {
21
+ const outcomes = new Map()
22
+ const limit = 3
23
+ const outcome = (/** @type {string} */ code) => ({code, attribution: undefined})
24
+
25
+ // Within the bound: every outcome is retained, in insertion order.
26
+ insertBoundedTerminalOutcome(outcomes, "s1\0a", outcome("RESUME_STREAM_OMITTED"), limit)
27
+ insertBoundedTerminalOutcome(outcomes, "s2\0b", outcome("BUFFER_LIMIT"), limit)
28
+ assert.equal(outcomes.size, 2)
29
+ assert.deepEqual([...outcomes.keys()], ["s1\0a", "s2\0b"])
30
+
31
+ // Exactly at the bound: no eviction yet.
32
+ insertBoundedTerminalOutcome(outcomes, "s3\0c", outcome("TARGET_CLOSED"), limit)
33
+ assert.equal(outcomes.size, 3)
34
+ assert.equal(outcomes.get("s1\0a")?.code, "RESUME_STREAM_OMITTED")
35
+
36
+ // One past the bound: the oldest (s1) is evicted, the newest (s3) is kept.
37
+ insertBoundedTerminalOutcome(outcomes, "s4\0d", outcome("RESUME_SESSION_LOST"), limit)
38
+ assert.equal(outcomes.size, 3)
39
+ assert.equal(outcomes.has("s1\0a"), false, "the oldest outcome must be evicted (fail-closed to RESUME_SESSION_LOST)")
40
+ assert.deepEqual([...outcomes.keys()], ["s2\0b", "s3\0c", "s4\0d"])
41
+ assert.equal(outcomes.get("s4\0d")?.code, "RESUME_SESSION_LOST", "the newest causal code must be retained")
42
+
43
+ // Further churn keeps the bound and evicts the next-oldest, deterministically.
44
+ insertBoundedTerminalOutcome(outcomes, "s5\0e", outcome("GATEWAY_MAINTENANCE"), limit)
45
+ assert.equal(outcomes.size, 3)
46
+ assert.deepEqual([...outcomes.keys()], ["s3\0c", "s4\0d", "s5\0e"])
47
+ assert.equal(outcomes.has("s2\0b"), false)
48
+
49
+ // Re-inserting a retained key updates it in place without disturbing the bound.
50
+ insertBoundedTerminalOutcome(outcomes, "s5\0e", outcome("BUFFER_LIMIT"), limit)
51
+ assert.equal(outcomes.size, 3)
52
+ assert.equal(outcomes.get("s5\0e")?.code, "BUFFER_LIMIT")
53
+ assert.deepEqual([...outcomes.keys()], ["s3\0c", "s4\0d", "s5\0e"])
54
+ })