socketduct 0.1.17 → 0.1.18

package/changelog.d/20260925120000-reverse-stream-transport-resume.md

@@ -0,0 +1 @@
+Native reverse-gateway streams now keep their app and target sockets alive across transient hub transport loss, preserve unread flow-control and terminal-close state, and replay durably spooled bytes exactly once after paired resume.

package/docs/architecture.md

@@ -106,14 +106,21 @@ resolved from the requested target for each `OPEN`. The hub maps frames without
 opening the target or nesting another Socketduct transport. Each home gateway
 applies the final local allowlist before connecting to its private Docker shim.
 
-Gateway registration reconnects with generation fencing. Established reverse
-streams are terminal on transport loss because replaying an ambiguous Docker
-operation would be unsafe. Live-transport buffers remain bounded; durable
-spooling and resume continue to apply to ordinary Socketduct sessions only.
+Gateway and client registrations reconnect with generation fencing. The hub
+retains each established bridge for a bounded window while the live gateway
+keeps its original target socket and both endpoints durably retain
+unacknowledged send chunks. Replacement physical generations exchange paired
+resume manifests; the hub reconciles receiver-committed sequence positions
+before either side replays. This resumes the same byte stream without reopening
+the target or issuing a second application request. Reverse process replacement
+remains terminal because the original target/app socket objects cannot move to
+a new process.
+
 The hub serializes output independently for each physical destination with a
 bounded queue, so one backpressured target gateway cannot stop client frames
 bound for a healthy sibling gateway. Exceeding that destination bound closes
-only the unhealthy physical owner and releases its mapped streams.
+only the unhealthy physical owner and fails its mapped streams with the causal
+limit error.
 See [Native reverse gateway](reverse-gateway.md).
 
 ### Edge

package/docs/protocol.md

@@ -43,13 +43,17 @@ emits it and ordinary handlers reject it as unnegotiated.
   before `AUTH_OK`.
 - `PING` / `PONG`: connection heartbeat and readiness.
 - `RESUME` / `RESUMED`: bounded pages of stream manifests. Each entry contains
-  stream ID, stable identifier and both directional acknowledgement positions.
+  stream ID, stable identifier and both directional sequence/acknowledgement
+  positions. Ordinary v3 uses one client manifest; native reverse resume pairs
+  independently authenticated client and gateway manifests at the hub.
 - `GOAWAY`: terminal physical-connection shutdown. Authentication,
   authorization and TLS failures are terminal and are not retried forever.
 - `GATEWAY_REGISTER` / `GATEWAY_REGISTERED`: reverse-only authenticated stable
-  gateway registration and generation fencing.
+  gateway registration and generation fencing. The response declares whether
+  retained logical streams require a resume manifest.
 - `GATEWAY_SELECT` / `GATEWAY_SELECTED`: reverse-only binding of a client
-  physical member to one currently registered gateway before readiness.
+  physical member to one exact gateway or route class before readiness. The
+  response likewise declares retained resume state.
 
 ### Stream control (nonzero stream ID)
 
@@ -72,11 +76,13 @@ bounded per stream and in aggregate per connection. Control frames have
 priority; DATA frames use round-robin scheduling with bounded frame sizes, so a
 large upload cannot monopolize the physical connection.
 
-Flow-control credit and scheduler position belong to one physical generation.
-After both peers reconcile cumulative durable acknowledgements in RESUME, each
-surviving stream begins the replacement generation with a fresh negotiated
-window and replay is rebuilt strictly above the reconciled ACK. Credit consumed
-by a failed transport is never carried forward.
+Scheduler position belongs to one physical generation, while receive-side
+flow-control debt belongs to the logical stream. After both peers reconcile in
+RESUME, a sender retains its negotiated window minus bytes the receiver already
+accepted but has not application-acknowledged. Those bytes are not replayed,
+but remain spooled until the durable ACK advances; replay starts strictly above
+the receiver's high-water sequence. A failed transport therefore cannot turn
+unread `VirtualSocket` bytes back into fresh credit.
 
 Relay retention is bounded independently per stream, per connection identity,
 and process-wide. The process-wide reservation is owned by the relay runtime
@@ -113,7 +119,31 @@ then authorizes the exact principal/gateway/target tuple and the gateway
 separately authorizes its local target. Zero, multiple, stale, disconnected, or
 mismatched resolutions reset the stream without weakening sibling routes.
 Registration reconnect uses a higher generation, but any established stream on
-the lost gateway is reset and never replayed.
+the lost gateway remains bound to its original client, gateway process, and
+target socket during the bounded resume window. The hub also replaces the
+corresponding client physical transport so neither endpoint can advance without
+paired reconciliation.
+
+On replacement, both reverse peers send paginated `RESUME` entries containing
+the stable identifier, each direction's allocated sequence and durable ACK,
+and directional END state. The hub verifies authenticated owner, stable stream
+ID/identifier, monotonic bounds, and cross-peer positions. It preserves both
+the receiver high-water sequence and its application-acknowledged boundary in
+each per-stream `RESUMED` result, and only then admits replay. Senders replay
+durable raw chunks strictly above the high-water sequence while retaining
+unacknowledged chunks and charging their outstanding bytes to the new window.
+A DATA frame that entered a failed physical socket but did not reach the
+logical receiver is therefore replayed; a byte already buffered by the receiver
+is not delivered twice. The target socket is never reopened and the application
+request is never recreated as a new stream.
+
+Reverse send replay is persisted before transmission on both the Server3 client
+and outbound gateway. ACKs truncate those private temporary spools. Missing
+targets, expired retention, identity or manifest mismatch, and configured
+buffer/spool/output bounds produce one terminal stream error with the causal
+code. Process-replacement recovery remains unsupported for reverse streams
+because a replacement gateway process cannot inherit the original target TCP
+socket.
 
 Hub output has one ordered owner per physical destination. Frames for a
 backpressured gateway remain in that gateway's bounded queue while frames for
@@ -124,8 +154,10 @@ destinations. Ordering within each destination and stream is preserved.
 Stream IDs must increase monotonically on each physical connection. END and
 CLOSE positions must match the sender's accepted raw-byte sequence; a
 gateway-originated CLOSE is retained until the client acknowledges it with
-CLOSE. RESET is immediately terminal. Repeats, data after END, and stale ACKs
-are protocol violations.
+CLOSE, including across physical resume. A client that echoes CLOSE immediately
+after resume uses the receiver-confirmed client-to-gateway position rather than
+a stale pre-resume transmit position. RESET is immediately terminal. Repeats,
+data after END, and stale ACKs are protocol violations.
 
 Reverse peers must negotiate `zstd` before selection/registration completes.
 A compressed DATA payload is `uint32_be(uncompressedLength)` followed by one

package/docs/reverse-gateway.md

@@ -65,8 +65,9 @@ The hub rejects unknown IDs, registration by the wrong principal, a second
 connection identity claiming the same ID, stale generations, repeated IDs,
 malformed frames, unauthorized routes, and configured limits. A higher
 generation from the same stable connection identity fences its prior transport.
-At the configured identity bound, the oldest disconnected client identity is
-evicted; active owners are never evicted and retain generation fencing.
+At the configured identity bound, the oldest disconnected, streamless client
+identity is evicted; retained-stream and active owners are never evicted and
+retain generation fencing.
 Zero eligible gateways denies the target, and multiple eligible gateways deny
 the stream as ambiguous. Unknown classes, class mismatches, stale registrations,
 and disconnected resolved gateways also fail closed. The home gateway
@@ -82,17 +83,30 @@ zstd runtime support and always uses TLS with certificate verification enabled;
 there is no plaintext or `rejectUnauthorized:false` reverse option.
 
 Each peer advertises its receive window during authentication and stream open;
-senders use the smaller local and peer window for stream credit. A graceful hub
-shutdown is transient and reconnects the client and gateway registrations, while
-authentication, authorization, and stale-generation rejections remain terminal.
+senders use the smaller local and peer window for stream credit. A hub-directed
+physical replacement is transient and reconnects the client and gateway
+registrations, while loss of the hub's retained state, authentication,
+authorization, and stale-generation rejections remain terminal.
+
+After an established stream loses either reverse physical transport, the hub
+retains its authenticated bridge for the configured resume window and fences the
+old generation. It makes both logical endpoints reconnect so each can present a
+paginated `RESUME` manifest containing the stable stream identifier, both
+directional sequence/ACK positions, and half-close state. The hub releases the
+replacement generation only after both manifests agree. Each sender then
+replays only durable chunks above the receiver-confirmed high-water position,
+with its flow-control window reduced by bytes already buffered but not yet
+consumed at the receiver. A registered gateway completing this paired resume is
+not healthy or eligible for a new selection or `OPEN`; retained peers alone may
+attach to submit their resume manifests.
 
 The hub owns a separate ordered output queue for every client and gateway
 physical connection. Each queue defaults to a 32 MiB bound and can be set with
 the programmatic `maxOutputBytesPerConnection` hub option. Backpressure on one
 gateway therefore cannot pause client input for a healthy gateway in the same
 route class. If one destination exceeds its bound, the hub closes that physical
-connection and its mapped streams through the normal detach path; sibling
-gateways and the route-class client connection remain usable.
+connection and fails its mapped streams with `OUTPUT_BACKPRESSURE_LIMIT`;
+sibling gateways and the route-class client connection remain usable.
 
 An application write may be larger than the configured reverse-stream buffer.
 The client admits it in bounded slices no larger than the negotiated peer window
@@ -110,21 +124,35 @@ in-memory gateway registry across independent cluster workers.
 ## Failure and replay boundary
 
 Gateway registration reconnects with bounded exponential delay and a higher
-generation. An established Docker stream does **not** survive that transport
-loss: the hub resets it with `GATEWAY_DISCONNECTED`, both endpoints release the
-stream, and the gateway destroys its local Docker socket. A route-class client
-physical connection stays authenticated, so streams through other gateways in
-the class and later opens to those targets remain usable. This is intentional.
-A Docker request whose write reached dockerd but whose response did not reach
-TensorBuzz has an ambiguous outcome and must never be replayed automatically.
-TensorBuzz must reconcile that Docker operation and then create a new stream.
-
-Consequently reverse streams have bounded in-memory flow-control/replay queues
-only for the live transport and report zero spool bytes. They do not persist an
-operation spool across disconnect or process replacement; `recoverConnection()`
-fails explicitly. Ordinary resumable Socketduct sessions retain their existing
-durable spool behavior. This split is the safety boundary between reconnecting
-the gateway registration and replaying privileged Docker operations.
+generation. An established Docker stream survives a transient physical loss
+when the hub process and its retained state, live client object, outbound gateway process, and
+original target socket remain available through the configured resume window.
+Socketduct keeps
+the app-facing socket open, durably spools each direction's unacknowledged raw
+chunks before sending them, reconciles the two receiver-committed ACK positions,
+and separately preserves each receiver's accepted high-water position. It
+replays only bytes above the high-water position, retains buffered-but-unread
+bytes against the flow-control window until ACK, and keeps terminal CLOSE state
+until the echoed CLOSE arrives. It never creates a replacement logical stream
+or reconnects dockerd, so an application request is not executed again merely
+because the WAN transport changed.
+
+This is transparent transport replay, not application-operation retry. Bytes
+already committed to dockerd are acknowledged by the surviving gateway and are
+not replayed; bytes already accepted by the surviving app-facing socket are
+likewise not delivered twice. Sequence gaps, overlaps, identity/ownership
+mismatches, invalid manifests, spool/buffer limits, and authorization failures
+fail closed.
+
+Reverse replay state is bounded per stream and connection and stored in private
+temporary spool directories. ACKed chunks are truncated, and terminal
+close/reset removes the spool. `status().spoolBytes` reports retained durable
+payload. `registerRecoveredConnection()` remains unsupported for reverse mode:
+a process replacement cannot inherit the live target socket or the prior
+`VirtualSocket`. If either endpoint process disappears, the target socket is
+lost, or the hub resume window expires, the logical stream fails once with the
+causal session-loss/reset code and the application must reconcile at its own
+protocol layer.
 
 Target bytes already observed by the gateway are ordered before a subsequent
 target-error reset. Stream `END` remains a directional half-close, while protocol
@@ -207,11 +235,18 @@ await starting
 Route-class sockets expose immutable `routeClass` allocation metadata and an
 undefined `gatewayId`. Exact-gateway sets retain immutable `gatewayId` metadata.
 Callers cannot override either selection per open. Reverse streams remain
-non-recoverable after an uncertain transport loss.
+sticky to their selected gateway/target but resume transparently after a
+transient physical transport replacement.
 
 The socket is returned immediately and waits inside Socketduct until the set
 reaches minimum readiness. New sockets rotate equally across ready members and
 stay on their assigned member. If all members are reconnecting, new sockets wait
 FIFO without selecting a connecting gateway; a recovered member resumes those
 opens. A disconnected member leaves readiness and rejoins after authenticated
-registration; existing streams on it fail for application-level reconciliation.
+registration and paired stream resume; existing sockets remain open unless the
+target/session is genuinely lost or a configured resume/buffer bound expires.
+
+The paired reverse-resume fields are a protocol compatibility boundary. Upgrade
+the hub, outbound gateways, and Server3/TensorBuzz Socketduct consumer together;
+0.1.17 and older reverse peers do not exchange these manifests and cannot
+provide transparent established-stream recovery.

package/docs/security.md

@@ -166,10 +166,18 @@ stops reading cannot grow an unbounded queue or stall authorization, DATA, or
 OPEN processing for a healthy sibling; exceeding the bound disconnects only
 that destination and resets its owned streams.
 
-Reverse registration can reconnect; established Docker streams cannot resume
-or spool across loss. They fail with an explicit uncertain-operation error so
-the caller must reconcile before retrying. This prevents privileged Docker API
-writes from being applied twice.
+Native reverse peers retain established logical streams for a bounded resume
+window when only a hub transport is lost. Both authenticated endpoints must
+reconnect with the same stable identities and newer generations, then present
+matching per-stream identifiers, byte positions, acknowledgements, and
+half-close state. Application bytes are durably spooled before transmission;
+the hub reconciles to each receiver's committed position and permits replay
+only above it. This preserves exact-once delivery without reopening or
+reissuing a privileged Docker request. Ownership mismatches, missing targets,
+invalid or incomplete manifests, resource-bound failures, and expired resume
+windows fail the affected logical stream once instead of falling back to a new
+stream. Process replacement remains terminal because the live app and target
+sockets cannot be inherited safely.
 
 ## Network exposure
 
@@ -268,4 +276,6 @@ To roll back TLS: remove the TLS env vars from `.env`. If the relay binds to a n
 - reverse registration rejects wrong principals, duplicate IDs, and stale generations
 - reverse clients reject unknown/unavailable gateway IDs and wrong TLS CAs
 - both hub route policy and the home-local allowlist deny unauthorized Docker targets
-- gateway loss cleans every stream and target without replay, then a fresh stream works after re-registration
+- transient reverse hub transport loss preserves each live stream and target,
+  replays only unacknowledged spooled bytes, and cleans all retained state after
+  close; target loss and resume expiry fail once with their causal codes

package/docs/session-lifecycle.md

@@ -11,22 +11,33 @@ spawn per-stream reconnects.
 ## Reverse-gateway lifecycle
 
 An outbound gateway moves through `connecting -> authenticated -> registering
--> ready`. Transport loss destroys every local target stream, advances the
-gateway generation, and reconnects with bounded exponential delay. Selected
-Server3 client members leave readiness when that gateway disappears and rejoin
-only after it registers again.
-
-Reverse stream loss is terminal even when registration later recovers. No
-established Docker bytes are persisted or replayed because their outcome may
-already be committed at dockerd. The application receives
-`GATEWAY_DISCONNECTED`, reconciles the operation, and opens a new stream.
+-> ready`. Transport loss advances the gateway generation and reconnects with
+bounded exponential delay without destroying established local targets. The hub
+retains authenticated bridge ownership for the configured resume window and
+replaces the affected Server3 client physical generation as well.
+
+Both reverse endpoints keep their logical sockets alive, persist unacknowledged
+send chunks, and submit their stable identifiers plus both directional
+sequence/ACK positions. The hub resumes a bridge only after the two manifests
+agree and returns both the receiver's accepted high-water sequence and its
+application-consumed ACK to each sender. Bytes above the high-water position
+replay on the same logical stream; accepted-but-unread bytes stay charged to the
+window and spooled until ACK. This preserves exact-once, in-order delivery
+without opening a second target or retrying an application operation.
+
+If the original target socket disappears, an owner/identifier or sequence does
+not match, a buffer/spool bound is exceeded, or the resume window expires, that
+logical stream fails once with the causal error and releases both sides. Reverse
+process-replacement recovery remains unsupported because the live target and
+app-facing socket cannot be transferred to replacement JavaScript objects.
 
 The bounded, paginated resume manifest carries stable identifiers and both
 directional acknowledgement positions. The relay returns an independent result
 per stream. A stream omitted because it was closed while disconnected is
 cleaned up at the relay. A target lost during interruption returns a stream-local
 resume reset and a typed session-loss error, while retained siblings continue.
-Only bytes above the peer acknowledgement are replayed.
+Only bytes above the peer's received high-water sequence are replayed; spool
+cleanup remains governed by its acknowledgement.
 
 ## States
 

package/docs/socket-api.md

@@ -190,8 +190,9 @@ becomes the stable, immutable configured ID. For a route-class set,
 the socket first removes the pending open, so later readiness cannot leak a
 member stream. An exact relay member selected immediately before its path drops
 retains the stream and performs member-local v3 recovery; the set never
-speculatively migrates it. A reverse route-class stream fails terminally on an
-uncertain transport loss and is never replayed.
+speculatively migrates it. A reverse route-class stream likewise remains on its
+resolved target gateway while Socketduct replaces and resumes the physical
+client/gateway transports underneath it.
 
 `createConnection(options, callback)` is pre-bound and invokes its callback once
 on `connect` or the terminal pre-connect error; the socket emits `connect` before
@@ -220,7 +221,9 @@ stickiness. Every returned raw or TLS socket exposes enumerable `gatewayId` and
 `routeClass` allocation metadata. Only the configured identity is set. Persist
 an exact set's immutable `gatewayId` after connection so a replacement can
 recover the stream on its original member. Reverse route-class streams are not
-recoverable after an uncertain transport loss.
+recoverable across process replacement, but they do recover transparently from
+a transient physical transport loss while the original objects and target
+socket remain alive.
 
 Process-replacement recovery is deliberately separate from ordinary selection.
 First detach the old set when an orderly handoff is possible. Construct the
@@ -399,8 +402,12 @@ It selects exactly one registered gateway during `connect()`. The existing
 `MultiplexedSocketductTransportSet` accepts `endpoint.reverseHub`, allowing
 three ready members to retain deterministic health-aware round-robin behavior.
 Reverse established operations deliberately do not support
-`registerRecoveredConnection()` or durable replay; a transport loss is a typed
-terminal result requiring application reconciliation.
+`registerRecoveredConnection()` across process replacement. They do durably
+spool unacknowledged send chunks during the lifetime of the client and gateway
+objects, then use paired v3 resume manifests to survive physical transport
+replacement without reopening the target or replaying an application request
+as a new logical stream. Target loss, resume expiry, identity mismatch, or a
+configured bound remains terminal and requires application reconciliation.
 
 Reverse virtual sockets accept individual writes larger than their configured
 retained-byte limit. The connection progresses those writes in slices bounded by

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "socketduct",
-  "version": "0.1.17",
+  "version": "0.1.18",
   "description": "Resilient authenticated tunnel/proxy for keeping socket streams alive across flaky links.",
   "type": "module",
   "main": "./src/index.js",

package/src/client/multiplexed-transport-set.js

@@ -270,6 +270,7 @@ export class MultiplexedSocketductTransportSet extends EventEmitter {
           tls: endpoint.tls,
           ...(endpoint.createPhysicalSocket === undefined ? {} : {createPhysicalSocket: endpoint.createPhysicalSocket}),
           connectionId: identity.connectionId,
+          spoolDirectory: join(validated.spoolDirectory, identity.spoolName),
           continuousReconnect: true,
           aggregateBudget: this.#budget,
           ...(this.#logger === undefined ? {} : {logger: this.#logger})