socketduct 0.1.17 → 0.1.19
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -2
- package/changelog.d/20260925120000-reverse-stream-transport-resume.md +1 -0
- package/changelog.d/20260925143000-reverse-client-process-recovery.md +1 -0
- package/docs/architecture.md +20 -11
- package/docs/protocol.md +56 -11
- package/docs/reverse-gateway.md +91 -24
- package/docs/running.md +2 -2
- package/docs/security.md +22 -5
- package/docs/session-lifecycle.md +32 -10
- package/docs/socket-api.md +57 -16
- package/package.json +1 -1
- package/src/client/multiplexed-transport-set.js +13 -5
- package/src/client/virtual-socket.js +42 -0
- package/src/reverse/client.js +524 -75
- package/src/reverse/gateway.js +265 -38
- package/src/reverse/hub.js +305 -24
- package/src/reverse/runtime.js +1 -0
- package/src/spool/spool-file.js +136 -7
- package/tensorbuzz.yml +2 -1
- package/test/e2e/reverse-client-recovery-process.test.js +170 -0
- package/test/fixtures/reverse-client-recovery.mjs +123 -0
- package/test/integration/reverse-gateway.test.js +996 -28
- package/test/integration/reverse-route-class.test.js +16 -12
- package/test/unit/client/multiplexed-transport-set.test.js +12 -1
- package/test/unit/spool/spool-file.test.js +133 -0
package/README.md
CHANGED
|
@@ -91,8 +91,8 @@ and automatically rejoin selection. A socket created before readiness is
|
|
|
91
91
|
returned immediately and waits inside Socketduct with normal callback, event,
|
|
92
92
|
timeout and backpressure behavior; assignment stays FIFO and occurs only on a
|
|
93
93
|
ready member. A virtual socket never changes members.
|
|
94
|
-
Durable process-replacement recovery uses `recoverConnection()` with
|
|
95
|
-
original gateway ID before the replacement set starts.
|
|
94
|
+
Durable process-replacement recovery uses `recoverConnection()` with exactly
|
|
95
|
+
the original gateway ID or reverse route class before the replacement set starts.
|
|
96
96
|
|
|
97
97
|
For home Docker hosts behind O2, Zyxel, and Starlink, run
|
|
98
98
|
`OutboundReverseGateway` in each router-pinned namespace and
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
Native reverse-gateway streams now keep their app and target sockets alive across transient hub transport loss, preserve unread flow-control and terminal-close state, and replay durably spooled bytes exactly once after paired resume.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
Reverse clients can now recover crash-consistent persistent streams in a replacement process with application-consumption ACKs and a quiescent detach handoff, without reopening the surviving gateway target, while unsupported hub, gateway, and target loss fails closed.
|
package/docs/architecture.md
CHANGED
|
@@ -85,12 +85,12 @@ gateway independently for each target `OPEN`, so adding target gateways does not
|
|
|
85
85
|
multiply client physical connections. Allocation metadata is `routeClass`;
|
|
86
86
|
legacy exact-gateway sets keep `gatewayId` and their existing identity derivation.
|
|
87
87
|
|
|
88
|
-
Replacement recovery is registered before set startup and addresses
|
|
89
|
-
original configured
|
|
90
|
-
identity/spool namespace and delegates recovery to that member
|
|
91
|
-
physical attempt; round-robin is never involved. An orderly set
|
|
92
|
-
active spools while retiring empty warmed lanes. Ordinary new
|
|
93
|
-
a
|
|
88
|
+
Replacement recovery is registered before set startup and addresses exactly one
|
|
89
|
+
original configured selector: `gatewayId` or `routeClass`. The set derives the
|
|
90
|
+
same member identity/spool namespace and delegates recovery to that member
|
|
91
|
+
before any physical attempt; round-robin is never involved. An orderly set
|
|
92
|
+
detach preserves active spools while retiring empty warmed lanes. Ordinary new
|
|
93
|
+
opens cannot name a member.
|
|
94
94
|
|
|
95
95
|
Protocol v2 retains the original one-session-per-tunnel architecture for
|
|
96
96
|
rollback. A relay listener selects v2 or v3 explicitly.
|
|
@@ -106,14 +106,23 @@ resolved from the requested target for each `OPEN`. The hub maps frames without
|
|
|
106
106
|
opening the target or nesting another Socketduct transport. Each home gateway
|
|
107
107
|
applies the final local allowlist before connecting to its private Docker shim.
|
|
108
108
|
|
|
109
|
-
Gateway
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
109
|
+
Gateway and client registrations reconnect with generation fencing. The hub
|
|
110
|
+
retains each established bridge for a bounded window while the live gateway
|
|
111
|
+
keeps its original target socket and both endpoints durably retain
|
|
112
|
+
unacknowledged send chunks. Replacement physical generations exchange paired
|
|
113
|
+
resume manifests; the hub reconciles receiver-committed sequence positions
|
|
114
|
+
before either side replays. This resumes the same byte stream without reopening
|
|
115
|
+
the target or issuing a second application request. The Server3 reverse-client
|
|
116
|
+
object/process may be replaced from persistent spool state because the home
|
|
117
|
+
gateway still owns the target socket; the replacement creates a new app-facing
|
|
118
|
+
`VirtualSocket` and resumes from its durable receive ACK. The hub process,
|
|
119
|
+
gateway process, and target socket are not reconstructable by this mechanism.
|
|
120
|
+
|
|
113
121
|
The hub serializes output independently for each physical destination with a
|
|
114
122
|
bounded queue, so one backpressured target gateway cannot stop client frames
|
|
115
123
|
bound for a healthy sibling gateway. Exceeding that destination bound closes
|
|
116
|
-
only the unhealthy physical owner and
|
|
124
|
+
only the unhealthy physical owner and fails its mapped streams with the causal
|
|
125
|
+
limit error.
|
|
117
126
|
See [Native reverse gateway](reverse-gateway.md).
|
|
118
127
|
|
|
119
128
|
### Edge
|
package/docs/protocol.md
CHANGED
|
@@ -43,13 +43,17 @@ emits it and ordinary handlers reject it as unnegotiated.
|
|
|
43
43
|
before `AUTH_OK`.
|
|
44
44
|
- `PING` / `PONG`: connection heartbeat and readiness.
|
|
45
45
|
- `RESUME` / `RESUMED`: bounded pages of stream manifests. Each entry contains
|
|
46
|
-
stream ID, stable identifier and both directional acknowledgement
|
|
46
|
+
stream ID, stable identifier and both directional sequence/acknowledgement
|
|
47
|
+
positions. Ordinary v3 uses one client manifest; native reverse resume pairs
|
|
48
|
+
independently authenticated client and gateway manifests at the hub.
|
|
47
49
|
- `GOAWAY`: terminal physical-connection shutdown. Authentication,
|
|
48
50
|
authorization and TLS failures are terminal and are not retried forever.
|
|
49
51
|
- `GATEWAY_REGISTER` / `GATEWAY_REGISTERED`: reverse-only authenticated stable
|
|
50
|
-
gateway registration and generation fencing.
|
|
52
|
+
gateway registration and generation fencing. The response declares whether
|
|
53
|
+
retained logical streams require a resume manifest.
|
|
51
54
|
- `GATEWAY_SELECT` / `GATEWAY_SELECTED`: reverse-only binding of a client
|
|
52
|
-
physical member to one
|
|
55
|
+
physical member to one exact gateway or route class before readiness. The
|
|
56
|
+
response likewise declares retained resume state.
|
|
53
57
|
|
|
54
58
|
### Stream control (nonzero stream ID)
|
|
55
59
|
|
|
@@ -72,11 +76,13 @@ bounded per stream and in aggregate per connection. Control frames have
|
|
|
72
76
|
priority; DATA frames use round-robin scheduling with bounded frame sizes, so a
|
|
73
77
|
large upload cannot monopolize the physical connection.
|
|
74
78
|
|
|
75
|
-
|
|
76
|
-
After both peers reconcile
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
79
|
+
Scheduler position belongs to one physical generation, while receive-side
|
|
80
|
+
flow-control debt belongs to the logical stream. After both peers reconcile in
|
|
81
|
+
RESUME, a sender retains its negotiated window minus bytes the receiver already
|
|
82
|
+
accepted but has not application-acknowledged. Those bytes are not replayed,
|
|
83
|
+
but remain spooled until the durable ACK advances; replay starts strictly above
|
|
84
|
+
the receiver's high-water sequence. A failed transport therefore cannot turn
|
|
85
|
+
unread `VirtualSocket` bytes back into fresh credit.
|
|
80
86
|
|
|
81
87
|
Relay retention is bounded independently per stream, per connection identity,
|
|
82
88
|
and process-wide. The process-wide reservation is owned by the relay runtime
|
|
@@ -113,7 +119,44 @@ then authorizes the exact principal/gateway/target tuple and the gateway
|
|
|
113
119
|
separately authorizes its local target. Zero, multiple, stale, disconnected, or
|
|
114
120
|
mismatched resolutions reset the stream without weakening sibling routes.
|
|
115
121
|
Registration reconnect uses a higher generation, but any established stream on
|
|
116
|
-
the lost gateway
|
|
122
|
+
the lost gateway remains bound to its original client, gateway process, and
|
|
123
|
+
target socket during the bounded resume window. The hub also replaces the
|
|
124
|
+
corresponding client physical transport so neither endpoint can advance without
|
|
125
|
+
paired reconciliation.
|
|
126
|
+
|
|
127
|
+
On replacement, both reverse peers send paginated `RESUME` entries containing
|
|
128
|
+
the stable identifier, each direction's allocated sequence and durable ACK,
|
|
129
|
+
and directional END state. The hub verifies authenticated owner, stable stream
|
|
130
|
+
ID/identifier, monotonic bounds, and cross-peer positions. It preserves both
|
|
131
|
+
the receiver high-water sequence and its application-acknowledged boundary in
|
|
132
|
+
each per-stream `RESUMED` result, and only then admits replay. Senders replay
|
|
133
|
+
durable raw chunks strictly above the high-water sequence while retaining
|
|
134
|
+
unacknowledged chunks and charging their outstanding bytes to the new window.
|
|
135
|
+
A DATA frame that entered a failed physical socket but did not reach the
|
|
136
|
+
logical receiver is therefore replayed; a byte already buffered by the receiver
|
|
137
|
+
is not delivered twice. The target socket is never reopened and the application
|
|
138
|
+
request is never recreated as a new stream.
|
|
139
|
+
|
|
140
|
+
Reverse send replay is persisted before transmission on both the Server3 client
|
|
141
|
+
and outbound gateway. ACKs truncate those private temporary spools. Missing
|
|
142
|
+
targets, expired retention, identity or manifest mismatch, and configured
|
|
143
|
+
buffer/spool/output bounds produce one terminal stream error with the causal
|
|
144
|
+
code. A replacement reverse client may restore its generation, selector,
|
|
145
|
+
lifecycle, directional sequences/ACKs, END state, and contiguous send replay
|
|
146
|
+
from persistent spool state before making a physical attempt. It presents the
|
|
147
|
+
durable receive ACK, not the dead object's readable-buffer high-water mark, so
|
|
148
|
+
the surviving gateway conservatively replays any target bytes that were not
|
|
149
|
+
durably acknowledged. A reverse client advances that ACK only after bytes are
|
|
150
|
+
returned to the application; merely buffering them below the readable
|
|
151
|
+
high-water mark is process-local and does not make them safe across replacement.
|
|
152
|
+
It never emits `OPEN`.
|
|
153
|
+
|
|
154
|
+
Only the reverse client is replaceable in this protocol boundary. The same hub
|
|
155
|
+
process and retained bridge, outbound gateway process, and original target TCP
|
|
156
|
+
socket must survive. Hub restart reconstruction, gateway process/socket-keeper
|
|
157
|
+
or FD transfer, target reconnection, and replay after target loss are not
|
|
158
|
+
defined. A fresh gateway that encounters retained bridges submits an empty
|
|
159
|
+
manifest; normal omitted-stream resets retire them without target creation.
|
|
117
160
|
|
|
118
161
|
Hub output has one ordered owner per physical destination. Frames for a
|
|
119
162
|
backpressured gateway remain in that gateway's bounded queue while frames for
|
|
@@ -124,8 +167,10 @@ destinations. Ordering within each destination and stream is preserved.
|
|
|
124
167
|
Stream IDs must increase monotonically on each physical connection. END and
|
|
125
168
|
CLOSE positions must match the sender's accepted raw-byte sequence; a
|
|
126
169
|
gateway-originated CLOSE is retained until the client acknowledges it with
|
|
127
|
-
CLOSE
|
|
128
|
-
|
|
170
|
+
CLOSE, including across physical resume. A client that echoes CLOSE immediately
|
|
171
|
+
after resume uses the receiver-confirmed client-to-gateway position rather than
|
|
172
|
+
a stale pre-resume transmit position. RESET is immediately terminal. Repeats,
|
|
173
|
+
data after END, and stale ACKs are protocol violations.
|
|
129
174
|
|
|
130
175
|
Reverse peers must negotiate `zstd` before selection/registration completes.
|
|
131
176
|
A compressed DATA payload is `uint32_be(uncompressedLength)` followed by one
|
package/docs/reverse-gateway.md
CHANGED
|
@@ -65,8 +65,9 @@ The hub rejects unknown IDs, registration by the wrong principal, a second
|
|
|
65
65
|
connection identity claiming the same ID, stale generations, repeated IDs,
|
|
66
66
|
malformed frames, unauthorized routes, and configured limits. A higher
|
|
67
67
|
generation from the same stable connection identity fences its prior transport.
|
|
68
|
-
At the configured identity bound, the oldest disconnected client
|
|
69
|
-
evicted; active owners are never evicted and
|
|
68
|
+
At the configured identity bound, the oldest disconnected, streamless client
|
|
69
|
+
identity is evicted; retained-stream and active owners are never evicted and
|
|
70
|
+
retain generation fencing.
|
|
70
71
|
Zero eligible gateways denies the target, and multiple eligible gateways deny
|
|
71
72
|
the stream as ambiguous. Unknown classes, class mismatches, stale registrations,
|
|
72
73
|
and disconnected resolved gateways also fail closed. The home gateway
|
|
@@ -82,17 +83,30 @@ zstd runtime support and always uses TLS with certificate verification enabled;
|
|
|
82
83
|
there is no plaintext or `rejectUnauthorized:false` reverse option.
|
|
83
84
|
|
|
84
85
|
Each peer advertises its receive window during authentication and stream open;
|
|
85
|
-
senders use the smaller local and peer window for stream credit. A
|
|
86
|
-
|
|
87
|
-
|
|
86
|
+
senders use the smaller local and peer window for stream credit. A hub-directed
|
|
87
|
+
physical replacement is transient and reconnects the client and gateway
|
|
88
|
+
registrations, while loss of the hub's retained state, authentication,
|
|
89
|
+
authorization, and stale-generation rejections remain terminal.
|
|
90
|
+
|
|
91
|
+
After an established stream loses either reverse physical transport, the hub
|
|
92
|
+
retains its authenticated bridge for the configured resume window and fences the
|
|
93
|
+
old generation. It makes both logical endpoints reconnect so each can present a
|
|
94
|
+
paginated `RESUME` manifest containing the stable stream identifier, both
|
|
95
|
+
directional sequence/ACK positions, and half-close state. The hub releases the
|
|
96
|
+
replacement generation only after both manifests agree. Each sender then
|
|
97
|
+
replays only durable chunks above the receiver-confirmed high-water position,
|
|
98
|
+
with its flow-control window reduced by bytes already buffered but not yet
|
|
99
|
+
consumed at the receiver. A registered gateway completing this paired resume is
|
|
100
|
+
not healthy or eligible for a new selection or `OPEN`; retained peers alone may
|
|
101
|
+
attach to submit their resume manifests.
|
|
88
102
|
|
|
89
103
|
The hub owns a separate ordered output queue for every client and gateway
|
|
90
104
|
physical connection. Each queue defaults to a 32 MiB bound and can be set with
|
|
91
105
|
the programmatic `maxOutputBytesPerConnection` hub option. Backpressure on one
|
|
92
106
|
gateway therefore cannot pause client input for a healthy gateway in the same
|
|
93
107
|
route class. If one destination exceeds its bound, the hub closes that physical
|
|
94
|
-
connection and its mapped streams
|
|
95
|
-
gateways and the route-class client connection remain usable.
|
|
108
|
+
connection and fails its mapped streams with `OUTPUT_BACKPRESSURE_LIMIT`;
|
|
109
|
+
sibling gateways and the route-class client connection remain usable.
|
|
96
110
|
|
|
97
111
|
An application write may be larger than the configured reverse-stream buffer.
|
|
98
112
|
The client admits it in bounded slices no larger than the negotiated peer window
|
|
@@ -110,21 +124,62 @@ in-memory gateway registry across independent cluster workers.
|
|
|
110
124
|
## Failure and replay boundary
|
|
111
125
|
|
|
112
126
|
Gateway registration reconnects with bounded exponential delay and a higher
|
|
113
|
-
generation. An established Docker stream
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
127
|
+
generation. An established Docker stream survives a transient physical loss
|
|
128
|
+
when the hub process and its retained state, outbound gateway process, and
|
|
129
|
+
original target socket remain available through the configured resume window.
|
|
130
|
+
The original client object may reconnect in place, or a replacement reverse
|
|
131
|
+
client process may reconstruct a new `VirtualSocket` from durable spool state.
|
|
132
|
+
Socketduct keeps
|
|
133
|
+
the app-facing socket open, durably spools each direction's unacknowledged raw
|
|
134
|
+
chunks before sending them, reconciles the two receiver-committed ACK positions,
|
|
135
|
+
and separately preserves each receiver's accepted high-water position. It
|
|
136
|
+
replays only bytes above the high-water position, retains buffered-but-unread
|
|
137
|
+
bytes against the flow-control window until ACK, and keeps terminal CLOSE state
|
|
138
|
+
until the echoed CLOSE arrives. It never creates a replacement logical stream
|
|
139
|
+
or reconnects dockerd, so an application request is not executed again merely
|
|
140
|
+
because the WAN transport changed.
|
|
141
|
+
|
|
142
|
+
This is transparent transport replay, not application-operation retry. Bytes
|
|
143
|
+
already committed to dockerd are acknowledged by the surviving gateway and are
|
|
144
|
+
not replayed; bytes already accepted by the surviving app-facing socket are
|
|
145
|
+
likewise not delivered twice. Sequence gaps, overlaps, identity/ownership
|
|
146
|
+
mismatches, invalid manifests, spool/buffer limits, and authorization failures
|
|
147
|
+
fail closed.
|
|
148
|
+
|
|
149
|
+
Reverse replay state is bounded per stream and connection and stored in private
|
|
150
|
+
spool directories. ACKed chunks are truncated, and terminal close/reset removes
|
|
151
|
+
the spool. `status().spoolBytes` reports retained durable payload. Reverse-client
|
|
152
|
+
replacement requires the same explicit `connectionId`, an explicit persistent
|
|
153
|
+
`spoolDirectory`, and the same stream identifier, name, target, and one original
|
|
154
|
+
selector (`gatewayId` or `routeClass`). Register every stream with
|
|
155
|
+
`registerRecoveredConnection({...options, recover: true})` before `connect()`;
|
|
156
|
+
registration synchronously rejects missing, expired, corrupt, wrong-owner,
|
|
157
|
+
mismatched, noncontiguous, or oversized state before any network attempt. The
|
|
158
|
+
stored generation is advanced and persisted before AUTH, recovered outbound
|
|
159
|
+
replay starts above the gateway high-water position, and inbound delivery starts
|
|
160
|
+
at the durable receive ACK because unread bytes in the dead `VirtualSocket` no
|
|
161
|
+
longer exist. Receive ACKs advance only after the application consumes bytes,
|
|
162
|
+
not when `VirtualSocket` merely accepts them into its readable buffer. Recovery
|
|
163
|
+
uses the committed recovery record to discard an uncommitted chunk tail and
|
|
164
|
+
repair derivable usage accounting after an abrupt process loss; structural,
|
|
165
|
+
identity, sequence, quota, and non-tail corruption still fail closed. Recovery
|
|
166
|
+
never sends `OPEN`.
|
|
167
|
+
|
|
168
|
+
The default reverse spool is under `/tmp`, and the sample reverse-gateway
|
|
169
|
+
Compose mounts `/tmp` as tmpfs. Those defaults do not survive a container or
|
|
170
|
+
host replacement. A caller claiming process/container recovery must supply a
|
|
171
|
+
private persistent directory; Socketduct maintains `0700` directories and
|
|
172
|
+
`0600` files and never writes hub credentials or session tokens there.
|
|
173
|
+
|
|
174
|
+
This boundary does not reconstruct a hub after hub process loss: the current
|
|
175
|
+
manifests have no durable hub epoch, hub-minted bridge identity, endpoint-pairing
|
|
176
|
+
journal, or split-brain fence. It also does not transfer gateway worker file
|
|
177
|
+
descriptors, recover an outbound gateway process, reopen a lost target socket,
|
|
178
|
+
or replay a Docker request after gateway/container/host loss. A newly constructed
|
|
179
|
+
gateway owns no inherited targets; if the hub reports retained bridges, it sends
|
|
180
|
+
an empty resume manifest so omitted bridges reset once, without calling the
|
|
181
|
+
target factory or emitting `OPEN`. Loss outside the supported boundary fails the
|
|
182
|
+
logical stream and leaves higher-level reconciliation to the application.
|
|
128
183
|
|
|
129
184
|
Target bytes already observed by the gateway are ordered before a subsequent
|
|
130
185
|
target-error reset. Stream `END` remains a directional half-close, while protocol
|
|
@@ -207,11 +262,23 @@ await starting
|
|
|
207
262
|
Route-class sockets expose immutable `routeClass` allocation metadata and an
|
|
208
263
|
undefined `gatewayId`. Exact-gateway sets retain immutable `gatewayId` metadata.
|
|
209
264
|
Callers cannot override either selection per open. Reverse streams remain
|
|
210
|
-
|
|
265
|
+
sticky to their selected gateway/target but resume transparently after a
|
|
266
|
+
transient physical transport replacement.
|
|
267
|
+
|
|
268
|
+
For process replacement, call `recoverConnection()` before `start()` with
|
|
269
|
+
exactly the socket's immutable original selector: `gatewayId` for an exact set
|
|
270
|
+
or `routeClass` for a route-class set. Recovery derives the original member
|
|
271
|
+
identity and spool namespace and never falls back to or probes a sibling.
|
|
211
272
|
|
|
212
273
|
The socket is returned immediately and waits inside Socketduct until the set
|
|
213
274
|
reaches minimum readiness. New sockets rotate equally across ready members and
|
|
214
275
|
stay on their assigned member. If all members are reconnecting, new sockets wait
|
|
215
276
|
FIFO without selecting a connecting gateway; a recovered member resumes those
|
|
216
277
|
opens. A disconnected member leaves readiness and rejoins after authenticated
|
|
217
|
-
registration; existing
|
|
278
|
+
registration and paired stream resume; existing sockets remain open unless the
|
|
279
|
+
target/session is genuinely lost or a configured resume/buffer bound expires.
|
|
280
|
+
|
|
281
|
+
The paired reverse-resume fields are a protocol compatibility boundary. Upgrade
|
|
282
|
+
the hub, outbound gateways, and Server3/TensorBuzz Socketduct consumer together;
|
|
283
|
+
0.1.17 and older reverse peers do not exchange these manifests and cannot
|
|
284
|
+
provide transparent established-stream recovery.
|
package/docs/running.md
CHANGED
|
@@ -210,6 +210,6 @@ inspect its ordered aggregate causes or privacy-safe per-member status codes.
|
|
|
210
210
|
No extra physical member is created. Configure the set-wide buffer/spool/queue
|
|
211
211
|
limits for the whole process rather than multiplying a desired budget by
|
|
212
212
|
gateway count.
|
|
213
|
-
For process replacement, call `recoverConnection()` with each stream's
|
|
214
|
-
gateway ID before `start()`; see the
|
|
213
|
+
For process replacement, call `recoverConnection()` with exactly each stream's
|
|
214
|
+
original gateway ID or reverse route class before `start()`; see the sequence in
|
|
215
215
|
[Node-compatible socket API design](socket-api.md#multiplexedsocketducttransportset-multi-gateway-v3-api).
|
package/docs/security.md
CHANGED
|
@@ -166,10 +166,25 @@ stops reading cannot grow an unbounded queue or stall authorization, DATA, or
|
|
|
166
166
|
OPEN processing for a healthy sibling; exceeding the bound disconnects only
|
|
167
167
|
that destination and resets its owned streams.
|
|
168
168
|
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
169
|
+
Native reverse peers retain established logical streams for a bounded resume
|
|
170
|
+
window when only a hub transport is lost. Both authenticated endpoints must
|
|
171
|
+
reconnect with the same stable identities and newer generations, then present
|
|
172
|
+
matching per-stream identifiers, byte positions, acknowledgements, and
|
|
173
|
+
half-close state. Application bytes are durably spooled before transmission;
|
|
174
|
+
the hub reconciles to each receiver's committed position and permits replay
|
|
175
|
+
only above it. This preserves exact-once delivery without reopening or
|
|
176
|
+
reissuing a privileged Docker request. Ownership mismatches, missing targets,
|
|
177
|
+
invalid or incomplete manifests, resource-bound failures, and expired resume
|
|
178
|
+
windows fail the affected logical stream once instead of falling back to a new
|
|
179
|
+
stream. Reverse-client process replacement is allowed only from caller-selected
|
|
180
|
+
private persistent storage with the same stable connection/stream identities
|
|
181
|
+
and exactly one original gateway or route-class selector. Recovery validates
|
|
182
|
+
ownership, age, identity, lifecycle, contiguous chunks, and bounds before
|
|
183
|
+
network activity; spool files contain no bearer credential or session token.
|
|
184
|
+
The same in-memory hub bridge, gateway process, and live target socket must
|
|
185
|
+
survive. Hub reconstruction, gateway FD transfer, and target reopen/retry remain
|
|
186
|
+
unsupported, and a replacement gateway resets retained bridges with an empty
|
|
187
|
+
manifest rather than recreating a privileged operation.
|
|
173
188
|
|
|
174
189
|
## Network exposure
|
|
175
190
|
|
|
@@ -268,4 +283,6 @@ To roll back TLS: remove the TLS env vars from `.env`. If the relay binds to a n
|
|
|
268
283
|
- reverse registration rejects wrong principals, duplicate IDs, and stale generations
|
|
269
284
|
- reverse clients reject unknown/unavailable gateway IDs and wrong TLS CAs
|
|
270
285
|
- both hub route policy and the home-local allowlist deny unauthorized Docker targets
|
|
271
|
-
-
|
|
286
|
+
- transient reverse hub transport loss preserves each live stream and target,
|
|
287
|
+
replays only unacknowledged spooled bytes, and cleans all retained state after
|
|
288
|
+
close; target loss and resume expiry fail once with their causal codes
|
|
@@ -11,22 +11,44 @@ spawn per-stream reconnects.
|
|
|
11
11
|
## Reverse-gateway lifecycle
|
|
12
12
|
|
|
13
13
|
An outbound gateway moves through `connecting -> authenticated -> registering
|
|
14
|
-
-> ready`. Transport loss
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
14
|
+
-> ready`. Transport loss advances the gateway generation and reconnects with
|
|
15
|
+
bounded exponential delay without destroying established local targets. The hub
|
|
16
|
+
retains authenticated bridge ownership for the configured resume window and
|
|
17
|
+
replaces the affected Server3 client physical generation as well.
|
|
18
|
+
|
|
19
|
+
Both reverse endpoints keep their logical sockets alive, persist unacknowledged
|
|
20
|
+
send chunks, and submit their stable identifiers plus both directional
|
|
21
|
+
sequence/ACK positions. The hub resumes a bridge only after the two manifests
|
|
22
|
+
agree and returns both the receiver's accepted high-water sequence and its
|
|
23
|
+
application-consumed ACK to each sender. Bytes above the high-water position
|
|
24
|
+
replay on the same logical stream; accepted-but-unread bytes stay charged to the
|
|
25
|
+
window and spooled until ACK. This preserves exact-once, in-order delivery
|
|
26
|
+
without opening a second target or retrying an application operation.
|
|
27
|
+
|
|
28
|
+
If the original target socket disappears, an owner/identifier or sequence does
|
|
29
|
+
not match, a buffer/spool bound is exceeded, or the resume window expires, that
|
|
30
|
+
logical stream fails once with the causal error and releases both sides. A dead
|
|
31
|
+
reverse-client process may be replaced: the new object creates a new
|
|
32
|
+
`VirtualSocket`, restores the durable generation/lifecycle and replay state, and
|
|
33
|
+
registers every stream before connecting. The same hub bridge, outbound gateway
|
|
34
|
+
process, and original target socket must still exist. The dead app-facing socket
|
|
35
|
+
is not transferred; target bytes above its durable receive ACK replay to the new
|
|
36
|
+
socket. That ACK covers bytes returned to the application, not bytes merely
|
|
37
|
+
buffered in the dead process. An orderly `detach()` drains already-queued frame
|
|
38
|
+
handling before its final durable snapshot and handoff.
|
|
39
|
+
|
|
40
|
+
Hub process reconstruction, gateway process or file-descriptor transfer, and
|
|
41
|
+
target reopen/retry are unsupported. A replacement gateway has no live target
|
|
42
|
+
sockets and therefore completes retained-state registration with an empty
|
|
43
|
+
manifest, resetting omitted bridges rather than issuing a second `OPEN`.
|
|
23
44
|
|
|
24
45
|
The bounded, paginated resume manifest carries stable identifiers and both
|
|
25
46
|
directional acknowledgement positions. The relay returns an independent result
|
|
26
47
|
per stream. A stream omitted because it was closed while disconnected is
|
|
27
48
|
cleaned up at the relay. A target lost during interruption returns a stream-local
|
|
28
49
|
resume reset and a typed session-loss error, while retained siblings continue.
|
|
29
|
-
Only bytes above the peer
|
|
50
|
+
Only bytes above the peer's received high-water sequence are replayed; spool
|
|
51
|
+
cleanup remains governed by its acknowledgement.
|
|
30
52
|
|
|
31
53
|
## States
|
|
32
54
|
|
package/docs/socket-api.md
CHANGED
|
@@ -190,8 +190,9 @@ becomes the stable, immutable configured ID. For a route-class set,
|
|
|
190
190
|
the socket first removes the pending open, so later readiness cannot leak a
|
|
191
191
|
member stream. An exact relay member selected immediately before its path drops
|
|
192
192
|
retains the stream and performs member-local v3 recovery; the set never
|
|
193
|
-
speculatively migrates it. A reverse route-class stream
|
|
194
|
-
|
|
193
|
+
speculatively migrates it. A reverse route-class stream likewise remains on its
|
|
194
|
+
resolved target gateway while Socketduct replaces and resumes the physical
|
|
195
|
+
client/gateway transports underneath it.
|
|
195
196
|
|
|
196
197
|
`createConnection(options, callback)` is pre-bound and invokes its callback once
|
|
197
198
|
on `connect` or the terminal pre-connect error; the socket emits `connect` before
|
|
@@ -218,22 +219,23 @@ the callback, which runs exactly once only after `secureConnect` or failure.
|
|
|
218
219
|
Both methods count only completed member assignments and preserve stream
|
|
219
220
|
stickiness. Every returned raw or TLS socket exposes enumerable `gatewayId` and
|
|
220
221
|
`routeClass` allocation metadata. Only the configured identity is set. Persist
|
|
221
|
-
an exact set's immutable
|
|
222
|
-
|
|
223
|
-
|
|
222
|
+
an exact set's immutable original selector after connection: `gatewayId` for a
|
|
223
|
+
gateway member or `routeClass` for a reverse member. A replacement uses that
|
|
224
|
+
selector to recover the stream on its original member without running selection
|
|
225
|
+
again.
|
|
224
226
|
|
|
225
227
|
Process-replacement recovery is deliberately separate from ordinary selection.
|
|
226
228
|
First detach the old set when an orderly handoff is possible. Construct the
|
|
227
|
-
replacement with the same base `connectionId`, spool directory,
|
|
228
|
-
|
|
229
|
-
`gatewayId` before calling `start()`:
|
|
229
|
+
replacement with the same base `connectionId`, spool directory, and configured
|
|
230
|
+
member endpoints. Then register each durable stream with exactly its original
|
|
231
|
+
`gatewayId` or `routeClass` before calling `start()`:
|
|
230
232
|
|
|
231
233
|
```js
|
|
232
234
|
await set.detach()
|
|
233
235
|
|
|
234
236
|
const replacement = new MultiplexedSocketductTransportSet(sameSetOptions)
|
|
235
237
|
const recovered = replacement.recoverConnection({
|
|
236
|
-
|
|
238
|
+
routeClass: durableRecord.routeClass, // exact sets use gatewayId instead
|
|
237
239
|
host: "api.internal",
|
|
238
240
|
port: 443,
|
|
239
241
|
identifier: "build_stream_01",
|
|
@@ -242,10 +244,11 @@ const recovered = replacement.recoverConnection({
|
|
|
242
244
|
await replacement.start()
|
|
243
245
|
```
|
|
244
246
|
|
|
245
|
-
`recoverConnection()` implies durable recovery and rejects a missing
|
|
246
|
-
|
|
247
|
-
begun. It bypasses
|
|
248
|
-
migrates or replays the stream on a sibling.
|
|
247
|
+
`recoverConnection()` implies durable recovery and rejects a missing,
|
|
248
|
+
ambiguous, invalid, or unknown original selector, an explicit `recover` option,
|
|
249
|
+
or registration after `start()` has begun. It bypasses selection only to reach
|
|
250
|
+
that configured member; it never migrates or replays the stream on a sibling.
|
|
251
|
+
Ordinary `createConnection()` and
|
|
249
252
|
`createTlsConnection()` reject `recover: true`, `gatewayId`, and `routeClass`,
|
|
250
253
|
so new opens cannot force a path. `detach()` coalesces with concurrent shutdown calls,
|
|
251
254
|
preserves active member spools, and retires empty warmed lanes; `close()` remains
|
|
@@ -398,9 +401,47 @@ Target `ca`, `cert`, and `key` accept Node TLS certificate material. `cert` and
|
|
|
398
401
|
It selects exactly one registered gateway during `connect()`. The existing
|
|
399
402
|
`MultiplexedSocketductTransportSet` accepts `endpoint.reverseHub`, allowing
|
|
400
403
|
three ready members to retain deterministic health-aware round-robin behavior.
|
|
401
|
-
|
|
402
|
-
|
|
403
|
-
|
|
404
|
+
An established reverse stream can be cold-registered in a replacement client
|
|
405
|
+
process before its first physical connection:
|
|
406
|
+
|
|
407
|
+
```js
|
|
408
|
+
const replacement = new ReverseGatewayConnection({
|
|
409
|
+
routeClass: durableRecord.routeClass,
|
|
410
|
+
connectionId: durableRecord.connectionId,
|
|
411
|
+
spoolDirectory: "/srv/socketduct/recovery",
|
|
412
|
+
hub,
|
|
413
|
+
tls
|
|
414
|
+
})
|
|
415
|
+
const socket = replacement.registerRecoveredConnection({
|
|
416
|
+
recover: true,
|
|
417
|
+
host: durableRecord.host,
|
|
418
|
+
port: durableRecord.port,
|
|
419
|
+
identifier: durableRecord.identifier,
|
|
420
|
+
sessionName: durableRecord.sessionName
|
|
421
|
+
})
|
|
422
|
+
await replacement.connect()
|
|
423
|
+
```
|
|
424
|
+
|
|
425
|
+
Recovery restores the durable stream lifecycle, sequences, acknowledgements,
|
|
426
|
+
and generation. It never emits a new `OPEN` or recreates the target; target data
|
|
427
|
+
above the durable receive acknowledgement is replayed by the surviving gateway.
|
|
428
|
+
Reverse receive ACKs cover bytes returned to the application, not unread bytes
|
|
429
|
+
held only in the old process. `detach()` resolves only after queued physical
|
|
430
|
+
frame handling is quiescent and the resulting recovery state is durable, so a
|
|
431
|
+
replacement cannot race a late write from the retired object.
|
|
432
|
+
The caller must supply stable `connectionId` and private persistent
|
|
433
|
+
`spoolDirectory` values. The default `/tmp` spool and the sample gateway
|
|
434
|
+
Compose tmpfs do not survive container or host replacement. Spool directories
|
|
435
|
+
remain mode 0700, files remain mode 0600, and credentials or session tokens are
|
|
436
|
+
not persisted.
|
|
437
|
+
|
|
438
|
+
This boundary requires the same hub bridge, reverse-gateway process, and target
|
|
439
|
+
TCP socket to survive. It does not reconstruct a restarted hub, keep or transfer
|
|
440
|
+
gateway sockets between processes, reopen a lost target, or replay an ambiguous
|
|
441
|
+
target request after gateway, container, or host loss. If a newly constructed
|
|
442
|
+
outbound gateway finds retained hub bridges but owns no live target sockets, it
|
|
443
|
+
sends an empty resume manifest so the hub resets those omitted streams instead
|
|
444
|
+
of creating targets.
|
|
404
445
|
|
|
405
446
|
Reverse virtual sockets accept individual writes larger than their configured
|
|
406
447
|
retained-byte limit. The connection progresses those writes in slices bounded by
|
package/package.json
CHANGED
|
@@ -270,6 +270,7 @@ export class MultiplexedSocketductTransportSet extends EventEmitter {
|
|
|
270
270
|
tls: endpoint.tls,
|
|
271
271
|
...(endpoint.createPhysicalSocket === undefined ? {} : {createPhysicalSocket: endpoint.createPhysicalSocket}),
|
|
272
272
|
connectionId: identity.connectionId,
|
|
273
|
+
spoolDirectory: join(validated.spoolDirectory, identity.spoolName),
|
|
273
274
|
continuousReconnect: true,
|
|
274
275
|
aggregateBudget: this.#budget,
|
|
275
276
|
...(this.#logger === undefined ? {} : {logger: this.#logger})
|
|
@@ -398,20 +399,27 @@ export class MultiplexedSocketductTransportSet extends EventEmitter {
|
|
|
398
399
|
}
|
|
399
400
|
|
|
400
401
|
/**
|
|
401
|
-
* Registers a durable stream on its original
|
|
402
|
-
* @param {import('./multiplexed-connection.js').VirtualConnectionOptions & {gatewayId: string}} options - Original gateway
|
|
402
|
+
* Registers a durable stream on its one original member before any member starts.
|
|
403
|
+
* @param {import('./multiplexed-connection.js').VirtualConnectionOptions & ({gatewayId: string, routeClass?: never} | {routeClass: string, gatewayId?: never})} options - Original gateway or route-class recovery identity.
|
|
403
404
|
* @param {(error: Error | null, socket: GatewayVirtualSocket) => void} [callback] - Existing callback convention.
|
|
404
405
|
* @returns {GatewayVirtualSocket} Original member-owned virtual socket with stable gateway metadata.
|
|
405
406
|
*/
|
|
406
407
|
recoverConnection(options, callback) {
|
|
407
408
|
if (this.#closed) throw closedError()
|
|
408
409
|
if (this.#started) throw new Error("Recovered streams must be registered before start()")
|
|
409
|
-
if (typeof options !== "object" || options === null
|
|
410
|
+
if (typeof options !== "object" || options === null) throw new TypeError("recoverConnection requires exactly one configured gatewayId or routeClass")
|
|
411
|
+
const hasGatewayId = Object.hasOwn(options, "gatewayId")
|
|
412
|
+
const hasRouteClass = Object.hasOwn(options, "routeClass")
|
|
413
|
+
if (hasGatewayId === hasRouteClass) throw new TypeError("recoverConnection requires exactly one configured gatewayId or routeClass")
|
|
414
|
+
if (hasGatewayId && (typeof options.gatewayId !== "string" || !GATEWAY_ID.test(options.gatewayId))) throw new TypeError("recoverConnection requires a valid configured gatewayId")
|
|
415
|
+
if (hasRouteClass && (typeof options.routeClass !== "string" || !ROUTE_CLASS.test(options.routeClass))) throw new TypeError("recoverConnection requires a valid configured routeClass")
|
|
410
416
|
if (Object.hasOwn(options, "recover")) throw new TypeError("recoverConnection owns the recover option")
|
|
411
|
-
const
|
|
412
|
-
|
|
417
|
+
const selector = hasGatewayId ? options.gatewayId : options.routeClass
|
|
418
|
+
const member = this.#members.find((candidate) => hasGatewayId ? candidate.gatewayId === selector : candidate.routeClass === selector)
|
|
419
|
+
if (member === undefined) throw new Error(`Unknown ${hasGatewayId ? "gatewayId" : "routeClass"}: ${selector}`)
|
|
413
420
|
const virtualOptions = {...options}
|
|
414
421
|
delete /** @type {{gatewayId?: string}} */ (virtualOptions).gatewayId
|
|
422
|
+
delete /** @type {{routeClass?: string}} */ (virtualOptions).routeClass
|
|
415
423
|
const allocation = memberAllocation(member)
|
|
416
424
|
/** @type {((error: Error | null, socket: import('./virtual-socket.js').VirtualSocket) => void) | undefined} */
|
|
417
425
|
const memberCallback = callback === undefined ? undefined : (error, socket) => callback(error, exposeAllocation(socket, allocation))
|