socketduct 0.1.15

package/docs/protocol.md

@@ -0,0 +1,558 @@
+# Socketduct wire protocol
+
+Socketduct has two separately selected wire protocols. Protocol v3 is the
+multiplexed transport. Protocol v2 remains available unchanged for migration
+and rollback. A listener is configured for one version; there is no ambiguous
+fallback after bytes have been sent.
+
+## Multiplexed protocol version 3
+
+V3 carries many independent virtual byte streams on one authenticated TCP or
+TLS connection. The physical connection performs TLS and bearer authentication
+once. Stream ID `0` is reserved for connection control; client-created stream
+IDs are monotonically allocated odd uint32 values and are never reused during a
+connection identity's lifetime.
+
+### Binary frame
+
+Every frame has a 24-byte big-endian header followed by its payload:
+
+```text
+offset  size  field
+0       4     ASCII magic "SDV3"
+4       1     version (3)
+5       1     frame type
+6       2     flags
+8       4     stream ID
+12      8     sequence (restricted to JavaScript safe integers)
+20      4     payload length
+24      N     payload
+```
+
+Unknown types, bad magic/version, unsafe sequences, stream/control ID
+confusion, oversized payloads, non-object control JSON, gaps, overlaps and
+invalid acknowledgements fail loudly. Control JSON and DATA frames are bounded
+independently. Ordinary v3 relay connections retain raw DATA. Native reverse
+connections negotiate zstd and reserve DATA flag `0x0001`; no ordinary-v3 peer
+emits it and ordinary handlers reject it as unnegotiated.
+
+### Connection control (stream 0)
+
+- `AUTH` / `AUTH_OK` / `AUTH_ERROR`: negotiate exactly version 3 and
+  authenticate a stable connection identity. No stream operation is accepted
+  before `AUTH_OK`.
+- `PING` / `PONG`: connection heartbeat and readiness.
+- `RESUME` / `RESUMED`: bounded pages of stream manifests. Each entry contains
+  stream ID, stable identifier and both directional acknowledgement positions.
+- `GOAWAY`: terminal physical-connection shutdown. Authentication,
+  authorization and TLS failures are terminal and are not retried forever.
+- `GATEWAY_REGISTER` / `GATEWAY_REGISTERED`: reverse-only authenticated stable
+  gateway registration and generation fencing.
+- `GATEWAY_SELECT` / `GATEWAY_SELECTED`: reverse-only binding of a client
+  physical member to one currently registered gateway before readiness.
+
+### Stream control (nonzero stream ID)
+
+- `OPEN` / `OPENED`: stable opaque identifier, human-readable name, exact
+  target and initial receive window. The relay authorizes the target for every
+  OPEN before creating its one local target socket.
+- `DATA`: raw bytes with a cumulative end sequence.
+- `ACK`: cumulative committed/delivered sequence; acknowledged replay bytes are
+  removed from the spool and never replayed.
+- `WINDOW`: explicit additional receive credit when needed.
+- `END`: directional half-close.
+- `CLOSE`: clean terminal close. The receiver echoes `CLOSE` after delivery;
+  the sender retains the stream until that terminal acknowledgement arrives,
+  so cleanup cannot race the last bytes or leak relay stream capacity.
+- `RESET`: stream-local error/cancellation. Resetting one stream never closes a
+  sibling or the shared physical transport.
+
+Each direction has an independent per-stream window. Send/replay bytes are
+bounded per stream and in aggregate per connection. Control frames have
+priority; DATA frames use round-robin scheduling with bounded frame sizes, so a
+large upload cannot monopolize the physical connection.
+
+Flow-control credit and scheduler position belong to one physical generation.
+After both peers reconcile cumulative durable acknowledgements in RESUME, each
+surviving stream begins the replacement generation with a fresh negotiated
+window and replay is rebuilt strictly above the reconciled ACK. Credit consumed
+by a failed transport is never carried forward.
+
+Relay retention is bounded independently per stream, per connection identity,
+and process-wide. The process-wide reservation is owned by the relay runtime
+and is released idempotently on acknowledgement, reset, close, error or expiry.
+
+### Physical reconnect and resume
+
+One `MultiplexedSocketductConnection` owns one connect/auth promise and one
+reconnect timer. Concurrent first opens share that promise. After a transport
+drop, active streams remain represented by their stable identifiers and spool
+state while exactly one new physical generation authenticates. The client sends
+the paginated resume manifest; the relay answers per stream and replays only
+bytes above the peer's acknowledgement. A missing target or invalid resume
+resets only that stream with a typed session-loss error. Streams locally closed
+during the outage are omitted and the relay closes their retained targets.
+The connect/authentication deadline remains active until `AUTH_OK`; a peer that
+accepts TCP/TLS and stays silent cannot retain the shared open promise.
+Replacement objects advertise only the durable receive acknowledgement because
+they do not own unread bytes buffered by the previous `VirtualSocket`; a
+physical reconnect on the same object may acknowledge its still-owned readable
+buffer. A durable `OPEN` without `OPENED` is resumed and reconciled by the
+relay's per-stream result rather than being sent again.
+
+### Native reverse-gateway extension
+
+Reverse mode reuses the v3 physical transport and stream frames directly. A
+home gateway registers outbound; a Server3 client selects it. Client streams
+are odd and the hub maps them to even gateway-side IDs. The hub authorizes the
+principal/gateway/target tuple and the gateway separately authorizes its local
+target. Registration reconnect uses a higher generation, but any established
+stream is reset on gateway loss and never replayed.
+
+Stream IDs must increase monotonically on each physical connection. END and
+CLOSE positions must match the sender's accepted raw-byte sequence; a
+gateway-originated CLOSE is retained until the client acknowledges it with
+CLOSE. RESET is immediately terminal. Repeats, data after END, and stale ACKs
+are protocol violations.
+
+Reverse peers must negotiate `zstd` before selection/registration completes.
+A compressed DATA payload is `uint32_be(uncompressedLength)` followed by one
+independent zstd frame and carries flag `0x0001`. Raw DATA carries flag zero.
+Sequences, acknowledgements, windows, and all buffer limits count uncompressed
+bytes. Invalid metadata, unavailable codecs, decompression bombs, or corrupt
+payloads terminate the physical connection. TLS certificate verification is
+mandatory for every reverse client and gateway transport. The complete role
+contract is in [Native reverse gateway](reverse-gateway.md).
+
+## Legacy protocol version 2
+
+V2 starts every connection with the length-prefixed JSON envelope
+described below. That envelope remains the bootstrap format, but authenticated
+sessions require compact framing version 2 for all stream records.
+
+## Compact framing version 2
+
+The client must add `"framing":[2]` to the length-prefixed `session.create` or
+`session.resume` header. The relay must reject an absent offer or `"framing":[1]`
+with `PROTOCOL_VIOLATION` before authentication and before creating or resuming
+a session. Every successful `session.created` and `session.resumed` response
+contains `"framing":2`; the client treats any response without that exact value
+as a failed handshake. The offer and selection are connection-local and are
+repeated on resume. There is no older framing fallback.
+
+After selection, every `stream.data` and `stream.ack` uses a compact record.
+Bootstrap, error, lifecycle (including `session.close`), and other control
+messages retain the length-prefixed JSON envelope because compact v2 has no
+equivalent record for them. A compact record starts with a byte whose high bit
+is set, while a valid JSON-envelope header length
+starts with a zero byte under the 64 KiB header limit, so controls and compact
+records can be mixed in one TCP byte stream.
+
+After a successful V2 handshake, both peers reject any length-prefixed
+`stream.data` or `stream.ack` input as a protocol violation. Parsers use the
+decoded `header.compact` marker to enforce this at the authenticated session
+boundary while continuing to accept length-prefixed control, lifecycle, and
+error messages.
+
+```text
+0x80  varint(endSequence) varint(payloadLength) rawPayload
+0x81  varint(endSequence) varint(uncompressedLength) varint(payloadLength) zstdPayload
+0x82  varint(ackSequence)
+```
+
+Integers are canonical unsigned LEB128 and must be JavaScript safe integers.
+Overlong encodings, overflow, truncated records, and payload lengths above the
+configured frame limit are protocol violations. Direction and session are
+implicit in the authenticated connection: data travels toward the receiving
+endpoint and an ACK acknowledges the opposite data direction. Decoders
+normalize these records into the same internal headers used by the envelope decoder, so zstd
+negotiation and expansion checks, cumulative uncompressed byte sequences,
+ACKs, replay retention, and raw spool storage have identical semantics.
+
+## No-delay coalescing
+
+A sole application write, or a sole chunk read from the relay's target socket,
+begins spool/compression/send work immediately.
+Socketduct does not use a timer, debounce, immediate, or microtask to wait for
+more data. If Node has already queued several application writes behind
+pre-existing async spool, compression, send, or drain work, the edge may merge
+those queued original bytes in order. Likewise, the relay may merge adjacent
+raw target chunks only when they are already waiting behind its per-session
+compression/send/drain queue. The first target chunk entering an idle queue is
+not held. Each merged record is bounded by the configured maximum decompressed
+size. Close, error, replay, ACK retention, and spool ordering use the same
+serialized pipelines as unmerged writes; cumulative sequences always count
+the exact uncompressed bytes.
+
+## Requirements
+
+The protocol must support:
+
+- authentication before session operations
+- session creation with `name`, `identifier`, and authorized `target`
+- session resume by `identifier`
+- ordered byte-stream transfer
+- acknowledgements
+- reconnection after short interruptions
+- replay from temporary spool files
+- clear failures when resume is impossible
+
+## Framing
+
+Use a length-prefixed frame format rather than raw JSON lines.
+
+A practical MVP can use:
+
+```text
+uint32_be headerLength
+json header
+bytes payload
+```
+
+Header fields:
+
+```json
+{
+  "version": 1,
+  "type": "stream.data",
+  "session": "018ff4c1-4cd9-7a25-b610-50cf8e10a4e7",
+  "sequence": 12345,
+  "payloadLength": 4096
+}
+```
+
+Later, this can move to a binary header if profiling shows JSON headers matter.
+
+## Authentication flow
+
+```text
+client -> server: auth.login
+server -> client: auth.ok | auth.error
+```
+
+Request:
+
+```json
+{
+  "type": "auth.login",
+  "username": "kasper",
+  "password": "<redacted>",
+  "clientName": "office-edge"
+}
+```
+
+Response:
+
+```json
+{
+  "type": "auth.ok",
+  "token": "<opaque-session-token>",
+  "expiresAt": "2026-07-07T12:00:00.000Z"
+}
+```
+
+After login, reconnects should prefer token auth:
+
+```json
+{
+  "type": "auth.resume",
+  "token": "<opaque-session-token>"
+}
+```
+
+## Session create
+
+```text
+client -> server: session.create
+server -> client: session.created | error
+```
+
+Request:
+
+```json
+{
+  "type": "session.create",
+  "name": "work-ssh",
+  "identifier": null,
+  "target": {
+    "host": "10.0.0.15",
+    "port": 22
+  },
+  "framing": [2]
+}
+```
+
+Response:
+
+```json
+{
+  "type": "session.created",
+  "identifier": "018ff4c1-4cd9-7a25-b610-50cf8e10a4e7",
+  "name": "work-ssh",
+  "state": "active",
+  "target": {
+    "host": "10.0.0.15",
+    "port": 22
+  },
+  "framing": 2
+}
+```
+
+The client may propose an identifier, but the server must verify that it is valid, owned by the user, and not already active for another incompatible session. The server must also validate `target` before creating the session. The relay is not configured with one static upstream target; it applies an explicit allowlist policy such as `authorizeTarget({principal, target})` for each create request.
+
+After authentication and authorization, the server stores the session in `creating` but withholds `session.created` until target TCP connect succeeds. Initial connection refusal is not a wire error: the relay retries at a bounded exponential rate while the authenticated tunnel remains open. Compact stream records received before readiness are a terminal `PROTOCOL_VIOLATION`. If the tunnel closes before the response is announced, the relay deletes the creating session immediately, so it cannot be resumed.
+
+`session.created` is the safety boundary. Before it, the relay has accepted no application bytes and may retry target acquisition. After it, target failure or write uncertainty is terminal and Socketduct does not reconnect the target or replay the application operation.
+
+## Session resume
+
+```text
+client -> server: session.resume
+server -> client: session.resumed | error
+client -> server: spool.manifest
+client -> server: stream.data frames for unacknowledged chunks
+```
+
+Request:
+
+```json
+{
+  "type": "session.resume",
+  "identifier": "018ff4c1-4cd9-7a25-b610-50cf8e10a4e7",
+  "name": "work-ssh",
+  "clientLastSequence": 180224,
+  "clientLastAck": 171520,
+  "framing": [2]
+}
+```
+
+Response:
+
+```json
+{
+  "type": "session.resumed",
+  "identifier": "018ff4c1-4cd9-7a25-b610-50cf8e10a4e7",
+  "serverLastSequence": 99120,
+  "serverLastAck": 171520,
+  "replayFromSequence": 171521,
+  "target": {
+    "host": "10.0.0.15",
+    "port": 22
+  },
+  "framing": 2
+}
+```
+
+The server tells the client which sequence to replay from. The edge then sends the relevant spooled data.
+
+## Stream data
+
+```json
+{
+  "type": "stream.data",
+  "session": "018ff4c1-4cd9-7a25-b610-50cf8e10a4e7",
+  "direction": "client-to-target",
+  "sequence": 171521,
+  "payloadLength": 4096
+}
+```
+
+The payload bytes follow the header.
+
+## Stream data compression
+
+`stream.data` payloads may be compressed. Nothing else is: authentication,
+session and other control frames are always sent raw.
+
+### Negotiation
+
+Compression is opt-in per connection and rides on the existing handshake frames,
+because unknown header fields are ignored by both peers while unknown frame
+types are a protocol violation.
+
+The client advertises the codecs it can use on `session.create` and
+`session.resume`:
+
+```json
+{
+  "type": "session.create",
+  "token": "<redacted>",
+  "name": "office-edge",
+  "target": {"host": "10.0.0.15", "port": 22},
+  "compression": ["zstd"]
+}
+```
+
+The relay answers on `session.created` / `session.resumed` with the single codec
+it selected, or omits the field entirely:
+
+```json
+{
+  "type": "session.created",
+  "identifier": "018ff4c1-4cd9-7a25-b610-50cf8e10a4e7",
+  "compression": "zstd"
+}
+```
+
+If compression is omitted, stream payloads are uncompressed but still use
+compact framing v2. Compression negotiation is redone on every reconnect, so a
+resumed session may land on a different codec (or none) than it started with.
+
+### Frame metadata
+
+A compressed compact `stream.data` record carries two extra normalized header
+fields. Both are absent on an uncompressed compact record.
+
+```json
+{
+  "type": "stream.data",
+  "direction": "target-to-client",
+  "sequence": 171521,
+  "encoding": "zstd",
+  "dataLength": 4096
+}
+```
+
+- `encoding` — the negotiated codec. Absent means the payload is uncompressed.
+- `dataLength` — the **uncompressed** byte length. Required whenever `encoding`
+  is present.
+
+Each frame is an independent zstd frame with no shared dictionary or window, so
+any frame can be decoded on its own. That is what keeps reconnect, resume and
+replay correct: spools and relay replay buffers store **raw** chunks, and a
+replayed chunk is re-encoded under whatever codec the current connection
+negotiated.
+
+`sequence`, `ackSequence`, spool offsets, byte limits and backpressure are all
+counted in **uncompressed** bytes. `dataLength`, not the wire payload length, is
+what the relay's contiguity check (`sequence == previous + length`) uses.
+
+### Validation
+
+A receiver fails closed. Any of the following is a `PROTOCOL_VIOLATION` that
+ends the connection; a frame that cannot be decoded is never written to the
+target, and there is no fallback to treating it as raw:
+
+- `encoding` present but not the codec negotiated for this connection.
+- `dataLength` missing, not a positive safe integer, or above the configured
+  maximum.
+- `dataLength` present without an `encoding`.
+- An uncompressed payload above the configured maximum `stream.data` size.
+- An expansion ratio (`dataLength` / compressed size) above the configured
+  maximum.
+- A payload that fails to decompress, or whose decompressed size does not
+  exactly equal the advertised `dataLength`.
+
+Decompression is bounded by the advertised `dataLength`, so a compression bomb
+can never produce more bytes than the sender promised.
+
+### Send policy
+
+Compression is adaptive and entirely send-side; a receiver needs no knowledge of
+it, because every frame declares its own encoding.
+
+- Payloads below `SOCKETDUCT_MIN_COMPRESS_BYTES` (default 256) are sent raw.
+- The compressed form is kept only if it actually saves bytes, so a frame never
+  grows.
+- Streams that do not compress (TLS, media, already-compressed data) stop being
+  compressed after a few probes and are re-probed periodically.
+
+### Runtime support
+
+zstd is provided by `node:zlib`, which only exposes it from Node 22.15 onward,
+while `engines` accepts Node >=22.0. Support is therefore feature-detected at
+runtime: a peer whose Node lacks zstd advertises no codec and the tunnel runs
+raw. No runtime dependency is added.
+
+## Acknowledgements
+
+```json
+{
+  "type": "stream.ack",
+  "session": "018ff4c1-4cd9-7a25-b610-50cf8e10a4e7",
+  "direction": "client-to-target",
+  "ackSequence": 175616
+}
+```
+
+Acked chunks can be truncated from the spool file.
+
+## Session close
+
+`session.close` remains an envelope control frame. A relay response confirming
+that it processed a client-requested logical close includes
+`"acknowledged": true`; a target-originated terminal close omits that field.
+Clients must not treat an unacknowledged terminal notification already in
+flight as confirmation of a later close request. A transport EOF is not a
+logical close and retains the announced session for normal resume.
+If the application requests a logical close while the relay transport is
+interrupted, the client first reconnects, authenticates, and receives
+`session.resumed`; only then does it send `session.close`. No application data
+is replayed by this close-only reconnect.
+
+## Backpressure
+
+The protocol should allow explicit pause/resume:
+
+```json
+{
+  "type": "stream.pause",
+  "session": "018ff4c1-4cd9-7a25-b610-50cf8e10a4e7",
+  "direction": "client-to-target",
+  "reason": "remote-buffer-full"
+}
+```
+
+```json
+{
+  "type": "stream.resume",
+  "session": "018ff4c1-4cd9-7a25-b610-50cf8e10a4e7",
+  "direction": "client-to-target"
+}
+```
+
+## Error handling
+
+Errors should be explicit and machine-readable:
+
+```json
+{
+  "type": "error",
+  "code": "SESSION_NOT_FOUND",
+  "message": "Session does not exist or is not owned by this user.",
+  "retryable": false
+}
+```
+
+Useful error codes:
+
+- `AUTH_REQUIRED`
+- `AUTH_FAILED`
+- `SESSION_NOT_FOUND`
+- `SESSION_ALREADY_ACTIVE`
+- `SESSION_EXPIRED`
+- `REPLAY_TOO_OLD`
+- `SPOOL_TOO_LARGE`
+- `TARGET_CLOSED`
+- `TARGET_BUFFER_OVERFLOW`
+- `TARGET_UNAUTHORIZED`
+- `PROTOCOL_VIOLATION`
+- `HANDSHAKE_TIMEOUT`
+
+`TARGET_CLOSED`, `TARGET_BUFFER_OVERFLOW`, spool overflow, authorization errors, and protocol errors are terminal (`retryable:false`). Initial target refusal does not emit `TARGET_UNAVAILABLE` or any other error frame.
+
+## Open protocol decisions
+
+- Use TLS-over-TCP, WebSocket, HTTP/2, or QUIC as the first edge-relay transport?
+- Should session identifiers be client-generated UUIDv7 or server-generated random IDs?
+- Should password auth be part of the tunnel protocol, or should the first version use pre-created bearer tokens?
+- How much target-side state should the relay keep after an edge disconnect?
+
+## Transport security
+
+The relay supports TLS via Node.js built-in `node:tls`. When configured with `SOCKETDUCT_TLS_CERT_FILE` and `SOCKETDUCT_TLS_KEY_FILE`, the protocol runs over a TLS socket. The mixed wire format (length-prefixed control envelopes and compact V2 stream records) is unchanged regardless of whether the transport is plain TCP or TLS.
+
+- Server certificate verification is enabled by default on the client (`rejectUnauthorized: true`).
+- Optional mutual TLS: when `SOCKETDUCT_TLS_CLIENT_CA_FILE` is set, the relay requests and verifies client certificates.
+- The relay fails fast at startup if certificate files are missing or unreadable.
+- Binding to a non-loopback address without TLS is refused unless `SOCKETDUCT_ALLOW_INSECURE_PUBLIC=1` or `true`.

package/docs/releasing.md

@@ -0,0 +1,18 @@
+# Releasing
+
+Socketduct publishes patch releases from a clean, current `master` checkout:
+
+```sh
+npm run release:patch
+```
+
+`release-patch` owns the version bump, lockfile update, annotated tag, atomic
+Git push, npm publication, and registry verification. Do not run `npm version`
+or create release tags manually for ordinary patch releases.
+
+Confirm CI is green and `npm run check` passes before releasing. If publication
+fails after the commit and tag are pushed, resume the exact release with:
+
+```sh
+npx release-patch --resume
+```