socketduct 0.1.15

package/docker-compose.yml

@@ -0,0 +1,73 @@
+services:
+  socketduct-relay:
+    image: ${SOCKETDUCT_IMAGE:?Set SOCKETDUCT_IMAGE to ghcr.io/kaspernj/socketduct@sha256 followed by 64 lowercase hex characters}
+    restart: unless-stopped
+    init: true
+    user: "1000:1000"
+    read_only: true
+    cap_drop:
+      - ALL
+    security_opt:
+      - no-new-privileges:true
+    pids_limit: 256
+    stop_grace_period: 45s
+    tmpfs:
+      - /tmp:rw,noexec,nosuid,nodev,size=64m
+    env_file:
+      - .env
+    environment:
+      SOCKETDUCT_MODE: relay
+      SOCKETDUCT_RELAY_HOST: 0.0.0.0
+      SOCKETDUCT_RELAY_PORT: ${SOCKETDUCT_RELAY_PORT:-3100}
+      SOCKETDUCT_PROTOCOL_VERSION: ${SOCKETDUCT_PROTOCOL_VERSION:-2}
+      SOCKETDUCT_TLS_CERT_FILE: ${SOCKETDUCT_TLS_CERT_FILE:-}
+      SOCKETDUCT_TLS_KEY_FILE: ${SOCKETDUCT_TLS_KEY_FILE:-}
+      SOCKETDUCT_TLS_CLIENT_CA_FILE: ${SOCKETDUCT_TLS_CLIENT_CA_FILE:-}
+      SOCKETDUCT_ALLOW_INSECURE_PUBLIC: ${SOCKETDUCT_ALLOW_INSECURE_PUBLIC:-}
+      SOCKETDUCT_STATUS_INTERVAL_MS: ${SOCKETDUCT_STATUS_INTERVAL_MS:-30000}
+      SOCKETDUCT_INITIAL_TARGET_RETRY_DELAY_MS: ${SOCKETDUCT_INITIAL_TARGET_RETRY_DELAY_MS:-250}
+      SOCKETDUCT_INITIAL_TARGET_RETRY_MAX_DELAY_MS: ${SOCKETDUCT_INITIAL_TARGET_RETRY_MAX_DELAY_MS:-5000}
+    ports:
+      - "${SOCKETDUCT_PUBLISH_HOST:-127.0.0.1}:${SOCKETDUCT_RELAY_PORT:-3100}:${SOCKETDUCT_RELAY_PORT:-3100}"
+    networks:
+      - socketduct-internal
+    volumes:
+      - ${SOCKETDUCT_TLS_CERT_DIR:-./certs}:/certs:ro
+      - ${SOCKETDUCT_LOG_HOST_DIRECTORY:-./logs}:/var/log/socketduct
+    healthcheck:
+      test:
+        - CMD-SHELL
+        - >-
+          node --input-type=module -e "import net from 'node:net'; const port=Number(process.env.SOCKETDUCT_RELAY_PORT || 3100); const socket=net.createConnection({host:'127.0.0.1', port}, () => { socket.end(); process.exit(0) }); socket.setTimeout(2000, () => { socket.destroy(); process.exit(1) }); socket.on('error', () => process.exit(1))"
+      interval: 30s
+      timeout: 3s
+      retries: 3
+      start_period: 10s
+
+  docker-socket-shim:
+    image: alpine/socat:1.8.0.0
+    restart: unless-stopped
+    init: true
+    read_only: true
+    cap_drop:
+      - ALL
+    security_opt:
+      - no-new-privileges:true
+    pids_limit: 128
+    tmpfs:
+      - /tmp:rw,noexec,nosuid,nodev,size=16m
+    profiles:
+      - docker-host
+    command:
+      - tcp-listen:2375,fork,reuseaddr
+      - unix-connect:/var/run/docker.sock
+    expose:
+      - "2375"
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+    networks:
+      - socketduct-internal
+
+networks:
+  socketduct-internal:
+    driver: bridge

package/docs/architecture.md

@@ -0,0 +1,264 @@
+# Architecture
+
+## Problem
+
+A normal app opens a TCP/socket connection through an internet path that sometimes drops for a few seconds. The app sees the socket close and must reconnect, often losing stream/session state.
+
+Socketduct moves the unstable part into a controlled tunnel:
+
+```text
+Stable LAN                         Unstable WAN                         Stable remote
+----------                         ------------                         -------------
+Application -> local edge/proxy -> resumable tunnel -> relay/server -> target service
+```
+
+The application only talks to the local edge over a stable LAN. The edge and relay speak Socketduct's resumable protocol across the flaky WAN.
+
+## Components
+
+### Protocol-v3 multiplexed connection
+
+`MultiplexedSocketductConnection` is the WAN ownership boundary. It owns
+exactly one `net.Socket` or `tls.TLSSocket`, authentication, heartbeat,
+generation negotiation, the shared reconnect lock, paginated resume manifests,
+aggregate buffer limits and fair scheduling. Concurrent virtual opens share its
+single connection/authentication promise.
+
+The relay runtime additionally owns one process-wide retained-byte reservation
+across all authenticated v3 connection identities. Per-stream and
+per-connection bounds remain independent safeguards. Physical-generation
+window credit is discarded and recomputed from durable ACK positions on resume.
+
+Each `VirtualSocket` is only a stream boundary. It owns a collision-free stream
+ID, stable resumable identifier/name, authorized target metadata, directional
+sequences and acknowledgements, flow-control windows, replay state, half-close,
+clean close and reset. On the relay each virtual stream owns one local target
+socket; those local sockets are not WAN Socketduct transports. A stream reset
+cannot close its siblings.
+
+For process replacement, callers opt in with a stable connection identity and
+stable stream identifiers. Bounded per-stream recovery metadata records the
+wire stream ID, target, generation and directional ACK state alongside the
+outbound spool. Fresh creation is exclusive; recovery never truncates an
+existing spool and rejects corrupt or identity-mismatched state.
+
+### Process-scoped multi-gateway transport set
+
+`MultiplexedSocketductTransportSet` is a client-side ownership layer, not a new
+wire protocol. It constructs one long-lived `MultiplexedSocketductConnection`
+for each explicit gateway ID and pinned endpoint/factory. Each member continues
+to own exactly one physical lane, connectionId, spool namespace, stream IDs,
+flow control, heartbeat, reconnect lock and v3 resume manifest. The set never
+copies stream/replay state between members.
+
+All configured members warm concurrently. Minimum readiness defaults to one;
+unready paths continue member-local reconnect attempts while ready siblings
+serve new opens. The set keeps a configured-order cursor and advances it once
+per successful virtual allocation, skipping members that are not ready. Thus
+continuously ready members differ by at most one assignment. A recovered member
+rejoins future selection at the cursor without taking existing sockets.
+
+The set also owns the application-facing readiness gap. It returns a bounded
+deferred `VirtualSocket` immediately when startup has not reached minimum
+readiness or every member is reconnecting, and holds opens FIFO without choosing
+a gateway. Once eligible, the facade is bridged to one ready member and its
+gateway ID becomes immutable. Destruction cancels an unassigned open; terminal
+startup, terminal all-member loss and set shutdown fail the facade through
+normal socket error/close behavior. Target TLS wraps the facade immediately, so
+the TLS layer continues to own its handshake and `secureConnect` lifecycle.
+
+Retained outbound bytes are both memory and durable-spool reservations. Pending
+pre-assignment bytes reserve only the shared memory and queued-byte budgets and
+are released before the selected member reserves its normal live/durable bytes.
+A shared set budget reserves those bytes before a member accepts them, while
+control frames reserve the shared queued-byte budget separately. ACK, reset and
+close paths release memory as soon as its buffer is gone, but release spool
+capacity only after the serialized durable truncate/remove succeeds. Cleanup
+failure retains the spool reservation and fails that member rather than
+advertising capacity that still exists on disk. These process bounds supplement,
+rather than replace, member and stream limits.
+
+Replacement recovery is registered before set startup and addresses the
+original configured gateway ID directly. The set derives the same member
+identity/spool namespace and delegates recovery to that member before any
+physical attempt; round-robin is never involved. An orderly set detach preserves
+active spools while retiring empty warmed lanes. Ordinary new opens cannot name
+a gateway.
+
+Protocol v2 retains the original one-session-per-tunnel architecture for
+rollback. A relay listener selects v2 or v3 explicitly.
+
+### Native reverse gateway
+
+The reverse hub is a rendezvous and stream-ID translation boundary on Server3.
+Authenticated home gateway processes connect outward, register stable IDs, and
+accept hub-created even stream IDs. Authenticated local application members
+select one ID and create odd-ID streams. The hub maps frames without opening
+the target or nesting another Socketduct transport. Each home gateway applies
+the final local allowlist before connecting to `docker-server:2375`.
+
+Gateway registration reconnects with generation fencing. Established reverse
+streams are terminal on transport loss because replaying an ambiguous Docker
+operation would be unsafe. Live-transport buffers remain bounded; durable
+spooling and resume continue to apply to ordinary Socketduct sessions only.
+See [Native reverse gateway](reverse-gateway.md).
+
+### Edge
+
+Runs near the application, usually on the same LAN.
+
+Responsibilities:
+
+- expose local TCP/SOCKS/HTTP proxy endpoints
+- authenticate to the relay
+- create named sessions with a requested target host/port and resume them by identifier
+- keep app-facing sockets open during short WAN interruptions
+- spool unacknowledged data to temporary files
+- replay spooled data after resume
+- apply local limits for buffering, reconnect duration, and memory usage
+
+### Relay/server
+
+Runs on a reachable stable machine/VPS/server.
+
+Responsibilities:
+
+- manage users and password hashes
+- authenticate clients
+- authorize session create/resume operations
+- validate each `session.create.target` against an allowlist policy
+- route stream frames to the target connection owned by that session
+- keep session target sockets alive across short edge-relay tunnel drops when limits allow
+- track session sequence/ack state
+- reject unauthorized session identifiers
+- in v3, authenticate each physical connection once and authorize every
+  `stream.open` target independently
+- clean up expired sessions and spool metadata
+- run across multiple CPU cores
+
+### Worker pool
+
+The server should use Node.js multi-core support from the start.
+
+Recommended MVP shape:
+
+- one primary/control process
+- `N = availableParallelism()` cluster workers
+- each worker accepts connections and owns the sessions assigned to it
+- shared durable state is minimal at first: user store + session metadata + spool files
+
+For the first version, session stickiness can be simple:
+
+- create/resume requests include `session.identifier`
+- primary process or a shared registry maps `identifier -> workerId`
+- if the original worker died, resume can recover from persisted spool/session metadata
+
+## Session identity
+
+Each session has:
+
+- `name`: human-readable label, e.g. `work-ssh`, `obs-stream`, `api-upload`
+- `identifier`: opaque stable id, e.g. UUIDv7 or 128-bit random token
+- `target`: authorized `{host, port}` metadata captured at creation time
+
+The name is for humans. The identifier is the real resume key.
+
+Example session creation:
+
+```json
+{
+  "type": "session.create",
+  "name": "work-ssh",
+  "target": {
+    "host": "10.0.0.15",
+    "port": 22
+  }
+}
+```
+
+Example response:
+
+```json
+{
+  "type": "session.created",
+  "identifier": "018ff4c1-4cd9-7a25-b610-50cf8e10a4e7",
+  "name": "work-ssh"
+}
+```
+
+The relay stores a new session as `creating` and withholds this response until the authorized target TCP connection succeeds. Only pre-response target acquisition is retried; after this response, target failure is terminal and application bytes are never automatically replayed against a replacement target.
+
+## Temporary spool files
+
+Spool files are used to survive brief disconnections and process restarts.
+
+Suggested default location:
+
+```text
+$XDG_RUNTIME_DIR/socketduct/sessions/<identifier>.spool
+```
+
+Fallback:
+
+```text
+/tmp/socketduct-$UID/sessions/<identifier>.spool
+```
+
+Spool file should contain:
+
+- protocol version
+- session identifier
+- user id or subject id
+- last local sequence
+- last remote acknowledgement
+- pending chunks not acknowledged by relay
+- checksum for corruption detection
+
+Spool files are not permanent storage. They should be deleted when:
+
+- the session closes cleanly
+- the session expires
+- the user deletes the session
+- the spool exceeds configured age/size limits
+
+## What Socketduct can and cannot guarantee
+
+Socketduct can hide short interruptions on the WAN leg if:
+
+- the app-facing LAN connection remains open
+- buffer limits are not exceeded
+- reconnect happens before the target/service side times out irrecoverably
+- the protocol being proxied tolerates short stalls
+
+Socketduct cannot perfectly resume every arbitrary upstream TCP connection if the remote target has already closed or invalidated its state. The relay can keep the target-facing connection open during short edge disconnects, but once the target closes, Socketduct can only reconnect at the application/protocol level if that protocol supports it.
+
+## Initial implementation direction
+
+Start with one transport and one proxy mode:
+
+- edge exposes a local TCP forwarder
+- edge includes the desired target in `session.create`
+- relay authorizes the target using deny-by-default policy such as `authorizeTarget({principal, target})`
+- relay connects to and retains target sockets per session identifier, not per relay instance
+- edge-relay protocol uses framed messages over TLS or QUIC-like transport
+- session resume handles edge-relay interruption first
+
+### Transport security model
+
+The relay enforces a fail-closed security posture:
+
+- Non-loopback public binds (any interface other than `127.0.0.0/8`, `::1`, `localhost`) without TLS are refused at startup unless explicitly opted out via `SOCKETDUCT_ALLOW_INSECURE_PUBLIC=1`.
+- Loopback plaintext binds remain compatible for local development and testing.
+- TLS is enabled via `SOCKETDUCT_TLS_CERT_FILE` + `SOCKETDUCT_TLS_KEY_FILE` (atomic pair).
+- Optional mTLS via `SOCKETDUCT_TLS_CLIENT_CA_FILE` enforces client certificate verification.
+- Certificate material is read at runtime; missing or unreadable files cause an immediate startup failure.
+- The client defaults to `rejectUnauthorized: true`.
+
+Later additions:
+
+- SOCKS5 proxy mode
+- HTTP CONNECT mode
+- destination allowlists per user
+- admin API/UI
+- metrics
+- pluggable auth backends

package/docs/docker-topology.md

@@ -0,0 +1,164 @@
+# Docker Topology for Socketduct
+
+> The topology below documents the ordinary edge-to-relay deployment. For the
+> TensorBuzz home-server design where Zyxel and Starlink have no public address,
+> use the [native reverse gateway](reverse-gateway.md): three home members
+> connect outbound to Server3, directly reach `docker-server:2375` on a private
+> Compose network, and require neither inbound forwarding nor a socket shim.
+
+This document describes how Socketduct should be used for Docker traffic from TensorBuzz through `docker-quack`.
+
+## Target topology
+
+```text
+TensorBuzz backend
+  -> docker-quack
+  -> Socketduct connection factory / HTTP agent
+  -> Socketduct relay endpoint for the selected Docker server
+  -> Docker server-local Socketduct container
+  -> Docker daemon TCP/Unix socket
+```
+
+Each Docker server owns its Socketduct endpoint. TensorBuzz chooses a Docker server, reads that server's Socketduct connection settings, and gives `docker-quack` a transport that opens Socketduct sessions to the Docker daemon reachable from that same server.
+
+## Why the relay lives beside Docker
+
+The critical requirement is that Docker traffic returns to the Docker server that originated the gateway container. Running Socketduct on or next to each Docker server keeps the target allowlist local and narrow:
+
+- relay target `docker-api:2375` or another private-network service name when Docker exposes a TCP endpoint reachable from the Socketduct container;
+- relay target `docker-socket-shim:2375` when Docker is only exposed as a Unix socket through a private sidecar shim;
+- relay target `127.0.0.1:2375` only when Socketduct shares the Docker host network namespace;
+- never a broad allowlist for arbitrary network hosts.
+
+The Docker server-local container becomes the stable remote endpoint. TensorBuzz and `docker-quack` connect through Socketduct rather than directly relying on a fragile long-lived TCP path.
+
+## Container modes
+
+### Mode A: TCP target
+
+Use this when the Docker daemon is already reachable from the Socketduct container as TCP through a private Docker network. `127.0.0.1` inside a normal bridge-networked Socketduct container is the Socketduct container itself, not the Docker host, so use a reachable service name or host address here.
+
+```env
+SOCKETDUCT_MODE=relay
+SOCKETDUCT_RELAY_HOST=0.0.0.0
+SOCKETDUCT_RELAY_PORT=3100
+SOCKETDUCT_AUTH_TOKENS=tensorbuzz:replace-with-generated-token
+SOCKETDUCT_TARGET_ALLOWLIST=docker-api:2375
+```
+
+### Mode B: Unix socket target through a local shim
+
+Use this when Docker is only available as `/var/run/docker.sock`.
+
+```text
+Socketduct relay container -> docker-socket-shim:2375 -> /var/run/docker.sock
+```
+
+The shim should be server-local, private, and only reachable from the Socketduct container network. The Socketduct target allowlist still points at exactly one host/port inside that private network.
+
+### Mode C: host network
+
+Use only where acceptable operationally. The relay container can run with host networking and allowlist `127.0.0.1:2375`. This is simple but less portable than an explicit private Compose network.
+
+## TensorBuzz configuration model
+
+TensorBuzz should add optional Socketduct fields to each Docker server configuration while keeping direct Docker connection as a fallback during rollout.
+
+Suggested model fields:
+
+- `socketduct_enabled` — boolean rollout flag.
+- `socketduct_relay_host` — hostname/IP TensorBuzz can reach.
+- `socketduct_relay_port` — relay port, default `3100`.
+- `socketduct_token_secret_ref` — backend secret reference, never raw token in UI logs.
+- `socketduct_target_host` — Docker target as seen from the relay, usually a private service name such as `docker-api` or `docker-socket-shim`; use `127.0.0.1` only for host-network deployments.
+- `socketduct_target_port` — Docker target port, usually `2375`.
+- `socketduct_resume_window_ms` — expected server-side hold window.
+- `socketduct_rollout_mode` — `off`, `shadow_probe`, `preferred`, or `required`.
+
+Rollout modes:
+
+- `off`: existing direct Docker transport only.
+- `shadow_probe`: direct Docker remains primary; background `_ping` probes Socketduct and records health.
+- `preferred`: try Socketduct first and fall back to direct Docker for safe idempotent operations.
+- `required`: use Socketduct only; direct fallback disabled.
+
+## docker-quack integration model
+
+`docker-quack` should not depend directly on TensorBuzz. It should gain a transport abstraction that can be passed to `SnapReq`/HTTP plumbing:
+
+```js
+const connection = new DockerConnection({
+  transport: createSocketductDockerTransport({
+    relay: {host, port, token},
+    target: {host: "docker-api", port: 2375}
+  })
+})
+```
+
+The transport should provide either:
+
+- an `http.Agent` compatible object; or
+- a `createConnection(options, callback)` hook accepted by the underlying HTTP client.
+
+The preferred shape is an agent because Docker API keep-alive, streaming responses, and hijack-style upgraded streams are all socket lifecycle problems that fit Node's HTTP client model.
+
+## Security requirements
+
+- Authenticate every Socketduct tunnel before session create/resume.
+- Keep target allowlists per Docker server and deny-by-default.
+- Store bearer tokens in TensorBuzz backend secrets, not code, changelogs, PR bodies, or task comments.
+- Redact token values in logs and error reports.
+- Treat Docker API traffic as privileged root-equivalent traffic.
+- Prefer private networks/firewalls around relay ports; add TLS or private overlay networking before internet exposure.
+- Keep fallback direct Docker credentials independent from Socketduct rollout flags so rollback is quick.
+
+## Observability
+
+TensorBuzz should record enough metadata to debug rollout without leaking secrets:
+
+- selected Docker server id/name;
+- transport mode: `direct` or `socketduct`;
+- Socketduct session id prefix/hash, not full identifier if considered sensitive;
+- connection phase: auth, session_create, http_request, stream, resume;
+- retry/resume counts and durations;
+- Docker endpoint path/method, not request body payload;
+- error code (`AUTH_FAILED`, `TARGET_UNAUTHORIZED`, `SESSION_EXPIRED`, `ECONNRESET`, etc.).
+
+## Test strategy
+
+### Socketduct repo
+
+- Unit tests for socket duck typing and state transitions.
+- Integration tests for HTTP over `SocketductHttpAgent` and strict HTTPS/mTLS over `SocketductHttpsAgent` against fake Docker API servers.
+- Process-level e2e tests with relay startup and allowed/denied target policies.
+- Streaming and upgrade-style tests that keep a `Duplex` alive after headers.
+
+### docker-quack repo
+
+- Transport abstraction tests with a fake socket factory/agent.
+- Buffered request tests (`/_ping`, `/version`).
+- Streaming response tests (`/events`, logs-style responses).
+- Hijack/upgrade conformance tests using a fake Docker daemon.
+
+### TensorBuzz repo
+
+- Configuration parsing/model tests for Socketduct rollout modes.
+- Build container handler tests that choose direct vs Socketduct transport.
+- Gateway lifecycle/provisioning tests for Docker server setup.
+- Cross-repo smoke test using local Socketduct relay + fake Docker server + docker-quack.
+
+## Rollback
+
+Every rollout step must be reversible:
+
+1. Set Docker server rollout mode to `off`.
+2. Keep direct Docker settings intact.
+3. Stop/remove the Socketduct sidecar after traffic drains.
+4. Rotate the Socketduct token if it may have been exposed during debugging.
+
+## Open decisions for implementation tasks
+
+- Whether Socketduct should ship its own Unix socket shim or document a small external shim first.
+- Whether TensorBuzz stores raw encrypted token values or references an existing secret store path.
+- Whether `docker-quack` should expose `agent`, `createConnection`, or both.
+- Whether Socketduct session ids should be logged as prefixes or stable hashes.