socketduct 0.1.15
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.dockerignore +11 -0
- package/.env.example +131 -0
- package/AGENTS.md +168 -0
- package/Dockerfile +30 -0
- package/README.md +134 -0
- package/changelog.d/20260716083144-v2-ack-wire-format.md +3 -0
- package/changelog.d/20260716092044-v2-only-stream-framing.md +1 -0
- package/changelog.d/20260727113000-relay-connect-timeout-destroy.md +1 -0
- package/changelog.d/20260813093000-private-relay-hardening.md +1 -0
- package/changelog.d/20260824183412-initial-target-recovery-https.md +1 -0
- package/changelog.d/20260824230000-tls-relay-session-close.md +1 -0
- package/changelog.d/20260825120000-reconnect-close-target-timeout.md +1 -0
- package/changelog.d/20260829120000-protocol-v3-multiplexing.md +1 -0
- package/changelog.d/20260829160000-v3-review-corrections.md +1 -0
- package/changelog.d/20260830000000-fix-socket-idle-timeout.md +1 -0
- package/changelog.d/20260830090000-multi-gateway-transport-set.md +1 -0
- package/changelog.d/20260830093000-relay-close-settlement.md +1 -0
- package/changelog.d/20260830120000-immutable-gateway-image.md +1 -0
- package/changelog.d/20260902210000-native-reverse-gateway.md +1 -0
- package/changelog.d/20260902230000-relay-resume-terminal-order.md +1 -0
- package/changelog.d/20260902230708-multiplexed-open-ownership.md +1 -0
- package/changelog.d/20260902231541-multiplexed-terminal-close.md +1 -0
- package/changelog.d/20260902231542-reverse-close-position.md +1 -0
- package/changelog.d/20260902231543-reverse-terminal-connect.md +1 -0
- package/changelog.d/20260902233000-reverse-gateway-bounds.md +1 -0
- package/changelog.d/20260903001000-reverse-data-ordering.md +1 -0
- package/changelog.d/20260903003000-reverse-hub-restart-window.md +1 -0
- package/changelog.d/20260903010000-reverse-terminal-lifecycle.md +1 -0
- package/changelog.d/20260904142743-transport-set-readiness-wait.md +1 -0
- package/changelog.d/20260904152855-deferred-open-snapshots-binary-views.md +1 -0
- package/changelog.d/20260904163041-tensorbuzz-container-workflow.md +1 -0
- package/changelog.d/20260904174955-transport-set-terminal-causes.md +1 -0
- package/changelog.d/20260905081000-contain-reverse-auth-write-errors.md +1 -0
- package/changelog.d/20260905100520-reverse-hub-close-race.md +1 -0
- package/changelog.d/20260905120000-persistent-jsonl-logging.md +1 -0
- package/changelog.d/20260908071009-jsonl-partial-append-recovery.md +1 -0
- package/changelog.d/20260908074916-private-log-directory-setup.md +1 -0
- package/changelog.d/20260909093057-reverse-large-write-backpressure.md +1 -0
- package/changelog.d/20260909135830-actionable-error-diagnostics.md +1 -0
- package/changelog.d/20260909150000-transport-set-error-redaction.md +1 -0
- package/docker-compose.dev.yml +19 -0
- package/docker-compose.reverse-gateway.yml +36 -0
- package/docker-compose.yml +73 -0
- package/docs/architecture.md +264 -0
- package/docs/docker-topology.md +164 -0
- package/docs/docker.md +281 -0
- package/docs/logging.md +185 -0
- package/docs/plans/initial-implementation.md +205 -0
- package/docs/plans/socket-docker-integration.md +221 -0
- package/docs/protocol.md +558 -0
- package/docs/releasing.md +18 -0
- package/docs/reverse-gateway.md +186 -0
- package/docs/runbooks/jsonl-integrity-rollout.md +144 -0
- package/docs/running.md +215 -0
- package/docs/security.md +258 -0
- package/docs/session-lifecycle.md +127 -0
- package/docs/socket-api.md +448 -0
- package/eslint.config.js +36 -0
- package/package.json +44 -0
- package/scripts/deploy-gateway.js +194 -0
- package/scripts/test.js +28 -0
- package/src/auth/passwords.js +38 -0
- package/src/auth/static-token-store.js +41 -0
- package/src/auth/user-store.js +139 -0
- package/src/client/connect.js +76 -0
- package/src/client/deferred-virtual-socket.js +312 -0
- package/src/client/http-agent.js +101 -0
- package/src/client/https-agent.js +366 -0
- package/src/client/multiplexed-connection.js +1504 -0
- package/src/client/multiplexed-transport-set.js +917 -0
- package/src/client/multiplexed.js +3 -0
- package/src/client/socketduct-socket.js +1174 -0
- package/src/client/virtual-socket.js +256 -0
- package/src/config/environment.js +400 -0
- package/src/edge/tcp-edge.js +82 -0
- package/src/index.js +101 -0
- package/src/logging/structured-logger.js +745 -0
- package/src/protocol/compression.js +325 -0
- package/src/protocol/frame.js +341 -0
- package/src/protocol/v3-compression.js +44 -0
- package/src/protocol/v3-frame.js +209 -0
- package/src/proxy/multiplexed-relay.js +798 -0
- package/src/proxy/tcp-relay.js +1191 -0
- package/src/relay.js +3 -0
- package/src/reverse/client.js +721 -0
- package/src/reverse/common.js +88 -0
- package/src/reverse/gateway.js +663 -0
- package/src/reverse/hub.js +563 -0
- package/src/reverse/index.js +3 -0
- package/src/reverse/runtime.js +78 -0
- package/src/server/control-handler.js +170 -0
- package/src/server/primary.js +64 -0
- package/src/server/relay-runtime.js +124 -0
- package/src/server/worker.js +39 -0
- package/src/sessions/session.js +212 -0
- package/src/spool/spool-file.js +451 -0
- package/tensorbuzz.yml +36 -0
- package/test/e2e/gateway-process.test.js +262 -0
- package/test/e2e/reverse-gateway-process.test.js +285 -0
- package/test/fixtures/logging/enospc-matrix.mjs +238 -0
- package/test/fixtures/logging/enospc-recovery.mjs +146 -0
- package/test/fixtures/logging/process-writer.mjs +20 -0
- package/test/helpers/tls-certificates.js +115 -0
- package/test/integration/initial-target-recovery.test.js +518 -0
- package/test/integration/logging-recovery.test.js +163 -0
- package/test/integration/multiplexed-compatibility.test.js +254 -0
- package/test/integration/multiplexed-corrections.test.js +551 -0
- package/test/integration/multiplexed-failures.test.js +339 -0
- package/test/integration/multiplexed-transport-set.test.js +456 -0
- package/test/integration/multiplexed-transport.test.js +208 -0
- package/test/integration/reverse-gateway.test.js +1293 -0
- package/test/integration/socketduct-http-agent.test.js +435 -0
- package/test/integration/socketduct-https-agent.test.js +669 -0
- package/test/integration/stream-compression.test.js +1267 -0
- package/test/integration/tcp-forwarding.test.js +2878 -0
- package/test/integration/tls-forwarding.test.js +748 -0
- package/test/unit/auth/auth.test.js +64 -0
- package/test/unit/auth/static-token-store.test.js +14 -0
- package/test/unit/client/connect.test.js +58 -0
- package/test/unit/client/multiplexed-spool-budget.test.js +309 -0
- package/test/unit/client/multiplexed-transport-set.test.js +906 -0
- package/test/unit/client/socketduct-socket.test.js +1124 -0
- package/test/unit/client/virtual-socket.test.js +98 -0
- package/test/unit/config/docker-gateway-files.test.js +274 -0
- package/test/unit/config/environment.test.js +507 -0
- package/test/unit/index.test.js +78 -0
- package/test/unit/logging/structured-logger.test.js +674 -0
- package/test/unit/protocol/compression.test.js +421 -0
- package/test/unit/protocol/frame.test.js +336 -0
- package/test/unit/protocol/v3-compression.test.js +31 -0
- package/test/unit/protocol/v3-frame.test.js +57 -0
- package/test/unit/release-configuration.test.js +21 -0
- package/test/unit/server/control-handler.test.js +291 -0
- package/test/unit/server/primary.test.js +59 -0
- package/test/unit/server/relay-runtime.test.js +434 -0
- package/test/unit/sessions/session.test.js +84 -0
- package/test/unit/spool/spool-file.test.js +295 -0
- package/tsconfig.json +20 -0
package/docs/docker.md
ADDED
|
@@ -0,0 +1,281 @@
|
|
|
1
|
+
# Docker gateway setup
|
|
2
|
+
|
|
3
|
+
This setup runs the Socketduct relay on a Docker server so edges can create authenticated, resumable sessions to targets reachable from that server. In Docker-host gateway mode the common target is the local Docker Engine API.
|
|
4
|
+
|
|
5
|
+
For the native outbound home topology—especially Zyxel and Starlink paths
|
|
6
|
+
behind CGNAT—use [Native reverse gateway](reverse-gateway.md) and
|
|
7
|
+
[`docker-compose.reverse-gateway.yml`](../docker-compose.reverse-gateway.yml).
|
|
8
|
+
That mode uses no socket shim and publishes no Docker or Socketduct port at
|
|
9
|
+
home: the gateway connects outbound to Server3 and directly to
|
|
10
|
+
`docker-server:2375` on the shared private control network.
|
|
11
|
+
|
|
12
|
+
## 1. Prepare the server
|
|
13
|
+
|
|
14
|
+
Install these host prerequisites before cloning the versioned deployment
|
|
15
|
+
contract:
|
|
16
|
+
|
|
17
|
+
- Docker Engine with Docker Compose v2.
|
|
18
|
+
- Node.js 24 or newer, including npm. Install it from your operating-system
|
|
19
|
+
vendor's supported packages or the official Node.js distribution; the
|
|
20
|
+
repository's `gateway:deploy` command runs on the host and does not run inside
|
|
21
|
+
Docker.
|
|
22
|
+
|
|
23
|
+
Verify the tools before continuing:
|
|
24
|
+
|
|
25
|
+
```sh
|
|
26
|
+
docker compose version
|
|
27
|
+
node --version
|
|
28
|
+
npm --version
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
`node --version` must report v24 or newer. Then clone the repository and copy
|
|
32
|
+
the example environment:
|
|
33
|
+
|
|
34
|
+
```sh
|
|
35
|
+
git clone https://github.com/kaspernj/socketduct.git
|
|
36
|
+
cd socketduct
|
|
37
|
+
cp .env.example .env
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
The production relay image is private in GHCR unless its package visibility has
|
|
41
|
+
been made public. Before deployment, authenticate Docker to `ghcr.io` using a
|
|
42
|
+
GitHub account or token that has package read access:
|
|
43
|
+
|
|
44
|
+
```sh
|
|
45
|
+
docker login ghcr.io
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Supply credentials only to Docker's login prompt or an external credential
|
|
49
|
+
helper. Do not put GHCR credentials in this repository, `.env`, Compose files,
|
|
50
|
+
shell history, or deployment logs.
|
|
51
|
+
|
|
52
|
+
Generate a bearer token for the edge:
|
|
53
|
+
|
|
54
|
+
```sh
|
|
55
|
+
openssl rand -base64 48 | tr '+/' '-_' | tr -d '=\n'
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Edit `.env`:
|
|
59
|
+
|
|
60
|
+
```sh
|
|
61
|
+
SOCKETDUCT_IMAGE=ghcr.io/kaspernj/socketduct@sha256:<64-lowercase-hex>
|
|
62
|
+
SOCKETDUCT_MODE=relay
|
|
63
|
+
SOCKETDUCT_RELAY_HOST=0.0.0.0
|
|
64
|
+
SOCKETDUCT_RELAY_PORT=3100
|
|
65
|
+
SOCKETDUCT_PROTOCOL_VERSION=3
|
|
66
|
+
SOCKETDUCT_PUBLISH_HOST=192.0.2.10
|
|
67
|
+
SOCKETDUCT_RESUME_WINDOW_MS=30000
|
|
68
|
+
SOCKETDUCT_AUTH_TOKENS=edge-user:replace-with-generated-token
|
|
69
|
+
SOCKETDUCT_TARGET_ALLOWLIST=docker-socket-shim:2375
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Replace `<64-lowercase-hex>` with the published digest for the exact Socketduct
|
|
73
|
+
image to deploy. `SOCKETDUCT_IMAGE` accepts only the untagged
|
|
74
|
+
`ghcr.io/kaspernj/socketduct@sha256:<digest>` form. Mutable `latest`, branch, and
|
|
75
|
+
commit-SHA tags are rejected before Docker runs. Keep `.env` private.
|
|
76
|
+
`SOCKETDUCT_AUTH_TOKENS` accepts comma-separated `username:token` entries.
|
|
77
|
+
`SOCKETDUCT_TARGET_ALLOWLIST` accepts comma-separated `host:port` entries. The
|
|
78
|
+
relay rejects any requested session target that is not on the allowlist. If the
|
|
79
|
+
relay listens on a non-loopback interface, configure the built-in TLS settings
|
|
80
|
+
in [TLS for Docker deployments](#tls-for-docker-deployments) before starting it;
|
|
81
|
+
public plaintext requires the explicit emergency opt-out documented there.
|
|
82
|
+
|
|
83
|
+
Compose publishes on `127.0.0.1` by default. Set `SOCKETDUCT_PUBLISH_HOST` explicitly to the private host interface that trusted edges use; avoid `0.0.0.0` unless the firewall policy and exposure are deliberate. The relay container runs as UID/GID 1000 with a read-only root filesystem, dropped capabilities, and `no-new-privileges`, so mounted TLS files must be readable by UID 1000 without becoming group/world writable.
|
|
84
|
+
|
|
85
|
+
The committed `.env.example` token is intentionally rejected by the runtime; replace it before starting the relay.
|
|
86
|
+
|
|
87
|
+
## 2. Choose the Docker target shape
|
|
88
|
+
|
|
89
|
+
### Docker socket shim, preferred for Docker-host gateway mode
|
|
90
|
+
|
|
91
|
+
The Compose file includes an optional `docker-socket-shim` service behind the `docker-host` profile. It mounts `/var/run/docker.sock` and exposes TCP port `2375` only on the private Compose network. The host does **not** publish `2375`, so Docker API access still has to pass through Socketduct authentication and the target allowlist.
|
|
92
|
+
|
|
93
|
+
Use this allowlist entry:
|
|
94
|
+
|
|
95
|
+
```sh
|
|
96
|
+
SOCKETDUCT_TARGET_ALLOWLIST=docker-socket-shim:2375
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
Deploy with the profile enabled as described below. The profile does not publish
|
|
100
|
+
the shim port on the host.
|
|
101
|
+
|
|
102
|
+
```sh
|
|
103
|
+
npm run gateway:deploy -- --docker-host
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
### Existing Docker TCP endpoint
|
|
107
|
+
|
|
108
|
+
If the Docker server already has a secured Docker TCP endpoint, allowlist that host and port instead:
|
|
109
|
+
|
|
110
|
+
```sh
|
|
111
|
+
SOCKETDUCT_TARGET_ALLOWLIST=docker.example.internal:2376
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
Use the hostname as resolved from the relay container, not from the edge machine.
|
|
115
|
+
|
|
116
|
+
### Host-network target
|
|
117
|
+
|
|
118
|
+
If you run Socketduct with host networking outside the provided Compose file, the target can be a host-local address such as `127.0.0.1:2375`. Do not use this with the default Compose network expecting container loopback to mean the Docker host; `127.0.0.1` inside the relay container is the relay container itself.
|
|
119
|
+
|
|
120
|
+
## 3. Start the relay
|
|
121
|
+
|
|
122
|
+
For a normal remote relay without the Docker socket shim:
|
|
123
|
+
|
|
124
|
+
```sh
|
|
125
|
+
npm run gateway:deploy
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
For Docker-host gateway mode:
|
|
129
|
+
|
|
130
|
+
```sh
|
|
131
|
+
npm run gateway:deploy -- --docker-host
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
The deployment command validates the image setting first, fixes the production
|
|
135
|
+
Compose file, `.env`, project directory/name, and optional profile explicitly,
|
|
136
|
+
then validates the rendered Compose JSON. It neutralizes ambient Compose model and
|
|
137
|
+
profile selectors while preserving Docker connection settings such as
|
|
138
|
+
`DOCKER_HOST` and `DOCKER_CONTEXT`. The rendered `socketduct-relay` image must
|
|
139
|
+
exactly equal the validated digest and must not contain a build source before the
|
|
140
|
+
command runs `docker compose pull`, then runs
|
|
141
|
+
`docker compose up -d --wait --no-build`. It cannot build from the checkout or
|
|
142
|
+
silently fall back to a local image. A missing or malformed image setting stops
|
|
143
|
+
the command before Docker is invoked. Automatic Compose override files and
|
|
144
|
+
ambient `COMPOSE_FILE`, `COMPOSE_ENV_FILES`, and `COMPOSE_PROFILES` values are
|
|
145
|
+
not part of this production path. Compose also rejects a missing setting if it
|
|
146
|
+
is invoked directly.
|
|
147
|
+
|
|
148
|
+
Check status and logs:
|
|
149
|
+
|
|
150
|
+
```sh
|
|
151
|
+
COMPOSE_PROFILES= docker compose --file docker-compose.yml --env-file .env --project-directory . --project-name socketduct-gateway ps
|
|
152
|
+
COMPOSE_PROFILES= docker compose --file docker-compose.yml --env-file .env --project-directory . --project-name socketduct-gateway logs -f socketduct-relay
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
For persistent structured logs, create a host directory writable by the
|
|
156
|
+
container's uid/gid 1000 and enable the opt-in sink in `.env`:
|
|
157
|
+
|
|
158
|
+
```sh
|
|
159
|
+
sudo install -d -o 1000 -g 1000 -m 0700 /srv/socketduct/logs
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
```text
|
|
163
|
+
SOCKETDUCT_LOG_HOST_DIRECTORY=/srv/socketduct/logs
|
|
164
|
+
SOCKETDUCT_LOG_DIRECTORY=/var/log/socketduct
|
|
165
|
+
SOCKETDUCT_LOG_MAX_FILE_BYTES=10485760
|
|
166
|
+
SOCKETDUCT_LOG_MAX_FILES=5
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
The production relay and reverse-gateway Compose files mount the host directory
|
|
170
|
+
at `/var/log/socketduct`. Existing container stdout/stderr remains enabled. See
|
|
171
|
+
[Persistent structured logging](logging.md) for file names, rotation, embedded
|
|
172
|
+
client configuration, and the redaction contract.
|
|
173
|
+
|
|
174
|
+
Expected startup log (formatted here for readability; the actual line is JSON):
|
|
175
|
+
|
|
176
|
+
```text
|
|
177
|
+
{"event":"relay.started","host":"0.0.0.0","port":3100,"transportSecurity":"TLS","compression":"auto"}
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
## 4. Connect an edge
|
|
181
|
+
|
|
182
|
+
Configure the local edge with:
|
|
183
|
+
|
|
184
|
+
- relay host: the remote server hostname or IP
|
|
185
|
+
- relay port: `SOCKETDUCT_RELAY_PORT`
|
|
186
|
+
- token: the generated token from `.env`
|
|
187
|
+
- target: one of the `SOCKETDUCT_TARGET_ALLOWLIST` entries
|
|
188
|
+
|
|
189
|
+
For Docker-host gateway mode the target should be:
|
|
190
|
+
|
|
191
|
+
```text
|
|
192
|
+
docker-socket-shim:2375
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
The target host is resolved from the relay container, not from the local edge machine. For TensorBuzz this means the allowlisted hostname must be reachable from the remote gateway server/container.
|
|
196
|
+
|
|
197
|
+
## 5. Update and rollback
|
|
198
|
+
|
|
199
|
+
To update, change only `SOCKETDUCT_IMAGE` in `.env` to the digest of the intended
|
|
200
|
+
new image, then re-apply with `npm run gateway:deploy` (or add
|
|
201
|
+
`-- --docker-host`). Verify the selected digest and service health before
|
|
202
|
+
discarding the previous digest.
|
|
203
|
+
|
|
204
|
+
For deterministic rollback, change the digest back to the previously verified
|
|
205
|
+
value and run the same deployment command again. The pull plus
|
|
206
|
+
`up -d --wait --no-build` sequence applies that exact image without rebuilding
|
|
207
|
+
or resolving a mutable tag.
|
|
208
|
+
|
|
209
|
+
## 6. Stop
|
|
210
|
+
|
|
211
|
+
```sh
|
|
212
|
+
COMPOSE_PROFILES= COMPOSE_REMOVE_ORPHANS=false docker compose --file docker-compose.yml --env-file .env --project-directory . --project-name socketduct-gateway --profile docker-host down
|
|
213
|
+
```
|
|
214
|
+
|
|
215
|
+
Add `-v` to that explicit `down` command only if you intentionally want to
|
|
216
|
+
remove named Socketduct state volumes.
|
|
217
|
+
|
|
218
|
+
## Security notes
|
|
219
|
+
|
|
220
|
+
- Do not commit real `.env` files or tokens.
|
|
221
|
+
- The Docker socket shim has Docker-host-equivalent privileges. Keep it private to the Compose network and expose only the Socketduct relay port.
|
|
222
|
+
- Expose the relay port only to trusted edge hosts where possible, for example with a firewall/security group.
|
|
223
|
+
- Put the relay behind TLS or a private network before production exposure.
|
|
224
|
+
- Keep `SOCKETDUCT_TARGET_ALLOWLIST` tight. This is the open-proxy guardrail.
|
|
225
|
+
|
|
226
|
+
## Source-independent development validation
|
|
227
|
+
|
|
228
|
+
The production Compose file is image-only and contains no `build` directive.
|
|
229
|
+
Local source builds remain explicit in `docker-compose.dev.yml`; development
|
|
230
|
+
gates run as UID 1000 from source copied into that separate image, without host
|
|
231
|
+
source mounts:
|
|
232
|
+
|
|
233
|
+
```sh
|
|
234
|
+
docker compose -f docker-compose.dev.yml build socketduct-dev
|
|
235
|
+
docker compose -f docker-compose.dev.yml run --rm socketduct-dev npm test
|
|
236
|
+
docker compose -f docker-compose.dev.yml run --rm socketduct-dev npm run typecheck
|
|
237
|
+
docker compose -f docker-compose.dev.yml run --rm socketduct-dev npm run lint
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
## TensorBuzz container workflow
|
|
241
|
+
|
|
242
|
+
TensorBuzz is the sole repository CI and private-container publication workflow
|
|
243
|
+
owner. Every build uses the checked-in `tensorbuzz.yml` to run the package gates
|
|
244
|
+
and build the Dockerfile against an isolated Docker-in-Docker service. The
|
|
245
|
+
validation image is local to that ephemeral service and is never pushed by a
|
|
246
|
+
pull-request build.
|
|
247
|
+
|
|
248
|
+
Private GHCR publication is currently disabled. Enabling it safely requires
|
|
249
|
+
TensorBuzz to provide trusted default-branch, tag, and manual-trigger metadata
|
|
250
|
+
to build commands, plus an established secret-backed Socketduct project input
|
|
251
|
+
containing a private-GHCR write credential. Ordinary TensorBuzz project
|
|
252
|
+
environment variables are readable project configuration and must not hold that
|
|
253
|
+
credential. Until both capabilities are configured, publication must remain
|
|
254
|
+
paused; repository CI must not infer trust from the checked-out commit or
|
|
255
|
+
expose registry credentials in source or logs.
|
|
256
|
+
|
|
257
|
+
## TLS for Docker deployments
|
|
258
|
+
|
|
259
|
+
To run the relay with TLS in Docker:
|
|
260
|
+
|
|
261
|
+
1. Place your PEM-encoded certificate (`relay.crt`) and private key (`relay.key`) in a `certs/` directory next to `docker-compose.yml`.
|
|
262
|
+
2. Set the following in `.env`:
|
|
263
|
+
```sh
|
|
264
|
+
SOCKETDUCT_TLS_CERT_FILE=/certs/relay.crt
|
|
265
|
+
SOCKETDUCT_TLS_KEY_FILE=/certs/relay.key
|
|
266
|
+
```
|
|
267
|
+
3. Optionally add mTLS:
|
|
268
|
+
```sh
|
|
269
|
+
SOCKETDUCT_TLS_CLIENT_CA_FILE=/certs/client-ca.crt
|
|
270
|
+
```
|
|
271
|
+
4. The Compose file automatically mounts `./certs` (or `SOCKETDUCT_TLS_CERT_DIR`) as `/certs:ro` in the container.
|
|
272
|
+
5. Restart:
|
|
273
|
+
```sh
|
|
274
|
+
npm run gateway:deploy
|
|
275
|
+
```
|
|
276
|
+
6. Expected startup log with TLS (the actual log is one JSON line):
|
|
277
|
+
```text
|
|
278
|
+
{"event":"relay.started","host":"0.0.0.0","port":3100,"transportSecurity":"TLS","compression":"auto"}
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
To run without TLS on a non-loopback interface (not recommended for production), set `SOCKETDUCT_ALLOW_INSECURE_PUBLIC=1` in `.env`. The relay will refuse to start on `0.0.0.0` without this explicit opt-out.
|
package/docs/logging.md
ADDED
|
@@ -0,0 +1,185 @@
|
|
|
1
|
+
# Persistent structured logging
|
|
2
|
+
|
|
3
|
+
Socketduct can write lifecycle diagnostics as newline-delimited JSON (JSONL) to
|
|
4
|
+
a directory mounted from persistent host storage. File logging is opt-in.
|
|
5
|
+
Existing startup/status output on stdout and errors on stderr remain unchanged
|
|
6
|
+
when file logging is enabled.
|
|
7
|
+
|
|
8
|
+
## Standalone runtime configuration
|
|
9
|
+
|
|
10
|
+
Set `SOCKETDUCT_LOG_DIRECTORY` for a `relay`, `reverse-hub`, or
|
|
11
|
+
`reverse-gateway` process:
|
|
12
|
+
|
|
13
|
+
```sh
|
|
14
|
+
SOCKETDUCT_LOG_DIRECTORY=/var/log/socketduct
|
|
15
|
+
SOCKETDUCT_LOG_MAX_FILE_BYTES=10485760
|
|
16
|
+
SOCKETDUCT_LOG_MAX_FILES=5
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
`SOCKETDUCT_LOG_MAX_FILE_BYTES` defaults to 10 MiB and must be at least 1024.
|
|
20
|
+
`SOCKETDUCT_LOG_MAX_FILES` defaults to 5, includes the active file, and must be
|
|
21
|
+
between 1 and 100. The size/count settings are ignored unless a non-empty log
|
|
22
|
+
directory enables the sink.
|
|
23
|
+
|
|
24
|
+
Each process writes its own `<role>-<pid>.jsonl` file, with older files renamed
|
|
25
|
+
to `.1`, `.2`, and so on. A writer never rotates another live process's active
|
|
26
|
+
file, which avoids cross-process rename races. Each file in the current
|
|
27
|
+
process/role family is bounded by the configured byte limit and the family
|
|
28
|
+
retains at most the configured file count. A reused PID continues the same
|
|
29
|
+
bounded filename family.
|
|
30
|
+
|
|
31
|
+
On its first write, a logger also caps files left by definitely exited
|
|
32
|
+
same-role process instances to the configured file count in total, retaining
|
|
33
|
+
the newest files. Repeated signal-0 PID probes must report `ESRCH`, and the file
|
|
34
|
+
must remain unchanged, before it can be deleted. A live/reused PID, permission
|
|
35
|
+
failure, unsupported probe, changed file, or other indeterminate result
|
|
36
|
+
preserves every file in that family. Files are created with mode `0600`; a newly
|
|
37
|
+
created log directory uses mode `0700`.
|
|
38
|
+
|
|
39
|
+
For Docker, bind mount a persistent host directory and point Socketduct at the
|
|
40
|
+
container path:
|
|
41
|
+
|
|
42
|
+
```yaml
|
|
43
|
+
services:
|
|
44
|
+
socketduct:
|
|
45
|
+
environment:
|
|
46
|
+
SOCKETDUCT_LOG_DIRECTORY: /var/log/socketduct
|
|
47
|
+
SOCKETDUCT_LOG_MAX_FILE_BYTES: "10485760"
|
|
48
|
+
SOCKETDUCT_LOG_MAX_FILES: "5"
|
|
49
|
+
volumes:
|
|
50
|
+
- /srv/socketduct/logs:/var/log/socketduct
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Create `/srv/socketduct/logs` with mode `0700` and ownership that permits the
|
|
54
|
+
container user to write it; for uid/gid 1000, use
|
|
55
|
+
`sudo install -d -o 1000 -g 1000 -m 0700 /srv/socketduct/logs`.
|
|
56
|
+
A permission, disk, or rotation error does not change tunnel,
|
|
57
|
+
session, target, or reconnect behavior. Socketduct reports only
|
|
58
|
+
`Socketduct logger failure (LOGGER_IO_ERROR)` to stderr, at most once per
|
|
59
|
+
logger, without including the path or underlying error text.
|
|
60
|
+
|
|
61
|
+
## Interrupted writes and retained evidence
|
|
62
|
+
|
|
63
|
+
Before appending, the file sink checks the last byte of a nonempty active file.
|
|
64
|
+
This bounded check also runs for a fresh logger or restarted process reusing
|
|
65
|
+
the same filename; it does not scan or parse historical records. A failed
|
|
66
|
+
append may have written a prefix even when the filesystem reports an error.
|
|
67
|
+
|
|
68
|
+
If the active file does not end in a newline, Socketduct moves the entire file
|
|
69
|
+
byte-for-byte into the existing numbered retention family and writes subsequent
|
|
70
|
+
events to a clean active file. The retained segment may still contain an
|
|
71
|
+
incomplete final record: it is incident evidence, not repaired JSONL. Normal
|
|
72
|
+
bounded retention eventually ages it out, so preserve incident evidence outside
|
|
73
|
+
the managed family before rollout or forensic work. New complete records must
|
|
74
|
+
parse independently; ignoring malformed historical lines is not proof of recovery.
|
|
75
|
+
|
|
76
|
+
If tail inspection, rotation, clean-file creation, or mode verification fails,
|
|
77
|
+
that logging call writes no new record. Later ordinary calls recheck the
|
|
78
|
+
filesystem; there is no background retry loop or replay of missed diagnostics.
|
|
79
|
+
If an append itself fails, it may leave another incomplete prefix that the next
|
|
80
|
+
call must separate. Logging failures still do not interrupt tunnel traffic.
|
|
81
|
+
|
|
82
|
+
With `maxFiles: 1`, there is no retained slot: an incomplete active file is left
|
|
83
|
+
untouched and file logging pauses. An incomplete active file already larger
|
|
84
|
+
than a newly lowered size limit also pauses logging **before startup pruning**;
|
|
85
|
+
it is not deleted to satisfy that limit. These pre-existing bytes can remain
|
|
86
|
+
outside the newly configured bound, but Socketduct will not grow them. An
|
|
87
|
+
operator must preserve the evidence separately and provide a clean path before
|
|
88
|
+
logging can continue. Socketduct never truncates or rewrites damaged records.
|
|
89
|
+
Already newline-terminated malformed historical records are not repaired.
|
|
90
|
+
|
|
91
|
+
The sink requires a regular active file with mode `0600` and a directory with
|
|
92
|
+
mode `0700`, including on reused paths. It checks the newly created file's
|
|
93
|
+
actual mode before writing and does not follow an active-file symlink. Fix
|
|
94
|
+
ownership/modes on an operator-preserved path rather than making logs public.
|
|
95
|
+
|
|
96
|
+
See [the private rollout runbook](runbooks/jsonl-integrity-rollout.md) for the
|
|
97
|
+
isolated fault commands and evidence-preserving Apoc-only rollout gates.
|
|
98
|
+
|
|
99
|
+
The checked-in production relay and reverse-gateway Compose files already mount
|
|
100
|
+
`${SOCKETDUCT_LOG_HOST_DIRECTORY:-./logs}` at `/var/log/socketduct`. Enable the
|
|
101
|
+
sink in `.env` after creating the host directory for the container's uid/gid
|
|
102
|
+
1000:
|
|
103
|
+
|
|
104
|
+
```sh
|
|
105
|
+
SOCKETDUCT_LOG_HOST_DIRECTORY=/srv/socketduct/logs
|
|
106
|
+
SOCKETDUCT_LOG_DIRECTORY=/var/log/socketduct
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
`SOCKETDUCT_STATUS_INTERVAL_MS` controls both console and file-backed periodic
|
|
110
|
+
status events. Its default remains 30000 milliseconds; `0` disables periodic
|
|
111
|
+
status only, not lifecycle logging.
|
|
112
|
+
|
|
113
|
+
## Embedded client configuration
|
|
114
|
+
|
|
115
|
+
Create one file logger per process/role and pass it to the client, agent, or
|
|
116
|
+
transport set through its `logger` option:
|
|
117
|
+
|
|
118
|
+
```js
|
|
119
|
+
import {createJsonlFileLogger} from "socketduct"
|
|
120
|
+
import {MultiplexedSocketductTransportSet} from "socketduct/multiplexed"
|
|
121
|
+
|
|
122
|
+
const logger = createJsonlFileLogger({
|
|
123
|
+
directory: "/var/log/socketduct",
|
|
124
|
+
role: "tensorbuzz-client",
|
|
125
|
+
maxFileBytes: 10 * 1024 * 1024,
|
|
126
|
+
maxFiles: 5
|
|
127
|
+
})
|
|
128
|
+
|
|
129
|
+
const transports = new MultiplexedSocketductTransportSet({
|
|
130
|
+
connectionId: "runner_01_stable_identity",
|
|
131
|
+
spoolDirectory: "/tmp/socketduct-v3-set",
|
|
132
|
+
gateways,
|
|
133
|
+
logger
|
|
134
|
+
})
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
The same `logger` option is supported by `MultiplexedSocketductConnection`,
|
|
138
|
+
`ReverseGatewayConnection`, `OutboundReverseGateway`, `connectSocketduct`,
|
|
139
|
+
`createSocketductConnectionFactory`, `SocketductHttpAgent`, and
|
|
140
|
+
`SocketductHttpsAgent`. A transport set owns logger propagation to its members;
|
|
141
|
+
do not put `logger` inside `memberOptions`.
|
|
142
|
+
|
|
143
|
+
For an existing logging system, use the exported line-writer adapter:
|
|
144
|
+
|
|
145
|
+
```js
|
|
146
|
+
import {createStructuredLogger} from "socketduct"
|
|
147
|
+
|
|
148
|
+
const logger = createStructuredLogger({
|
|
149
|
+
role: "tensorbuzz-client",
|
|
150
|
+
writeLine(line) {
|
|
151
|
+
destination.append(line)
|
|
152
|
+
}
|
|
153
|
+
})
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
Alternatively, pass an object with `log(event, fields)` directly. Socketduct
|
|
157
|
+
calls it only with a known event and already-sanitized fields. A direct custom
|
|
158
|
+
logger is responsible for adding its own timestamp/process/role envelope and
|
|
159
|
+
for preserving one-object-per-line framing. Synchronous throws and rejected
|
|
160
|
+
promises from custom loggers/writers are isolated from connection behavior.
|
|
161
|
+
|
|
162
|
+
## Event and data contract
|
|
163
|
+
|
|
164
|
+
The built-in logger emits one JSON object per line, capped at 1024 bytes. Every
|
|
165
|
+
entry contains an ISO-8601 `timestamp`, an allowlisted `event`, numeric
|
|
166
|
+
`processId`, and validated `role`. Depending on the component, events cover:
|
|
167
|
+
|
|
168
|
+
- process startup and periodic status counters
|
|
169
|
+
- physical transport connect, disconnect, retry scheduling, recovery, and
|
|
170
|
+
terminal failure
|
|
171
|
+
- transport-set readiness, member state/generation replacement, and terminal
|
|
172
|
+
loss
|
|
173
|
+
- reverse-gateway target open success, open failure, and close
|
|
174
|
+
- virtual-stream opening, opened, closed, and error transitions
|
|
175
|
+
|
|
176
|
+
Only finite allowlisted booleans, non-negative integer counters, validated
|
|
177
|
+
gateway IDs, states, and error classifications can be included. Unknown events
|
|
178
|
+
are dropped. Unknown error codes become `UNKNOWN`. Errors supplied through
|
|
179
|
+
`errorDiagnostics()` retain their name, message, and stack so terminal
|
|
180
|
+
transport and virtual-stream failures are actionable.
|
|
181
|
+
|
|
182
|
+
The diagnostic path never serializes protocol payloads or arbitrary caller
|
|
183
|
+
fields. It serializes error messages and stack traces produced by
|
|
184
|
+
`errorDiagnostics()` after redacting bearer credentials, URL userinfo, and
|
|
185
|
+
password/token/authorization values.
|
|
@@ -0,0 +1,205 @@
|
|
|
1
|
+
# Initial Implementation Plan
|
|
2
|
+
|
|
3
|
+
> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task.
|
|
4
|
+
|
|
5
|
+
**Goal:** Build the first Node.js foundation for Socketduct: authenticated session create/resume, framed protocol primitives, temporary spool files, and a multi-core server skeleton.
|
|
6
|
+
|
|
7
|
+
**Architecture:** Start with a small standalone Node.js codebase. Use a primary process plus `node:cluster` workers for multi-core connection handling. Keep protocol/session/spool logic isolated and unit-tested before adding real proxy streaming.
|
|
8
|
+
|
|
9
|
+
**Tech Stack:** Node.js, TypeScript preferred, built-in `node:net`, `node:cluster`, `node:fs`, `node:stream`, test runner TBD (`node:test` is fine for MVP).
|
|
10
|
+
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
## Task 1: Initialize Node.js project
|
|
14
|
+
|
|
15
|
+
**Objective:** Add the minimal package structure for a Node.js service.
|
|
16
|
+
|
|
17
|
+
**Files:**
|
|
18
|
+
|
|
19
|
+
- Create: `package.json`
|
|
20
|
+
- Create: `src/index.ts` or `src/index.js`
|
|
21
|
+
- Create: `test/`
|
|
22
|
+
|
|
23
|
+
**Steps:**
|
|
24
|
+
|
|
25
|
+
1. Choose TypeScript or plain Node.js. Prefer TypeScript unless setup friction slows the MVP.
|
|
26
|
+
2. Add scripts:
|
|
27
|
+
- `test`
|
|
28
|
+
- `lint` once linting exists
|
|
29
|
+
- `start`
|
|
30
|
+
3. Add a trivial startup entrypoint.
|
|
31
|
+
4. Run `npm test` and verify it passes.
|
|
32
|
+
5. Commit: `chore: initialize node project`
|
|
33
|
+
|
|
34
|
+
## Task 2: Add framed protocol encoder/decoder
|
|
35
|
+
|
|
36
|
+
**Objective:** Encode/decode length-prefixed protocol frames.
|
|
37
|
+
|
|
38
|
+
**Files:**
|
|
39
|
+
|
|
40
|
+
- Create: `src/protocol/frame.*`
|
|
41
|
+
- Create: `test/unit/protocol/frame.test.*`
|
|
42
|
+
|
|
43
|
+
**Acceptance criteria:**
|
|
44
|
+
|
|
45
|
+
- Encodes JSON header + binary payload.
|
|
46
|
+
- Decodes frames split across multiple TCP chunks.
|
|
47
|
+
- Rejects invalid header length and oversized payload.
|
|
48
|
+
- Preserves payload bytes exactly.
|
|
49
|
+
|
|
50
|
+
**Commit:** `feat: add protocol frame codec`
|
|
51
|
+
|
|
52
|
+
## Task 3: Add session model and state transitions
|
|
53
|
+
|
|
54
|
+
**Objective:** Represent sessions with name, identifier, owner, sequence positions, and state.
|
|
55
|
+
|
|
56
|
+
**Files:**
|
|
57
|
+
|
|
58
|
+
- Create: `src/sessions/session.*`
|
|
59
|
+
- Create: `src/sessions/session-store.*`
|
|
60
|
+
- Create: `test/unit/sessions/session.test.*`
|
|
61
|
+
|
|
62
|
+
**Acceptance criteria:**
|
|
63
|
+
|
|
64
|
+
- Create session with `name` and generated `identifier`.
|
|
65
|
+
- Resume lookup by `identifier`.
|
|
66
|
+
- Enforce legal state transitions.
|
|
67
|
+
- Reject resume for missing/expired session.
|
|
68
|
+
|
|
69
|
+
**Commit:** `feat: add session lifecycle model`
|
|
70
|
+
|
|
71
|
+
## Task 4: Add local user authentication
|
|
72
|
+
|
|
73
|
+
**Objective:** Add local users with password hashing and login validation.
|
|
74
|
+
|
|
75
|
+
**Files:**
|
|
76
|
+
|
|
77
|
+
- Create: `src/auth/user-store.*`
|
|
78
|
+
- Create: `src/auth/passwords.*`
|
|
79
|
+
- Create: `src/auth/tokens.*`
|
|
80
|
+
- Create: `test/unit/auth/*.test.*`
|
|
81
|
+
|
|
82
|
+
**Acceptance criteria:**
|
|
83
|
+
|
|
84
|
+
- Passwords are hashed, never stored plaintext.
|
|
85
|
+
- Login returns an opaque token.
|
|
86
|
+
- Invalid username/password fails.
|
|
87
|
+
- Token maps back to user until expiry/revocation.
|
|
88
|
+
|
|
89
|
+
**Commit:** `feat: add local authentication primitives`
|
|
90
|
+
|
|
91
|
+
## Task 5: Add spool file writer/reader
|
|
92
|
+
|
|
93
|
+
**Objective:** Persist interrupted session data to temp files and read it back for resume.
|
|
94
|
+
|
|
95
|
+
**Files:**
|
|
96
|
+
|
|
97
|
+
- Create: `src/spool/spool-file.*`
|
|
98
|
+
- Create: `test/unit/spool/spool-file.test.*`
|
|
99
|
+
|
|
100
|
+
**Acceptance criteria:**
|
|
101
|
+
|
|
102
|
+
- Creates user-private temp spool directory.
|
|
103
|
+
- Writes metadata and chunks.
|
|
104
|
+
- Reads only chunks after a requested sequence.
|
|
105
|
+
- Truncates/removes acked chunks.
|
|
106
|
+
- Rejects path traversal or mismatched session identifiers.
|
|
107
|
+
|
|
108
|
+
**Commit:** `feat: add session spool files`
|
|
109
|
+
|
|
110
|
+
## Task 6: Add multi-core server skeleton
|
|
111
|
+
|
|
112
|
+
**Objective:** Start a primary process and cluster workers.
|
|
113
|
+
|
|
114
|
+
**Files:**
|
|
115
|
+
|
|
116
|
+
- Create: `src/server/primary.*`
|
|
117
|
+
- Create: `src/server/worker.*`
|
|
118
|
+
- Modify: `src/index.*`
|
|
119
|
+
|
|
120
|
+
**Acceptance criteria:**
|
|
121
|
+
|
|
122
|
+
- Primary starts one worker per available CPU by default.
|
|
123
|
+
- Worker count can be configured.
|
|
124
|
+
- Worker crash is logged and replaced.
|
|
125
|
+
- Single-worker mode is available for tests/dev.
|
|
126
|
+
|
|
127
|
+
**Commit:** `feat: add clustered server skeleton`
|
|
128
|
+
|
|
129
|
+
## Task 7: Add authenticated session create/resume messages
|
|
130
|
+
|
|
131
|
+
**Objective:** Wire auth + session store + protocol messages without real proxying yet.
|
|
132
|
+
|
|
133
|
+
**Files:**
|
|
134
|
+
|
|
135
|
+
- Create: `src/server/control-handler.*`
|
|
136
|
+
- Create: `test/integration/session-create-resume.test.*`
|
|
137
|
+
|
|
138
|
+
**Acceptance criteria:**
|
|
139
|
+
|
|
140
|
+
- Unauthenticated `session.create` is rejected.
|
|
141
|
+
- Authenticated `session.create` returns identifier.
|
|
142
|
+
- Authenticated `session.resume` by owner succeeds.
|
|
143
|
+
- Different user cannot resume the session.
|
|
144
|
+
- Unknown identifier returns machine-readable error.
|
|
145
|
+
|
|
146
|
+
**Commit:** `feat: add authenticated session control flow`
|
|
147
|
+
|
|
148
|
+
## Task 8: Add first TCP forwarding prototype
|
|
149
|
+
|
|
150
|
+
**Objective:** Forward bytes from a local edge connection through relay to target using framed stream data.
|
|
151
|
+
|
|
152
|
+
**Files:**
|
|
153
|
+
|
|
154
|
+
- Create: `src/edge/*`
|
|
155
|
+
- Create: `src/proxy/*`
|
|
156
|
+
- Create: `test/integration/tcp-forwarding.test.*`
|
|
157
|
+
|
|
158
|
+
**Acceptance criteria:**
|
|
159
|
+
|
|
160
|
+
- Local app connects to edge port.
|
|
161
|
+
- Relay connects to configured target port.
|
|
162
|
+
- Bytes flow both ways.
|
|
163
|
+
- Sequence numbers and acknowledgements are emitted.
|
|
164
|
+
|
|
165
|
+
**Commit:** `feat: add tcp forwarding prototype`
|
|
166
|
+
|
|
167
|
+
## Task 9: Add interruption/resume integration test
|
|
168
|
+
|
|
169
|
+
**Objective:** Prove the core Socketduct behavior in a controlled test.
|
|
170
|
+
|
|
171
|
+
**Files:**
|
|
172
|
+
|
|
173
|
+
- Create: `test/integration/interruption-resume.test.*`
|
|
174
|
+
|
|
175
|
+
**Acceptance criteria:**
|
|
176
|
+
|
|
177
|
+
- Start app socket -> edge -> relay -> fake target.
|
|
178
|
+
- Send bytes.
|
|
179
|
+
- Simulate edge-relay connection drop.
|
|
180
|
+
- Edge writes spool file.
|
|
181
|
+
- Reconnect and resume by identifier.
|
|
182
|
+
- Edge sends spool data.
|
|
183
|
+
- Fake target receives the stream exactly once and in order.
|
|
184
|
+
|
|
185
|
+
**Commit:** `test: cover interrupted session resume`
|
|
186
|
+
|
|
187
|
+
## Task 10: Add docs for running and security limits
|
|
188
|
+
|
|
189
|
+
**Objective:** Document local dev usage and known limitations.
|
|
190
|
+
|
|
191
|
+
**Files:**
|
|
192
|
+
|
|
193
|
+
- Modify: `README.md`
|
|
194
|
+
- Create: `docs/running.md`
|
|
195
|
+
- Modify: `docs/security.md`
|
|
196
|
+
|
|
197
|
+
**Acceptance criteria:**
|
|
198
|
+
|
|
199
|
+
- Local dev command documented.
|
|
200
|
+
- Example user creation documented.
|
|
201
|
+
- Example edge/relay run documented.
|
|
202
|
+
- Open proxy prevention documented.
|
|
203
|
+
- Known non-guarantees documented.
|
|
204
|
+
|
|
205
|
+
**Commit:** `docs: add running guide`
|