socketduct 0.1.15

package/docs/plans/socket-docker-integration.md

@@ -0,0 +1,221 @@
+# Socket API and Docker Topology Implementation Plan
+
+> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task.
+
+**Goal:** Add a Node socket-compatible Socketduct client API and carry it through docker-quack into TensorBuzz's Docker server workflow.
+
+**Architecture:** Extract the existing per-connection logic from `startTcpEdge()` into a reusable `Duplex`-compatible client socket, layer a connection factory and HTTP agent on top, then adapt docker-quack and TensorBuzz to opt into that transport per Docker server. Keep relay target authorization deny-by-default and keep direct Docker fallback available during rollout.
+
+**Tech Stack:** Node.js ES modules, normal JavaScript with JSDoc, Node `Duplex`/`http.Agent`, built-in `node:test`, existing Socketduct framed TCP protocol, docker-quack, TensorBuzz.
+
+## Cross-repository scope
+
+Tasks 1-6 happen in this Socketduct repository. Tasks 7-9 happen in `/opt/data/projects/docker-quack`. Tasks 10-13 happen in `/opt/data/projects/tensorbuzz` with local Socketduct and docker-quack checkouts used by the smoke harness.
+
+---
+
+## Task 1: Implement `SocketductSocket` skeleton
+
+**Objective:** Create a socket-shaped `Duplex` that owns one Socketduct relay session.
+
+**Files:**
+- Create: `src/client/socketduct-socket.js`
+- Create: `test/unit/client/socketduct-socket.test.js`
+- Modify: `package.json` exports if needed
+
+**Steps:**
+1. Write tests asserting the object is a `Duplex`, exposes common socket methods/properties, has a public `.connect()` method that returns `this`, and no-op compatibility methods return `this`.
+2. Implement constructor state, `.connect()`, `setTimeout`, `setNoDelay`, `setKeepAlive`, `address`, `ref`, `unref`, `_read`, `_write`, `_final`, and `_destroy` skeletons.
+3. Verify with `node --test test/unit/client/socketduct-socket.test.js`, `npm run lint`, `npm run typecheck`.
+
+**Commit:** `feat: add Socketduct socket client skeleton`
+
+## Task 2: Move edge connection protocol into `SocketductSocket`
+
+**Objective:** Make `SocketductSocket` create/resume sessions and stream bytes without a local TCP listener.
+
+**Files:**
+- Modify: `src/client/socketduct-socket.js`
+- Modify: `src/edge/tcp-edge.js`
+- Create/modify: `test/integration/socketduct-socket-forwarding.test.js`
+
+**Steps:**
+1. Write integration test using fake target + `startTcpRelay()` + `SocketductSocket` directly.
+2. Port session create/resume, frame handling, spooling, sequence, ack, and reconnect logic from `EdgeConnection`.
+3. Refactor `startTcpEdge()` to create a `SocketductSocket` per app socket and pipe both directions.
+4. Verify existing edge integration tests still pass.
+
+**Commit:** `feat: route edge traffic through SocketductSocket`
+
+## Task 3: Add `connectSocketduct()` and connection factory
+
+**Objective:** Provide ergonomic public APIs that mirror `net.connect()` and `createConnection` hooks.
+
+**Files:**
+- Create: `src/client/connect.js`
+- Modify: package exports / `src/index.js` as appropriate
+- Create: `test/unit/client/connect.test.js`
+
+**Steps:**
+1. Test `connectSocketduct(options)` returns a `SocketductSocket` immediately and calls `.connect()` asynchronously.
+2. Test `createSocketductConnectionFactory(baseOptions)` merges per-call target/session options.
+3. Document that relay token is required and never logged.
+
+**Commit:** `feat: add Socketduct connection factory`
+
+## Task 4: Add `SocketductHttpAgent`
+
+**Objective:** Let Node HTTP clients route requests over Socketduct sockets.
+
+**Files:**
+- Create: `src/client/http-agent.js`
+- Create: `test/integration/socketduct-http-agent.test.js`
+
+**Steps:**
+1. Test `http.request({agent})` can call a fake Docker-style `/_ping` endpoint through relay.
+2. Test keep-alive reuses a live Socketduct socket when the HTTP agent keeps it open.
+3. Test unauthorized targets fail as HTTP connection errors.
+4. Verify with full test suite.
+
+**Commit:** `feat: add HTTP agent for Socketduct sockets`
+
+## Task 5: Harden socket reconnection semantics
+
+**Objective:** Make reconnect behavior predictable for socket-style consumers.
+
+**Files:**
+- Modify: `src/client/socketduct-socket.js`
+- Modify/create: reconnect tests under `test/integration/`
+- Modify docs: `docs/socket-api.md`, `docs/session-lifecycle.md`
+
+**Steps:**
+1. Add tests for relay tunnel drop with target retained and socket staying open.
+2. Add tests for resume expiry causing `error` then `close`.
+3. Add max interrupted duration and spool size guards.
+4. Add backpressure-focused tests for large writes.
+
+**Commit:** `feat: harden Socketduct socket reconnect behavior`
+
+## Task 6: Add Docker-host gateway container mode
+
+**Objective:** Support a documented server-local Socketduct relay container for Docker servers.
+
+**Files:**
+- Modify: `docker-compose.yml`, `.env.example`, `docs/docker.md`
+- Create optional docs/scripts for Docker socket shim if chosen
+- Add runtime config tests if env shape changes
+
+**Steps:**
+1. Add Compose example for Docker target allowlist.
+2. Document TCP target, private shim target, and host-network target options.
+3. Ensure placeholders remain rejected and secrets remain ignored.
+4. Validate with process-level tests when Docker is unavailable.
+
+**Commit:** `feat: document Docker server gateway mode`
+
+## Task 7: Add docker-quack transport abstraction
+
+**Objective:** Let docker-quack use custom agents/socket factories without knowing about TensorBuzz.
+
+**Files:**
+- In `docker-quack`: modify `src/docker-connection.js` and exports
+- Add tests under `spec/`
+
+**Steps:**
+1. Add `agent` and/or `createConnection` options to `DockerConnection`.
+2. Pass the transport to SnapReq/HTTP plumbing or adapt SnapReq if needed.
+3. Test buffered requests against a fake HTTP server using a custom transport.
+
+**Commit:** `feat: support custom Docker transports`
+
+## Task 8: Wire docker-quack to Socketduct transport
+
+**Objective:** Provide a docker-quack helper or documented integration that builds a Socketduct-backed Docker connection.
+
+**Files:**
+- In `docker-quack`: add helper module or docs
+- Add integration tests with local Socketduct relay and fake Docker API
+
+**Steps:**
+1. Use `SocketductHttpAgent` or `createSocketductConnectionFactory` as docker-quack transport.
+2. Test `/_ping` and `/version` over Socketduct.
+3. Keep Socketduct optional so docker-quack still works standalone.
+
+**Commit:** `feat: add Socketduct Docker transport support`
+
+## Task 9: Add Docker streaming and hijack conformance tests
+
+**Objective:** Prove Docker streaming endpoints survive the Socketduct transport layer.
+
+**Files:**
+- In `docker-quack`: streaming/hijack tests
+- In `socketduct`: HTTP agent upgrade/stream tests if missing
+
+**Steps:**
+1. Test chunked logs/events-style streaming.
+2. Test upgraded bidirectional attach/exec-style stream against a fake daemon.
+3. Verify no buffering layer consumes the stream prematurely.
+
+**Commit:** `test: cover Docker streaming over Socketduct`
+
+## Task 10: Add TensorBuzz Socketduct configuration model
+
+**Objective:** Let each Docker server choose direct or Socketduct Docker transport.
+
+**Files:**
+- In `tensorbuzz`: Docker server model/config files
+- Tests for validation and redaction
+
+**Steps:**
+1. Add rollout mode fields and secret references.
+2. Validate target host/port and relay host/port.
+3. Redact tokens/secrets in serializers, logs, and errors.
+
+**Commit:** `feat: add Socketduct Docker server configuration`
+
+## Task 11: Add TensorBuzz gateway lifecycle/provisioning
+
+**Objective:** Provision or document the Docker-server-local Socketduct gateway container.
+
+**Files:**
+- In `tensorbuzz`: services/scripts/admin flows touching Docker server setup
+- Tests for generated Compose/env config without real secrets
+
+**Steps:**
+1. Generate or document server-local gateway settings.
+2. Add safe lifecycle commands/status probes.
+3. Keep final destructive operations confirmation-gated where UI/admin flows require it.
+
+**Commit:** `feat: add Socketduct gateway lifecycle support`
+
+## Task 12: Add TensorBuzz observability and fallback behavior
+
+**Objective:** Make rollout safe and debuggable.
+
+**Files:**
+- In `tensorbuzz`: build container handler / Docker connection factory / logging
+- Tests for direct, shadow, preferred, and required modes
+
+**Steps:**
+1. Add direct fallback for safe operations in `preferred` mode.
+2. Add health probes and structured telemetry fields.
+3. Ensure logs never include tokens or raw Docker payloads.
+
+**Commit:** `feat: add Socketduct Docker rollout controls`
+
+## Task 13: Add cross-repo e2e smoke test
+
+**Objective:** Prove TensorBuzz + docker-quack + Socketduct work together against a fake Docker daemon.
+
+**Files:**
+- In `tensorbuzz` or CI scripts: cross-repo smoke harness
+- Docs describing how to run locally
+
+**Steps:**
+1. Start fake Docker API server.
+2. Start Socketduct relay with target allowlist to the fake daemon.
+3. Configure docker-quack through Socketduct.
+4. Run TensorBuzz build-server probe path.
+5. Verify green CI and document local Docker limitations.
+
+**Commit:** `test: add Socketduct Docker cross-repo smoke coverage`