@twinsearth/w2m-dsh-plugin 0.0.0-stage → 0.4.7

package/PROTOCOL-v0.3.9.md

@@ -0,0 +1,207 @@
+# W2M Protocol — v0.3.9: P2P hole punching
+
+> Status: **the transport is implemented and tested; the task-path integration is
+> partial.** Read §7 before relying on it. Protocol version stays **1** — v0.3.9
+> adds endpoints and an out-of-band channel, and removes nothing.
+
+## 0. Why
+
+v1 is a star: every task offer and every result crosses the relay. That is what
+makes it work through NAT with no inbound port and no `sshd`, and it is also the
+design's two real costs — the relay carries all payload traffic, and it is a single
+point of failure for every byte.
+
+v0.3.9 adds a direct path. Both machines send to each other's reflexive address at
+the same time, which opens both NAT mappings, and the payload then travels
+peer-to-peer. The relay keeps doing what only it can do: rendezvous, the task
+ledger, and the fallback.
+
+## 1. Path selection
+
+| `p2p.mode` | Behaviour |
+|---|---|
+| `auto` (**default**) | Discover, announce, try to punch. On success use the direct path. On failure use the relay and **say so**. |
+| `direct` | Require the direct path. If the punch fails, the task fails with `P2P_UNAVAILABLE`. No silent fallback. |
+| `relay` | Never attempt a direct path. Byte-for-byte the v0.3.8 behaviour. |
+
+`auto` is the default because punching **cannot** succeed against a symmetric NAT.
+That is not a theoretical caveat: measured on the authoring machine behind an iPhone
+hotspot, three STUN servers reported three different mapped ports for the same
+socket, i.e. endpoint-dependent mapping, so no punch from that network can work.
+A default that hard-required the direct path would break every deployment on such a
+network the moment it was upgraded.
+
+### 1.1 Path reporting is mandatory, not decorative
+
+Every offer and every result carries the path it actually took:
+
+```json
+{ "transport": "p2p",    "p2p": { "rtt_ms": 14.2, "peer": "203.0.113.7:51234" } }
+{ "transport": "relay",  "p2p": { "reason": "P2P_PUNCH_TIMEOUT", "attempts": 8 } }
+```
+
+A fallback that is not reported is indistinguishable from a direct path that is
+slow, and the operator then debugs the wrong thing. `w2m_wait` surfaces the field
+per machine.
+
+## 2. Discovery — STUN (RFC 5389)
+
+Each machine, holding the same UDP socket it will later punch from:
+
+1. sends a Binding Request to every configured STUN server in turn;
+2. reads `XOR-MAPPED-ADDRESS` from the Binding Success Response;
+3. classifies **mapping behaviour** by comparing the mapped port across servers:
+
+| Observation | `mapping` | Punchable |
+|---|---|---|
+| server reports the socket's own address and port | `none` | yes (no NAT) |
+| every server agrees on one port | `endpoint-independent` | yes |
+| different port per server | `endpoint-dependent` | **no** — this is a symmetric NAT |
+| fewer than two answers | `unknown` | attempted anyway |
+
+`unknown` is attempted on purpose: one unreachable STUN server must not be the
+reason a direct path is never tried. The punch is the test.
+
+**Only mapping behaviour is detected.** Filtering behaviour (address-restricted vs
+port-restricted cone) would need a second host to probe from, and claiming to
+detect what was not probed is the failure mode this project refuses elsewhere.
+**IPv4 only** in v0.3.9.
+
+## 3. Signalling — two new endpoints
+
+### 3.1 `POST /v1/peer/announce` — device token
+
+```json
+{ "candidates": [ { "address": "203.0.113.7", "port": 51234 },
+                  { "address": "192.168.1.20", "port": 51234 } ],
+  "nat": { "mapping": "endpoint-independent" },
+  "ttl_ms": 60000 }
+```
+
+* The entry is keyed by the **authenticated** `device.machine_id`. A `machine_id`
+  in the body is ignored: otherwise any paired machine could aim a peer's punch at
+  a third party.
+* `address` must be dotted-quad IPv4 (no leading zeros, no hostnames, no IPv6);
+  `port` must be 1–65535. At most **16** candidates.
+* `ttl_ms` is clamped to 1000 … 600000; the default is 60000.
+
+### 3.2 `GET /v1/peer/{machine_id}` — device token
+
+`200` with `{ peer: { machine_id, candidates, nat, announced_at, expires_at } }`,
+or `404 NOT_FOUND` when the machine has never announced or its announcement expired.
+A **named 404 rather than an empty list**: "nobody has announced" and "announced
+nothing usable" require different responses from the caller, and an empty list would
+send it into a punch that cannot work.
+
+### 3.3 Announcements are ephemeral and never persisted
+
+A candidate *is* a NAT mapping, and the mapping dies with the process that held it.
+Persisting candidates would survive a restart and then hand out addresses that cannot
+work. `/healthz` reports the live count as `peers_announced`.
+
+## 4. Punching
+
+Both peers send a `HELLO` to every one of the other's candidates every 250 ms until
+one gets through (default timeout 5 s). The peer that receives a `HELLO` replies
+`HELLO_ACK` and considers the path open — the arrival of a datagram is the entire
+question the punch was asking. A peer that was never told about the initiator can
+still answer, which is what lets the relay broker a punch with no extra round trip.
+
+### 4.1 The session id
+
+Every frame carries a 32-bit session id, so frames from an earlier punch are ignored
+rather than corrupting a new one. Two peers that each invented their own id would
+each discard the other's `HELLO` and time out while both insist they were sending —
+which looks exactly like a NAT problem. So:
+
+* both peers MUST pass the same `session`, normally from `deriveSession(...)` over
+  inputs both already share (the task id and the two machine ids);
+* a peer that has no session may pass `allowSessionAdoption: true` and adopt the
+  initiator's id, echoing it back;
+* `punch()` **throws** if neither is given, rather than generating one and failing
+  later with a misleading diagnosis.
+
+## 5. Channel framing
+
+```
+u8  kind
+u32 session
+... body
+```
+
+| kind | id | body |
+|---|---|---|
+| `HELLO` | 1 | — |
+| `HELLO_ACK` | 2 | — |
+| `DATA` | 3 | `u32 messageId, u32 fragmentIndex, u32 fragmentTotal, payload` |
+| `ACK` | 4 | `u32 messageId, u32 fragmentIndex` (selective, per fragment) |
+| `DONE` | 5 | `u32 messageId` |
+| `PING` | 6 | — |
+| `PONG` | 7 | — (the answer to `PING`, and never a `PING`) |
+| `BYE` | 8 | — |
+
+* Payload is at most **1160 bytes** per fragment, keeping a frame inside 1200 so it
+  passes a 1280-byte IPv6 minimum MTU and a 1500-byte Ethernet MTU with room for a
+  WireGuard tunnel.
+* Every fragment is acknowledged and retransmitted (RTO 300 ms, doubling to 3 s,
+  at most 8 attempts, window 32). A message is delivered only when all fragments
+  have arrived.
+* Messages sent **one at a time** arrive in order; concurrent `send()` calls may
+  complete in either order.
+* Largest message: 8 MiB (a single stdout sample is capped at 2 MiB upstream).
+* `PING` is answered with `PONG`. Answering with `PING` makes two peers bounce
+  datagrams between them forever, and the loop looks exactly like healthy keep-alive.
+
+## 6. Security
+
+* Signalling requires a **device token**; the two new routes are covered by the
+  existing `v0.3.0` request-signing layer when it is enabled.
+* A machine can only announce for itself (§3.1).
+* Candidate addresses are validated (IPv4 shape, port range, count ceiling) because
+  this is the one place where a peer supplies an address that another peer will send
+  UDP datagrams to.
+* **The P2P channel is not authenticated.** Session ids are random 32-bit values and
+  UDP source addresses are spoofable. Do not treat a direct path as more trustworthy
+  than the relay path: correlate payloads with the task they belong to, and enable
+  request signing if the threat model includes an on-path attacker. This is stated
+  rather than fixed because fixing it properly needs the `v0.3.0` signing secret to
+  be available on both machines, which is a separate design decision.
+
+## 7. What is verified, and what is not
+
+**Verified on macOS 27.0 / arm64, Node 24.21.0:**
+
+| Item | Evidence |
+|---|---|
+| STUN against real public servers | Cloudflare, Google and Nextcloud all answered; `XOR-MAPPED-ADDRESS` decoded correctly |
+| NAT mapping classification | measured `endpoint-dependent` on an iPhone hotspot (three servers, three ports) |
+| Wire format, including a frozen captured response | `test/p2p-stun.test.mjs` |
+| Punch, fragmentation, selective ACK, retransmit of an injected loss, timeout reporting, close semantics | `test/p2p-transport.test.mjs`, real UDP sockets |
+| Both signalling endpoints, validation, 404 semantics, self-only announce, non-persistence | `test/p2p-signaling.test.mjs` |
+
+**NOT verified — and cannot be, on one machine:**
+
+| Claim | Why |
+|---|---|
+| A punch across a **real** NAT | needs two hosts behind two *different* NATs. Two sockets on one host share a loopback path with no translator between them: "the punch succeeded" says nothing about NAT traversal. |
+| Punching through a cone NAT | as above |
+| The direct path under packet loss / reordering on a real WAN | loopback loss was injected deterministically; a real path was not exercised |
+| IPv6 | not implemented |
+| Filtering behaviour (restricted cone) | needs a second host to probe from |
+
+## 8. Integration status
+
+Implemented and tested: STUN discovery and mapping classification, punching, the
+reliable channel, and the two signalling endpoints.
+
+**Not yet wired:** the `p2p.mode` configuration surface (its semantics are defined
+in §1 and default to `auto`, but nothing reads the setting yet), the Localside
+announcer, and direct-path preference in `w2m_run`. Until those land, the task path
+behaves exactly as v0.3.8 and the signalling layer is live but idle — a caller that
+tries `transport: "p2p"` gets the relay path, because there is no `transport` field
+being produced at all.
+
+This is stated here, in `CHANGELOG.md` and in the README rather than left for a
+reader to discover from a `transport` field that never appears. The one thing this
+section exists to prevent is a deployment that believes it is running over a direct
+path when it is not.

package/PROTOCOL-v0.4.0.md

@@ -0,0 +1,255 @@
+# W2M Protocol — v0.4.0: the direct path, on by default
+
+> Status: **implemented and wired into the task path.** v0.3.9 shipped the P2P transport and
+> admitted in §8 that nothing read `p2p.mode` and the Localside announcer did not exist. This
+> release is that admission being paid off: the announcer exists, the offer can arrive over a
+> punched path, the result can return over it, and every message says which path it took.
+> Protocol version stays **1** — this adds endpoints and fields and removes nothing. Read §8
+> for what is measured, and §9 for what is still not.
+
+## 0. Why the default changed
+
+v0.3.9 documented the intent and shipped none of it. A deployment could set `p2p.mode: auto`,
+see no error, and get relay-only behaviour — the worst outcome the feature can produce,
+because an operator then debugs a NAT that was never used.
+
+v0.4.0 makes the direct path the **default connection method between machines** and makes the
+relay what the design always said it was: the rendezvous, the ledger, and the fallback.
+
+## 1. Path selection
+
+| `p2p.mode` | Dispatcher (the machine running `w2m_run`) | Executor (Localside) |
+|---|---|---|
+| `auto` (**default**) | Announce, punch to each target, push the offer over the direct channel. The relay's own offer is still emitted, so a failed punch costs latency rather than the task. | Accept an offer from **either** path. The first copy wins; a later copy of the same `task_id`+`attempt` is never executed again. |
+| `direct` | Same, but a failed punch is not hidden: the executor refuses the relay copy. | **Refuse** an offer that did not arrive over the direct path: `status: refused`, `refusal_reason: P2P_UNAVAILABLE`. No silent fallback. |
+| `relay` | Never dials. No UDP socket is bound at all. | v0.3.9 behaviour, byte for byte. |
+
+Why `auto` and not `direct`: punching **cannot** succeed behind a symmetric NAT, and that is
+not theoretical. Measured twice now, on two different networks, with three STUN servers
+returning three different mapped ports for one socket:
+
+* the authoring machine behind an iPhone hotspot (`endpoint-dependent`, v0.3.9);
+* the reference Windows client behind China Mobile (`endpoint-dependent`, v0.4.0) — three
+  servers, three ports: 36028, 5855, 26713.
+
+A default that hard-required the direct path would break every deployment on such a network
+the moment it was upgraded. `direct` is there for an operator who has measured their own path
+and wants a failure to be loud.
+
+### 1.1 The table is per machine, and it is the executor's own mode that refuses
+
+`direct` is enforced where the refusal can be honest: the executor knows how the offer
+arrived, so it is the executor that refuses a relay offer. The dispatcher's mode decides
+whether it bothers to punch.
+
+## 2. The shared server
+
+A machine behind NAT cannot discover its own public address, and two such machines cannot find
+each other. One always-on host provides both, and its address is now the default:
+
+| Constant | Value | Used for |
+|---|---|---|
+| `SHARED_SERVER.rabbitUrl` | `http://202.182.123.154:8787` | The relay, when `rabbitUrl` is not configured |
+| `SHARED_SERVER.stun` | `202.182.123.154:3478` | First entry of the default STUN list |
+
+Precedence, highest first: `rabbitUrl` in the profile patch → `W2M_RABBIT_URL` →
+the shared default. Which one was used is reported by `w2m_status` as `rabbit_source`
+(`config` | `env` | `shared-default`), so "it works" and "it works because someone set it"
+are distinguishable.
+
+The public STUN servers (Cloudflare, Google, Nextcloud) stay in the list after the shared one.
+That is deliberate: the shared responder is one machine, and a punch that is never attempted
+because that one machine is down would be a self-inflicted outage.
+
+## 3. Announcement — the announcer v0.3.9 did not have
+
+Each machine, holding the same UDP socket it will punch from:
+
+1. binds an ephemeral UDP port (`0.0.0.0:0`);
+2. asks the STUN servers for its reflexive address and classifies mapping behaviour;
+3. announces its candidates (`POST /v1/peer/announce`, device token):
+   the **reflexive address first**, then its private address, capped at 8 and de-duplicated;
+4. re-announces every 20 s with a 60 s TTL, on an `unref()`ed timer, so the entry stays live
+   without keeping a process alive.
+
+A STUN server that does not answer is **not** fatal: the machine announces its private
+candidate, records `announce_ok: false` and the error, and a LAN punch can still work. This
+is the same refusal-to-give-up that makes `unknown` punchable in v0.3.9 §2.
+
+Announcements remain ephemeral and are still never persisted — a candidate is a NAT mapping,
+and a mapping dies with the process that held it.
+
+## 4. The offer over the direct path
+
+The dispatcher, after `POST /v1/task` returns `{task_id, seq, leases[]}`:
+
+1. reads the target's lease (`index`, `attempt`, `dedupe_key`) — the pushed offer must be
+   recognisably the *same* work item the relay is offering, or exactly-once becomes
+   at-least-twice;
+2. `GET /v1/peer/{machine_id}` for the target's live candidates;
+3. punches (`HELLO` to every candidate, 250 ms apart, 5 s default budget);
+4. on success, sends `{type: 'task.offer', ...}` — the same JSON object the relay would have
+   sent over SSE, with `origin_machine_id` and `p2p` added — over the channel;
+5. on failure, does nothing further. The relay's offer is already on its way.
+
+### 4.1 The relay holds its own offer back, briefly, or the direct path could never win
+
+An SSE offer is a push over a connection that is **already open**; a punch costs at least one
+round trip. Emitting both at the same moment therefore means the relay copy arrives first
+essentially always, and `transport` reads `relay` on networks where the punch succeeded. That
+was measured on loopback while writing this contract: the direct copy consistently lost.
+
+So when — and only when — the task carries both `origin_machine_id` and a `p2p.mode` of `auto`
+or `direct`, the relay defers its own offer by `p2pOfferGraceMs` (default **1200 ms**) and
+cancels it if the lease has moved out of `queued` by then. No new message is needed for that
+test: the executor's first act on any offer it accepts is a lease heartbeat, which is exactly
+the fact that says "the direct copy arrived".
+
+Consequences, stated because they are the cost of the feature:
+
+* a punch that fails costs one grace period of extra latency per task, and nothing else;
+* a duplicate can still arrive in both directions — the grace is a window, not a lock — so the
+  executor's dedupe remains load-bearing and is tested in both arrival orders;
+* a v0.3.9 dispatcher (no `origin_machine_id`, no `p2p`) gets the old timing byte for byte.
+
+**Both copies can arrive, in either order.** The executor dedupes on `task_id` + `attempt`
+(+ `dedupe_key` when present) rather than on the arrival path: the second copy of a task that
+is queued, running or finished is not executed again. A duplicate is not an error to report;
+it is the mechanism working.
+
+## 5. The result, and the ledger copy that is not optional
+
+The executor returns the result on the channel the offer arrived on, when there is one:
+
+```
+executor                                     dispatcher
+   |  {type:'task.result', task_id, machine_id}  ->
+   |  <-  {type:'result.ack', task_id}            |   (only after the dispatcher has stored it)
+```
+
+* **What travels on that frame, and what deliberately does not.** It carries the frame type, the
+  task id and the machine id — *not* the envelope. `result_path` is one of the envelope's own
+  fields and `envelope_sha256` covers the whole envelope, so the acknowledgement has to be known
+  before the envelope is built; sending the envelope first and stamping `result_path` afterwards
+  would produce a hash that does not cover the bytes it claims to. The direct path therefore
+  saves the dispatcher the wait for the relay's copy, **not the bytes**: the payload still reaches
+  it through the ledger. This is a real narrowing of what "the payload path is direct" means, and
+  it is written here rather than left for a reader to infer from a frame that looks short.
+* The direct copy is bounded: if the dispatcher does not ack within the timeout (2 s default), the
+  channel is not waited on any longer and the envelope records `result_path: 'relay'`.
+* **The relay copy is always sent, from the executor, unchanged in shape and ordering.** The
+  relay is the ledger, the report source and the only writer allowed to store a machine's own
+  result. v0.4.0 does not move the ledger, and a direct path that silently replaced it would
+  make the aggregate less trustworthy, not more.
+* The dispatcher stores what the frame told it and acks; the durable record arrives on its own
+  schedule. A dead dispatcher — crashed DSH, closed laptop — costs the direct copy and nothing
+  else: the relay copy is sent after a bounded wait, never instead of it.
+* A task's channel outlives its attempt by `p2pChannelLingerMs` (30 s default) so a duplicate of a
+  finished attempt can still be answered on it, and is released afterwards rather than accumulating
+  one live session per task an agent has ever run.
+
+## 6. Path reporting is mandatory, not decorative
+
+Every result carries the path it actually took:
+
+```json
+{
+  "transport": "p2p",
+  "p2p": {
+    "mode": "auto",
+    "offer_path": "p2p",
+    "result_path": "p2p",
+    "reason": null,
+    "rtt_ms": 43.8,
+    "peer": "202.182.123.154:51234",
+    "session": 3141592653,
+    "mapping": "endpoint-dependent"
+  }
+}
+```
+
+```json
+{
+  "transport": "relay",
+  "p2p": { "mode": "auto", "offer_path": "relay", "result_path": "relay",
+           "reason": "P2P_PUNCH_TIMEOUT", "rtt_ms": null, "peer": null,
+           "session": null, "mapping": "endpoint-dependent" }
+}
+```
+
+* `transport` is how the **offer** arrived (`p2p` | `relay`); `result_path` is how the result
+  returned. They are separate because a punch can open in one direction only in principle, and
+  a single field would have to lie about one of them.
+* `peer` and `session` describe the channel **this machine opened**. On the responding side — the
+  machine that accepted a punch — they are `null`: a HELLO carries a session, not an identity, and
+  reporting the initiator's address as a peer of the responder would be inventing a fact. `mapping`
+  is always this machine's own measurement.
+* `reason` names the failure (`P2P_NO_CANDIDATES`, `P2P_PUNCH_TIMEOUT`, `P2P_DISABLED`,
+  `P2P_UNAVAILABLE`) instead of leaving "not p2p" to be interpreted, and is **absent** (not null)
+  when the direct path worked: a key that is present is a claim.
+* The task aggregate and the report's per-machine object carry both fields; the markdown
+  report's table has a trailing `路径` column.
+* **Neither field is comparable.** A path is not a result: two machines that ran the same
+  bytes and disagree only about how the offer reached them are `consistent`, and a test says
+  so explicitly. Adding a transport field to `COMPARABLE_FIELDS` would turn a network
+  difference into a false bug report, which is the exact failure the six aggregation states
+  exist to prevent.
+
+## 7. The relay, extended
+
+Additive only:
+
+* `POST /v1/task` accepts `origin_machine_id` (string ≤ 128 chars) and `p2p` (`{mode}` with
+  `mode ∈ auto|direct|relay`). Both are validated shallowly and loudly — a bad value is
+  `BAD_REQUEST`, names the field, and creates **no** task and emits **no** offer.
+* Every emitted offer carries both fields (`null` when the caller sent nothing).
+* They are stored in the task and in the `task.created` ledger entry, so a relay restart
+  re-emitting offers (`redeliverPendingOffers`, lease takeover) does not quietly strip the
+  direct-push hint.
+* `GET /v1/tasks/{id}` and `GET /v1/tasks/{id}/report` expose `transport`/`p2p` per machine.
+  A result envelope with a malformed `transport` or `p2p` is refused rather than stored: a
+  ledger entry reading `transport: "satellite"` would render as a path that does not exist.
+
+## 8. `w2m-stun` — the responder
+
+The shared server also answers STUN itself (RFC 5389 Binding, UDP `:3478`), so a fleet does
+not depend on three third-party servers being reachable from every network it runs in. It
+replies `0x0101` with `XOR-MAPPED-ADDRESS`, `MAPPED-ADDRESS` and `SOFTWARE`, echoes the
+transaction id, drops anything whose magic cookie is wrong, and answers `0x0111`/400 only for
+an unknown comprehension-required attribute.
+
+Deliberately **not** implemented: RFC 5780 (`CHANGE-REQUEST` / `OTHER-ADDRESS`), so this
+responder cannot classify filtering behaviour. That would need a second address, and claiming
+to detect what was not probed is the failure mode this project keeps refusing.
+
+The v0.3.9 client needed no change to talk to it — the responder was written against the same
+message layout, and the test that proves it is a frozen byte-level vector, not a round trip
+through the encoder it was written next to.
+
+## 9. What is verified, and what is not
+
+**Verified:**
+
+| Claim | Evidence |
+|---|---|
+| Announce → rendezvous → punch → offer → execute → result, over real UDP sockets | `test/p2p-node.test.mjs`, `test/p2p-agent.test.mjs` |
+| The responder's wire format, including a frozen byte vector and error path | `test/stun-server.test.mjs` |
+| The relay's pass-through, validation and restart durability | `test/p2p-transport-fields.test.mjs` |
+| A transport difference never changes a verdict | `test/p2p-transport-fields.test.mjs`, `test/p2p-plugin.test.mjs` |
+| Exactly-once under double delivery (relay + direct, both orders) | `test/p2p-agent.test.mjs` |
+| One real WAN hop, with numbers: dispatcher on a Chinese residential network behind a **symmetric** NAT, executor on the shared server in Tokyo — `HELLO`→`HELLO_ACK` in **331.61 ms**, ledger `transport: "p2p"`, `result_path: "p2p"` | `_ops/wan-p2p-verify.mjs`, recorded in `RELEASE-STATUS.md` and `CHANGELOG.md` |
+| The shared server's relay, STUN responder and proxy, reachable from the public internet | `deploy/shared-server/`, live instance |
+
+**Not verified — and the release says so rather than implying otherwise:**
+
+| Claim | Why it is still open |
+|---|---|
+| A punch between **two** hosts behind two **different** NATs | Needs two such networks at once. Measured here: the client network is symmetric, so a two-NAT punch from it cannot succeed — which is exactly what the fallback is for. |
+| Punching through a cone NAT | Same reason. |
+| The direct path under real packet loss / reordering | Loopback loss is injected deterministically; a lossy WAN path was not exercised. |
+| IPv6 | Not implemented in the transport, and the responder's IPv6 branch is unexercised (no `udp6` socket was created while testing). |
+| The UDP channel's authenticity | Unchanged from v0.3.9 §6: session ids are random 32-bit values and UDP source addresses are spoofable. A direct path is not more trustworthy than the relay; enable request signing when the threat model includes an on-path attacker. |
+
+A loopback punch is **not** evidence of NAT traversal: two sockets on one host share a path
+with no translator between them. The WAN measurement is one hop with one NAT, and it is
+labelled as one hop with one NAT.