@twinsearth/w2m-dsh-plugin 0.4.9 → 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +100 -0
- package/deploy/networking/LESSONS.md +90 -0
- package/deploy/networking/README.md +130 -0
- package/deploy/networking/acceptance.mjs +306 -0
- package/deploy/networking/wireguard/server-setup.sh +106 -0
- package/deploy/windows/install-service.ps1 +25 -1
- package/deploy/windows/w2m-agent-supervisor.mjs +31 -9
- package/package.json +1 -1
- package/src/agent/agent.mjs +474 -17
- package/src/agent/p2p-node.mjs +118 -23
- package/src/agent/p2p.mjs +137 -9
- package/src/plugin/tools.mjs +136 -15
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,106 @@ All notable changes to this project are documented here. The format follows
|
|
|
4
4
|
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project
|
|
5
5
|
adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
6
|
|
|
7
|
+
## [0.5.1] — 2026-10-10
|
|
8
|
+
|
|
9
|
+
### Fixed
|
|
10
|
+
|
|
11
|
+
* **The agent no longer dies with its console.** Measured twice on the fleet: the whole process
|
|
12
|
+
tree — supervisor and agent together — was killed with code `0xC000013A` (`STATUS_CONTROL_C_EXIT`,
|
|
13
|
+
its console went away), the log ended mid-sentence, and Task Scheduler did not restart it, so the
|
|
14
|
+
fleet silently lost the machine. A supervisor that shares the console dies with the thing it
|
|
15
|
+
supervises, which was its one job. The supervisor now spawns the agent with `detached: true` and
|
|
16
|
+
**file-descriptor stdio** — never `inherit`, never pipes, because a detached child writing into a
|
|
17
|
+
pipe whose reader has died blocks instead of failing loudly.
|
|
18
|
+
* **The scheduled task heals itself, without elevation.** The interactive mode now also registers a
|
|
19
|
+
repeating trigger (default every 5 minutes) which, with `MultipleInstances IgnoreNew`, means "be
|
|
20
|
+
running, or start": a death from any cause is repaired at the next tick. It exists because
|
|
21
|
+
`RestartCount 999` was measured not to fire at all.
|
|
22
|
+
|
|
23
|
+
### Added
|
|
24
|
+
|
|
25
|
+
* **`deploy/networking/LESSONS.md`** — a read-only study of 2dust/v2rayN (GPL-3.0, so description
|
|
26
|
+
with citations only and nothing copied) recording the transport disciplines this project was
|
|
27
|
+
missing, and the decision it changed: **no TUN/VPN mode.** v2rayN never creates an adapter itself
|
|
28
|
+
and contains no peer-to-peer anywhere; a tunnel is hub-and-spoke by construction, so it would make
|
|
29
|
+
our "the data path is peer-to-peer" claim false for exactly the traffic a tunnel exists to carry.
|
|
30
|
+
|
|
31
|
+
### Notes
|
|
32
|
+
|
|
33
|
+
* A regression introduced and caught during the fix is recorded in the code: the first version also
|
|
34
|
+
called `child.unref()`, which leaves the supervisor with no handle at all, so its event loop
|
|
35
|
+
drains and it exits immediately after spawning the agent. The evidence was that no supervisor
|
|
36
|
+
process existed — found while looking for a different property.
|
|
37
|
+
* Verified on the machine that had the defect: supervisor and agent both present, and killing the
|
|
38
|
+
supervisor leaves the agent alive with its punch socket still bound (`0.0.0.0:41235`).
|
|
39
|
+
* Also included: the test fix for `test/outbound-only.test.mjs`, whose port was being read from the
|
|
40
|
+
timestamp rather than from the field that states it (commit `57dbc5f`).
|
|
41
|
+
## [0.5.0] — 2026-10-10
|
|
42
|
+
|
|
43
|
+
The release that stops treating "the relay carried it" as the only answer for a machine behind a
|
|
44
|
+
symmetric NAT. Measured before it: a dispatcher behind carrier-grade NAT (an iPhone hotspot, in this
|
|
45
|
+
fleet) could not push an offer to a peer, the peer's result could not come back, and both halves fell
|
|
46
|
+
to the relay on **every** task — `P2P_PUNCH_TIMEOUT` on the push, `transport: relay` on the result.
|
|
47
|
+
|
|
48
|
+
### Added
|
|
49
|
+
|
|
50
|
+
* **Simultaneous two-way punch.** The dispatcher publishes its punch in the offer it puts on the relay
|
|
51
|
+
(`POST /v1/task`'s `p2p`: `{mode, punch, candidates?}`), and the executor — which previously could
|
|
52
|
+
only wait in `accept()` — **dials the dispatcher back** on the same derived session, concurrently
|
|
53
|
+
with execution, bounded by the new `tuning.reverseDialTimeoutMs` (default **8000 ms**). Two NATs
|
|
54
|
+
that each open a mapping toward the other can then meet; one of them no longer has to be dialable.
|
|
55
|
+
* **Reverse dial on the result path.** `planDirectDelivery` dials the dispatcher before falling back
|
|
56
|
+
to the relay, reusing the offer-time attempt when there was one (one bounded window per task, never
|
|
57
|
+
two) and starting its own when the channel died in between. This is the direction that works for a
|
|
58
|
+
symmetric-NAT machine: it dials *out*.
|
|
59
|
+
* **The ladder is now recorded, rung by rung.** `p2p.opened_by` names the mechanism
|
|
60
|
+
(`dispatcher-punch` | `simultaneous-punch` | `result-reverse-dial`); `p2p.reverse_dial`,
|
|
61
|
+
`p2p.reverse_dial_ms` and `p2p.reverse_dial_reason` say whether a dial was attempted, what it cost
|
|
62
|
+
and why it lost (`P2P_REVERSE_DIAL_TIMEOUT`, `P2P_REVERSE_DIAL_FAILED`, `P2P_CHANNEL_LIVE`,
|
|
63
|
+
`P2P_NO_SESSION`, `P2P_NO_ORIGIN`, `P2P_DISABLED`, `P2P_UNREACHABLE`, `P2P_NO_TASK_ID`). The
|
|
64
|
+
mechanism is shared with the plugin (`punchSession()` lives in one place, imported, not copied).
|
|
65
|
+
* **`deploy/networking/`** — the ladder written down: five rungs with what the ledger shows for each,
|
|
66
|
+
`acceptance.mjs` to measure which rung a real dispatch takes (it mints a real punch id, because a
|
|
67
|
+
body without one exercises neither new path), and `wireguard/server-setup.sh` for rung 4.
|
|
68
|
+
|
|
69
|
+
### Fixed
|
|
70
|
+
|
|
71
|
+
* **`P2PChannel.close()` could never send its BYE.** It set `closed = true` and *then* called
|
|
72
|
+
`sendFrame(BYE, …)`, and `sendFrame` refuses to send once closed — so the goodbye was unreachable
|
|
73
|
+
dead code. The measured consequence sat exactly on the path this release is about: `pushOffer`
|
|
74
|
+
closes the channel it pushed over, the executor kept a channel it believed was live, sent its result
|
|
75
|
+
into it and fell back only after `P2P_FRAGMENT_UNACKED … after 8 attempts` — **with no reverse dial
|
|
76
|
+
attempted**, because a channel for the task still existed. Change B alone could not have fixed the
|
|
77
|
+
measured failure; this is the other half. `close(reason, {silent: true})` exists for discarding a
|
|
78
|
+
duplicate local view of a live session (a simultaneous punch legitimately creates two channel
|
|
79
|
+
objects on one side, and a BYE there would tear down the peer's working channel).
|
|
80
|
+
|
|
81
|
+
### Changed
|
|
82
|
+
|
|
83
|
+
* **`transport` widens.** It is `p2p` when the direct path carried **either** half — the offer over a
|
|
84
|
+
channel **or** the result acknowledged over one — and `relay` only when the relay carried both.
|
|
85
|
+
`p2p.offer_path` keeps its exact meaning ("how the offer arrived"), which is what keeps the two
|
|
86
|
+
halves distinguishable: `transport: p2p` + `offer_path: relay` is a result that found its own way
|
|
87
|
+
home. Anything reading `transport` alone should read `offer_path`/`opened_by` beside it.
|
|
88
|
+
|
|
89
|
+
### Compatibility
|
|
90
|
+
|
|
91
|
+
No protocol version bump: `envelope_version` stays `1.0`, the new keys live inside the optional `p2p`
|
|
92
|
+
object, and `transport`'s value set is unchanged. A v0.4.x relay stores and republishes the extended
|
|
93
|
+
object **verbatim**, and a v0.4.x agent ignores keys it does not know — both are pinned by tests, on
|
|
94
|
+
both sides, rather than argued from intent.
|
|
95
|
+
|
|
96
|
+
### Verification
|
|
97
|
+
|
|
98
|
+
* Required suites: **424 tests, 421 pass, 1 fail, 2 skipped** (baseline 407/404/1/2 → +17 new tests,
|
|
99
|
+
all passing). The failure is the pre-existing bundled-python case.
|
|
100
|
+
* Every other suite: 340 tests, 335 pass, 2 fail (the pre-existing `pipeline-e2e` cases).
|
|
101
|
+
* `eslint .`: 0 errors, the same 79 pre-existing warnings.
|
|
102
|
+
* **Not verified, and it matters:** every punch in the suites is loopback, which crosses no NAT, and
|
|
103
|
+
no genuine v0.4.x binary was run — compatibility is argued from the relay's validation code plus
|
|
104
|
+
round-trip tests. Real traversal needs a fleet running this build; that is the next measurement, not
|
|
105
|
+
a claim in this entry.
|
|
106
|
+
|
|
7
107
|
## [0.4.9] — 2026-10-10
|
|
8
108
|
|
|
9
109
|
### Fixed
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
# What a mature proxy client taught us about transport — and what it did not
|
|
2
|
+
|
|
3
|
+
This file records a **read-only study of [2dust/v2rayN](https://github.com/2dust/v2rayN)**, done to
|
|
4
|
+
find out which transport disciplines this project is missing. It exists because the study's
|
|
5
|
+
conclusions are only worth anything if they can be checked, and because one of them is a decision we
|
|
6
|
+
would otherwise have made wrongly.
|
|
7
|
+
|
|
8
|
+
## Provenance and licence
|
|
9
|
+
|
|
10
|
+
* Read at branch **`master`**, commit **`b501efbc45bca3c465df99af7d4ec092c2f4c40f`**. Paths below are
|
|
11
|
+
relative to that clone, so every claim can be checked at that revision.
|
|
12
|
+
* **v2rayN is GPL-3.0** (`LICENSE`, the verbatim GNU GPL v3 text; no other licence statement in the
|
|
13
|
+
repository). **This project is MIT.** No code, text or file was copied from it — not a fragment,
|
|
14
|
+
not a paraphrase of code. Everything below is *description with citations*, and any implementation
|
|
15
|
+
we write from it is our own.
|
|
16
|
+
* We did not modify v2rayN, and we did not open issues or pull requests against it.
|
|
17
|
+
|
|
18
|
+
## The decision this study changed: do not add a TUN/VPN mode
|
|
19
|
+
|
|
20
|
+
The tempting idea was "add a VPN so two machines behind hard NAT can reach each other". The study
|
|
21
|
+
says plainly that this does not do what it sounds like:
|
|
22
|
+
|
|
23
|
+
* v2rayN never creates an adapter or touches the routing table itself; it writes a declarative config
|
|
24
|
+
and the external core (Xray/sing-box/mihomo) does the privileged work
|
|
25
|
+
(`v2rayN/ServiceLib/Services/CoreConfig/V2ray/V2rayInboundService.cs:75-77`;
|
|
26
|
+
`Sample/tun_singbox_inbound:10-13`).
|
|
27
|
+
* Its local inbound is loopback-only (`Sample/SampleInbound:5`), and every path is client → remote
|
|
28
|
+
server. There is **no peer-to-peer anywhere in the codebase**: no hole punching, no candidate
|
|
29
|
+
exchange, no relay of our kind — a grep for the concepts returns nothing, and the only STUN code is
|
|
30
|
+
a UDP reachability *diagnostic* (`ServiceLib.UdpTest/Tester/StunService.cs:8-48`), not a data path.
|
|
31
|
+
* So a TUN interface is **hub-and-spoke by construction**: traffic between two of our machines would
|
|
32
|
+
transit the hub. Our promise — "the data path is peer-to-peer" — would become **false for exactly
|
|
33
|
+
the traffic the tunnel exists to carry**.
|
|
34
|
+
|
|
35
|
+
**Therefore: no TUN in this project for now.** It would cost administrator rights, a kernel driver,
|
|
36
|
+
and the truth of our central claim, while solving only part of what the delivery ladder already
|
|
37
|
+
addresses. If it is ever added, two invariants have to hold, and both belong in the ladder's
|
|
38
|
+
documentation rather than in a commit message: the control plane and the punch sockets must be
|
|
39
|
+
exempted from the tunnel, and tunnelled traffic must never be reported as `p2p`.
|
|
40
|
+
|
|
41
|
+
## Disciplines worth adopting, in the order we should adopt them
|
|
42
|
+
|
|
43
|
+
1. **Declare policy; let one worker enforce it.** v2rayN's central shape is an ordered, declarative
|
|
44
|
+
policy consumed by a single enforcer (`Services/CoreConfig/…`), with validation that aborts a start
|
|
45
|
+
rather than launching something broken (`Manager/CoreManager.cs:75-80`), mutating a deep copy
|
|
46
|
+
rather than live config. Our ladder should be *data* — ordered rungs, each with its own bound and
|
|
47
|
+
its own reason code — executed in one place.
|
|
48
|
+
2. **One named constant per network bound, and a bigger budget for the relayed leg.** `LocalFetch`
|
|
49
|
+
5 s, `DirectDownloadConnect` 5 s, **`ProxyDownloadConnect` 10 s**, selected per hop
|
|
50
|
+
(`Global.cs:100,103-104`; `Helper/DownloaderHelper.cs:336-339`). We cross NAT *and* a relay; the
|
|
51
|
+
relay leg deserves its own larger bound instead of sharing one timeout with a LAN attempt.
|
|
52
|
+
3. **A distinguished failure value, never a silent zero.** `-1` as a failure sentinel
|
|
53
|
+
(`Handler/ConnectionHandler.cs:34,72`), with dependent work explicitly skipped when a probe fails
|
|
54
|
+
(`Services/SpeedtestService.cs:499-509`). Our `HELLO`/`HELLO_ACK` path should separate *no answer
|
|
55
|
+
yet* / *refused* / *timed out* / *answered wrong*, and the ledger should name **which rung** failed
|
|
56
|
+
rather than reporting a single boolean.
|
|
57
|
+
4. **Bounded attempts, then fall down the ladder.** `MaxTryAgainOnFailure = 2`, twice with a 500 ms
|
|
58
|
+
gap (`DownloaderHelper.cs:37,71`; `ConnectionHandler.cs:39-47`). Give the simultaneous punch and
|
|
59
|
+
the reverse dial two or three attempts each, and record the reason each one lost.
|
|
60
|
+
5. **Exempt the transport's own traffic from the transport.** v2rayN forces its cores and its own
|
|
61
|
+
directory to go direct (`Services/CoreConfig/V2ray/V2rayRoutingService.cs:239-278`, emitted at
|
|
62
|
+
`:16-30`). This is the most load-bearing lesson in the whole study: a control plane routed through
|
|
63
|
+
the thing it controls is the most likely way to brick a machine.
|
|
64
|
+
6. **Teardown that is enforced by the OS, plus reaping at start.** Cores are assigned to a
|
|
65
|
+
kill-on-close job object (`Services/WindowsJobService.cs:15-18`; `Manager/CoreManager.cs:360,365-376`)
|
|
66
|
+
so a GUI crash still kills them, and stale devices are swept at the *next* launch
|
|
67
|
+
(`Common/WindowsUtils.cs:55-75`). Our analogue: reap stale state and half-open punches on start
|
|
68
|
+
instead of trusting a graceful exit.
|
|
69
|
+
7. **Validate, warn, fall back to a safe value.** Errors and warnings are separate lists; an
|
|
70
|
+
unresolvable tag falls back with a warning; invalid values are blanked and reported
|
|
71
|
+
(`Handler/Builder/CoreConfigContextBuilder.cs:3-26,70-136`).
|
|
72
|
+
8. **Ordered rules with a documented catch-all.** Tunnel/DNS/loop rules first, user rules next,
|
|
73
|
+
catch-all last (`V2rayRoutingService.cs:9-64,215-237`). Only relevant to us if routing ever enters
|
|
74
|
+
the picture — which, per the decision above, it has not.
|
|
75
|
+
|
|
76
|
+
## What we deliberately are not copying
|
|
77
|
+
|
|
78
|
+
Everything requiring an external core or administrator rights: Wintun/WFP plumbing, WinINet/RAS proxy
|
|
79
|
+
configuration with a registry fallback, PAC serving, route supernet arithmetic, TUN address pools,
|
|
80
|
+
per-process/per-domain routing, and core-level observatories/balancers. None of it is our shape, and
|
|
81
|
+
most of it is licensed work we must not take even if it were.
|
|
82
|
+
|
|
83
|
+
## One thing v2rayN does *not* do, which we should not admire
|
|
84
|
+
|
|
85
|
+
There is **no core auto-restart and no core-death notification**: the `Exited` handler only
|
|
86
|
+
unsubscribes log handlers (`Services/ProcessService.cs:132-142`). Its safety net is the job object
|
|
87
|
+
killing things reliably — not noticing and repairing. We already learned this the hard way: this
|
|
88
|
+
project's agent was twice found dead with a `0xC000013A` and nothing restarting it. Reliability by
|
|
89
|
+
construction is better than reliability by hope, and where we cannot have the former we now at least
|
|
90
|
+
have a watchdog that notices.
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
# Networking: how the fleet gets from one machine to another
|
|
2
|
+
|
|
3
|
+
This directory is about the one thing that decides whether a fleet is pleasant or infuriating:
|
|
4
|
+
**which path the bytes actually take**, and what to do when the direct one is not available.
|
|
5
|
+
|
|
6
|
+
The control plane is never in question. Devices, tasks, leases, results and the ledger all go through
|
|
7
|
+
the relay on the shared server (`http://202.182.123.154:8787`), because that is what makes a task
|
|
8
|
+
*auditable*: one place holds what was asked, who took it, what came back and by which path. The data
|
|
9
|
+
plane is where the interesting failures live, so it is arranged as a ladder — each rung is tried,
|
|
10
|
+
and a rung that cannot be opened is recorded with its reason rather than guessed at.
|
|
11
|
+
|
|
12
|
+
## The ladder
|
|
13
|
+
|
|
14
|
+
| rung | mechanism | what the ledger shows | what it costs |
|
|
15
|
+
|---|---|---|---|
|
|
16
|
+
| 1 | a channel is already open | `transport: p2p`, `offer_path: p2p` | nothing |
|
|
17
|
+
| 2 | **simultaneous punch (A)** — the dispatcher punches, and the executor dials the same session it was told about | `transport: p2p`, `offer_path: p2p` | two NATs must agree, or one side must be dialable |
|
|
18
|
+
| 3 | **reverse dial (B)** — no channel existed, so the receiving side dialled out to deliver the result | `transport: p2p`, `offer_path: relay` | the result found its own way home while the offer needed the relay |
|
|
19
|
+
| 4 | **tunnel (C)** — WireGuard on the shared server, and every machine gets a routable `10.66.0.x` | `transport: p2p` — **indistinguishable from rung 1–2** | see the warning below |
|
|
20
|
+
| 5 | relay | `transport: relay` + a `p2p.reason` naming why | the relay carries the data |
|
|
21
|
+
|
|
22
|
+
**Rung 4 is the one to read carefully.** WireGuard has no NAT traversal of its own — that is exactly
|
|
23
|
+
what Tailscale adds on top of it (disco/DERP) and what rungs 2 and 3 are for. So with two machines
|
|
24
|
+
behind NAT, traffic between their `10.66.0.x` addresses is **forwarded through the VPS**. The ledger
|
|
25
|
+
will say `p2p`, and it will not be lying: from W2M's point of view that socket is direct. But the
|
|
26
|
+
bytes cross a server. If you asked for "the data goes peer to peer", rung 4 does not satisfy that —
|
|
27
|
+
it satisfies "the fleet works regardless of NAT, encrypted, with no third party", which is a
|
|
28
|
+
different and still useful promise. Rung 2/3 are the ones that make the data path direct.
|
|
29
|
+
|
|
30
|
+
## What is measured, and by what
|
|
31
|
+
|
|
32
|
+
`acceptance.mjs` dispatches a real command to every online machine through the relay's HTTP API and
|
|
33
|
+
then reads back what each machine recorded. It is deliberately blunt: it does not synthesise packets
|
|
34
|
+
or poke at NATs itself, because a script that punched its own holes would prove that the script can
|
|
35
|
+
punch, which nobody doubts.
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
node deploy/networking/acceptance.mjs \
|
|
39
|
+
--rabbit http://202.182.123.154:8787 \
|
|
40
|
+
--token <operator token> \
|
|
41
|
+
--project /path/to/the/fleet/project \
|
|
42
|
+
--command 'git rev-parse HEAD' \
|
|
43
|
+
--require-direct # optional: exit non-zero if any machine lands on rung 5
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
Requirements and behaviour worth knowing before you read its output:
|
|
47
|
+
|
|
48
|
+
* **Both credentials.** Dispatching needs the **operator** token (`--token`, or
|
|
49
|
+
`W2M_OPERATOR_TOKEN`); reading a task back is a device-scoped read and needs the **device** token
|
|
50
|
+
from `device.json` (`--state-dir`, default `$DSH_HOME/xclient`). Measured: the operator token alone
|
|
51
|
+
answers `401 missing or invalid device_token`. The split is deliberate — the credential that can
|
|
52
|
+
order work around cannot also read every machine's results.
|
|
53
|
+
* **Every machine must be on the same commit.** The script fingerprints `--project` with
|
|
54
|
+
`git-temp-index-tree/v1` and sends it as the anchor; machines that are not there refuse or report
|
|
55
|
+
`unverifiable`. That is the mechanism working, not the script failing.
|
|
56
|
+
* **`--command` takes plain text** (`--command 'git rev-parse HEAD'`). A JSON array is the natural
|
|
57
|
+
spelling and it does not survive a Windows command line: `'["git","rev-parse","HEAD"]'` arrives as
|
|
58
|
+
`[git,rev-parse,HEAD]`. The script still accepts JSON when the shell passes it through intact.
|
|
59
|
+
* **Transient network failures retry.** An acceptance run that reported "the fleet is broken" because
|
|
60
|
+
one TCP connect to another continent timed out would be worse than useless.
|
|
61
|
+
* **What it exercises is rungs 3 and 5 (and 2 once A lands).** Because the script dispatches *through
|
|
62
|
+
the relay*, the offer arrives by relay — which is exactly the situation the reverse-dial work
|
|
63
|
+
addresses. The dispatcher-side punch is performed by the plugin's own `w2m_run`, which pushes to
|
|
64
|
+
each peer directly before falling back. So the two commands answer different questions:
|
|
65
|
+
|
|
66
|
+
| command | question |
|
|
67
|
+
|---|---|
|
|
68
|
+
| `w2m_run` + `w2m_wait` (the DSH tools) | what does the whole delivery path do, including the dispatcher's push? |
|
|
69
|
+
| `acceptance.mjs` | what do the **machines** do when the offer arrives by relay? |
|
|
70
|
+
|
|
71
|
+
Run both. If `w2m_run` shows `p2p` while `acceptance.mjs` shows rung 5 for the same pair, the
|
|
72
|
+
dispatcher's push is working and the receiving side is not dialling back — which is precisely the
|
|
73
|
+
gap rungs 2 and 3 close.
|
|
74
|
+
|
|
75
|
+
## Baseline, measured 2026-10-10 (before A/B landed)
|
|
76
|
+
|
|
77
|
+
```
|
|
78
|
+
machine outcome exit rung path offer note
|
|
79
|
+
jp-shared ok 0 5 relay relay relay fallback -- P2P_NO_CANDIDATES ...
|
|
80
|
+
win-desktop ok 0 5 relay relay relay fallback -- P2P_NO_CANDIDATES ...
|
|
81
|
+
macmini ok 0 5 relay relay relay fallback -- P2P_NO_CANDIDATES ...
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
The same fleet, dispatched through the plugin's `w2m_run` instead (which pushes directly), gets
|
|
85
|
+
`p2p` for the two machines that can be dialled. Both readings are true; they are readings of
|
|
86
|
+
different layers. Keep the baseline above and re-run it after the reverse-dial work lands: rung 5
|
|
87
|
+
becoming rung 2 or 3 for a machine that used to fall back is the evidence that it worked.
|
|
88
|
+
|
|
89
|
+
## Rung 4: the WireGuard hub on the shared server
|
|
90
|
+
|
|
91
|
+
Set up and running on the Tokyo VPS. `deploy/networking/wireguard/server-setup.sh` is the whole
|
|
92
|
+
thing: it generates keys once (and reuses them afterwards, so a client that already has a config
|
|
93
|
+
keeps working), writes `/etc/wireguard/wg0.conf`, opens the port and starts the hub. It prints a
|
|
94
|
+
client config on demand — that is the only thing a client needs from this host:
|
|
95
|
+
|
|
96
|
+
```bash
|
|
97
|
+
bash server-setup.sh # on the always-on host, as root
|
|
98
|
+
bash server-setup.sh --client macmini # prints a config to paste into the WireGuard app
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
```
|
|
102
|
+
interface wg0 listening 51820/udp server 10.66.0.1/24
|
|
103
|
+
peers: 10.66.0.2 (win-desktop), 10.66.0.3 (macmini) ufw: 51820/udp allowed
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
Each client routes **only** `10.66.0.0/24`, so nothing of your ordinary traffic can travel through
|
|
107
|
+
the VPS and no DNS is touched. Client install: Windows `winget install WireGuard.WireGuard` (needs
|
|
108
|
+
elevation) then import the `.conf`; macOS the WireGuard app or `brew install --cask wireguard-tools`.
|
|
109
|
+
Verify from the server with `wg show`: a `latest handshake` line per peer is the proof it is up.
|
|
110
|
+
|
|
111
|
+
Two port notes, both learned the hard way:
|
|
112
|
+
|
|
113
|
+
* **51820 is WireGuard's default, so do not give it to anything else.** The W2M agent on this machine
|
|
114
|
+
was briefly pinned to 51820 (to match an inbound firewall rule) and was moved to **41235** when the
|
|
115
|
+
tunnel arrived.
|
|
116
|
+
* **An inbound firewall rule is not a route.** A rule for `51820/udp` on a machine behind
|
|
117
|
+
carrier-grade NAT cannot make that machine dialable: the carrier drops the packet before the host
|
|
118
|
+
firewall ever sees it. Rules matter again the moment the machine has a real mapping (a normal
|
|
119
|
+
router, or the tunnel).
|
|
120
|
+
|
|
121
|
+
## The honesty section
|
|
122
|
+
|
|
123
|
+
* **Rung 5 is not a failure.** A task that completes over the relay completed. The ledger naming the
|
|
124
|
+
path is the feature; pretending every path is direct would be the bug.
|
|
125
|
+
* **A `p2p` reading on a tunnel (rung 4) does not mean the bytes avoided a server.** See above.
|
|
126
|
+
* **A symmetric NAT cannot be dialled into, ever.** Not by this project, not by Tailscale. What can
|
|
127
|
+
be done is to have the symmetric side dial *out* (rungs 2 and 3) or to stop being behind it
|
|
128
|
+
(rung 4, or another network).
|
|
129
|
+
* **Nothing here injects prompts into another machine's model session.** The fleet transports
|
|
130
|
+
commands, their output, and the comparison between them.
|
|
@@ -0,0 +1,306 @@
|
|
|
1
|
+
// ============================================================================
|
|
2
|
+
// acceptance.mjs -- does the fleet actually take the direct path?
|
|
3
|
+
// ============================================================================
|
|
4
|
+
//
|
|
5
|
+
// This script answers one question with evidence instead of optimism: for a real
|
|
6
|
+
// dispatched task, which rung of the delivery ladder did each machine land on, and
|
|
7
|
+
// why. It dispatches a command to every online machine through the relay's own
|
|
8
|
+
// HTTP API, waits for the aggregate, and maps each machine's recorded facts
|
|
9
|
+
// (`transport`, `p2p.offer_path`, `p2p.reason`) onto the ladder in README.md.
|
|
10
|
+
//
|
|
11
|
+
// THE LADDER, and what the ledger calls it:
|
|
12
|
+
//
|
|
13
|
+
// 1 live channel transport p2p + offer_path p2p (a channel was already open)
|
|
14
|
+
// 2 simultaneous punch transport p2p + offer_path p2p (both sides dialled -- A)
|
|
15
|
+
// 3 reverse dial transport p2p + offer_path relay (the result found its own way back -- B)
|
|
16
|
+
// ...and the result of a level-3 task may itself report result_path p2p in the agent's log;
|
|
17
|
+
// that log is on the machine, which is why this script reads the aggregate the relay keeps.
|
|
18
|
+
// 4 tunnel transport p2p on a WireGuard address. NOT DETECTABLE FROM THE LEDGER:
|
|
19
|
+
// level 4 looks exactly like level 1-2 from here. See README.md -- a
|
|
20
|
+
// tunnel makes the fleet reachable, and the bytes still cross the VPS.
|
|
21
|
+
// 5 relay transport relay (the fallback, always available)
|
|
22
|
+
//
|
|
23
|
+
// USAGE
|
|
24
|
+
//
|
|
25
|
+
// node deploy/networking/acceptance.mjs \
|
|
26
|
+
// --rabbit http://202.182.123.154:8787 \
|
|
27
|
+
// --token <operator token> \
|
|
28
|
+
// --project /path/to/the/fleet/project \
|
|
29
|
+
// --command 'git rev-parse HEAD'
|
|
30
|
+
//
|
|
31
|
+
// --require-direct exit non-zero when any machine falls to the relay, so this can
|
|
32
|
+
// be a gate rather than a report
|
|
33
|
+
// --json print the aggregate as JSON instead of a table
|
|
34
|
+
//
|
|
35
|
+
// WHY --command TAKES PLAIN TEXT
|
|
36
|
+
//
|
|
37
|
+
// A JSON array is the natural spelling and it does not survive the trip: measured on Windows,
|
|
38
|
+
// `--command '["git","rev-parse","HEAD"]'` arrives as `[git,rev-parse,HEAD]` -- the quotes are
|
|
39
|
+
// consumed before the script ever sees them (the same trap that produced the config file in
|
|
40
|
+
// deploy/windows). So plain whitespace-separated text is the documented form; a string that does
|
|
41
|
+
// start with `[` is still parsed as JSON for callers whose shell does not mangle it.
|
|
42
|
+
//
|
|
43
|
+
// WHAT IT DELIBERATELY DOES NOT DO
|
|
44
|
+
//
|
|
45
|
+
// It does not test the punch itself, inject packets, or bypass the relay. The punch is the
|
|
46
|
+
// protocol's business; this only reads what the protocol recorded after a real task. A script
|
|
47
|
+
// that synthesised its own punches would prove that the script can punch, which nobody doubts.
|
|
48
|
+
// ============================================================================
|
|
49
|
+
import { headCommit, treeFingerprint } from '../../src/agent/git.mjs';
|
|
50
|
+
import { mintPunchId } from '../../src/agent/p2p.mjs';
|
|
51
|
+
import { readFileSync } from 'node:fs';
|
|
52
|
+
import { join } from 'node:path';
|
|
53
|
+
|
|
54
|
+
const USAGE = `acceptance.mjs --rabbit <url> [--token <operator token>] [--project <dir>]
|
|
55
|
+
[--command '<json argv>'] [--timeout-ms <n>] [--poll-ms <n>]
|
|
56
|
+
[--require-direct] [--json] [--state-dir <dir>]`;
|
|
57
|
+
|
|
58
|
+
function parseArgs(argv) {
|
|
59
|
+
const out = {};
|
|
60
|
+
for (let i = 0; i < argv.length; i += 1) {
|
|
61
|
+
const token = argv[i];
|
|
62
|
+
if (!token.startsWith('--')) continue;
|
|
63
|
+
const key = token.slice(2);
|
|
64
|
+
const next = argv[i + 1];
|
|
65
|
+
if (next === undefined || next.startsWith('--')) out[key] = true;
|
|
66
|
+
else {
|
|
67
|
+
out[key] = next;
|
|
68
|
+
i += 1;
|
|
69
|
+
}
|
|
70
|
+
}
|
|
71
|
+
return out;
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
const args = parseArgs(process.argv.slice(2));
|
|
75
|
+
if (args.help) {
|
|
76
|
+
console.log(USAGE);
|
|
77
|
+
process.exit(0);
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
const rabbit = String(args.rabbit ?? 'http://202.182.123.154:8787').replace(/\/+$/, '');
|
|
81
|
+
const token = typeof args.token === 'string' ? args.token : (process.env.W2M_OPERATOR_TOKEN ?? '');
|
|
82
|
+
const project = String(args.project ?? process.cwd());
|
|
83
|
+
const timeoutMs = Number(args['timeout-ms'] ?? 180_000);
|
|
84
|
+
const pollMs = Number(args['poll-ms'] ?? 1500);
|
|
85
|
+
const requireDirect = args['require-direct'] === true;
|
|
86
|
+
const asJson = args.json === true;
|
|
87
|
+
const rawCommand = typeof args.command === 'string' ? args.command.trim() : '';
|
|
88
|
+
const commandArgv = rawCommand === ''
|
|
89
|
+
? ['git', 'rev-parse', 'HEAD']
|
|
90
|
+
: rawCommand.startsWith('[')
|
|
91
|
+
? JSON.parse(rawCommand)
|
|
92
|
+
: rawCommand.split(/\s+/).filter(Boolean);
|
|
93
|
+
|
|
94
|
+
if (!token) {
|
|
95
|
+
console.error('no operator token: pass --token or set W2M_OPERATOR_TOKEN (the relay prints it once)');
|
|
96
|
+
process.exit(2);
|
|
97
|
+
}
|
|
98
|
+
if (!Array.isArray(commandArgv) || commandArgv.length === 0) {
|
|
99
|
+
console.error("--command must be a command, e.g. --command 'git rev-parse HEAD' (or a JSON array)");
|
|
100
|
+
process.exit(2);
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
const headers = { 'content-type': 'application/json', authorization: `Bearer ${token}` };
|
|
104
|
+
|
|
105
|
+
/**
|
|
106
|
+
* Reading a task back is a DEVICE-scoped read, so it needs the device token; dispatching is what the
|
|
107
|
+
* operator token is for. Measured: `GET /v1/tasks/<id>` carrying the operator token answers
|
|
108
|
+
* `401 UNAUTHORIZED: missing or invalid device_token`. The split is deliberate -- the credential that
|
|
109
|
+
* can order work around cannot also read every machine's results -- so this script carries both,
|
|
110
|
+
* exactly as the plugin does.
|
|
111
|
+
*/
|
|
112
|
+
async function get(path) {
|
|
113
|
+
// Transient transport failures are the norm on a link between continents (measured: this exact
|
|
114
|
+
// script dispatched fine and then had its first poll time out connecting to the relay). An
|
|
115
|
+
// acceptance run that reports "the fleet is broken" because one TCP connect timed out would be
|
|
116
|
+
// worse than useless, so polls retry; an HTTP error still fails immediately, because that is the
|
|
117
|
+
// relay telling us something.
|
|
118
|
+
let lastError = null;
|
|
119
|
+
for (let attempt = 1; attempt <= 5; attempt += 1) {
|
|
120
|
+
try {
|
|
121
|
+
const res = await fetch(`${rabbit}${path}`, {
|
|
122
|
+
headers: { accept: 'application/json', authorization: `Bearer ${deviceToken ?? token}` },
|
|
123
|
+
});
|
|
124
|
+
if (!res.ok) {
|
|
125
|
+
const text = (await res.text()).slice(0, 200);
|
|
126
|
+
if (res.status === 401) {
|
|
127
|
+
throw new Error(
|
|
128
|
+
`GET ${path} -> 401 ${text}\n` +
|
|
129
|
+
'reading a task needs a device token: point --state-dir at the directory holding device.json',
|
|
130
|
+
);
|
|
131
|
+
}
|
|
132
|
+
throw new Error(`GET ${path} -> ${res.status} ${text}`);
|
|
133
|
+
}
|
|
134
|
+
return res.json();
|
|
135
|
+
} catch (error) {
|
|
136
|
+
lastError = error;
|
|
137
|
+
const transient = error?.cause?.code !== undefined || /fetch failed|timeout/i.test(String(error?.message ?? ''));
|
|
138
|
+
if (!transient) throw error;
|
|
139
|
+
await new Promise((resolve) => {
|
|
140
|
+
setTimeout(resolve, 2000 * attempt);
|
|
141
|
+
});
|
|
142
|
+
}
|
|
143
|
+
}
|
|
144
|
+
throw new Error(`GET ${path} failed after 5 attempts: ${lastError?.message ?? lastError}`);
|
|
145
|
+
}
|
|
146
|
+
|
|
147
|
+
/** This machine's own identity, when it has one -- `origin_machine_id` must be a real machine. */
|
|
148
|
+
function localIdentity() {
|
|
149
|
+
const stateDir = typeof args['state-dir'] === 'string'
|
|
150
|
+
? String(args['state-dir'])
|
|
151
|
+
: join(process.env.DSH_HOME ?? join(process.env.USERPROFILE ?? process.env.HOME ?? '.', '.dsh'), 'xclient');
|
|
152
|
+
try {
|
|
153
|
+
const device = JSON.parse(readFileSync(join(stateDir, 'device.json'), 'utf8').replace(/^\uFEFF/, ''));
|
|
154
|
+
return {
|
|
155
|
+
machine_id: device.machine_id ?? null,
|
|
156
|
+
name: device.machine_name ?? device.name ?? null,
|
|
157
|
+
device_token: typeof device.device_token === 'string' && device.device_token !== '' ? device.device_token : null,
|
|
158
|
+
};
|
|
159
|
+
} catch {
|
|
160
|
+
return { machine_id: null, name: null, device_token: null };
|
|
161
|
+
}
|
|
162
|
+
}
|
|
163
|
+
|
|
164
|
+
/**
|
|
165
|
+
* One machine's recorded path, read as a rung of the ladder.
|
|
166
|
+
*
|
|
167
|
+
* `opened_by` is the precise answer when the build provides it: it names which mechanism opened the
|
|
168
|
+
* channel (`dispatcher-punch`, `simultaneous-punch`, `result-reverse-dial`). Without it, `transport`
|
|
169
|
+
* plus `offer_path` still separate "the result found its own way back" (rung 3) from "the relay
|
|
170
|
+
* carried everything" (rung 5) -- that is why the two halves are recorded separately and why this
|
|
171
|
+
* script never reads `transport` alone.
|
|
172
|
+
*/
|
|
173
|
+
function classify(machine) {
|
|
174
|
+
const transport = machine.transport ?? null;
|
|
175
|
+
const offerPath = machine.p2p?.offer_path ?? null;
|
|
176
|
+
const openedBy = machine.p2p?.opened_by ?? null;
|
|
177
|
+
const reason = machine.p2p?.reverse_dial_reason ?? machine.p2p?.reason ?? null;
|
|
178
|
+
if (machine.outcome === 'ok' && transport === 'p2p') {
|
|
179
|
+
if (openedBy === 'result-reverse-dial') return { rung: 3, label: 'reverse dial (B)', reason };
|
|
180
|
+
if (openedBy === 'simultaneous-punch') return { rung: 2, label: 'simultaneous punch (A)', reason };
|
|
181
|
+
if (openedBy === 'dispatcher-punch') return { rung: 2, label: 'dispatcher punch', reason };
|
|
182
|
+
return {
|
|
183
|
+
rung: offerPath === 'relay' ? 3 : 2,
|
|
184
|
+
label: offerPath === 'relay' ? 'reverse dial (offer by relay, result direct)' : 'direct channel',
|
|
185
|
+
reason,
|
|
186
|
+
};
|
|
187
|
+
}
|
|
188
|
+
if (machine.outcome === 'ok' && transport === 'relay') return { rung: 5, label: 'relay fallback', reason };
|
|
189
|
+
if (machine.outcome === 'ok') return { rung: null, label: `ok over ${transport ?? 'an unknown path'}`, reason };
|
|
190
|
+
return { rung: null, label: `${machine.outcome ?? 'no outcome'}`, reason: machine.refusal_reason ?? reason };
|
|
191
|
+
}
|
|
192
|
+
|
|
193
|
+
const identity = localIdentity();
|
|
194
|
+
// Declared before the first read: reading a task needs the device token, and this machine's own
|
|
195
|
+
// device.json is where it lives (`--state-dir` when it is somewhere else).
|
|
196
|
+
const deviceToken = identity.device_token;
|
|
197
|
+
|
|
198
|
+
const health = await get('/healthz');
|
|
199
|
+
const devices = health.devices ?? health.device_count ?? '?';
|
|
200
|
+
|
|
201
|
+
const commit = await headCommit({ cwd: project });
|
|
202
|
+
const tree = await treeFingerprint({ cwd: project });
|
|
203
|
+
if (!commit || !tree.fingerprint) {
|
|
204
|
+
console.error(`cannot fingerprint ${project}: commit=${commit} tree=${tree.fingerprint} error=${tree.error ?? 'none'}`);
|
|
205
|
+
console.error('every machine is compared against these anchors, so the project must be a git work tree');
|
|
206
|
+
console.error('sitting at the same commit as the machines you are testing.');
|
|
207
|
+
process.exit(2);
|
|
208
|
+
}
|
|
209
|
+
|
|
210
|
+
const cwdRel = '.';
|
|
211
|
+
const body = {
|
|
212
|
+
mode: 'replicate',
|
|
213
|
+
command_argv: commandArgv,
|
|
214
|
+
cwd_rel: cwdRel,
|
|
215
|
+
index_total: 1,
|
|
216
|
+
timeout_ms: timeoutMs,
|
|
217
|
+
write: false,
|
|
218
|
+
base_commit: commit,
|
|
219
|
+
base_tree: tree.fingerprint,
|
|
220
|
+
requirements: { toolchain: {}, platform: [] },
|
|
221
|
+
compare_policy: { strip_ansi: true, normalize_crlf: true, strip_trailing_blank_lines: true, redact: [] },
|
|
222
|
+
halt: 'never',
|
|
223
|
+
created_by: identity.name ?? 'acceptance',
|
|
224
|
+
// THE PUNCH FACTS ARE THE WHOLE POINT OF THIS SCRIPT.
|
|
225
|
+
//
|
|
226
|
+
// A v0.4.x-style body (`p2p: {mode}`) exercises neither A nor B: the executor only dials back when
|
|
227
|
+
// the offer actually publishes a punch id (or candidates), so a task without one lands on rung 5 by
|
|
228
|
+
// construction and would make this script report the old behaviour forever. Measured, by the very
|
|
229
|
+
// work that added the rungs. `mintPunchId` is imported from the node's own module rather than
|
|
230
|
+
// re-implemented, so the script mints exactly what a real dispatcher mints.
|
|
231
|
+
p2p: { mode: 'auto', punch: mintPunchId() },
|
|
232
|
+
...(identity.machine_id ? { origin_machine_id: identity.machine_id } : {}),
|
|
233
|
+
};
|
|
234
|
+
|
|
235
|
+
const created = await fetch(`${rabbit}/v1/task`, { method: 'POST', headers, body: JSON.stringify(body) });
|
|
236
|
+
if (!created.ok) {
|
|
237
|
+
console.error(`POST /v1/task -> ${created.status} ${(await created.text()).slice(0, 300)}`);
|
|
238
|
+
process.exit(2);
|
|
239
|
+
}
|
|
240
|
+
const { task_id: taskId, seq } = await created.json();
|
|
241
|
+
if (!asJson) {
|
|
242
|
+
console.log(`task ${taskId} (seq ${seq}) -- ${commandArgv.join(' ')}`);
|
|
243
|
+
console.log(`anchors ${commit.slice(0, 12)} / ${tree.fingerprint.slice(0, 12)} (${tree.fingerprint_algo ?? 'git-temp-index-tree/v1'})`);
|
|
244
|
+
console.log(`relay ${rabbit} devices ${devices}\n`);
|
|
245
|
+
}
|
|
246
|
+
|
|
247
|
+
const TERMINAL = new Set(['done', 'refused', 'expired', 'failed']);
|
|
248
|
+
let aggregate = null;
|
|
249
|
+
const started = Date.now();
|
|
250
|
+
while (Date.now() - started < timeoutMs) {
|
|
251
|
+
const status = await get(`/v1/tasks/${encodeURIComponent(taskId)}`);
|
|
252
|
+
aggregate = status.aggregate;
|
|
253
|
+
const leases = status.leases ?? [];
|
|
254
|
+
if (leases.length > 0 && leases.every((lease) => TERMINAL.has(lease.state))) break;
|
|
255
|
+
if (aggregate && aggregate.state && aggregate.state !== 'pending' && leases.length === 0) break;
|
|
256
|
+
await new Promise((resolve) => {
|
|
257
|
+
setTimeout(resolve, pollMs);
|
|
258
|
+
});
|
|
259
|
+
}
|
|
260
|
+
|
|
261
|
+
if (!aggregate) {
|
|
262
|
+
console.error('no aggregate came back before the timeout');
|
|
263
|
+
process.exit(1);
|
|
264
|
+
}
|
|
265
|
+
|
|
266
|
+
if (asJson) {
|
|
267
|
+
console.log(JSON.stringify({ task_id: taskId, anchors: { base_commit: commit, base_tree: tree.fingerprint }, aggregate }, null, 2));
|
|
268
|
+
} else {
|
|
269
|
+
// `|| 'unknown'` rather than `??`: an empty `states` array joins to '' and `??` would happily
|
|
270
|
+
// print nothing, which is the one thing a verdict line must not do.
|
|
271
|
+
const verdict = aggregate.state ?? aggregate.verdict ?? ((aggregate.states ?? []).join(', ') || 'unknown');
|
|
272
|
+
console.log(`verdict ${verdict}\n`);
|
|
273
|
+
const rows = (aggregate.machines ?? []).map((machine) => {
|
|
274
|
+
const verdict = classify(machine);
|
|
275
|
+
return {
|
|
276
|
+
machine: machine.machine_name ?? machine.machine_id,
|
|
277
|
+
outcome: machine.outcome ?? '',
|
|
278
|
+
exit: machine.exit_code ?? '',
|
|
279
|
+
rung: verdict.rung ?? '-',
|
|
280
|
+
path: machine.transport ?? '-',
|
|
281
|
+
offer: machine.p2p?.offer_path ?? '-',
|
|
282
|
+
note: verdict.label + (verdict.reason ? ` -- ${verdict.reason}` : ''),
|
|
283
|
+
};
|
|
284
|
+
});
|
|
285
|
+
const width = Math.max(...rows.map((row) => row.machine.length), 7);
|
|
286
|
+
console.log(`${'machine'.padEnd(width)} outcome exit rung path offer note`);
|
|
287
|
+
for (const row of rows) {
|
|
288
|
+
console.log(
|
|
289
|
+
`${String(row.machine).padEnd(width)} ${String(row.outcome).padEnd(7)} ${String(row.exit).padEnd(4)} ` +
|
|
290
|
+
`${String(row.rung).padEnd(4)} ${String(row.path).padEnd(5)} ${String(row.offer).padEnd(5)} ${row.note}`,
|
|
291
|
+
);
|
|
292
|
+
}
|
|
293
|
+
const ladderRungs = rows.map((row) => row.rung).filter((rung) => typeof rung === 'number');
|
|
294
|
+
console.log(
|
|
295
|
+
`\nladder: ${ladderRungs.length > 0 ? ladderRungs.join(', ') : 'no rung recorded'} ` +
|
|
296
|
+
`(5 = relay; see deploy/networking/README.md for what each rung claims)`,
|
|
297
|
+
);
|
|
298
|
+
if (aggregate.notes?.length) for (const note of aggregate.notes) console.log(`note: ${note}`);
|
|
299
|
+
}
|
|
300
|
+
|
|
301
|
+
const fellBack = (aggregate.machines ?? []).some((machine) => classify(machine).rung === 5);
|
|
302
|
+
const incomplete = (aggregate.machines ?? []).some((machine) => machine.outcome !== 'ok');
|
|
303
|
+
if (requireDirect && (fellBack || incomplete)) {
|
|
304
|
+
console.error('\n--require-direct: at least one machine did not take a direct path (or did not answer)');
|
|
305
|
+
process.exit(1);
|
|
306
|
+
}
|