@twinsearth/w2m-dsh-plugin 0.14.0 → 0.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +91 -0
- package/package.json +5 -4
- package/src/agent/fleet-scale.mjs +63 -0
- package/src/agent/fleet-state.mjs +167 -0
- package/src/plugin/tools.mjs +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,97 @@ All notable changes to this project are documented here. The format follows
|
|
|
4
4
|
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project
|
|
5
5
|
adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
6
|
|
|
7
|
+
## [0.16.0] — 2026-10-15
|
|
8
|
+
|
|
9
|
+
**A non-zero exit code is not a fleet failure, and this session conflated them repeatedly.**
|
|
10
|
+
|
|
11
|
+
Twelve real task reports were fetched one request at a time — the only correct input, since `w2m_history`
|
|
12
|
+
records current lease state rather than outcomes — and the measured result was **12 acting leases, 12 `done`,
|
|
13
|
+
0 expired, 0 refused**. Yet several runs in the same session produced no result at all. Both facts are true.
|
|
14
|
+
|
|
15
|
+
They are different failures, and the difference decides where a reader looks:
|
|
16
|
+
|
|
17
|
+
| Observation | What it means | Where the fault is |
|
|
18
|
+
|---|---|---|
|
|
19
|
+
| `exit_code: 70`, `lease_state: done` | **The command ran and reported 70.** The fleet worked. | the command |
|
|
20
|
+
| `exit_code: null`, `lease_state: expired` | **No result was ever produced.** | the machine, the lease, or the network |
|
|
21
|
+
|
|
22
|
+
`classifyLease()` and `isFleetFailure()` now encode that distinction, and **`exit_code` is the decisive field,
|
|
23
|
+
not the lease state** — a lease can be `done` while its command returned non-zero, which is a *successful
|
|
24
|
+
execution of a failing command*. `isFleetFailure` returns true only for the no-result case; a refusal is the
|
|
25
|
+
request being wrong, and an observer was never asked to act.
|
|
26
|
+
|
|
27
|
+
A rate that merged the two would report a healthy fleet as broken, or a broken one as healthy, depending on
|
|
28
|
+
which it happened to count.
|
|
29
|
+
|
|
30
|
+
### The measured fleet, this round
|
|
31
|
+
|
|
32
|
+
| Measure | Value | Note |
|
|
33
|
+
|---|---|---|
|
|
34
|
+
| acting leases | 12 of 12 | a bounded sample, fetched per task |
|
|
35
|
+
| expiry rate | **0.0 %** | 0 expired in the sample |
|
|
36
|
+
| refusal rate | **0.0 %** | 0 refused |
|
|
37
|
+
| direct share | **0.0 %** | **every task went by relay** |
|
|
38
|
+
| P2P delivery rate | **0.0 %** | no direct channel was ever established |
|
|
39
|
+
| fleet size | **3 usable of 5 paired** | two `win-desktop` silent ≈ 17.2 h |
|
|
40
|
+
|
|
41
|
+
**The 0 % direct share is the finding worth acting on**: not one task in the sample used a direct path, so the
|
|
42
|
+
entire fleet has been running on the relay at RTT **354 ms** (macmini) and **336 ms** (ToFuture) where
|
|
43
|
+
jp-shared answers in **20 ms**.
|
|
44
|
+
|
|
45
|
+
### Still not fixed
|
|
46
|
+
|
|
47
|
+
The two relay endpoints still disagree about whether the silent machines are online — `reconcile()` detects it
|
|
48
|
+
and says which to trust, but the disagreement itself needs a change on the relay, and this session has no shell
|
|
49
|
+
on that host.
|
|
50
|
+
|
|
51
|
+
**24 tests, 0 failures.**
|
|
52
|
+
|
|
53
|
+
## [0.15.0] — 2026-10-15
|
|
54
|
+
|
|
55
|
+
**The fleet is three machines, not five — and two relay endpoints disagree about it.**
|
|
56
|
+
|
|
57
|
+
### The measurement, which is the whole point of this release
|
|
58
|
+
|
|
59
|
+
`w2m_stats` and `w2m_devices` both answer "is this machine usable?" and **they do not agree**:
|
|
60
|
+
|
|
61
|
+
| Machine | `w2m_devices` | `w2m_stats` | silent for |
|
|
62
|
+
|---|---|---|---|
|
|
63
|
+
| jp-shared | online | online, RTT **6 ms** | 2.4 s |
|
|
64
|
+
| macmini | online | online, RTT **538 ms** | 4.4 s |
|
|
65
|
+
| ToFuture | online | online, RTT **122 ms** | 0 s |
|
|
66
|
+
| win-desktop ×2 | **online** | **offline, `silent`** | **61 384 s ≈ 17 h** |
|
|
67
|
+
|
|
68
|
+
**A fleet whose size is reported two ways is a fleet whose plans are wrong by a factor of 1.7** — and the
|
|
69
|
+
failure is silent, because both answers look authoritative. `src/agent/fleet-state.mjs` prefers `w2m_stats`
|
|
70
|
+
(it is the one carrying a *reason*), treats a listing without a heartbeat age as unverified, and
|
|
71
|
+
`reconcile()` reports **every machine the two endpoints disagree about** rather than picking one and staying
|
|
72
|
+
quiet.
|
|
73
|
+
|
|
74
|
+
### The second divergence, which would have flattered the fleet
|
|
75
|
+
|
|
76
|
+
`w2m_history` lists a `lease_states` map per task. **It is not the execution outcome.** Of forty tasks in the
|
|
77
|
+
sample, every one recorded macmini as `done` — while the session's own result envelopes included several
|
|
78
|
+
`expired` with no exit code at all. The map records the *current* lease state.
|
|
79
|
+
|
|
80
|
+
`historyIsOutcomeRecord()` returns `false` unconditionally with the reason, and its suite asserts on the
|
|
81
|
+
measured payload. The finding matters because of its direction: **a reliability measurement built on the
|
|
82
|
+
history listing would have shown a perfectly reliable fleet** — the most dangerous possible error in a module
|
|
83
|
+
whose whole purpose is to notice trouble.
|
|
84
|
+
|
|
85
|
+
The number to plan against is therefore **three usable machines**, and `readStats()` takes it from the
|
|
86
|
+
per-machine entries rather than from `counts.online`, while still reporting the relay's own claim rather than
|
|
87
|
+
hiding it.
|
|
88
|
+
|
|
89
|
+
### A fixture typo that invented a disagreement
|
|
90
|
+
|
|
91
|
+
Two tests failed because a machine id in a fixture was written `tob` where the payload says `tof` — so the
|
|
92
|
+
reconciler correctly reported a *third* machine claimed online but not usable. **The code was right and the
|
|
93
|
+
fixture was wrong**, which is the third time in this project that a failing test was the test's fault; the
|
|
94
|
+
argument for testing refusals is that they fail loudly enough to be told apart.
|
|
95
|
+
|
|
96
|
+
**16 tests, 0 failures.**
|
|
97
|
+
|
|
7
98
|
## [0.14.0] — 2026-10-15
|
|
8
99
|
|
|
9
100
|
**Fleet scale: the arithmetic the observations support, and the refusals where they support nothing.**
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@twinsearth/w2m-dsh-plugin",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.16.0",
|
|
4
4
|
"description": "DSH: Windows2MacOS — drive the same project on online Windows and macOS machines from one DeepSeek account, across networks and regions. P2P hole punching is the default path (relay + shared server as rendezvous, ledger and fallback), with a bundled STUN responder, Tailscale/public-VPS/tunnel shapes, Rabbit relay + per-machine Localside + eight DSH tools (replicate / split), nine aggregate verdicts and signed requests.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "lib/index.js",
|
|
@@ -113,13 +113,13 @@
|
|
|
113
113
|
}
|
|
114
114
|
},
|
|
115
115
|
"scripts": {
|
|
116
|
-
"test": "node --test --test-force-exit test/relay.test.mjs test/broadcast.test.mjs test/agent.test.mjs test/authority.test.mjs test/blocks.test.mjs test/approval-index.test.mjs test/fleet-sync.test.mjs test/health.test.mjs test/storage.test.mjs test/exit-channel.test.mjs test/council.test.mjs test/log-bounds.test.mjs test/test-report.test.mjs test/fleet-scale.test.mjs test/layer-model.test.mjs test/memory.test.mjs test/tools.test.mjs test/claims.test.mjs test/fleet-target.test.mjs test/client-bundle.test.mjs test/schedule.test.mjs test/auto-update.test.mjs test/update-source.test.mjs test/update-install.test.mjs test/update-wiring.test.mjs test/signing.test.mjs test/stress.test.mjs test/shared-config.test.mjs test/metrics.test.mjs test/metrics-route.test.mjs test/tool-cards.test.mjs test/p2p-stun.test.mjs test/p2p-transport.test.mjs test/p2p-signaling.test.mjs test/p2p-node.test.mjs test/p2p-transport-fields.test.mjs test/p2p-plugin.test.mjs test/p2p-crypto.test.mjs test/stun-server.test.mjs test/loader-entry.test.mjs test/mdns-wire.test.mjs test/mdns-discover.test.mjs test/chatops-approval.test.mjs test/chatops-policy.test.mjs test/chatops-file-safety.test.mjs test/chatops-outbound-queue.test.mjs test/chatops-commands.test.mjs test/chatops-bridge.test.mjs test/chatops-config-route.test.mjs test/chatops-apply.test.mjs test/chatops-ilink-contract.test.mjs test/chatops-fleet.test.mjs test/chatops-account.test.mjs test/chatops-auth.test.mjs test/chatops-fleet-scope.test.mjs test/chatops-handoff.test.mjs test/chatops-fleet-wiring.test.mjs test/chatops-fleet-live.test.mjs test/chatops-takeover.test.mjs test/chatops-device-signing.test.mjs test/chatops-relay-verify.test.mjs test/chatops-durability.test.mjs test/chatops-run-command.test.mjs test/secrets-guard.test.mjs test/chatops-ilink-fleet-bridge.test.mjs test/chatops-fleet-join.test.mjs",
|
|
116
|
+
"test": "node --test --test-force-exit test/relay.test.mjs test/broadcast.test.mjs test/agent.test.mjs test/authority.test.mjs test/blocks.test.mjs test/approval-index.test.mjs test/fleet-sync.test.mjs test/health.test.mjs test/storage.test.mjs test/exit-channel.test.mjs test/council.test.mjs test/log-bounds.test.mjs test/test-report.test.mjs test/fleet-scale.test.mjs test/fleet-state.test.mjs test/layer-model.test.mjs test/memory.test.mjs test/tools.test.mjs test/claims.test.mjs test/fleet-target.test.mjs test/client-bundle.test.mjs test/schedule.test.mjs test/auto-update.test.mjs test/update-source.test.mjs test/update-install.test.mjs test/update-wiring.test.mjs test/signing.test.mjs test/stress.test.mjs test/shared-config.test.mjs test/metrics.test.mjs test/metrics-route.test.mjs test/tool-cards.test.mjs test/p2p-stun.test.mjs test/p2p-transport.test.mjs test/p2p-signaling.test.mjs test/p2p-node.test.mjs test/p2p-transport-fields.test.mjs test/p2p-plugin.test.mjs test/p2p-crypto.test.mjs test/stun-server.test.mjs test/loader-entry.test.mjs test/mdns-wire.test.mjs test/mdns-discover.test.mjs test/chatops-approval.test.mjs test/chatops-policy.test.mjs test/chatops-file-safety.test.mjs test/chatops-outbound-queue.test.mjs test/chatops-commands.test.mjs test/chatops-bridge.test.mjs test/chatops-config-route.test.mjs test/chatops-apply.test.mjs test/chatops-ilink-contract.test.mjs test/chatops-fleet.test.mjs test/chatops-account.test.mjs test/chatops-auth.test.mjs test/chatops-fleet-scope.test.mjs test/chatops-handoff.test.mjs test/chatops-fleet-wiring.test.mjs test/chatops-fleet-live.test.mjs test/chatops-takeover.test.mjs test/chatops-device-signing.test.mjs test/chatops-relay-verify.test.mjs test/chatops-durability.test.mjs test/chatops-run-command.test.mjs test/secrets-guard.test.mjs test/chatops-ilink-fleet-bridge.test.mjs test/chatops-fleet-join.test.mjs",
|
|
117
117
|
"test:e2e": "node --test --test-force-exit test/e2e.test.mjs test/crossnetwork.test.mjs test/signing-e2e.test.mjs test/outbound-only.test.mjs test/recovery.test.mjs test/pipeline-e2e.test.mjs test/p2p-agent.test.mjs",
|
|
118
118
|
"test:relay": "node --test --test-force-exit test/relay.test.mjs",
|
|
119
119
|
"test:agent": "node --test --test-force-exit test/agent.test.mjs test/memory.test.mjs",
|
|
120
120
|
"test:tools": "node --test --test-force-exit test/tools.test.mjs",
|
|
121
121
|
"test:crossnetwork": "node --test --test-force-exit test/crossnetwork.test.mjs",
|
|
122
|
-
"test:all": "node --test --test-force-exit test/relay.test.mjs test/broadcast.test.mjs test/agent.test.mjs test/authority.test.mjs test/blocks.test.mjs test/approval-index.test.mjs test/fleet-sync.test.mjs test/health.test.mjs test/storage.test.mjs test/exit-channel.test.mjs test/council.test.mjs test/log-bounds.test.mjs test/test-report.test.mjs test/fleet-scale.test.mjs test/layer-model.test.mjs test/memory.test.mjs test/tools.test.mjs test/claims.test.mjs test/fleet-target.test.mjs test/client-bundle.test.mjs test/schedule.test.mjs test/auto-update.test.mjs test/update-source.test.mjs test/update-install.test.mjs test/update-wiring.test.mjs test/signing.test.mjs test/stress.test.mjs test/e2e.test.mjs test/crossnetwork.test.mjs test/signing-e2e.test.mjs test/outbound-only.test.mjs test/recovery.test.mjs test/pipeline-e2e.test.mjs test/shared-config.test.mjs test/metrics.test.mjs test/metrics-route.test.mjs test/tool-cards.test.mjs test/p2p-stun.test.mjs test/p2p-transport.test.mjs test/p2p-signaling.test.mjs test/p2p-node.test.mjs test/p2p-transport-fields.test.mjs test/p2p-plugin.test.mjs test/p2p-agent.test.mjs test/stun-server.test.mjs test/loader-entry.test.mjs test/mdns-wire.test.mjs test/mdns-discover.test.mjs test/p2p-crypto.test.mjs test/chatops-approval.test.mjs test/chatops-policy.test.mjs test/chatops-file-safety.test.mjs test/chatops-outbound-queue.test.mjs test/chatops-commands.test.mjs test/chatops-bridge.test.mjs test/chatops-config-route.test.mjs test/chatops-apply.test.mjs test/chatops-ilink-contract.test.mjs test/chatops-fleet.test.mjs test/chatops-account.test.mjs test/chatops-auth.test.mjs test/chatops-fleet-scope.test.mjs test/chatops-handoff.test.mjs test/chatops-fleet-wiring.test.mjs test/chatops-fleet-live.test.mjs test/chatops-takeover.test.mjs test/chatops-device-signing.test.mjs test/chatops-relay-verify.test.mjs test/chatops-durability.test.mjs test/chatops-run-command.test.mjs test/secrets-guard.test.mjs test/chatops-ilink-fleet-bridge.test.mjs test/chatops-fleet-join.test.mjs",
|
|
122
|
+
"test:all": "node --test --test-force-exit test/relay.test.mjs test/broadcast.test.mjs test/agent.test.mjs test/authority.test.mjs test/blocks.test.mjs test/approval-index.test.mjs test/fleet-sync.test.mjs test/health.test.mjs test/storage.test.mjs test/exit-channel.test.mjs test/council.test.mjs test/log-bounds.test.mjs test/test-report.test.mjs test/fleet-scale.test.mjs test/fleet-state.test.mjs test/layer-model.test.mjs test/memory.test.mjs test/tools.test.mjs test/claims.test.mjs test/fleet-target.test.mjs test/client-bundle.test.mjs test/schedule.test.mjs test/auto-update.test.mjs test/update-source.test.mjs test/update-install.test.mjs test/update-wiring.test.mjs test/signing.test.mjs test/stress.test.mjs test/e2e.test.mjs test/crossnetwork.test.mjs test/signing-e2e.test.mjs test/outbound-only.test.mjs test/recovery.test.mjs test/pipeline-e2e.test.mjs test/shared-config.test.mjs test/metrics.test.mjs test/metrics-route.test.mjs test/tool-cards.test.mjs test/p2p-stun.test.mjs test/p2p-transport.test.mjs test/p2p-signaling.test.mjs test/p2p-node.test.mjs test/p2p-transport-fields.test.mjs test/p2p-plugin.test.mjs test/p2p-agent.test.mjs test/stun-server.test.mjs test/loader-entry.test.mjs test/mdns-wire.test.mjs test/mdns-discover.test.mjs test/p2p-crypto.test.mjs test/chatops-approval.test.mjs test/chatops-policy.test.mjs test/chatops-file-safety.test.mjs test/chatops-outbound-queue.test.mjs test/chatops-commands.test.mjs test/chatops-bridge.test.mjs test/chatops-config-route.test.mjs test/chatops-apply.test.mjs test/chatops-ilink-contract.test.mjs test/chatops-fleet.test.mjs test/chatops-account.test.mjs test/chatops-auth.test.mjs test/chatops-fleet-scope.test.mjs test/chatops-handoff.test.mjs test/chatops-fleet-wiring.test.mjs test/chatops-fleet-live.test.mjs test/chatops-takeover.test.mjs test/chatops-device-signing.test.mjs test/chatops-relay-verify.test.mjs test/chatops-durability.test.mjs test/chatops-run-command.test.mjs test/secrets-guard.test.mjs test/chatops-ilink-fleet-bridge.test.mjs test/chatops-fleet-join.test.mjs",
|
|
123
123
|
"pack": "node scripts/pack.mjs",
|
|
124
124
|
"verify:join": "node scripts/verify-join.mjs",
|
|
125
125
|
"lint": "eslint .",
|
|
@@ -141,7 +141,8 @@
|
|
|
141
141
|
"test:log-bounds": "node --test --test-force-exit test/log-bounds.test.mjs",
|
|
142
142
|
"layer-check": "node scripts/layer-check.mjs",
|
|
143
143
|
"test:test-report": "node --test --test-force-exit test/test-report.test.mjs",
|
|
144
|
-
"test:fleet-scale": "node --test --test-force-exit test/fleet-scale.test.mjs"
|
|
144
|
+
"test:fleet-scale": "node --test --test-force-exit test/fleet-scale.test.mjs",
|
|
145
|
+
"test:fleet-state": "node --test --test-force-exit test/fleet-state.test.mjs"
|
|
145
146
|
},
|
|
146
147
|
"keywords": [
|
|
147
148
|
"dsh-bundle",
|
|
@@ -48,6 +48,69 @@ export const FAILURE_STATES = Object.freeze(['expired', 'timeout']);
|
|
|
48
48
|
/** Outcomes that are not the fleet's fault, so they are counted separately. */
|
|
49
49
|
export const REQUEST_FAULT_STATES = Object.freeze(['refused']);
|
|
50
50
|
|
|
51
|
+
/**
|
|
52
|
+
* How a lease ended, told apart properly.
|
|
53
|
+
*
|
|
54
|
+
* ## The conflation this exists to prevent
|
|
55
|
+
*
|
|
56
|
+
* This session repeatedly read `exit_code: 70` as a failure and `lease_state: expired` as the same kind of
|
|
57
|
+
* failure. **They are not the same kind of thing at all**, and the difference decides where a reader looks:
|
|
58
|
+
*
|
|
59
|
+
* * `exit_code: 70` with `lease_state: done` means **the command ran and reported 70**. The fleet worked.
|
|
60
|
+
* The number is the command's answer, and the machine is fine.
|
|
61
|
+
* * `exit_code: null` with `lease_state: expired` means **no result was ever produced**. The command may not
|
|
62
|
+
* have started. Nothing is known about what it did, and the machine — or the lease, or the network — is
|
|
63
|
+
* where the fault is.
|
|
64
|
+
*
|
|
65
|
+
* Measured: twelve consecutive real reports showed **12 acting leases, 12 `done`, 0 expired** — while several
|
|
66
|
+
* runs in the same session had produced no result at all. Both facts are true; they are different failures.
|
|
67
|
+
* A rate that merged them would report a healthy fleet as broken, or a broken one as healthy, depending on
|
|
68
|
+
* which it happened to count.
|
|
69
|
+
*/
|
|
70
|
+
export const LEASE_OUTCOME = Object.freeze({
|
|
71
|
+
RAN: 'ran', // a result was produced, whatever it said
|
|
72
|
+
NO_RESULT: 'no_result', // nothing came back; the command's behaviour is unknown
|
|
73
|
+
REFUSED: 'refused', // the allow-list rejected it before it could run
|
|
74
|
+
OBSERVED_ONLY: 'observed', // the machine was told the outcome without being asked to act
|
|
75
|
+
UNKNOWN: 'unknown',
|
|
76
|
+
});
|
|
77
|
+
|
|
78
|
+
/**
|
|
79
|
+
* Classify one machine entry from a task report.
|
|
80
|
+
*
|
|
81
|
+
* `exit_code` is the decisive field, **not** the lease state: a lease can be `done` while its command
|
|
82
|
+
* returned non-zero, and that is a successful execution of a failing command.
|
|
83
|
+
*/
|
|
84
|
+
export function classifyLease(entry) {
|
|
85
|
+
if (entry === null || typeof entry !== 'object') return { outcome: LEASE_OUTCOME.UNKNOWN, ran: false, exit_code: null, detail: 'no machine entry' };
|
|
86
|
+
const state = String(entry.lease_state ?? entry.outcome ?? '');
|
|
87
|
+
const code = entry.exit_code;
|
|
88
|
+
if (state === 'observing') return { outcome: LEASE_OUTCOME.OBSERVED_ONLY, ran: false, exit_code: null, detail: 'told the outcome, not asked to act' };
|
|
89
|
+
if (state === 'refused') return { outcome: LEASE_OUTCOME.REFUSED, ran: false, exit_code: null, detail: entry.refusal_reason ?? 'refused' };
|
|
90
|
+
if (state === 'done') {
|
|
91
|
+
return {
|
|
92
|
+
outcome: LEASE_OUTCOME.RAN,
|
|
93
|
+
ran: true,
|
|
94
|
+
exit_code: Number.isFinite(code) ? code : null,
|
|
95
|
+
detail: Number.isFinite(code) && code !== 0
|
|
96
|
+
? `ran and reported ${code} -- the command's answer, not a fleet failure`
|
|
97
|
+
: 'ran and reported success',
|
|
98
|
+
};
|
|
99
|
+
}
|
|
100
|
+
if (state === 'expired' || state === 'timeout') {
|
|
101
|
+
return { outcome: LEASE_OUTCOME.NO_RESULT, ran: false, exit_code: null, detail: 'no result was produced; what the command did is unknown' };
|
|
102
|
+
}
|
|
103
|
+
return { outcome: LEASE_OUTCOME.UNKNOWN, ran: false, exit_code: null, detail: `unrecognised lease state ${JSON.stringify(state)}` };
|
|
104
|
+
}
|
|
105
|
+
|
|
106
|
+
/**
|
|
107
|
+
* Whether a non-zero exit code is a fleet failure. It is not, and this function exists so a caller does not
|
|
108
|
+
* have to decide that for itself each time.
|
|
109
|
+
*/
|
|
110
|
+
export function isFleetFailure(entry) {
|
|
111
|
+
return classifyLease(entry).outcome === LEASE_OUTCOME.NO_RESULT;
|
|
112
|
+
}
|
|
113
|
+
|
|
51
114
|
/**
|
|
52
115
|
* Summarise a set of task reports.
|
|
53
116
|
*
|
|
@@ -0,0 +1,167 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Fleet state: what the relay's own accounting does and does not say, and the one place it disagrees with itself.
|
|
3
|
+
*
|
|
4
|
+
* ## The disagreement, measured
|
|
5
|
+
*
|
|
6
|
+
* Two relay endpoints answer "is this machine usable?" and they do not agree:
|
|
7
|
+
*
|
|
8
|
+
* * `w2m_devices` reports every paired machine as `state: "online"` -- including two that had been silent
|
|
9
|
+
* for **61,384 seconds (about 17 hours)** at the time of measurement.
|
|
10
|
+
* * `w2m_stats` reports the same two as `online: false` with `offline_reason: "silent"`, and counts
|
|
11
|
+
* `online: 3` out of `devices: 5`.
|
|
12
|
+
*
|
|
13
|
+
* A caller that asked the first one would plan work for five machines and get three. **A fleet whose size is
|
|
14
|
+
* reported two ways is a fleet whose plans are wrong by a factor of 1.7**, and the failure is silent because
|
|
15
|
+
* both answers look authoritative.
|
|
16
|
+
*
|
|
17
|
+
* So this module prefers `w2m_stats` — it is the one that carries a *reason* — and treats a device listing
|
|
18
|
+
* without a heartbeat age as **unverified rather than online**.
|
|
19
|
+
*
|
|
20
|
+
* ## The second disagreement, which is about history
|
|
21
|
+
*
|
|
22
|
+
* `w2m_history` lists, per task, a `lease_states` map. It is NOT the execution outcome: a task that expired
|
|
23
|
+
* mid-run can show `done` afterwards, because the map records the *current* lease state rather than what
|
|
24
|
+
* happened. Measured directly: of forty tasks in the sample, every one recorded macmini as `done`, while the
|
|
25
|
+
* session's own result envelopes included several `expired` with no exit code at all.
|
|
26
|
+
*
|
|
27
|
+
* Extracting the outcome requires `w2m_report`, which is a request per task. **A history listing is therefore
|
|
28
|
+
* a poor input for a reliability measurement, and using it as one would have produced a fleet that looked
|
|
29
|
+
* perfectly reliable** — the most dangerous possible error in a module whose purpose is to notice trouble.
|
|
30
|
+
*/
|
|
31
|
+
|
|
32
|
+
/** How a machine's usability was determined, in descending order of trust. */
|
|
33
|
+
export const EVIDENCE = Object.freeze({
|
|
34
|
+
HEARTBEAT_WITH_RTT: 'a live heartbeat and a measured round trip',
|
|
35
|
+
HEARTBEAT_ALONE: 'a live heartbeat with no round trip measured',
|
|
36
|
+
STALE_OR_ABSENT: 'no recent heartbeat',
|
|
37
|
+
UNKNOWN: 'the relay did not say',
|
|
38
|
+
});
|
|
39
|
+
|
|
40
|
+
/**
|
|
41
|
+
* Classify one machine from a `w2m_stats` entry.
|
|
42
|
+
*
|
|
43
|
+
* `silent_ms` is the decisive field. A machine that is "online" but silent is not online, and the reason is
|
|
44
|
+
* reported alongside so a caller can distinguish "went away" from "never arrived".
|
|
45
|
+
*/
|
|
46
|
+
export function classifyMachine(entry) {
|
|
47
|
+
if (entry === null || typeof entry !== 'object' || typeof entry.machine_id !== 'string') {
|
|
48
|
+
return { machine_id: null, usable: false, evidence: EVIDENCE.UNKNOWN, reason: 'no machine entry', detail: 'the relay did not describe this machine' };
|
|
49
|
+
}
|
|
50
|
+
const online = entry.online === true;
|
|
51
|
+
const rtt = entry.rtt_ms;
|
|
52
|
+
const stale = entry.rtt_stale === true;
|
|
53
|
+
if (online && !stale && Number.isFinite(rtt)) {
|
|
54
|
+
return { machine_id: entry.machine_id, name: entry.machine_name ?? null, usable: true, evidence: EVIDENCE.HEARTBEAT_WITH_RTT, rtt_ms: rtt, reason: null, detail: `live, ${rtt} ms` };
|
|
55
|
+
}
|
|
56
|
+
if (online && !stale) {
|
|
57
|
+
return { machine_id: entry.machine_id, name: entry.machine_name ?? null, usable: true, evidence: EVIDENCE.HEARTBEAT_ALONE, rtt_ms: null, reason: null, detail: 'live, round trip not measured' };
|
|
58
|
+
}
|
|
59
|
+
return {
|
|
60
|
+
machine_id: entry.machine_id,
|
|
61
|
+
name: entry.machine_name ?? null,
|
|
62
|
+
usable: false,
|
|
63
|
+
evidence: EVIDENCE.STALE_OR_ABSENT,
|
|
64
|
+
rtt_ms: Number.isFinite(rtt) && !stale ? rtt : null,
|
|
65
|
+
reason: entry.offline_reason ?? 'not online',
|
|
66
|
+
silent_ms: Number.isFinite(entry.silent_ms) ? entry.silent_ms : null,
|
|
67
|
+
detail: entry.offline_reason === 'silent'
|
|
68
|
+
? `silent for ${Math.round((entry.silent_ms ?? 0) / 1000)}s`
|
|
69
|
+
: String(entry.offline_reason ?? 'not online'),
|
|
70
|
+
};
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
/**
|
|
74
|
+
* Read a `w2m_stats` payload.
|
|
75
|
+
*
|
|
76
|
+
* @returns {{
|
|
77
|
+
* paired: number, online: number, usable: number, silent: Array<object>,
|
|
78
|
+
* usable_machines: Array<object>, detail: string
|
|
79
|
+
* }}
|
|
80
|
+
*/
|
|
81
|
+
export function readStats(stats) {
|
|
82
|
+
const machines = Array.isArray(stats?.machines) ? stats.machines : [];
|
|
83
|
+
const classified = machines.map(classifyMachine);
|
|
84
|
+
const paired = Number.isFinite(stats?.counts?.devices) ? stats.counts.devices : machines.length;
|
|
85
|
+
const usable = classified.filter((m) => m.usable);
|
|
86
|
+
const silent = classified.filter((m) => !m.usable);
|
|
87
|
+
// The number a caller should plan against is the number of machines that can actually run something.
|
|
88
|
+
return {
|
|
89
|
+
paired,
|
|
90
|
+
online: Number.isFinite(stats?.counts?.online) ? stats.counts.online : usable.length,
|
|
91
|
+
usable: usable.length,
|
|
92
|
+
usable_machines: usable,
|
|
93
|
+
silent,
|
|
94
|
+
runner_up_rtt_ms: usable.filter((m) => Number.isFinite(m.rtt_ms)).sort((a, b) => a.rtt_ms - b.rtt_ms)[0]?.rtt_ms ?? null,
|
|
95
|
+
detail: `${usable.length} of ${paired} paired machine(s) can run work`,
|
|
96
|
+
};
|
|
97
|
+
}
|
|
98
|
+
|
|
99
|
+
/**
|
|
100
|
+
* Compare a device listing against the stats reading, and report every machine they disagree about.
|
|
101
|
+
*
|
|
102
|
+
* **The disagreement is the product.** A caller that saw only the device listing would believe the fleet is
|
|
103
|
+
* larger than it is; a caller that saw only the stats would not know the listing claimed otherwise. Reporting
|
|
104
|
+
* the gap makes the gap fixable, and until it is fixed the smaller number is the true one.
|
|
105
|
+
*/
|
|
106
|
+
export function reconcile(devices, stats) {
|
|
107
|
+
const listed = Array.isArray(devices?.devices) ? devices.devices : [];
|
|
108
|
+
const reading = readStats(stats);
|
|
109
|
+
const usableIds = new Set(reading.usable_machines.map((m) => m.machine_id));
|
|
110
|
+
const claimedOnline = listed.filter((d) => d?.state === 'online');
|
|
111
|
+
const disagreements = claimedOnline
|
|
112
|
+
.filter((d) => !usableIds.has(d.machine_id))
|
|
113
|
+
.map((d) => {
|
|
114
|
+
const m = reading.silent.find((s) => s.machine_id === d.machine_id);
|
|
115
|
+
return {
|
|
116
|
+
machine_id: d.machine_id,
|
|
117
|
+
name: d.name ?? null,
|
|
118
|
+
devices_says: 'online',
|
|
119
|
+
stats_says: m?.reason ?? 'not usable',
|
|
120
|
+
silent_ms: m?.silent_ms ?? null,
|
|
121
|
+
};
|
|
122
|
+
});
|
|
123
|
+
return {
|
|
124
|
+
devices_claims_online: claimedOnline.length,
|
|
125
|
+
stats_says_usable: reading.usable,
|
|
126
|
+
disagreements,
|
|
127
|
+
// The trust decision, stated as a field rather than left to the reader.
|
|
128
|
+
trust: 'stats',
|
|
129
|
+
reason: disagreements.length === 0
|
|
130
|
+
? 'the two endpoints agree'
|
|
131
|
+
: `a device listing reports a machine as online when it has not been heard from; the stats reading carries a reason and is the one to plan against`,
|
|
132
|
+
detail: `${claimedOnline.length} claimed online, ${reading.usable} usable, ${disagreements.length} disagreement(s)`,
|
|
133
|
+
};
|
|
134
|
+
}
|
|
135
|
+
|
|
136
|
+
/**
|
|
137
|
+
* Whether a `w2m_history` payload can be used as a reliability measurement.
|
|
138
|
+
*
|
|
139
|
+
* It cannot, and this function exists so that conclusion is a checked fact rather than a comment. Every entry
|
|
140
|
+
* in the sample recorded the same state for the same machine, which is exactly what a current-state map looks
|
|
141
|
+
* like and exactly what an outcome record does not.
|
|
142
|
+
*
|
|
143
|
+
* @returns {{usable_as_outcome_record: boolean, reason: string, homogeneous: Array<string>}}
|
|
144
|
+
*/
|
|
145
|
+
export function historyIsOutcomeRecord(history) {
|
|
146
|
+
const tasks = Array.isArray(history?.tasks) ? history.tasks : [];
|
|
147
|
+
if (tasks.length === 0) {
|
|
148
|
+
return { usable_as_outcome_record: false, reason: 'no tasks in the sample', homogeneous: [] };
|
|
149
|
+
}
|
|
150
|
+
// Per machine, how many distinct states appear across tasks. A machine at "done" in every task, including
|
|
151
|
+
// tasks that expired, is a current-state map rather than a record of what happened.
|
|
152
|
+
const states = new Map();
|
|
153
|
+
for (const task of tasks) {
|
|
154
|
+
for (const [id, state] of Object.entries(task?.lease_states ?? {})) {
|
|
155
|
+
if (!states.has(id)) states.set(id, new Set());
|
|
156
|
+
states.get(id).add(String(state));
|
|
157
|
+
}
|
|
158
|
+
}
|
|
159
|
+
const homogeneous = [...states.entries()].filter(([, set]) => set.size === 1).map(([id]) => id);
|
|
160
|
+
return {
|
|
161
|
+
usable_as_outcome_record: false,
|
|
162
|
+
homogeneous,
|
|
163
|
+
reason: homogeneous.length === states.size && states.size > 0
|
|
164
|
+
? `every one of the ${states.size} machine(s) holds a single state across ${tasks.length} task(s) -- a current-state map, not an outcome record; use w2m_report per task instead`
|
|
165
|
+
: `a lease_states map reports the current state, not what happened; use w2m_report per task instead`,
|
|
166
|
+
};
|
|
167
|
+
}
|
package/src/plugin/tools.mjs
CHANGED
|
@@ -185,7 +185,7 @@ const MAX_STAGES = 16;
|
|
|
185
185
|
* `package.json` during `scripts/pack.mjs` rather than hand-maintained, and `W2M_PLUGIN_VERSION`
|
|
186
186
|
* overrides it at runtime for tests and packagers.
|
|
187
187
|
*/
|
|
188
|
-
const PLUGIN_VERSION = '0.
|
|
188
|
+
const PLUGIN_VERSION = '0.16.0';
|
|
189
189
|
|
|
190
190
|
/** How long any single HTTP call may take before it is aborted, in milliseconds. */
|
|
191
191
|
const DEFAULT_REQUEST_TIMEOUT_MS = 15_000;
|