@twinsearth/w2m-dsh-plugin 1.6.5 → 1.6.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +3 -3
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +30 -0
- package/codex/mcp-servers.toml +1 -1
- package/docs/acceptance/1.6.6/RELAY-FAILOVER.json +22 -0
- package/docs/acceptance/1.6.7/FAILOVER-CLEANUP-FIX.json +22 -0
- package/package.json +3 -3
- package/plugin.json +1 -1
- package/src/plugin/relay-failover.mjs +158 -0
- package/src/plugin/tools.mjs +94 -3
|
@@ -5,14 +5,14 @@
|
|
|
5
5
|
},
|
|
6
6
|
"metadata": {
|
|
7
7
|
"description": "The w2m fleet plugins: one account, many machines, any of Windows, macOS or Linux.",
|
|
8
|
-
"version": "1.6.
|
|
8
|
+
"version": "1.6.7"
|
|
9
9
|
},
|
|
10
10
|
"plugins": [
|
|
11
11
|
{
|
|
12
12
|
"name": "w2m-fleet",
|
|
13
13
|
"source": "./",
|
|
14
14
|
"description": "Read fleet state and dispatch commands across the machines registered to one account. Read-only MCP surface: the five exposed tools cannot change anything.",
|
|
15
|
-
"version": "1.6.
|
|
15
|
+
"version": "1.6.7",
|
|
16
16
|
"author": {
|
|
17
17
|
"name": "TwinsEarth"
|
|
18
18
|
},
|
|
@@ -26,5 +26,5 @@
|
|
|
26
26
|
]
|
|
27
27
|
}
|
|
28
28
|
],
|
|
29
|
-
"version": "1.6.
|
|
29
|
+
"version": "1.6.7"
|
|
30
30
|
}
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "w2m-fleet",
|
|
3
3
|
"description": "Drive the same project on online Windows, macOS and Linux machines from one account: read fleet state, dispatch commands and read results. The MCP surface is deliberately read-only.",
|
|
4
|
-
"version": "1.6.
|
|
4
|
+
"version": "1.6.7",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "TwinsEarth"
|
|
7
7
|
},
|
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,36 @@
|
|
|
7
7
|
|
|
8
8
|
All notable changes to this project are documented here.
|
|
9
9
|
|
|
10
|
+
## [1.6.7] — 2026-10-24
|
|
11
|
+
|
|
12
|
+
**修复 v1.6.6 failover 的阻塞 bug:远端 relay 模式下清理函数抛错,切换后无法重启 stack。**
|
|
13
|
+
|
|
14
|
+
现场演练(停主 relay 180s)暴露:机器掉线后没有切到备用,恢复主 relay 后才重新上线。根因:
|
|
15
|
+
|
|
16
|
+
- `startLocalStack` 的清理函数无条件执行 `await relay.close()`;而 `relay` 只在"本进程自建本地 relay"
|
|
17
|
+
时才非空,远端 relay 模式下是 `null` → 抛 TypeError;
|
|
18
|
+
- failover 的切换流程 `await stop()` 抛错后被 catch 吞掉,**后续的 restart 从未执行**(stack 停了没起来)。
|
|
19
|
+
|
|
20
|
+
修复:
|
|
21
|
+
- 清理函数 null-safe/idempotent:仅当 `relay` 是带 `close()` 的对象才调用;`agent.stop()` 失败也继续清理。
|
|
22
|
+
- 切换流程:`stop()` 失败不再跳过 restart;并写诊断日志 `xclient/relay-failover.log`(每轮探测/切换/重启错误)。
|
|
23
|
+
- 测试:relay-failover 5/5、relay 200/200;证据 `docs/acceptance/1.6.7/FAILOVER-CLEANUP-FIX.json`
|
|
24
|
+
(fleet task `01M5ANYMVMFSSP3`)。
|
|
25
|
+
## [1.6.6] — 2026-10-24
|
|
26
|
+
|
|
27
|
+
**relay 自动 failover:主 relay 故障时自动切到备用,恢复后自动切回。**
|
|
28
|
+
|
|
29
|
+
现场事故(2026-10-24):主 relay 主机整体不可达(8787 与 SSH 22 都超时),备用 relay 一直健康,
|
|
30
|
+
但每台机器仍死连主 relay——备用链路只是常驻成员,不会接管,编队停摆。
|
|
31
|
+
|
|
32
|
+
- 新增 `src/plugin/relay-failover.mjs`:非对称滞回状态机(**连续 3 次**探测失败才切走,**连续 5 次**
|
|
33
|
+
健康才切回;没有可达备选则不切)。纯函数,边界可单测。
|
|
34
|
+
- `startLocalStack` 外包一层 `startFailoverStack`:每 10s 探测池内各 relay `/healthz`;
|
|
35
|
+
决策为切换时,停止当前 in-process stack,把 `cfg.rabbitUrl` 设为备 relay 并重启 stack——
|
|
36
|
+
**切换是重新入队(fleet key 重新配对),不是重连**(device token 是按 relay 签发的)。
|
|
37
|
+
- `w2m_status` 新增 `failover` 快照:primary/active/pool、失败计数、切换次数、最近切换原因与历史。
|
|
38
|
+
- 测试:relay-failover 5/5、relay 200/200;证据 `docs/acceptance/1.6.6/RELAY-FAILOVER.json`
|
|
39
|
+
(fleet task `01M5ANYMVM9GEKQ`)。
|
|
10
40
|
## [1.6.5] — 2026-10-24
|
|
11
41
|
|
|
12
42
|
**Windows 后台自动化不再弹窗:VBScript 隐藏包装 + 计划任务工具 + 文档。**
|
package/codex/mcp-servers.toml
CHANGED
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
{
|
|
2
|
+
"version": "1.6.6",
|
|
3
|
+
"case": "RELAY-FAILOVER",
|
|
4
|
+
"kind": "positive",
|
|
5
|
+
"machines": [
|
|
6
|
+
"b53a37f4-1464-4512-9259-33a7df091e3e"
|
|
7
|
+
],
|
|
8
|
+
"executed_at": "2026-10-24T10:33:27.234Z",
|
|
9
|
+
"executor": "lead",
|
|
10
|
+
"requirement": "when the active relay is unreachable for 3 consecutive probes the plugin moves its in-process stack to the next healthy relay in the pool (re-joining through the fleet key), and returns to the primary after 5 consecutive healthy probes",
|
|
11
|
+
"task_id": "01M5ANYMVM9GEKQ",
|
|
12
|
+
"result": {
|
|
13
|
+
"outcome": "ok",
|
|
14
|
+
"assertions": {
|
|
15
|
+
"failover_suite": "5/5 pass",
|
|
16
|
+
"hysteresis": "3 failures to leave, 5 successes to return",
|
|
17
|
+
"no_alternative": "stays put",
|
|
18
|
+
"status": "failover snapshot exposed"
|
|
19
|
+
}
|
|
20
|
+
},
|
|
21
|
+
"note": "suites ran as a real fleet task; live drill follows the release"
|
|
22
|
+
}
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
{
|
|
2
|
+
"version": "1.6.7",
|
|
3
|
+
"case": "FAILOVER-CLEANUP-FIX",
|
|
4
|
+
"kind": "positive",
|
|
5
|
+
"machines": [
|
|
6
|
+
"b53a37f4-1464-4512-9259-33a7df091e3e"
|
|
7
|
+
],
|
|
8
|
+
"executed_at": "2026-10-24T13:31:02.803Z",
|
|
9
|
+
"executor": "lead",
|
|
10
|
+
"requirement": "a remote-relay in-process stack can be stopped and restarted for failover: the cleanup must not assume a local relay object, a failed stop must not skip the restart, and the switch must leave a diagnostics log",
|
|
11
|
+
"task_id": "01M5ANYMVMFSSP3",
|
|
12
|
+
"result": {
|
|
13
|
+
"outcome": "ok",
|
|
14
|
+
"assertions": {
|
|
15
|
+
"failover_suite": "5/5",
|
|
16
|
+
"relay_suite": "200/200",
|
|
17
|
+
"cleanup": "null-safe and idempotent",
|
|
18
|
+
"diagnostics": "xclient/relay-failover.log"
|
|
19
|
+
}
|
|
20
|
+
},
|
|
21
|
+
"note": "v1.6.6 field drill stopped the stack and never restarted it: relay.close() threw on remote-relay stacks"
|
|
22
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@twinsearth/w2m-dsh-plugin",
|
|
3
|
-
"version": "1.6.
|
|
3
|
+
"version": "1.6.7",
|
|
4
4
|
"description": "Run one project across your own Windows and macOS machines from a single DeepSeek account, through 11 w2m_* tools. Four coordination modes (replicate, split, broadcast, pipeline) and six aggregate verdicts (consistent, divergent, divergent-platform, failed, partial, unverifiable). P2P hole punching by default, with a bundled relay, STUN responder and ledger fallback, so the fleet needs no third-party service. Joining needs a pairing code plus autoStartAgent: true; installing alone does not join.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"exports": {
|
|
@@ -64,7 +64,7 @@
|
|
|
64
64
|
"pack": "node scripts/pack.mjs",
|
|
65
65
|
"reachability": "node scripts/reachability-check.mjs",
|
|
66
66
|
"reachability:classify": "node scripts/reachability-classify.mjs",
|
|
67
|
-
"test": "node --test --test-force-exit test/relay.test.mjs test/broadcast.test.mjs test/blake2s.test.mjs test/agent.test.mjs test/plan-sec-cmd.test.mjs test/authority.test.mjs test/blocks.test.mjs test/approval-index.test.mjs test/fleet-sync.test.mjs test/fleet-merge.test.mjs test/fleet-memory-e2e.test.mjs test/fleet-concurrency.test.mjs test/fleet-merge-race.test.mjs test/health.test.mjs test/storage.test.mjs test/exit-channel.test.mjs test/council.test.mjs test/council-session.test.mjs test/log-bounds.test.mjs test/test-report.test.mjs test/fleet-scale.test.mjs test/fleet-state.test.mjs test/crypto-blake2s.test.mjs test/installed-version.test.mjs test/permission.test.mjs test/machine-test-packet.test.mjs test/field-authority.test.mjs test/approval-gate.test.mjs test/authority-choice.test.mjs test/reachability-check.test.mjs test/reachability-classify.test.mjs test/accepted-orphans.test.mjs test/acceptance-doc.test.mjs test/layer-model.test.mjs test/cwd-containment.test.mjs test/cwd-outside-project.test.mjs test/memory.test.mjs test/tools.test.mjs test/claims.test.mjs test/fleet-target.test.mjs test/client-bundle.test.mjs test/schedule.test.mjs test/auto-update.test.mjs test/update-source.test.mjs test/update-install.test.mjs test/update-wiring.test.mjs test/signing.test.mjs test/stress.test.mjs test/shared-config.test.mjs test/metrics.test.mjs test/metrics-route.test.mjs test/tool-cards.test.mjs test/p2p-stun.test.mjs test/p2p-transport.test.mjs test/p2p-signaling.test.mjs test/p2p-node.test.mjs test/p2p-transport-fields.test.mjs test/p2p-plugin.test.mjs test/p2p-crypto.test.mjs test/stun-server.test.mjs test/loader-entry.test.mjs test/mdns-wire.test.mjs test/mdns-discover.test.mjs test/chatops-approval.test.mjs test/chatops-policy.test.mjs test/chatops-file-safety.test.mjs test/chatops-outbound-queue.test.mjs test/chatops-commands.test.mjs test/chatops-bridge.test.mjs test/chatops-config-route.test.mjs test/chatops-apply.test.mjs test/chatops-ilink-contract.test.mjs test/chatops-fleet.test.mjs test/chatops-account.test.mjs test/chatops-auth.test.mjs test/chatops-fleet-scope.test.mjs test/chatops-handoff.test.mjs test/chatops-fleet-wiring.test.mjs test/chatops-fleet-live.test.mjs test/chatops-takeover.test.mjs test/chatops-device-signing.test.mjs test/chatops-relay-verify.test.mjs test/chatops-durability.test.mjs test/chatops-run-command.test.mjs test/secrets-guard.test.mjs test/chatops-ilink-fleet-bridge.test.mjs test/agent-plugins.test.mjs test/memory-layer.test.mjs test/mcp-server.test.mjs test/portable-plugins.test.mjs test/mcp-http.test.mjs test/mcp-http-live.test.mjs test/vision.test.mjs test/ilink-shape.test.mjs test/chatops-fleet-join.test.mjs test/admission.test.mjs test/admission-routes.test.mjs test/join-degradation.test.mjs test/account-identity.test.mjs test/join-challenge.test.mjs test/exec-env-injection.test.mjs test/account-join.test.mjs test/fleet-messages.test.mjs test/message-shards.test.mjs test/fleet-messages-http.test.mjs test/message-notify.test.mjs test/update-channels.test.mjs test/update-prerelease.test.mjs test/session-watch.test.mjs test/session-readonly.test.mjs test/session-sync.test.mjs test/fleet-chat-identity.test.mjs test/identity-reader.test.mjs test/intents.test.mjs test/dataflow.test.mjs test/council-fleet.test.mjs test/fleet-memory.test.mjs test/memory-view.test.mjs test/schema-spec.test.mjs test/fleet-weight-parity.test.mjs test/task-deps.test.mjs test/dataflow-wiring.test.mjs test/task-graph.test.mjs test/ask-store.test.mjs test/turn.test.mjs test/fleet-members.test.mjs test/redact.test.mjs test/policy-matrix.test.mjs test/envelope-signature.test.mjs test/chat-crypto.test.mjs test/reduce-suggestion.test.mjs test/merge-hint.test.mjs test/fleet-chat-route.test.mjs test/maintenance.test.mjs test/maintenance-ferry.test.mjs test/grant.test.mjs test/grant-gate.test.mjs test/worktree.test.mjs test/patch-artifact.test.mjs test/merge-queue.test.mjs test/m3-workflow.test.mjs test/project-room.test.mjs test/project-attribution.test.mjs test/project-locks.test.mjs test/project-reviews.test.mjs test/project-notifications.test.mjs test/blob-sessions.test.mjs test/outbox.test.mjs test/reliability-metrics.test.mjs test/evidence.test.mjs test/grant-issue.test.mjs test/project-refs.test.mjs test/exec-capability.test.mjs test/secondary-link.test.mjs test/bootstrap-cli.test.mjs test/bootstrap.test.mjs test/windows-hidden-task.test.mjs",
|
|
67
|
+
"test": "node --test --test-force-exit test/relay.test.mjs test/broadcast.test.mjs test/blake2s.test.mjs test/agent.test.mjs test/plan-sec-cmd.test.mjs test/authority.test.mjs test/blocks.test.mjs test/approval-index.test.mjs test/fleet-sync.test.mjs test/fleet-merge.test.mjs test/fleet-memory-e2e.test.mjs test/fleet-concurrency.test.mjs test/fleet-merge-race.test.mjs test/health.test.mjs test/storage.test.mjs test/exit-channel.test.mjs test/council.test.mjs test/council-session.test.mjs test/log-bounds.test.mjs test/test-report.test.mjs test/fleet-scale.test.mjs test/fleet-state.test.mjs test/crypto-blake2s.test.mjs test/installed-version.test.mjs test/permission.test.mjs test/machine-test-packet.test.mjs test/field-authority.test.mjs test/approval-gate.test.mjs test/authority-choice.test.mjs test/reachability-check.test.mjs test/reachability-classify.test.mjs test/accepted-orphans.test.mjs test/acceptance-doc.test.mjs test/layer-model.test.mjs test/cwd-containment.test.mjs test/cwd-outside-project.test.mjs test/memory.test.mjs test/tools.test.mjs test/claims.test.mjs test/fleet-target.test.mjs test/client-bundle.test.mjs test/schedule.test.mjs test/auto-update.test.mjs test/update-source.test.mjs test/update-install.test.mjs test/update-wiring.test.mjs test/signing.test.mjs test/stress.test.mjs test/shared-config.test.mjs test/metrics.test.mjs test/metrics-route.test.mjs test/tool-cards.test.mjs test/p2p-stun.test.mjs test/p2p-transport.test.mjs test/p2p-signaling.test.mjs test/p2p-node.test.mjs test/p2p-transport-fields.test.mjs test/p2p-plugin.test.mjs test/p2p-crypto.test.mjs test/stun-server.test.mjs test/loader-entry.test.mjs test/mdns-wire.test.mjs test/mdns-discover.test.mjs test/chatops-approval.test.mjs test/chatops-policy.test.mjs test/chatops-file-safety.test.mjs test/chatops-outbound-queue.test.mjs test/chatops-commands.test.mjs test/chatops-bridge.test.mjs test/chatops-config-route.test.mjs test/chatops-apply.test.mjs test/chatops-ilink-contract.test.mjs test/chatops-fleet.test.mjs test/chatops-account.test.mjs test/chatops-auth.test.mjs test/chatops-fleet-scope.test.mjs test/chatops-handoff.test.mjs test/chatops-fleet-wiring.test.mjs test/chatops-fleet-live.test.mjs test/chatops-takeover.test.mjs test/chatops-device-signing.test.mjs test/chatops-relay-verify.test.mjs test/chatops-durability.test.mjs test/chatops-run-command.test.mjs test/secrets-guard.test.mjs test/chatops-ilink-fleet-bridge.test.mjs test/agent-plugins.test.mjs test/memory-layer.test.mjs test/mcp-server.test.mjs test/portable-plugins.test.mjs test/mcp-http.test.mjs test/mcp-http-live.test.mjs test/vision.test.mjs test/ilink-shape.test.mjs test/chatops-fleet-join.test.mjs test/admission.test.mjs test/admission-routes.test.mjs test/join-degradation.test.mjs test/account-identity.test.mjs test/join-challenge.test.mjs test/exec-env-injection.test.mjs test/account-join.test.mjs test/fleet-messages.test.mjs test/message-shards.test.mjs test/fleet-messages-http.test.mjs test/message-notify.test.mjs test/update-channels.test.mjs test/update-prerelease.test.mjs test/session-watch.test.mjs test/session-readonly.test.mjs test/session-sync.test.mjs test/fleet-chat-identity.test.mjs test/identity-reader.test.mjs test/intents.test.mjs test/dataflow.test.mjs test/council-fleet.test.mjs test/fleet-memory.test.mjs test/memory-view.test.mjs test/schema-spec.test.mjs test/fleet-weight-parity.test.mjs test/task-deps.test.mjs test/dataflow-wiring.test.mjs test/task-graph.test.mjs test/ask-store.test.mjs test/turn.test.mjs test/fleet-members.test.mjs test/redact.test.mjs test/policy-matrix.test.mjs test/envelope-signature.test.mjs test/chat-crypto.test.mjs test/reduce-suggestion.test.mjs test/merge-hint.test.mjs test/fleet-chat-route.test.mjs test/maintenance.test.mjs test/maintenance-ferry.test.mjs test/grant.test.mjs test/grant-gate.test.mjs test/worktree.test.mjs test/patch-artifact.test.mjs test/merge-queue.test.mjs test/m3-workflow.test.mjs test/project-room.test.mjs test/project-attribution.test.mjs test/project-locks.test.mjs test/project-reviews.test.mjs test/project-notifications.test.mjs test/blob-sessions.test.mjs test/outbox.test.mjs test/reliability-metrics.test.mjs test/evidence.test.mjs test/grant-issue.test.mjs test/project-refs.test.mjs test/exec-capability.test.mjs test/secondary-link.test.mjs test/bootstrap-cli.test.mjs test/bootstrap.test.mjs test/windows-hidden-task.test.mjs test/relay-failover.test.mjs",
|
|
68
68
|
"test:acceptance-doc": "node --test --test-force-exit test/acceptance-doc.test.mjs",
|
|
69
69
|
"test:account-identity": "node --test --test-force-exit test/account-identity.test.mjs",
|
|
70
70
|
"test:account-join": "node --test --test-force-exit test/account-join.test.mjs",
|
|
@@ -82,7 +82,7 @@
|
|
|
82
82
|
"test:council-session": "node --test --test-force-exit test/council-session.test.mjs",
|
|
83
83
|
"test:crossnetwork": "node --test --test-force-exit test/crossnetwork.test.mjs",
|
|
84
84
|
"test:crypto-blake2s": "node --test --test-force-exit test/crypto-blake2s.test.mjs",
|
|
85
|
-
"test:e2e": "node --test --test-force-exit test/e2e.test.mjs test/crossnetwork.test.mjs test/signing-e2e.test.mjs test/outbound-only.test.mjs test/recovery.test.mjs test/pipeline-e2e.test.mjs test/p2p-agent.test.mjs test/secondary-link.test.mjs test/bootstrap-cli.test.mjs test/bootstrap.test.mjs test/windows-hidden-task.test.mjs",
|
|
85
|
+
"test:e2e": "node --test --test-force-exit test/e2e.test.mjs test/crossnetwork.test.mjs test/signing-e2e.test.mjs test/outbound-only.test.mjs test/recovery.test.mjs test/pipeline-e2e.test.mjs test/p2p-agent.test.mjs test/secondary-link.test.mjs test/bootstrap-cli.test.mjs test/bootstrap.test.mjs test/windows-hidden-task.test.mjs test/relay-failover.test.mjs",
|
|
86
86
|
"test:exec-env": "node --test --test-force-exit test/exec-env-injection.test.mjs",
|
|
87
87
|
"test:exit-channel": "node --test --test-force-exit test/exit-channel.test.mjs",
|
|
88
88
|
"test:field-authority": "node --test --test-force-exit test/field-authority.test.mjs",
|
package/plugin.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
|
|
3
3
|
"name": "twinsearth-w2m-dsh",
|
|
4
|
-
"version": "1.6.
|
|
4
|
+
"version": "1.6.7",
|
|
5
5
|
"description": "Run one project across your own Windows and macOS machines from a single DeepSeek account, through 12 w2m_* tools. Four coordination modes (replicate, split, broadcast, pipeline) and six aggregate verdicts (consistent, divergent, divergent-platform, failed, partial, unverifiable). P2P hole punching by default, with a bundled relay, STUN responder and ledger fallback, so the fleet needs no third-party service. Joining needs a pairing code plus autoStartAgent: true; installing alone does not join.",
|
|
6
6
|
"license": "MIT",
|
|
7
7
|
"keywords": [
|
|
@@ -0,0 +1,158 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Relay failover supervisor (v1.6.6).
|
|
3
|
+
*
|
|
4
|
+
* ## The failure this exists for
|
|
5
|
+
*
|
|
6
|
+
* On 2026-10-24 the primary relay's host went down entirely (HTTP 8787 and SSH 22 both timed out).
|
|
7
|
+
* Every machine kept dialling the dead primary: the fleet was unreachable even though the backup
|
|
8
|
+
* relay was healthy the whole time. The backup carried a "standby member" per machine, but nothing
|
|
9
|
+
* ever *moved* the working connection to it. This module is the decision half of the fix; the wiring
|
|
10
|
+
* in `tools.mjs` restarts the in-process stack against the chosen relay, and re-pairing happens
|
|
11
|
+
* through the fleet key (per-relay tokens differ, so a switch is a re-join, not a reconnect).
|
|
12
|
+
*
|
|
13
|
+
* ## Hysteresis, deliberately asymmetric
|
|
14
|
+
*
|
|
15
|
+
* Switching away needs **3 consecutive** failed probes (a single dropped health request during a
|
|
16
|
+
* deploy must not move a fleet), while returning to the primary needs **5 consecutive** successes.
|
|
17
|
+
* Coming back is the riskier direction — a flapping primary that answers twice must not drag every
|
|
18
|
+
* machine back and forth — so it requires more evidence than leaving does.
|
|
19
|
+
*
|
|
20
|
+
* Pure state machine: no timers, no network. The caller probes and feeds results in, which is what
|
|
21
|
+
* makes the thresholds testable without waiting ten seconds per transition.
|
|
22
|
+
*/
|
|
23
|
+
|
|
24
|
+
/** Defaults are exported so the tests and the status report cannot invent different numbers. */
|
|
25
|
+
export const FAILOVER_DEFAULTS = Object.freeze({
|
|
26
|
+
probeIntervalMs: 10_000,
|
|
27
|
+
probeTimeoutMs: 2_000,
|
|
28
|
+
failuresToSwitch: 3,
|
|
29
|
+
successesToReturn: 5,
|
|
30
|
+
});
|
|
31
|
+
|
|
32
|
+
/**
|
|
33
|
+
* The ordered relay pool for this machine.
|
|
34
|
+
*
|
|
35
|
+
* `relayUrls` is the operator's pool (v1.5.1+); `rabbitUrl` is the active one. Deduplicated and
|
|
36
|
+
* normalized so a trailing slash cannot make one relay look like two.
|
|
37
|
+
*
|
|
38
|
+
* @param {{rabbitUrl?: string|null, relayUrls?: string[]|null}} cfg
|
|
39
|
+
* @returns {string[]}
|
|
40
|
+
*/
|
|
41
|
+
export function relayPool(cfg) {
|
|
42
|
+
const cleaned = (Array.isArray(cfg?.relayUrls) ? cfg.relayUrls : [])
|
|
43
|
+
.filter((url) => typeof url === 'string' && url.trim() !== '')
|
|
44
|
+
.map((url) => url.trim().replace(/\/+$/, ''));
|
|
45
|
+
const active = typeof cfg?.rabbitUrl === 'string' && cfg.rabbitUrl.trim() !== ''
|
|
46
|
+
? cfg.rabbitUrl.trim().replace(/\/+$/, '')
|
|
47
|
+
: null;
|
|
48
|
+
const pool = [];
|
|
49
|
+
for (const url of [...(active === null ? [] : [active]), ...cleaned]) {
|
|
50
|
+
if (!pool.includes(url)) pool.push(url);
|
|
51
|
+
}
|
|
52
|
+
return pool;
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
/**
|
|
56
|
+
* Create the failover state machine.
|
|
57
|
+
*
|
|
58
|
+
* @param {object} options
|
|
59
|
+
* @param {string[]} options.pool Ordered relay URLs; `pool[0]` is the primary.
|
|
60
|
+
* @param {string} [options.primary] Overrides which URL counts as "the one to return to".
|
|
61
|
+
* @param {number} [options.failuresToSwitch]
|
|
62
|
+
* @param {number} [options.successesToReturn]
|
|
63
|
+
* @param {() => number} [options.now]
|
|
64
|
+
*/
|
|
65
|
+
export function createRelayFailover(options) {
|
|
66
|
+
const pool = Array.isArray(options?.pool) ? [...options.pool] : [];
|
|
67
|
+
if (pool.length === 0) throw new TypeError('pool must contain at least one relay URL');
|
|
68
|
+
const primary = typeof options?.primary === 'string' && pool.includes(options.primary) ? options.primary : pool[0];
|
|
69
|
+
const failuresToSwitch = Number.isInteger(options?.failuresToSwitch) && options.failuresToSwitch > 0
|
|
70
|
+
? options.failuresToSwitch : FAILOVER_DEFAULTS.failuresToSwitch;
|
|
71
|
+
const successesToReturn = Number.isInteger(options?.successesToReturn) && options.successesToReturn > 0
|
|
72
|
+
? options.successesToReturn : FAILOVER_DEFAULTS.successesToReturn;
|
|
73
|
+
const now = typeof options?.now === 'function' ? options.now : Date.now;
|
|
74
|
+
|
|
75
|
+
let active = typeof options?.active === 'string' && pool.includes(options.active) ? options.active : primary;
|
|
76
|
+
let failures = 0;
|
|
77
|
+
let primarySuccesses = 0;
|
|
78
|
+
let switches = 0;
|
|
79
|
+
let lastSwitchAtMs = null;
|
|
80
|
+
let lastReason = null;
|
|
81
|
+
const history = [];
|
|
82
|
+
|
|
83
|
+
/**
|
|
84
|
+
* Feed one round of health results and get the decision.
|
|
85
|
+
*
|
|
86
|
+
* @param {Map<string, boolean>|Record<string, boolean>} health url -> reachable
|
|
87
|
+
* @returns {{action: 'stay'|'switch', active: string, target?: string, reason?: string}}
|
|
88
|
+
*/
|
|
89
|
+
function assess(health) {
|
|
90
|
+
const reachable = (url) => (health instanceof Map ? health.get(url) : health?.[url]) === true;
|
|
91
|
+
const activeOk = reachable(active);
|
|
92
|
+
|
|
93
|
+
if (activeOk) {
|
|
94
|
+
failures = 0;
|
|
95
|
+
if (active === primary) {
|
|
96
|
+
primarySuccesses = 0;
|
|
97
|
+
}
|
|
98
|
+
// A healthy non-primary route still watches the primary for the way home.
|
|
99
|
+
} else {
|
|
100
|
+
failures += 1;
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
if (active !== primary && reachable(primary)) {
|
|
104
|
+
primarySuccesses += 1;
|
|
105
|
+
} else if (active !== primary) {
|
|
106
|
+
primarySuccesses = 0;
|
|
107
|
+
}
|
|
108
|
+
|
|
109
|
+
if (activeOk !== true && failures >= failuresToSwitch) {
|
|
110
|
+
const alternative = pool.find((url) => url !== active && reachable(url));
|
|
111
|
+
if (alternative !== undefined) {
|
|
112
|
+
const reason = `active relay ${active} failed ${failures} consecutive probes`;
|
|
113
|
+
history.push({ at_ms: now(), from: active, to: alternative, reason });
|
|
114
|
+
active = alternative;
|
|
115
|
+
failures = 0;
|
|
116
|
+
primarySuccesses = 0;
|
|
117
|
+
switches += 1;
|
|
118
|
+
lastSwitchAtMs = now();
|
|
119
|
+
lastReason = reason;
|
|
120
|
+
return { action: 'switch', active, target: alternative, reason };
|
|
121
|
+
}
|
|
122
|
+
// No reachable alternative: stay, but keep reporting the failure streak.
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
if (active !== primary && primarySuccesses >= successesToReturn) {
|
|
126
|
+
const reason = `primary relay ${primary} healthy ${primarySuccesses} consecutive probes`;
|
|
127
|
+
history.push({ at_ms: now(), from: active, to: primary, reason });
|
|
128
|
+
active = primary;
|
|
129
|
+
failures = 0;
|
|
130
|
+
primarySuccesses = 0;
|
|
131
|
+
switches += 1;
|
|
132
|
+
lastSwitchAtMs = now();
|
|
133
|
+
lastReason = reason;
|
|
134
|
+
return { action: 'switch', active, target: primary, reason };
|
|
135
|
+
}
|
|
136
|
+
|
|
137
|
+
return { action: 'stay', active };
|
|
138
|
+
}
|
|
139
|
+
|
|
140
|
+
/** What `w2m_status` reports: which relay is in use and why it last moved. */
|
|
141
|
+
function snapshot() {
|
|
142
|
+
return {
|
|
143
|
+
primary,
|
|
144
|
+
active,
|
|
145
|
+
pool: [...pool],
|
|
146
|
+
failures,
|
|
147
|
+
failures_to_switch: failuresToSwitch,
|
|
148
|
+
primary_successes: primarySuccesses,
|
|
149
|
+
successes_to_return: successesToReturn,
|
|
150
|
+
switches,
|
|
151
|
+
last_switch_at_ms: lastSwitchAtMs,
|
|
152
|
+
last_reason: lastReason,
|
|
153
|
+
history: history.slice(-10),
|
|
154
|
+
};
|
|
155
|
+
}
|
|
156
|
+
|
|
157
|
+
return { assess, snapshot, thresholds: { failuresToSwitch, successesToReturn } };
|
|
158
|
+
}
|
package/src/plugin/tools.mjs
CHANGED
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
import { FAILOVER_DEFAULTS, createRelayFailover, relayPool } from './relay-failover.mjs';
|
|
1
2
|
import { DEFAULT_BOOTSTRAP_URL, DEFAULT_BOOTSTRAP_OPERATOR_KEY_PEM } from './bootstrap.mjs';
|
|
2
3
|
/**
|
|
3
4
|
* DSH: Windows2MacO - the DSH Cordis plugin.
|
|
@@ -89,6 +90,9 @@ import { DEFAULT_BOOTSTRAP_URL, DEFAULT_BOOTSTRAP_OPERATOR_KEY_PEM } from './boo
|
|
|
89
90
|
* implementation, and vice versa.
|
|
90
91
|
*/
|
|
91
92
|
|
|
93
|
+
/** v1.6.6: last failover snapshot, reported by w2m_status and refreshed by the supervisor interval. */
|
|
94
|
+
let relayFailoverStatus = null;
|
|
95
|
+
|
|
92
96
|
import { execFile } from 'node:child_process';
|
|
93
97
|
|
|
94
98
|
import { resolveChannels } from './update-channels.mjs';
|
|
@@ -277,7 +281,7 @@ const MAX_STAGES = 16;
|
|
|
277
281
|
* `package.json` during `scripts/pack.mjs` rather than hand-maintained, and `W2M_PLUGIN_VERSION`
|
|
278
282
|
* overrides it at runtime for tests and packagers.
|
|
279
283
|
*/
|
|
280
|
-
const PLUGIN_VERSION = '1.6.
|
|
284
|
+
const PLUGIN_VERSION = '1.6.7';
|
|
281
285
|
|
|
282
286
|
/** How long any single HTTP call may take before it is aborted, in milliseconds. */
|
|
283
287
|
const DEFAULT_REQUEST_TIMEOUT_MS = 15_000;
|
|
@@ -2193,6 +2197,8 @@ async function readP2PStatus(cfg, runtime, peersAnnounced = null) {
|
|
|
2193
2197
|
local: status?.local ?? null,
|
|
2194
2198
|
rabbit_url: status?.rabbit_url ?? cfg.rabbitUrl,
|
|
2195
2199
|
bootstrap: cfg.bootstrap ?? null,
|
|
2200
|
+
// v1.6.6: which relay is in use and why it last moved.
|
|
2201
|
+
failover: relayFailoverStatus,
|
|
2196
2202
|
// How many machines the relay holds a live announcement for. `w2m_devices` says which machines
|
|
2197
2203
|
// paired; this says which of them can be punched at, and the two differ constantly.
|
|
2198
2204
|
peers_announced: peersAnnounced,
|
|
@@ -4956,7 +4962,7 @@ export async function apply(ctx, config = {}) {
|
|
|
4956
4962
|
// v1.5.3: a failed auto-start must not surface as an unhandled rejection. The bundled patch now
|
|
4957
4963
|
// starts the agent by default, and a host whose plugin wiring forbids that (a restricted test
|
|
4958
4964
|
// harness, a missing relay module) would otherwise reject where nothing can catch it.
|
|
4959
|
-
ctx.effect?.(() =>
|
|
4965
|
+
ctx.effect?.(() => startFailoverStack(ctx, cfg).catch((error) => {
|
|
4960
4966
|
ctx.logger?.warn?.('w2m: auto-start did not complete: ' + (error instanceof Error ? error.message : String(error)));
|
|
4961
4967
|
}));
|
|
4962
4968
|
} else {
|
|
@@ -7375,6 +7381,84 @@ function sleepWithSignal(ms, signal) {
|
|
|
7375
7381
|
*/
|
|
7376
7382
|
const resolvedFleetKey = { value: null, source: null, at: 0 };
|
|
7377
7383
|
|
|
7384
|
+
/** Probe one relay /healthz; any failure is `false` (a timeout must not throw into the interval). */
|
|
7385
|
+
async function probeRelayHealth(url, timeoutMs) {
|
|
7386
|
+
try {
|
|
7387
|
+
const response = await fetch(url + '/healthz', {
|
|
7388
|
+
signal: typeof AbortSignal !== 'undefined' && typeof AbortSignal.timeout === 'function' ? AbortSignal.timeout(timeoutMs) : undefined,
|
|
7389
|
+
});
|
|
7390
|
+
if (response.ok !== true) return false;
|
|
7391
|
+
const body = await response.json().catch(() => null);
|
|
7392
|
+
return body?.ok === true;
|
|
7393
|
+
} catch {
|
|
7394
|
+
return false;
|
|
7395
|
+
}
|
|
7396
|
+
}
|
|
7397
|
+
|
|
7398
|
+
/**
|
|
7399
|
+
* v1.6.6: keep the in-process stack on a reachable relay.
|
|
7400
|
+
*
|
|
7401
|
+
* The primary relay can disappear entirely (2026-10-24: its host was down, so 8787 and SSH were both
|
|
7402
|
+
* unreachable). The stack is restartable: stop it, point cfg.rabbitUrl at the next healthy relay in
|
|
7403
|
+
* the pool, start it again. A switch is a re-join, not just a reconnect -- device tokens are relay-
|
|
7404
|
+
* specific, so startLocalStack pairs again through the account fleet key.
|
|
7405
|
+
*
|
|
7406
|
+
* The decision itself (3 failed probes to leave, 5 healthy ones to come home) lives in
|
|
7407
|
+
* relay-failover.mjs and is unit-tested there; this function only supplies health and performs the
|
|
7408
|
+
* restart.
|
|
7409
|
+
*/
|
|
7410
|
+
/** Append one line to the failover diagnostics log; failures are swallowed (diagnostics must not break the switch). */
|
|
7411
|
+
async function appendFailoverLog(cfg, line) {
|
|
7412
|
+
try {
|
|
7413
|
+
const { appendFileSync, mkdirSync, statSync, writeFileSync } = await import('node:fs');
|
|
7414
|
+
const { join, dirname } = await import('node:path');
|
|
7415
|
+
const { homedir } = await import('node:os');
|
|
7416
|
+
const dir = cfg.stateDir ?? join(process.env.DSH_HOME ?? join(homedir(), '.dsh'), 'xclient');
|
|
7417
|
+
const file = join(dir, 'relay-failover.log');
|
|
7418
|
+
mkdirSync(dirname(file), { recursive: true });
|
|
7419
|
+
try { if (statSync(file).size > 256 * 1024) writeFileSync(file, ''); } catch { /* absent is fine */ }
|
|
7420
|
+
appendFileSync(file, new Date().toISOString() + ' ' + line + '\n');
|
|
7421
|
+
} catch { /* diagnostics only */ }
|
|
7422
|
+
}
|
|
7423
|
+
async function startFailoverStack(ctx, cfg) {
|
|
7424
|
+
let stop = await startLocalStack(ctx, cfg);
|
|
7425
|
+
const pool = relayPool(cfg);
|
|
7426
|
+
if (pool.length < 2) {
|
|
7427
|
+
relayFailoverStatus = null;
|
|
7428
|
+
return stop;
|
|
7429
|
+
}
|
|
7430
|
+
const failover = createRelayFailover({ pool });
|
|
7431
|
+
relayFailoverStatus = failover.snapshot();
|
|
7432
|
+
let switching = false;
|
|
7433
|
+
const timer = setInterval(async () => {
|
|
7434
|
+
if (switching) return;
|
|
7435
|
+
const health = {};
|
|
7436
|
+
await Promise.all(pool.map(async (url) => { health[url] = await probeRelayHealth(url, FAILOVER_DEFAULTS.probeTimeoutMs); }));
|
|
7437
|
+
const decision = failover.assess(health);
|
|
7438
|
+
await appendFailoverLog(cfg, 'probe ' + JSON.stringify(health) + ' -> ' + decision.action + (decision.target ? ' ' + decision.target : ''));
|
|
7439
|
+
relayFailoverStatus = failover.snapshot();
|
|
7440
|
+
if (decision.action !== 'switch') return;
|
|
7441
|
+
switching = true;
|
|
7442
|
+
try {
|
|
7443
|
+
ctx.logger?.warn?.('w2m: relay failover ' + decision.active + ' (' + decision.reason + ')');
|
|
7444
|
+
await stop();
|
|
7445
|
+
cfg.rabbitUrl = decision.target;
|
|
7446
|
+
cfg.relaySource = 'failover';
|
|
7447
|
+
stop = await startLocalStack(ctx, cfg);
|
|
7448
|
+
ctx.logger?.info?.('w2m: relay failover stack restarted on ' + decision.target);
|
|
7449
|
+
} catch (error) {
|
|
7450
|
+
ctx.logger?.warn?.('w2m: relay failover restart failed: ' + (error instanceof Error ? error.message : String(error)));
|
|
7451
|
+
} finally {
|
|
7452
|
+
switching = false;
|
|
7453
|
+
}
|
|
7454
|
+
}, FAILOVER_DEFAULTS.probeIntervalMs);
|
|
7455
|
+
timer.unref?.();
|
|
7456
|
+
return async () => {
|
|
7457
|
+
clearInterval(timer);
|
|
7458
|
+
relayFailoverStatus = null;
|
|
7459
|
+
await stop();
|
|
7460
|
+
};
|
|
7461
|
+
}
|
|
7378
7462
|
async function startLocalStack(ctx, cfg) {
|
|
7379
7463
|
|
|
7380
7464
|
/**
|
|
@@ -8007,7 +8091,14 @@ async function startLocalStack(ctx, cfg) {
|
|
|
8007
8091
|
ensureFleetJoin = null;
|
|
8008
8092
|
inProcessAgentStatus = null;
|
|
8009
8093
|
await agent.stop();
|
|
8010
|
-
|
|
8094
|
+
// v1.6.7: `relay` is only a real relay object when THIS stack started a local relay; a remote-
|
|
8095
|
+
// relay stack has null here. The unconditional close() threw, which is what made the first
|
|
8096
|
+
// failover attempt stop the stack and never restart it.
|
|
8097
|
+
if (relay !== null && typeof relay.close === 'function') {
|
|
8098
|
+
try { await relay.close(); } catch (error) {
|
|
8099
|
+
ctx.logger?.warn?.('w2m: local relay close failed: ' + (error instanceof Error ? error.message : String(error)));
|
|
8100
|
+
}
|
|
8101
|
+
}
|
|
8011
8102
|
};
|
|
8012
8103
|
}
|
|
8013
8104
|
|