@gotcos/glasses-server 6.87.0 → 6.89.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.env.example +7 -0
- package/CHANGELOG.md +97 -0
- package/SECURITY.md +5 -0
- package/docs/tailscale-self-heal.md +159 -0
- package/managed-runtime-contract.json +1 -0
- package/package.json +4 -1
- package/server/index.ts +20 -2
- package/server/lib/tailscale-nudge.ts +536 -0
- package/server/lib/transfer-integrity.ts +59 -0
- package/server/routes/health.ts +3 -0
- package/server/routes/meeting.ts +20 -1
package/.env.example
CHANGED
|
@@ -53,6 +53,13 @@ BIND_HOST=0.0.0.0
|
|
|
53
53
|
# server after Wi-Fi/Tailscale changes and process restarts.
|
|
54
54
|
# COS_SERVER_INSTANCE_ID_PATH=/path/to/server-instance-id
|
|
55
55
|
|
|
56
|
+
# 6.89.0 Tailscale self-heal. After a restart, a wake from sleep, a Tailscale
|
|
57
|
+
# restart or a network change, or when a recently active phone goes quiet while
|
|
58
|
+
# Tailscale says it is online, the server runs a bounded `tailscale ping` at it
|
|
59
|
+
# (at most one a minute per phone, 30 an hour). On by default; 0 turns it off.
|
|
60
|
+
# See docs/tailscale-self-heal.md.
|
|
61
|
+
# COS_TAILSCALE_NUDGE=0
|
|
62
|
+
|
|
56
63
|
# Build 204+ uses server-owned durable query jobs by default. The server fsyncs an
|
|
57
64
|
# accepted prompt before returning 202, then keeps provider work running if the
|
|
58
65
|
# phone backgrounds, reloads, or changes networks. Reopening COS reattaches to
|
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,100 @@
|
|
|
1
|
+
## 6.89.0 - LAN first removed; the Mac nudges a stale Tailscale tunnel back to life
|
|
2
|
+
|
|
3
|
+
Miles, 2026-10-23 19:57: no LAN-only code, because he uses COS away from home; and checks around the nudge so the
|
|
4
|
+
Tailscale connection always comes back. After the 10/22 Mac restart the phone's Tailscale extension kept a stale tunnel:
|
|
5
|
+
Tailscale's own ping went through, no COS request did, and only toggling Tailscale on the phone fixed it. Contract and
|
|
6
|
+
field canary: `docs/tailscale-self-heal.md`.
|
|
7
|
+
|
|
8
|
+
- **LAN first is gone.** Removed: public `GET /api/route-proof`, `routes` on `/api/models`, `capabilities.routeProof` and
|
|
9
|
+
`capabilities.lanRoutes`, `COS_LAN_FIRST`, `server/lib/lan-first.ts`, `server/routes/route-proof.ts`,
|
|
10
|
+
`docs/lan-first.md`, the SECURITY.md note, their tests and gate (reverts c593b91). `/api/models` is the 6.87.0 shape
|
|
11
|
+
path for path, and `/api/health` is 6.87.0 plus `tailscaleNudge`, both checked against goldens captured from
|
|
12
|
+
601d807. `/api/route-proof` is an unknown path behind the token again. `COS_LAN_FIRST` is read by nothing.
|
|
13
|
+
- **Kept:** 6.88.0's transfer integrity against the phone's final index and the origin-wipe recovery save that waits
|
|
14
|
+
for it (fc52f49), and the older opt-in "Allow pairing on this Wi-Fi" pairing switch (6.67.0).
|
|
15
|
+
- **The nudge** (`server/lib/tailscale-nudge.ts`). `tailscale ping -c 3 --timeout 5s <peer>` (disco: both ends redo
|
|
16
|
+
path discovery), then, only after a pong, `tailscale ping --icmp -c 1 --timeout 5s <peer>` (one packet through
|
|
17
|
+
WireGuard, which starts a fresh handshake on a stale session). `execFile` from a fixed path, never a shell.
|
|
18
|
+
- **Triggers:** 30 s after boot; a wake from sleep (the wall clock outran the monotonic clock by 20 s); a change in
|
|
19
|
+
this Mac's IPv4 interfaces; a change in Tailscale's own state (BackendState, this Mac's Tailscale IPs, or peer byte
|
|
20
|
+
counters going backwards, meaning tailscaled restarted); and a phone that made an authenticated request in the last
|
|
21
|
+
5 minutes, or sent meeting chunks in the last 15 minutes while a meeting is open, silent for 45 s. Every one needs
|
|
22
|
+
the peer `Online` in a status read at most 15 s old and Tailscale `Running`: absence of requests alone never nudges.
|
|
23
|
+
- **Caps:** one nudge per peer per 60 s, doubling per unanswered nudge to 15 minutes, reset by any request; 30 an hour
|
|
24
|
+
in all; one pass at a time; hard timeouts (5, 20 and 10 s, SIGKILL). `COS_TAILSCALE_NUDGE=0` turns it off, read
|
|
25
|
+
every tick, forwarded by the managed runtime contract. No LLM, no disk write, no new route.
|
|
26
|
+
- **What it says.** `[tailscale-nudge]` log lines with trigger, outcome, timing, handshake age and byte movement, never
|
|
27
|
+
an address; `tailscaleNudge { enabled, available, lastAt, lastTrigger, lastOutcome, lastDataPath, lastRecoveredAt,
|
|
28
|
+
nudgesLastHour }` on public `/api/health`.
|
|
29
|
+
- **Phone contract for glasses 6.10.653** (not built here): retry with fresh `no-store` requests on a 2, 4, 8, 15, 30 s
|
|
30
|
+
ladder, show 6.10.652's stuck-Tailscale message only after 90 s, read `tailscaleNudge`, and the iOS VPN On Demand note.
|
|
31
|
+
|
|
32
|
+
## 6.88.0 - LAN first: a pre-token route proof and the Mac's addresses for the phone
|
|
33
|
+
|
|
34
|
+
The server half of the glasses "LAN first" reconnect plan (plan of 2026-10-22, Miles's decisions of 2026-10-23: "Go
|
|
35
|
+
with your picks on the reconnect plan. No regressions or loss of existing functionality."). After the 10/22 Mac restart
|
|
36
|
+
the phone stayed on a stale Tailscale tunnel while the Mac answered fine on the same Wi-Fi. These two additions let a
|
|
37
|
+
phone build reach the Mac over the home network first, without ever sending its token to the wrong machine. Contract
|
|
38
|
+
for the phone side: `docs/lan-first.md`.
|
|
39
|
+
|
|
40
|
+
- **`GET /api/route-proof?nonce=<22 to 64 base64url characters>`, public.** Answers `{ v: 1, serverInstanceId, proof }`,
|
|
41
|
+
where `proof` is lowercase hex HMAC-SHA256(token, "cos-route-proof:v1:" + nonce + ":" + serverInstanceId). The phone
|
|
42
|
+
computes the same HMAC with its stored token and sends `X-Cos-Token` to that address only on a match. The token is never
|
|
43
|
+
in the response, the nonce is never echoed or logged, and the caller chooses only the nonce, so the route is not a
|
|
44
|
+
general HMAC oracle; the fixed prefix also keeps a proof from ever being a display-stream ticket. Exact public door,
|
|
45
|
+
GET only (`server/lib/api-auth.ts`); every other verb and path stays behind the token. Errors: 400 `bad_nonce`,
|
|
46
|
+
404 `disabled`, 405 `method_not_allowed`, 429 `rate_limited` (with `Retry-After`), 503 `not_ready`.
|
|
47
|
+
- **Rate limited.** 20 a minute per socket address and 240 a minute in all, in memory; bad nonces spend the same budget.
|
|
48
|
+
- **Only for a strong token.** A proof lets anyone who reaches the port test guesses offline, so it is served (and
|
|
49
|
+
advertised) only when the token is 32 characters or more. Tokens made by COS Control (64) and by the server (44) qualify;
|
|
50
|
+
a short hand-set legacy token keeps today's behaviour.
|
|
51
|
+
- **`routes` on authenticated `/api/models` only.** `{ v: 1, serverInstanceId, learnedAt, lan: [{ip, port, scheme}],
|
|
52
|
+
tailscale: [{ip, port, scheme}] }`: RFC1918 IPv4 on real interfaces (tunnels, Internet Sharing and VM bridges, Docker,
|
|
53
|
+
AirDrop and link-local left out) and 100.64/10 on Tailscale's own interface, plain http on `PORT` (3141). Re-read at
|
|
54
|
+
most every 10 s, so a new DHCP lease shows up; a failed interface read omits the field and leaves the rest of the
|
|
55
|
+
response alone. Never on public `/api/health`. The MagicDNS name is left out (reading it needs a `tailscale`
|
|
56
|
+
process per request).
|
|
57
|
+
- **Capability flags.** `capabilities.routeProof: { protocol: 1 }` and `capabilities.lanRoutes: { protocol: 1 }` on
|
|
58
|
+
`/api/health` and `/api/models`, from one helper. An older server omits both, and a phone keeps today's single URL.
|
|
59
|
+
- **Switch.** `COS_LAN_FIRST=0` turns both pieces off, read per request; the responses are then the 6.87.0 shape exactly.
|
|
60
|
+
- **No regression.** Every existing field of `/api/health` and `/api/models` is unchanged, checked path for path against
|
|
61
|
+
a golden captured from the unmodified 6.87.0 base, and byte for byte against the same response with the switch off.
|
|
62
|
+
- **Not built:** Phase 3's loopback `/api/glasses/link` for COS Control's banner. It needs new tracking of the phone's
|
|
63
|
+
last path and check-in, so it is not a purely additive change.
|
|
64
|
+
- **No spend.** No LLM, no new caller, nothing written to disk.
|
|
65
|
+
- **Checks (6.88.0 branch).** Full suite, serial (`--maxWorkers=1`), HOME, COS_DATA_DIR and COS_PROFILE_PATH in a scratch folder,
|
|
66
|
+
COS_SCRIPTS_DIR unset. Unmodified 6.87.0 base (601d807): 584 files (583 passed, 1 skipped), 8815 tests, 8801 passed,
|
|
67
|
+
14 skipped, 0 failed. This branch: 588 files (587 passed, 1 skipped), 8911 tests, 8897 passed, 14 skipped, 0 failed.
|
|
68
|
+
The 96 new tests are the only change per file (`lan-first.test.ts` 58, `route-proof.test.ts` 18, `health-lan-first.test.ts`
|
|
69
|
+
9, `lan-first-wiring.test.ts` 2, `api-auth.test.ts` 40 to 49); no existing test was changed. Mutation gate
|
|
70
|
+
`lan-first-mutation-gate.mjs` against a green baseline (136 tests in 5 files): 56 of 56 killed, each by a named test,
|
|
71
|
+
every target landing exactly once and restored byte for byte. Its first run left two survivors (the IPv4-only filter and
|
|
72
|
+
the de-duplication), each now covered by its own test and killed. `tsc --noEmit` clean. Pack dry run 549 files
|
|
73
|
+
(6.87.0: 546; the three new files are `server/lib/lan-first.ts`, `server/routes/route-proof.ts` and `docs/lan-first.md`).
|
|
74
|
+
|
|
75
|
+
### Also in 6.88.0: transfer integrity counts what the phone captured, and recovery saves wait for it
|
|
76
|
+
|
|
77
|
+
For glasses 6.10.652 (meeting t2xs9u, 2026-10-23 07:59): a relaunched COS window saved a meeting from the Mac's 74 chunks
|
|
78
|
+
while the window still recording it had captured through index 120, and the file said "Transfer integrity 100%". One new
|
|
79
|
+
file (`server/lib/transfer-integrity.ts`) and two edits in `performSave` (`server/routes/meeting.ts`).
|
|
80
|
+
|
|
81
|
+
- **Integrity counts the phone's final index.** When the phone has said how far it captured (the Stop intent's
|
|
82
|
+
`finalChunkIndex`, or the save's own new optional `finalChunkIndex`), expected is that index plus one and every index past
|
|
83
|
+
the Mac's highest chunk is missing. 74 of 121 now reads 61.1% with 47 chunks not received, in the saved markdown, the
|
|
84
|
+
sidecar and the receipt. A final index more than 20,000 past the Mac's highest chunk is capped there; a malformed one is
|
|
85
|
+
ignored.
|
|
86
|
+
- **A recovery save waits.** A save with `recovery: 'origin_wipe'` (glasses 6.10.652) is refused with 409
|
|
87
|
+
`chunks_outstanding` (`missingCount`, `finalChunkIndex`) while the phone's final index says the Mac lacks chunks, and
|
|
88
|
+
nothing is saved or closed. `acceptMissing: true` (the phone waited its 10 minutes) saves what the Mac has, honestly counted.
|
|
89
|
+
- **Unchanged:** a save without `recovery` is never refused here; a save with no final index from anywhere writes exactly
|
|
90
|
+
what 6.87.0 wrote; an already saved meeting replays its receipt first; the autonomous save, the stop route and the status
|
|
91
|
+
route are untouched. No new log line, no LLM, no new disk write. Contract: `docs/meeting-autonomous-save.md`, section 6.88.0.
|
|
92
|
+
- **Checks (integrity, on top of c593b91).** Full suite, serial, the same scratch folder rules: 590 files, 8926 tests, 8912
|
|
93
|
+
passed, 14 skipped, 0 failed (the LAN-first record above is 8911 tests; the 15 new ones are `transfer-integrity.test.ts` 5
|
|
94
|
+
and `meeting-save-integrity.test.ts` 10; no existing test was changed). Mutation gate `integrity-6880-mutation-gate.mjs`
|
|
95
|
+
against a green baseline (15 tests in 2 files): 15 of 15 killed, each by a named test. `tsc --noEmit` clean. Pack dry run
|
|
96
|
+
551 files (the LAN-first 549 plus `server/lib/transfer-integrity.ts` and the gate script).
|
|
97
|
+
|
|
1
98
|
## 6.87.0 - A rough level per chunk for COS Control's glasses meeting strip
|
|
2
99
|
|
|
3
100
|
The server half of the level strip COS Control 0.5.339 draws in a glasses meeting's bottom bar (plan of 2026-10-23,
|
package/SECURITY.md
CHANGED
|
@@ -43,6 +43,11 @@ reproduction you have. Do not include real transcripts.
|
|
|
43
43
|
turns itself off after 10 minutes. Over Tailscale the traffic is encrypted.
|
|
44
44
|
- Behind `tailscale serve` or `funnel`, requests arrive from a loopback proxy, so
|
|
45
45
|
every claim is refused (`not_allowed_network`): pairing fails closed there.
|
|
46
|
+
- 6.89.0 Tailscale self-heal (`docs/tailscale-self-heal.md`): the server runs `tailscale status --json`
|
|
47
|
+
and `tailscale ping` at a peer that Tailscale reports online, with `execFile` from a fixed path and a
|
|
48
|
+
fixed argument list (the only variable is a 100.64.0.0/10 address read from Tailscale's own status).
|
|
49
|
+
Never a shell. Public `/api/health` shows when it last ran and the outcome, never an address.
|
|
50
|
+
`COS_TAILSCALE_NUDGE=0` turns it off.
|
|
46
51
|
- `GET /api/display-stream` is public by design and returns lifecycle events
|
|
47
52
|
only. Content requires a ticket or the token header. See
|
|
48
53
|
`server/routes/display.ts` for the allowlist.
|
|
@@ -0,0 +1,159 @@
|
|
|
1
|
+
# Tailscale self-heal (server 6.89.0)
|
|
2
|
+
|
|
3
|
+
COS is used away from home, over Tailscale. The problem this solves: after the Mac restarts, wakes
|
|
4
|
+
from sleep, restarts Tailscale or changes network, the phone can stay on a stale Tailscale tunnel.
|
|
5
|
+
Tailscale says the phone is online and its own ping gets through, yet no COS request arrives, and
|
|
6
|
+
the only fix used to be toggling Tailscale on the phone. 6.89.0 adds a bounded Mac-side "nudge" and
|
|
7
|
+
a contract for the phone side. Code: `server/lib/tailscale-nudge.ts`.
|
|
8
|
+
|
|
9
|
+
6.89.0 also removes 6.88.0's LAN-first pieces (`GET /api/route-proof`, `routes` on `/api/models`,
|
|
10
|
+
`capabilities.routeProof` and `capabilities.lanRoutes`, `COS_LAN_FIRST`). Tailscale already takes
|
|
11
|
+
the direct path over the home Wi-Fi when both ends are home, so a separate LAN route added nothing
|
|
12
|
+
for the away-from-home case. `COS_LAN_FIRST` is now read by nothing.
|
|
13
|
+
|
|
14
|
+
## What the Mac can see (read-only probes, 2026-10-23, Tailscale 1.102.2 on macOS)
|
|
15
|
+
|
|
16
|
+
`tailscale status --json`, the phone's peer entry while COS was working normally:
|
|
17
|
+
|
|
18
|
+
| Field | Value seen | What it tells us |
|
|
19
|
+
|---|---|---|
|
|
20
|
+
| `Online` | `true` | Control-plane presence. It stays true while the iOS extension runs, stale tunnel or not, so on its own it proves nothing about the data path. |
|
|
21
|
+
| `Active` | `true` | Recent traffic in the engine. |
|
|
22
|
+
| `CurAddr` | the phone's home Wi-Fi address and port | A direct path. Away from home it is a public address, or empty when relayed. |
|
|
23
|
+
| `Relay` | the home DERP region | The fallback relay. |
|
|
24
|
+
| `LastHandshake` | 60 s old, refreshing about every 2 minutes | The WireGuard session is being renewed. A handshake much older than about 3 minutes while `Active` is true is suspicious. |
|
|
25
|
+
| `LastSeen` | zero time | Only set while the peer is offline. |
|
|
26
|
+
| `RxBytes` / `TxBytes` | grew by about 9 KB and 18 KB in 12 s | Per-peer counters for the life of tailscaled. They reset when tailscaled restarts, which is how the server detects a Mac-side Tailscale restart. |
|
|
27
|
+
|
|
28
|
+
`tailscale ping -c 3 --timeout 5s <phone>` answered once, `via <home address>:41641 in 91ms`, and
|
|
29
|
+
stopped: it stops at the first direct pong. `tailscale ping --icmp -c 1 --timeout 5s <phone>`
|
|
30
|
+
answered `via ICMP in 27ms`. The CLI's own help says the plain (disco) ping does not go through the
|
|
31
|
+
tunnel; the `--icmp` ping does go through WireGuard.
|
|
32
|
+
|
|
33
|
+
The local signal for "the peer looks online but no COS request has arrived": the server's own last
|
|
34
|
+
request time from that Tailscale address (any `/api` request, public routes included, because any
|
|
35
|
+
request that arrives proves the tunnel carries COS traffic), together with `Online: true` from a
|
|
36
|
+
status read no more than 15 s old. Each nudge log line also records the peer's handshake age and its
|
|
37
|
+
`RxBytes` movement between the last two status reads, so the field canary can show which of them
|
|
38
|
+
tells a stuck tunnel apart. The stuck state itself was not reproduced here.
|
|
39
|
+
|
|
40
|
+
## The nudge
|
|
41
|
+
|
|
42
|
+
A nudge is two bounded commands at one peer, run with `execFile` from a fixed path (never a shell,
|
|
43
|
+
never a PATH lookup):
|
|
44
|
+
|
|
45
|
+
1. `tailscale ping -c 3 --timeout 5s <peer>`: a disco ping. Both ends redo path discovery.
|
|
46
|
+
2. Only after a pong: `tailscale ping --icmp -c 1 --timeout 5s <peer>`: one packet through
|
|
47
|
+
WireGuard, which makes WireGuard start a fresh handshake when its session is stale. On 10/22 the
|
|
48
|
+
disco ping got through while COS did not, so the disco ping alone is not enough.
|
|
49
|
+
|
|
50
|
+
The CLI is looked for at `/Applications/Tailscale.app/Contents/MacOS/Tailscale`,
|
|
51
|
+
`/opt/homebrew/bin/tailscale`, `/usr/local/bin/tailscale` and `/usr/bin/tailscale`, in that order.
|
|
52
|
+
Without one the loop does nothing and health says `available: false`.
|
|
53
|
+
|
|
54
|
+
Whether a Mac-side nudge un-sticks the iOS extension is not proven by tests. The field canary below
|
|
55
|
+
is the proof.
|
|
56
|
+
|
|
57
|
+
### Triggers
|
|
58
|
+
|
|
59
|
+
Every trigger needs the positive signal: the peer is `Online` in a status read at most 15 s old, and
|
|
60
|
+
Tailscale's `BackendState` is `Running`. Absence of requests alone never nudges.
|
|
61
|
+
|
|
62
|
+
| Trigger | When | Who |
|
|
63
|
+
|---|---|---|
|
|
64
|
+
| `boot` | 30 s after the server starts | Event peers (below) |
|
|
65
|
+
| `wake` | The wall clock moved 20 s or more past the monotonic clock between two 10 s ticks, which only happens across a sleep. Runs 15 s later. | Event peers |
|
|
66
|
+
| `network` | This Mac's external IPv4 interfaces changed, Tailscale's tunnel interface included. Runs 10 s later. | Event peers |
|
|
67
|
+
| `tailscale` | Tailscale's own state changed between status reads: `BackendState`, this Mac's Tailscale IPs, or the peer byte counters went backwards (tailscaled restarted). Runs 10 s later. | Event peers |
|
|
68
|
+
| `silent` | A Tailscale address that made an authenticated request in the last 5 minutes, or sent meeting chunks in the last 15 minutes while a meeting is open, has made no request at all for 45 s. | That address |
|
|
69
|
+
|
|
70
|
+
Event peers: Online peers this server has seen authenticate in the last 12 hours, plus Online iOS
|
|
71
|
+
and Android peers (after a restart the server remembers no one), at most 4, known ones first. A peer
|
|
72
|
+
that has already made a request since the event is skipped. An event that cannot run because
|
|
73
|
+
Tailscale is not `Running` waits, and is dropped (with one log line) after 2 minutes.
|
|
74
|
+
|
|
75
|
+
### Caps
|
|
76
|
+
|
|
77
|
+
- One nudge per peer per 60 s. Each nudge that brings no request doubles the gap: 60, 120, 240,
|
|
78
|
+
480 s, then 900 s. Any request from that peer resets it.
|
|
79
|
+
- 30 nudges an hour in all, across peers.
|
|
80
|
+
- One pass at a time; each command has a hard timeout (status 5 s, disco 20 s, ICMP 10 s, killed
|
|
81
|
+
with SIGKILL).
|
|
82
|
+
- A quiet phone outside a meeting is nudged at most 3 times (at about 45 s, 1:50 and 3:50) before its
|
|
83
|
+
5-minute window closes. In a meeting, at most 4 in 15 minutes.
|
|
84
|
+
- Status is read once a minute while the loop is on, and otherwise only when a nudge could run.
|
|
85
|
+
- `COS_TAILSCALE_NUDGE=0` (or `false`, `off`) stops all of it, read every tick. On by default.
|
|
86
|
+
Forwarded by the managed runtime contract.
|
|
87
|
+
- No LLM, no disk write, no new route.
|
|
88
|
+
|
|
89
|
+
### What it says
|
|
90
|
+
|
|
91
|
+
Log lines, outcome and timing only, never an address:
|
|
92
|
+
|
|
93
|
+
```
|
|
94
|
+
[tailscale-nudge] trigger=silent outcome=pong_direct dataPath=ok pingMs=91 icmpMs=27 silentS=50 handshakeAgeS=61 rxDelta=9040/60s inRow=1 lastHour=1
|
|
95
|
+
[tailscale-nudge] peer answered 4.2s after the silent nudge
|
|
96
|
+
[tailscale-nudge] trigger=wake dropped: Tailscale not running for 121s
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
`outcome` is `pong_direct`, `pong_relay`, `no_pong`, `timeout`, `error` or `unavailable`.
|
|
100
|
+
`dataPath` is `ok`, `failed` or `skipped` (no ICMP ping without a disco pong). A failed ping also
|
|
101
|
+
logs `code`, `signal` and `killed`.
|
|
102
|
+
|
|
103
|
+
Public `/api/health` carries:
|
|
104
|
+
|
|
105
|
+
```json
|
|
106
|
+
"tailscaleNudge": {
|
|
107
|
+
"enabled": true,
|
|
108
|
+
"available": true,
|
|
109
|
+
"lastAt": "2026-10-23T20:05:10.000Z",
|
|
110
|
+
"lastTrigger": "silent",
|
|
111
|
+
"lastOutcome": "pong_direct",
|
|
112
|
+
"lastDataPath": "ok",
|
|
113
|
+
"lastRecoveredAt": "2026-10-23T20:05:14.200Z",
|
|
114
|
+
"nudgesLastHour": 1
|
|
115
|
+
}
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
`available` is `null` until the first look for the CLI. `lastRecoveredAt` is when a nudged peer next
|
|
119
|
+
made a request, if that was within 90 s of the nudge. No address or device name is published.
|
|
120
|
+
|
|
121
|
+
## The phone side (contract for glasses 6.10.653; not built in this release)
|
|
122
|
+
|
|
123
|
+
- **Retry with a fresh request.** On a reconnect failure, retry `/api/health` and then the
|
|
124
|
+
authenticated `/api/models` with `cache: 'no-store'` and a cache-busting query, on a fresh
|
|
125
|
+
connection where the WebView allows it (do not reuse a stalled keep-alive socket; a new
|
|
126
|
+
`EventSource` for the display stream). Backoff ladder: 2, 4, 8, 15, then every 30 s while the app
|
|
127
|
+
is in the foreground. Never a tight loop.
|
|
128
|
+
- **Wait out the Mac-side window before blaming Tailscale.** The Mac's nudge fires 45 s into a
|
|
129
|
+
silence (or 30 s after a Mac boot, 15 s after a wake) and its pings take up to about 30 s. Show
|
|
130
|
+
6.10.652's stuck-Tailscale message only after 90 s of failed fresh requests while the phone has
|
|
131
|
+
network (the gotcos.com reachability check). Before that, show the ordinary reconnecting state.
|
|
132
|
+
- **Read the outcome.** `/api/health` is public, so once the phone reaches it, `tailscaleNudge`
|
|
133
|
+
tells it whether the Mac nudged and when. If health is reachable but authenticated calls are not,
|
|
134
|
+
the tunnel is fine and the problem is elsewhere: do not show the Tailscale message.
|
|
135
|
+
- **Keep working offline.** Local-first capture and backfill are unchanged: record locally, upload
|
|
136
|
+
when the Mac answers.
|
|
137
|
+
- **What to tell Miles about VPN On Demand.** In the iOS Tailscale app (Settings, VPN On Demand),
|
|
138
|
+
turning it on makes iOS bring the tunnel back by itself after it drops, for example after a network
|
|
139
|
+
change. It is a phone setting he can choose; COS cannot change it. It does not replace the nudge: a
|
|
140
|
+
tunnel iOS considers "up" but stale is not a drop, which is the case the Mac-side nudge targets.
|
|
141
|
+
|
|
142
|
+
## Field canary (Miles)
|
|
143
|
+
|
|
144
|
+
1. Note the time. Phone on cellular or away from home, COS open on the phone, Tailscale on.
|
|
145
|
+
2. On the Mac, either quit and reopen Tailscale (menu bar, Quit, then open it), or restart the Mac.
|
|
146
|
+
3. Do not touch Tailscale on the phone. Use COS on the phone within about two minutes.
|
|
147
|
+
|
|
148
|
+
Success:
|
|
149
|
+
|
|
150
|
+
- **Log** (the server log; with COS Control, `~/Library/Logs/COS Glasses/server.log`): a `[tailscale-nudge] trigger=tailscale` (Tailscale restart) or
|
|
151
|
+
`trigger=boot` (Mac restart) line with `outcome=pong_direct` or `pong_relay` and `dataPath=ok`,
|
|
152
|
+
then `peer answered ...s after the ... nudge`.
|
|
153
|
+
- **Health:** `curl -s http://127.0.0.1:3141/api/health` shows `tailscaleNudge.lastTrigger` matching,
|
|
154
|
+
`lastOutcome` a pong, and `lastRecoveredAt` set a few seconds after `lastAt`.
|
|
155
|
+
- **Phone:** COS reconnects by itself, without a Tailscale toggle, and a meeting started before the
|
|
156
|
+
restart keeps its chunks (transfer integrity 100%).
|
|
157
|
+
|
|
158
|
+
If the phone still needs a toggle: record the nudge log line (its `handshakeAgeS` and `rxDelta` are
|
|
159
|
+
the evidence for the next step) and the time of the toggle.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@gotcos/glasses-server",
|
|
3
|
-
"version": "6.
|
|
3
|
+
"version": "6.89.0",
|
|
4
4
|
"description": "COS Glasses: self-hosted AI heads-up-display server for Even G2 smart glasses, powered by Claude Code, Codex, Cursor Agent CLI, or local Ollama",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -136,6 +136,9 @@
|
|
|
136
136
|
"!server/scripts/question-hold-mutation-gate.mjs",
|
|
137
137
|
"!server/scripts/live-meeting-6860-mutation-gate.mjs",
|
|
138
138
|
"!server/scripts/live-meeting-6870-mutation-gate.mjs",
|
|
139
|
+
"!server/scripts/integrity-6880-mutation-gate.mjs",
|
|
140
|
+
"!server/scripts/tailscale-nudge-6890-mutation-gate.mjs",
|
|
141
|
+
"docs/tailscale-self-heal.md",
|
|
139
142
|
"docs/test-plan-6.71.8.md",
|
|
140
143
|
"docs/test-plan-6.71.9.md",
|
|
141
144
|
"docs/test-plan-6.71.10.md",
|
package/server/index.ts
CHANGED
|
@@ -14,7 +14,8 @@ import { execSync } from 'node:child_process'
|
|
|
14
14
|
import { randomBytes, randomUUID } from 'node:crypto'
|
|
15
15
|
import { healthRouter } from './routes/health.js'
|
|
16
16
|
import { diagRouter } from './routes/diag.js'
|
|
17
|
-
import { createClientInstanceRouter } from './routes/client-instance.js'
|
|
17
|
+
import { createClientInstanceRouter, isMeetingChunkPost } from './routes/client-instance.js'
|
|
18
|
+
import { installTailscaleNudger } from './lib/tailscale-nudge.js'
|
|
18
19
|
import { queryRouter } from './routes/query.js'
|
|
19
20
|
import { providerProofRouter } from './routes/provider-proof.js'
|
|
20
21
|
import { transcribeRouter } from './routes/transcribe.js'
|
|
@@ -188,7 +189,7 @@ import {
|
|
|
188
189
|
isLoopbackSocket,
|
|
189
190
|
reachableIpv4Addresses,
|
|
190
191
|
} from './lib/network-policy.js'
|
|
191
|
-
import { requireApiToken } from './lib/api-auth.js'
|
|
192
|
+
import { isPublicApiRequest, requireApiToken } from './lib/api-auth.js'
|
|
192
193
|
import { PairingState, pairingHostsFrom } from './lib/glasses-pairing.js'
|
|
193
194
|
import { createPairingRouter } from './routes/pairing.js'
|
|
194
195
|
import { isManagedRuntime } from './lib/managed-runtime.js'
|
|
@@ -355,9 +356,23 @@ const pairingState = new PairingState({
|
|
|
355
356
|
serverName: () => hostname().replace(/\.local$/i, ''),
|
|
356
357
|
hosts: lanArmed => pairingHostsFrom(reachableIpv4Addresses(networkInterfaces()), PORT, lanArmed),
|
|
357
358
|
})
|
|
359
|
+
// 6.89.0 Tailscale self-heal (lib/tailscale-nudge.ts, docs/tailscale-self-heal.md). Every /api
|
|
360
|
+
// request from a Tailscale address is noted BEFORE the token check, public routes included: any
|
|
361
|
+
// request that arrives proves the tunnel carries COS traffic. Authenticated ones also mark the
|
|
362
|
+
// address as a paired phone. Notes only, never answers; noteRequest never throws.
|
|
363
|
+
const tailscaleNudger = installTailscaleNudger({ activeMeetings: () => getActiveTranscriptionSessionCount() })
|
|
364
|
+
app.use('/api', (req, _res, next) => {
|
|
365
|
+
tailscaleNudger.noteRequest(req.socket?.remoteAddress, { meetingChunk: isMeetingChunkPost(req.method, req.path) })
|
|
366
|
+
next()
|
|
367
|
+
})
|
|
358
368
|
app.use('/api', requireApiToken(API_TOKEN, {
|
|
359
369
|
onAuthenticated: req => pairingState.noteAuthenticated(req.socket?.remoteAddress),
|
|
360
370
|
}))
|
|
371
|
+
// Past the token gate, a request on a non-public path carried the token: mark a paired phone.
|
|
372
|
+
app.use('/api', (req, _res, next) => {
|
|
373
|
+
if (!isPublicApiRequest(req.method, req.path)) tailscaleNudger.noteRequest(req.socket?.remoteAddress, { authenticated: true })
|
|
374
|
+
next()
|
|
375
|
+
})
|
|
361
376
|
|
|
362
377
|
// Pairing routes sit HERE on purpose: after the token gate (mint, status and decision
|
|
363
378
|
// need it; the claim and its poll are exact public doors in api-auth.ts), and BEFORE the
|
|
@@ -1274,6 +1289,7 @@ async function gracefulShutdown(): Promise<void> {
|
|
|
1274
1289
|
await meetingAskRuntime.shutdown()
|
|
1275
1290
|
stopMorningBriefScheduler()
|
|
1276
1291
|
stopRoutineScheduler()
|
|
1292
|
+
tailscaleNudger.stop()
|
|
1277
1293
|
await workReviewRuntime?.close()
|
|
1278
1294
|
workIntakeStore?.close()
|
|
1279
1295
|
workHandoffRequestStore?.close()
|
|
@@ -1476,6 +1492,8 @@ listenRequiredServers(listeners).then(() => {
|
|
|
1476
1492
|
// Hash the pinned FluidAudio CLI once at boot so the first Settings GET already knows `installable`.
|
|
1477
1493
|
void findPinnedCli().catch(() => null)
|
|
1478
1494
|
startPreviewWarmLoop()
|
|
1495
|
+
// 6.89.0: the Tailscale nudge loop. Its boot trigger is due 30 s from here. COS_TAILSCALE_NUDGE=0 stops it.
|
|
1496
|
+
tailscaleNudger.start()
|
|
1479
1497
|
}
|
|
1480
1498
|
|
|
1481
1499
|
// Durable recovery starts with the admitted runtime by default. A literal
|
|
@@ -0,0 +1,536 @@
|
|
|
1
|
+
// 6.89.0 Tailscale self-heal: the Mac-side "nudge" (docs/tailscale-self-heal.md).
|
|
2
|
+
//
|
|
3
|
+
// Why: on 2026-10-22 the Mac restarted at 14:09 and the server answered on every address from
|
|
4
|
+
// 14:10:58, but the iPhone's Tailscale extension kept a stale tunnel. Tailscale's own ping got
|
|
5
|
+
// through, no COS request did, and only toggling Tailscale on the phone (15:09) fixed it. The goal
|
|
6
|
+
// is that a Mac restart, a wake from sleep, a Tailscale restart on the Mac or a network change heals
|
|
7
|
+
// without anyone touching Tailscale.
|
|
8
|
+
//
|
|
9
|
+
// What a nudge is: two bounded `tailscale ping`s at one peer, run with execFile from a fixed path,
|
|
10
|
+
// never a shell:
|
|
11
|
+
// 1. `ping -c 3 --timeout 5s <ip>`: a disco ping. It makes both ends redo path discovery, and it
|
|
12
|
+
// stops early once a direct path answers.
|
|
13
|
+
// 2. only after a pong, `ping --icmp -c 1 --timeout 5s <ip>`: a packet THROUGH WireGuard. The disco
|
|
14
|
+
// ping alone can succeed while the data path is stale (it does not use the tunnel, which is what
|
|
15
|
+
// 10/22 looked like); a data packet makes WireGuard start a fresh handshake when its session is
|
|
16
|
+
// stale.
|
|
17
|
+
// Whether this un-sticks the iOS extension in the field is NOT proven here; the field canary is.
|
|
18
|
+
//
|
|
19
|
+
// When (triggers), each needing the POSITIVE signal "tailscale status says this peer is Online"
|
|
20
|
+
// from a status read at most STATUS_FRESH_MS old. Absence of requests alone never nudges.
|
|
21
|
+
// boot 30 s after the server starts.
|
|
22
|
+
// wake the wall clock moved 20 s+ more than the monotonic clock (the Mac slept).
|
|
23
|
+
// network this Mac's IPv4 interfaces changed (Tailscale's utun included).
|
|
24
|
+
// tailscale Tailscale's own state changed: BackendState, this Mac's Tailscale IPs, or the peer
|
|
25
|
+
// byte counters went backwards (tailscaled restarted).
|
|
26
|
+
// silent a phone that made an authenticated request in the last 5 minutes, or sent meeting
|
|
27
|
+
// chunks in the last 15 minutes while a meeting is open, has made no request at all
|
|
28
|
+
// for 45 s, while its peer is Online.
|
|
29
|
+
// Event triggers (boot, wake, network, tailscale) aim at peers this process has seen authenticate in
|
|
30
|
+
// the last 12 hours, plus Online iOS and Android peers (after a restart the Mac remembers no one),
|
|
31
|
+
// at most 4, and skip any peer that has already made a request since the event.
|
|
32
|
+
//
|
|
33
|
+
// Caps: one nudge per peer per 60 s, doubling per nudge that brings no request (60, 120, 240, 480,
|
|
34
|
+
// then 900 s), reset by any request from that peer; 30 nudges an hour in all; one at a time.
|
|
35
|
+
// COS_TAILSCALE_NUDGE=0 (or false/off) stops everything, read every tick, no restart needed.
|
|
36
|
+
// No LLM, no disk write, no shell. Logs carry trigger, outcome and timing only, never an address.
|
|
37
|
+
|
|
38
|
+
import { execFile } from 'node:child_process'
|
|
39
|
+
import { accessSync, constants as fsConstants } from 'node:fs'
|
|
40
|
+
import { networkInterfaces } from 'node:os'
|
|
41
|
+
import { readSwitch } from './env-switch.js'
|
|
42
|
+
import { isTailscaleIpv4, normalizeRemoteIp } from './network-policy.js'
|
|
43
|
+
|
|
44
|
+
export const TAILSCALE_NUDGE_LIMITS = Object.freeze({
|
|
45
|
+
tickMs: 10_000,
|
|
46
|
+
bootDelayMs: 30_000,
|
|
47
|
+
wakeSettleMs: 15_000,
|
|
48
|
+
networkSettleMs: 10_000,
|
|
49
|
+
/** An event trigger that cannot run (Tailscale not Running yet) is dropped after this. */
|
|
50
|
+
eventExpiryMs: 120_000,
|
|
51
|
+
/** The monotonic clock stops while the Mac sleeps; a wall-clock lead past this is a wake. */
|
|
52
|
+
wakeGapMs: 20_000,
|
|
53
|
+
silenceMs: 45_000,
|
|
54
|
+
activeWindowMs: 5 * 60_000,
|
|
55
|
+
meetingWindowMs: 15 * 60_000,
|
|
56
|
+
knownPeerWindowMs: 12 * 60 * 60_000,
|
|
57
|
+
minIntervalMs: 60_000,
|
|
58
|
+
maxBackoffMs: 15 * 60_000,
|
|
59
|
+
maxPerHour: 30,
|
|
60
|
+
eventMaxPeers: 4,
|
|
61
|
+
statusPollMs: 60_000,
|
|
62
|
+
statusFreshMs: 15_000,
|
|
63
|
+
recoveredWithinMs: 90_000,
|
|
64
|
+
discoTimeoutMs: 20_000,
|
|
65
|
+
icmpTimeoutMs: 10_000,
|
|
66
|
+
statusTimeoutMs: 5_000,
|
|
67
|
+
binRecheckMs: 10 * 60_000,
|
|
68
|
+
maxTrackedPeers: 16,
|
|
69
|
+
})
|
|
70
|
+
|
|
71
|
+
/** Fixed places the Tailscale CLI lives. No PATH lookup and no override: the path is never input. */
|
|
72
|
+
export const TAILSCALE_BIN_CANDIDATES = Object.freeze([
|
|
73
|
+
'/Applications/Tailscale.app/Contents/MacOS/Tailscale',
|
|
74
|
+
'/opt/homebrew/bin/tailscale',
|
|
75
|
+
'/usr/local/bin/tailscale',
|
|
76
|
+
'/usr/bin/tailscale',
|
|
77
|
+
])
|
|
78
|
+
|
|
79
|
+
export const DISCO_PING_ARGS = (ip: string): string[] => ['ping', '-c', '3', '--timeout', '5s', ip]
|
|
80
|
+
export const ICMP_PING_ARGS = (ip: string): string[] => ['ping', '--icmp', '-c', '1', '--timeout', '5s', ip]
|
|
81
|
+
export const STATUS_ARGS: readonly string[] = Object.freeze(['status', '--json'])
|
|
82
|
+
|
|
83
|
+
export type NudgeTrigger = 'boot' | 'wake' | 'network' | 'tailscale' | 'silent'
|
|
84
|
+
export type NudgeOutcome = 'pong_direct' | 'pong_relay' | 'no_pong' | 'timeout' | 'error' | 'unavailable'
|
|
85
|
+
export type DataPathOutcome = 'ok' | 'failed' | 'skipped'
|
|
86
|
+
|
|
87
|
+
export interface RunResult {
|
|
88
|
+
code: number | null
|
|
89
|
+
signal: string | null
|
|
90
|
+
killed: boolean
|
|
91
|
+
stdout: string
|
|
92
|
+
stderr: string
|
|
93
|
+
/** Set when the process could not be started at all (ENOENT, EACCES). */
|
|
94
|
+
spawnError?: string
|
|
95
|
+
elapsedMs: number
|
|
96
|
+
}
|
|
97
|
+
/** The one way this module runs Tailscale: a binary path and an argument array, never a shell. */
|
|
98
|
+
export type TailscaleRunner = (bin: string, args: readonly string[], timeoutMs: number) => Promise<RunResult>
|
|
99
|
+
|
|
100
|
+
export const execFileRunner: TailscaleRunner = (bin, args, timeoutMs) => new Promise(resolve => {
|
|
101
|
+
const started = Date.now()
|
|
102
|
+
execFile(bin, [...args], { timeout: timeoutMs, killSignal: 'SIGKILL', maxBuffer: 1024 * 1024, windowsHide: true, encoding: 'utf8' },
|
|
103
|
+
(error, stdout, stderr) => {
|
|
104
|
+
const err = error as (NodeJS.ErrnoException & { code?: number | string; killed?: boolean; signal?: string }) | null
|
|
105
|
+
const spawnError = err && typeof err.code === 'string' ? err.code : undefined
|
|
106
|
+
resolve({
|
|
107
|
+
code: err ? (typeof err.code === 'number' ? err.code : null) : 0,
|
|
108
|
+
signal: err?.signal ?? null,
|
|
109
|
+
killed: !!err?.killed,
|
|
110
|
+
stdout: String(stdout ?? ''),
|
|
111
|
+
stderr: String(stderr ?? ''),
|
|
112
|
+
...(spawnError ? { spawnError } : {}),
|
|
113
|
+
elapsedMs: Date.now() - started,
|
|
114
|
+
})
|
|
115
|
+
})
|
|
116
|
+
})
|
|
117
|
+
|
|
118
|
+
export function resolveTailscaleBin(isExecutable: (p: string) => boolean = defaultIsExecutable): string | null {
|
|
119
|
+
for (const candidate of TAILSCALE_BIN_CANDIDATES) if (isExecutable(candidate)) return candidate
|
|
120
|
+
return null
|
|
121
|
+
}
|
|
122
|
+
function defaultIsExecutable(p: string): boolean {
|
|
123
|
+
try { accessSync(p, fsConstants.X_OK); return true } catch { return false }
|
|
124
|
+
}
|
|
125
|
+
|
|
126
|
+
export interface PeerStatus {
|
|
127
|
+
ip: string
|
|
128
|
+
online: boolean
|
|
129
|
+
os: string
|
|
130
|
+
rxBytes: number
|
|
131
|
+
txBytes: number
|
|
132
|
+
/** Epoch ms of the last WireGuard handshake, or 0 for none. */
|
|
133
|
+
lastHandshakeMs: number
|
|
134
|
+
}
|
|
135
|
+
export interface TailscaleStatus {
|
|
136
|
+
backendState: string
|
|
137
|
+
selfIps: string[]
|
|
138
|
+
peers: PeerStatus[]
|
|
139
|
+
at: number
|
|
140
|
+
}
|
|
141
|
+
|
|
142
|
+
/** Parse `tailscale status --json`. Returns null for anything that is not that shape. */
|
|
143
|
+
export function parseTailscaleStatus(text: string, at: number): TailscaleStatus | null {
|
|
144
|
+
let raw: any
|
|
145
|
+
try { raw = JSON.parse(text) } catch { return null }
|
|
146
|
+
if (!raw || typeof raw !== 'object' || typeof raw.BackendState !== 'string') return null
|
|
147
|
+
const v4 = (ips: unknown): string[] => Array.isArray(ips) ? ips.filter((ip): ip is string => typeof ip === 'string' && isTailscaleIpv4(ip)) : []
|
|
148
|
+
const peers: PeerStatus[] = []
|
|
149
|
+
const peerMap = raw.Peer && typeof raw.Peer === 'object' ? raw.Peer : {}
|
|
150
|
+
for (const p of Object.values(peerMap) as any[]) {
|
|
151
|
+
const ip = v4(p?.TailscaleIPs)[0]
|
|
152
|
+
if (!ip) continue
|
|
153
|
+
const hs = typeof p.LastHandshake === 'string' ? Date.parse(p.LastHandshake) : NaN
|
|
154
|
+
peers.push({
|
|
155
|
+
ip,
|
|
156
|
+
online: p.Online === true,
|
|
157
|
+
os: typeof p.OS === 'string' ? p.OS : '',
|
|
158
|
+
rxBytes: Number.isFinite(p.RxBytes) ? p.RxBytes : 0,
|
|
159
|
+
txBytes: Number.isFinite(p.TxBytes) ? p.TxBytes : 0,
|
|
160
|
+
lastHandshakeMs: Number.isFinite(hs) && hs > 0 ? hs : 0,
|
|
161
|
+
})
|
|
162
|
+
}
|
|
163
|
+
return { backendState: raw.BackendState, selfIps: v4(raw.Self?.TailscaleIPs).sort(), peers, at }
|
|
164
|
+
}
|
|
165
|
+
|
|
166
|
+
/** Disco ping output: "pong from <name> (<ip>) via <DERP(x) | ip:port | peer-relay(...)> in 91ms". */
|
|
167
|
+
export function classifyDiscoPing(result: RunResult): NudgeOutcome {
|
|
168
|
+
if (result.spawnError) return 'unavailable'
|
|
169
|
+
const pong = /pong from .* via (\S+) in [\d.]+ms/.exec(result.stdout)
|
|
170
|
+
if (pong) return /^(DERP|peer-relay)/i.test(pong[1]) ? 'pong_relay' : 'pong_direct'
|
|
171
|
+
if (result.killed || result.signal) return 'timeout'
|
|
172
|
+
if (/no reply|timed out|timeout/i.test(result.stdout + result.stderr)) return 'no_pong'
|
|
173
|
+
return result.code === 0 ? 'no_pong' : 'error'
|
|
174
|
+
}
|
|
175
|
+
export function classifyIcmpPing(result: RunResult): DataPathOutcome {
|
|
176
|
+
return !result.spawnError && /pong from .* via ICMP/i.test(result.stdout) ? 'ok' : 'failed'
|
|
177
|
+
}
|
|
178
|
+
|
|
179
|
+
/** Every external IPv4 interface address, sorted: changes when a network or the Tailscale tunnel changes. */
|
|
180
|
+
type InterfaceInfo = { address: string; family: string | number; internal: boolean }
|
|
181
|
+
export function networkFingerprint(nets: Record<string, InterfaceInfo[] | undefined> = networkInterfaces()): string {
|
|
182
|
+
const out: string[] = []
|
|
183
|
+
for (const [name, infos] of Object.entries(nets)) {
|
|
184
|
+
for (const info of infos ?? []) {
|
|
185
|
+
const family = typeof info.family === 'number' ? (info.family === 4 ? 'IPv4' : 'IPv6') : String(info.family)
|
|
186
|
+
if (family === 'IPv4' && !info.internal) out.push(`${name}=${info.address}`)
|
|
187
|
+
}
|
|
188
|
+
}
|
|
189
|
+
return out.sort().join(',')
|
|
190
|
+
}
|
|
191
|
+
|
|
192
|
+
/** Did Tailscale's own state change between two status reads? A failed read is not a change. */
|
|
193
|
+
export function tailscaleStateChanged(prev: TailscaleStatus | null, next: TailscaleStatus | null): boolean {
|
|
194
|
+
if (!prev || !next) return false
|
|
195
|
+
if (prev.backendState !== next.backendState) return true
|
|
196
|
+
if (prev.selfIps.join(',') !== next.selfIps.join(',')) return true
|
|
197
|
+
// tailscaled keeps per-peer byte counters for its lifetime: a total that went DOWN means it restarted.
|
|
198
|
+
const total = (s: TailscaleStatus) => s.peers.reduce((n, p) => n + p.rxBytes + p.txBytes, 0)
|
|
199
|
+
return total(next) < total(prev)
|
|
200
|
+
}
|
|
201
|
+
|
|
202
|
+
/** Seconds between nudges for a peer, given how many nudges in a row brought no request. */
|
|
203
|
+
export function backoffMs(consecutive: number): number {
|
|
204
|
+
if (consecutive <= 0) return 0
|
|
205
|
+
const { minIntervalMs, maxBackoffMs } = TAILSCALE_NUDGE_LIMITS
|
|
206
|
+
return Math.min(minIntervalMs * 2 ** (consecutive - 1), maxBackoffMs)
|
|
207
|
+
}
|
|
208
|
+
|
|
209
|
+
export function isMobileOs(os: string): boolean {
|
|
210
|
+
return /^(ios|android)$/i.test(os.trim())
|
|
211
|
+
}
|
|
212
|
+
|
|
213
|
+
interface PeerTrack {
|
|
214
|
+
/** Any /api request from this address, public routes included: proof the data path works. */
|
|
215
|
+
lastRequestAt: number
|
|
216
|
+
lastAuthenticatedAt: number
|
|
217
|
+
lastMeetingChunkAt: number
|
|
218
|
+
lastNudgeAt: number
|
|
219
|
+
/** Nudges since the last request from this peer. */
|
|
220
|
+
consecutive: number
|
|
221
|
+
}
|
|
222
|
+
|
|
223
|
+
export interface TailscaleNudgeDeps {
|
|
224
|
+
runner?: TailscaleRunner
|
|
225
|
+
now?: () => number
|
|
226
|
+
/** Monotonic milliseconds; it does not advance while the Mac sleeps. */
|
|
227
|
+
monotonic?: () => number
|
|
228
|
+
env?: () => NodeJS.ProcessEnv
|
|
229
|
+
resolveBin?: () => string | null
|
|
230
|
+
fingerprint?: () => string
|
|
231
|
+
activeMeetings?: () => number
|
|
232
|
+
log?: (line: string) => void
|
|
233
|
+
}
|
|
234
|
+
|
|
235
|
+
export interface TailscaleNudgeHealth {
|
|
236
|
+
enabled: boolean
|
|
237
|
+
/** null until the first look for the Tailscale CLI; false when it is not installed. */
|
|
238
|
+
available: boolean | null
|
|
239
|
+
lastAt: string | null
|
|
240
|
+
lastTrigger: NudgeTrigger | null
|
|
241
|
+
lastOutcome: NudgeOutcome | null
|
|
242
|
+
lastDataPath: DataPathOutcome | null
|
|
243
|
+
/** When a nudged peer next made a request, within 90 s of the nudge. */
|
|
244
|
+
lastRecoveredAt: string | null
|
|
245
|
+
nudgesLastHour: number
|
|
246
|
+
}
|
|
247
|
+
|
|
248
|
+
interface PendingEvent { trigger: Exclude<NudgeTrigger, 'silent'>; firedAt: number; dueAt: number }
|
|
249
|
+
|
|
250
|
+
export class TailscaleNudger {
|
|
251
|
+
private readonly runner: TailscaleRunner
|
|
252
|
+
private readonly now: () => number
|
|
253
|
+
private readonly monotonic: () => number
|
|
254
|
+
private readonly env: () => NodeJS.ProcessEnv
|
|
255
|
+
private readonly resolveBinFn: () => string | null
|
|
256
|
+
private readonly fingerprintFn: () => string
|
|
257
|
+
private readonly activeMeetings: () => number
|
|
258
|
+
private readonly log: (line: string) => void
|
|
259
|
+
|
|
260
|
+
private readonly peers = new Map<string, PeerTrack>()
|
|
261
|
+
private readonly hourLog: number[] = []
|
|
262
|
+
private pending: PendingEvent | null = null
|
|
263
|
+
private bin: string | null | undefined = undefined
|
|
264
|
+
private binCheckedAt = 0
|
|
265
|
+
private status: TailscaleStatus | null = null
|
|
266
|
+
private previousStatus: TailscaleStatus | null = null
|
|
267
|
+
private lastStatusAttemptAt = 0
|
|
268
|
+
private lastFingerprint: string | null = null
|
|
269
|
+
private lastTick: { wall: number; mono: number } | null = null
|
|
270
|
+
private busy = false
|
|
271
|
+
private timer: ReturnType<typeof setInterval> | null = null
|
|
272
|
+
private lastNudge: { at: number; ip: string; trigger: NudgeTrigger; outcome: NudgeOutcome; dataPath: DataPathOutcome } | null = null
|
|
273
|
+
private lastRecoveredAt: number | null = null
|
|
274
|
+
|
|
275
|
+
constructor(deps: TailscaleNudgeDeps = {}) {
|
|
276
|
+
this.runner = deps.runner ?? execFileRunner
|
|
277
|
+
this.now = deps.now ?? Date.now
|
|
278
|
+
this.monotonic = deps.monotonic ?? (() => Number(process.hrtime.bigint() / 1_000_000n))
|
|
279
|
+
this.env = deps.env ?? (() => process.env)
|
|
280
|
+
this.resolveBinFn = deps.resolveBin ?? (() => resolveTailscaleBin())
|
|
281
|
+
this.fingerprintFn = deps.fingerprint ?? (() => networkFingerprint())
|
|
282
|
+
this.activeMeetings = deps.activeMeetings ?? (() => 0)
|
|
283
|
+
this.log = deps.log ?? (line => console.log(line))
|
|
284
|
+
}
|
|
285
|
+
|
|
286
|
+
enabled(): boolean {
|
|
287
|
+
return readSwitch(this.env().COS_TAILSCALE_NUDGE, true)
|
|
288
|
+
}
|
|
289
|
+
|
|
290
|
+
/** Start the tick. The boot trigger is due 30 s from now. Timers are unref'd. */
|
|
291
|
+
start(): void {
|
|
292
|
+
if (this.timer) return
|
|
293
|
+
const at = this.now()
|
|
294
|
+
this.pending = { trigger: 'boot', firedAt: at, dueAt: at + TAILSCALE_NUDGE_LIMITS.bootDelayMs }
|
|
295
|
+
this.timer = setInterval(() => { void this.tick() }, TAILSCALE_NUDGE_LIMITS.tickMs)
|
|
296
|
+
this.timer.unref?.()
|
|
297
|
+
}
|
|
298
|
+
|
|
299
|
+
stop(): void {
|
|
300
|
+
if (this.timer) clearInterval(this.timer)
|
|
301
|
+
this.timer = null
|
|
302
|
+
}
|
|
303
|
+
|
|
304
|
+
/**
|
|
305
|
+
* Called for every /api request (before the token check) with the socket address. Only
|
|
306
|
+
* Tailscale addresses are tracked. Never throws.
|
|
307
|
+
*/
|
|
308
|
+
noteRequest(remoteAddress: string | undefined | null, kind: { authenticated?: boolean; meetingChunk?: boolean } = {}): void {
|
|
309
|
+
try {
|
|
310
|
+
const ip = normalizeRemoteIp(remoteAddress ?? '')
|
|
311
|
+
if (!isTailscaleIpv4(ip)) return
|
|
312
|
+
const at = this.now()
|
|
313
|
+
let track = this.peers.get(ip)
|
|
314
|
+
if (!track) {
|
|
315
|
+
track = { lastRequestAt: 0, lastAuthenticatedAt: 0, lastMeetingChunkAt: 0, lastNudgeAt: 0, consecutive: 0 }
|
|
316
|
+
}
|
|
317
|
+
this.peers.delete(ip)
|
|
318
|
+
this.peers.set(ip, track)
|
|
319
|
+
while (this.peers.size > TAILSCALE_NUDGE_LIMITS.maxTrackedPeers) this.peers.delete(this.peers.keys().next().value as string)
|
|
320
|
+
if (track.consecutive > 0 && this.lastNudge?.ip === ip && at - this.lastNudge.at <= TAILSCALE_NUDGE_LIMITS.recoveredWithinMs
|
|
321
|
+
&& (this.lastRecoveredAt === null || this.lastRecoveredAt < this.lastNudge.at)) {
|
|
322
|
+
this.lastRecoveredAt = at
|
|
323
|
+
this.log(`[tailscale-nudge] peer answered ${((at - this.lastNudge.at) / 1000).toFixed(1)}s after the ${this.lastNudge.trigger} nudge`)
|
|
324
|
+
}
|
|
325
|
+
track.lastRequestAt = at
|
|
326
|
+
track.consecutive = 0
|
|
327
|
+
if (kind.authenticated) track.lastAuthenticatedAt = at
|
|
328
|
+
if (kind.meetingChunk) track.lastMeetingChunkAt = at
|
|
329
|
+
} catch { /* observer only */ }
|
|
330
|
+
}
|
|
331
|
+
|
|
332
|
+
health(): TailscaleNudgeHealth {
|
|
333
|
+
const at = this.now()
|
|
334
|
+
return {
|
|
335
|
+
enabled: this.enabled(),
|
|
336
|
+
available: this.bin === undefined ? null : this.bin !== null,
|
|
337
|
+
lastAt: this.lastNudge ? new Date(this.lastNudge.at).toISOString() : null,
|
|
338
|
+
lastTrigger: this.lastNudge?.trigger ?? null,
|
|
339
|
+
lastOutcome: this.lastNudge?.outcome ?? null,
|
|
340
|
+
lastDataPath: this.lastNudge?.dataPath ?? null,
|
|
341
|
+
lastRecoveredAt: this.lastRecoveredAt !== null ? new Date(this.lastRecoveredAt).toISOString() : null,
|
|
342
|
+
nudgesLastHour: this.hourLog.filter(t => at - t < 3_600_000).length,
|
|
343
|
+
}
|
|
344
|
+
}
|
|
345
|
+
|
|
346
|
+
/** One decision pass. Public for tests; the interval calls it. Never throws, never overlaps. */
|
|
347
|
+
async tick(): Promise<void> {
|
|
348
|
+
if (this.busy) return
|
|
349
|
+
this.busy = true
|
|
350
|
+
try { await this.tickInner() } catch { /* a failed pass waits for the next tick */ } finally { this.busy = false }
|
|
351
|
+
}
|
|
352
|
+
|
|
353
|
+
private async tickInner(): Promise<void> {
|
|
354
|
+
const at = this.now()
|
|
355
|
+
const mono = this.monotonic()
|
|
356
|
+
const L = TAILSCALE_NUDGE_LIMITS
|
|
357
|
+
// Wake and network detection run even when switched off, so turning it on later starts clean.
|
|
358
|
+
if (this.lastTick && (at - this.lastTick.wall) - (mono - this.lastTick.mono) > L.wakeGapMs) {
|
|
359
|
+
this.raise('wake', at, L.wakeSettleMs)
|
|
360
|
+
}
|
|
361
|
+
this.lastTick = { wall: at, mono }
|
|
362
|
+
const fp = safe(this.fingerprintFn, '')
|
|
363
|
+
if (this.lastFingerprint !== null && fp !== this.lastFingerprint) this.raise('network', at, L.networkSettleMs)
|
|
364
|
+
this.lastFingerprint = fp
|
|
365
|
+
|
|
366
|
+
if (!this.enabled()) return
|
|
367
|
+
if (!this.ensureBin(at)) return
|
|
368
|
+
|
|
369
|
+
if (at - this.lastStatusAttemptAt >= L.statusPollMs) {
|
|
370
|
+
// Compared only on a successful read: a failed one leaves the last good status in place.
|
|
371
|
+
const before = this.status
|
|
372
|
+
const read = await this.readStatus(at)
|
|
373
|
+
if (read && tailscaleStateChanged(before, read)) this.raise('tailscale', at, L.networkSettleMs)
|
|
374
|
+
}
|
|
375
|
+
|
|
376
|
+
const pending = this.pending
|
|
377
|
+
if (pending && at >= pending.dueAt) {
|
|
378
|
+
await this.runEvent(pending, at)
|
|
379
|
+
return
|
|
380
|
+
}
|
|
381
|
+
await this.runSilent(at)
|
|
382
|
+
}
|
|
383
|
+
|
|
384
|
+
private raise(trigger: PendingEvent['trigger'], at: number, settleMs: number): void {
|
|
385
|
+
// A boot still waiting keeps its own due time; any later event replaces an earlier one.
|
|
386
|
+
if (this.pending?.trigger === 'boot' && trigger !== 'boot') {
|
|
387
|
+
this.pending = { trigger, firedAt: this.pending.firedAt, dueAt: Math.max(this.pending.dueAt, at + settleMs) }
|
|
388
|
+
return
|
|
389
|
+
}
|
|
390
|
+
this.pending = { trigger, firedAt: at, dueAt: at + settleMs }
|
|
391
|
+
}
|
|
392
|
+
|
|
393
|
+
private ensureBin(at: number): boolean {
|
|
394
|
+
if (this.bin === undefined || (this.bin === null && at - this.binCheckedAt >= TAILSCALE_NUDGE_LIMITS.binRecheckMs)) {
|
|
395
|
+
this.bin = safe(this.resolveBinFn, null)
|
|
396
|
+
this.binCheckedAt = at
|
|
397
|
+
}
|
|
398
|
+
return this.bin !== null
|
|
399
|
+
}
|
|
400
|
+
|
|
401
|
+
private async readStatus(at: number): Promise<TailscaleStatus | null> {
|
|
402
|
+
this.lastStatusAttemptAt = at
|
|
403
|
+
const result = await this.runner(this.bin as string, STATUS_ARGS, TAILSCALE_NUDGE_LIMITS.statusTimeoutMs)
|
|
404
|
+
if (result.spawnError === 'ENOENT') this.bin = null
|
|
405
|
+
const parsed = result.spawnError ? null : parseTailscaleStatus(result.stdout, at)
|
|
406
|
+
if (parsed) {
|
|
407
|
+
this.previousStatus = this.status
|
|
408
|
+
this.status = parsed
|
|
409
|
+
}
|
|
410
|
+
return parsed
|
|
411
|
+
}
|
|
412
|
+
|
|
413
|
+
/** A status no older than STATUS_FRESH_MS, read now if needed; null when Tailscale cannot say. */
|
|
414
|
+
private async freshStatus(at: number): Promise<TailscaleStatus | null> {
|
|
415
|
+
if (this.status && at - this.status.at <= TAILSCALE_NUDGE_LIMITS.statusFreshMs) return this.status
|
|
416
|
+
const before = this.status
|
|
417
|
+
const read = await this.readStatus(at)
|
|
418
|
+
if (read && tailscaleStateChanged(before, read)) this.raise('tailscale', at, TAILSCALE_NUDGE_LIMITS.networkSettleMs)
|
|
419
|
+
return read
|
|
420
|
+
}
|
|
421
|
+
|
|
422
|
+
private async runEvent(pending: PendingEvent, at: number): Promise<void> {
|
|
423
|
+
const L = TAILSCALE_NUDGE_LIMITS
|
|
424
|
+
const status = await this.freshStatus(at)
|
|
425
|
+
if (!status || status.backendState !== 'Running') {
|
|
426
|
+
if (at - pending.firedAt > L.eventExpiryMs) {
|
|
427
|
+
this.pending = null
|
|
428
|
+
this.log(`[tailscale-nudge] trigger=${pending.trigger} dropped: Tailscale ${status ? 'not running' : 'did not answer'} for ${Math.round((at - pending.firedAt) / 1000)}s`)
|
|
429
|
+
}
|
|
430
|
+
return
|
|
431
|
+
}
|
|
432
|
+
if (this.pending === pending) this.pending = null
|
|
433
|
+
const known = (ip: string) => {
|
|
434
|
+
const t = this.peers.get(ip)
|
|
435
|
+
return !!t && t.lastAuthenticatedAt > 0 && at - t.lastAuthenticatedAt <= L.knownPeerWindowMs
|
|
436
|
+
}
|
|
437
|
+
const candidates = status.peers
|
|
438
|
+
.filter(p => p.online && (known(p.ip) || isMobileOs(p.os)))
|
|
439
|
+
// Already back since the event: nothing to heal.
|
|
440
|
+
.filter(p => (this.peers.get(p.ip)?.lastRequestAt ?? 0) < pending.firedAt)
|
|
441
|
+
.sort((a, b) => Number(known(b.ip)) - Number(known(a.ip)))
|
|
442
|
+
.slice(0, L.eventMaxPeers)
|
|
443
|
+
for (const peer of candidates) {
|
|
444
|
+
if (!this.enabled()) return
|
|
445
|
+
await this.maybeNudge(peer, pending.trigger, status, at, 0)
|
|
446
|
+
}
|
|
447
|
+
}
|
|
448
|
+
|
|
449
|
+
private async runSilent(at: number): Promise<void> {
|
|
450
|
+
const L = TAILSCALE_NUDGE_LIMITS
|
|
451
|
+
const meetingOpen = safe(this.activeMeetings, 0) > 0
|
|
452
|
+
const silent: Array<[string, PeerTrack]> = []
|
|
453
|
+
for (const [ip, t] of this.peers) {
|
|
454
|
+
const expected = (t.lastAuthenticatedAt > 0 && at - t.lastAuthenticatedAt <= L.activeWindowMs)
|
|
455
|
+
|| (meetingOpen && t.lastMeetingChunkAt > 0 && at - t.lastMeetingChunkAt <= L.meetingWindowMs)
|
|
456
|
+
// Only a peer the caps would let us nudge now: a status read for one we cannot nudge is waste.
|
|
457
|
+
if (expected && at - t.lastRequestAt >= L.silenceMs && this.allowed(ip, at)) silent.push([ip, t])
|
|
458
|
+
}
|
|
459
|
+
if (silent.length === 0) return
|
|
460
|
+
// Never on absence alone: only a peer that Tailscale itself reports Online is nudged.
|
|
461
|
+
const status = await this.freshStatus(at)
|
|
462
|
+
if (!status || status.backendState !== 'Running') return
|
|
463
|
+
for (const [ip, t] of silent) {
|
|
464
|
+
const peer = status.peers.find(p => p.ip === ip)
|
|
465
|
+
if (!peer || !peer.online) continue
|
|
466
|
+
if (!this.enabled()) return
|
|
467
|
+
await this.maybeNudge(peer, 'silent', status, at, at - t.lastRequestAt)
|
|
468
|
+
}
|
|
469
|
+
}
|
|
470
|
+
|
|
471
|
+
private allowed(ip: string, at: number): boolean {
|
|
472
|
+
const L = TAILSCALE_NUDGE_LIMITS
|
|
473
|
+
while (this.hourLog.length > 0 && at - this.hourLog[0] >= 3_600_000) this.hourLog.shift()
|
|
474
|
+
if (this.hourLog.length >= L.maxPerHour) return false
|
|
475
|
+
const t = this.peers.get(ip)
|
|
476
|
+
if (!t || t.lastNudgeAt === 0) return true
|
|
477
|
+
return at - t.lastNudgeAt >= Math.max(L.minIntervalMs, backoffMs(t.consecutive))
|
|
478
|
+
}
|
|
479
|
+
|
|
480
|
+
private async maybeNudge(peer: PeerStatus, trigger: NudgeTrigger, status: TailscaleStatus, at: number, silentMs: number): Promise<void> {
|
|
481
|
+
if (!this.allowed(peer.ip, at)) return
|
|
482
|
+
let track = this.peers.get(peer.ip)
|
|
483
|
+
if (!track) {
|
|
484
|
+
track = { lastRequestAt: 0, lastAuthenticatedAt: 0, lastMeetingChunkAt: 0, lastNudgeAt: 0, consecutive: 0 }
|
|
485
|
+
this.peers.set(peer.ip, track)
|
|
486
|
+
while (this.peers.size > TAILSCALE_NUDGE_LIMITS.maxTrackedPeers) this.peers.delete(this.peers.keys().next().value as string)
|
|
487
|
+
}
|
|
488
|
+
track.lastNudgeAt = at
|
|
489
|
+
track.consecutive += 1
|
|
490
|
+
this.hourLog.push(at)
|
|
491
|
+
const bin = this.bin as string
|
|
492
|
+
const disco = await this.runner(bin, DISCO_PING_ARGS(peer.ip), TAILSCALE_NUDGE_LIMITS.discoTimeoutMs)
|
|
493
|
+
const outcome = classifyDiscoPing(disco)
|
|
494
|
+
if (disco.spawnError === 'ENOENT') this.bin = null
|
|
495
|
+
let dataPath: DataPathOutcome = 'skipped'
|
|
496
|
+
let icmpMs = 0
|
|
497
|
+
if (outcome === 'pong_direct' || outcome === 'pong_relay') {
|
|
498
|
+
const icmp = await this.runner(bin, ICMP_PING_ARGS(peer.ip), TAILSCALE_NUDGE_LIMITS.icmpTimeoutMs)
|
|
499
|
+
dataPath = classifyIcmpPing(icmp)
|
|
500
|
+
icmpMs = icmp.elapsedMs
|
|
501
|
+
}
|
|
502
|
+
this.lastNudge = { at, ip: peer.ip, trigger, outcome, dataPath }
|
|
503
|
+
const prev = this.previousStatus?.peers.find(p => p.ip === peer.ip)
|
|
504
|
+
const rx = prev && this.previousStatus ? ` rxDelta=${peer.rxBytes - prev.rxBytes}/${Math.round((status.at - this.previousStatus.at) / 1000)}s` : ''
|
|
505
|
+
const hs = peer.lastHandshakeMs > 0 ? ` handshakeAgeS=${Math.round((status.at - peer.lastHandshakeMs) / 1000)}` : ' handshakeAgeS=none'
|
|
506
|
+
const failure = outcome === 'error' || outcome === 'timeout' || outcome === 'unavailable'
|
|
507
|
+
? ` code=${disco.code} signal=${disco.signal ?? 'none'} killed=${disco.killed}${disco.spawnError ? ` spawn=${disco.spawnError}` : ''}`
|
|
508
|
+
: ''
|
|
509
|
+
this.log(`[tailscale-nudge] trigger=${trigger} outcome=${outcome} dataPath=${dataPath} pingMs=${disco.elapsedMs} icmpMs=${icmpMs}`
|
|
510
|
+
+ `${silentMs > 0 ? ` silentS=${Math.round(silentMs / 1000)}` : ''}${hs}${rx} inRow=${track.consecutive} lastHour=${this.hourLog.length}${failure}`)
|
|
511
|
+
}
|
|
512
|
+
}
|
|
513
|
+
|
|
514
|
+
function safe<T>(fn: () => T, fallback: T): T {
|
|
515
|
+
try { return fn() } catch { return fallback }
|
|
516
|
+
}
|
|
517
|
+
|
|
518
|
+
let singleton: TailscaleNudger | null = null
|
|
519
|
+
/** The server's nudger. Created on first use so /api/health can read it before start(). */
|
|
520
|
+
export function getTailscaleNudger(): TailscaleNudger {
|
|
521
|
+
if (!singleton) singleton = new TailscaleNudger()
|
|
522
|
+
return singleton
|
|
523
|
+
}
|
|
524
|
+
/** index.ts installs the server's nudger with its real meeting probe, before any request is noted. */
|
|
525
|
+
export function installTailscaleNudger(deps: TailscaleNudgeDeps = {}): TailscaleNudger {
|
|
526
|
+
singleton?.stop()
|
|
527
|
+
singleton = new TailscaleNudger(deps)
|
|
528
|
+
return singleton
|
|
529
|
+
}
|
|
530
|
+
export function __setTailscaleNudgerForTests(nudger: TailscaleNudger | null): void {
|
|
531
|
+
singleton = nudger
|
|
532
|
+
}
|
|
533
|
+
/** `{ tailscaleNudge }` for /api/health. Public route: no address, no peer name, counts and times only. */
|
|
534
|
+
export function tailscaleNudgeHealthField(): { tailscaleNudge: TailscaleNudgeHealth } {
|
|
535
|
+
return { tailscaleNudge: getTailscaleNudger().health() }
|
|
536
|
+
}
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
// 6.88.0: transfer integrity counts what the phone captured, not only what the Mac heard of.
|
|
2
|
+
//
|
|
3
|
+
// WHAT HAPPENED. 2026-10-23 07:59, meeting t2xs9u (glasses 6.10.651). A relaunched COS window saved
|
|
4
|
+
// the meeting from the 74 chunks the Mac held while the window still recording it had captured
|
|
5
|
+
// through index 120. The saved file said "Transfer integrity 100%": the gap report only looks for
|
|
6
|
+
// holes in [0, highest index received], so 47 chunks the Mac never heard of did not count.
|
|
7
|
+
//
|
|
8
|
+
// THE RULE. When the phone has said how far it captured (the Stop's `finalChunkIndex`, or a save's
|
|
9
|
+
// own `finalChunkIndex`, glasses 6.10.652), the expected count is that index plus one, and every
|
|
10
|
+
// index from the Mac's highest received chunk to it is missing. 74 of 121 reads 61.1%.
|
|
11
|
+
//
|
|
12
|
+
// And a phone's ORIGIN-WIPE RECOVERY save (`recovery: 'origin_wipe'`, glasses 6.10.652) is refused
|
|
13
|
+
// with 409 `chunks_outstanding` while the phone has said it captured chunks the Mac does not hold,
|
|
14
|
+
// unless it says it waited them out (`acceptMissing: true`). The phone checks the same thing before
|
|
15
|
+
// it asks; this closes the moment between its status read and its save (a Stop landing in between).
|
|
16
|
+
// A save without `recovery` is never refused here: every existing caller behaves as before.
|
|
17
|
+
|
|
18
|
+
import type { TranscriptGapReport } from '../routes/transcribe-stream.js'
|
|
19
|
+
|
|
20
|
+
/** The stop route's own bound on a final index. */
|
|
21
|
+
export const PHONE_FINAL_INDEX_LIMIT = 1_000_000
|
|
22
|
+
/** A phone-said index this far past the Mac's highest chunk is capped (about 30 hours of 6 s chunks),
|
|
23
|
+
* so one bad number cannot write a huge missing list into the sidecar. */
|
|
24
|
+
export const PHONE_FINAL_INDEX_MAX_AHEAD = 20_000
|
|
25
|
+
|
|
26
|
+
/** A final index as the stop route accepts it (a whole number, -1 or more), else null. */
|
|
27
|
+
export function parsePhoneFinalChunkIndex(value: unknown): number | null {
|
|
28
|
+
return typeof value === 'number' && Number.isInteger(value) && value >= -1 && value <= PHONE_FINAL_INDEX_LIMIT ? value : null
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
/** The highest index any phone window said it captured, or null when none said. */
|
|
32
|
+
export function phoneFinalChunkIndex(...said: Array<number | null | undefined>): number | null {
|
|
33
|
+
const known = said.filter((v): v is number => typeof v === 'number' && Number.isInteger(v) && v >= 0)
|
|
34
|
+
return known.length > 0 ? Math.max(...known) : null
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
/** The gap report with the phone's final index counted. Unchanged when the phone said nothing new. */
|
|
38
|
+
export function withPhoneFinalIndex(report: TranscriptGapReport | null, phoneFinal: number | null): TranscriptGapReport | null {
|
|
39
|
+
if (!report || phoneFinal === null || phoneFinal <= report.maxIndex) return report
|
|
40
|
+
const final = Math.min(phoneFinal, Math.max(report.maxIndex, 0) + PHONE_FINAL_INDEX_MAX_AHEAD)
|
|
41
|
+
const missingIndices = [...report.missingIndices]
|
|
42
|
+
for (let index = report.maxIndex + 1; index <= final; index++) missingIndices.push(index)
|
|
43
|
+
const expected = final + 1
|
|
44
|
+
return {
|
|
45
|
+
...report,
|
|
46
|
+
maxIndex: final,
|
|
47
|
+
expected,
|
|
48
|
+
missingIndices,
|
|
49
|
+
completeness: expected > 0 ? report.received / expected : 1,
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
/** Chunks in [0, phoneFinal] the Mac does not hold. */
|
|
54
|
+
export function chunksOutstanding(report: TranscriptGapReport | null, phoneFinal: number | null): number {
|
|
55
|
+
if (phoneFinal === null) return 0
|
|
56
|
+
const counted = withPhoneFinalIndex(report, phoneFinal)
|
|
57
|
+
if (!counted) return phoneFinal + 1
|
|
58
|
+
return counted.missingIndices.filter(index => index <= phoneFinal).length
|
|
59
|
+
}
|
package/server/routes/health.ts
CHANGED
|
@@ -75,6 +75,7 @@ import { liveCuesCapability } from '../lib/live-cues-capability.js'
|
|
|
75
75
|
import { liveCuesV2Advertised } from '../lib/live-cues-v2.js'
|
|
76
76
|
import { liveMeetingCapability } from '../lib/live-provisional.js'
|
|
77
77
|
import { CUE_KINDS } from '../../shared/live-cues-v2.js'
|
|
78
|
+
import { tailscaleNudgeHealthField } from '../lib/tailscale-nudge.js'
|
|
78
79
|
|
|
79
80
|
/** 6.86.0: `capabilities.liveCues.kinds`, the cue kinds this server can write (additive; an older server omits it, which
|
|
80
81
|
* is how a client knows to expect only reinforce and challenge). Which kinds a given meeting gets is in its arm answer. */
|
|
@@ -445,6 +446,8 @@ healthRouter.get('/health', async (_req, res) => {
|
|
|
445
446
|
...continueLiveHealthFields(),
|
|
446
447
|
// 6.62.0 /qa (W6): spawn isolation, posture and Cursor model read counters.
|
|
447
448
|
...continueReadHealthFields(),
|
|
449
|
+
// 6.89.0 Tailscale self-heal: switch, last nudge and its outcome, count this hour. No address.
|
|
450
|
+
...tailscaleNudgeHealthField(),
|
|
448
451
|
// 6.62.0: ONE providers object per engine. `act` (plan 3.11): what each provider can do,
|
|
449
452
|
// with a reason for every "no". `observe` (plan 1.10): what COS can see of its sessions.
|
|
450
453
|
providers: mergeProviderSections(
|
package/server/routes/meeting.ts
CHANGED
|
@@ -12,6 +12,7 @@ import { Router } from 'express'
|
|
|
12
12
|
import { emitDisplay } from '../lib/display-bus.js'
|
|
13
13
|
import { createStopRequestStore, isStopRequestRefusal, STOP_REQUEST_CLAIM_WINDOW_MS, type StopRequest, type StopRequestStore } from '../lib/meeting-stop-requests.js'
|
|
14
14
|
import { appendAutonomousSaveLog, getStopIntentStore, type StopIntent, type StopIntentStore } from '../lib/meeting-stop-intents.js'
|
|
15
|
+
import { chunksOutstanding, parsePhoneFinalChunkIndex, phoneFinalChunkIndex, withPhoneFinalIndex } from '../lib/transfer-integrity.js'
|
|
15
16
|
import {
|
|
16
17
|
AUTONOMOUS_FINALIZE_POLL_MS,
|
|
17
18
|
AUTONOMOUS_FINALIZE_SETTLE_MS,
|
|
@@ -1128,6 +1129,24 @@ export function createMeetingRouter(deps: MeetingRouteDependencies = {}): Router
|
|
|
1128
1129
|
if (MAC_CAPTURE_ID.test(sessionId)) {
|
|
1129
1130
|
requireMacRecordingForSave(sessionId, body?.expectedChunkCount)
|
|
1130
1131
|
}
|
|
1132
|
+
// 6.88.0 (lib/transfer-integrity.ts): how far a phone window said it captured, from its Stop or
|
|
1133
|
+
// from this save (glasses 6.10.652). The saved file's integrity counts against it.
|
|
1134
|
+
const phoneFinal = phoneFinalChunkIndex(stopIntents.get(sessionId)?.finalChunkIndex, parsePhoneFinalChunkIndex(body?.finalChunkIndex))
|
|
1135
|
+
// An origin-wipe recovery never finalizes a meeting a phone window still holds chunks for, unless
|
|
1136
|
+
// it says it waited them out. A save without `recovery` is never refused here.
|
|
1137
|
+
if (body?.recovery === 'origin_wipe' && body?.acceptMissing !== true) {
|
|
1138
|
+
const report = sessions.getIntegrity(sessionId)
|
|
1139
|
+
const outstanding = report ? chunksOutstanding(report, phoneFinal) : 0
|
|
1140
|
+
if (outstanding > 0) {
|
|
1141
|
+
respond(409, {
|
|
1142
|
+
error: 'A phone window captured chunks this Mac does not hold yet',
|
|
1143
|
+
reason: 'chunks_outstanding',
|
|
1144
|
+
missingCount: outstanding,
|
|
1145
|
+
finalChunkIndex: phoneFinal,
|
|
1146
|
+
})
|
|
1147
|
+
return null
|
|
1148
|
+
}
|
|
1149
|
+
}
|
|
1131
1150
|
if (savingSessions.has(sessionId)) {
|
|
1132
1151
|
respond(409, { error: 'Meeting save already in progress', reason: 'save_in_progress' })
|
|
1133
1152
|
return null
|
|
@@ -1166,7 +1185,7 @@ export function createMeetingRouter(deps: MeetingRouteDependencies = {}): Router
|
|
|
1166
1185
|
const durationMs = durationFromTimeline > 0
|
|
1167
1186
|
? durationFromTimeline
|
|
1168
1187
|
: Math.max(0, Date.now() - startTime)
|
|
1169
|
-
const integrity = sessions.getIntegrity(sessionId)
|
|
1188
|
+
const integrity = withPhoneFinalIndex(sessions.getIntegrity(sessionId), phoneFinal)
|
|
1170
1189
|
const needsOperations = cosOpsPipelineConfigured() && !macRecording
|
|
1171
1190
|
// Standalone saves need a finalization pass too, for summary enrichment.
|
|
1172
1191
|
// Before 6.37 a standalone save with no audio nulled the job below and
|