@celilo/cli 5.1.1 → 5.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CELILO_SUBSYSTEMS.md +1 -0
- package/package.json +2 -2
- package/src/api/serve.ts +55 -2
- package/src/api/sessions-reap-cost.test.ts +218 -0
- package/src/api/sessions-retention.test.ts +120 -0
- package/src/api/sessions.ts +226 -2
- package/src/cli/commands/api.ts +22 -0
- package/src/cli/commands/system-migrate.ts +6 -0
- package/src/cli/completion.ts +1 -1
- package/src/cli/index.ts +6 -0
- package/src/policy/module-business-baseline.ts +0 -6
- package/src/services/module-subscriptions.test.ts +23 -0
- package/src/services/module-subscriptions.ts +6 -0
- package/src/templates/copy-role-files.test.ts +20 -2
- package/src/templates/generator.ts +56 -35
package/CELILO_SUBSYSTEMS.md
CHANGED
|
@@ -375,6 +375,7 @@ Run any celilo command on celilo-mgr over SSH instead of screen-scraping `ssh <h
|
|
|
375
375
|
- **Client** — `packages/core/src/remote-client.ts` (`@celilo/core`) — `resolveRemote` (`--remote <dest>` / `CELILO_REMOTE`), `runRemoteClient` (`ssh -T`, renders progress via the local ProgressDisplay, answers interviews via the `@celilo/cli-display` prompts). Refuses to prompt on a non-TTY stdin, replying `unanswerable` rather than submitting a default as if a human had chosen it.
|
|
376
376
|
- **Access control** — `apps/celilo/src/services/api-access.ts` — `grantPrincipal`, `isAuthorized` (deny-by-default, `command:subcommand` grants), `renderAuthorizedKeys`. Table: `api_principals` (`apps/celilo/src/db/schema.ts`). CLI: `apps/celilo/src/cli/commands/api.ts` (`api grant|list|revoke|authorized-keys|key new`).
|
|
377
377
|
- **Principal enrolment for a module (`control_plane_api`)** — `apps/celilo/src/services/api-principal-enrolment.ts` — `enrolControlPlanePrincipal`, `revokeControlPlanePrincipal`, and `buildControlPlaneApi`, the method table a consuming module's hooks receive. The consumer generates an ed25519 pair on its own system and presents the public half; nothing here accepts a private key. Grants are DERIVED from `readOnlyGrants(COMMANDS)` and are not a parameter, so a caller cannot ask for more, and a write verb is never granted however it is named. **Framework-granted, so no module provides it** — enrolment writes celilo's own `api_principals` row and a module script may import nothing but `@celilo/capabilities`, which rules out celilo-mgmt as much as anyone else (`web-ui-console` D7b). Injected by `capability-loader.ts` ONLY for a module whose stored manifest declares it under `requires`/`optional`, unlike every other capability the loader hands out, and scoped to that module: a caller may not name a neighbour's principal. Contract: `packages/capabilities/src/control-plane-api.ts`.
|
|
378
|
+
- **Parked-session registry** — `apps/celilo/src/api/sessions.ts` — the on-disk handoff that lets a command parked on an unanswerable question outlive the ssh session that started it (`api-sessions/<id>/session.json` + `output.ndjson`; each connection is its own process, so the registry cannot be in memory). `SessionWriter` owns a live session; `reapExpiredSessions` abandons parked sessions past their 30-minute TTL by ANSWERING the bus query `{abandoned}` rather than killing the child, which is what releases the module-operation lock on the way out. `pruneRetiredSessions` applies `SESSION_RETENTION_MS` (24h) to RETIRED records only — `running` and `parked` are left alone at any age, because a long command is not garbage and an old parked record can still be live. **The reap reads an INDEX of parked sessions (`api-parked/<id>`, one empty marker per parked session, maintained inside `write()` so every state transition already goes through it), NOT every record.** It is on the connection path on purpose — it is the only reaper that runs on a host with no dispatcher — and that is only defensible while it is O(parked): scanning every record there is what made connection cost track session count (celilo#1440's 94,706 records and 11.4s in front of a 240ms command; the shape itself is celilo#1454). The index is a cache, never the truth — `listParkedSessions` verifies each marker against its record and drops the ones that disagree, and an absent index is rebuilt from one scan. Retention pruning is O(every record) and nothing waits on it, so it belongs to `runApiSessionSweep` on the hourly `celilo-api-session-sweep` subscriber (`timer.tick.1h` → `celilo api sweep`, armed by `ensureApiSessionSweepSubscriber` from module registration and `celilo system migrate`, exactly like the backup and operations sweeps); `api-serve` keeps a stamp-gated fallback prune that runs AFTER a command is served, so a dispatcher-less host stays bounded without paying on connection setup. Gates: `sessions-reap-cost.test.ts` (the cost ratio across two magnitudes of backlog is what fails if O(records) work returns to the path), `sessions-retention.test.ts`, `sessions.test.ts`.
|
|
378
379
|
- **Mid-run interview bridge (`kind:daemon` responder)** — `apps/celilo/src/services/remote-responder.ts` — `startRemoteResponder` bridges bus `interview.required.*` ↔ wire.
|
|
379
380
|
- **Server provisioning** — the `celilo-bootstrap` deb (`packaging/celilo-bootstrap/scripts/postinst`) creates the non-root `celilo-api` landing account + sshd; membership in the `celilo` group + `/etc/sudoers.d/celilo` (`!use_pty`) gives api-serve DB access via the wrapper's sudo-drop.
|
|
380
381
|
- **Self-upgrade (apt)** — `celilo apt-upgrade` (`apps/celilo/src/cli/commands/apt-upgrade.ts`) upgrades the deb-installed `celilo`/`celilo-bootstrap` packages (`apt-get update` → `--only-upgrade install`) then spawns a fresh `celilo system migrate` (ISS-0100), then `celilo events restart-daemon` so the dispatcher actually runs the code just installed — a failure there fails the whole command and names which steps DID complete, because "upgraded" while the dispatcher serves stale code is the silent state celilo#604 documents. It's the RW target behind the MCP's registry-derived `celilo_apt_upgrade` tool; the celilo user's two apt invocations are scoped-sudo'd by `/etc/sudoers.d/celilo-apt-upgrade`, shipped by `celilo-bootstrap`. **This upgrades celilo ITSELF — not the modules it manages. For those, see Module auto-upgrade below; the two are routinely confused.**
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@celilo/cli",
|
|
3
|
-
"version": "5.
|
|
3
|
+
"version": "5.2.1",
|
|
4
4
|
"description": "Celilo — home lab orchestration CLI",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -61,7 +61,7 @@
|
|
|
61
61
|
"@aws-sdk/lib-storage": "^3.1101.0",
|
|
62
62
|
"@celilo/capabilities": "^6.1.0",
|
|
63
63
|
"@celilo/cli-display": "^0.2.0",
|
|
64
|
-
"@celilo/core": "^0.
|
|
64
|
+
"@celilo/core": "^0.15.0",
|
|
65
65
|
"@celilo/event-bus": "^0.7.0",
|
|
66
66
|
"ajv": "^8.18.0",
|
|
67
67
|
"drizzle-orm": "^0.36.4",
|
package/src/api/serve.ts
CHANGED
|
@@ -17,6 +17,8 @@
|
|
|
17
17
|
* answers by event id; a later `attach` collects the outcome.
|
|
18
18
|
*/
|
|
19
19
|
|
|
20
|
+
import { existsSync, statSync, writeFileSync } from 'node:fs';
|
|
21
|
+
import { join } from 'node:path';
|
|
20
22
|
import { createInterface } from 'node:readline';
|
|
21
23
|
import {
|
|
22
24
|
API_PROTOCOL_VERSION,
|
|
@@ -26,7 +28,7 @@ import {
|
|
|
26
28
|
translateOutputLine,
|
|
27
29
|
} from '@celilo/core';
|
|
28
30
|
import { parseArguments } from '../cli/parser';
|
|
29
|
-
import { getEventBusPath } from '../config/paths';
|
|
31
|
+
import { getDataDir, getEventBusPath } from '../config/paths';
|
|
30
32
|
import { getDb } from '../db/client';
|
|
31
33
|
import { isAuthorized } from '../services/api-access';
|
|
32
34
|
import { EVENT_TYPES } from '../services/bus-interview';
|
|
@@ -36,6 +38,7 @@ import {
|
|
|
36
38
|
SessionWriter,
|
|
37
39
|
abandonSession,
|
|
38
40
|
expiryReason,
|
|
41
|
+
pruneRetiredSessions,
|
|
39
42
|
readSession,
|
|
40
43
|
reapExpiredSessions,
|
|
41
44
|
replayOutput,
|
|
@@ -234,12 +237,61 @@ function handleCancel(principal: string, sessionId: string): void {
|
|
|
234
237
|
send(resultMessage(true, 0));
|
|
235
238
|
}
|
|
236
239
|
|
|
240
|
+
/**
|
|
241
|
+
* How often a connection will pay for the retention prune when nothing else is
|
|
242
|
+
* pruning. Far finer than the 24h retention it enforces.
|
|
243
|
+
*/
|
|
244
|
+
const RETENTION_FALLBACK_INTERVAL_MS = 60 * 60 * 1000;
|
|
245
|
+
|
|
246
|
+
function retentionStampPath(): string {
|
|
247
|
+
return join(getDataDir(), 'api-sessions-pruned-at');
|
|
248
|
+
}
|
|
249
|
+
|
|
250
|
+
/**
|
|
251
|
+
* The daemonless safety net for retention.
|
|
252
|
+
*
|
|
253
|
+
* The hourly `celilo-api-session-sweep` subscriber owns pruning now, but a host
|
|
254
|
+
* with no dispatcher running has no subscribers at all — and unbounded retention
|
|
255
|
+
* is precisely what celilo#1440 was: 94,706 records, 1.2 GB, every connection
|
|
256
|
+
* paying for all of it. Moving the prune to a timer without this would re-open
|
|
257
|
+
* that on exactly the hosts least likely to notice.
|
|
258
|
+
*
|
|
259
|
+
* Two things keep it off the critical path, and both matter:
|
|
260
|
+
*
|
|
261
|
+
* 1. It runs AFTER the command's result has been sent, not before the first
|
|
262
|
+
* command is read. The caller is already served; this is teardown.
|
|
263
|
+
* 2. A stamp file gates it to once an hour, so the cost is not per-connection.
|
|
264
|
+
* One `statSync` is what a connection pays in the common case.
|
|
265
|
+
*
|
|
266
|
+
* The stamp is written FIRST, so a prune that throws still backs off rather than
|
|
267
|
+
* having every subsequent connection retry a sweep that cannot work.
|
|
268
|
+
*/
|
|
269
|
+
function sweepRetentionIfDue(): void {
|
|
270
|
+
try {
|
|
271
|
+
const stamp = retentionStampPath();
|
|
272
|
+
const lastRun = existsSync(stamp) ? statSync(stamp).mtimeMs : 0;
|
|
273
|
+
if (Date.now() - lastRun < RETENTION_FALLBACK_INTERVAL_MS) return;
|
|
274
|
+
writeFileSync(stamp, '');
|
|
275
|
+
pruneRetiredSessions();
|
|
276
|
+
} catch {
|
|
277
|
+
// Housekeeping. Nothing about this is worth failing a served command over,
|
|
278
|
+
// and the hourly subscriber is the primary path regardless.
|
|
279
|
+
}
|
|
280
|
+
}
|
|
281
|
+
|
|
237
282
|
export async function apiServeMode(principal: string): Promise<void> {
|
|
238
283
|
send({ type: 'ready', protocolVersion: API_PROTOCOL_VERSION });
|
|
239
284
|
|
|
240
285
|
const busDbPath = getEventBusPath();
|
|
241
286
|
// A session whose owning process died (reboot, OOM) would otherwise stay
|
|
242
|
-
// parked forever, holding whatever its command holds.
|
|
287
|
+
// parked forever, holding whatever its command holds. This is the only reaper
|
|
288
|
+
// that runs on a host with no dispatcher, which is why it is still here and
|
|
289
|
+
// not only on the hourly sweep.
|
|
290
|
+
//
|
|
291
|
+
// It reads the parked index rather than every session record, so its cost
|
|
292
|
+
// tracks parked sessions (usually none) and not how many have ever run — the
|
|
293
|
+
// property celilo#1454 is about. Nothing O(records) may go back in front of
|
|
294
|
+
// this line.
|
|
243
295
|
reapExpiredSessions({ busDbPath });
|
|
244
296
|
|
|
245
297
|
// When the ssh client goes away, sshd hangs up its forced command. Ignoring
|
|
@@ -405,5 +457,6 @@ export async function apiServeMode(principal: string): Promise<void> {
|
|
|
405
457
|
// command finishes or the reaper abandons it.
|
|
406
458
|
live.clientAttached = false;
|
|
407
459
|
if (commandRunning) await commandRunning;
|
|
460
|
+
sweepRetentionIfDue();
|
|
408
461
|
process.exit(0);
|
|
409
462
|
}
|
|
@@ -0,0 +1,218 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Opening an API connection must not cost one read per session that has ever run.
|
|
3
|
+
*
|
|
4
|
+
* `api-serve` calls `reapExpiredSessions` after sending `ready` and before it
|
|
5
|
+
* will read a command, so whatever that call scans is charged to every caller of
|
|
6
|
+
* the remote API — the web console twice per panel. It used to scan every
|
|
7
|
+
* record: celilo#1440 was the extreme form (94,706 records, 11.4s before a
|
|
8
|
+
* 240ms command), and bounding retention to 24h fixed the magnitude while
|
|
9
|
+
* leaving the shape, which is celilo#1454.
|
|
10
|
+
*
|
|
11
|
+
* The reap now reads the parked INDEX instead, so its working set is the number
|
|
12
|
+
* of PARKED sessions (usually none) rather than the number of records.
|
|
13
|
+
*
|
|
14
|
+
* Two gates here, deliberately different in kind:
|
|
15
|
+
*
|
|
16
|
+
* MECHANISM — the index holds one entry per parked session and nothing else,
|
|
17
|
+
* so what the reap opens is bounded by parked count. Deterministic.
|
|
18
|
+
*
|
|
19
|
+
* COST — the wall-clock of the reap across two magnitudes of backlog, which is
|
|
20
|
+
* what the issue asks for and what fails if any O(records) work returns to
|
|
21
|
+
* this path, by whatever route.
|
|
22
|
+
*
|
|
23
|
+
* WATCHED RED, and the result is worth recording because it says which of these
|
|
24
|
+
* tests is actually the gate. Restoring `listSessions()` as the reap's source —
|
|
25
|
+
* the pre-fix shape — fails the COST test at a ratio of 114.4 against a
|
|
26
|
+
* threshold of 10, and fails the daemonless test. It does NOT fail the mechanism
|
|
27
|
+
* test, which keeps passing: a full scan still finds the one parked session, so
|
|
28
|
+
* asserting on the reap's RESULT cannot see how much it read to get there. The
|
|
29
|
+
* mechanism test documents the index and guards its self-healing; the ratio is
|
|
30
|
+
* the only thing here that fails when the cost comes back.
|
|
31
|
+
*
|
|
32
|
+
* Restoring the `pruneRetiredSessions` call inside `reapExpiredSessions` fails
|
|
33
|
+
* 'the reap does not prune' at 0 records remaining against 20.
|
|
34
|
+
*/
|
|
35
|
+
|
|
36
|
+
import { afterEach, beforeEach, expect, test } from 'bun:test';
|
|
37
|
+
import { mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs';
|
|
38
|
+
import { tmpdir } from 'node:os';
|
|
39
|
+
import { join } from 'node:path';
|
|
40
|
+
import {
|
|
41
|
+
SESSION_RETENTION_MS,
|
|
42
|
+
type SessionState,
|
|
43
|
+
SessionWriter,
|
|
44
|
+
getParkedDir,
|
|
45
|
+
getSessionsDir,
|
|
46
|
+
listParkedSessions,
|
|
47
|
+
reapExpiredSessions,
|
|
48
|
+
runApiSessionSweep,
|
|
49
|
+
} from './sessions';
|
|
50
|
+
|
|
51
|
+
const NOW = 1_800_000_000_000;
|
|
52
|
+
const OLD = NOW - SESSION_RETENTION_MS - 60_000;
|
|
53
|
+
|
|
54
|
+
let dataDir: string;
|
|
55
|
+
let busDbPath: string;
|
|
56
|
+
let previous: string | undefined;
|
|
57
|
+
|
|
58
|
+
/**
|
|
59
|
+
* Write a record the way a retired session leaves one behind: on disk, with no
|
|
60
|
+
* process attached. `expiresAt` in the past is what makes a parked one reapable.
|
|
61
|
+
*/
|
|
62
|
+
function seed(sessionId: string, state: SessionState, startedAt = OLD): void {
|
|
63
|
+
const dir = join(getSessionsDir(), sessionId);
|
|
64
|
+
mkdirSync(dir, { recursive: true });
|
|
65
|
+
writeFileSync(
|
|
66
|
+
join(dir, 'session.json'),
|
|
67
|
+
JSON.stringify({
|
|
68
|
+
sessionId,
|
|
69
|
+
principal: 'celilo-web-console',
|
|
70
|
+
argv: ['module', 'list'],
|
|
71
|
+
startedAt,
|
|
72
|
+
expiresAt: startedAt + 30 * 60 * 1000,
|
|
73
|
+
state,
|
|
74
|
+
parkedEventId: null,
|
|
75
|
+
parkedEventType: null,
|
|
76
|
+
question: null,
|
|
77
|
+
questionKey: null,
|
|
78
|
+
}),
|
|
79
|
+
);
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
/** A parked record AND its index entry — what `write()` produces together. */
|
|
83
|
+
function seedParked(sessionId: string, startedAt = OLD): void {
|
|
84
|
+
seed(sessionId, 'parked', startedAt);
|
|
85
|
+
mkdirSync(getParkedDir(), { recursive: true });
|
|
86
|
+
writeFileSync(join(getParkedDir(), sessionId), '');
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
/** Wall-clock of the exact call `api-serve` makes before reading a command. */
|
|
90
|
+
function timeReap(): number {
|
|
91
|
+
const started = performance.now();
|
|
92
|
+
reapExpiredSessions({ busDbPath, now: NOW });
|
|
93
|
+
return performance.now() - started;
|
|
94
|
+
}
|
|
95
|
+
|
|
96
|
+
beforeEach(() => {
|
|
97
|
+
previous = process.env.CELILO_DATA_DIR;
|
|
98
|
+
dataDir = mkdtempSync(join(tmpdir(), 'celilo-reap-cost-'));
|
|
99
|
+
process.env.CELILO_DATA_DIR = dataDir;
|
|
100
|
+
busDbPath = join(dataDir, 'bus.db');
|
|
101
|
+
mkdirSync(getSessionsDir(), { recursive: true });
|
|
102
|
+
mkdirSync(getParkedDir(), { recursive: true });
|
|
103
|
+
});
|
|
104
|
+
|
|
105
|
+
afterEach(() => {
|
|
106
|
+
if (previous === undefined) delete process.env.CELILO_DATA_DIR;
|
|
107
|
+
else process.env.CELILO_DATA_DIR = previous;
|
|
108
|
+
rmSync(dataDir, { recursive: true, force: true });
|
|
109
|
+
});
|
|
110
|
+
|
|
111
|
+
test('the reap opens the parked index, not every record', () => {
|
|
112
|
+
for (let i = 0; i < 4000; i++) seed(`retired-${i}`, 'finished');
|
|
113
|
+
seedParked('the-only-parked-one');
|
|
114
|
+
|
|
115
|
+
// The premise, stated so a reader can see the two numbers that matter.
|
|
116
|
+
expect(readdirSync(getSessionsDir())).toHaveLength(4001);
|
|
117
|
+
expect(readdirSync(getParkedDir())).toHaveLength(1);
|
|
118
|
+
|
|
119
|
+
// What the reap can even consider is the index, so 1 — not 4001.
|
|
120
|
+
expect(listParkedSessions().map((s) => s.sessionId)).toEqual(['the-only-parked-one']);
|
|
121
|
+
expect(reapExpiredSessions({ busDbPath, now: NOW }).map((r) => r.sessionId)).toEqual([
|
|
122
|
+
'the-only-parked-one',
|
|
123
|
+
]);
|
|
124
|
+
});
|
|
125
|
+
|
|
126
|
+
test('the reap does not prune — retention is the hourly sweep, not the caller', () => {
|
|
127
|
+
for (let i = 0; i < 20; i++) seed(`retired-${i}`, 'finished');
|
|
128
|
+
|
|
129
|
+
reapExpiredSessions({ busDbPath, now: NOW });
|
|
130
|
+
// Still all there: nothing O(records) ran in front of the caller's command.
|
|
131
|
+
expect(readdirSync(getSessionsDir())).toHaveLength(20);
|
|
132
|
+
|
|
133
|
+
// The sweep is what reclaims them, and it is the same records.
|
|
134
|
+
expect(runApiSessionSweep({ busDbPath, now: NOW })).toEqual({ reaped: 0, pruned: 20 });
|
|
135
|
+
expect(readdirSync(getSessionsDir())).toEqual([]);
|
|
136
|
+
});
|
|
137
|
+
|
|
138
|
+
test('connection cost does not track the number of session records', () => {
|
|
139
|
+
const SMALL = 50;
|
|
140
|
+
const LARGE = 5000; // 100x
|
|
141
|
+
|
|
142
|
+
for (let i = 0; i < SMALL; i++) seed(`small-${i}`, 'finished');
|
|
143
|
+
timeReap(); // discard: first call pays for cold caches, not for N
|
|
144
|
+
const small = Math.min(timeReap(), timeReap(), timeReap());
|
|
145
|
+
|
|
146
|
+
for (let i = SMALL; i < LARGE; i++) seed(`large-${i}`, 'finished');
|
|
147
|
+
expect(readdirSync(getSessionsDir())).toHaveLength(LARGE);
|
|
148
|
+
const large = Math.min(timeReap(), timeReap(), timeReap());
|
|
149
|
+
|
|
150
|
+
// A floor, because at these speeds the measurement is mostly timer noise and a
|
|
151
|
+
// ratio against ~0ms means nothing.
|
|
152
|
+
const ratio = large / Math.max(small, 0.05);
|
|
153
|
+
|
|
154
|
+
// O(records) would put this near 100. Ten is a wide margin that still cannot
|
|
155
|
+
// be reached by anything proportional to a 100x change in N.
|
|
156
|
+
expect(ratio).toBeLessThan(10);
|
|
157
|
+
});
|
|
158
|
+
|
|
159
|
+
test('a parked session past its TTL is reaped with no dispatcher running', () => {
|
|
160
|
+
// The property the inline call buys and the whole reason it stays on the
|
|
161
|
+
// connection path: nothing else reaps on a host with no bus dispatcher. There
|
|
162
|
+
// is deliberately no subscriber, no responder and no daemon in this test.
|
|
163
|
+
seedParked('owner-died-in-a-reboot');
|
|
164
|
+
for (let i = 0; i < 500; i++) seed(`retired-${i}`, 'finished');
|
|
165
|
+
|
|
166
|
+
const reaped = reapExpiredSessions({ busDbPath, now: NOW });
|
|
167
|
+
|
|
168
|
+
expect(reaped.map((r) => r.sessionId)).toEqual(['owner-died-in-a-reboot']);
|
|
169
|
+
// Abandoned, and retained as history rather than deleted.
|
|
170
|
+
const record = JSON.parse(
|
|
171
|
+
readFileSync(join(getSessionsDir(), 'owner-died-in-a-reboot', 'session.json'), 'utf-8'),
|
|
172
|
+
);
|
|
173
|
+
expect(record.state).toBe('abandoned');
|
|
174
|
+
// And it leaves the index, so the next connection does not reconsider it.
|
|
175
|
+
expect(readdirSync(getParkedDir())).toEqual([]);
|
|
176
|
+
});
|
|
177
|
+
|
|
178
|
+
test('a marker whose session is gone is dropped, not paid for forever', () => {
|
|
179
|
+
// A prune removes the directory and the marker outlives it. The index is a
|
|
180
|
+
// cache, so it self-heals rather than making every future reap read a hole.
|
|
181
|
+
mkdirSync(getParkedDir(), { recursive: true });
|
|
182
|
+
writeFileSync(join(getParkedDir(), 'pruned-out-from-under-us'), '');
|
|
183
|
+
|
|
184
|
+
expect(listParkedSessions()).toEqual([]);
|
|
185
|
+
expect(readdirSync(getParkedDir())).toEqual([]);
|
|
186
|
+
});
|
|
187
|
+
|
|
188
|
+
test('an index that does not exist yet is built from one scan, then reused', () => {
|
|
189
|
+
// The upgrade path: a host coming into this code may already hold a parked
|
|
190
|
+
// session, and it must still be reaped. Pay for one full scan, once.
|
|
191
|
+
rmSync(getParkedDir(), { recursive: true, force: true });
|
|
192
|
+
seed('parked-before-the-upgrade', 'parked');
|
|
193
|
+
for (let i = 0; i < 100; i++) seed(`retired-${i}`, 'finished');
|
|
194
|
+
|
|
195
|
+
expect(listParkedSessions().map((s) => s.sessionId)).toEqual(['parked-before-the-upgrade']);
|
|
196
|
+
// The index now exists and holds exactly that session, so the scan was once.
|
|
197
|
+
expect(readdirSync(getParkedDir())).toEqual(['parked-before-the-upgrade']);
|
|
198
|
+
});
|
|
199
|
+
|
|
200
|
+
test('the index follows the real lifecycle, not a separate bookkeeping call', () => {
|
|
201
|
+
// The index is maintained inside `write()`, which every state transition
|
|
202
|
+
// already goes through. That is what makes it hard to drift: there is no
|
|
203
|
+
// second call for a future edit to forget.
|
|
204
|
+
const session = SessionWriter.create({ principal: 'tester', argv: ['module', 'list'] });
|
|
205
|
+
expect(readdirSync(getParkedDir())).toEqual([]);
|
|
206
|
+
|
|
207
|
+
session.park({ eventId: '1', eventType: 'interview.required.x.y', question: 'well?' });
|
|
208
|
+
expect(readdirSync(getParkedDir())).toEqual([session.id]);
|
|
209
|
+
|
|
210
|
+
session.unpark();
|
|
211
|
+
expect(readdirSync(getParkedDir())).toEqual([]);
|
|
212
|
+
|
|
213
|
+
session.park({ eventId: '2', eventType: 'interview.required.x.y', question: 'again?' });
|
|
214
|
+
expect(readdirSync(getParkedDir())).toEqual([session.id]);
|
|
215
|
+
|
|
216
|
+
session.finish('finished');
|
|
217
|
+
expect(readdirSync(getParkedDir())).toEqual([]);
|
|
218
|
+
});
|
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Session records are retained as history on purpose. The bug was that the
|
|
3
|
+
* retention had no bound.
|
|
4
|
+
*
|
|
5
|
+
* Nothing in `sessions.ts` ever deleted a directory, so the set grew by one per
|
|
6
|
+
* API connection and never shrank — and `reapExpiredSessions` reads EVERY
|
|
7
|
+
* record, on the path `api-serve` takes before it will accept a command. So the
|
|
8
|
+
* cost of every session that had ever run was charged to every caller, on every
|
|
9
|
+
* connection. Measured on celilo-mgr 2026-09-27: 94,706 directories, 1.2 GB,
|
|
10
|
+
* and 11.4s between `ready` and the first command being read, for a command
|
|
11
|
+
* that then took 240ms. celilo#1440.
|
|
12
|
+
*
|
|
13
|
+
* WATCHED RED: with the `running`/`parked` guard removed, the live-session tests
|
|
14
|
+
* fail.
|
|
15
|
+
*
|
|
16
|
+
* The prune no longer runs from `reapExpiredSessions` — that coupling was the
|
|
17
|
+
* other half of the same defect, since it put O(every record) in front of every
|
|
18
|
+
* caller's first command. `runApiSessionSweep` owns it on an hourly subscriber
|
|
19
|
+
* now, and `sessions-reap-cost.test.ts` is what holds that line. celilo#1454.
|
|
20
|
+
*/
|
|
21
|
+
|
|
22
|
+
import { afterEach, beforeEach, expect, test } from 'bun:test';
|
|
23
|
+
import { mkdirSync, mkdtempSync, readdirSync, rmSync, writeFileSync } from 'node:fs';
|
|
24
|
+
import { tmpdir } from 'node:os';
|
|
25
|
+
import { join } from 'node:path';
|
|
26
|
+
import {
|
|
27
|
+
SESSION_RETENTION_MS,
|
|
28
|
+
type SessionState,
|
|
29
|
+
listSessions,
|
|
30
|
+
pruneRetiredSessions,
|
|
31
|
+
} from './sessions';
|
|
32
|
+
|
|
33
|
+
const NOW = 1_800_000_000_000;
|
|
34
|
+
const OLD = NOW - SESSION_RETENTION_MS - 60_000; // comfortably past the bound
|
|
35
|
+
const RECENT = NOW - 60_000;
|
|
36
|
+
|
|
37
|
+
let dataDir: string;
|
|
38
|
+
let previous: string | undefined;
|
|
39
|
+
|
|
40
|
+
function seed(sessionId: string, state: SessionState, startedAt: number): void {
|
|
41
|
+
const dir = join(dataDir, 'api-sessions', sessionId);
|
|
42
|
+
mkdirSync(dir, { recursive: true });
|
|
43
|
+
writeFileSync(
|
|
44
|
+
join(dir, 'session.json'),
|
|
45
|
+
JSON.stringify({
|
|
46
|
+
sessionId,
|
|
47
|
+
principal: 'celilo-web-console',
|
|
48
|
+
argv: ['api', 'list'],
|
|
49
|
+
startedAt,
|
|
50
|
+
expiresAt: startedAt + 30 * 60 * 1000,
|
|
51
|
+
state,
|
|
52
|
+
parkedEventId: null,
|
|
53
|
+
parkedEventType: null,
|
|
54
|
+
question: null,
|
|
55
|
+
questionKey: null,
|
|
56
|
+
}),
|
|
57
|
+
);
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
const remaining = () => readdirSync(join(dataDir, 'api-sessions')).sort();
|
|
61
|
+
|
|
62
|
+
beforeEach(() => {
|
|
63
|
+
previous = process.env.CELILO_DATA_DIR;
|
|
64
|
+
dataDir = mkdtempSync(join(tmpdir(), 'celilo-sessions-'));
|
|
65
|
+
process.env.CELILO_DATA_DIR = dataDir;
|
|
66
|
+
mkdirSync(join(dataDir, 'api-sessions'), { recursive: true });
|
|
67
|
+
});
|
|
68
|
+
|
|
69
|
+
afterEach(() => {
|
|
70
|
+
if (previous === undefined) delete process.env.CELILO_DATA_DIR;
|
|
71
|
+
else process.env.CELILO_DATA_DIR = previous;
|
|
72
|
+
rmSync(dataDir, { recursive: true, force: true });
|
|
73
|
+
});
|
|
74
|
+
|
|
75
|
+
test('removes retired sessions past the retention bound', () => {
|
|
76
|
+
seed('old-finished', 'finished', OLD);
|
|
77
|
+
seed('old-abandoned', 'abandoned', OLD);
|
|
78
|
+
|
|
79
|
+
expect(pruneRetiredSessions({ now: NOW })).toBe(2);
|
|
80
|
+
expect(remaining()).toEqual([]);
|
|
81
|
+
});
|
|
82
|
+
|
|
83
|
+
test('keeps a retired session that is still inside the bound', () => {
|
|
84
|
+
seed('just-finished', 'finished', RECENT);
|
|
85
|
+
|
|
86
|
+
expect(pruneRetiredSessions({ now: NOW })).toBe(0);
|
|
87
|
+
expect(remaining()).toEqual(['just-finished']);
|
|
88
|
+
});
|
|
89
|
+
|
|
90
|
+
test('never removes a live session, however old it is', () => {
|
|
91
|
+
// A long command is not garbage, and an old `parked` record can still be
|
|
92
|
+
// live — abandonSession declines to abandon one whose question was answered.
|
|
93
|
+
seed('ancient-running', 'running', OLD);
|
|
94
|
+
seed('ancient-parked', 'parked', OLD);
|
|
95
|
+
|
|
96
|
+
expect(pruneRetiredSessions({ now: NOW })).toBe(0);
|
|
97
|
+
expect(remaining()).toEqual(['ancient-parked', 'ancient-running']);
|
|
98
|
+
});
|
|
99
|
+
|
|
100
|
+
test('removes a corrupt directory, which listSessions cannot even see', () => {
|
|
101
|
+
const dir = join(dataDir, 'api-sessions', 'corrupt');
|
|
102
|
+
mkdirSync(dir, { recursive: true });
|
|
103
|
+
writeFileSync(join(dir, 'session.json'), '{ not json');
|
|
104
|
+
|
|
105
|
+
// The premise: it is invisible to every other function here, so nothing else
|
|
106
|
+
// could ever reclaim it — which is why it is pruned on its directory mtime.
|
|
107
|
+
expect(listSessions().map((s) => s.sessionId)).not.toContain('corrupt');
|
|
108
|
+
expect(pruneRetiredSessions({ now: NOW, retentionMs: -1 })).toBe(1);
|
|
109
|
+
expect(remaining()).toEqual([]);
|
|
110
|
+
});
|
|
111
|
+
|
|
112
|
+
test('a backlog collapses to just the live sessions', () => {
|
|
113
|
+
for (let i = 0; i < 500; i++) seed(`retired-${i}`, 'finished', OLD);
|
|
114
|
+
seed('live', 'running', OLD);
|
|
115
|
+
|
|
116
|
+
expect(pruneRetiredSessions({ now: NOW })).toBe(500);
|
|
117
|
+
expect(remaining()).toEqual(['live']);
|
|
118
|
+
// The point of the bound: the next caller's scan is one record, not 501.
|
|
119
|
+
expect(listSessions()).toHaveLength(1);
|
|
120
|
+
});
|
package/src/api/sessions.ts
CHANGED
|
@@ -25,6 +25,8 @@ import {
|
|
|
25
25
|
mkdirSync,
|
|
26
26
|
readFileSync,
|
|
27
27
|
readdirSync,
|
|
28
|
+
rmSync,
|
|
29
|
+
statSync,
|
|
28
30
|
writeFileSync,
|
|
29
31
|
} from 'node:fs';
|
|
30
32
|
import { join } from 'node:path';
|
|
@@ -32,6 +34,7 @@ import type { ServerMessage } from '@celilo/core';
|
|
|
32
34
|
import { defineEvents, openBus } from '@celilo/event-bus';
|
|
33
35
|
import { z } from 'zod';
|
|
34
36
|
import { getDataDir } from '../config/paths';
|
|
37
|
+
import type { SubscriberRegistrar } from '../services/module-operations';
|
|
35
38
|
|
|
36
39
|
const NO_SCHEMAS = defineEvents({});
|
|
37
40
|
|
|
@@ -82,7 +85,21 @@ export function readSession(sessionId: string): SessionRecord | null {
|
|
|
82
85
|
if (!existsSync(path)) return null;
|
|
83
86
|
// Untrusted only in the sense of "written by another process" — validate it
|
|
84
87
|
// rather than trusting the shape (Rule 3.7).
|
|
85
|
-
|
|
88
|
+
//
|
|
89
|
+
// The SHAPE was validated and the SYNTAX was not, so a record that is not
|
|
90
|
+
// JSON at all threw straight out of here. `listSessions` maps this over every
|
|
91
|
+
// directory, and `api-serve` calls that before it will accept a command, so
|
|
92
|
+
// one truncated write — a crash mid-write, a full disk — took down every API
|
|
93
|
+
// connection on the box, not just the session it belonged to. Unparseable and
|
|
94
|
+
// malformed are the same event to every caller here, and both already mean
|
|
95
|
+
// null.
|
|
96
|
+
let raw: unknown;
|
|
97
|
+
try {
|
|
98
|
+
raw = JSON.parse(readFileSync(path, 'utf-8'));
|
|
99
|
+
} catch {
|
|
100
|
+
return null;
|
|
101
|
+
}
|
|
102
|
+
const parsed = SessionRecordSchema.safeParse(raw);
|
|
86
103
|
return parsed.success ? parsed.data : null;
|
|
87
104
|
}
|
|
88
105
|
|
|
@@ -103,6 +120,94 @@ export function sessionParkedOn(eventId: string): SessionRecord | null {
|
|
|
103
120
|
function write(record: SessionRecord): void {
|
|
104
121
|
mkdirSync(sessionDir(record.sessionId), { recursive: true });
|
|
105
122
|
writeFileSync(recordPath(record.sessionId), `${JSON.stringify(record, null, 2)}\n`);
|
|
123
|
+
syncParkedMarker(record);
|
|
124
|
+
}
|
|
125
|
+
|
|
126
|
+
/**
|
|
127
|
+
* The index of PARKED sessions: one empty file per parked session, named by its
|
|
128
|
+
* id, in a directory of its own.
|
|
129
|
+
*
|
|
130
|
+
* It exists so the inline reap at `api-serve` startup costs O(parked) instead of
|
|
131
|
+
* O(every record that has ever run). Reaping has to find parked sessions, and
|
|
132
|
+
* finding them by reading every `session.json` is what made connection setup
|
|
133
|
+
* scale with session count (celilo#1454; celilo#1440 was its extreme form, at
|
|
134
|
+
* 94,706 records and 11.4s per connection). Parked is a rare state — usually
|
|
135
|
+
* none — so a `readdir` of this directory is the O(1) the path should always
|
|
136
|
+
* have been.
|
|
137
|
+
*
|
|
138
|
+
* ponytail: a directory of empty marker files rather than one index file. A
|
|
139
|
+
* single JSON index needs a lock — every `api-serve` process is a separate
|
|
140
|
+
* writer — while create/unlink of a named file is atomic and needs none.
|
|
141
|
+
* Upgrade to an index only if the parked set ever gets big enough to readdir
|
|
142
|
+
* slowly, which would mean something else is very wrong.
|
|
143
|
+
*
|
|
144
|
+
* It is a CACHE, never the truth: `listParkedSessions` verifies every marker
|
|
145
|
+
* against the record and drops the ones that no longer agree, so a marker lost
|
|
146
|
+
* to a crash or left behind by a prune self-heals on the next pass.
|
|
147
|
+
*
|
|
148
|
+
* ⚠️ It lives BESIDE `api-sessions`, not inside it. `listSessions` and
|
|
149
|
+
* `pruneRetiredSessions` both `readdir` that directory and treat every entry as
|
|
150
|
+
* a session id, so a marker directory nested in there would read as a corrupt
|
|
151
|
+
* session — and the prune would delete the whole index on its own mtime.
|
|
152
|
+
*/
|
|
153
|
+
export function getParkedDir(): string {
|
|
154
|
+
return join(getDataDir(), 'api-parked');
|
|
155
|
+
}
|
|
156
|
+
|
|
157
|
+
function parkedMarkerPath(sessionId: string): string {
|
|
158
|
+
return join(getParkedDir(), sessionId);
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
function syncParkedMarker(record: SessionRecord): void {
|
|
162
|
+
if (record.state === 'parked') {
|
|
163
|
+
mkdirSync(getParkedDir(), { recursive: true });
|
|
164
|
+
// Deliberately unguarded. A parked session that fails to enter the index is
|
|
165
|
+
// one nothing will reap, which is the outage this index has to not cause —
|
|
166
|
+
// so it fails the park, loudly, exactly as a failed record write does.
|
|
167
|
+
writeFileSync(parkedMarkerPath(record.sessionId), '');
|
|
168
|
+
return;
|
|
169
|
+
}
|
|
170
|
+
try {
|
|
171
|
+
rmSync(parkedMarkerPath(record.sessionId), { force: true });
|
|
172
|
+
} catch {
|
|
173
|
+
// A marker we cannot remove is only a stale entry, and `listParkedSessions`
|
|
174
|
+
// drops those on sight. Nothing here is worth failing an unpark over.
|
|
175
|
+
}
|
|
176
|
+
}
|
|
177
|
+
|
|
178
|
+
/**
|
|
179
|
+
* Every session the index says is parked, confirmed against its record.
|
|
180
|
+
*
|
|
181
|
+
* A marker whose record is gone, unreadable, or no longer parked is stale — the
|
|
182
|
+
* session retired, or a prune removed it — so it is removed here rather than
|
|
183
|
+
* paid for on every future pass.
|
|
184
|
+
*/
|
|
185
|
+
export function listParkedSessions(): SessionRecord[] {
|
|
186
|
+
const markerDir = getParkedDir();
|
|
187
|
+
if (!existsSync(markerDir)) {
|
|
188
|
+
// No index yet: a host upgrading into this code may already hold parked
|
|
189
|
+
// sessions, and they must still be reaped. Pay for one full scan, build the
|
|
190
|
+
// index from it, and never scan again.
|
|
191
|
+
const parked = listSessions().filter((s) => s.state === 'parked');
|
|
192
|
+
mkdirSync(markerDir, { recursive: true });
|
|
193
|
+
for (const record of parked) writeFileSync(parkedMarkerPath(record.sessionId), '');
|
|
194
|
+
return parked;
|
|
195
|
+
}
|
|
196
|
+
|
|
197
|
+
const parked: SessionRecord[] = [];
|
|
198
|
+
for (const sessionId of readdirSync(markerDir)) {
|
|
199
|
+
const record = readSession(sessionId);
|
|
200
|
+
if (record?.state === 'parked') {
|
|
201
|
+
parked.push(record);
|
|
202
|
+
continue;
|
|
203
|
+
}
|
|
204
|
+
try {
|
|
205
|
+
rmSync(parkedMarkerPath(sessionId), { force: true });
|
|
206
|
+
} catch {
|
|
207
|
+
// Same as above: a stale marker costs one failed read on the next pass.
|
|
208
|
+
}
|
|
209
|
+
}
|
|
210
|
+
return parked;
|
|
106
211
|
}
|
|
107
212
|
|
|
108
213
|
/**
|
|
@@ -255,16 +360,93 @@ export function expiryReason(record: SessionRecord): string {
|
|
|
255
360
|
* parked child on its own timer, and again at `api-serve` startup so a session
|
|
256
361
|
* whose owner died (reboot, OOM) doesn't sit parked forever — the failure mode
|
|
257
362
|
* that left a `module deploy` holding every backup lock on the fleet for 20 days.
|
|
363
|
+
*
|
|
364
|
+
* Reads the parked INDEX, not every record, so its cost tracks the number of
|
|
365
|
+
* parked sessions (usually none) rather than the number that have ever run. It
|
|
366
|
+
* stays on the connection path for the reason above — it is the only thing that
|
|
367
|
+
* reaps on a host with no dispatcher running — and staying there is only
|
|
368
|
+
* defensible because it is now O(parked). celilo#1454.
|
|
369
|
+
*
|
|
370
|
+
* It no longer runs the retention prune. Pruning is housekeeping, is O(every
|
|
371
|
+
* record) by nature, and nothing waits on it, so it belongs on the hourly sweep
|
|
372
|
+
* (`runApiSessionSweep`) rather than in front of a caller's first command.
|
|
258
373
|
*/
|
|
259
374
|
export function reapExpiredSessions(opts: { busDbPath: string; now?: number }): SessionRecord[] {
|
|
260
375
|
const now = opts.now ?? Date.now();
|
|
261
|
-
const expired =
|
|
376
|
+
const expired = listParkedSessions().filter((s) => s.expiresAt <= now);
|
|
262
377
|
for (const record of expired) {
|
|
263
378
|
abandonSession(record, { busDbPath: opts.busDbPath, reason: expiryReason(record) });
|
|
264
379
|
}
|
|
265
380
|
return expired;
|
|
266
381
|
}
|
|
267
382
|
|
|
383
|
+
/**
|
|
384
|
+
* How long a RETIRED session's directory is kept.
|
|
385
|
+
*
|
|
386
|
+
* A finished record is retained as history on purpose (see `finish`), but the
|
|
387
|
+
* retention was unbounded: nothing in this file ever deleted a directory, so
|
|
388
|
+
* the set grew by one per API connection and never shrank. That is expensive in
|
|
389
|
+
* a way nothing about it looks expensive, because `reapExpiredSessions` reads
|
|
390
|
+
* EVERY record, and `api-serve` calls it before it will accept a command — so
|
|
391
|
+
* every caller pays for every session that has ever run, on every connection.
|
|
392
|
+
*
|
|
393
|
+
* Measured on celilo-mgr, 2026-09-27: 94,706 directories, 1.2 GB, and an 11.4s
|
|
394
|
+
* gap between `api-serve` sending `ready` and reading its first command, on a
|
|
395
|
+
* box whose root is an SD card. The command itself took 240ms. The web console
|
|
396
|
+
* makes two such connections per panel, so every panel cost 23-36s against a
|
|
397
|
+
* shorter client timeout and read as "unavailable — Request Timeout".
|
|
398
|
+
* celilo#1440.
|
|
399
|
+
*
|
|
400
|
+
* A day is far past the 30-minute TTL, so nothing live can be caught by it.
|
|
401
|
+
*/
|
|
402
|
+
export const SESSION_RETENTION_MS = 24 * 60 * 60 * 1000;
|
|
403
|
+
|
|
404
|
+
/** A directory's own mtime, for records too corrupt to carry their own. */
|
|
405
|
+
function dirModifiedAt(sessionId: string): number | null {
|
|
406
|
+
try {
|
|
407
|
+
return statSync(sessionDir(sessionId)).mtimeMs;
|
|
408
|
+
} catch {
|
|
409
|
+
return null;
|
|
410
|
+
}
|
|
411
|
+
}
|
|
412
|
+
|
|
413
|
+
/**
|
|
414
|
+
* Remove the directories of sessions that retired longer than `retentionMs`
|
|
415
|
+
* ago, and return how many went.
|
|
416
|
+
*
|
|
417
|
+
* Only RETIRED sessions: `running` and `parked` are left alone at any age. A
|
|
418
|
+
* long-running command is not garbage, and a parked one is the reaper's
|
|
419
|
+
* business — `abandonSession` deliberately declines to abandon a session whose
|
|
420
|
+
* question has since been answered, so an old parked record can be a live one.
|
|
421
|
+
*
|
|
422
|
+
* A directory whose record will not parse is pruned on its own mtime. Such a
|
|
423
|
+
* directory is invisible to `listSessions`, so nothing else here can ever see
|
|
424
|
+
* it, and it would sit on disk forever while still costing every scan a
|
|
425
|
+
* `readdir` entry and a failed read.
|
|
426
|
+
*/
|
|
427
|
+
export function pruneRetiredSessions(opts: { now?: number; retentionMs?: number } = {}): number {
|
|
428
|
+
const dir = getSessionsDir();
|
|
429
|
+
if (!existsSync(dir)) return 0;
|
|
430
|
+
const cutoff = (opts.now ?? Date.now()) - (opts.retentionMs ?? SESSION_RETENTION_MS);
|
|
431
|
+
|
|
432
|
+
let removed = 0;
|
|
433
|
+
for (const sessionId of readdirSync(dir)) {
|
|
434
|
+
const record = readSession(sessionId);
|
|
435
|
+
if (record && (record.state === 'running' || record.state === 'parked')) continue;
|
|
436
|
+
const retiredAt = record ? record.startedAt : dirModifiedAt(sessionId);
|
|
437
|
+
if (retiredAt === null || retiredAt > cutoff) continue;
|
|
438
|
+
try {
|
|
439
|
+
rmSync(sessionDir(sessionId), { recursive: true, force: true });
|
|
440
|
+
removed++;
|
|
441
|
+
} catch {
|
|
442
|
+
// A directory we cannot remove (permissions, a racing writer) is not a
|
|
443
|
+
// reason to abandon the sweep: the next caller pays for it, and that is
|
|
444
|
+
// strictly better than leaving the rest of the backlog in place.
|
|
445
|
+
}
|
|
446
|
+
}
|
|
447
|
+
return removed;
|
|
448
|
+
}
|
|
449
|
+
|
|
268
450
|
/** Buffered messages an attaching client should be replayed. */
|
|
269
451
|
export function replayOutput(
|
|
270
452
|
sessionId: string,
|
|
@@ -276,3 +458,45 @@ export function replayOutput(
|
|
|
276
458
|
const start = Math.max(fromLine, lines.length - REPLAY_LIMIT);
|
|
277
459
|
return { messages: lines.slice(start), next: lines.length };
|
|
278
460
|
}
|
|
461
|
+
|
|
462
|
+
/**
|
|
463
|
+
* The hourly pass: reap what the TTL has expired, then apply retention.
|
|
464
|
+
*
|
|
465
|
+
* This is where the O(every record) prune lives now. It used to run in front of
|
|
466
|
+
* every API connection's first command, which is what made connection cost track
|
|
467
|
+
* session count (celilo#1454).
|
|
468
|
+
*
|
|
469
|
+
* Registered as an ordinary bus subscriber whose handler is an existing CLI
|
|
470
|
+
* command, exactly like `celilo-backup-sweep` and `celilo-operations-sweep` — no
|
|
471
|
+
* new scheduler. Hourly is far finer than the 24h retention, and the pass is
|
|
472
|
+
* idempotent: with nothing expired and nothing retired it is one `readdir` of an
|
|
473
|
+
* empty index plus one of the record set.
|
|
474
|
+
*
|
|
475
|
+
* The reap stays duplicated between here and `api-serve` startup on purpose.
|
|
476
|
+
* This subscriber is the efficient path; the inline call is the one that still
|
|
477
|
+
* works when no dispatcher is running, and a host in that state is exactly the
|
|
478
|
+
* host whose owning process just died.
|
|
479
|
+
*/
|
|
480
|
+
export function runApiSessionSweep(opts: {
|
|
481
|
+
busDbPath: string;
|
|
482
|
+
now?: number;
|
|
483
|
+
retentionMs?: number;
|
|
484
|
+
}): { reaped: number; pruned: number } {
|
|
485
|
+
const now = opts.now ?? Date.now();
|
|
486
|
+
const reaped = reapExpiredSessions({ busDbPath: opts.busDbPath, now });
|
|
487
|
+
const pruned = pruneRetiredSessions({ now, retentionMs: opts.retentionMs });
|
|
488
|
+
return { reaped: reaped.length, pruned };
|
|
489
|
+
}
|
|
490
|
+
|
|
491
|
+
export const API_SESSION_SWEEP_SUBSCRIBER = 'celilo-api-session-sweep';
|
|
492
|
+
export const API_SESSION_SWEEP_PATTERN = 'timer.tick.1h';
|
|
493
|
+
|
|
494
|
+
/** Idempotent: `bus.subscribe` upserts by name. */
|
|
495
|
+
export function ensureApiSessionSweepSubscriber(bus: SubscriberRegistrar): void {
|
|
496
|
+
bus.subscribe({
|
|
497
|
+
name: API_SESSION_SWEEP_SUBSCRIBER,
|
|
498
|
+
pattern: API_SESSION_SWEEP_PATTERN,
|
|
499
|
+
handler: 'celilo api sweep',
|
|
500
|
+
registeredBy: 'celilo-api-sessions',
|
|
501
|
+
});
|
|
502
|
+
}
|
package/src/cli/commands/api.ts
CHANGED
|
@@ -245,3 +245,25 @@ export async function handleApiKeyNew(args: string[]): Promise<CommandResult> {
|
|
|
245
245
|
return { success: false, error: `Failed to generate API key: ${errMsg(error)}` };
|
|
246
246
|
}
|
|
247
247
|
}
|
|
248
|
+
|
|
249
|
+
/**
|
|
250
|
+
* `celilo api sweep` — the hourly session pass: reap expired parked sessions,
|
|
251
|
+
* then apply retention to retired ones.
|
|
252
|
+
*
|
|
253
|
+
* This is the handler behind the `celilo-api-session-sweep` subscriber. It holds
|
|
254
|
+
* the O(every record) prune that used to run in front of every API connection's
|
|
255
|
+
* first command (celilo#1454).
|
|
256
|
+
*/
|
|
257
|
+
export async function handleApiSweep(): Promise<CommandResult> {
|
|
258
|
+
try {
|
|
259
|
+
const { runApiSessionSweep } = await import('../../api/sessions');
|
|
260
|
+
const { getEventBusPath } = await import('../../config/paths');
|
|
261
|
+
const { reaped, pruned } = runApiSessionSweep({ busDbPath: getEventBusPath() });
|
|
262
|
+
return {
|
|
263
|
+
success: true,
|
|
264
|
+
message: `Session sweep: ${reaped} expired session(s) abandoned, ${pruned} retired record(s) removed`,
|
|
265
|
+
};
|
|
266
|
+
} catch (error) {
|
|
267
|
+
return { success: false, error: `Session sweep failed: ${errMsg(error)}` };
|
|
268
|
+
}
|
|
269
|
+
}
|
|
@@ -9,6 +9,7 @@
|
|
|
9
9
|
|
|
10
10
|
import type { Database } from 'bun:sqlite';
|
|
11
11
|
import { defineEvents, openBus } from '@celilo/event-bus';
|
|
12
|
+
import { ensureApiSessionSweepSubscriber } from '../../api/sessions';
|
|
12
13
|
import { getEventBusPath } from '../../config/paths';
|
|
13
14
|
import { createDbClient, findMigrationsFolder, getDb } from '../../db/client';
|
|
14
15
|
import { runMigrationsOn } from '../../db/migrate';
|
|
@@ -45,6 +46,11 @@ function ensureCoreSubscribers(): void {
|
|
|
45
46
|
// pause exists only because staging leaked, and the reaper that stops it
|
|
46
47
|
// leaking ships in this same binary.
|
|
47
48
|
ensureBackupSweepSubscriber(bus);
|
|
49
|
+
// Core, not module-dependent: any fleet that answers the remote API at all
|
|
50
|
+
// accumulates session records. This is the hourly pass that retires them,
|
|
51
|
+
// and arming it here is what gets it onto fleets that upgrade without
|
|
52
|
+
// installing anything (celilo#1454).
|
|
53
|
+
ensureApiSessionSweepSubscriber(bus);
|
|
48
54
|
} finally {
|
|
49
55
|
bus.close();
|
|
50
56
|
}
|
package/src/cli/completion.ts
CHANGED
|
@@ -409,7 +409,7 @@ export async function getCompletions(words: string[], current: number): Promise<
|
|
|
409
409
|
|
|
410
410
|
// API subcommands
|
|
411
411
|
if (command === 'api' && currentIndex === 1) {
|
|
412
|
-
const subcommands = ['grant', 'list', 'revoke', 'authorized-keys', 'key'];
|
|
412
|
+
const subcommands = ['grant', 'list', 'revoke', 'authorized-keys', 'key', 'sweep'];
|
|
413
413
|
return filterSuggestions(subcommands, args[1] || '');
|
|
414
414
|
}
|
|
415
415
|
|
package/src/cli/index.ts
CHANGED
|
@@ -19,6 +19,7 @@ import {
|
|
|
19
19
|
handleApiKeyNew,
|
|
20
20
|
handleApiList,
|
|
21
21
|
handleApiRevoke,
|
|
22
|
+
handleApiSweep,
|
|
22
23
|
} from './commands/api';
|
|
23
24
|
import { handleAptUpgrade } from './commands/apt-upgrade';
|
|
24
25
|
import { handleCapabilityInfo } from './commands/capability-info';
|
|
@@ -2176,6 +2177,7 @@ export async function runCli(argv: string[]): Promise<CommandResult> {
|
|
|
2176
2177
|
' celilo api revoke <principal>',
|
|
2177
2178
|
' celilo api authorized-keys',
|
|
2178
2179
|
' celilo api key new <name>',
|
|
2180
|
+
' celilo api sweep',
|
|
2179
2181
|
'',
|
|
2180
2182
|
'Grants are command:subcommand (module:deploy), command:* (service:*), or * (all).',
|
|
2181
2183
|
].join('\n'),
|
|
@@ -2208,6 +2210,10 @@ export async function runCli(argv: string[]): Promise<CommandResult> {
|
|
|
2208
2210
|
return handleApiAuthorizedKeys();
|
|
2209
2211
|
}
|
|
2210
2212
|
|
|
2213
|
+
if (parsed.subcommand === 'sweep') {
|
|
2214
|
+
return handleApiSweep();
|
|
2215
|
+
}
|
|
2216
|
+
|
|
2211
2217
|
if (parsed.subcommand === 'key') {
|
|
2212
2218
|
if (parsed.args[0] === 'new') {
|
|
2213
2219
|
return handleApiKeyNew(parsed.args.slice(1));
|
|
@@ -345,12 +345,6 @@ export const CAPABILITY_NAME_BASELINE: readonly CapabilityNameRow[] = [
|
|
|
345
345
|
count: 1,
|
|
346
346
|
why: 'S13 — ZONE_REQUIREMENTS, a second hand-maintained copy of well-known.ts; Phase 2 (#937)',
|
|
347
347
|
},
|
|
348
|
-
{
|
|
349
|
-
file: 'apps/celilo/src/templates/generator.ts',
|
|
350
|
-
capability: 'dns_internal',
|
|
351
|
-
count: 1,
|
|
352
|
-
why: 'X4 — declaration-driven and the model for the rest; only the error string names the capability (#945)',
|
|
353
|
-
},
|
|
354
348
|
{
|
|
355
349
|
file: 'apps/celilo/src/variables/context.ts',
|
|
356
350
|
capability: 'dns_internal',
|
|
@@ -218,6 +218,29 @@ describe('register / unregister roundtrip', () => {
|
|
|
218
218
|
}
|
|
219
219
|
});
|
|
220
220
|
|
|
221
|
+
// Every module deploy goes through an API session, so the same argument
|
|
222
|
+
// applies: registering ANY module arms the pass that retires session records.
|
|
223
|
+
// Asserted as a ROW, because "we called ensure…" is not the same fact
|
|
224
|
+
// (celilo#1454).
|
|
225
|
+
it('arms the api-session sweep for any module', () => {
|
|
226
|
+
registerModuleSubscriptions(baseManifest({}), '/p');
|
|
227
|
+
|
|
228
|
+
const bus = openBus({ dbPath, events: defineEvents({}) });
|
|
229
|
+
try {
|
|
230
|
+
const row = bus.db
|
|
231
|
+
.query<{ pattern: string; handler: string }, []>(
|
|
232
|
+
"SELECT pattern, handler FROM subscribers WHERE name = 'celilo-api-session-sweep'",
|
|
233
|
+
)
|
|
234
|
+
.get();
|
|
235
|
+
expect(row).toEqual({
|
|
236
|
+
pattern: 'timer.tick.1h',
|
|
237
|
+
handler: 'celilo api sweep',
|
|
238
|
+
});
|
|
239
|
+
} finally {
|
|
240
|
+
bus.close();
|
|
241
|
+
}
|
|
242
|
+
});
|
|
243
|
+
|
|
221
244
|
it('registers each subscription as a row, names scoped to module id', () => {
|
|
222
245
|
const result = registerModuleSubscriptions(
|
|
223
246
|
baseManifest({
|
|
@@ -17,6 +17,7 @@
|
|
|
17
17
|
import { join } from 'node:path';
|
|
18
18
|
import { defineEvents, openBus } from '@celilo/event-bus';
|
|
19
19
|
import { inArray } from 'drizzle-orm';
|
|
20
|
+
import { ensureApiSessionSweepSubscriber } from '../api/sessions';
|
|
20
21
|
import { getEventBusPath, getModuleStoragePath } from '../config/paths';
|
|
21
22
|
import { getDb } from '../db/client';
|
|
22
23
|
import { modules } from '../db/schema';
|
|
@@ -98,6 +99,11 @@ export function registerModuleSubscriptions(
|
|
|
98
99
|
// reclaims abandoned rows. Idempotent.
|
|
99
100
|
ensureOperationsSweepSubscriber(bus);
|
|
100
101
|
|
|
102
|
+
// Also unconditional, and for the same reason: every module deploy goes
|
|
103
|
+
// through an API session, so the first module on a fleet is what arms the
|
|
104
|
+
// pass that retires session records. Idempotent.
|
|
105
|
+
ensureApiSessionSweepSubscriber(bus);
|
|
106
|
+
|
|
101
107
|
// A module that can be backed up is also what switches the scheduled
|
|
102
108
|
// backup sweep on. Registering here rather than at system init means the
|
|
103
109
|
// sweep appears the moment the fleet has something to back up, and — since
|
|
@@ -21,9 +21,9 @@ import { tmpdir } from 'node:os';
|
|
|
21
21
|
import { join } from 'node:path';
|
|
22
22
|
import { copyAnsibleRoleFilesDirs } from './generator';
|
|
23
23
|
|
|
24
|
-
function moduleWithRoleFile(binary: string): string {
|
|
24
|
+
function moduleWithRoleFile(binary: string, layout = 'ansible'): string {
|
|
25
25
|
const root = mkdtempSync(join(tmpdir(), 'celilo-rolefiles-'));
|
|
26
|
-
const filesDir = join(root,
|
|
26
|
+
const filesDir = join(root, layout, 'roles', 'demo', 'files');
|
|
27
27
|
mkdirSync(filesDir, { recursive: true });
|
|
28
28
|
writeFileSync(join(filesDir, 'demo-linux-x86_64'), binary);
|
|
29
29
|
return root;
|
|
@@ -59,6 +59,24 @@ describe('copyAnsibleRoleFilesDirs', () => {
|
|
|
59
59
|
expect(generatedBinary(outputPath)).toBe('v2');
|
|
60
60
|
});
|
|
61
61
|
|
|
62
|
+
/**
|
|
63
|
+
* celilo#1461. The template pipeline accepts BOTH `ansible/` and
|
|
64
|
+
* `celilo/ansible/`; this copy accepted one, so a celilo/-layout module got
|
|
65
|
+
* its role's tasks/ and templates/ and never its files/. `existsSync` was
|
|
66
|
+
* false, the function returned, and nothing anywhere said so.
|
|
67
|
+
*
|
|
68
|
+
* Generated output is the standard layout either way — the source layout is
|
|
69
|
+
* what varies.
|
|
70
|
+
*/
|
|
71
|
+
test('finds role files/ under the celilo/ layout too', async () => {
|
|
72
|
+
const modulePath = moduleWithRoleFile('v1', 'celilo/ansible');
|
|
73
|
+
const outputPath = mkdtempSync(join(tmpdir(), 'celilo-out-'));
|
|
74
|
+
|
|
75
|
+
await copyAnsibleRoleFilesDirs(modulePath, outputPath);
|
|
76
|
+
|
|
77
|
+
expect(generatedBinary(outputPath)).toBe('v1');
|
|
78
|
+
});
|
|
79
|
+
|
|
62
80
|
test('a module with no role files/ directory is not an error', async () => {
|
|
63
81
|
const root = mkdtempSync(join(tmpdir(), 'celilo-norole-'));
|
|
64
82
|
mkdirSync(join(root, 'ansible', 'roles', 'demo', 'tasks'), { recursive: true });
|
|
@@ -60,9 +60,18 @@ const COPY_AS_IS_EXTENSIONS = ['.yml', '.yaml', '.tf'];
|
|
|
60
60
|
/**
|
|
61
61
|
* Template directories to process
|
|
62
62
|
*/
|
|
63
|
+
// A module may keep its deployment tree at the root or under `celilo/`.
|
|
64
|
+
// This is THE list of supported layouts — everything that has to find a
|
|
65
|
+
// module's ansible/ or terraform/ derives it from here. celilo#1461 was two
|
|
66
|
+
// functions deciding that independently and disagreeing.
|
|
67
|
+
const LAYOUT_ROOTS = ['', 'celilo'];
|
|
68
|
+
|
|
69
|
+
const layoutDirs = (name: string): string[] =>
|
|
70
|
+
LAYOUT_ROOTS.map((root) => (root ? `${root}/${name}` : name));
|
|
71
|
+
|
|
63
72
|
// Directories to scan for template files. Supports both root-level and
|
|
64
73
|
// celilo/ subdirectory layouts (e.g., ansible/ or celilo/ansible/).
|
|
65
|
-
const TEMPLATE_DIRS = ['terraform', 'ansible'
|
|
74
|
+
const TEMPLATE_DIRS = [...layoutDirs('terraform'), ...layoutDirs('ansible')];
|
|
66
75
|
|
|
67
76
|
/**
|
|
68
77
|
* Check if file is a template that needs variable resolution
|
|
@@ -513,8 +522,9 @@ export async function readTemplateFiles(
|
|
|
513
522
|
* @param files - Generated files to write
|
|
514
523
|
*/
|
|
515
524
|
/**
|
|
516
|
-
* Copy each
|
|
517
|
-
* to the generated output, verbatim (preserving binary content + mode
|
|
525
|
+
* Copy each `<layout>/ansible/roles/<role>/files/` directory from the module
|
|
526
|
+
* source to the generated output, verbatim (preserving binary content + mode
|
|
527
|
+
* bits). Both supported layouts are scanned — see LAYOUT_ROOTS.
|
|
518
528
|
*
|
|
519
529
|
* These hold Ansible role-local static assets — they may be binaries and
|
|
520
530
|
* don't need template variable resolution, so the standard template
|
|
@@ -524,38 +534,49 @@ export async function copyAnsibleRoleFilesDirs(
|
|
|
524
534
|
modulePath: string,
|
|
525
535
|
outputPath: string,
|
|
526
536
|
): Promise<void> {
|
|
527
|
-
|
|
528
|
-
|
|
529
|
-
|
|
530
|
-
|
|
531
|
-
|
|
532
|
-
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
|
|
536
|
-
|
|
537
|
-
|
|
538
|
-
|
|
539
|
-
|
|
540
|
-
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
|
|
544
|
-
|
|
545
|
-
|
|
546
|
-
|
|
547
|
-
|
|
548
|
-
|
|
549
|
-
|
|
550
|
-
|
|
551
|
-
|
|
552
|
-
|
|
553
|
-
|
|
554
|
-
|
|
555
|
-
|
|
556
|
-
|
|
557
|
-
|
|
558
|
-
|
|
537
|
+
let copied = 0;
|
|
538
|
+
for (const ansibleDir of layoutDirs('ansible')) {
|
|
539
|
+
const rolesDir = join(modulePath, ansibleDir, 'roles');
|
|
540
|
+
if (!existsSync(rolesDir)) continue;
|
|
541
|
+
const roles = await readdir(rolesDir, { withFileTypes: true });
|
|
542
|
+
for (const role of roles) {
|
|
543
|
+
if (!role.isDirectory()) continue;
|
|
544
|
+
const srcFilesDir = join(rolesDir, role.name, 'files');
|
|
545
|
+
if (!existsSync(srcFilesDir)) continue;
|
|
546
|
+
const destFilesDir = join(outputPath, 'ansible', 'roles', role.name, 'files');
|
|
547
|
+
await mkdir(dirname(destFilesDir), { recursive: true });
|
|
548
|
+
// `force: true` is LOAD-BEARING on bun, and its absence was celilo#925.
|
|
549
|
+
//
|
|
550
|
+
// Node defaults `force` to true, so this looked correct and is correct
|
|
551
|
+
// under Node. Bun 1.3.3 does not, on this path specifically — measured,
|
|
552
|
+
// copying "NEW" over an existing "OLD":
|
|
553
|
+
//
|
|
554
|
+
// recursive only -> NEW
|
|
555
|
+
// recursive + force -> NEW
|
|
556
|
+
// recursive + preserveTimestamps -> OLD <- what this was
|
|
557
|
+
// recursive + force + preserveTimestamps -> NEW
|
|
558
|
+
//
|
|
559
|
+
// celilo runs on bun. So a module's built binary landed in `generated/`
|
|
560
|
+
// exactly once, at first generate, and no later version ever replaced it —
|
|
561
|
+
// silently, because `cp` reports no error, so the caller's try/catch has
|
|
562
|
+
// nothing to catch. Ansible then copies that first binary forever and
|
|
563
|
+
// reports `ok`, unchanged, while the module's version field advances.
|
|
564
|
+
//
|
|
565
|
+
// The sibling call in `storage-set-path.ts:153` already passes `force`.
|
|
566
|
+
await cp(srcFilesDir, destFilesDir, {
|
|
567
|
+
recursive: true,
|
|
568
|
+
force: true,
|
|
569
|
+
preserveTimestamps: true,
|
|
570
|
+
});
|
|
571
|
+
copied++;
|
|
572
|
+
}
|
|
573
|
+
}
|
|
574
|
+
|
|
575
|
+
// "I looked and found nothing" must not read the same as "I copied
|
|
576
|
+
// everything". A module with no role files/ is legitimate; a module that has
|
|
577
|
+
// one somewhere this function cannot see is celilo#1461, and it was silent.
|
|
578
|
+
if (copied === 0) {
|
|
579
|
+
log.info(`No Ansible role files/ directories found under ${modulePath}`);
|
|
559
580
|
}
|
|
560
581
|
}
|
|
561
582
|
|