@ours.network/fleet 0.7.1 → 0.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -51,9 +51,11 @@ roles:
51
51
 
52
52
  Each role gets a state dir (`~/.ours-fleet/agents/<Name>/`) holding its briefing,
53
53
  logs, routines, and session markers. On boot the agent reads its briefing: bind
54
- identity, publish bio/persona, arm a mail monitor (`ours-mcp watch`), announce to
55
- its coordinator, work. On crash the supervisor relaunches it and the harness
56
- resumes the same session.
54
+ identity, publish bio/persona, announce to its coordinator, work while the
55
+ supervisor delivers its mail wakes as `[fleet-monitor]` console lines (see
56
+ [Mail monitor](#mail-monitor); `monitor.enabled: false` reverts to the agent
57
+ arming its own `ours-mcp watch`). On crash the supervisor relaunches it and the
58
+ harness resumes the same session.
57
59
 
58
60
  The state dir contract:
59
61
 
@@ -178,16 +180,27 @@ the default `~/fleet.yaml` on its very first crash-restart and fail to resolve.
178
180
 
179
181
  ```yaml
180
182
  vars: { work_root: /home/me/work } # ${var} substitution anywhere below
183
+ start_stagger_ms: 0 # delay between agent LAUNCHES (host-wide, ms); 0 = no stagger
181
184
  defaults:
182
185
  harness: claude-code # for roles that don't set one
183
186
  model: claude-fable-5 # default model for roles that don't set one (per-role model / --model wins)
184
187
  max_tokens: 500000 # session cap (harness-interpreted)
188
+ monitor: # supervisor-owned mail wake (fleet-wide default)
189
+ enabled: true # default true; a role block overrides key-by-key
185
190
  roles:
186
191
  Name: # [A-Za-z0-9_-]+
187
192
  harness: claude-code
188
193
  identity: "Display Name" # ours identity to bind (default: Name)
189
194
  cwd: ${work_root}/repo # where the harness process runs
190
195
  coordinator: FleetCoordinator # announce target on boot
196
+ monitor: # deterministic wake, owned by the supervisor
197
+ enabled: true # default (defaults.monitor.enabled ?? true);
198
+ # # false = legacy in-session `ours-mcp watch`
199
+ wake_sources: # which daemon events wake the console (default:
200
+ - message_received # message_received, file_received,
201
+ - file_received # local_contact_request, pending_message)
202
+ batch_ms: 2000 # coalesce a burst into one line (default 2000)
203
+ inject: notification # notification (default) | full (bodies inline; roadmap)
191
204
  model: claude-fable-5 # launch on a specific model (pass-through id; default: launcher default)
192
205
  mission: one line
193
206
  persona: | # operating contract (published as persona)
@@ -225,7 +238,41 @@ Merge order: `fleet.yaml` ← `fleet.d/*.yaml`; a duplicate role name is a hard
225
238
  error naming both files. Identities and roles are decoupled — removing a role
226
239
  never deletes an identity. `defaults.harness_options` is shallow-merged with each
227
240
  role's `harness_options`, so a fleet can set common Codex permission/profile defaults
228
- and override individual keys per role.
241
+ and override individual keys per role. `monitor` merges the same way — a role block
242
+ overrides `defaults.monitor` key-by-key.
243
+
244
+ ### Start staggering
245
+
246
+ `start_stagger_ms` (top-level, host-wide, default `0`) spaces out agent **launches**
247
+ so a burst of boots doesn't hit the harness/API rate limit (429) all at once. When
248
+ set, each launch is held until at least `start_stagger_ms` after the previous one
249
+ across the whole host. It is enforced at the harness-launch point inside the runner,
250
+ so — unlike a delay in the `up`/`restart` command loop — it also covers **systemd
251
+ host boot**, where every agent's unit starts concurrently. The gate is time-based:
252
+ a lone start or a solo crash-restart waits **zero**; only genuinely concurrent
253
+ launches are spread out. Example: `start_stagger_ms: 4000` on a 7-agent fleet spaces
254
+ their boots ~4 s apart instead of firing all seven at once.
255
+
256
+ > **Migration (v0.9+):** this replaces the old `FLEET_START_STAGGER` environment
257
+ > variable, which only staggered the `ours-fleet up`/`restart` command loop (not
258
+ > host boot) and defaulted to 5 s. `FLEET_START_STAGGER` is **retired** — set
259
+ > `start_stagger_ms` in `fleet.yaml` instead (note: milliseconds, and the default
260
+ > is now `0`, so add it explicitly if you relied on the old implicit 5 s spacing).
261
+
262
+ ### Mail monitor
263
+
264
+ With `monitor.enabled` (the default), the **supervisor** delivers a role's mail
265
+ wakes: the per-role runner long-polls the ours daemon's notification API and
266
+ injects a single `[fleet-monitor] N new messages from … — run get_messages` line
267
+ straight into the console. It is a deterministic program whose lifetime is fused to
268
+ the tmux session — it primes the notification cursor *before* the session launches
269
+ (no missed arrivals), cannot be orphaned or left deaf-but-armed, and writes its
270
+ health to `<agentDir>/.monitor-status` (`armed | degraded | failed`), surfaced in
271
+ `ours-fleet status`/`doctor`. The agent's briefing tells it **not** to arm an
272
+ in-session Monitor. Set `monitor.enabled: false` to keep the legacy behavior where
273
+ the agent arms its own `ours-mcp watch`. `inject: full` (pushing message bodies
274
+ inline) is on the roadmap and needs two new ours-mcp daemon endpoints; today all
275
+ roles deliver `notification` lines and drain via `get_messages`.
229
276
 
230
277
  ## Codex roles
231
278
 
package/dist/briefing.js CHANGED
@@ -34,7 +34,12 @@ export function generateBriefing(role, v, opts) {
34
34
  : ' with a 1–2 sentence summary of your Charter above. Skip if it already matches.');
35
35
  L.push(`5. SET your **persona** (local operating contract, never shared in invites) via`);
36
36
  L.push(` **${v.setPersonaTool}** with the **Charter** section above, verbatim. Skip if it matches.`);
37
- L.push(`6. ${v.monitorInstruction(id, role)}`);
37
+ // When the supervisor owns the monitor (monitor.enabled), the agent must NOT arm
38
+ // its own in-session watch — wakes are injected as [fleet-monitor] lines (design §5).
39
+ const wakeNote = role.monitor?.enabled
40
+ ? v.supervisedWakeNote(id, role)
41
+ : v.monitorInstruction(id, role);
42
+ L.push(`6. ${wakeNote}`);
38
43
  if (role.coordinator) {
39
44
  L.push(`7. ANNOUNCE yourself: call **${v.sendTool}** to contact "${role.coordinator}" with text:`);
40
45
  L.push(` "${role.name} online — identity '${id}' bound, ready."`);
@@ -70,7 +75,7 @@ export function generateBriefing(role, v, opts) {
70
75
  L.push('change between wakes without a restart; treat the file, not your memory of it, as current.');
71
76
  L.push('', '## On restart (you run under a supervised launcher)');
72
77
  L.push(`On restart, WITHOUT asking: re-bind (**${v.bindTool}** name "${id}" force=true), then`);
73
- L.push(`${v.monitorInstruction(id, role)} Then continue from your WORKLOG.`);
78
+ L.push(`${wakeNote} Then continue from your WORKLOG.`);
74
79
  L.push('Do not blindly re-run whatever may have crashed you.');
75
80
  L.push('', '## House rules');
76
81
  L.push('- Never broad `rm -rf` on home/critical paths; quote globs; use explicit paths.');
package/dist/cli.js CHANGED
@@ -26,7 +26,6 @@ catch {
26
26
  const deps = () => ({
27
27
  backend: pickBackend(),
28
28
  binPath,
29
- sleep: ms => new Promise(r => setTimeout(r, ms)),
30
29
  log: l => console.log(l),
31
30
  });
32
31
  const die = (e) => { console.error(String(e instanceof Error ? e.message : e)); process.exit(1); };
package/dist/config.d.ts CHANGED
@@ -3,6 +3,21 @@ export interface OverseeEntry {
3
3
  role: string;
4
4
  interval: string;
5
5
  }
6
+ /** The 8 content-free event types the ours daemon appends to notifications.log. */
7
+ export declare const NOTIFY_EVENT_TYPES: readonly ["message_received", "file_received", "sibling_contact_added", "local_contact_request", "pending_message", "contact_restored", "inbound_error", "state_import_failed"];
8
+ export type NotifyEventType = (typeof NOTIFY_EVENT_TYPES)[number];
9
+ export type InjectMode = 'notification' | 'full';
10
+ /** Resolved per-role supervisor-monitor config (see DESIGN-external-monitor §2). */
11
+ export interface MonitorConfig {
12
+ enabled: boolean;
13
+ wake_sources: string[];
14
+ batch_ms: number;
15
+ inject: InjectMode;
16
+ }
17
+ /** Default wake sources when a role does not list its own (design §2). */
18
+ export declare const DEFAULT_WAKE_SOURCES: NotifyEventType[];
19
+ /** Validate a raw (role-level or merged) `monitor:` block; returns human-readable problems. */
20
+ export declare function validateMonitorConfig(raw: unknown): string[];
6
21
  export interface RoleConfig {
7
22
  harness?: string;
8
23
  identity?: string;
@@ -19,21 +34,37 @@ export interface RoleConfig {
19
34
  oversee?: OverseeEntry[];
20
35
  harness_options?: Record<string, unknown>;
21
36
  isolation?: IsolationConfig;
37
+ monitor?: Partial<MonitorConfig>;
22
38
  }
23
39
  export interface ResolvedRole extends RoleConfig {
24
40
  name: string;
25
41
  harness: string;
26
42
  identity: string;
27
43
  sourceFile: string;
44
+ monitor: MonitorConfig;
28
45
  }
29
46
  export interface FleetConfig {
30
47
  roles: ResolvedRole[];
31
48
  vars: Record<string, string>;
32
49
  defaults: Record<string, unknown>;
33
50
  files: string[];
51
+ /** Fleet-wide delay (ms) enforced between agent launches to avoid boot bursts (0 = none). */
52
+ startStaggerMs: number;
34
53
  }
35
54
  export declare class ConfigError extends Error {
36
55
  }
37
56
  /** Load ~/fleet.yaml (or an explicit path) merged with ~/fleet.d/*.yaml drop-ins. */
38
57
  export declare function loadConfig(configPath?: string): FleetConfig;
58
+ /**
59
+ * Merge `defaults.monitor` under the role's own `monitor:` key-by-key, validate the
60
+ * result, and fill code-constant defaults (design §2). `defaults.monitor.enabled`
61
+ * is the fleet-wide default; absent everywhere ⇒ enabled. Throws ConfigError on a
62
+ * malformed block so a typo fails loudly rather than silently disarming a monitor.
63
+ * Exported so temp-spawn (which builds a ResolvedRole by hand) resolves identically.
64
+ */
65
+ export declare function resolveMonitorConfig(defMonitor: unknown, roleMonitor?: Partial<MonitorConfig>, labels?: {
66
+ base?: string;
67
+ file?: string;
68
+ name?: string;
69
+ }): MonitorConfig;
39
70
  export declare function findRole(cfg: FleetConfig, name: string): ResolvedRole;
package/dist/config.js CHANGED
@@ -3,13 +3,52 @@ import { join } from 'node:path';
3
3
  import { parse } from 'yaml';
4
4
  import { defaultConfigPath, fleetDDir } from './paths.js';
5
5
  import { validateIsolationConfig } from './isolation/policy.js';
6
+ /** The 8 content-free event types the ours daemon appends to notifications.log. */
7
+ export const NOTIFY_EVENT_TYPES = [
8
+ 'message_received', 'file_received', 'sibling_contact_added', 'local_contact_request',
9
+ 'pending_message', 'contact_restored', 'inbound_error', 'state_import_failed',
10
+ ];
11
+ /** Default wake sources when a role does not list its own (design §2). */
12
+ export const DEFAULT_WAKE_SOURCES = ['message_received', 'file_received', 'local_contact_request', 'pending_message'];
13
+ const MONITOR_KEYS = ['enabled', 'wake_sources', 'batch_ms', 'inject'];
14
+ const INJECT_MODES = ['notification', 'full'];
15
+ const MONITOR_DEFAULT_BATCH_MS = 2000;
16
+ const isPlainObject = (v) => v !== null && typeof v === 'object' && !Array.isArray(v);
17
+ /** Validate a raw (role-level or merged) `monitor:` block; returns human-readable problems. */
18
+ export function validateMonitorConfig(raw) {
19
+ const problems = [];
20
+ if (!isPlainObject(raw))
21
+ return ['monitor: must be a mapping'];
22
+ const m = raw;
23
+ const bad = Object.keys(m).filter(k => !MONITOR_KEYS.includes(k));
24
+ if (bad.length)
25
+ problems.push(`monitor: unknown key(s) ${bad.join(', ')}; allowed: ${MONITOR_KEYS.join(', ')}`);
26
+ if (m.enabled !== undefined && typeof m.enabled !== 'boolean')
27
+ problems.push('monitor.enabled: must be true or false');
28
+ if (m.batch_ms !== undefined
29
+ && (typeof m.batch_ms !== 'number' || !Number.isFinite(m.batch_ms) || m.batch_ms < 0))
30
+ problems.push('monitor.batch_ms: must be a non-negative number');
31
+ if (m.inject !== undefined && !INJECT_MODES.includes(m.inject))
32
+ problems.push(`monitor.inject: invalid value '${m.inject}'; allowed: ${INJECT_MODES.join(', ')}`);
33
+ if (m.wake_sources !== undefined) {
34
+ if (!Array.isArray(m.wake_sources))
35
+ problems.push('monitor.wake_sources: must be a list');
36
+ else {
37
+ const unknown = m.wake_sources.filter(w => !NOTIFY_EVENT_TYPES.includes(w));
38
+ if (unknown.length)
39
+ problems.push(`monitor.wake_sources: unknown source(s) ${unknown.join(', ')}; ` +
40
+ `allowed: ${NOTIFY_EVENT_TYPES.join(', ')}`);
41
+ }
42
+ }
43
+ return problems;
44
+ }
6
45
  export class ConfigError extends Error {
7
46
  }
8
47
  const NAME_RE = /^[A-Za-z0-9_-]+$/;
9
48
  const ROLE_KEYS = [
10
49
  'harness', 'identity', 'cwd', 'coordinator', 'mission', 'persona', 'bio',
11
50
  'briefing_file', 'model', 'max_tokens', 'autocompact_pct', 'env', 'oversee', 'harness_options',
12
- 'isolation',
51
+ 'isolation', 'monitor',
13
52
  ];
14
53
  function deepSub(v, vars) {
15
54
  if (typeof v === 'string')
@@ -47,6 +86,7 @@ export function loadConfig(configPath) {
47
86
  const baseDoc = docs.length && docs[0].file === base ? docs[0].doc : {};
48
87
  const vars = (baseDoc.vars ?? {});
49
88
  const defaults = (baseDoc.defaults ?? {});
89
+ const startStaggerMs = resolveStartStaggerMs(baseDoc.start_stagger_ms, base);
50
90
  const seen = new Map();
51
91
  const roles = [];
52
92
  for (const { file, doc } of docs) {
@@ -78,6 +118,7 @@ export function loadConfig(configPath) {
78
118
  if (problems.length)
79
119
  throw new ConfigError(`${file}: role '${name}' ${problems.join('; ')}`);
80
120
  }
121
+ const monitor = resolveMonitorConfig(defaults.monitor, r.monitor, { base, file, name });
81
122
  roles.push({
82
123
  ...r,
83
124
  name,
@@ -88,10 +129,46 @@ export function loadConfig(configPath) {
88
129
  max_tokens: r.max_tokens ?? defaults.max_tokens,
89
130
  harness_options: harnessOptions,
90
131
  isolation,
132
+ monitor,
91
133
  });
92
134
  }
93
135
  }
94
- return { roles, vars, defaults, files };
136
+ return { roles, vars, defaults, files, startStaggerMs };
137
+ }
138
+ /** Validate the top-level `start_stagger_ms` (supervisor launch spacing); default 0. */
139
+ function resolveStartStaggerMs(raw, base) {
140
+ if (raw === undefined || raw === null)
141
+ return 0;
142
+ if (typeof raw !== 'number' || !Number.isFinite(raw) || raw < 0)
143
+ throw new ConfigError(`${base}: start_stagger_ms must be a non-negative number of milliseconds (got ${JSON.stringify(raw)})`);
144
+ return raw;
145
+ }
146
+ /**
147
+ * Merge `defaults.monitor` under the role's own `monitor:` key-by-key, validate the
148
+ * result, and fill code-constant defaults (design §2). `defaults.monitor.enabled`
149
+ * is the fleet-wide default; absent everywhere ⇒ enabled. Throws ConfigError on a
150
+ * malformed block so a typo fails loudly rather than silently disarming a monitor.
151
+ * Exported so temp-spawn (which builds a ResolvedRole by hand) resolves identically.
152
+ */
153
+ export function resolveMonitorConfig(defMonitor, roleMonitor, labels = {}) {
154
+ const where = labels.file && labels.name ? `${labels.file}: role '${labels.name}' ` : '';
155
+ if (defMonitor !== undefined && !isPlainObject(defMonitor))
156
+ throw new ConfigError(`${labels.base ?? 'config'}: defaults.monitor must be a map`);
157
+ if (roleMonitor !== undefined && !isPlainObject(roleMonitor))
158
+ throw new ConfigError(`${where}monitor: must be a mapping`);
159
+ const merged = {
160
+ ...(defMonitor ?? {}),
161
+ ...(roleMonitor ?? {}),
162
+ };
163
+ const problems = validateMonitorConfig(merged);
164
+ if (problems.length)
165
+ throw new ConfigError(`${where}${problems.join('; ')}`);
166
+ return {
167
+ enabled: merged.enabled ?? true,
168
+ wake_sources: merged.wake_sources ?? [...DEFAULT_WAKE_SOURCES],
169
+ batch_ms: merged.batch_ms ?? MONITOR_DEFAULT_BATCH_MS,
170
+ inject: merged.inject ?? 'notification',
171
+ };
95
172
  }
96
173
  export function findRole(cfg, name) {
97
174
  const r = cfg.roles.find(r => r.name === name);
package/dist/doctor.d.ts CHANGED
@@ -1,7 +1,8 @@
1
1
  import { type Exec } from './exec.js';
2
+ import { type FetchLike } from './monitor.js';
2
3
  import type { PrereqReport } from './harness/types.js';
3
4
  /** Host-level + per-harness prerequisite report with actionable messages. */
4
5
  export declare function doctor(opts?: {
5
6
  harness?: string;
6
7
  configPath?: string;
7
- }, exec?: Exec, platform?: NodeJS.Platform): Promise<PrereqReport>;
8
+ }, exec?: Exec, platform?: NodeJS.Platform, fetchImpl?: FetchLike): Promise<PrereqReport>;
package/dist/doctor.js CHANGED
@@ -20,7 +20,7 @@ function cgroupDelegationDetail() {
20
20
  }
21
21
  }
22
22
  /** Host-level + per-harness prerequisite report with actionable messages. */
23
- export async function doctor(opts = {}, exec = realExec, platform = process.platform) {
23
+ export async function doctor(opts = {}, exec = realExec, platform = process.platform, fetchImpl = (u, i) => globalThis.fetch(u, i)) {
24
24
  const checks = [];
25
25
  const major = Number(process.versions.node.split('.')[0]);
26
26
  checks.push({
@@ -107,6 +107,38 @@ export async function doctor(opts = {}, exec = realExec, platform = process.plat
107
107
  detail = `backend=${policy.backend} (not yet implemented)`;
108
108
  checks.push({ name: `isolation: ${r.name}`, ok, detail });
109
109
  }
110
+ // Monitor daemon-API reachability (design §5): only when a role is supervised.
111
+ // /state-dir is unauthenticated (liveness); /identities exercises the token so a
112
+ // shared-mode misconfig (401) surfaces here rather than as a silent deaf monitor.
113
+ if (roles.some(r => r.monitor?.enabled)) {
114
+ const port = Number(process.env.OURS_PORT) || 3050;
115
+ const token = process.env.OURS_API_TOKEN;
116
+ const headers = token ? { 'x-ours-api-token': token } : {};
117
+ let ok = false, detail;
118
+ try {
119
+ const live = await fetchImpl(`http://127.0.0.1:${port}/state-dir`, {});
120
+ if (!live.ok) {
121
+ detail = `daemon on :${port} answered /state-dir with HTTP ${live.status} — not the ours daemon?`;
122
+ }
123
+ else {
124
+ const auth = await fetchImpl(`http://127.0.0.1:${port}/identities`, { headers });
125
+ if (auth.status === 401)
126
+ detail = `reachable on :${port} but the API token was rejected (401) — set OURS_API_TOKEN to the ` +
127
+ `daemon's token (shared mode) or run the fleet as the daemon owner`;
128
+ else if (!auth.ok)
129
+ detail = `reachable on :${port} but /identities returned HTTP ${auth.status}`;
130
+ else {
131
+ ok = true;
132
+ detail = `reachable on :${port}, authorized — supervisor wake stream available`;
133
+ }
134
+ }
135
+ }
136
+ catch (e) {
137
+ detail = `unreachable on :${port} — monitored roles run degraded until it is up ` +
138
+ `(start it: ours-mcp start) [${e?.message ?? e}]`;
139
+ }
140
+ checks.push({ name: 'monitor: daemon API', ok, detail });
141
+ }
110
142
  const harnesses = opts.harness
111
143
  ? [opts.harness]
112
144
  : [...new Set(roles.map(r => r.harness))];
@@ -109,10 +109,15 @@ export function makeClaudeCodeAdapter(exec = realExec) {
109
109
  getMessagesTool: 'get_messages',
110
110
  watchCommand: id => `ours-mcp watch "${id}"`,
111
111
  monitorInstruction: id => `Arm a **persistent Monitor** running the shell command \`ours-mcp watch "${id}"\` so inbound ours mail wakes you.`,
112
+ supervisedWakeNote: () => 'Your mail wake-ups are delivered by the fleet supervisor directly into this console as ' +
113
+ '`[fleet-monitor]` lines — do NOT arm an in-session Monitor. When such a line appears, run ' +
114
+ '**get_messages** to drain the mail.',
112
115
  launchNote: name => `You were launched with \`--remote-control ${name}\`. Confirm you are running.`,
113
- restartPrompt: (id, worklog) => `Session restarted. Re-bind your ours identity now (choose_identity name "${id}" force=true), ` +
114
- `re-arm your monitor (ours-mcp watch "${id}"), then continue from ${worklog}. ` +
115
- 'Do not re-run whatever crashed you.',
116
+ restartPrompt: (id, worklog, role) => `Session restarted. Re-bind your ours identity now (choose_identity name "${id}" force=true), ` +
117
+ (role?.monitor?.enabled
118
+ ? 'then continue from '
119
+ : `re-arm your monitor (ours-mcp watch "${id}"), then continue from `) +
120
+ `${worklog}. Do not re-run whatever crashed you.`,
116
121
  },
117
122
  exitPolicy: { cleanExitIsFresh: true, fastFailSecs: 20 },
118
123
  };
@@ -197,8 +197,15 @@ export function makeCodexAdapter(exec = realExec) {
197
197
  `fallback. After each arrival, call **get_messages**, handle the mail, and re-enter ` +
198
198
  `**foreground_monitor** while the approved monitoring session remains armed.`;
199
199
  },
200
+ supervisedWakeNote: () => 'Your mail wake-ups are delivered by the fleet supervisor directly into this console as ' +
201
+ '`[fleet-monitor]` lines — do NOT arm arm_monitor or foreground_monitor. When such a line ' +
202
+ 'appears, call **get_messages**, handle the mail, and reply with send_message.',
200
203
  launchNote: name => `You were launched as the fleet role \`${name}\` under a Codex session. Confirm you are running.`,
201
204
  restartPrompt: (id, worklog, configuredRole) => {
205
+ if (configuredRole?.monitor?.enabled)
206
+ return `Session restarted. Re-bind your ours identity now (choose_identity name "${id}" force=true); ` +
207
+ 'your mail wakes are delivered by the fleet supervisor as `[fleet-monitor]` console lines, so do ' +
208
+ `NOT arm arm_monitor/foreground_monitor. Continue from ${worklog}. Do not re-run whatever crashed you.`;
202
209
  const consented = configuredRole?.harness_options?.monitor === true;
203
210
  return `Session restarted. Re-bind your ours identity now (choose_identity name "${id}" force=true), ` +
204
211
  (consented
@@ -37,6 +37,8 @@ export interface BriefingVocab {
37
37
  getMessagesTool: string;
38
38
  watchCommand(identity: string): string;
39
39
  monitorInstruction(identity: string, role?: ResolvedRole): string;
40
+ /** Wake-source wording for a role whose monitor is supervisor-owned (monitor.enabled). */
41
+ supervisedWakeNote(identity: string, role?: ResolvedRole): string;
40
42
  launchNote(name: string): string;
41
43
  restartPrompt(identity: string, worklogPath: string, role?: ResolvedRole): string;
42
44
  }
@@ -0,0 +1,103 @@
1
+ import type { MonitorConfig, NotifyEventType } from './config.js';
2
+ /** A content-free arrival event as the daemon serves it over the notifications API. */
3
+ export interface NotifyEvent {
4
+ event?: NotifyEventType | string;
5
+ from?: string;
6
+ msg_id?: number | string;
7
+ file_id?: number | string;
8
+ date?: string;
9
+ queued?: number | string;
10
+ }
11
+ export interface FetchResponse {
12
+ status: number;
13
+ ok: boolean;
14
+ json(): Promise<{
15
+ cursor?: number;
16
+ events?: NotifyEvent[];
17
+ }>;
18
+ }
19
+ export type FetchLike = (url: string, init?: {
20
+ headers?: Record<string, string>;
21
+ signal?: AbortSignal;
22
+ }) => Promise<FetchResponse>;
23
+ export interface MonitorTmux {
24
+ has(name: string): Promise<boolean>;
25
+ capture(name: string, lines?: number): Promise<string>;
26
+ sendText(name: string, text: string): Promise<void>;
27
+ sendKey(name: string, key: string): Promise<void>;
28
+ }
29
+ export interface MonitorDeps {
30
+ fetch: FetchLike;
31
+ tmux: MonitorTmux;
32
+ isAlive(pid: number): boolean;
33
+ sleep(ms: number): Promise<void>;
34
+ now(): number;
35
+ log(line: string): void;
36
+ env: NodeJS.ProcessEnv;
37
+ timers: {
38
+ set(fn: () => void, ms: number): ReturnType<typeof setTimeout>;
39
+ clear(t: ReturnType<typeof setTimeout>): void;
40
+ };
41
+ }
42
+ /** Resolve the daemon endpoint + auth header from the environment (design §6c). */
43
+ export declare function resolveEndpoint(env: NodeJS.ProcessEnv): {
44
+ url(name: string): string;
45
+ headers: Record<string, string>;
46
+ };
47
+ /** Keep only the events whose type the role asked to wake on. */
48
+ export declare function filterEvents(events: NotifyEvent[], wakeSources: string[]): NotifyEvent[];
49
+ /**
50
+ * Summarize a (coalesced) batch of events into one content-free console line —
51
+ * count + senders + ids, ending in the call to action. Falls back to compact
52
+ * counts when a burst would blow past the length cap (design §3, edge: burst).
53
+ */
54
+ export declare function formatNotificationLine(events: NotifyEvent[]): string;
55
+ /**
56
+ * Heuristic: does the pane show a modal selection dialog we must not `Enter`
57
+ * into? Markers are the deployed Claude Code trust/permission dialogs — a `❯`
58
+ * pointer beside numbered options, or a "Do you want …" prompt (design §3.2,
59
+ * open question (a): refine empirically). A running turn is NOT modal.
60
+ */
61
+ export declare function looksModal(pane: string): boolean;
62
+ export interface MonitorOpts {
63
+ name: string;
64
+ agentDir: string;
65
+ cfg: MonitorConfig;
66
+ deps: MonitorDeps;
67
+ }
68
+ /** The lifecycle surface the runner drives: prime pre-launch, run, stop on pid death. */
69
+ export interface MonitorHandle {
70
+ prime(): Promise<void>;
71
+ run(pid: number): Promise<void>;
72
+ stop(): void;
73
+ }
74
+ export declare class Monitor {
75
+ private readonly name;
76
+ private readonly cfg;
77
+ private readonly deps;
78
+ private readonly ep;
79
+ private readonly statusPath;
80
+ private readonly cursorPath;
81
+ private cursor;
82
+ private fatal;
83
+ private stopped;
84
+ private bootDeadline;
85
+ private currentAbort;
86
+ constructor(o: MonitorOpts);
87
+ /** Prime at the stream tip (or resume a persisted cursor if the daemon is down). */
88
+ prime(): Promise<void>;
89
+ /** Long-poll → filter → coalesce → inject, until the pane pid dies or stop(). */
90
+ run(pid: number): Promise<void>;
91
+ stop(): void;
92
+ /** Gather stragglers arriving within batch_ms so a burst lands as one line. */
93
+ private coalesce;
94
+ private deliver;
95
+ /** Block until the console can accept input; classify offline/stopped/ready. */
96
+ private awaitInjectable;
97
+ private doFetch;
98
+ private advance;
99
+ private persistCursor;
100
+ private readPersistedCursor;
101
+ private setStatus;
102
+ }
103
+ export declare function createMonitor(o: MonitorOpts): Monitor;
@@ -0,0 +1,298 @@
1
+ import { existsSync, readFileSync, writeFileSync } from 'node:fs';
2
+ import { join } from 'node:path';
3
+ // Code constants (not config — YAGNI, design §2).
4
+ const DEFAULT_PORT = 3050;
5
+ const LONGPOLL_TIMEOUT_MS = 35_000; // > the daemon's 25s hold
6
+ const COALESCE_HOLD_MS = 500; // straggler poll must not block
7
+ const BOOT_GRACE_MS = 15_000; // hold injection until the TUI is up
8
+ const POST_VERIFY_MS = 1_000;
9
+ const MAX_ENTER_RETRIES = 2;
10
+ const MODAL_RETRY_MS = 5_000;
11
+ const BACKOFF_STEP_MS = 1_000;
12
+ const BACKOFF_MAX_MS = 5_000;
13
+ const PREFIX = '[fleet-monitor]';
14
+ const MAX_LINE = 260;
15
+ class AuthError extends Error {
16
+ }
17
+ /** Resolve the daemon endpoint + auth header from the environment (design §6c). */
18
+ export function resolveEndpoint(env) {
19
+ const port = Number(env.OURS_PORT) || DEFAULT_PORT;
20
+ const token = env.OURS_API_TOKEN;
21
+ return {
22
+ url: (name) => `http://127.0.0.1:${port}/identities/${encodeURIComponent(name)}/notifications`,
23
+ headers: token ? { 'x-ours-api-token': token } : {},
24
+ };
25
+ }
26
+ /** Keep only the events whose type the role asked to wake on. */
27
+ export function filterEvents(events, wakeSources) {
28
+ const set = new Set(wakeSources);
29
+ return events.filter(e => e.event !== undefined && set.has(e.event));
30
+ }
31
+ const uniq = (xs) => [...new Set(xs)];
32
+ const plural = (n, one, many = one + 's') => (n === 1 ? one : many);
33
+ /**
34
+ * Summarize a (coalesced) batch of events into one content-free console line —
35
+ * count + senders + ids, ending in the call to action. Falls back to compact
36
+ * counts when a burst would blow past the length cap (design §3, edge: burst).
37
+ */
38
+ export function formatNotificationLine(events) {
39
+ const of = (t) => events.filter(e => e.event === t);
40
+ const msgs = of('message_received');
41
+ const files = of('file_received');
42
+ const intros = of('local_contact_request');
43
+ const pending = of('pending_message');
44
+ const known = new Set(['message_received', 'file_received', 'local_contact_request', 'pending_message']);
45
+ const others = events.filter(e => !known.has(e.event ?? ''));
46
+ const senders = (list) => uniq(list.map(e => e.from ?? '?')).join(', ');
47
+ const ids = (list) => {
48
+ const xs = list.map(e => e.msg_id).filter(v => v !== undefined);
49
+ return xs.length ? ` (${xs.map(x => `#${x}`).join(', ')})` : '';
50
+ };
51
+ const clauses = [];
52
+ if (msgs.length)
53
+ clauses.push(`${msgs.length} new ${plural(msgs.length, 'message')} from ${senders(msgs)}${ids(msgs)}`);
54
+ if (files.length)
55
+ clauses.push(`${files.length} ${plural(files.length, 'file')} from ${senders(files)}`);
56
+ if (intros.length)
57
+ clauses.push(`${intros.length} pending ${plural(intros.length, 'introduction')} from ${senders(intros)}`);
58
+ if (pending.length)
59
+ clauses.push(`${pending.length} queued ${plural(pending.length, 'message')} from ${senders(pending)}`);
60
+ if (others.length)
61
+ clauses.push(`${others.length} other ${plural(others.length, 'event')} (${uniq(others.map(e => e.event ?? '?')).join(', ')})`);
62
+ const line = `${PREFIX} ${clauses.join(', ')} — run get_messages`;
63
+ if (line.length <= MAX_LINE)
64
+ return line;
65
+ const compact = [
66
+ msgs.length && `${msgs.length} messages`,
67
+ files.length && `${files.length} files`,
68
+ intros.length && `${intros.length} introductions`,
69
+ pending.length && `${pending.length} queued`,
70
+ others.length && `${others.length} other`,
71
+ ].filter(Boolean).join(', ');
72
+ return `${PREFIX} ${compact} — run get_messages`;
73
+ }
74
+ /**
75
+ * Heuristic: does the pane show a modal selection dialog we must not `Enter`
76
+ * into? Markers are the deployed Claude Code trust/permission dialogs — a `❯`
77
+ * pointer beside numbered options, or a "Do you want …" prompt (design §3.2,
78
+ * open question (a): refine empirically). A running turn is NOT modal.
79
+ */
80
+ export function looksModal(pane) {
81
+ if (/Do you want\b/i.test(pane))
82
+ return true;
83
+ const hasPointer = /❯/.test(pane);
84
+ const hasNumbered = /(^|\n)\s*[❯>]?\s*\d+[.)]\s+\S/.test(pane);
85
+ return hasPointer && hasNumbered;
86
+ }
87
+ /** Is the injected line still sitting unsubmitted in the composer (bottom of pane)? */
88
+ function stillInComposer(pane, line) {
89
+ const frag = line.slice(0, 48);
90
+ const tail = pane.split('\n').slice(-4).join('\n');
91
+ return tail.includes(frag);
92
+ }
93
+ export class Monitor {
94
+ name;
95
+ cfg;
96
+ deps;
97
+ ep;
98
+ statusPath;
99
+ cursorPath;
100
+ cursor = null;
101
+ fatal = false;
102
+ stopped = false;
103
+ bootDeadline = 0;
104
+ currentAbort = null;
105
+ constructor(o) {
106
+ this.name = o.name;
107
+ this.cfg = o.cfg;
108
+ this.deps = o.deps;
109
+ this.ep = resolveEndpoint(o.deps.env);
110
+ this.statusPath = join(o.agentDir, '.monitor-status');
111
+ this.cursorPath = join(o.agentDir, '.notify-cursor');
112
+ }
113
+ /** Prime at the stream tip (or resume a persisted cursor if the daemon is down). */
114
+ async prime() {
115
+ try {
116
+ const body = await this.doFetch('tip', LONGPOLL_TIMEOUT_MS);
117
+ this.cursor = typeof body.cursor === 'number' ? body.cursor : 0;
118
+ this.persistCursor();
119
+ this.setStatus('armed');
120
+ }
121
+ catch (e) {
122
+ if (e instanceof AuthError) {
123
+ this.fatal = true;
124
+ this.setStatus(`failed: ${e.message}`);
125
+ }
126
+ else {
127
+ this.cursor = this.readPersistedCursor();
128
+ this.setStatus(`degraded: prime failed (${msg(e)})`);
129
+ }
130
+ }
131
+ }
132
+ /** Long-poll → filter → coalesce → inject, until the pane pid dies or stop(). */
133
+ async run(pid) {
134
+ if (this.fatal)
135
+ return;
136
+ this.bootDeadline = this.deps.now() + BOOT_GRACE_MS;
137
+ let backoff = 0;
138
+ while (!this.stopped) {
139
+ if (!this.deps.isAlive(pid)) {
140
+ this.setStatus('degraded: session offline');
141
+ return;
142
+ }
143
+ let body;
144
+ try {
145
+ body = await this.doFetch(String(this.cursor ?? 0), LONGPOLL_TIMEOUT_MS);
146
+ backoff = 0;
147
+ }
148
+ catch (e) {
149
+ if (this.stopped)
150
+ return;
151
+ if (e instanceof AuthError) {
152
+ this.fatal = true;
153
+ this.setStatus(`failed: ${e.message}`);
154
+ return;
155
+ }
156
+ backoff = Math.min(backoff + BACKOFF_STEP_MS, BACKOFF_MAX_MS);
157
+ this.setStatus(`degraded: stream hiccup (${msg(e)})`);
158
+ await this.deps.sleep(backoff);
159
+ continue;
160
+ }
161
+ this.advance(body.cursor);
162
+ const batch = filterEvents(body.events ?? [], this.cfg.wake_sources);
163
+ if (batch.length === 0)
164
+ continue;
165
+ await this.coalesce(batch);
166
+ await this.deliver(pid, batch);
167
+ }
168
+ }
169
+ stop() {
170
+ this.stopped = true;
171
+ this.currentAbort?.abort();
172
+ }
173
+ // ── internals ──────────────────────────────────────────────────────────────
174
+ /** Gather stragglers arriving within batch_ms so a burst lands as one line. */
175
+ async coalesce(batch) {
176
+ if (this.cfg.batch_ms <= 0 || this.stopped)
177
+ return;
178
+ await this.deps.sleep(this.cfg.batch_ms);
179
+ if (this.stopped)
180
+ return;
181
+ try {
182
+ const more = await this.doFetch(String(this.cursor ?? 0), COALESCE_HOLD_MS);
183
+ this.advance(more.cursor);
184
+ batch.push(...filterEvents(more.events ?? [], this.cfg.wake_sources));
185
+ }
186
+ catch { /* no stragglers / abort — deliver what we have */ }
187
+ }
188
+ async deliver(pid, batch) {
189
+ const state = await this.awaitInjectable(pid);
190
+ if (state !== 'ready') {
191
+ if (state === 'offline')
192
+ this.setStatus('degraded: offline during delivery');
193
+ return; // events remain covered by unread.json / SessionStart backlog
194
+ }
195
+ const line = formatNotificationLine(batch);
196
+ await this.deps.tmux.sendText(this.name, line); // send-keys -l + Enter
197
+ let delivered = false;
198
+ // Verify submission for THIS line even if stop() arrives mid-flight: the text
199
+ // is already in the composer and we want it submitted (at-least-once). A truly
200
+ // dead pane makes safeCapture return '' ⇒ not-in-composer ⇒ breaks, no wasted Enter.
201
+ for (let i = 0; i < MAX_ENTER_RETRIES; i++) {
202
+ await this.deps.sleep(POST_VERIFY_MS);
203
+ const pane = await safeCapture(this.deps.tmux, this.name);
204
+ if (!stillInComposer(pane, line)) {
205
+ delivered = true;
206
+ break;
207
+ }
208
+ await this.deps.tmux.sendKey(this.name, 'Enter');
209
+ }
210
+ this.setStatus(delivered ? 'armed' : 'degraded: injection unverified');
211
+ }
212
+ /** Block until the console can accept input; classify offline/stopped/ready. */
213
+ async awaitInjectable(pid) {
214
+ for (;;) {
215
+ if (this.stopped)
216
+ return 'stopped';
217
+ if (!this.deps.isAlive(pid) || !(await this.deps.tmux.has(this.name)))
218
+ return 'offline';
219
+ const now = this.deps.now();
220
+ if (now < this.bootDeadline) {
221
+ await this.deps.sleep(this.bootDeadline - now);
222
+ continue;
223
+ }
224
+ const pane = await safeCapture(this.deps.tmux, this.name);
225
+ if (looksModal(pane)) {
226
+ await this.deps.sleep(MODAL_RETRY_MS);
227
+ continue;
228
+ }
229
+ return 'ready';
230
+ }
231
+ }
232
+ async doFetch(since, holdMs) {
233
+ const ctrl = new AbortController();
234
+ this.currentAbort = ctrl;
235
+ const timer = this.deps.timers.set(() => ctrl.abort(), holdMs);
236
+ let resp;
237
+ try {
238
+ resp = await this.deps.fetch(`${this.ep.url(this.name)}?since=${since}`, { headers: this.ep.headers, signal: ctrl.signal });
239
+ }
240
+ finally {
241
+ this.deps.timers.clear(timer);
242
+ this.currentAbort = null;
243
+ }
244
+ if (resp.status === 401)
245
+ throw new AuthError('daemon rejected the API token (401) — set OURS_API_TOKEN or run as the daemon owner');
246
+ if (!resp.ok)
247
+ throw new Error(`daemon returned HTTP ${resp.status}`);
248
+ return resp.json();
249
+ }
250
+ advance(cursor) {
251
+ if (typeof cursor === 'number' && cursor !== this.cursor) {
252
+ this.cursor = cursor;
253
+ this.persistCursor();
254
+ }
255
+ }
256
+ persistCursor() {
257
+ try {
258
+ if (this.cursor !== null)
259
+ writeFileSync(this.cursorPath, `${this.cursor}\n`);
260
+ }
261
+ catch (e) {
262
+ this.deps.log(`[${this.name}] monitor: failed to persist cursor: ${msg(e)}`);
263
+ }
264
+ }
265
+ readPersistedCursor() {
266
+ try {
267
+ if (!existsSync(this.cursorPath))
268
+ return null;
269
+ const n = parseInt(readFileSync(this.cursorPath, 'utf8').trim(), 10);
270
+ return Number.isFinite(n) ? n : null;
271
+ }
272
+ catch {
273
+ return null;
274
+ }
275
+ }
276
+ setStatus(s) {
277
+ try {
278
+ writeFileSync(this.statusPath, `${s}\n`);
279
+ }
280
+ catch (e) {
281
+ this.deps.log(`[${this.name}] monitor: failed to write status: ${msg(e)}`);
282
+ }
283
+ if (!s.startsWith('armed'))
284
+ this.deps.log(`[${this.name}] monitor ${s}`);
285
+ }
286
+ }
287
+ export function createMonitor(o) {
288
+ return new Monitor(o);
289
+ }
290
+ async function safeCapture(tmux, name) {
291
+ try {
292
+ return await tmux.capture(name);
293
+ }
294
+ catch {
295
+ return '';
296
+ }
297
+ }
298
+ const msg = (e) => e?.message ?? String(e);
package/dist/ops.d.ts CHANGED
@@ -3,7 +3,6 @@ import type { SupervisorBackend } from './supervisor/types.js';
3
3
  export interface OpsDeps {
4
4
  backend: SupervisorBackend;
5
5
  binPath: string;
6
- sleep(ms: number): Promise<void>;
7
6
  log(line: string): void;
8
7
  }
9
8
  /** Materialize a role's state dir from config: briefing + markers. Returns the dir. */
package/dist/ops.js CHANGED
@@ -5,7 +5,11 @@ import { agentDir, fleetDDir } from './paths.js';
5
5
  import { findRole } from './config.js';
6
6
  import { getAdapter } from './harness/registry.js';
7
7
  import { generateBriefing } from './briefing.js';
8
- const STAGGER_MS = () => 1000 * Number(process.env.FLEET_START_STAGGER ?? 5);
8
+ // Launch staggering now lives at the harness-launch point (the runner's start
9
+ // gate, driven by `start_stagger_ms`), so it covers systemd host-boot too — not
10
+ // just the `up`/`restart` command loop below. The old in-loop FLEET_START_STAGGER
11
+ // sleep is retired; `up`/`restart` fire installs promptly and the gate spaces the
12
+ // resulting launches.
9
13
  /** Materialize a role's state dir from config: briefing + markers. Returns the dir. */
10
14
  export function applyRole(role, opts = {}) {
11
15
  const adapter = getAdapter(role.harness);
@@ -43,11 +47,7 @@ function selectRoles(cfg, names) {
43
47
  }
44
48
  /** Create/start roles declaratively. Idempotent; active roles keep their context. */
45
49
  export async function up(cfg, names, deps, configPath) {
46
- let first = true;
47
50
  for (const role of selectRoles(cfg, names)) {
48
- if (!first)
49
- await deps.sleep(STAGGER_MS());
50
- first = false;
51
51
  const dir = applyRole(role, { configPath });
52
52
  // If the role isn't running, boot fresh so it reads the briefing we just wrote.
53
53
  const status = await deps.backend.status(role.name).catch(() => '');
@@ -70,11 +70,7 @@ export async function down(cfg, names, deps) {
70
70
  }
71
71
  /** Re-sync from config + bounce. mode 'keep' resumes context; 'fresh' wipes it. */
72
72
  export async function restartRoles(cfg, names, deps, mode, configPath) {
73
- let first = true;
74
73
  for (const role of selectRoles(cfg, names)) {
75
- if (!first)
76
- await deps.sleep(STAGGER_MS());
77
- first = false;
78
74
  applyRole(role, { fresh: mode === 'fresh', configPath });
79
75
  await deps.backend.restart(role.name);
80
76
  deps.log(mode === 'fresh'
package/dist/runner.d.ts CHANGED
@@ -1,6 +1,7 @@
1
1
  import { type ResolvedRole } from './config.js';
2
2
  import type { Launch } from './harness/types.js';
3
3
  import { Tmux } from './tmux.js';
4
+ import { type MonitorHandle, type MonitorOpts, type FetchLike } from './monitor.js';
4
5
  import { type Exec } from './exec.js';
5
6
  export interface RunnerDeps {
6
7
  tmux: Tmux;
@@ -10,6 +11,10 @@ export interface RunnerDeps {
10
11
  sleep(ms: number): Promise<void>;
11
12
  now(): number;
12
13
  log(line: string): void;
14
+ /** HTTP transport for the monitor's daemon long-poll (injectable for tests). */
15
+ fetch: FetchLike;
16
+ /** Construct the supervisor mail monitor (injectable so tests stub it out). */
17
+ createMonitor(opts: MonitorOpts): MonitorHandle;
13
18
  }
14
19
  /**
15
20
  * Compose the tmux pane shell command: env prefix + argv + exit-status capture.
@@ -19,6 +24,17 @@ export interface RunnerDeps {
19
24
  * still sees the real exit code.
20
25
  */
21
26
  export declare function buildPaneCommand(launch: Launch, roleEnv: Record<string, string> | undefined, exitStatusPath: string, paneArgv?: string[]): string;
27
+ /** Filename spawnTemp writes into a temp agent dir to carry the fleet start-stagger. */
28
+ export declare const START_STAGGER_FILE = ".start-stagger-ms";
29
+ /**
30
+ * Reserve this process's launch slot on the host-wide start gate and return the
31
+ * wall-clock time it may launch at. A tiny atomic mutex (mkdir is atomic across
32
+ * processes) guards a single `.last-launch` timestamp: each launcher takes the
33
+ * next slot = max(now, last + staggerMs), so concurrent boots serialize and spread
34
+ * out by staggerMs while a lone/idle start returns `now` (zero wait). A crashed
35
+ * launcher's stale lock is broken so the gate can never deadlock the fleet.
36
+ */
37
+ export declare function reserveLaunchSlot(root: string, staggerMs: number, deps: Pick<RunnerDeps, 'now' | 'sleep' | 'log'>): Promise<number>;
22
38
  /** Read a temp role's config snapshot written by spawnTemp. */
23
39
  export declare function loadTempRole(name: string): ResolvedRole;
24
40
  /** One supervised session lifecycle. The supervisor re-invokes us after we return. */
package/dist/runner.js CHANGED
@@ -2,10 +2,11 @@ import { existsSync, readFileSync, writeFileSync, rmSync, mkdirSync } from 'node
2
2
  import { join } from 'node:path';
3
3
  import { randomUUID } from 'node:crypto';
4
4
  import { parse } from 'yaml';
5
- import { agentDir, home } from './paths.js';
5
+ import { agentDir, home, stateRoot } from './paths.js';
6
6
  import { loadConfig, findRole } from './config.js';
7
7
  import { getAdapter } from './harness/registry.js';
8
8
  import { Tmux } from './tmux.js';
9
+ import { createMonitor } from './monitor.js';
9
10
  import { realExec, shq } from './exec.js';
10
11
  import { resolveIsolation } from './isolation/policy.js';
11
12
  import { selectIsolationBackend } from './isolation/registry.js';
@@ -24,6 +25,8 @@ const defaultDeps = () => ({
24
25
  sleep: ms => new Promise(r => setTimeout(r, ms)),
25
26
  now: () => Date.now(),
26
27
  log: line => process.stderr.write(line + '\n'),
28
+ fetch: (url, init) => globalThis.fetch(url, init),
29
+ createMonitor: opts => createMonitor(opts),
27
30
  });
28
31
  /**
29
32
  * Compose the tmux pane shell command: env prefix + argv + exit-status capture.
@@ -38,6 +41,85 @@ export function buildPaneCommand(launch, roleEnv, exitStatusPath, paneArgv = lau
38
41
  const cmd = paneArgv.map(shq).join(' ');
39
42
  return `${envPfx} ${cmd}; echo $? > ${shq(exitStatusPath)}`;
40
43
  }
44
+ /** Adapt the runner's injected deps into the monitor's dependency surface. */
45
+ function monitorDeps(deps) {
46
+ return {
47
+ fetch: deps.fetch,
48
+ tmux: deps.tmux,
49
+ isAlive: deps.isAlive,
50
+ sleep: deps.sleep,
51
+ now: deps.now,
52
+ log: deps.log,
53
+ env: process.env,
54
+ timers: { set: (fn, ms) => setTimeout(fn, ms), clear: t => clearTimeout(t) },
55
+ };
56
+ }
57
+ /** Filename spawnTemp writes into a temp agent dir to carry the fleet start-stagger. */
58
+ export const START_STAGGER_FILE = '.start-stagger-ms';
59
+ /** Read the start-stagger a temp agent was spawned with (0 if none / unreadable). */
60
+ function readStartStagger(dir) {
61
+ try {
62
+ const n = parseInt(readFileSync(join(dir, START_STAGGER_FILE), 'utf8').trim(), 10);
63
+ return Number.isFinite(n) && n > 0 ? n : 0;
64
+ }
65
+ catch {
66
+ return 0;
67
+ }
68
+ }
69
+ /**
70
+ * Reserve this process's launch slot on the host-wide start gate and return the
71
+ * wall-clock time it may launch at. A tiny atomic mutex (mkdir is atomic across
72
+ * processes) guards a single `.last-launch` timestamp: each launcher takes the
73
+ * next slot = max(now, last + staggerMs), so concurrent boots serialize and spread
74
+ * out by staggerMs while a lone/idle start returns `now` (zero wait). A crashed
75
+ * launcher's stale lock is broken so the gate can never deadlock the fleet.
76
+ */
77
+ export async function reserveLaunchSlot(root, staggerMs, deps) {
78
+ mkdirSync(root, { recursive: true });
79
+ const lockDir = join(root, '.launch-gate.lock');
80
+ const lockTsFile = join(lockDir, 'ts');
81
+ const tsFile = join(root, '.last-launch');
82
+ const staleMs = Math.max(staggerMs * 4, 10_000);
83
+ const POLL_MS = 50;
84
+ const readTs = (p) => {
85
+ try {
86
+ const n = parseInt(readFileSync(p, 'utf8').trim(), 10);
87
+ return Number.isFinite(n) ? n : null;
88
+ }
89
+ catch {
90
+ return null;
91
+ }
92
+ };
93
+ for (let waited = 0;;) {
94
+ try {
95
+ mkdirSync(lockDir);
96
+ writeFileSync(lockTsFile, String(deps.now()));
97
+ break;
98
+ }
99
+ catch (e) {
100
+ if (e.code !== 'EEXIST')
101
+ throw e;
102
+ const lockTs = readTs(lockTsFile);
103
+ const stale = lockTs !== null && deps.now() - lockTs > staleMs;
104
+ if (stale || waited > staleMs * 2) { // break a crashed launcher's lock; never deadlock
105
+ rmSync(lockDir, { recursive: true, force: true });
106
+ continue;
107
+ }
108
+ await deps.sleep(POLL_MS);
109
+ waited += POLL_MS;
110
+ }
111
+ }
112
+ try {
113
+ const now = deps.now();
114
+ const last = readTs(tsFile);
115
+ const target = last === null ? now : Math.max(now, last + staggerMs);
116
+ writeFileSync(tsFile, String(target));
117
+ return target;
118
+ }
119
+ finally {
120
+ rmSync(lockDir, { recursive: true, force: true });
121
+ }
122
+ }
41
123
  /** Read a temp role's config snapshot written by spawnTemp. */
42
124
  export function loadTempRole(name) {
43
125
  const p = join(agentDir(name, true), 'role.yaml');
@@ -70,7 +152,20 @@ export async function runOnce(name, opts = {}, partialDeps = {}) {
70
152
  const temp = opts.temp === true;
71
153
  const dir = agentDir(name, temp);
72
154
  const configPath = temp ? opts.configPath : resolveConfigPath(dir, opts.configPath);
73
- const role = temp ? loadTempRole(name) : findRole(loadConfig(configPath), name);
155
+ // Resolve the role and the fleet-wide start-stagger. Permanent roles read the
156
+ // live config; temp/detached agents read the value spawnTemp snapshotted into
157
+ // their dir (they have no config path threaded through the detached supervisor).
158
+ let role;
159
+ let staggerMs;
160
+ if (temp) {
161
+ role = loadTempRole(name);
162
+ staggerMs = readStartStagger(dir);
163
+ }
164
+ else {
165
+ const cfg = loadConfig(configPath);
166
+ role = findRole(cfg, name);
167
+ staggerMs = cfg.startStaggerMs;
168
+ }
74
169
  const adapter = getAdapter(role.harness);
75
170
  mkdirSync(dir, { recursive: true });
76
171
  const sidFile = join(dir, '.session-id');
@@ -116,6 +211,31 @@ export async function runOnce(name, opts = {}, partialDeps = {}) {
116
211
  if (rprefix.length)
117
212
  paneArgv = [...rprefix, ...paneArgv];
118
213
  }
214
+ // Start-stagger: space this launch at least start_stagger_ms after the previous
215
+ // agent launch across the whole host, so a burst of boots (systemd starts every
216
+ // user unit concurrently on boot; `ours-fleet up`/restart-all bulk-start) does not
217
+ // hit the harness/API rate limit at once. Time-based via a shared launch gate, so
218
+ // a lone start or a solo crash-restart waits zero. Applied right before the harness
219
+ // launch (tmux.newSession); the cheap monitor prime still runs immediately after.
220
+ if (staggerMs > 0) {
221
+ const slot = await reserveLaunchSlot(stateRoot(), staggerMs, deps);
222
+ const wait = slot - deps.now();
223
+ if (wait > 0) {
224
+ deps.log(`[${name}] start-stagger: holding ${wait}ms before launch`);
225
+ await deps.sleep(wait);
226
+ }
227
+ }
228
+ // Supervisor mail monitor (design §1): prime the notification cursor at the
229
+ // stream tip BEFORE the session launches so no arrival is missed during boot
230
+ // (backlog before the tip is the SessionStart hook's job). Disabled roles keep
231
+ // the legacy in-session watch. Temp snapshots predating `monitor:` are treated
232
+ // as disabled (monitor may be undefined on an old role.yaml).
233
+ const monitor = role.monitor?.enabled ? deps.createMonitor({
234
+ name, agentDir: dir, cfg: role.monitor,
235
+ deps: monitorDeps(deps),
236
+ }) : null;
237
+ if (monitor)
238
+ await monitor.prime();
119
239
  rmSync(exitFile, { force: true });
120
240
  await deps.tmux.kill(name);
121
241
  await deps.tmux.newSession(name, runCwd, buildPaneCommand(launch, role.env, exitFile, paneArgv));
@@ -128,9 +248,16 @@ export async function runOnce(name, opts = {}, partialDeps = {}) {
128
248
  if (pid === null)
129
249
  throw new Error(`[${name}] could not resolve tmux pane pid`);
130
250
  deps.log(`[${name}] up; pid=${pid} cwd=${runCwd} harness=${role.harness} mode=${mode}`);
251
+ // The monitor loop lives exactly as long as the session: it starts once the
252
+ // pane pid is known and is stopped when that pid dies (task dies with runner).
253
+ const monitorLoop = monitor?.run(pid);
131
254
  const start = deps.now();
132
255
  while (deps.isAlive(pid))
133
256
  await deps.sleep(2000);
257
+ if (monitor) {
258
+ monitor.stop();
259
+ await monitorLoop;
260
+ }
134
261
  const elapsed = (deps.now() - start) / 1000;
135
262
  const code = existsSync(exitFile) ? readFileSync(exitFile, 'utf8').trim() : 'crash';
136
263
  const rotate = (why) => {
package/dist/spawn.js CHANGED
@@ -3,8 +3,9 @@ import { existsSync, mkdirSync, openSync, readFileSync, writeFileSync } from 'no
3
3
  import { join } from 'node:path';
4
4
  import { stringify } from 'yaml';
5
5
  import { agentDir, fleetDDir } from './paths.js';
6
- import { loadConfig } from './config.js';
6
+ import { loadConfig, resolveMonitorConfig } from './config.js';
7
7
  import { applyRole, up } from './ops.js';
8
+ import { START_STAGGER_FILE } from './runner.js';
8
9
  function roleFromOpts(o, defaultHarness) {
9
10
  const r = {};
10
11
  if (o.harness)
@@ -90,10 +91,17 @@ export async function spawnTemp(o, binPath, launch = detachedSupervisor) {
90
91
  identity: o.identity ?? o.name,
91
92
  model: o.model?.trim() || cfg.defaults.model,
92
93
  harness_options: Object.keys(mergedHarnessOptions).length ? mergedHarnessOptions : undefined,
94
+ // Temp agents inherit the fleet-wide monitor defaults via the snapshot (design §2).
95
+ monitor: resolveMonitorConfig(cfg.defaults.monitor, fromOpts.monitor),
93
96
  sourceFile: '(temp)',
94
97
  };
95
98
  const dir = applyRole(role, { temp: true });
96
99
  writeFileSync(join(dir, 'role.yaml'), stringify(role));
100
+ // Snapshot the fleet start-stagger so the detached temp supervisor (no config path
101
+ // threaded through it) honors the same launch gate — a burst of temp spawns spaces
102
+ // out; a lone temp spawn still waits zero (time-based gate).
103
+ if (cfg.startStaggerMs > 0)
104
+ writeFileSync(join(dir, START_STAGGER_FILE), String(cfg.startStaggerMs));
97
105
  // Run the supervisor DETACHED — NOT inside a tmux session named <name>.
98
106
  // `_run-temp` -> runOnce() creates AND kills the tmux session <name> for the
99
107
  // agent itself; a supervisor sharing that session name would SIGHUP its own
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@ours.network/fleet",
3
- "version": "0.7.1",
3
+ "version": "0.9.0",
4
4
  "description": "Harness-agnostic fleet of persistent, identity-bound AI agents. Declarative fleet.yaml, tmux consoles, systemd/launchd supervision, ours.network messaging.",
5
5
  "type": "module",
6
6
  "license": "FSL-1.1-Apache-2.0",