@awebai/oats 0.69.18 → 0.69.19

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -10,7 +10,8 @@
10
10
  * - Only a loss the kernel can vouch for is resumed (lib/session-seen.mjs resumeEligible, from what
11
11
  * local listings recorded of the home): seen running for its current start, not judged by an
12
12
  * earlier sweep, first seen lost less than 24 hours ago, and, within one boot, lost less than 24
13
- * hours after it was last seen running. Any other is `skipped`, `not-seen-recently`. A sweep that
13
+ * hours after it was last seen running. Any other is `skipped`, `not-seen-recently`. A home the
14
+ * sweep started is recorded as seen running for its new start. A sweep that
14
15
  * is not a plan judges each lost home it answered (`judgedBy`), so it is answered once.
15
16
  * - Each home starts as a child process of this kernel (`session start --resume --home <home>
16
17
  * --json`, its cwd the home's deployment), so each start resolves its own deployment's scope; at
@@ -20,18 +21,31 @@
20
21
  * and boot id; a lock whose holder is gone, or that was taken in another boot, is taken over at
21
22
  * once. A second sweep answers E_RESUME_BUSY; a lock that needs a person, E_RESUME_LOCK_STUCK,
22
23
  * which `oats doctor` reports too (acquireResumeLock, resumeLockProblem).
23
- * - No retry storm. A loss the sweep answered is judged (above) and not started again, whatever the
24
- * boot; one note per boot (`note.json`) backs that: a refused resume is noted, and a later sweep
25
- * in the same boot reports it again (`refused-this-boot`) without starting it, also when its
26
- * judgement was lost (a home whose start id changed since is not matched); and a home is started by sweeps at most RESUME_LIMIT
27
- * times in a boot (counted by home: each start makes a new start id), then reported
28
- * `keeps-getting-lost` without a start. A start counts when it launched something (`resumed`,
29
- * `fresh`, `blocked`), noted as soon as that start has answered, so a sweep that is killed while
30
- * other starts run has counted the ones that answered. `session start --resume` run by hand is
31
- * neither bounded nor counted.
24
+ * - No retry storm, and no loss given up for a cause that passes.
25
+ * A start counts when it launched something (`resumed`, `fresh`, `blocked`), noted as soon as
26
+ * that start has answered, so a sweep that is killed while other starts run has counted the
27
+ * ones that answered. A home is started by sweeps at most RESUME_LIMIT times in
28
+ * RESUME_WINDOW_MS (counted by home: each start makes a new start id), then reported
29
+ * `keeps-getting-lost` without a start, with `nextAttemptAt`: when the oldest counted start
30
+ * leaves the window a sweep starts it again.
31
+ * That is not for ever: a home that uses up its starts in RESUME_SPENT_HOURS hours in a row
32
+ * is answered `keeps-getting-lost` with no `nextAttemptAt`, and that loss is judged: no sweep
33
+ * starts it again, and a person does. Hours are in a row when each begins less than
34
+ * RESUME_WINDOW_MS after the one before it cleared; only an hour without one begins the
35
+ * count again. No start does, whoever makes it: a person may start a home the sweep gave up,
36
+ * and if it is lost again in a used-up hour it is given up again at once.
37
+ * A refused start launched nothing: it is not counted, and it does not judge the loss. It is
38
+ * noted, and a sweep tries that home and start again no sooner than RETRY_FIRST_MS after the
39
+ * refusal, doubled at each further refusal up to RETRY_CEILING_MS; until then a sweep reports
40
+ * it `start-refused-retrying` with the refusal's `code` and `message` and `nextAttemptAt`,
41
+ * without a start. The retries end when the loss is no longer one the kernel vouches for
42
+ * (first seen 24 hours ago: `not-seen-recently`).
43
+ * The note is one boot's (`note.json`): another boot's is replaced whole, so a reboot starts
44
+ * the count and the delays again. `session start --resume` run by hand is neither bounded
45
+ * nor counted.
32
46
  * A current-start host-stop receipt identifies an administrative restore: a successful one
33
47
  * bypasses the crash ceiling without spending or clearing the carried budget. A blocked or
34
- * failed restore keeps its failure accounting, and eligibility/prior refusal checks still hold.
48
+ * failed restore keeps its failure accounting, and eligibility checks still hold.
35
49
  * Without a boot id (neither Linux nor macOS) nothing is noted or counted: each sweep tries again.
36
50
  * The lock and the note are kernel-private, not a contract. `plan` predicts from the note, starting
37
51
  * nothing, taking no lock and writing no note. */
@@ -45,7 +59,7 @@ import { acquireDirLock, dirLockHeld, stuckLockRemedy } from "./dir-lock.mjs";
45
59
  import { homeClaimHolder, listInstances, launchEnvRefs, missingLaunchEnvRefs, owedRetireSaid, resumeConversation } from "./core.mjs";
46
60
  import { checkStartEnv, launchEnvChangedMessage, launchEnvMissingMessage, launchEnvUnverifiedWarning } from "./start-env.mjs";
47
61
  import { FRESH_REASON, NOT_SWEPT, OUTCOME, RESUME_WAITING, ROW_REASON, SESSION_END } from "./session-resume-contract.mjs";
48
- import { bootId, homeKey, judgeLosses, readSeen, resumeEligible, resumeStateDir, sameBoot, takenInAnotherBoot } from "./session-seen.mjs";
62
+ import { bootId, homeKey, judgeLosses, readSeen, recordSeen, resumeEligible, resumeStateDir, sameBoot, takenInAnotherBoot } from "./session-seen.mjs";
49
63
  import { isAgentScopeWarning } from "./tmux-server-scope.mjs";
50
64
  import { hostStoppedAt as readHostStoppedAt } from "./session-end.mjs";
51
65
 
@@ -54,8 +68,16 @@ export { bootId, resumeStateDir };
54
68
  export const RESUME_SCHEMA = 1;
55
69
  /** At most this many starts run at once. */
56
70
  export const RESUME_CONCURRENCY = 4;
57
- /** At most this many sweep starts of one home in one boot. */
71
+ /** At most this many sweep starts of one home in RESUME_WINDOW_MS. */
58
72
  export const RESUME_LIMIT = 3;
73
+ /** The span over which a home's sweep starts are counted. */
74
+ export const RESUME_WINDOW_MS = 60 * 60_000;
75
+ /** A home that uses up its starts in this many hours in a row is not started by a sweep again. */
76
+ export const RESUME_SPENT_HOURS = 3;
77
+ /** How long after a refused start a sweep tries it again: this after the first refusal, doubled at
78
+ * each further one, never more than RETRY_CEILING_MS. */
79
+ export const RETRY_FIRST_MS = 60_000;
80
+ export const RETRY_CEILING_MS = 15 * 60_000;
59
81
  const lockPath = () => join(resumeStateDir(), "lock");
60
82
  const notePath = () => join(resumeStateDir(), "note.json");
61
83
 
@@ -106,32 +128,66 @@ export function resumeLockProblem({ boot = bootId() } = {}) {
106
128
  return { code: RESUME_LOCK_STUCK, message: `${stuck.said}: oats session resume starts nothing on this machine while it is there`, remedy: stuck.remedy };
107
129
  }
108
130
 
109
- /** This boot's note: `{ bootId, refused: [{ home, startId, code, message, at }], resumes: { <home>: n } }`,
110
- * each home by its real path (lib/session-seen.mjs homeKey);
111
- * empty for another boot's note, an unreadable one, or no boot id. */
112
- export function readBootNote(boot = bootId()) {
131
+ /** This boot's note as it is stored: `{ bootId, refused: [{ home, startId, code, message, at,
132
+ * attempts }], starts: { <home>: [<ISO time of a sweep start that launched something>] },
133
+ * spent: { <home>: { hours, since } } }` (`since` the oldest counted start of the home's last
134
+ * used-up hour, `hours` how many such hours in a row that makes), each home by its real path
135
+ * (lib/session-seen.mjs homeKey); empty for another boot's note, an unreadable one, or no boot id. */
136
+ function readNote(boot) {
113
137
  const note = boot ? readJson(notePath()) : undefined;
114
- if (!note || !sameBoot(note.bootId, boot)) return { bootId: boot, refused: [], resumes: {} };
115
- const resumes = {};
116
- if (note.resumes && typeof note.resumes === "object" && !Array.isArray(note.resumes)) {
117
- for (const [home, n] of Object.entries(note.resumes)) if (Number.isSafeInteger(n) && n > 0) resumes[home] = n;
138
+ const map = (v) => (v && typeof v === "object" && !Array.isArray(v) ? v : {});
139
+ if (!note || !sameBoot(note.bootId, boot)) return { bootId: boot, refused: [], starts: {}, spent: {} };
140
+ const starts = {};
141
+ if (note.starts && typeof note.starts === "object" && !Array.isArray(note.starts)) {
142
+ for (const [home, times] of Object.entries(note.starts)) if (Array.isArray(times)) starts[home] = times.filter((t) => typeof t === "string" && Number.isFinite(Date.parse(t)));
118
143
  }
119
- return { bootId: boot, refused: Array.isArray(note.refused) ? note.refused.filter((r) => r && typeof r.home === "string") : [], resumes };
144
+ const spent = Object.fromEntries(Object.entries(map(note.spent)).filter(([, v]) => v && Number.isSafeInteger(v.hours) && v.hours > 0 && Number.isFinite(Date.parse(v.since))));
145
+ return { bootId: boot, refused: Array.isArray(note.refused) ? note.refused.filter((r) => r && typeof r.home === "string") : [], starts, spent };
120
146
  }
121
- /** Note a sweep's `refusals` (each replacing its home's earlier entry) and the `launched` homes (each
122
- * counted once more) for this boot; another boot's note is replaced whole. Nothing is written
123
- * without a boot id. */
124
- export function noteSweep({ refusals = [], launched = [] }, boot = bootId()) {
125
- if (!boot || (!refusals.length && !launched.length)) return;
126
- const note = readBootNote(boot);
147
+ /** The used-up hour that begins with the counted start `since`, after the home's earlier mark
148
+ * `mark` (a note's `spent` entry, or none) → `{ hours, since }`: the same hour again is the mark
149
+ * itself; an hour that begins less than RESUME_WINDOW_MS after the marked one cleared is one
150
+ * more in a row; any other is the first. */
151
+ export function spentHours(mark, since) {
152
+ if (mark && mark.since === since) return { hours: mark.hours, since };
153
+ const inARow = mark && Date.parse(since) - (Date.parse(mark.since) + RESUME_WINDOW_MS) < RESUME_WINDOW_MS;
154
+ return { hours: inARow ? mark.hours + 1 : 1, since };
155
+ }
156
+ /** The times in `times` that are inside the counting window at `now`, oldest first. */
157
+ const inWindow = (times = [], now) => times.filter((t) => now - Date.parse(t) < RESUME_WINDOW_MS).sort();
158
+ /** This boot's note: `{ bootId, refused, resumes: { <home>: n } }`, `refused` as it is stored
159
+ * (readNote) and `n` the sweep starts of the home counted at `now`: those of the last
160
+ * RESUME_WINDOW_MS. */
161
+ export function readBootNote(boot = bootId(), now = Date.now()) {
162
+ const note = readNote(boot), resumes = {};
163
+ for (const [home, times] of Object.entries(note.starts)) { const n = inWindow(times, now).length; if (n > 0) resumes[home] = n; }
164
+ return { bootId: boot, refused: note.refused, resumes };
165
+ }
166
+ /** When a sweep tries the refused start `prior` (a note's entry) again, in ms: RETRY_FIRST_MS
167
+ * after its refusal, doubled for each refusal before it, at most RETRY_CEILING_MS. An entry
168
+ * whose time does not read is due at once. */
169
+ export function retryDueAt(prior) {
170
+ const at = Date.parse(prior?.at);
171
+ if (!Number.isFinite(at)) return 0;
172
+ const refusals = Number.isSafeInteger(prior.attempts) && prior.attempts > 0 ? prior.attempts : 1;
173
+ return at + Math.min(RETRY_FIRST_MS * 2 ** Math.min(refusals - 1, 30), RETRY_CEILING_MS);
174
+ }
175
+ /** Note a sweep's `refusals` (each replacing its home's earlier entry), the `launched` homes (each
176
+ * a start at `now`, and the starts that left the counting window dropped) and the used-up hours
177
+ * it marked (`spent`: { <home>: { hours, since } }) for this boot; another boot's note is
178
+ * replaced whole. Nothing is written without a boot id. */
179
+ export function noteSweep({ refusals = [], launched = [], spent = {} }, boot = bootId(), now = Date.now()) {
180
+ if (!boot || (!refusals.length && !launched.length && !Object.keys(spent).length)) return;
181
+ const note = readNote(boot);
127
182
  const homes = new Set(refusals.map((r) => r.home));
128
- const resumes = { ...note.resumes };
129
- for (const home of launched) resumes[home] = (resumes[home] ?? 0) + 1;
183
+ const starts = {};
184
+ for (const [home, times] of Object.entries(note.starts)) { const kept = inWindow(times, now); if (kept.length) starts[home] = kept; }
185
+ for (const home of launched) starts[home] = [...(starts[home] ?? []), new Date(now).toISOString()];
130
186
  const path = notePath();
131
187
  mkdirSync(resumeStateDir(), { recursive: true, mode: 0o700 });
132
188
  const tmp = `${path}.${process.pid}.tmp`;
133
189
  try {
134
- writeFileSync(tmp, JSON.stringify({ bootId: boot, refused: [...note.refused.filter((r) => !homes.has(r.home)), ...refusals], resumes }, null, 2) + "\n", { mode: 0o600 });
190
+ writeFileSync(tmp, JSON.stringify({ bootId: boot, refused: [...note.refused.filter((r) => !homes.has(r.home)), ...refusals], starts, spent: { ...note.spent, ...spent } }, null, 2) + "\n", { mode: 0o600 });
135
191
  renameSync(tmp, path);
136
192
  } catch (e) { rmSync(tmp, { force: true }); throw e; }
137
193
  }
@@ -166,10 +222,12 @@ function readDeployment(dir) {
166
222
 
167
223
  /** The reasons of the rows a sweep answers without judging the loss, so that a later sweep may
168
224
  * resume it: a home OATS was stopping, upgrading or retiring (`lifecycle-busy`), whose next state
169
- * is that command's to decide. A loop guard's refusal (`keeps-getting-lost`) judges the loss like
170
- * any refusal, and nothing brings it back automatically, a reboot included (decided for #1567);
171
- * naming it here would let the next boot retry it. */
172
- const NOT_JUDGED = new Set([ROW_REASON.LIFECYCLE_BUSY]);
225
+ * is that command's to decide; a start that was refused, which launched nothing and says how the
226
+ * host was at that moment (`start-refused`, and `start-refused-retrying` while its delay runs);
227
+ * and a home whose starts of the last RESUME_WINDOW_MS used up the budget (`keeps-getting-lost`),
228
+ * which a sweep starts again when the window has moved on; unless it did so RESUME_SPENT_HOURS
229
+ * hours in a row (the sweep's `givenUp`), which is judged. */
230
+ const NOT_JUDGED = new Set([ROW_REASON.LIFECYCLE_BUSY, ROW_REASON.START_REFUSED, ROW_REASON.START_REFUSED_RETRYING, ROW_REASON.KEEPS_GETTING_LOST]);
173
231
  /** Outcomes that can consume the crash budget; a successful host-stop restore is exempt below. */
174
232
  const LAUNCHED = new Set([OUTCOME.RESUMED, OUTCOME.FRESH, OUTCOME.BLOCKED]);
175
233
  const base = (row) => ({ deployment: row.deployment, instance: row.instance, home: row.home });
@@ -262,10 +320,11 @@ export async function resumeSessions({ deployment, plan = false, kernel, env = p
262
320
  else if (row.sessionEnd === SESSION_END.LOST || (row.retirePending && row.launched === true && row.running === false)) candidates.push(row);
263
321
  }
264
322
  }
265
- const note = readBootNote(boot), now = Date.now();
266
- const toStart = [], hostStopped = new Set();
267
- // Precedence, the first that applies: not seen recently, refused this boot, keeps getting lost,
268
- // started. The record and the note name a home by its real path (homeKey), whatever spelling the
323
+ const now = Date.now(), note = readNote(boot);
324
+ const toStart = [], hostStopped = new Set(), refusedBefore = new Map(), marked = {}, givenUp = new Set();
325
+ const iso = (ms) => new Date(ms).toISOString();
326
+ // Precedence, the first that applies: not seen recently, refused and not yet due again, keeps
327
+ // getting lost, started. The record and the note name a home by its real path (homeKey), whatever spelling the
269
328
  // sweep was given; a row shows the path it listed.
270
329
  // A home OATS is stopping, upgrading or retiring, or whose retire is owed, comes first: nothing
271
330
  // is started there. This read is only a filter (the sweep is not atomic with it): the start's
@@ -282,11 +341,21 @@ export async function resumeSessions({ deployment, plan = false, kernel, env = p
282
341
  if (hostStoppedAt !== null) hostStopped.add(key);
283
342
  if (busy) rows.push({ ...base(row), outcome: OUTCOME.SKIPPED, reason: ROW_REASON.LIFECYCLE_BUSY, message: busy });
284
343
  else if (!fit.eligible) rows.push({ ...base(row), outcome: OUTCOME.SKIPPED, reason: ROW_REASON.NOT_SEEN_RECENTLY, message: `${row.instance} is not resumed automatically: ${fit.why}; start it by hand (oats session start --resume --home ${row.home}) if it should run` });
285
- else if (prior) rows.push({ ...base(row), outcome: OUTCOME.REFUSED, reason: ROW_REASON.REFUSED_THIS_BOOT, code: prior.code, message: prior.message });
286
- else if (!hostStopped.has(key) && (note.resumes[key] ?? 0) >= RESUME_LIMIT) rows.push({ ...base(row), outcome: OUTCOME.REFUSED, reason: ROW_REASON.KEEPS_GETTING_LOST, message: `${row.instance} was resumed ${note.resumes[key]} times in this boot and was lost again each time; no sweep starts this loss again: start it by hand (oats session start --resume --home ${row.home})` });
344
+ else if (prior && now < retryDueAt(prior)) rows.push({ ...base(row), outcome: OUTCOME.REFUSED, reason: ROW_REASON.START_REFUSED_RETRYING, code: prior.code, message: prior.message, nextAttemptAt: iso(retryDueAt(prior)) });
345
+ else if (!hostStopped.has(key) && inWindow(note.starts[key], now).length >= RESUME_LIMIT) {
346
+ const counted = inWindow(note.starts[key], now), again = iso(Date.parse(counted[0]) + RESUME_WINDOW_MS);
347
+ const hour = spentHours(note.spent[key], counted[0]);
348
+ if (!plan && (note.spent[key]?.since !== hour.since || note.spent[key]?.hours !== hour.hours)) marked[key] = hour;
349
+ const by = `start it by hand (oats session start --resume --home ${row.home})`;
350
+ if (hour.hours >= RESUME_SPENT_HOURS) {
351
+ givenUp.add(row.home);
352
+ rows.push({ ...base(row), outcome: OUTCOME.REFUSED, reason: ROW_REASON.KEEPS_GETTING_LOST, message: `${row.instance} was resumed ${counted.length} times an hour for ${hour.hours} hours in a row and was lost again each time; no sweep starts it again: ${by}` });
353
+ } else rows.push({ ...base(row), outcome: OUTCOME.REFUSED, reason: ROW_REASON.KEEPS_GETTING_LOST, nextAttemptAt: again, message: `${row.instance} was resumed ${counted.length} times in the last hour and was lost again each time; a sweep starts it again after ${again}, or ${by}` });
354
+ }
287
355
  else if (plan) rows.push(predicted(row, env));
288
- else toStart.push(row);
356
+ else { toStart.push(row); if (prior) refusedBefore.set(key, prior); }
289
357
  }
358
+ if (Object.keys(marked).length) noteSweep({ spent: marked }, boot, now);
290
359
  // Each start's answer is noted as it comes, under the lock (noteSweep is synchronous: two answers
291
360
  // never interleave in it), before the sweep waits on the others. A note that cannot be written
292
361
  // takes no more home; the lock is held until every start already running has answered, and the
@@ -299,24 +368,38 @@ export async function resumeSessions({ deployment, plan = false, kernel, env = p
299
368
  // A home removed between the read and its start (a retire finished in between) is not lost:
300
369
  // whatever its start answered.
301
370
  const gone = answer?.ok !== true && !existsSync(row.home);
302
- const answered = gone ? { ...base(row), outcome: OUTCOME.SKIPPED, reason: ROW_REASON.NOT_LOST, ...(typeof answer?.error?.code === "string" ? { code: answer.error.code } : {}) } : rowFromStart(row, answer);
303
371
  const key = homeKey(row.home);
372
+ // A refusal is noted with how many times this home and start were refused, which sets when a
373
+ // sweep tries it again; the row says when.
374
+ const refusal = { home: key, startId: row.startId ?? null, at: iso(Date.now()), attempts: (refusedBefore.get(key)?.attempts ?? 0) + 1 };
375
+ const from = gone ? { ...base(row), outcome: OUTCOME.SKIPPED, reason: ROW_REASON.NOT_LOST, ...(typeof answer?.error?.code === "string" ? { code: answer.error.code } : {}) } : rowFromStart(row, answer);
376
+ const answered = from.outcome === OUTCOME.REFUSED && boot ? { ...from, nextAttemptAt: iso(retryDueAt(refusal)) } : from;
304
377
  try {
305
378
  noteSweep({
306
- refusals: answered.outcome === OUTCOME.REFUSED ? [{ home: key, startId: row.startId ?? null, code: answered.code, message: answered.message, at: new Date().toISOString() }] : [],
379
+ refusals: answered.outcome === OUTCOME.REFUSED ? [{ ...refusal, code: answered.code, message: answered.message }] : [],
307
380
  launched: LAUNCHED.has(answered.outcome) && (!hostStopped.has(key) || answered.outcome === OUTCOME.BLOCKED) ? [key] : [],
308
381
  }, boot);
309
382
  } catch (e) { noteFailure ??= e; }
310
383
  return answered;
311
384
  }, () => noteFailure !== null || !hostRunAdmissionOpen())).filter(Boolean);
312
385
  rows.push(...started);
313
- // Each lost home this sweep answered is judged: no later sweep resumes this loss (a status read
314
- // never uses one up). A plan judges nothing.
386
+ // Each lost home this sweep answered is judged, but for the rows of NOT_JUDGED: no later sweep
387
+ // resumes a judged loss (a status read never uses one up). A plan judges nothing. Judge the
388
+ // lost start before recording its replacement: a new start resets the record and its judgement.
315
389
  if (!plan) {
316
390
  const lost = new Set(candidates.map((c) => c.home)), byDeployment = new Map();
317
- for (const r of rows) if (lost.has(r.home) && !NOT_JUDGED.has(r.reason)) byDeployment.set(r.deployment, [...(byDeployment.get(r.deployment) ?? []), { home: r.home, startId: candidates.find((c) => c.home === r.home).startId ?? null, outcome: r.outcome }]);
391
+ for (const r of rows) if (lost.has(r.home) && (!NOT_JUDGED.has(r.reason) || givenUp.has(r.home))) byDeployment.set(r.deployment, [...(byDeployment.get(r.deployment) ?? []), { home: r.home, startId: candidates.find((c) => c.home === r.home).startId ?? null, outcome: r.outcome }]);
318
392
  for (const [dir, judged] of byDeployment) judgeLosses(join(dir, "agents"), judged, { boot });
319
393
  }
394
+ // A home this sweep started is recorded as seen running for its new start. Without this,
395
+ // a home lost again before the next listing would have no record for its start: no later
396
+ // sweep would resume it, and its starts would never be counted.
397
+ const launchedIn = new Map();
398
+ for (const r of started) if (r.outcome === OUTCOME.RESUMED || r.outcome === OUTCOME.FRESH) launchedIn.set(r.deployment, (launchedIn.get(r.deployment) ?? new Set()).add(r.home));
399
+ for (const [dir, homes] of launchedIn) {
400
+ const root = join(dir, "agents");
401
+ try { recordSeen(root, listInstances(root).flatMap((a) => a.instances || []).map((i) => (homes.has(i.home) ? { ...i, running: true } : i)), { boot }); } catch { /* the next listing records it */ }
402
+ }
320
403
  if (noteFailure) throw noteFailure;
321
404
  rows.sort((a, b) => (a.deployment < b.deployment ? -1 : a.deployment > b.deployment ? 1 : a.home < b.home ? -1 : a.home > b.home ? 1 : 0));
322
405
  return { schemaVersion: RESUME_SCHEMA, plan: !!plan, deployments, rows, ...(hostWide.size ? { warnings: [...hostWide].sort() } : {}) };
@@ -6,9 +6,9 @@
6
6
  * firstNotRunningAt, firstNotRunningBoot, judgedBy } } }
7
7
  * (times ISO; boots bootId's, null without one; `judgedBy` `{ at, boot, outcome }` or null).
8
8
  *
9
- * - Observations have one producer, a local `listInstances` (`oats status`, the sweep's own read
10
- * and every other local listing): recordSeen. A status routed to another machine reads nothing
11
- * here; a start writes nothing.
9
+ * - Observations come from local `listInstances` calls (`oats status`, the sweep's own read
10
+ * and every other local listing) and from the sweep recording the homes it started: recordSeen.
11
+ * A status routed to another machine reads nothing here; `session start` writes nothing.
12
12
  * - `judgedBy` has one writer, a sweep that is not a plan (judgeLosses), once its rows are known:
13
13
  * a status read never uses up a loss.
14
14
  * - Never in the way of what reads it: an unreadable file is an empty record (which resumes
@@ -8,7 +8,7 @@
8
8
  },
9
9
  "oats.aweb": {
10
10
  "url": "https://github.com/awebai/oats-aweb.git",
11
- "ref": "v1.41.0",
11
+ "ref": "v1.42.1",
12
12
  "path": "oats-package"
13
13
  },
14
14
  "oats.jira": {
@@ -48,7 +48,7 @@
48
48
  },
49
49
  "oats.framework": {
50
50
  "url": "https://github.com/awebai/oats.git",
51
- "ref": "oats-framework/v1.25.0",
51
+ "ref": "oats-framework/v1.25.2",
52
52
  "path": "oats-package"
53
53
  }
54
54
  },
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@awebai/oats",
3
- "version": "0.69.18",
3
+ "version": "0.69.19",
4
4
  "description": "OATS (Open Agent Team Specification) — durable souls, disposable instances, targetable capability packages, and the runtime-neutral oats CLI/kernel.",
5
5
  "keywords": [
6
6
  "agents",
@@ -37,6 +37,8 @@
37
37
  "skills/",
38
38
  "injects/",
39
39
  "docs/",
40
+ "package.json",
41
+ "LICENSE",
40
42
  "README.md",
41
43
  "package-catalog.json",
42
44
  "packages/record/bin/",
@@ -10,8 +10,8 @@ const RUNTIME_ORDER = Object.freeze({ running: 0, unknown: 1, stopped: 2 });
10
10
  /** The panel's runtime order, a comparator for `sort`: running, then unknown (`running` null or anything not
11
11
  * a boolean: it may be running, so it stands above the rows nobody needs to watch), then stopped; by name
12
12
  * within each. Each row's place depends on that row alone, so it is a total order whatever `running` holds.
13
- * Read by `panelInstances` (deployment-panel.mjs). The sidebar's roster orders by name (instance-tree.mjs
14
- * `rosterOrder`, #1262). */
13
+ * Read by `panelInstances` (deployment-panel.mjs). The sidebar's roster orders by spawn time, newest first
14
+ * (instance-tree.mjs `rosterOrder`). */
15
15
  export function byRuntimeState(a, b) {
16
16
  return RUNTIME_ORDER[runtimeState(a)] - RUNTIME_ORDER[runtimeState(b)] || String(a.instance).localeCompare(String(b.instance));
17
17
  }
@@ -66,11 +66,23 @@ export function resolveLinkId(fromInstance, name, byName) {
66
66
  return sameRoot.length === 1 ? instanceId(sameRoot[0]) : null;
67
67
  }
68
68
 
69
- /** The roster's order, a comparator for `sort`: by instance name, ties broken by `instanceId`, so it is total and
70
- * deterministic. It ranks roots, siblings under one parent, and so clusters (by their first root). A start or a
71
- * stop moves no row; only adding or removing one does (Pepe's decision on #1262, comment 6196531154, which
72
- * replaces #1083's running-first order for the roster). The one place the roster's order is decided. */
69
+ /** When a row was spawned, in ms: its `createdAt` (the kernel's ISO time from instance.json), or null when it has
70
+ * none or the reader can't parse it. */
71
+ function spawnTime(instance) {
72
+ const time = typeof instance.createdAt === "string" ? Date.parse(instance.createdAt) : NaN;
73
+ return Number.isFinite(time) ? time : null;
74
+ }
75
+
76
+ /** The roster's order, a comparator for `sort`: by spawn time, newest first (Pepe, 2026-11-13: "by spawn time not
77
+ * alphabetically, i.e. new to old"); a row with no readable spawn time after every row with one; then by instance
78
+ * name, ties broken by `instanceId`, so it is total and deterministic. It ranks roots, siblings under one parent,
79
+ * and so clusters (by their first root: a bundle sits where its newest root sits, so a new child never moves it).
80
+ * A spawn time never changes, so a start or a stop moves no row; only adding or removing one does (#1262's
81
+ * recognition rule, whose name order this replaces; #1083's running-first order stays out of the roster). The one
82
+ * place the roster's order is decided. */
73
83
  function rosterOrder(a, b) {
84
+ const ta = spawnTime(a), tb = spawnTime(b);
85
+ if (ta !== tb) return ta === null ? 1 : tb === null ? -1 : tb - ta;
74
86
  const ia = instanceId(a), ib = instanceId(b);
75
87
  return String(a.instance).localeCompare(String(b.instance)) || (ia < ib ? -1 : ia > ib ? 1 : 0);
76
88
  }
@@ -117,8 +129,7 @@ export function instanceSiblings(instances) {
117
129
  * duplicate names across repos render as distinct nodes.
118
130
  * Under one parent, the children that draw their parent line come first (same-work.mjs `drawingFirst`: on a
119
131
  * host that declares instance-same-work, the flagged ones), each part in the roster's order (#1577). The product
120
- * owner's contract (awebai/oats#1577, comment 6441606665) refines the name order (the Desktop expert's decision
121
- * roster-name-order-preserves-recognition) inside one parent's children only: roots, clusters and groups keep it.
132
+ * owner's contract (awebai/oats#1577, comment 6441606665) refines the roster's order inside one parent's children only: roots, clusters and groups keep it.
122
133
  * Returns [{ key, instances: [{...instance, depth}] }] with clusters ranked by
123
134
  * their first ROOT in the roster's order (`rosterOrder`), never by another member (#1262): a bundle sits where
124
135
  * its root sits, and gaining or losing a child, or a start or a stop, moves nothing. A cluster without a root
@@ -185,7 +196,7 @@ export function clusterInstances(instances, { links = instanceLinks, roster = in
185
196
  // Deterministic cluster label: the lexically-smallest ROOT name —
186
197
  // independent of liveness, so the visible cluster name does not flip
187
198
  // when a different member starts/stops running (review f921f7d nit).
188
- // `rank` (rosterOrder: by name) governs display ORDER within the cluster.
199
+ // `rank` (rosterOrder: by spawn time, newest first) governs display ORDER within the cluster.
189
200
  const rootNames = (roots.length ? roots : members).map((i) => i.instance).sort();
190
201
  clusters.push({ key: rootNames[0], instances: ordered, head: roots[0] ?? [...members].sort(rank)[0] });
191
202
  }
@@ -533,7 +533,7 @@ installed CLI's catalog:
533
533
  node -e 'const c=require(process.argv[1]);console.log(c.packages["oats.framework"].ref,c.packages["oats.aweb"].ref)' "$(oats root)/package-catalog.json"
534
534
  ```
535
535
 
536
- It prints the two refs, for example `oats-framework/v1.25.0 v1.41.0`. If the
536
+ It prints the two refs, for example `oats-framework/v1.25.2 v1.42.1`. If the
537
537
  framework's version (after the `/`) is below `1.19.0`, or with aweb
538
538
  `oats.aweb`'s is below `v1.33.1`, this OATS release is older than this
539
539
  skill: stop, and tell the person to wait for the next OATS
@@ -770,8 +770,8 @@ name: acme
770
770
  members:
771
771
  - git:github.com/acme/acme-oats # the host; the setup expert adds the others
772
772
  packages:
773
- oats.framework: v1.25.0 # the installed CLI's catalog versions (below)
774
- oats.aweb: v1.41.0 # messaging: agent identities, mail and chat
773
+ oats.framework: v1.25.2 # the installed CLI's catalog versions (below)
774
+ oats.aweb: v1.42.1 # messaging: agent identities, mail and chat
775
775
  oats.machine: v0.1.1 # on a machine found-list only: the machine's steward
776
776
  defaults:
777
777
  capabilities: