shraga 0.1.32 → 0.1.34
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/defaults/skills/scheduler.md +45 -10
- package/defaults/skills/self-aware.md +4 -2
- package/package.json +1 -1
- package/src/server/boot.ts +14 -0
- package/src/server/downtime.ts +104 -19
- package/src/server/scheduler/engine.ts +53 -31
|
@@ -140,8 +140,9 @@ Schedule: { id, name, enabled, trigger, task, scope, createdBy, nextRun?, lastRu
|
|
|
140
140
|
|
|
141
141
|
## Missed windows (`onMissed` / `missedRun`)
|
|
142
142
|
|
|
143
|
-
If
|
|
144
|
-
|
|
143
|
+
If a window elapses with nothing running (deploy, crash, power cut — or the host **suspending**
|
|
144
|
+
with the process frozen), the scheduler decides whether to replay it. `onMissed` is settable on
|
|
145
|
+
`POST`/`PUT`:
|
|
145
146
|
|
|
146
147
|
| `onMissed` | Behaviour |
|
|
147
148
|
|---|---|
|
|
@@ -157,10 +158,20 @@ curl -s -X PUT -H "Content-Type: application/json" -H "x-internal-token: $INTERN
|
|
|
157
158
|
- **A staleness ceiling overrides every policy, `run` included**: a window more than
|
|
158
159
|
`SCHEDULER_MAX_MISSED_AGE_MS` (default **6h**) late is never replayed. Finishing an 08:00 report
|
|
159
160
|
at 22:00 is not the job the schedule describes.
|
|
160
|
-
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
161
|
+
- One decision (`judgeMissed` in `scheduler/engine.ts`) governs **all three** paths that could
|
|
162
|
+
replay a window:
|
|
163
|
+
1. **Boot catch-up** (cron only) — the process was down when the window passed.
|
|
164
|
+
2. **In-place resume** of a run interrupted mid-flight — every trigger kind
|
|
165
|
+
(`interval`/`once`/`event` have no catch-up, so resume is their only gate; their window is
|
|
166
|
+
the one the interrupted run recorded when it started).
|
|
167
|
+
3. **A late timer fire** — the process was *frozen*, not killed (laptop lid/standby, NTP step,
|
|
168
|
+
VM pause): same pid, no restart, so catch-up never ran, and the overdue `setTimeout` fires on
|
|
169
|
+
wake. The window is the schedule's `nextRun` (when it *should* have fired), not the wake time.
|
|
170
|
+
A punctual fire is untouched — `onMissed` is a missed-window policy, never a mute — the gate
|
|
171
|
+
engages only once the fire is past the ceiling.
|
|
172
|
+
- After a refusal the schedule is still re-armed normally: an `interval` job loses exactly one
|
|
173
|
+
cycle and fires again next period; a `once` is retired (disabled, `nextRun` cleared) with its
|
|
174
|
+
`missedRun` left for an on-demand run. Nothing is left enabled-but-never-firing.
|
|
164
175
|
- Whenever a window is not replayed, the schedule gets
|
|
165
176
|
`missedRun: { at, reason: "skip"|"offer"|"stale", noticedAt }` — visible on
|
|
166
177
|
`GET /api/schedules` and `GET /api/schedules/{id}`. **That is the `offer` affordance**: read it,
|
|
@@ -173,9 +184,32 @@ curl -s -X PUT -H "Content-Type: application/json" -H "x-internal-token: $INTERN
|
|
|
173
184
|
## "What did I miss?" — downtime recovery (`GET /api/downtime`)
|
|
174
185
|
|
|
175
186
|
This box is deliberately not always-on. The server heartbeats to `data/state/heartbeat.json` every
|
|
176
|
-
`HEARTBEAT_INTERVAL_MS` (default **60s**)
|
|
177
|
-
|
|
178
|
-
`{ from, to, ms, slackCursors }` (last **20** kept). A clean restart records nothing.
|
|
187
|
+
`HEARTBEAT_INTERVAL_MS` (default **60s**). A gap larger than `DOWNTIME_THRESHOLD_MS` (default
|
|
188
|
+
**3 intervals = 3m**) is appended to `data/state/downtime.json` as
|
|
189
|
+
`{ from, to, ms, cause, slackCursors? }` (last **20** kept). A clean restart records nothing.
|
|
190
|
+
|
|
191
|
+
Two `cause`s, one code path:
|
|
192
|
+
|
|
193
|
+
- **`cause: 'boot'`** — the process died (crash, power cut). The gap is measured at startup, from
|
|
194
|
+
the last persisted heartbeat.
|
|
195
|
+
- **`cause: 'suspend'`** — the process never died: the **host slept** (lid closed, standby) and the
|
|
196
|
+
process was *frozen*. Nothing restarts, so a boot check would never see it — instead the next
|
|
197
|
+
heartbeat tick comes back late by more than the threshold and records the gap itself. One entry
|
|
198
|
+
per outage, however long (the tick writes the heartbeat forward before the next one compares).
|
|
199
|
+
Two consequences of a `suspend` gap, both flagged in the report's `note`:
|
|
200
|
+
|
|
201
|
+
- **`missedSchedules[]` is INCOMPLETE, not empty** — the boot-time catch-up scan never ran (nothing
|
|
202
|
+
restarted), so the only records are the ones the **late timer fire** wrote itself (see "Missed
|
|
203
|
+
windows" above: a fire past the ceiling is refused and recorded as `missedRun`, at a timestamp
|
|
204
|
+
*inside* the gap, which the report then joins). Windows still inside the ceiling on wake simply
|
|
205
|
+
ran; anything else that elapsed in the gap left **no record at all** — check those by hand. The
|
|
206
|
+
note is emitted for any suspend in the last `SUSPEND_NOTE_MAX_AGE_MS` (**24h**), not just the
|
|
207
|
+
newest entry: a restart afterwards appends a `boot` entry but rescans nothing.
|
|
208
|
+
- **No `slackCursors` snapshot** — the snapshot is only taken on the `boot` path, where it is
|
|
209
|
+
provably older than any post-recovery traffic. A suspend gap is recorded from a heartbeat tick up
|
|
210
|
+
to 60s *after* the wake, by which time the reconnected socket may already have pushed the cursor
|
|
211
|
+
past the whole backlog. So the entry carries none and the backfill floors at the gap's `from` —
|
|
212
|
+
it may re-report a few already-seen messages, never skip unseen ones.
|
|
179
213
|
|
|
180
214
|
```bash
|
|
181
215
|
curl -s -H "x-internal-token: $INTERNAL_API_TOKEN" "http://localhost:$PORT/api/downtime" | jq .
|
|
@@ -192,7 +226,8 @@ Returns `{ heartbeat, downtime[], lastDowntime, missedSchedules[], slack }`:
|
|
|
192
226
|
Events API retries a dropped delivery for only ~30 minutes then discards it forever, while
|
|
193
227
|
`conversations.history` stays readable for days.
|
|
194
228
|
- **Where it starts:** the outage's `from`, raised only by the last-seen cursor **as
|
|
195
|
-
snapshotted into the downtime entry at boot** (`slackCursors`
|
|
229
|
+
snapshotted into the downtime entry at boot** (`slackCursors`, `cause: 'boot'` only — a
|
|
230
|
+
`suspend` entry has none, so it starts at `from`). The live cursors in
|
|
196
231
|
`data/state/slack-cursors.json` are a tail pointer — one message after recovery pushes them
|
|
197
232
|
past the whole backlog — so they are never used as the floor.
|
|
198
233
|
- **Which channels:** every channel with a cursor (at the time of the outage or since), plus
|
|
@@ -123,12 +123,14 @@ Schedules sync across all envs but only **execute** where `DATA_SYNC_SCHEDULER_A
|
|
|
123
123
|
Two layers cause you to run — know both, and which to reach for:
|
|
124
124
|
|
|
125
125
|
**Schedules (`schedules.json`)** — a schedule is `trigger` + `task` (`prompt`/`bash`/`job`). One execution path, two trigger families:
|
|
126
|
-
- **Time**: `cron` / `interval` / `once`. A window missed while you were down
|
|
126
|
+
- **Time**: `cron` / `interval` / `once`. A window missed while you were down — or while the host
|
|
127
|
+
was **suspended** and you were merely frozen — obeys `onMissed`
|
|
127
128
|
(`run` default / `skip` / `offer`) plus a hard 6h staleness ceiling that overrides *every*
|
|
128
129
|
policy — so an 08:00 job never silently replays at 22:00. Anything not replayed is recorded as
|
|
129
130
|
`missedRun` on the schedule (visible in `GET /api/schedules`) for an on-demand run. See the
|
|
130
131
|
**scheduler** skill.
|
|
131
|
-
- **Asked "what did I miss?"** (after a power cut
|
|
132
|
+
- **Asked "what did I miss?"** (after a power cut, or a laptop **suspend** — a slept host is
|
|
133
|
+
caught too, by a late heartbeat tick, `cause: 'suspend'`): `GET /api/downtime`. It
|
|
132
134
|
joins the recorded outage ranges to those `missedRun` windows and to the Slack messages that
|
|
133
135
|
arrived while you were down, and it **reports and proposes only** — run a missed window with
|
|
134
136
|
an explicit `POST /api/schedules/{id}/run`. Never replay everything you missed; a 14h-late
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "shraga",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.34",
|
|
4
4
|
"description": "The teammate you delegate coding to — a self-hostable, multi-user AI coding agent web UI (Claude Code, with a pluggable engine seam).",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./src/index.ts",
|
package/src/server/boot.ts
CHANGED
|
@@ -2016,6 +2016,20 @@ try {
|
|
|
2016
2016
|
|
|
2017
2017
|
const PORT = Number(process.env.PORT) || 3032;
|
|
2018
2018
|
await new Promise<void>((resolve) => {
|
|
2019
|
+
// A listen failure MUST kill the process. Without this the error reaches the `uncaughtException`
|
|
2020
|
+
// handler above, which by design keeps us alive — so the promise never settles and we sit there
|
|
2021
|
+
// forever: process running, port unbound, nothing served. The service manager sees a healthy job
|
|
2022
|
+
// and a port watchdog sees a dead one, so it kickstarts on a loop and shreds in-flight runs.
|
|
2023
|
+
// EADDRINUSE is the common case: `kickstart -k` starts the replacement while the old process is
|
|
2024
|
+
// still draining (up to 90s). Exiting non-zero is the correct answer — the manager restarts us,
|
|
2025
|
+
// and by then the port is free.
|
|
2026
|
+
server.once('error', (err: NodeJS.ErrnoException) => {
|
|
2027
|
+
const why = err.code === 'EADDRINUSE'
|
|
2028
|
+
? `port ${PORT} is already in use (previous instance still draining?)`
|
|
2029
|
+
: (err.message ?? String(err));
|
|
2030
|
+
console.error(`[server] FATAL: cannot listen on ${PORT} — ${why}. Exiting so the service manager restarts us.`);
|
|
2031
|
+
process.exit(1);
|
|
2032
|
+
});
|
|
2019
2033
|
server.listen(PORT, () => {
|
|
2020
2034
|
console.log(`[server] Running on http://0.0.0.0:${PORT}`);
|
|
2021
2035
|
resolve();
|
package/src/server/downtime.ts
CHANGED
|
@@ -7,7 +7,8 @@
|
|
|
7
7
|
* 22:00; this module supplies the other half — the ledger and the on-demand report.
|
|
8
8
|
*
|
|
9
9
|
* Deliberately inert: nothing here starts a run, answers a Slack message, or mutates a schedule.
|
|
10
|
-
* The only writers are the heartbeat, the boot gap
|
|
10
|
+
* The only writers are the heartbeat, the gap ledger (boot gap + late-tick/suspend gap), and the
|
|
11
|
+
* Slack last-seen cursor.
|
|
11
12
|
* Acting on a finding is always an explicit follow-up (`POST /api/schedules/:id/run`).
|
|
12
13
|
*/
|
|
13
14
|
import { readFileSync, writeFileSync, existsSync, mkdirSync, renameSync } from 'node:fs';
|
|
@@ -42,6 +43,18 @@ export const DOWNTIME_THRESHOLD_MS = Number(process.env.DOWNTIME_THRESHOLD_MS ??
|
|
|
42
43
|
*/
|
|
43
44
|
export const DOWNTIME_HISTORY_MAX = 20;
|
|
44
45
|
|
|
46
|
+
/**
|
|
47
|
+
* How long a `suspend` gap keeps colouring the report's `note`.
|
|
48
|
+
*
|
|
49
|
+
* A suspend is never retroactively scanned — no restart happens, so no catch-up ever covers it, and
|
|
50
|
+
* a later `boot` entry does not make it honest. The warning therefore cannot be tied to "is the
|
|
51
|
+
* newest entry a suspend?"; it has to age out on its own. 24h is the horizon of the manual check
|
|
52
|
+
* the note asks for ("which schedule windows fell in the gap?"): nearly every schedule here is
|
|
53
|
+
* daily or tighter, so after a day the same window has come round again and run normally, and the
|
|
54
|
+
* gap is no longer the thing to look at.
|
|
55
|
+
*/
|
|
56
|
+
export const SUSPEND_NOTE_MAX_AGE_MS = Number(process.env.SUSPEND_NOTE_MAX_AGE_MS ?? 24 * 60 * 60 * 1000);
|
|
57
|
+
|
|
45
58
|
export interface DowntimeEntry {
|
|
46
59
|
/** Last proven-alive moment (the final heartbeat before the gap). */
|
|
47
60
|
from: number;
|
|
@@ -49,10 +62,26 @@ export interface DowntimeEntry {
|
|
|
49
62
|
to: number;
|
|
50
63
|
ms: number;
|
|
51
64
|
/**
|
|
52
|
-
*
|
|
53
|
-
*
|
|
54
|
-
*
|
|
55
|
-
*
|
|
65
|
+
* How the gap was noticed. `boot` = the process died (crash, power cut, deploy) and the gap was
|
|
66
|
+
* measured at startup. `suspend` = the process never died — the host slept (lid closed, standby)
|
|
67
|
+
* and a heartbeat tick came back late by more than the threshold. Both are real outages; the
|
|
68
|
+
* distinction matters to the reader, because a `suspend` gap means the process was FROZEN, so
|
|
69
|
+
* nothing at all ran at boot afterwards (no scheduler catch-up scan — see `missedSchedules`).
|
|
70
|
+
* Absent on entries written before this field existed.
|
|
71
|
+
*/
|
|
72
|
+
cause?: 'boot' | 'suspend';
|
|
73
|
+
/**
|
|
74
|
+
* channelId → last Slack ts we had seen when this outage was recorded. Present on `boot` entries
|
|
75
|
+
* ONLY, where it is snapshotted before any live traffic can move the cursors (`recordBootGap()`
|
|
76
|
+
* runs before the Slack ingress mounts — see boot.ts). The live cursor is a tail pointer: one
|
|
77
|
+
* normal message after recovery pushes it past the entire backlog, so a snapshot taken after
|
|
78
|
+
* traffic resumed would hide the outage rather than bound it.
|
|
79
|
+
*
|
|
80
|
+
* Absent on `suspend` entries by design: that gap is recorded from a heartbeat tick, up to
|
|
81
|
+
* HEARTBEAT_INTERVAL_MS after the wake — exactly when the reconnected socket delivers the
|
|
82
|
+
* backlog — so the cursor is no longer provably pre-gap. With no snapshot the backfill floors at
|
|
83
|
+
* the outage start (`from`), which can re-report a few already-seen messages but can never skip
|
|
84
|
+
* unseen ones.
|
|
56
85
|
*/
|
|
57
86
|
slackCursors?: Record<string, string>;
|
|
58
87
|
}
|
|
@@ -90,11 +119,19 @@ export function writeHeartbeat(at = Date.now()): void {
|
|
|
90
119
|
writeJsonAtomic(HEARTBEAT_FILE, { at, pid: process.pid });
|
|
91
120
|
}
|
|
92
121
|
|
|
93
|
-
/**
|
|
122
|
+
/**
|
|
123
|
+
* Stamp liveness every interval, and NOTICE when a tick comes back late.
|
|
124
|
+
*
|
|
125
|
+
* A boot-time check only catches an outage that killed the process. The common failure on this box
|
|
126
|
+
* is the opposite: a MacBook that sleeps with the lid closed. The process is frozen, not killed —
|
|
127
|
+
* it never restarts, so `recordBootGap()` never runs, and a 2.5h outage was completely invisible.
|
|
128
|
+
* A late tick is the one signal that survives a suspend, because the timer resumes on wake and the
|
|
129
|
+
* wall clock has moved on. Unref'd — a heartbeat must never hold the process open.
|
|
130
|
+
*/
|
|
94
131
|
export function startHeartbeat(intervalMs = HEARTBEAT_INTERVAL_MS): () => void {
|
|
95
132
|
writeHeartbeat();
|
|
96
133
|
const timer = setInterval(() => {
|
|
97
|
-
try {
|
|
134
|
+
try { recordTickGap(); } catch (err) { console.error('[downtime] heartbeat write failed:', err); }
|
|
98
135
|
}, intervalMs);
|
|
99
136
|
timer.unref?.();
|
|
100
137
|
return () => clearInterval(timer);
|
|
@@ -106,6 +143,37 @@ export function listDowntime(): DowntimeEntry[] {
|
|
|
106
143
|
return readJson<DowntimeFile>(DOWNTIME_FILE, { entries: [] }).entries ?? [];
|
|
107
144
|
}
|
|
108
145
|
|
|
146
|
+
/**
|
|
147
|
+
* The one gap-recording path, shared by the boot check and the late-tick check.
|
|
148
|
+
*
|
|
149
|
+
* Both ask the same question — "the last proven-alive moment is `last`, it is now `now`, is that a
|
|
150
|
+
* hole?" — so they share the same threshold, the same bounds, and the same Slack-cursor snapshot.
|
|
151
|
+
* Writing the heartbeat forward is the caller's job and happens either way (this process is alive
|
|
152
|
+
* NOW regardless of the verdict), which is also what makes an outage record exactly ONCE: the next
|
|
153
|
+
* comparison starts from `now`, not from the stale pre-gap stamp.
|
|
154
|
+
*
|
|
155
|
+
* Never fires anything. It appends a row to a JSON file and logs — that is the whole contract.
|
|
156
|
+
*/
|
|
157
|
+
function recordGap(last: number, now: number, cause: 'boot' | 'suspend'): DowntimeEntry | null {
|
|
158
|
+
const ms = now - last;
|
|
159
|
+
// `<=` also covers a backwards clock jump (negative ms): degrade to "no outage", never invent one.
|
|
160
|
+
if (ms <= DOWNTIME_THRESHOLD_MS) return null;
|
|
161
|
+
|
|
162
|
+
// Freeze the Slack cursors into the entry — but only on the boot path, where they are still
|
|
163
|
+
// provably pre-gap (recordBootGap runs before the Slack ingress mounts). On the suspend path the
|
|
164
|
+
// socket has been back for up to a heartbeat interval and may already have pushed the cursor past
|
|
165
|
+
// the whole backlog; recording that would be worse than recording nothing, so the entry carries
|
|
166
|
+
// no snapshot and the backfill floors at the gap start instead. See DowntimeEntry.slackCursors.
|
|
167
|
+
const snapshot: Record<string, string> = {};
|
|
168
|
+
if (cause === 'boot') for (const [channel, cursor] of Object.entries(listSlackCursors())) snapshot[channel] = cursor.ts;
|
|
169
|
+
|
|
170
|
+
const entry: DowntimeEntry = { from: last, to: now, ms, cause, ...(Object.keys(snapshot).length ? { slackCursors: snapshot } : {}) };
|
|
171
|
+
const entries = [...listDowntime(), entry].slice(-DOWNTIME_HISTORY_MAX);
|
|
172
|
+
writeJsonAtomic(DOWNTIME_FILE, { entries } satisfies DowntimeFile);
|
|
173
|
+
console.log(`[downtime] ${cause} gap of ${Math.round(ms / 60_000)}m recorded — down from ${new Date(last).toISOString()} to ${new Date(now).toISOString()}`);
|
|
174
|
+
return entry;
|
|
175
|
+
}
|
|
176
|
+
|
|
109
177
|
/**
|
|
110
178
|
* Compare the last heartbeat against boot time and, if the gap is real, record it. Returns the
|
|
111
179
|
* entry it recorded, or null — a clean restart (and a first-ever boot, which has no heartbeat to
|
|
@@ -116,19 +184,26 @@ export function recordBootGap(bootTime = Date.now()): DowntimeEntry | null {
|
|
|
116
184
|
// Write the new heartbeat regardless: whatever we conclude, this process is alive now.
|
|
117
185
|
writeHeartbeat(bootTime);
|
|
118
186
|
if (last === null) return null;
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
// Freeze the Slack cursors into the entry now, at boot, before the first inbound message can
|
|
123
|
-
// advance them past the backlog this outage left behind.
|
|
124
|
-
const snapshot: Record<string, string> = {};
|
|
125
|
-
for (const [channel, cursor] of Object.entries(listSlackCursors())) snapshot[channel] = cursor.ts;
|
|
187
|
+
return recordGap(last, bootTime, 'boot');
|
|
188
|
+
}
|
|
126
189
|
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
190
|
+
/**
|
|
191
|
+
* One heartbeat tick: stamp liveness, and record an outage if the tick is late past the threshold.
|
|
192
|
+
*
|
|
193
|
+
* The reference point is the PERSISTED heartbeat — the same value `recordBootGap()` measures from,
|
|
194
|
+
* so boot and suspend can never disagree about where the last proven-alive moment was, and can
|
|
195
|
+
* never double-record one outage: whichever check sees the gap first advances the stamp, and the
|
|
196
|
+
* other then measures from the new one.
|
|
197
|
+
*
|
|
198
|
+
* A long suspend yields ONE entry, not one per missed interval, because a frozen process fires no
|
|
199
|
+
* timers while it sleeps — `setInterval` does not accumulate a backlog of missed ticks — and even
|
|
200
|
+
* if it did, this writes the heartbeat forward before the next tick can compare.
|
|
201
|
+
*/
|
|
202
|
+
export function recordTickGap(now = Date.now()): DowntimeEntry | null {
|
|
203
|
+
const last = readHeartbeat();
|
|
204
|
+
writeHeartbeat(now);
|
|
205
|
+
if (last === null) return null;
|
|
206
|
+
return recordGap(last, now, 'suspend');
|
|
132
207
|
}
|
|
133
208
|
|
|
134
209
|
// ── Slack last-seen cursors ───────────────────────────────────────────────────────────────────
|
|
@@ -354,6 +429,16 @@ export async function buildReport(opts: { slack?: boolean; backfill?: BackfillOp
|
|
|
354
429
|
missedSchedules: missedSchedules(entries),
|
|
355
430
|
note: 'Report only — nothing here has been run, answered, or replayed. Act on an item explicitly.',
|
|
356
431
|
};
|
|
432
|
+
// A suspend gap froze the process instead of killing it, so the scheduler's boot-time catch-up
|
|
433
|
+
// scan never ran. The late timer fire on wake DOES record the windows it refuses, so the list is
|
|
434
|
+
// not empty — it is INCOMPLETE: it holds only what that fire judged. Say that, and say it for
|
|
435
|
+
// every recent suspend, not just the newest entry: a restart afterwards appends a `boot` entry
|
|
436
|
+
// but scans nothing retroactively, so gating on `last` would silently drop the warning.
|
|
437
|
+
const suspends = entries.filter(e => e.cause === 'suspend' && now - e.to <= SUSPEND_NOTE_MAX_AGE_MS);
|
|
438
|
+
if (suspends.length) {
|
|
439
|
+
const when = suspends.map(e => `${new Date(e.from).toISOString()}→${new Date(e.to).toISOString()}`).join(', ');
|
|
440
|
+
report.note += ` NOTE: a recent outage was a host SUSPEND (the process was frozen, not restarted: ${when}), so no scheduler catch-up scan ran. missedSchedules is INCOMPLETE for that gap — it lists only the windows the late timer fire itself judged on wake; any other window that elapsed inside the gap left no record at all. Check schedules whose window falls in the gap by hand.`;
|
|
441
|
+
}
|
|
357
442
|
if (opts.slack !== false) {
|
|
358
443
|
report.slack = last
|
|
359
444
|
// The snapshot frozen at boot, not the live cursors — see DowntimeEntry.slackCursors.
|
|
@@ -111,22 +111,10 @@ export function start(broadcast: Broadcast): void {
|
|
|
111
111
|
continue;
|
|
112
112
|
}
|
|
113
113
|
|
|
114
|
-
const
|
|
115
|
-
if (
|
|
116
|
-
console.log(`[scheduler]
|
|
117
|
-
noteMissed(s, prev,
|
|
118
|
-
continue;
|
|
119
|
-
}
|
|
120
|
-
if (policy === 'offer') {
|
|
121
|
-
console.log(`[scheduler] missed window ${new Date(prev).toISOString()} for ${s.id} — onMissed=offer, recorded for on-demand run`);
|
|
122
|
-
noteMissed(s, prev, 'offer');
|
|
123
|
-
continue;
|
|
124
|
-
}
|
|
125
|
-
const age = Date.now() - prev;
|
|
126
|
-
const grace = maxMissedAgeMs();
|
|
127
|
-
if (age > grace) {
|
|
128
|
-
console.log(`[scheduler] missed window ${new Date(prev).toISOString()} for ${s.id} is ${Math.round(age / 60_000)}m stale (grace ${Math.round(grace / 60_000)}m) — not replaying`);
|
|
129
|
-
noteMissed(s, prev, 'stale');
|
|
114
|
+
const verdict = judgeMissed(s, prev);
|
|
115
|
+
if (!verdict.replay) {
|
|
116
|
+
console.log(`[scheduler] not replaying ${s.id} — ${verdict.message}`);
|
|
117
|
+
noteMissed(s, prev, verdict.reason);
|
|
130
118
|
continue;
|
|
131
119
|
}
|
|
132
120
|
catchUps.push({ id: s.id, window: prev });
|
|
@@ -347,20 +335,11 @@ export function resumeRun(scheduleId: string, sessionId: string, prompt: string)
|
|
|
347
335
|
console.log(`[scheduler] not resuming ${scheduleId} — ${msg}`);
|
|
348
336
|
return refuse('already-completed', msg);
|
|
349
337
|
}
|
|
350
|
-
const
|
|
351
|
-
if (
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
return refuse(policy === 'skip' ? 'policy-skip' : 'policy-offer', msg);
|
|
356
|
-
}
|
|
357
|
-
const age = Date.now() - window;
|
|
358
|
-
const grace = maxMissedAgeMs();
|
|
359
|
-
if (age > grace) {
|
|
360
|
-
const msg = `window ${new Date(window).toISOString()} is ${Math.round(age / 60_000)}m stale (ceiling ${Math.round(grace / 60_000)}m)`;
|
|
361
|
-
console.log(`[scheduler] not resuming ${scheduleId} — ${msg}`);
|
|
362
|
-
noteMissed(s, window, 'stale');
|
|
363
|
-
return refuse('stale', msg);
|
|
338
|
+
const verdict = judgeMissed(s, window);
|
|
339
|
+
if (!verdict.replay) {
|
|
340
|
+
console.log(`[scheduler] not resuming ${scheduleId} — ${verdict.message}`);
|
|
341
|
+
noteMissed(s, window, verdict.reason);
|
|
342
|
+
return refuse(verdict.refusal, verdict.message);
|
|
364
343
|
}
|
|
365
344
|
}
|
|
366
345
|
console.log(`[scheduler] resuming ${scheduleId} in-place on session ${sessionId.slice(0, 30)}…`);
|
|
@@ -423,6 +402,32 @@ function missedPolicy(s: Schedule): MissedPolicy {
|
|
|
423
402
|
return s.onMissed ?? 'run';
|
|
424
403
|
}
|
|
425
404
|
|
|
405
|
+
type MissedVerdict =
|
|
406
|
+
| { replay: true }
|
|
407
|
+
| { replay: false; reason: 'skip' | 'offer' | 'stale'; refusal: 'policy-skip' | 'policy-offer' | 'stale'; message: string };
|
|
408
|
+
|
|
409
|
+
/**
|
|
410
|
+
* The one decision behind "this window already elapsed — may it still run?", shared by all three
|
|
411
|
+
* replay paths: the boot catch-up scan, the in-place resume, and the late timer fire.
|
|
412
|
+
*
|
|
413
|
+
* `onMissed` first (an explicit skip/offer is a standing instruction, not an age question), then
|
|
414
|
+
* the absolute staleness ceiling, which overrides even `run`. Callers do the recording, because
|
|
415
|
+
* only they know how to refuse (a return value, or `continue`).
|
|
416
|
+
*/
|
|
417
|
+
function judgeMissed(s: Schedule, window: number, now: number = Date.now()): MissedVerdict {
|
|
418
|
+
const when = new Date(window).toISOString();
|
|
419
|
+
const policy = missedPolicy(s);
|
|
420
|
+
if (policy !== 'run') {
|
|
421
|
+
return { replay: false, reason: policy, refusal: policy === 'skip' ? 'policy-skip' : 'policy-offer', message: `window ${when} was missed and onMissed=${policy}` };
|
|
422
|
+
}
|
|
423
|
+
const age = now - window;
|
|
424
|
+
const grace = maxMissedAgeMs();
|
|
425
|
+
if (age > grace) {
|
|
426
|
+
return { replay: false, reason: 'stale', refusal: 'stale', message: `window ${when} is ${Math.round(age / 60_000)}m stale (ceiling ${Math.round(grace / 60_000)}m)` };
|
|
427
|
+
}
|
|
428
|
+
return { replay: true };
|
|
429
|
+
}
|
|
430
|
+
|
|
426
431
|
function noteMissed(s: Schedule, at: number, reason: 'skip' | 'offer' | 'stale'): void {
|
|
427
432
|
s.missedRun = { at, reason, noticedAt: Date.now() };
|
|
428
433
|
saveSchedules(state.schedules);
|
|
@@ -455,7 +460,24 @@ function fireDue(): void {
|
|
|
455
460
|
const now = Date.now();
|
|
456
461
|
for (const s of state.schedules) {
|
|
457
462
|
if (!s.enabled || s.nextRun === undefined) continue;
|
|
458
|
-
if (s.nextRun
|
|
463
|
+
if (s.nextRun > now) continue;
|
|
464
|
+
// A timer fire is normally punctual, so it is NOT treated as a missed window — `onMissed`
|
|
465
|
+
// must never gate healthy operation. But a setTimeout does not survive a wall-clock jump:
|
|
466
|
+
// when the host sleeps (lid closed) or NTP steps the clock, the process is FROZEN, not
|
|
467
|
+
// killed — no restart, so the boot catch-up scan never runs — and the overdue timer fires
|
|
468
|
+
// the elapsed window the moment the machine wakes. Firing an 08:00 job at 21:36 is exactly
|
|
469
|
+
// what the catch-up ceiling exists to prevent, reached by the one path it didn't cover.
|
|
470
|
+
// Past the ceiling we hand the window to the same judgement the other two paths use.
|
|
471
|
+
// Past the ceiling `judgeMissed` can only refuse (it replays a `run` window only while it is
|
|
472
|
+
// within the ceiling), so there is no replay branch here — the shared decision is used for WHAT
|
|
473
|
+
// to record, not whether to fire.
|
|
474
|
+
if (now - s.nextRun > maxMissedAgeMs()) {
|
|
475
|
+
const verdict = judgeMissed(s, s.nextRun, now) as Extract<MissedVerdict, { replay: false }>;
|
|
476
|
+
console.log(`[scheduler] not firing ${s.id} — ${verdict.message} (timer fired late; host suspended or clock jumped)`);
|
|
477
|
+
noteMissed(s, s.nextRun, verdict.reason);
|
|
478
|
+
continue; // the advance loop below still re-arms / retires this schedule
|
|
479
|
+
}
|
|
480
|
+
enqueueFire(s, s.nextRun);
|
|
459
481
|
}
|
|
460
482
|
// Advance nextRun for recurring triggers; disable fired `once` triggers
|
|
461
483
|
for (const s of state.schedules) {
|