shraga 0.1.31 → 0.1.33

package/defaults/skills/mcp-server.md

@@ -56,6 +56,7 @@ Add to `~/Library/Application Support/Claude/claude_desktop_config.json` (uses s
 | `put_skills_write` | Create/update a skill |
 | `get_schedules` | List scheduled jobs |
 | `post_schedules_run` | Trigger a schedule run |
+| `get_downtime` | "What did I miss?" — outage ranges + missed schedule windows + Slack backfill (reports/proposes only, fires nothing) |
 | `post_chat` | Talk to the agent — conversational, multi-turn via sessionId |
 | `get_config` | Read agent configuration |
 

package/defaults/skills/scheduler.md

@@ -134,9 +134,113 @@ Task:
   { kind: "prompt", prompt: "<text>" }
   { kind: "bash", command: "<cmd>" }
 
-Schedule: { id, name, enabled, trigger, task, scope, createdBy, nextRun?, lastRun?, runCount }
+Schedule: { id, name, enabled, trigger, task, scope, createdBy, nextRun?, lastRun?, runCount,
+            onMissed?, missedRun? }
 ```
 
+## Missed windows (`onMissed` / `missedRun`)
+
+If a window elapses with nothing running (deploy, crash, power cut — or the host **suspending**
+with the process frozen), the scheduler decides whether to replay it. `onMissed` is settable on
+`POST`/`PUT`:
+
+| `onMissed` | Behaviour |
+|---|---|
+| `run` (default) | Replay the missed window — the pre-existing behaviour. |
+| `skip` | Never replay. The window is simply lost. |
+| `offer` | Don't auto-fire, but **record** it so a human/agent can run it on demand. |
+
+```bash
+curl -s -X PUT -H "Content-Type: application/json" -H "x-internal-token: $INTERNAL_API_TOKEN" \
+  http://localhost:$PORT/api/schedules/{id} -d '{ "onMissed": "offer" }' | jq .
+```
+
+- **A staleness ceiling overrides every policy, `run` included**: a window more than
+  `SCHEDULER_MAX_MISSED_AGE_MS` (default **6h**) late is never replayed. Finishing an 08:00 report
+  at 22:00 is not the job the schedule describes.
+- One decision (`judgeMissed` in `scheduler/engine.ts`) governs **all three** paths that could
+  replay a window:
+  1. **Boot catch-up** (cron only) — the process was down when the window passed.
+  2. **In-place resume** of a run interrupted mid-flight — every trigger kind
+     (`interval`/`once`/`event` have no catch-up, so resume is their only gate; their window is
+     the one the interrupted run recorded when it started).
+  3. **A late timer fire** — the process was *frozen*, not killed (laptop lid/standby, NTP step,
+     VM pause): same pid, no restart, so catch-up never ran, and the overdue `setTimeout` fires on
+     wake. The window is the schedule's `nextRun` (when it *should* have fired), not the wake time.
+     A punctual fire is untouched — `onMissed` is a missed-window policy, never a mute — the gate
+     engages only once the fire is past the ceiling.
+- After a refusal the schedule is still re-armed normally: an `interval` job loses exactly one
+  cycle and fires again next period; a `once` is retired (disabled, `nextRun` cleared) with its
+  `missedRun` left for an on-demand run. Nothing is left enabled-but-never-firing.
+- Whenever a window is not replayed, the schedule gets
+  `missedRun: { at, reason: "skip"|"offer"|"stale", noticedAt }` — visible on
+  `GET /api/schedules` and `GET /api/schedules/{id}`. **That is the `offer` affordance**: read it,
+  then `POST /api/schedules/{id}/run` to run the missed work on demand. It clears the moment the
+  schedule next starts a run.
+- `POST /api/schedules/{id}/run` returns **409** `{ error, reason }` when a run can't start
+  (`locked` — a run is already in flight, incl. one held across a restart; `already-completed` /
+  `already-attempted`). A `404` there really does mean the schedule doesn't exist.
+
+## "What did I miss?" — downtime recovery (`GET /api/downtime`)
+
+This box is deliberately not always-on. The server heartbeats to `data/state/heartbeat.json` every
+`HEARTBEAT_INTERVAL_MS` (default **60s**). A gap larger than `DOWNTIME_THRESHOLD_MS` (default
+**3 intervals = 3m**) is appended to `data/state/downtime.json` as
+`{ from, to, ms, cause, slackCursors? }` (last **20** kept). A clean restart records nothing.
+
+Two `cause`s, one code path:
+
+- **`cause: 'boot'`** — the process died (crash, power cut). The gap is measured at startup, from
+  the last persisted heartbeat.
+- **`cause: 'suspend'`** — the process never died: the **host slept** (lid closed, standby) and the
+  process was *frozen*. Nothing restarts, so a boot check would never see it — instead the next
+  heartbeat tick comes back late by more than the threshold and records the gap itself. One entry
+  per outage, however long (the tick writes the heartbeat forward before the next one compares).
+Two consequences of a `suspend` gap, both flagged in the report's `note`:
+
+- **`missedSchedules[]` is INCOMPLETE, not empty** — the boot-time catch-up scan never ran (nothing
+  restarted), so the only records are the ones the **late timer fire** wrote itself (see "Missed
+  windows" above: a fire past the ceiling is refused and recorded as `missedRun`, at a timestamp
+  *inside* the gap, which the report then joins). Windows still inside the ceiling on wake simply
+  ran; anything else that elapsed in the gap left **no record at all** — check those by hand. The
+  note is emitted for any suspend in the last `SUSPEND_NOTE_MAX_AGE_MS` (**24h**), not just the
+  newest entry: a restart afterwards appends a `boot` entry but rescans nothing.
+- **No `slackCursors` snapshot** — the snapshot is only taken on the `boot` path, where it is
+  provably older than any post-recovery traffic. A suspend gap is recorded from a heartbeat tick up
+  to 60s *after* the wake, by which time the reconnected socket may already have pushed the cursor
+  past the whole backlog. So the entry carries none and the backfill floors at the gap's `from` —
+  it may re-report a few already-seen messages, never skip unseen ones.
+
+```bash
+curl -s -H "x-internal-token: $INTERNAL_API_TOKEN" "http://localhost:$PORT/api/downtime" | jq .
+curl -s -H "x-internal-token: $INTERNAL_API_TOKEN" "http://localhost:$PORT/api/downtime?slack=0" | jq .   # skip the Slack read
+```
+
+Returns `{ heartbeat, downtime[], lastDowntime, missedSchedules[], slack }`:
+
+- **`missedSchedules[]`** — the `missedRun` records above, joined to the outage that covers them,
+  each with a `proposal` string. It does **not** re-derive which windows were missed; the
+  scheduler owns that.
+- **`slack`** — messages that arrived during the last outage, fetched **on demand only** (never at
+  boot) via `conversations.history`, deduped by `client_msg_id`. History, not event replay: Slack's
+  Events API retries a dropped delivery for only ~30 minutes then discards it forever, while
+  `conversations.history` stays readable for days.
+  - **Where it starts:** the outage's `from`, raised only by the last-seen cursor **as
+    snapshotted into the downtime entry at boot** (`slackCursors`, `cause: 'boot'` only — a
+    `suspend` entry has none, so it starts at `from`). The live cursors in
+    `data/state/slack-cursors.json` are a tail pointer — one message after recovery pushes them
+    past the whole backlog — so they are never used as the floor.
+  - **Which channels:** every channel with a cursor (at the time of the outage or since), plus
+    `SLACK_AGENT_CHANNEL` — so a first deploy, or a channel quiet before the outage, is not
+    invisible.
+  - **`channels[].truncated: true`** means the page cap (5 × 200 per channel) was hit with more
+    still waiting. `conversations.history` returns newest-first, so what's missing is the
+    **start** of the outage. The report's `note` also says `INCOMPLETE` — treat it as such.
+
+**It reports and proposes — nothing auto-fires.** Acting on an item is always an explicit second
+call: `POST /api/schedules/{id}/run` (409 + `reason` if it can't start). Same data is exposed as
+the MCP `get_downtime` tool when `MCP_ALL_TOOLS=true`.
+
 ## Emitting events (to fire event-triggered schedules)
 
 Any caller that can present shraga auth can push an event onto the bus:

package/defaults/skills/self-aware.md

@@ -123,7 +123,18 @@ Schedules sync across all envs but only **execute** where `DATA_SYNC_SCHEDULER_A
 Two layers cause you to run — know both, and which to reach for:
 
 **Schedules (`schedules.json`)** — a schedule is `trigger` + `task` (`prompt`/`bash`/`job`). One execution path, two trigger families:
-- **Time**: `cron` / `interval` / `once`.
+- **Time**: `cron` / `interval` / `once`. A window missed while you were down — or while the host
+  was **suspended** and you were merely frozen — obeys `onMissed`
+  (`run` default / `skip` / `offer`) plus a hard 6h staleness ceiling that overrides *every*
+  policy — so an 08:00 job never silently replays at 22:00. Anything not replayed is recorded as
+  `missedRun` on the schedule (visible in `GET /api/schedules`) for an on-demand run. See the
+  **scheduler** skill.
+  - **Asked "what did I miss?"** (after a power cut, or a laptop **suspend** — a slept host is
+    caught too, by a late heartbeat tick, `cause: 'suspend'`): `GET /api/downtime`. It
+    joins the recorded outage ranges to those `missedRun` windows and to the Slack messages that
+    arrived while you were down, and it **reports and proposes only** — run a missed window with
+    an explicit `POST /api/schedules/{id}/run`. Never replay everything you missed; a 14h-late
+    08:00 job is the incident this exists to prevent. Details: the **scheduler** skill.
 - **Event** (`{ kind:'event', source, match? }`) — fires when a matching event hits the event bus. `match` is an AND-filter of payload dot-paths → values. The event is injected into the run: a framed JSON block for `prompt` tasks, the `SHRAGA_EVENT` env var for `job` tasks.
 - Events arrive via `POST /api/events/:source` (auth-gated) or `ctx.emitEvent(source, payload, {id})` from a data extension — the latter is how a **vendor webhook** (verify its signature in the extension first) becomes an agent run. Bus + dispatcher: `src/server/events/`; fire path: `scheduler/engine.ts` `fireEvent()`. Full how-to (create / match / emit): the **scheduler** skill.
 - **Built-in lifecycle source**: the system auto-emits `schedule.finished` (`{ scheduleId, name, status, sessionId, sessionUrl?, error? }`) when a time/manual run completes — react to your own runs (e.g. `match: { status: "error" }` → notify). Event-triggered runs don't emit it (loop guard). More internal sources can be added with one `emitEvent()` at the milestone.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "shraga",
-  "version": "0.1.31",
+  "version": "0.1.33",
   "description": "The teammate you delegate coding to — a self-hostable, multi-user AI coding agent web UI (Claude Code, with a pluggable engine seam).",
   "type": "module",
   "main": "./src/index.ts",

package/src/server/boot.ts

@@ -41,6 +41,7 @@ import { registerModuleRoutes, reconcileInstalledModules } from './modules/index
 import { hydrateSlackUserToken } from './slack/oauth.ts';
 import { registerMcpOAuthRoutes } from './mcp-oauth.ts';
 import { registerEventRoutes } from './events/routes.ts';
+import { startHeartbeat, recordBootGap, buildReport } from './downtime.ts';
 import { registerWebhook } from './events/webhook.ts';
 import { startEventDispatcher } from './events/dispatcher.ts';
 import { seedOperators } from './contacts.ts';
@@ -390,6 +391,20 @@ function scheduleIfVisible(id: string, uid: string, isOwner = false): Schedule |
   return undefined;
 }
 
+// "What did I miss?" — the on-demand downtime report. Read-only and INERT: it returns outage
+// ranges, the windows phase 1 deliberately did not replay (`missedRun`), and the Slack messages
+// that arrived while we were down, each with a PROPOSAL. Running any of it is a separate,
+// explicit call (POST /api/schedules/:id/run). `?slack=0` skips the Slack read.
+app.get('/api/downtime', requireAuth, async (req, res) => {
+  try {
+    const slack = String(req.query.slack ?? '1') !== '0';
+    res.json(await buildReport({ slack }));
+  } catch (err) {
+    console.error('[downtime] report failed:', err);
+    res.status(500).json({ error: (err as Error).message });
+  }
+});
+
 app.get('/api/schedules', requireAuth, (req, res) => {
   const user = (req as any).user;
   const schedules = scheduler.listSchedules().filter((s) => user.isOwner || s.scope === 'system' || s.createdBy.uid === user.uid);
@@ -414,6 +429,7 @@ app.post('/api/schedules', requireAuth, (req, res) => {
     enabled: body.enabled ?? true,
     trigger: body.trigger as Schedule['trigger'],
     task: body.task as Schedule['task'],
+    onMissed: body.onMissed,
     scope: 'user',
     createdBy: { uid: user.uid, email: user.email },
     createdAt: now,
@@ -438,6 +454,7 @@ app.put('/api/schedules/:id', requireAuth, (req, res) => {
     enabled: body.enabled ?? existing.enabled,
     trigger: (body.trigger ?? existing.trigger) as Schedule['trigger'],
     task: (body.task ?? existing.task) as Schedule['task'],
+    onMissed: body.onMissed ?? existing.onMissed,
   };
   const result = scheduler.upsertSchedule(updated);
   if (!result.ok) return res.status(400).json({ error: result.error });
@@ -469,9 +486,15 @@ app.post('/api/schedules/:id/run', requireAuth, (req, res) => {
   const id = String(req.params.id);
   if (!scheduleIfVisible(id, user.uid, user.isOwner)) return res.status(404).json({ error: 'Not found' });
   const override = typeof req.body?.override === 'string' ? req.body.override.trim() || undefined : undefined;
-  const sessionId = scheduler.runNow(id, override);
-  if (!sessionId) return res.status(404).json({ error: 'Not found' });
-  res.json({ sessionId });
+  const outcome = scheduler.runNow(id, override);
+  // A refusal here is a CONFLICT with the schedule's current state (a run already holds the
+  // cross-restart lock, the period is already done), not a missing resource — 404 sent operators
+  // hunting for a schedule that plainly exists.
+  if (!outcome.ok) {
+    if (outcome.reason === 'unknown-schedule') return res.status(404).json({ error: 'Not found' });
+    return res.status(409).json({ error: outcome.message, reason: outcome.reason });
+  }
+  res.json({ sessionId: outcome.sessionId, queued: outcome.queued ?? false });
 });
 
 app.post('/api/schedules/:id/cancel', requireAuth, (req, res) => {
@@ -1042,6 +1065,13 @@ setBroadcaster(broadcast); // let session-bus push async events (e.g. an add-on'
 ensureWorkspaceDir();
 watchWorkspace((event) => broadcast({ type: 'workspace_change', ...event }));
 if (!PASSIVE) {
+  // BEFORE anything that could take time: the gap is measured as (now - last heartbeat), so it must
+  // be read and re-stamped while "now" still means boot — a slow start would otherwise inflate the
+  // gap, or (once startHeartbeat is running) erase it. Order vs the scheduler's catch-up doesn't
+  // matter for the join: missedSchedules() reads the two files independently, at report time.
+  // Passive twins never write here — single-active-writer, same rule as the scheduler.
+  recordBootGap();
+  startHeartbeat();
   scheduler.start(broadcast);
   startEventDispatcher();
   // Modules reconcile MUST follow scheduler.start(): upsertSchedule mutates the engine's
@@ -1102,6 +1132,8 @@ async function activateConsumers() {
   console.log('[server] ACTIVATING — starting consumers and background writers');
   await dataSync.init();
   syncVendorRepos().catch(err => console.warn('[vendor-sync] error:', (err as Error).message));
+  recordBootGap();
+  startHeartbeat();
   scheduler.start(broadcast);
   startEventDispatcher();
   mountFeatures({ app, requireAuth, broadcast, passive: false });
@@ -1806,7 +1838,22 @@ async function recoverInterruptedSessions() {
       });
       const resumePrompt = 'Your previous response was cut off by a server restart. The partial response has been preserved above. Continue from where you left off, and avoid repeating any side-effects (e.g. messages already sent) that may have completed before the interruption.';
       setRunStatus(s.sessionId, 'idle');
-      scheduler.resumeRun(s.scheduleId!, s.sessionId, resumePrompt);
+      const outcome = scheduler.resumeRun(s.scheduleId!, s.sessionId, resumePrompt);
+      if (!outcome.ok) {
+        // The refusal is now the EXPECTED outcome after a long outage (stale window / onMissed).
+        // Close the session out exactly like the not-resumable branch above — otherwise
+        // scheduleRunStatus stays 'running', the sidebar shows it busy forever, and it can never
+        // self-heal because getRunningSessions() filters on runStatus, which is already idle.
+        appendMessage(s.sessionId, {
+          id: crypto.randomUUID(),
+          role: 'assistant',
+          blocks: [{ type: 'error', text: `Not resumed after the server restart — ${outcome.message} (${outcome.reason}).` }],
+        });
+        updateScheduledSessionStatus(s.sessionId, 'error');
+        scheduler.clearRunningMarker(s.scheduleId!);
+        console.log(`[recovery] scheduler session ${s.sessionId.slice(0, 8)} (${s.scheduleId}) — resume refused (${outcome.reason}): ${outcome.message}`);
+        continue;
+      }
       console.log(`[recovery] resuming scheduler session ${s.sessionId.slice(0, 8)} (${s.scheduleId}) in-place`);
       continue;
     }