@turmind/halo 2.1.6 → 2.2.0

package/bundled-docs/design/cron.md

@@ -1,26 +1,25 @@
 # Cron — Scheduled Tasks Design
 
-Run an agent on a schedule (daily report, periodic broadcast) and fan the result out to channels. Jobs live in the global db; [croner](https://github.com/hexagon/croner) only computes their fire times, the runner's own timers fire them, and each fire executes as a fresh `halo cli` child process.
+Run an agent on a schedule (daily report, periodic broadcast) or on each incoming webhook, and fan the result out to channels. Jobs live in the global db; [croner](https://github.com/hexagon/croner) only computes their fire times, the runner's own timers fire them, and each fire runs **in the server process** as a turn in the job's session (since 2.2; before, a fresh `halo cli` child).
 
 ## Architecture
 
 ```
-cron_jobs (global db)                      registered at boot:
-      │                                    ┌── slack/cron-dispatcher
-      ▼  reloadAll / reconcileFromDb       ├── feishu/cron-dispatcher
-  runner.ts ──── runner timers ────┐       ├── telegram/cron-dispatcher
-      │                            │       ├── wechat/cron-dispatcher
-      │                            │       └── wecom/cron-dispatcher
-      │ fire                       │              ▲
-      ▼                            │              │ registerCronDispatcher
-  spawn `halo cli -s <session>`    │       dispatcher.ts (registry)
-      │ stdout                     │              ▲
-      ▼                            └──────────────┘ dispatchToTargets(rawText, targets, workspacePath)
-  cron_runs (audit row + log file) ───────────────┘
+cron_jobs (global db)   POST /cron-hooks/:id     registered at boot:
+      │                   → cron_events            ┌── slack/cron-dispatcher
+      ▼  reloadAll / reconcileFromDb │             ├── feishu/cron-dispatcher
+  runner.ts ── timers / startOrInject┤             ├── telegram/cron-dispatcher
+      │ fire                                       ├── wechat/cron-dispatcher
+      ▼                                            └── wecom/cron-dispatcher
+  SessionManager (job's session,                          ▲
+    reply_to = { cron: runId })                           │ registerCronDispatcher
+      │ tree quiet → relay CronRunDoor → finishRun  dispatcher.ts (registry)
+      ▼                                                   ▲
+  cron_runs (audit row; the session is the log) ──────────┘ dispatchToTargets(text, targets, workspacePath)
 ```
 
 Two halves, cleanly split:
-- **runner.ts** — the scheduler. Owns the fire timers (croner only computes fire times), spawns the cli, persists audit rows.
+- **runner.ts** — the scheduler. Owns the fire timers (croner only computes fire times), sends the fire into the job's session, wraps the run up, persists audit rows.
 - **dispatcher.ts** — a channel-agnostic registry. Knows how to look up a dispatcher by `channelType` and hand off; knows nothing about any specific channel.
 
 ## Scheduling model
@@ -30,12 +29,12 @@ Schedules are **durable** — the source of truth is the `cron_jobs` table; in-m
 The runner arms its own `setTimeout` toward each target in hops of at most 30 s; a hop that wakes early re-arms toward the **same** target, and the next target is computed (`nextRun(now)`) only after a fire — a sleep / clock jump over several fires runs once, no catch-up. Why not croner's own timers: in croner 10.0.1 a timer waking a fraction of a ms early re-plans from a fresh clock read, and when the clock crosses the fire instant mid-re-plan, an instance that has never fired (every job right after a restart, every one-shot) silently skips that fire — ~10% of first fires after a restart in production; the upstream fix is only in 11.0.0-dev (`test/cron-fire-race.test.ts`).
 
 - **Recurring**: standard cron expression (`job.schedule`), optional `timezone`, optional **active window** `[activeFrom, activeUntil)` (epoch ms, `active_from` / `active_until`, NULL = unbounded; `CRON_MIGRATIONS` slot 2). The window is passed to croner as native `startAt` / `stopAt` (`activeWindowOptions` in `runner.ts`; `startAt` is `activeFrom − 1 ms` because croner's startAt is exclusive), so fires outside it simply never happen — no `cron_runs` row. "Pause until X" = push `activeFrom` to X; there is no skip-list for a gap in the middle. Past `activeUntil` the job stays `enabled` (so it can be extended later) but `scheduleJob` creates no croner and logs `past activeUntil; not scheduled` once — the id goes into `_expired` so reconcile's "fingerprint cached but not in `_active`" retry branch treats it as handled instead of re-scheduling (and re-logging) every 10s. A schedule or timezone croner rejects is handled the same way: one `schedule invalid` warn per fingerprint (the timezone in it), retried once the row changes. REST POST / PUT validate both: a non-string `timezone` → 400 `timezone must be a string`; a zone `Intl.DateTimeFormat` rejects → 400 `invalid timezone: <value>` (croner resolves zones through the same Intl call, so the two accept the same set; unset / `null` / `''` = host timezone). Only an out-of-band write reaches this path, and croner throws on a bad timezone only at `nextRun()`. Known gap: `manage-cron.py` stores `--timezone` unchecked (it resolves the zone through ZoneInfo only to parse `--active-from` / `--active-until`), so a typo there lands in this warn-once fallback. That's deliberate: Windows has no system tz database, so without the `tzdata` package ZoneInfo would reject valid zones. The REST `nextRunAt` uses the same options, so the list shows the first post-pause fire; the admin badges a job "active from …" / "ended". Run-now ignores the window. Recurring only: a window together with `runAt` is rejected at every write surface (REST on the merged row, skill helper).
-- **One-shot**: `job.runAt` (epoch ms) — the runner's timer fires once at that instant (no croner). After it completes, `finalize` sets `enabled=0` so it never re-fires. A `runAt` already in the past at schedule time is marked `lastRunStatus='missed'` and disabled (rather than firing immediately or retrying every reconcile) — unless its own run is still in flight (`_inflight`, e.g. `POST /api/cron/reload` mid-run): `finalize` then records the real status. At boot, the orphan sweep already finalized a one-shot whose run it found (`enabled=0`, `failed`), so the reload never sees it as missed. The `run_at` column is `CRON_MIGRATIONS` slot 0 in `db/cron-db.ts`, run through the shared `runMigrations` / `PRAGMA user_version` mechanism (see [storage.md](storage.md#schema-change-rules)).
-- **Folder + stateless** (`CRON_MIGRATIONS` slot 4, v5): `cron_jobs.working_dir` — the folder the run focuses on, passed as `halo cli --dir` (same semantics as a sub-agent's `working_dir`: the folder's `.halo/INSTRUCTIONS.md` chain joins the prompt, plus a `Working directory:` line); stored workspace-relative, NULL = the workspace root; validated at every write surface (inside the workspace, existing — REST re-checks a stored folder when `workspacePath` moves). `cron_jobs.stateless` (0/1) — **new session every run**: each fire gets a fresh root session `cron-<id>-<base36 ms>` (`perRunSessionId`, same no-double-prefix rule as `effectiveSessionId`), so nothing accumulates across fires (each job keeps only its own newest 10 runs; per-run sessions are deleted along with their run rows, see [Retention](#retention)); mutually exclusive with `session_id` (`400` on the merged row). `cron_runs.session_id` — the session that run used, the only record of a stateless run's id (NULL on rows from before v5). `cron_runs.workspace_path` (slot 5, v6) — the workspace the run fired against (the job's at fire time), i.e. where a stateless run's session lives even after the job moves workspace or is deleted; v6 backfills it from the job row, leaving NULL on rows whose job was already gone.
+- **One-shot**: `job.runAt` (epoch ms) — the runner's timer fires once at that instant (no croner). After it completes, `finalize` sets `enabled=0` so it never re-fires. A `runAt` already in the past at schedule time is marked `lastRunStatus='missed'` and disabled (rather than firing immediately or retrying every reconcile) — unless its own run is still in flight (`_inflight`, e.g. `POST /api/cron/reload` mid-run): `finalize` then records the real status. At boot, a one-shot whose run was cut off is still in `_inflight` (recovery holds it until the run resumes or is failed — either way `finalize` sets `enabled=0`), so the reload never sees it as missed. The `run_at` column is `CRON_MIGRATIONS` slot 0 in `db/cron-db.ts`, run through the shared `runMigrations` / `PRAGMA user_version` mechanism (see [storage.md](storage.md#schema-change-rules)).
+- **Folder + stateless** (`CRON_MIGRATIONS` slot 4, v5): `cron_jobs.working_dir` — the folder the run focuses on, set as the session's `working_dir` (same semantics as a sub-agent's `working_dir`: the folder's `.halo/INSTRUCTIONS.md` chain joins the prompt, plus a `Working directory:` line); stored workspace-relative, NULL = the workspace root; validated at every write surface (inside the workspace, existing — REST re-checks a stored folder when `workspacePath` moves). `cron_jobs.stateless` (0/1) — **new session every run**: each fire gets a fresh root session `cron-<id>-<base36 ms>` (`perRunSessionId`, same no-double-prefix rule as `effectiveSessionId`), so nothing accumulates across fires (each job keeps only its own newest 10 runs; per-run sessions are deleted along with their run rows, see [Retention](#retention)); mutually exclusive with `session_id` (`400` on the merged row). `cron_runs.session_id` — the session that run used, the only record of a stateless run's id (NULL on rows from before v5). `cron_runs.workspace_path` (slot 5, v6) — the workspace the run fired against (the job's at fire time), i.e. where a stateless run's session lives even after the job moves workspace or is deleted; v6 backfills it from the job row, leaving NULL on rows whose job was already gone.
 
 ### Trigger-mode exclusivity (the db contract)
 
-Exactly **one** of `schedule` / `runAt` is set per job. `runner.ts` `scheduleJob` silently prefers `runAt` when both are present, and at-mode's post-fire auto-disable (`enabled=0`) would then kill the recurring schedule too — so the invariant is enforced at every write point, not just at create. `PUT /api/cron/jobs/:id` validates the **merged** row (partial PUT bodies make this non-obvious):
+Exactly **one** of `schedule` / `runAt` / `webhookSecret` is set per job (webhook: see [Webhook trigger](#webhook-trigger) — `webhook: true` switches to it, a schedule / runAt alone switches away, `400 schedule, runAt or webhook required` when none would be left). `runner.ts` `scheduleJob` silently prefers `runAt` when both are present, and at-mode's post-fire auto-disable (`enabled=0`) would then kill the recurring schedule too — so the invariant is enforced at every write point, not just at create. `PUT /api/cron/jobs/:id` validates the **merged** row (partial PUT bodies make this non-obvious):
 
 - Both sides in one body → `400 schedule and runAt are mutually exclusive`.
 - One side only → treated as a **mode switch**: the other side is implicitly cleared (`schedule` set ⇒ `runAt=null`; `runAt` set ⇒ `schedule=''`).
@@ -49,53 +48,77 @@ The skill helper `manage-cron.py update` follows the same rules (`--schedule` /
 REST route mutations call `scheduleJob` / `unscheduleJob` directly. But the `cron` skill (and manual ops) edit `cron.db` over a *different* sqlite connection. `reconcileFromDb` (10s timer) catches those:
 
 - Fast path: `PRAGMA data_version` flips only when *another* connection commits. If unchanged since last pass, skip the full select entirely — one pragma read (~µs) vs. a select-all + per-row fingerprint compare. The pragma (and the run-pruning `NOT IN (subquery)` below) needs the raw better-sqlite3 handle, which the runner gets from the shared `rawSqlite(db)` helper (`db/raw-sqlite.ts`) instead of re-spelling drizzle's `$client` / `.session.client` cast at each site — it did, three times, each with its own hand-written method shape.
-- On change: diff db rows against the in-memory `_fingerprint` map (`jobFingerprint`: enabled, schedule, runAt, timezone, activeFrom, activeUntil, agentId, userPrompt, workingDir, stateless); schedule new rows, unschedule deleted ones, re-instantiate croner only for rows whose fingerprint changed. Broadcasts a coalesced `cron:job_changed` per affected job.
+- On change: diff db rows against the in-memory `_fingerprint` map (`jobFingerprint`: enabled, schedule, runAt, webhook, timezone, activeFrom, activeUntil, agentId, userPrompt, workingDir, stateless); schedule new rows, unschedule deleted ones, re-instantiate croner only for rows whose fingerprint changed. Broadcasts a coalesced `cron:job_changed` per affected job.
 
 ## Execution (`runJob`)
 
-Each fire:
+A fire runs **in-process**: the prompt goes into the job's session on the server's own `SessionManager` for that workspace (reached through `setCronSessionRegistry(registry)`, called in `index.ts` next to `setRelayRegistry`), and the run ends when relay's turn-end hook sees that session's tree go quiet. Before 2.2 each fire spawned a `halo cli` child; that is gone, except for the boot-time reaping of a leftover child (see [Recovery](#recovery-on-boot-recoverruns)). One writer per session also removes the old double-write window on a picked session.
 
-1. **Concurrency guard**: resolve the job's session first (`effectiveSessionId` — `job.sessionId`, else the job's own session, see step 4; a stateless job's fresh `perRunSessionId`, which the two session-level checks never hit), then check three conditions in order; the first hit inserts a `cron_runs` row with `status='skipped'` and that `failureReason`, broadcasts, and bails without spawning. Applies to both scheduled fires and manual run-now clicks.
-   - Same `jobId` already running in this process (in-memory `_inflight` set) → `previous run still in progress`.
-   - Another job's cli already on the same session (`_inflightSessions`, keyed `realpath(workspace) + sessionId` so two spellings of one workspace collide) → `session <id> busy: another cron run is in progress`. Two jobs may pick the same session.
-   - The server itself is mid-turn anywhere in that session's tree (`hasActiveWorkInTree` on the workspace's live SessionManager, looked up with the registry's `peek` — cache-only, never builds a manager, whose constructor would run the boot reconcile as a side effect) → `session <id> busy: a turn is running in it`. In practice this is a picked session someone is chatting in.
+1. **Job guard**: same `jobId` already running in this process (`_inflight`) → a `skipped` row, `previous run still in progress`. Applies to scheduled fires and run-now.
+2. **Session guards** — the session is resolved first (`effectiveSessionId`: `job.sessionId`, else the job's own; a stateless job's fresh `perRunSessionId`, which these never hit), then `sessionBusy` on the workspace's manager (`registry.getOrCreate(ws)`):
+   - `reply_to` set — someone is owed this session's wrap-up → `session <id> busy: owed to <cron run X | relay from … | A2A task … | invocation …>`. `reply_to` lives in the workspace db, so this holds across restarts; it also replaces the old in-memory `_inflightSessions` for two jobs picking one session.
+   - a turn running anywhere in its tree (`hasActiveWorkInTree`) → `session <id> busy: a turn is running in it` (in practice someone chatting in a picked session).
 
-   Two overlapping writers would double-write the same on-disk session state; SessionManager's per-session lock is in-process and can't see across cli children, so these sets are the cheapest place to enforce serialization. They are in-memory, but their restart blind spot is covered: the boot-time orphan sweep (below) re-enters stale jobIds and session keys before the runner re-arms its timers. The runner reaches the server's managers through `setCronSessionRegistry(registry)` (called in `index.ts` next to `setRelayRegistry`); unset — tests, a bare CLI — means there are no server-side sessions to guard.
-2. Insert a `cron_runs` row with `status='running'` up front (UI sees it mid-flight), recording the session it runs in (`cron_runs.session_id`) and the job's current workspace (`cron_runs.workspace_path`; step 1's `skipped` row records the workspace too); broadcast `cron:run_changed`.
-3. Check `job.workspacePath` exists — fail loudly if the workspace was moved/deleted. Same for the job folder (`job.workingDir`, re-resolved with `resolveWorkingDir`): moved / deleted since the job was saved → `failed` with `folder not found: …`, nothing spawned.
-4. Spawn `halo cli -a <agent> -s <session> -w <workspace> --dir=<folder|.> [--access <level>]`, prompt on **stdin** (avoids Windows argv length limits), with a fixed `[Unattended cron run — nobody will answer questions …]` line stamped ahead of `user_prompt` on every fire (`UNATTENDED_PREAMBLE` in `runner.ts`) — the model can't see the `cron-*` session id, and prompts typed into the admin Cron form carry no such notice, so the runner adds it rather than trusting the prompt author. **Session strategy**: every job runs in one stable session — its own by default, or an existing **root** session picked on the job (`cron_jobs.session_id`, nullable; `CRON_MIGRATIONS` slot 1 in `db/cron-db.ts`). The job's own session id **is the job id** (generated ids are already `cron-<ts>-<rand>`; a custom id without that prefix gets `cron-<id>`) — `effectiveSessionId` in `runner.ts` is the one copy of the rule, and the REST list returns its result as `effectiveSessionId` so the admin never re-derives it. Older builds always prefixed, giving `cron-cron-…`; `CRON_MIGRATIONS` slot 3 (v4) pins every job that ever spawned a cli (`last_run_id` set, or a non-skipped run row — covers a first run in flight across the upgrade restart) to that session via `session_id`, so its history and the orphan sweep's `-s` fingerprint stay the same; jobs that never ran take the new rule. The cli's `-s` is create-on-missing, so the first fire creates the session with the job's agent; every later fire *resumes the same session*, so the conversation accumulates and is reviewable in the admin Sessions tab. A **stateless** job (`cron_jobs.stateless`) skips all of this: every fire creates a fresh session (`perRunSessionId`), recorded on the run row.
+   A timer / run-now fire is only ever skipped, never injected into a busy session: its folder can't change mid-turn, its timeout would stop the other turn, and the reply would be whose? Webhook fires differ — see [Webhook trigger](#webhook-trigger).
+3. Insert the `running` row (`cron_runs.session_id`, `workspace_path`), broadcast `cron:run_changed`. A missing workspace / folder (`resolveWorkingDir`) → `failed` (`workspace not found` / `folder not found: …`); a workspace db that won't open → `could not open workspace <ws>: …`.
+4. **Create or resume** the session. Missing → `createSession(job.agentId, null, 'Cron: <label|id>', undefined, sid, folderAbs, null[, title])` — access `null` = full; the display name resolves from agent.yaml; a webhook job's session is titled `Webhook: <label|id>` up front (its first message is a payload), a scheduled one takes its title from the first message. Existing with another agent → `failed` (`session X runs agent Y, not the job's Z; …`). Existing with the same agent → `setWorkingDir(sid, folderAbs|null)` re-points its folder in place (changing a job's folder moves its session too) — except a webhook fire on a busy session, which leaves it for a later run (see [Webhook trigger](#webhook-trigger)); its **access level is kept** — the send passes no level.
+5. **Re-check, then take the pointer** — past the awaits `sessionBusy` runs again, then in the same synchronous slice `takeReplyTo(db, sid, { cron: runId })`.
+6. **Track** (`trackRun`, before the send — the turn can end before `sendUserMessage` resolves): `_runs` entry, the root event listener (notices, interrupted replies), the timeout timer.
+7. **Send**: `appendUserMessage(sid, job.userPrompt)` (the UI row shows the prompt alone) and `sendUserMessage(sid, UNATTENDED_PREAMBLE + prompt, …, stamp)`. The `[Unattended cron run — nobody will answer questions …]` line is added on every fire: the model can't see the `cron-*` id, and prompts typed into the admin form carry no such notice. A send that throws → own pointer cleared, `failed`. `runJob` returns the runId here; it does not wait for the run.
 
-   **Agent lock.** The runner always passes `-a <job agent>`, and the cli refuses `-s <existing>` with a different explicit `-a` (a session's agent never changes). So once a stateful job's session exists, the job's agent is fixed to the session's: the REST create / update routes return `400 session <id> already runs agent <x>; …` when the **merged** row's agent differs from the session's (`sessionAgentConflict`; the skill helper exits with the same message). The check only runs when the body actually changes one of its inputs — agent, session, stateless or workspace, compared as stored values (the admin form resends every field) — so a job saved before the rule, or hand-edited into a mismatch, still takes unrelated edits such as `{enabled:false}`; disabling is always allowed. Such a legacy mismatch is not repaired: each of its runs fails with the cli's refusal on stderr as `failureReason`, until the agent is changed back, the session deleted, or the job switched to stateless (which has no lock). The list GET returns the session's agent as `sessionAgentId` (null: stateless / session not created yet), which the admin form locks the agent field on, or flags with an amber notice when it already differs.
+**Session strategy.** A stateful job runs in one stable session — its own by default, or an existing **root** session picked on the job (`cron_jobs.session_id`, `CRON_MIGRATIONS` slot 1). The job's own session id **is the job id** (generated ids are already `cron-<ts>-<rand>`; a custom id without that prefix gets `cron-<id>`) — `effectiveSessionId` in `runner.ts` is the one copy of the rule, returned by the REST list as `effectiveSessionId`. Older builds always prefixed (`cron-cron-…`); slot 3 (v4) pins every job that ever ran to that session via `session_id`. The conversation accumulates and is reviewable (live) in the Sessions tab. A **stateless** job (`cron_jobs.stateless`) gets a fresh session every fire.
 
-   **Folder.** `--dir=<folder>` is always passed, as **one** token (a folder starting with `-` would otherwise read as a flag), with `.` for the workspace root, so the cli applies it on create *and* re-points a resumed session in place (`SessionManager.setWorkingDir`) — changing a job's folder moves its existing session too. The log header records `dir=`.
+**Agent lock.** A session's agent never changes, so once a stateful job's session exists the job's agent is fixed to it: REST create / update return `400 session <id> already runs agent <x>; …` when the **merged** row's agent differs (`sessionAgentConflict`; the skill helper exits with the same message). Only checked when the body changes one of its inputs (agent, session, stateless, workspace — stored values compared, the admin form resends every field), so a legacy mismatch still takes unrelated edits; each of its runs fails at step 4 until fixed. The list GET returns the session's agent as `sessionAgentId`, which the admin form locks on.
 
-   **Access level.** For a session that already exists the runner reads its `access_level` from the workspace db (`readSessionAccessLevel`, null column = `full`) and passes it as `--access`: the cli defaults to `full` and persists any level that differs from the row, so without it one cron fire would permanently promote a `readonly` / `workspace` session. A session the cli is about to create gets no `--access` (full, as before). The log header records `session=` (with `(stateless)`) / `access=`.
+**Session lookup** (`readSessionRow`, behind `sessionAgentId` and the agent lock). It runs per job on every list GET and `cron:job_changed` push, so it never goes through `getWorkspaceDb` (realpath + scaffolding per call — a GET must not scaffold): one read-only better-sqlite3 handle per workspace path, rows cached and dropped whenever `PRAGMA data_version` moves. **Absent vs unreadable**: no `.halo/halo.db` or no row → null; any other failure throws `could not read session <id> from <ws>: …` — the REST write points return it as the `400` (the skill helper reads an unreadable db as no session); display reads it as null.
 
-   **Session lookup** (`readSessionRow`, shared by the access level, `sessionAgentId` and the agent lock). It runs per job on every list GET and `cron:job_changed` push, so it never goes through `getWorkspaceDb` (realpath + `ensureWorkspaceHalo` scaffolding per call — and a GET must not scaffold): one read-only better-sqlite3 handle per workspace path, with the rows cached and dropped whenever `PRAGMA data_version` moves. The handle never writes, so any commit by anyone — the cli child creating the session (`finalize` reads it right after), a Sessions-tab delete, another process — moves the version. **Absent vs unreadable**: no `.halo/halo.db` or no row → null ("no session yet"); any other failure (corrupt / locked db) throws `could not read session <id> from <ws>: …`, so a read error never passes for "no session" — which would spawn a restricted session at `full`, or let the agent lock through. At fire time the run is finalized `failed` with that message and nothing is spawned; the REST write points return it as the `400` (the skill helper reads an unreadable db as no session — the fire still refuses); display (GET `sessionAgentId`, `cron:job_changed`) reads it as null.
+### Wrap-up: `reply_to = { cron: runId }`
 
-   The child's pid is written back onto the `running` row (`cron_runs.pid`, nullable) — this is what makes the orphan sweep possible after a server crash. Note the **dual-writer layout** this creates: the cli child and the server share the workspace's session files. The server side is view-only for these sessions — admin subscribes seed a UIState from disk, and the WS detach-save skips it via the dirty gate (see [session.md](session.md#ws-disconnect-resilience)); without that gate a frozen server snapshot would overwrite the cli child's newer messages (the cron-session UI-log truncation incident). Live streaming of a cli-driven session into the admin is out of scope — events don't cross processes; the Sessions tab re-reads the file on open.
+The run's pointer is a `ReplyTo` branch next to relay / A2A / invocation (`agents/relay.ts`). relay reaches the runner through `CronRunDoor` (`setCronRunDoor({ owns, finish, cancel, released })`, set when `runner.ts` loads — relay can't import the runner, which imports relay); the CLI / TUI never load the runner, so there the door is null and a `{cron}` pointer is left as is.
 
-   A **picked** session breaks the view-only assumption: the server may hold a live in-memory copy (it was chatted in / opened before, and `ensureSession` keeps it resident), and its raw history predates the cli's turn — the next server-side save would write the cron turn away. So once the cli exits the runner re-`peek`s the workspace's manager (it may have been built mid-run) and calls `SessionManager.forgetExternalWrite(sessionId)`: if the tree is idle, it drops the root + `root>…` sessions from `sessions` **without saving** and drops the UIState, so the next view / message re-reads disk. If a turn started in the session while the cli ran (step 1 only covers turns already running at spawn), it returns false and nothing is dropped — both processes wrote the same files, the last save wins, and the runner logs a `console.warn` that one of the two turns may be missing. That window is the accepted cost of sharing a session with a job; the admin form hint and the `cron` skill both say so.
-5. Tee stdout + stderr live to `~/.halo/global/logs/cron/<runId>.log`; capture stdout to memory for dispatch. Since 1.1.5 the cli's stdout is the **final reply of the last root turn** as raw markdown (stdout is not a TTY, so no terminal styling; the interim text the model emits before tool calls and the wrap-ups of earlier drained turns are not included — see [session.md](session.md#message-queue-and-drain) "Final-text flag"). Before that it was every streamed text block of the whole run, which for a director session absorbing a dozen sub-agent reports reached ~40 KB and blew the wechat gateway ceiling.
-6. Enforce a **per-job timeout** (`job.timeoutSec ?? 3600`, the nullable `cron_jobs.timeout_sec` validated 60–21600 at every write surface; a change takes effect on the job's next fire, no croner re-instantiation needed) via a Node timer (cross-platform, not the Linux-only `timeout(1)`), with a **two-phase kill**: SIGTERM the direct child first — the cli has a SIGTERM handler that stops the harness, repairs + flushes the session, kills its own shell_exec process groups, and exits 143 — then, if it's still alive after a 30s grace, SIGKILL the entire descendant tree (POSIX ppid-walk over one `ps` snapshot; `taskkill /T /F` on Windows) so no grandchild survives to double-write the workspace on the next fire. A ppid walk, not a process-group kill: the cli sandbox spawns shell_exec workers `detached` into their own groups, so `kill(-pid)` would miss them. Timed-out runs still report exit code 124. Generous because overlap-of-the-same-job is already blocked up-front in step 1; this is just the long-stop reaper for a truly stuck child.
-7. Classify: exit 124 → `timeout`; non-zero → `failed`; zero but empty stdout → `failed` ("nothing to dispatch" — with the final-only stdout contract this means the last turn produced no wrap-up text, not that the agent never spoke); else `succeeded`.
-8. On success only, `dispatchToTargets(stdout, targets, job.workspacePath)` — the raw stdout goes down as-is; `MEDIA:` extraction happens per-target inside the dispatcher (see Dispatch model). If every target fails, downgrade to `failed`.
-9. `finalize`: write the terminal `cron_runs` row, update `cron_jobs.lastRun*`, disable one-shot jobs, broadcast. The `cron:job_changed` it sends carries `workingDir` / `stateless` / `sessionAgentId` (`jobChangedFields`, as do the REST mutations' and reconcile's) — `sessionAgentId` read after the run, so a first fire's push already has the agent of the session it just created. The `_inflight` and `_inflightSessions` entries are released in `finally`, regardless of outcome.
+- **`deliverRelayReport`** (the session's tree went quiet): `owns(runId)` = that row is `running` in **this process's** `cron.db` — a DEV server sharing workspace dbs with prod, or an AgentCore process, doesn't own it and leaves the pointer (`left for its owner`). Otherwise `finishRun`. `deliverRelayInterim` sends nothing for `{cron}` — final only.
+- **`finishRun`** — db-driven (run row, job row, targets read now; the `RunCtx` only adds notices, interrupted replies and the timer, so a run resumed after a restart finishes the same way). Synchronous up to the dispatch: `_finishing` claim, untrack, own pointer cleared. Text = what `halo cli` printed: root notices as `ℹ️ <n>` lines (not a sub-session's, not the drain-time sibling-status line, not an out-of-band `notifySystem` one — the evo ticker's auto-apply line landing mid-run is marked `outOfBand` and stays in chat only), a blank line, then the reply — `finalOutput || output`, after a webhook run's replies to turns its events cut short (`answers`, see below). `turnError` → `failed` (500 chars; a refusal on a stateful job appends a hint — delete the job's own session, or for a picked session pick another / clear the pick, or turn on "new session every run" — the refusal sticks to that history); empty → `failed` `run produced no output (nothing to dispatch)`; else `succeeded`, then `dispatchToTargets(text, targets, run workspace)` — every target failing downgrades to `failed`. `finalize` is guarded (`WHERE status='running'`): timeout, stop, finish and supersede race, the first wins. `test/cron-cli-parity.test.ts` replays each run's event stream through the cli's `runCli` and asserts the same text.
+- **Self-evolution**: a turn under a `{cron}` pointer never enqueues a pre-compact evo run (`enqueueEvoForCompactedTurn`), and a session that took webhook events enqueues later runs — pre-compact or `/evo` — with `source_access = 'webhook'`, so L2/L3 never auto-apply them (reason `source_webhook`; [evolution.md](evolution.md#auto-apply-l2--l3)). The evo ticker's auto-apply line (`notifySystem`, `outOfBand`) never joins a run's notices.
+- **Stop / delete / archive** of the session here: `cancelOwedForSession(db, sid, 'stop'|'delete'|'archive')` (the renamed `cancelA2AForSession`, same 4 call sites in `session-manager.ts`) → `cancelRun`: `failed` `session stopped/deleted/archived before the run finished`, nothing dispatched — a Stop no longer ships the partial output.
+- **Supersede**: `takeReplyTo` over a `{cron}` pointer (a relay / A2A taking the session) → the run `failed` `superseded by …`, the new pointer stays.
+- **Timeout** (`job.timeoutSec ?? 3600`, the nullable `cron_jobs.timeout_sec` validated 60–21600 at every write surface): outcome first — own pointer cleared, `timeout` `run exceeded <t>s timeout`, nothing dispatched — then `stopSession`, raced against 30 s (`STOP_SETTLE_MS`); one that doesn't settle (a tool ignoring its abort) logs `stopSession did not settle in 30s; the turn keeps running detached`, and later fires skip on the busy session. The abort reaches `shell_exec` as a process-group SIGTERM → SIGKILL (`spawnGroupExec`); bwrap adds `--die-with-parent`. The old SIGKILL-the-tree fallback is gone (accepted risk).
+- `exit_code` is NULL on every new row (status carries timeout); `log_path` NULL — the session is the log, live in the admin. The admin shows "Open session" for such runs; `logs/cron/` only ages out pre-2.2 files.
 
-The cli executable is resolved via `resolveHaloCli()` (`$HALO_CLI` override; `halo.cmd` on Windows to dodge the GUI `Halo.exe` on PATH).
+The `cron:job_changed` that `finalize` sends carries `workingDir` / `stateless` / `sessionAgentId` (`jobChangedFields`), read after the run.
 
-### Orphan sweep on boot (`sweepOrphanRuns`)
+## Recovery on boot (`recoverRuns`)
 
-The step-1 guard is per-process, so it has one blind spot: the server dies (SIGKILL, crash) while a cli child is mid-run. On POSIX the child is re-parented and survives; the restarted server has an empty `_inflight` and the runner re-arms the schedule — the next fire would double-write the orphan's session. Per the "persistent operations must not depend on in-memory state" rule, recovery is driven from the db: any `cron_runs` row still at `status='running'` at boot can only be a previous incarnation's leftover (this process hasn't run anything yet).
+Runs are recorded in the run ledger like any root session (`runs.db`; see [session.md](session.md#run-ledger--restart-nudge-for-interrupted-roots-haloglobalrunsdb)), and the `{cron}` pointer survives in the workspace db, so a run cut off by a restart resumes: the ledger's boot sweep nudges the session, and the resumed turn's end reaches `finishRun` through the door as usual.
 
-`startCronDaemon` calls `sweepOrphanRuns` **before** `reloadAll`. For each stale row:
+`startCronDaemon` calls `recoverRuns` (not awaited) before `reloadAll`; its first synchronous slice puts the jobId of every `running` row into `_inflight`, so a timer that fires right away only skips. Per row:
 
-1. Take the session the orphan was spawned on from the run row (`cron_runs.session_id` — the only source for a stateless job's per-run id; a row from before that column falls back to the job row's effective session, a job deleted since to the default rule on its id), then re-enter its jobId into `_inflight` and its session key into `_inflightSessions` first — the core invariant: while the orphan may be alive, no new cli for that job *or for any other job on that session* is spawned (a fire in that window is recorded as `skipped`, same as normal overlap).
-2. If the row has a pid (POSIX): **identity-check before killing** — `ps -p <pid> -o args=` must carry the session id as a whole argv token (the `-s <session>` value; whole-token so `cron-a` never matches `cron-ab`), so a recycled pid never kills an innocent process. If the session came from the job row (pre-v5 run row) and was re-pointed after the orphan was spawned, the fingerprint won't match and the orphan is left alone — the safe direction. Verified → SIGTERM (graceful path, same as timeout phase 1); still alive after a 30s async grace (`unref`ed timer, identity re-checked — pid reuse again) → SIGKILL the descendant tree.
-3. No pid / already dead / fingerprint mismatch / Windows: no kill, log only. Windows deliberately marks-only — there's no cheap command-line identity check (WMI is slow and sometimes policy-blocked), and a full-tree `taskkill /T /F` on a reused pid is irreversible while a double-written session is fixable; the orphan-survives-server-death window is also naturally smaller there.
-4. Either way the row is finalized as `failed` (`failureReason` notes the orphan disposition, `completedAt` stamped) so the UI stops showing a ghost `running` — for a one-shot its job row too, as `finalize` would (`enabled=0`, `lastRunStatus='failed'`, `lastRun*` = this run, `cron:job_changed` pushed); a recurring job's row is left alone — and the jobId / session key are released once the orphan is confirmed dead (or immediately when nothing was killable).
+- **In-process** (no pid): the workspace exists, the row has a session, the session's `reply_to` still names this run → wait for the ledger nudges of that workspace (`nudgesSettled`) → if the tree has work in flight, re-track the run with what's left of its timeout (`trackRun`; ≤ 0 times out right away) and redeliver its webhook events whose stamp never reached the model history (`rawHasStamp`, see below). Otherwise `failed` `interrupted by server restart (…)` — `not resumed: this server does not own the workspace runtime, or the session could not be woken` (a DEV server: ledger off), `before the run reached its session` (no session on the row), `workspace not found`; a pointer that no longer names the run (cut off before `takeReplyTo`, or superseded) → plain `interrupted by server restart`. A run whose resumed turn already ended during an await (this row's or an earlier row's) is left as recorded (`_finishing` / status re-checked) and its own end path settles its events and releases the job.
+- **Legacy** (pid set — a pre-2.2 `halo cli` child, which can outlive its server outside a systemd cgroup): reaped as before — `ps -p <pid> -o args=` must carry the session id as a whole token (pid reuse), SIGTERM, SIGKILL the descendant tree after a 30 s grace, job + session (`_inflightSessions`) blocked meanwhile; never killed on Windows. Marked failed regardless. Remove one minor release after 2.2.
 
-Only the kill grace is asynchronous; the boot-blocking part is one small select + row updates.
+A run not resumed hands its webhook events back to pending; at the end every job with pending events gets `startOrInject`.
+
+## Webhook trigger
+
+The third trigger mode (`CRON_MIGRATIONS` slot 6, v7): `cron_jobs.webhook_secret` non-null = a webhook job — no schedule, no `runAt`, no timer (`scheduleJob` parks it in `_expired`; `jobFingerprint` includes the flag). Excludes a picked session (picked sessions are often owed to relay / A2A; events would wait with no wake-up point) and an active window, at every write surface.
+
+**Receiver** — `POST /cron-hooks/:jobId` (`routes/cron-hooks.ts`, outside `/api`, public; mounted only when not AgentCore). Body ≤ 1 MB (413). Auth: GitHub `X-Hub-Signature-256` (HMAC-SHA256 of the raw body) or `Authorization: Bearer <secret>`, both `timingSafeEqual`; no job / not a webhook job / bad auth are one `404`. A GitHub `ping` → 200 before the enabled check (so creating the hook on a disabled job passes); disabled → 409. Delivery id = `X-GitHub-Delivery` / `Idempotency-Key` / `X-Request-Id`, else `sha256:<hex of body>#<receivedAt>` — matched by prefix within the last 10 minutes (`BODY_DEDUPE_MS`), so a header-less sender's retry burst is one event and the same body later a new one; the receipt time keeps it clear of `UNIQUE(job_id, delivery_id)`. A duplicate → 200 `{duplicate:true, eventId}` (insert `onConflictDoNothing` for a race) — except a repeat of a `failed` event: flipped back to `pending` (run / session / stamp cleared) and handled like a new one, 202 `{eventId, redelivered:true}`. More than 100 pending → 429. Stored → 202 `{eventId}`, then `startOrInject(jobId)` asynchronously.
+
+**`cron_events`** — `id, job_id, delivery_id, event_type, received_at, state, run_id, session_id, stamp, payload`; `state`: `pending` → `injected` (handed to `run_id`'s session, `stamp` = its UI row / raw turn key) → `done` | `failed`. Indexed by job and by run.
+
+**`startOrInject(jobId)`** — batching:
+- A run of the job is tracked here and still owns its session → claim every pending event and inject each as its own message (`appendUserMessage` + `stampEvents` + `sendUserMessage` with the stamp): folded into the running turn or queued, so one run reports a batch. An inject that throws puts that event back to pending. A manual run (run-now) counts: events arriving during it are injected into it, though its opening message carried the prompt alone.
+- Otherwise (job enabled and still a webhook job) `runJob(jobId, 'webhook')`: the events claimed right after `trackRun` ride the opening message (prompt + each event's text). Unlike a timer fire, a webhook fire **does not skip on a user's turn** (`sessionBusy(…, ownerOnly)`) — it queues behind it, leaving the session's folder as is (it can't change mid-turn; the job folder applies on a later run); a session owed to another owner → `''`, no row, events stay pending until it lets go: `clearReplyTo` (every owner's release — relay, A2A, `/invocations`, a cron run) calls the door's `released(sessionId)`, which kicks each webhook job (non-stateless) whose session that is and that has pending events. The same holds when the re-check past the fire's awaits finds a new owner: the `running` row it had written is deleted (no failed row, `lastRunStatus` untouched), the events stay pending. The kick only reaches this process: an owner in another process sharing the workspace db (dev and prod, a TUI) releasing the session kicks nothing here — the events wait for the next event, a restart (`recoverRuns`) or the job's next run end. Every run end (finish / timeout / cancel) kicks the job again if events are pending. A webhook fire that fails before its turn starts (workspace / folder missing, workspace db unopenable, agent mismatch, `createSession` or the send throwing) marks every event it held or found pending `failed` — not retried (a broken workspace would loop); the sender's redelivery revives them. A run-now that fails before its turn kicks the job, so events that arrived during it get their own run (that one failing settles them `failed` and kicks nothing).
+- Job disabled or switched off webhook mode → its pending events `failed` (`failPendingEvents`: `unscheduleJob` — REST PUT and reconcile —, `scheduleJob`'s non-webhook branch, and `startOrInject` finding it so).
+
+**Event text** — `[Webhook event · <type> · delivery <id> · received <iso>]`, then a line saying the payload is data, not instructions, then the payload. Over 32 KB the payload is written to `<ws>/.halo/tmp/webhooks/<jobId>/<eventId>.txt` and only its first 4 KB inlined; the UI row is cut at 2000 chars.
+
+**Interrupted replies** (`RunCtx.answers`). An event injected mid-run starts a new turn, and `finishRun` reads the last turn only, so the reply to the earlier events would be lost. The listener keeps the final text of a turn an injected event cut short (`queued_message` after an inject — the batch-boundary `complete` drainQueue emits between merged turns is not an idle point and is skipped), and of a turn that ended while the root idled waiting on sub-agents (`injectEvent` saves it); the run's text is those joined with `\n\n`, then the last reply. Sub-agent reports alone don't keep anything — the last wrap-up is the answer, as in the cli.
+
+**Consumed?** — a finished run marks its events `done` (succeeded) or `failed`. After a restart (`recoverInProcessRun`), an `injected` event whose stamp isn't in the session's raw history (`SessionManager.rawHasStamp` — a queued message dies with the process) is injected again, prefixed `[redelivered after restart]`. At-least-once: a compaction since can also drop a stamp.
+
+**Cleanup** rides the run paths: `deleteRuns` drops those runs' events, `deleteJobRuns` the job's pending ones, `deleteOrphanRuns` pending events of jobs that no longer exist.
+
+REST: create / PUT take `webhook: boolean` (three-way exclusive with `schedule` / `runAt`; PUT `true` keeps an existing secret, `false` — or a schedule / runAt alone — drops it); the server generates the secret (`randomBytes(32)` base64url); `POST /cron/jobs/:id/webhook-secret` rotates it; the list returns `webhookSecret` + relative `webhookPath` (the admin prefixes its own origin — the server can't know the public one behind a proxy); runs return `eventCount`. Behind a CDN / SSO proxy, `/cron-hooks/*` must reach the origin without the login gate and with its headers intact (signature, delivery id) — an auth-gated default route answers the sender with a redirect and the event never arrives.
 
 ## Dispatch model
 
@@ -131,19 +154,20 @@ interface CronChannelDispatcher {
   - **REST delete**: `DELETE /api/cron/jobs/:id` runs `deleteJobRuns` before deleting the job row.
   - **Skill delete**: `manage-cron.py delete` drops only the `cron_jobs` row. The server's next full `reconcileFromDb` pass then calls `deleteOrphanRuns` without awaiting it (failures are logged as warnings). That function handles every run whose `job_id` is no longer in `cron_jobs`, so rows and sessions go within about 10 s. A full pass runs after any other connection writes `cron.db`, and once on the first tick after boot, which also covers jobs deleted while the server was down.
   - **Deleted mid-run** (REST or skill): the delete keeps the in-progress row and its session. After the run finalizes, `pruneOldRuns` finds the job row gone and calls `deleteJobRuns`, which removes both.
-- **Logs**: files older than 30 days deleted on a daily timer (db row kept; UI shows "log unavailable"). `cron_runs` rows older than 60 days are also dropped, through `deleteRuns`, so the session rule above applies.
+- **Logs**: in-process runs write none; pre-2.2 files older than 30 days are deleted on a daily timer (db row kept). `cron_runs` rows older than 60 days are also dropped, through `deleteRuns`, so the session rule above applies.
 
 ## Key files
 
-- `packages/server/src/cron/runner.ts` — scheduler: fire timers (croner computes times), `runJob` spawn/capture/finalize, reconcile, retention; `effectiveSessionId` / `perRunSessionId`, the read-only session lookup (`readSessionRow`) behind `sessionAgentIdFor` / `sessionAgentConflict`.
+- `packages/server/src/cron/runner.ts` — scheduler: fire timers (croner computes times), `runJob` / `finishRun` / timeout / `cancelRun`, `recoverRuns`, `startOrInject`, reconcile, retention; `effectiveSessionId` / `perRunSessionId`, the read-only session lookup (`readSessionRow`) behind `sessionAgentIdFor` / `sessionAgentConflict`.
 - `packages/server/src/cron/dispatcher.ts` — channel-agnostic dispatch registry (`registerCronDispatcher`, `dispatchToTargets`, `listAllCronTargets`).
-- `packages/server/src/db/cron-db.ts` — `cron_jobs` + `cron_runs` schema, `CRON_MIGRATIONS`, `getCronDb`.
-- `packages/server/src/agents/session-manager.ts` — `forgetExternalWrite` (drop the stale in-memory copy of a session a cli child just wrote).
+- `packages/server/src/cron/events.ts` — webhook event queue (`claimPendingEvents`, `settleRunEvents`, `requeueEvents`, `eventText`); `routes/cron-hooks.ts` — the receiver.
+- `packages/server/src/db/cron-db.ts` — `cron_jobs` + `cron_runs` + `cron_events` schema, `CRON_MIGRATIONS`, `getCronDb`.
+- `packages/server/src/agents/relay.ts` — the `{cron}` `ReplyTo` branch and `CronRunDoor`; `session-manager.ts` — `rawHasStamp` (webhook redelivery check).
 - `packages/server/src/channels/<ch>/cron-dispatcher.ts` — per-channel `dispatch` + `listTargets`.
 - `packages/server/templates/skills/cron/SKILL.md` — the agent-facing skill that CRUDs `cron_jobs` (edits the db directly; `reconcileFromDb` picks it up). It receives `{{channel.type/account_id/chat_id}}` placeholders, so a job created from a chat defaults its target to that chat, and "delete my cron" inside a chat reverse-looks-up its subscriptions with `list --chat-id <id>`.
 
 ## Scope
 
-Supported: recurring (cron expr) + one-shot (`runAt`) schedules; per-timezone; optional active window for recurring jobs (`activeFrom` / `activeUntil` — "pause until", "send until"); per-job configurable timeout (`timeoutSec`, 60–21600s, default 3600); multi-target fan-out; stable accumulating session per job — its own (id = job id) or a picked existing root session (`sessionId`) — or a fresh session every run (`stateless`); a per-job folder (`workingDir`, the run's `--dir`); run-now (manual trigger); admin UI history with per-run logs + dispatch results.
+Supported: recurring (cron expr) + one-shot (`runAt`) schedules + webhook triggers (batched events, redelivered after a restart); per-timezone; optional active window for recurring jobs (`activeFrom` / `activeUntil` — "pause until", "send until"); per-job configurable timeout (`timeoutSec`, 60–21600s, default 3600); multi-target fan-out; stable accumulating session per job — its own (id = job id) or a picked existing root session (`sessionId`) — or a fresh session every run (`stateless`); a per-job folder (`workingDir`); run-now (manual trigger); runs resumed after a server restart; admin UI history with the run's session + dispatch results.
 
-Not supported: sub-minute schedules below croner's resolution; in-process job execution (always a fresh cli child); running in a sub-session (`>` paths — root sessions only); changing a stateful job's agent once its session exists (agent lock); a conflict-free shared session — a message sent into a picked session while its cron run is in flight can lose one of the two turns (logged, not prevented; see step 4).
+Not supported: sub-minute schedules below croner's resolution; running in a sub-session (`>` paths — root sessions only); changing a stateful job's agent once its session exists (agent lock); a webhook job on a picked session; running cron from the CLI / TUI (server only).