@aion0/forge 0.10.20 → 0.10.23

package/lib/help-docs/24-watch.md

@@ -0,0 +1,77 @@
+# Background Watches
+
+A **watch** is Forge's lightweight async primitive for long-running jobs:
+you (or the assistant) kick something off that finishes minutes later —
+a device firmware upgrade, a Jenkins build, a test run — and instead of
+the assistant sitting in the conversation polling (burning tokens,
+getting stuck), Forge polls it in the **background** and posts the result
+back into this chat when it's done.
+
+You don't manage watches directly most of the time — they appear when a
+long job is started and clear themselves when it finishes.
+
+## What you see
+
+- **Progress chip** — while a watch runs, a small status pill shows
+  above the chat composer (e.g. "Upgrading NAC 10.15.52.152… poll 3/15,
+  build 6956"). It updates in place, is **not** a chat message (doesn't
+  clutter the thread), and disappears when the job finishes.
+- **Completion message** — when the job reaches its goal (or fails /
+  times out), a normal assistant message lands in the chat:
+  "NAC … upgrade confirmed — now running build 6957." If your session
+  came from Telegram, that message arrives in Telegram.
+- **Watch list** — see and control active watches in two places:
+  - **/chat** web: the "Background watches" panel in the left sidebar.
+  - **Forge web**: user menu → **Monitor** → "Background Watches".
+  Each entry shows status (active / done / failed / timed_out), poll
+  count, and **Cancel** / **Delete** buttons.
+
+Watches survive a Forge restart (persisted in SQLite) — an in-flight
+upgrade keeps being watched after a restart, no chat needed.
+
+## Two ways a watch starts
+
+**1. Connector built-in (`async` block).** Some connector tools are
+pre-declared as long tasks by their author — e.g. `nac.upgrade`
+automatically registers a watch that polls `nac.get_version` until the
+new build is running. Nothing for you to do; it just works.
+
+**2. The assistant decides (`start_watch`).** For long jobs that aren't
+pre-declared — a Jenkins build, a one-off cross-connector flow — the
+assistant calls the `start_watch` builtin on the fly: it triggers the
+job, registers a watch to poll the right status tool, and then stops
+talking. You get the result later in chat.
+
+Both use the same background machinery; you experience them identically.
+
+## `start_watch` (for the assistant)
+
+When you've just started a long job and have a tool that reports its
+status, register a watch instead of polling in the conversation:
+
+```
+start_watch({
+  poll: "jenkins.get_build",                       // <connector>.<status tool>
+  poll_args: { job_path: "job/foo", build_number: 18 },
+  done_match: { path: "result", equals: "SUCCESS" },  // or done_path (truthy)
+  interval_sec: 60, timeout_sec: 1800,
+  message: "Build 18 finished: {poll.result}"      // {poll.<path>} = latest result
+})
+```
+
+Then **stop** — do not keep calling get_build in the conversation. A
+completion message arrives in chat; the user can cancel it from the watch
+list. Pick `done_match`/`done_path` from a field you saw in the status
+tool's output (you usually called it once already). Queue/startup errors
+(e.g. a build number 404 while still queued) are tolerated for a while.
+Guards (max polls, timeout, lifetime, active cap) keep it from running
+away; worst case it times out and reports that.
+
+## Limits
+
+- Watches are short-lived; they end at done / failed / timed_out /
+  cancelled and stop polling.
+- `start_watch` reports back only via chat — it doesn't open a separate
+  notification channel. Telegram-origin sessions reply on Telegram.
+- A watch can't *fix* a job that's stuck waiting for human action (a
+  build needing approval); it only watches to a final state and reports.

package/lib/help-docs/CLAUDE.md

@@ -50,6 +50,7 @@ The token is valid for 24 hours. Store it in a variable and reuse for all API ca
 | `21-build-connector.md` | **Authoring** a custom connector — interview script, manifest template (browser / http / shell protocols), how to install locally via the Forge data dir or a zip upload. Use this when the user asks to BUILD a connector, not when they ask about an existing one. |
 | `18-chrome-mcp.md` | Connect Forge Claude Code sessions to a real Chrome via chrome-devtools-mcp — dev-time browser access for connector authoring |
 | `23-automation-states.md` | Fortinet pipeline automation: GitLab MR stage labels, Mantis status flow, Teams notify policy |
+| `24-watch.md` | Background watches — async polling of long jobs (device upgrade, Jenkins build, test run) that report back in chat. Two triggers: connectors' declarative `async` block, and the `start_watch` builtin the assistant calls on the fly. Where to see/cancel them. |
 
 ## Matching questions to docs
 
@@ -83,3 +84,4 @@ The token is valid for 24 hours. Store it in a variable and reuse for all API ca
 - Job / scheduled job / connector poll / "Jobs tab" → tell user: **Jobs is deprecated**; use Schedules (`13-schedules.md`) instead.
 - Recipe / "From recipe" form / parameterized job → tell user: **recipes deprecated** along with Jobs; fire pipelines manually or via Schedules.
 - Mantis bug fix / fortinet-mantis-bug-fix / open MR for Mantis bug / fortinet-mr-review / pre-review / GitLab stage labels → `23-automation-states.md` (kept Fortinet pipelines)
+- Background watch / "watch this build" / "tell me when the upgrade is done" / progress chip / start_watch / async long job / why is the assistant polling → `24-watch.md`

package/lib/watch/register.ts

@@ -0,0 +1,108 @@
+/**
+ * registerWatch — turn a just-run `async` tool into a background watch.
+ *
+ * Called by the tool-dispatcher right after a tool with an `async` block
+ * runs (and detaches). Resolves the spec's templates against the
+ * trigger's args/result, enforces the global active cap and chain-depth
+ * guard, and persists a watch row. The watch-runner (chat-standalone)
+ * picks it up on the next tick. No dependency on the runner or
+ * tool-dispatcher → no import cycle.
+ */
+
+import type { AsyncWatchSpec } from '../connectors/types';
+import { resolveDeep, resolveTemplate } from './template';
+import { createWatch, countActive, type WatchAction } from './watch-store';
+
+export const MAX_ACTIVE_WATCHES = 50;
+export const DEFAULT_CHAIN_DEPTH = 3;
+const MIN_INTERVAL_SEC = 30;
+const DEFAULT_INTERVAL_SEC = 60;
+const DEFAULT_TIMEOUT_SEC = 1200;
+const DEFAULT_MAX_POLLS = 40;
+
+export interface RegisterWatchCtx {
+  spec: AsyncWatchSpec;
+  connectorId: string;
+  toolName: string;
+  args: Record<string, unknown>;          // trigger tool input
+  result: unknown;                          // trigger tool result (parsed)
+  settings: Record<string, unknown>;
+  sessionId: string | null;
+  chainDepth: number;                       // remaining chain budget
+}
+
+export type RegisterResult =
+  | { ok: true; watch_id: string; label: string }
+  | { ok: false; reason: string };
+
+function num(v: unknown, dflt: number): number {
+  const n = Number(v);
+  return Number.isFinite(n) && n > 0 ? n : dflt;
+}
+
+export function registerWatch(ctx: RegisterWatchCtx): RegisterResult {
+  const { spec } = ctx;
+  if (!spec || !spec.poll) return { ok: false, reason: 'no async.poll declared' };
+  if (ctx.chainDepth <= 0) return { ok: false, reason: 'chain depth exhausted' };
+  if (countActive() >= MAX_ACTIVE_WATCHES) {
+    return { ok: false, reason: `active watch limit reached (${MAX_ACTIVE_WATCHES})` };
+  }
+
+  // Templating context for register-time resolution.
+  const regCtx = { args: ctx.args, result: ctx.result, settings: ctx.settings };
+
+  const pollArgs = (resolveDeep(spec.poll_args || {}, regCtx) as Record<string, unknown>) || {};
+
+  let doneMatch: { path: string; equals?: string; contains?: string } | null = null;
+  if (spec.done_match && spec.done_match.path) {
+    doneMatch = {
+      path: spec.done_match.path,
+      ...(spec.done_match.equals != null ? { equals: resolveTemplate(String(spec.done_match.equals), regCtx) } : {}),
+      ...(spec.done_match.contains != null ? { contains: resolveTemplate(String(spec.done_match.contains), regCtx) } : {}),
+    };
+  }
+
+  const interval = Math.max(MIN_INTERVAL_SEC, num(spec.interval_sec, DEFAULT_INTERVAL_SEC));
+  const timeout = num(spec.timeout_sec, DEFAULT_TIMEOUT_SEC);
+  const maxPolls = num(spec.max_polls, DEFAULT_MAX_POLLS);
+
+  // Label: connector.tool + a hint (host/lab/ip) pulled from args if present.
+  const hint = ['host', 'lab', 'ip', 'name'].map((k) => ctx.args[k]).find((v) => typeof v === 'string' && v);
+  const label = `${ctx.connectorId}.${ctx.toolName}${hint ? ` ${hint}` : ''}`;
+
+  // Pre-resolve {args.*}/{result.*}/{settings.*} in messages/action-args NOW
+  // (they're fixed at register time). {poll.*}/{poll_count}/{max_polls} are
+  // left literal for the runner to fill per poll — resolveTemplate keeps
+  // tokens whose namespace isn't in the context, so the two passes compose.
+  const preAction = (a: WatchAction | undefined): WatchAction | null => {
+    if (!a) return null;
+    return {
+      ...a,
+      ...(a.message ? { message: resolveTemplate(a.message, regCtx) } : {}),
+      ...(a.args ? { args: resolveDeep(a.args, regCtx) as Record<string, unknown> } : {}),
+    };
+  };
+  const preProgress = spec.progress
+    ? { ...spec.progress, ...(spec.progress.message ? { message: resolveTemplate(spec.progress.message, regCtx) } : {}) }
+    : null;
+
+  const w = createWatch({
+    session_id: ctx.sessionId,
+    label,
+    connector_id: ctx.connectorId,
+    poll_tool: spec.poll,
+    poll_args: pollArgs,
+    done_path: spec.done_path ?? null,
+    done_match: doneMatch,
+    fail_path: spec.fail_path ?? null,
+    on_done: preAction(spec.on_done as WatchAction),
+    on_fail: preAction(spec.on_fail as WatchAction),
+    progress: preProgress,
+    interval_sec: interval,
+    timeout_sec: timeout,
+    max_polls: maxPolls,
+    chain_depth: ctx.chainDepth,
+    now: Date.now(),
+  });
+  return { ok: true, watch_id: w.id, label };
+}

package/lib/watch/start-watch-tool.ts

@@ -0,0 +1,116 @@
+/**
+ * `start_watch` — a builtin tool that lets the LLM register a background
+ * watch on the fly, instead of relying on a connector author's static
+ * `async` block. Use case: the model just triggered a long job (jenkins
+ * build, a test run) and, rather than polling in-conversation (burning
+ * tokens, getting stuck), it calls start_watch to have Forge poll a
+ * given tool until a done/fail condition, then report back in chat.
+ *
+ * This is the dynamic counterpart to manifest `async` blocks — same
+ * watch backend (store + runner + chip), just driven by the model. The
+ * handler closes over the originating session id so completion routes to
+ * the right conversation.
+ */
+
+import type { BuiltinToolDef } from '../chat/tool-dispatcher';
+import { createWatch, countActive } from './watch-store';
+import { MAX_ACTIVE_WATCHES, DEFAULT_CHAIN_DEPTH } from './register';
+
+const MIN_INTERVAL_SEC = 30;
+const DEFAULT_INTERVAL_SEC = 60;
+const DEFAULT_TIMEOUT_SEC = 1800;
+const DEFAULT_MAX_POLLS = 40;
+
+export interface StartWatchTool {
+  def: BuiltinToolDef;
+  handle: (input: unknown) => Promise<string>;
+}
+
+function num(v: unknown, dflt: number): number {
+  const n = Number(v);
+  return Number.isFinite(n) && n > 0 ? n : dflt;
+}
+
+export function buildStartWatchTool(sessionId: string | null): StartWatchTool {
+  const def: BuiltinToolDef = {
+    name: 'start_watch',
+    description:
+      'Register a BACKGROUND WATCH that polls a tool until done, then posts the result back here — for long-running jobs you just kicked off (a Jenkins build, a test run, a device upgrade). Use this INSTEAD of polling in conversation: call the trigger tool, then call start_watch and STOP — Forge polls in the background and a completion message arrives in this chat. ' +
+      'Pick `poll` = the read tool that reports status (e.g. "jenkins.get_build") and `poll_args` to call it with (e.g. the build number you predicted via get_next_build_number). Give a done condition: `done_match` {path, equals} on the poll result (e.g. path "result" equals "SUCCESS"), or `done_path` (a result path that becomes truthy). You usually already saw the poll tool\'s output once, so you know the right field. Optional `fail_path` (truthy = failed). Tune `interval_sec`/`timeout_sec` to the job (build ≈ 60s / 1800s).',
+    input_schema: {
+      type: 'object',
+      properties: {
+        poll: { type: 'string', description: 'Tool to poll, "<connector>.<tool>" e.g. "jenkins.get_build". Must be a read/status tool.' },
+        poll_args: { type: 'object', description: 'Args to call the poll tool with each tick, e.g. {"job_path":"job/foo","build_number":18}. Concrete values, not templates.' },
+        done_match: {
+          type: 'object',
+          description: 'Done when poll-result <path> equals/contains this. e.g. {"path":"result","equals":"SUCCESS"}.',
+          properties: {
+            path: { type: 'string' },
+            equals: { type: 'string' },
+            contains: { type: 'string' },
+          },
+        },
+        done_path: { type: 'string', description: 'Alternative to done_match: done when this poll-result path is truthy.' },
+        fail_path: { type: 'string', description: 'Optional: poll-result path that, when truthy, means failed.' },
+        message: { type: 'string', description: 'Message posted to chat on completion. May reference {poll.<path>}, e.g. "Build 18 finished: {poll.result}".' },
+        fail_message: { type: 'string', description: 'Optional message on failure/timeout.' },
+        progress_message: { type: 'string', description: 'Optional per-poll status chip text (ambient, not a chat message). e.g. "Build 18 — {poll.result}".' },
+        interval_sec: { type: 'number', description: 'Seconds between polls (default 60, min 30).' },
+        timeout_sec: { type: 'number', description: 'Overall deadline in seconds (default 1800).' },
+        max_polls: { type: 'number', description: 'Hard cap on poll count (default 40).' },
+      },
+      required: ['poll'],
+    },
+  };
+
+  const handle = async (input: unknown): Promise<string> => {
+    const a = (input ?? {}) as Record<string, any>;
+    const poll = String(a.poll || '').trim();
+    const dot = poll.indexOf('.');
+    if (dot < 1) return JSON.stringify({ ok: false, error: 'poll must be "<connector>.<tool>", e.g. jenkins.get_build' });
+    const connectorId = poll.slice(0, dot);
+    const pollTool = poll.slice(dot + 1);
+
+    const doneMatch = a.done_match && typeof a.done_match === 'object' && a.done_match.path
+      ? { path: String(a.done_match.path), ...(a.done_match.equals != null ? { equals: String(a.done_match.equals) } : {}), ...(a.done_match.contains != null ? { contains: String(a.done_match.contains) } : {}) }
+      : null;
+    const donePath = typeof a.done_path === 'string' && a.done_path ? a.done_path : null;
+    if (!doneMatch && !donePath) {
+      return JSON.stringify({ ok: false, error: 'give a done condition: done_match {path,equals} or done_path' });
+    }
+    if (countActive() >= MAX_ACTIVE_WATCHES) {
+      return JSON.stringify({ ok: false, error: `active watch limit reached (${MAX_ACTIVE_WATCHES})` });
+    }
+
+    const hint = ['build_number', 'host', 'ip', 'lab', 'id', 'name'].map((k) => a.poll_args?.[k]).find((v) => v != null && v !== '');
+    const label = `${connectorId}.${pollTool}${hint != null ? ` ${hint}` : ''}`;
+
+    const w = createWatch({
+      session_id: sessionId,
+      label,
+      connector_id: connectorId,
+      poll_tool: pollTool,
+      poll_args: (a.poll_args && typeof a.poll_args === 'object') ? a.poll_args : {},
+      done_path: donePath,
+      done_match: doneMatch,
+      fail_path: typeof a.fail_path === 'string' && a.fail_path ? a.fail_path : null,
+      on_done: { mode: 'chat', message: String(a.message || `${label}: done.`) },
+      on_fail: { mode: 'chat', message: String(a.fail_message || `${label}: did not complete in time — please check.`) },
+      progress: { show: true, ...(a.progress_message ? { message: String(a.progress_message) } : {}) },
+      interval_sec: Math.max(MIN_INTERVAL_SEC, num(a.interval_sec, DEFAULT_INTERVAL_SEC)),
+      timeout_sec: num(a.timeout_sec, DEFAULT_TIMEOUT_SEC),
+      max_polls: num(a.max_polls, DEFAULT_MAX_POLLS),
+      chain_depth: DEFAULT_CHAIN_DEPTH,
+      now: Date.now(),
+    });
+    return JSON.stringify({
+      ok: true,
+      watch_id: w.id,
+      polling: poll,
+      note: 'Background watch registered. STOP polling in conversation — a completion message will arrive in this chat. The user can see/cancel it in the watch list.',
+    });
+  };
+
+  return { def, handle };
+}

package/lib/watch/template.ts

@@ -0,0 +1,40 @@
+/**
+ * Tiny namespaced template resolver for watch specs. Replaces tokens
+ * like {args.x}, {result.fired_at}, {poll.build}, {settings.host} from a
+ * context of namespaces. Dot paths supported. Unknown tokens are left
+ * literal (so a typo is visible rather than silently empty). No eval.
+ */
+
+export function getPath(obj: unknown, path: string): unknown {
+  let v: any = obj;
+  for (const p of path.split('.')) {
+    if (v == null || typeof v !== 'object') return undefined;
+    v = v[p];
+  }
+  return v;
+}
+
+export function resolveTemplate(str: string, ctx: Record<string, unknown>): string {
+  return str.replace(/\{([^{}]+)\}/g, (full, raw) => {
+    const key = String(raw).trim();
+    const dot = key.indexOf('.');
+    const ns = dot < 0 ? key : key.slice(0, dot);
+    const rest = dot < 0 ? '' : key.slice(dot + 1);
+    if (!(ns in ctx)) return full;
+    const v = rest ? getPath(ctx[ns], rest) : ctx[ns];
+    if (v == null) return full;
+    return typeof v === 'object' ? JSON.stringify(v) : String(v);
+  });
+}
+
+/** Resolve templates throughout a value (strings, arrays, plain objects). */
+export function resolveDeep(val: unknown, ctx: Record<string, unknown>): unknown {
+  if (typeof val === 'string') return resolveTemplate(val, ctx);
+  if (Array.isArray(val)) return val.map((v) => resolveDeep(v, ctx));
+  if (val && typeof val === 'object') {
+    const out: Record<string, unknown> = {};
+    for (const [k, v] of Object.entries(val)) out[k] = resolveDeep(v, ctx);
+    return out;
+  }
+  return val;
+}

package/lib/watch/watch-runner.ts

@@ -0,0 +1,158 @@
+/**
+ * Watch runner — the background ticker that drives long-task watches.
+ *
+ * Runs inside chat-standalone (single instance, guarded). Each tick it
+ * polls every due active watch (single-flight, bounded concurrency),
+ * evaluates done/fail/timeout, and on a terminal state runs the watch's
+ * action: feed the result back into the originating chat session
+ * (mode=chat, via the runChat callback), chain a tool (mode=tool), or
+ * just record it (mode=none). Per-poll it emits ambient progress through
+ * onProgress — that lands as a status chip, NOT a chat message.
+ *
+ * Guards (see design doc §3): max_polls, timeout, max_lifetime, single-
+ * flight, consecutive-error cutoff, chain-depth on tool callbacks.
+ */
+
+import { dispatchTool } from '../chat/tool-dispatcher';
+import { resolveTemplate, resolveDeep, getPath } from './template';
+import {
+  listDue, updateWatch, getWatch, type Watch, type WatchState,
+} from './watch-store';
+
+const TICK_MS = 20_000;            // scan cadence; each watch paces itself via next_poll_at
+const MAX_CONCURRENT = 8;          // simultaneous in-flight polls
+const MAX_CONSEC_ERRORS = 10;      // device unreachable (e.g. mid-reboot) tolerated this many polls
+
+export interface WatchRunnerHooks {
+  /** Emit ambient progress (status chip) for a watch's session. */
+  onProgress?: (sessionId: string, payload: Record<string, unknown>) => void;
+  /** Feed a completion message back into a chat session (assistant replies). */
+  runChat?: (sessionId: string, text: string) => void;
+}
+
+function truthy(v: unknown): boolean {
+  if (v == null || v === false || v === 0) return false;
+  if (typeof v === 'string') { const s = v.trim().toLowerCase(); return s !== '' && s !== 'false' && s !== '0' && s !== 'null'; }
+  return true;
+}
+
+function parseResult(content: string): any {
+  try { return JSON.parse(content); } catch { return { _raw: content }; }
+}
+
+const g = globalThis as any;
+
+export function startWatchRunner(hooks: WatchRunnerHooks = {}): void {
+  if (g.__forgeWatchRunner) return;       // single instance per process
+  const running = new Set<string>();      // watch ids currently polling (single-flight)
+
+  const finish = (w: Watch, state: WatchState, obj: unknown, summary: string) => {
+    const now = Date.now();
+    updateWatch(w.id, { state, last_result: obj, last_text: summary }, now);
+    // Terminal watch_status — tells the UI to drop the progress chip
+    // immediately (otherwise it lingers until the 150s prune). The real
+    // completion text goes via on_done below; this is just the chip kill.
+    if (hooks.onProgress && w.session_id) {
+      hooks.onProgress(w.session_id, { watch_id: w.id, state, done: true, text: summary });
+    }
+    const action = state === 'done' ? w.on_done : w.on_fail;
+    const mode = action?.mode || 'chat';
+    if (mode === 'none' || !action) return;
+    const ctx = { poll: obj };
+    if (mode === 'chat') {
+      if (!w.session_id || !hooks.runChat) return;
+      const msg = action.message ? resolveTemplate(action.message, ctx) : summary;
+      const tag = state === 'done' ? '✅' : '⚠️';
+      hooks.runChat(w.session_id, `[background watch ${w.label}] ${tag} ${msg}`);
+    } else if (mode === 'tool' && action.tool) {
+      const input = (resolveDeep(action.args || {}, ctx) as Record<string, unknown>) || {};
+      void dispatchTool(
+        { id: `watch-chain-${w.id}`, name: action.tool, input },
+        { sessionId: w.session_id || undefined, chainDepth: Math.max(0, w.chain_depth - 1) } as any,
+      ).catch((e) => console.error('[watch] on_done tool chain failed', w.id, e));
+    }
+  };
+
+  const emitProgress = (w: Watch, obj: unknown, pollCount: number) => {
+    if (!w.session_id || !hooks.onProgress) return;
+    if (w.progress && w.progress.show === false) return;
+    const tmpl = w.progress?.message || 'Watching {label} … poll {poll_count}/{max_polls}';
+    const text = resolveTemplate(tmpl, { poll: obj, poll_count: pollCount, max_polls: w.max_polls, label: w.label });
+    hooks.onProgress(w.session_id, { watch_id: w.id, state: 'active', poll_count: pollCount, text });
+  };
+
+  const pollOne = async (w: Watch): Promise<void> => {
+    const now = Date.now();
+    // Hard lifetime backstop.
+    if (now - w.created_at > 2 * w.timeout_sec * 1000) {
+      return finish(w, 'timed_out', w.last_result, `${w.label}: watch lifetime exceeded — please verify manually.`);
+    }
+    let res;
+    try {
+      res = await dispatchTool({ id: `watch-${w.id}-${w.polls}`, name: `${w.connector_id}.${w.poll_tool}`, input: w.poll_args }, { noTruncation: false } as any);
+    } catch (e) {
+      res = { content: String(e), is_error: true };
+    }
+
+    if (res.is_error) {
+      // Poll failed (often expected: device rebooting). Tolerate up to N
+      // consecutive, then give up.
+      const errs = w.err_count + 1;
+      if (errs >= MAX_CONSEC_ERRORS) {
+        return finish(w, 'errored', { error: res.content }, `${w.label}: ${MAX_CONSEC_ERRORS} consecutive poll errors — giving up. Last: ${String(res.content).slice(0, 200)}`);
+      }
+      updateWatch(w.id, { err_count: errs, next_poll_at: now + w.interval_sec * 1000 }, now);
+      emitProgress(w, { _error: String(res.content).slice(0, 120) }, w.polls);
+      return;
+    }
+
+    const obj = parseResult(res.content);
+    const polls = w.polls + 1;
+
+    // fail check
+    if (w.fail_path && truthy(getPath(obj, w.fail_path))) {
+      return finish(w, 'failed', obj, `${w.label}: failure condition met.`);
+    }
+    // done check
+    let done = false;
+    if (w.done_match) {
+      const v = getPath(obj, w.done_match.path);
+      if (w.done_match.equals != null) done = String(v) === String(w.done_match.equals);
+      else if (w.done_match.contains != null) done = String(v ?? '').toLowerCase().includes(String(w.done_match.contains).toLowerCase());
+    } else if (w.done_path) {
+      done = truthy(getPath(obj, w.done_path));
+    }
+    if (done) {
+      return finish(w, 'done', obj, `${w.label}: done.`);
+    }
+    // not done — bound by polls / timeout, else reschedule
+    if (polls >= w.max_polls || now - w.created_at > w.timeout_sec * 1000) {
+      return finish(w, 'timed_out', obj, `${w.label}: not done within ${w.max_polls} polls / ${w.timeout_sec}s — please verify manually.`);
+    }
+    updateWatch(w.id, { polls, err_count: 0, next_poll_at: now + w.interval_sec * 1000 }, now);
+    emitProgress(w, obj, polls);
+  };
+
+  const tick = async () => {
+    try {
+      const slots = MAX_CONCURRENT - running.size;
+      if (slots <= 0) return;
+      const due = listDue(Date.now()).filter((w) => !running.has(w.id)).slice(0, slots);
+      for (const w of due) {
+        running.add(w.id);
+        // Re-read inside to avoid acting on a row cancelled since listDue.
+        void Promise.resolve()
+          .then(() => { const cur = getWatch(w.id); return cur && cur.state === 'active' ? pollOne(cur) : undefined; })
+          .catch((e) => console.error('[watch] poll error', w.id, e))
+          .finally(() => running.delete(w.id));
+      }
+    } catch (e) {
+      console.error('[watch] tick error', e);
+    }
+  };
+
+  const timer = setInterval(() => { void tick(); }, TICK_MS);
+  if (typeof timer.unref === 'function') timer.unref();
+  g.__forgeWatchRunner = timer;
+  console.log('[watch] runner started (tick ' + TICK_MS / 1000 + 's)');
+}