@aion0/forge 0.10.22 → 0.10.25

package/docs/tp-automation-api-v2.md

@@ -0,0 +1,482 @@
+# TP Automation API — 3-Step Workflow for Dev/QA
+
+Minimum HTTP contract for driving a `build → upgrade → run tests →
+collect results` cycle against an automation testbed on TP. All
+endpoints are stateless except for the durable `PytestExecution` /
+`SingleDevice` rows the celery workers update — callers just POST then
+poll, no websockets/sse/callbacks.
+
+```
+<TP-base-url>/<endpoint>
+```
+
+`<TP-base-url>`:
+- Production: `https://nac-tp.fortinet-us.com`
+- Test: `http://10.15.33.25:8000`
+- Dev (.11): `http://10.15.33.11:8000`
+
+## Authentication
+
+Every endpoint requires a JWT in the `Authorization` header:
+
+```
+Authorization: JWT <token>
+```
+
+Mint a token:
+
+```bash
+T=$(curl -s -X POST <TP-base-url>/token-auth/ \
+       -H 'Content-Type: application/json' \
+       --data-binary @- <<JSON | jq -r .token
+{"username":"<user>","password":"<pw>"}
+JSON
+)
+```
+
+In examples below `$T` stands for the JWT.
+
+> **Note on prod:** `/token-auth/` is **not** exposed at the public
+> reverse proxy on prod (SSO-only). Automation scripts on prod-facing
+> endpoints need to be run from a host that can reach the internal
+> port, or they need an SSO-issued token. On the dev `.11` server,
+> `/token-auth/` works directly with username + password.
+
+---
+
+## Step 1 — Upgrade the testbed
+
+```
+POST /adc/automation/upgrade/                — kick off (HTTP 202)
+GET  /adc/automation/upgrade/<testbed>/      — poll status
+```
+
+The dev team has three modes available; pick whichever matches your
+build pipeline:
+
+| `mode`     | When to use it | Required extra fields |
+|---|---|---|
+| `command`  | You have a Jenkins-built image and Jenkins already produces the full `execute restore image scp ...` command string. | `command` |
+| `build`    | You have a build number from the official build server. | `build_number` |
+| `ga`       | You want the latest GA build for a specific FortiNAC version. The system resolves the build number via `BuildHistory.GA=True`. | `version` |
+
+### Request
+
+Content-Type: `application/json`.
+
+```json
+{
+  "testbed": "AT16_Combined_FSW",
+  "mode":    "command",
+  "command": "execute restore image scp /var/lib/jenkins/jobs/fortinac-build-7.6-1/builds/3983/archive/nacos/FNAC_ESX-v7-build6956-FORTINET_job1_3983.out 10.15.33.5 jenkins fortinet"
+}
+```
+
+```json
+{
+  "testbed": "AT16_Combined_FSW",
+  "mode":    "build",
+  "build_number": "0815"
+}
+```
+
+```json
+{
+  "testbed": "AT16_Combined_FSW",
+  "mode":    "ga",
+  "version": "7.6.5"
+}
+```
+
+Internal behavior:
+
+1. Looks up the testbed's `deployinfo`, extracts every device whose
+   name contains `nac` or `ncm`. A multi-NAC testbed gets one celery
+   task per device, running in parallel.
+2. `mode=ga` resolves `version` → `build_number` via `BuildHistory`,
+   then dispatches the same path as `mode=build`.
+3. Concurrency guard: HTTP **409** if any target IP is already
+   `PROGRESS`.
+4. `mode=command` validates the command starts with
+   `execute restore image` (security guard — keeps arbitrary CLI from
+   riding through this endpoint).
+
+### Response (HTTP 202)
+
+```json
+{
+  "testbed": "AT16_Combined_FSW",
+  "mode": "command",
+  "build_number": "",
+  "target_ips": ["10.15.52.152"],
+  "tasks": [
+    {"ip": "10.15.52.152", "celery_id": "f62a4cb7-d435-41de-adb6-12cca405d507"}
+  ]
+}
+```
+
+Returns in <500 ms. The actual upgrade work runs asynchronously in
+celery workers — the HTTP call returns the moment the task is queued.
+
+### Poll for status
+
+```
+GET /adc/automation/upgrade/<testbed>/
+```
+
+```json
+{
+  "testbed": "AT16_Combined_FSW",
+  "status":  "PROGRESS",
+  "target_ips": ["10.15.52.152"],
+  "per_device": [
+    {
+      "ip": "10.15.52.152",
+      "status": "PROGRESS",
+      "last_task_type": "command",
+      "last_build_number": null,
+      "updated_at": "2026-05-28T18:14:19+00:00",
+      "log_tail": "...last 2 KB of the NAC's SSH output..."
+    }
+  ]
+}
+```
+
+Aggregate `status` values:
+
+| Value | Meaning |
+|---|---|
+| `UNKNOWN`  | No upgrade has been dispatched against this testbed (yet). |
+| `PROGRESS` | At least one device is still mid-upgrade. |
+| `SUCCESS`  | Every device finished with `SUCCESS`. |
+| `FAILURE`  | At least one device finished with `FAILURE` / `TIMEOUT` (and no device is still in flight). |
+
+`per_device[].log_tail` is the last ~2 KB of the device's SSH session
+output, updated every ~3 s by the worker. Useful for live debugging
+without flooding the response.
+
+### Concrete polling loop
+
+```bash
+while :; do
+  R=$(curl -sH "Authorization: JWT $T" $TP/adc/automation/upgrade/AT16_Combined_FSW/)
+  STATUS=$(echo "$R" | jq -r .status)
+  echo "$(date +%H:%M:%S) $STATUS"
+  case "$STATUS" in
+    PROGRESS) sleep 15 ;;
+    SUCCESS)  echo "upgrade complete"; break ;;
+    FAILURE)  echo "upgrade FAILED"; echo "$R" | jq .per_device; exit 1 ;;
+    *)        echo "$R"; exit 1 ;;
+  esac
+done
+```
+
+A real upgrade takes 2-15 minutes depending on the image and how many
+devices the testbed has.
+
+---
+
+## Step 2 — Run pytest cases on that testbed
+
+```
+POST /adc/automation/pytest/             — kick off (HTTP 202)
+GET  /adc/automation/pytest/<exec_id>/   — poll status + results
+```
+
+### Request
+
+Content-Type: `application/json`.
+
+```json
+{
+  "user":     "<TP username>",
+  "lab":      "AT16_Combined_FSW",
+  "testcase": [
+    "/Tests_CLI/test_cli_sanity.py::TestGetBasic::test_get_system_status"
+  ],
+  "argument": "-vv --tb=short",
+  "extra_pytest_options": ""
+}
+```
+
+| Field | Type | Notes |
+|---|---|---|
+| `user`     | string | TP username of the caller. Must own the lab (be in `users` or `usedby` on the `AutomationTBUser` row). |
+| `lab`      | string | AT lab name. Same `<testbed>` value as Step 1. |
+| `testcase` | **list of strings** | Each entry is a pytest test-id path **relative to the tests repo root** (`/root/fnac_auto/tests` on the controller). Leading slash is OK; both `/Tests_CLI/...` and `Tests_CLI/...` work. The handler iterates the list, joins with spaces. |
+| `argument` | string | Raw pytest CLI args injected between the testcase paths and the framework's `--html`/`--rack-file` flags. `-k`, `-m`, `-vv`, `--tb=short`, `--maxfail=N`, etc. |
+| `extra_pytest_options` | string | (optional) Extra options appended *after* `--html`/`--rack-file` on the real run. Use for env-specific flags that shouldn't apply to the `--collect-only` dry run. |
+
+### Response (HTTP 202)
+
+```json
+{
+  "exec_id": 446,
+  "status":  "Initiating",
+  "lab":     "AT16_Combined_FSW",
+  "testcase_count": 1
+}
+```
+
+Returns in ~300 ms. Save `exec_id` — it's the handle for Step 3.
+
+### What runs on the controller
+
+The celery worker SFTP-uploads this script to the controller VM and
+launches it under `nohup setsid`:
+
+```bash
+#!/bin/bash
+set +e
+cd /root/fnac_auto/tests          && git pull origin main || true
+cd /root/fnac_auto/test-framework && git pull origin main || true
+cd /root/fnac_auto
+source venv/bin/activate
+export PYTHONPATH=/root/fnac_auto/test-framework:/root/fnac_auto/tests
+export DISPLAY=:99
+pytest <expanded testcase paths> <argument> --collect-only
+pytest <expanded testcase paths> <argument> --html=<report> --rack-file <rack> <extra_pytest_options>
+echo $? > /tmp/pytest_<exec_id>.exit
+```
+
+Key properties:
+
+- **Pytest is fully detached** from the launching ssh session via
+  `nohup setsid`. The ssh session can drop, the network can blip, the
+  TP backend can restart — pytest keeps running on the controller.
+- **TP polls via short, separate ssh round-trips** to read the log
+  file and check for the terminal `/tmp/pytest_<exec_id>.exit` marker.
+- **Surviving a TP backend restart** is automatic — the celery worker
+  is a separate process and just keeps polling. Verified end-to-end on
+  `.11` (exec_id 447: killed runserver mid-run, exec completed
+  normally and report was fetched back).
+
+### Concurrency guard
+
+If another `PytestExecution` on the same lab has `status` =
+`Running` or `Initiating`, the POST returns HTTP **409**:
+
+```json
+{"error": "another execution is already in flight on 'AT16_Combined_FSW'"}
+```
+
+---
+
+## Step 3 — Read per-test results
+
+```
+GET /adc/automation/pytest/<exec_id>/
+```
+
+Same endpoint as the Step-2 poll — there's nothing distinct to call.
+
+### Response
+
+```json
+{
+  "exec_id":          446,
+  "status":           "Done",
+  "lab":              "AT16_Combined_FSW",
+  "controller":       "10.15.52.159",
+  "remote_pid":       "177854",
+  "pass_count":       1,
+  "fail_count":       0,
+  "skip_count":       0,
+  "error_count":      0,
+  "total_count":      1,
+  "start_timestamp":  "1780090011",
+  "end_timestamp":    "1780090041",
+  "report_file_path": "data/AutomationTest/test_446_report_2026-05-29_21:26:50.html",
+  "log_file_path":    "/tmp/pytest_446.log",
+  "log_size":         25338,
+  "log_tail":         "...last 4 KB of pytest stdout..."
+}
+```
+
+| Field | Meaning |
+|---|---|
+| `status` | One of `Initiating`, `Running`, `Done`, `Failed`, `Cancelled`. **Treat `Done` as terminal — success/failure is inferred from counts.** |
+| `pass_count`, `fail_count`, `skip_count`, `error_count`, `total_count` | Parsed from pytest's own summary line. Populated **mid-run** (updated every 15 s), so you can see partial progress without waiting for the run to finish. |
+| `start_timestamp` / `end_timestamp` | Unix epoch seconds, written by the worker. |
+| `report_file_path` | Server-relative path to the pytest-html report. Fetch via `<TP>/<report_file_path>` with the JWT for per-test rows + tracebacks. |
+| `log_file_path` | On-controller path to the live log. Mostly informational — use `log_tail` to read the last 4 KB without an extra round-trip. |
+| `log_tail` | Last 4 KB of pytest stdout. |
+| `remote_pid` | The detached pytest tree's pid on the controller. Useful if you ever need to ssh to the controller and kill it manually. |
+
+### Concrete polling loop
+
+```bash
+EXEC=$(curl -sX POST -H "Authorization: JWT $T" \
+     -H 'Content-Type: application/json' \
+     -d '{"user":"alice","lab":"AT16_Combined_FSW","testcase":["/Tests_CLI/test_cli_sanity.py::TestGetBasic::test_get_system_status"],"argument":"-vv"}' \
+     $TP/adc/automation/pytest/ | jq -r .exec_id)
+echo "started exec $EXEC"
+
+while :; do
+  R=$(curl -sH "Authorization: JWT $T" $TP/adc/automation/pytest/$EXEC/)
+  STATUS=$(echo "$R" | jq -r .status)
+  COUNTS=$(echo "$R" | jq -r '"P\(.pass_count)/F\(.fail_count)/S\(.skip_count)/E\(.error_count) of T\(.total_count)"')
+  echo "$(date +%H:%M:%S) $STATUS $COUNTS"
+  case "$STATUS" in
+    Initiating|Running) sleep 15 ;;
+    Done)
+      FAIL=$(echo "$R" | jq -r .fail_count)
+      ERR=$(echo "$R"  | jq -r .error_count)
+      if [ "$FAIL" -gt 0 ] || [ "$ERR" -gt 0 ]; then
+        echo "tests had failures — fetch report at $TP/$(echo "$R" | jq -r .report_file_path)"
+        exit 1
+      fi
+      echo "all pass"; break ;;
+    Failed|Cancelled)
+      echo "execution terminated: $STATUS"
+      echo "$R" | jq -r .log_tail; exit 1 ;;
+  esac
+done
+```
+
+---
+
+## End-to-end script
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+TP=http://10.15.33.11:8000
+USER=admin
+TESTBED=AT16_Combined_FSW
+
+T=$(curl -s -X POST $TP/token-auth/ -H 'Content-Type: application/json' \
+       --data-binary @- <<JSON | jq -r .token
+{"username":"$USER","password":"<your-pw>"}
+JSON
+)
+
+# ---- Step 1: upgrade testbed ----
+COMMAND='execute restore image scp /var/lib/jenkins/jobs/fortinac-build-7.6-1/builds/3983/archive/nacos/FNAC_ESX-v7-build6956-FORTINET_job1_3983.out 10.15.33.5 jenkins fortinet'
+
+curl -s -X POST -H "Authorization: JWT $T" \
+     -H 'Content-Type: application/json' \
+     -d "$(jq -n --arg cmd "$COMMAND" --arg tb "$TESTBED" \
+            '{testbed:$tb, mode:"command", command:$cmd}')" \
+     $TP/adc/automation/upgrade/ > /dev/null
+
+while :; do
+  R=$(curl -sH "Authorization: JWT $T" $TP/adc/automation/upgrade/$TESTBED/)
+  S=$(echo "$R" | jq -r .status)
+  echo "upgrade: $S"
+  [ "$S" = "SUCCESS" ] && break
+  [ "$S" = "FAILURE" ] && { echo "$R" | jq .per_device; exit 1; }
+  sleep 30
+done
+
+# ---- Step 2 + 3: run tests, collect results ----
+EXEC=$(curl -s -X POST -H "Authorization: JWT $T" \
+     -H 'Content-Type: application/json' \
+     -d "$(jq -n --arg tb "$TESTBED" --arg user "$USER" \
+            '{user:$user, lab:$tb,
+              testcase:["/Tests_CLI/test_cli_sanity.py::TestGetBasic::test_get_system_status"],
+              argument:"-vv --tb=short"}')" \
+     $TP/adc/automation/pytest/ | jq -r .exec_id)
+echo "started exec $EXEC"
+
+while :; do
+  R=$(curl -sH "Authorization: JWT $T" $TP/adc/automation/pytest/$EXEC/)
+  S=$(echo "$R" | jq -r .status)
+  echo "pytest: $S $(echo "$R" | jq -r '"P\(.pass_count)/F\(.fail_count) of T\(.total_count)"')"
+  case "$S" in
+    Done)
+      F=$(echo "$R" | jq -r .fail_count)
+      E=$(echo "$R" | jq -r .error_count)
+      [ "$F" -gt 0 ] || [ "$E" -gt 0 ] && {
+        echo "report: $TP/$(echo "$R" | jq -r .report_file_path)"
+        exit 1
+      }
+      echo "PASS"; break ;;
+    Failed|Cancelled)
+      echo "exec terminated: $S"; echo "$R" | jq -r .log_tail; exit 1 ;;
+    *) sleep 15 ;;
+  esac
+done
+```
+
+---
+
+## Behaviors worth knowing
+
+### Concurrency
+
+For each `<testbed>`:
+- **Upgrade**: HTTP 409 if any target IP currently has
+  `SingleDevice.last_task_status='PROGRESS'`.
+- **Pytest**: HTTP 409 if any `PytestExecution` with this `lab` has
+  `status` in `{Running, Initiating}`.
+
+You can have an upgrade AND a pytest running simultaneously on
+*different* testbeds. The guards are per-testbed.
+
+### Survival semantics
+
+| Failure | Upgrade | Pytest v2 |
+|---|---|---|
+| ssh session drops mid-run | N/A — TP only ssh's briefly | Pytest keeps running (`nohup setsid` on controller) |
+| TP backend restart mid-run | Celery worker keeps going, completes the run, writes terminal state | Same — celery `pytest_poll` keeps short-polling the controller |
+| Controller VM reboot mid-run | Upgrade reboots the NAC; this is expected/desired. | Pytest is lost (controller died); polling will eventually fail with the max-runtime safety net (6 h cap). |
+| Celery worker restart | In-flight task may be lost (celery uses ack_late=false by default). Worth a follow-up. | Same. |
+
+### Per-test pass/fail visibility
+
+The `passed_nodes` / `failed_nodes` fields on the legacy `pytest_run`
+response shape don't apply to v2. v2 surfaces:
+
+- **aggregate counts** (`pass_count`, `fail_count`, etc.) in the GET
+  response — populated mid-run as pytest emits them
+- **the HTML report** at `report_file_path` — the standard
+  pytest-html document with per-test rows, durations, tracebacks, and
+  captured logs. Fetch via `<TP>/<report_file_path>` with the JWT.
+
+If you need per-test JSON instead of the HTML report, ask — it's a
+trivial extra endpoint that parses the HTML and returns
+`{passed: [...], failed: [...]}`.
+
+### Test-case path format
+
+Whatever you'd type after `pytest` on the command line, prefixed (or
+not) with `/`. Relative to `/root/fnac_auto/tests/` on the controller.
+
+| Want to | Use |
+|---|---|
+| Run one test | `["/Tests_CLI/test_cli_sanity.py::TestGetBasic::test_get_system_status"]` |
+| Run several | `["/Tests_CLI/test_cli_sanity.py::TestGetBasic::test_get_system_status", "/Tests_CLI/test_cli_sanity.py::TestShowFullConfig::test_nacos_show_config"]` |
+| Run a whole file | `["/Tests_CLI/test_cli_sanity.py"]` |
+| Run by marker | `["/Tests_CLI"]` + `argument: "-m smoke"` |
+| Dry-run / list | `["/Tests_CLI"]` + `argument: "--collect-only"` |
+
+Tip: discover paths via `GET /adc/get_testcases` (legacy endpoint;
+returns the full test tree). Note: this endpoint does a `git pull` on
+TP's local mirror of the tests repo as a side effect — fine in normal
+use, just be aware.
+
+---
+
+## Source files
+
+| Concern | File |
+|---|---|
+| URL routes | `backend/adc/urls.py` |
+| Upgrade API | `backend/adc/views/automation/upgrade_api.py` |
+| Upgrade tasks | `backend/adc/tasks.py` (`upgrade_nac_command`, `upgrade_nac_build`) |
+| Pytest v2 API | `backend/adc/views/automation/pytest_api.py` |
+| Pytest v2 tasks | `backend/adc/tasks.py` (`pytest_launch`, `pytest_poll`) |
+| Per-device state | `backend/adc/models.py` (`SingleDevice`, `PytestExecution`) |
+| GA-build resolution | `backend/adc/models.py` (`BuildHistory.GA=True`) |
+
+Legacy endpoints (still live, used by the `/automation` UI page, kept
+for backward compatibility):
+
+- `POST /adc/nac-upgrade-testbed/`
+- `POST /adc/pytest_run`
+- `POST /adc/get_test_execution_by_id`
+
+Avoid these for new automation — they're synchronous-in-the-request,
+don't survive TP restarts, and the upgrade variant blocks the HTTP
+request for the full restore duration.

package/lib/chat/agent-loop.ts

@@ -26,6 +26,7 @@ import {
 import { getMemoryStore } from './memory-store';
 import { buildMemoryContext } from './build-memory-context';
 import { buildMemoryTools } from './memory-tools';
+import { buildStartWatchTool } from '../watch/start-watch-tool';
 import { estimateTokens } from '../memory/token-estimate';
 import {
   listInstalledConnectors,
@@ -48,10 +49,25 @@ const MAX_TOKENS = 16000;
 // and recalled via buildMemoryContext as compact blocks instead.
 const HISTORY_MSG_BUDGET = 60;
 const HISTORY_TOKEN_BUDGET = 8000;
+// Hard cap on a single tool_result stored into the conversation (chars).
+// A giant result (e.g. a connector returning a full test tree) would
+// otherwise blow the whole HISTORY_TOKEN_BUDGET, push its paired
+// assistant tool_use out of the window, and leave an orphan tool_result
+// that trimOrphanToolResults strips — yielding an empty history and an
+// "messages must not be empty" provider error. ~16k chars ≈ 4k tokens,
+// half the budget, so a complete tool_use+result pair always survives.
+const MAX_TOOL_RESULT_CHARS = 16000;
 
 // After clipping to last N, the first kept message may be a tool_result
 // whose tool_use was cut. Anthropic/OpenAI both reject that, so drop
 // leading tool_result-bearing user messages until the slice starts clean.
+function truncateToolResult(s: string): string {
+  if (s.length <= MAX_TOOL_RESULT_CHARS) return s;
+  return s.slice(0, MAX_TOOL_RESULT_CHARS) +
+    `\n\n[… tool result truncated: ${s.length} chars total, showing first ${MAX_TOOL_RESULT_CHARS}. ` +
+    `Refine the call (filter / paginate / flatten) to get a smaller, complete result.]`;
+}
+
 function trimOrphanToolResults(history: Message[]): Message[] {
   let i = 0;
   while (i < history.length) {
@@ -73,6 +89,7 @@ export interface AgentEvent {
     | 'message_saved'   // a full message persisted (assistant or tool-results carrier)
     | 'memory_status'   // pinned/blocks/hits snapshot from Temper for the UI strip
     | 'turn_done'       // loop finished
+    | 'watch_status'    // ambient background-watch progress (status chip, NOT a message)
     | 'error';          // unrecoverable
   message_id?: string;
   data?: any;
@@ -392,6 +409,11 @@ export async function runTurn(args: RunTurnArgs): Promise<{ ok: boolean; error?:
   const memHandlers: Record<string, BuiltinHandler> = {};
   for (const t of memTools) memHandlers[t.def.name] = t.handle;
 
+  // start_watch — LLM-driven background watch (always available). Bound
+  // to this session so completion reports back here.
+  const watchTool = buildStartWatchTool(args.sessionId);
+  memHandlers[watchTool.def.name] = watchTool.handle;
+
   if (memStore.enabled) {
     // Inspector strip (memory_status event) wants the full inventory —
     // keep its own listBlocks call. The prompt-injection text comes
@@ -466,6 +488,7 @@ export async function runTurn(args: RunTurnArgs): Promise<{ ok: boolean; error?:
   const builtinDefsAll = [
     ...BUILTIN_TOOL_DEFS,
     ...memTools.map((m) => m.def),
+    watchTool.def,
   ];
   const allTools: LlmTool[] = [
     ...builtinDefsAll.map((t) => ({
@@ -500,6 +523,13 @@ export async function runTurn(args: RunTurnArgs): Promise<{ ok: boolean; error?:
       const history = trimOrphanToolResults(
         listMessagesCapped(args.sessionId, HISTORY_MSG_BUDGET, HISTORY_TOKEN_BUDGET, estimateTokens),
       );
+      // Belt-and-suspenders: tool_result truncation should keep a complete
+      // pair in-window, but if history is somehow empty, fail clearly
+      // instead of letting the provider throw "messages must not be empty".
+      if (history.length === 0) {
+        cb({ type: 'error', data: { error: 'Conversation context is empty after trimming an oversized result. Clear the chat or retry with a narrower query.' } });
+        return { ok: false, error: 'empty history' };
+      }
 
       assistantBlocksAccum = [];
       let currentTextBuf = '';
@@ -543,11 +573,11 @@ export async function runTurn(args: RunTurnArgs): Promise<{ ok: boolean; error?:
       const toolUses = result.content.filter((b): b is ToolUseBlock => b.type === 'tool_use');
       const toolResults: ToolResultBlock[] = [];
       for (const t of toolUses) {
-        const r = await dispatchTool({ id: t.id, name: t.name, input: t.input }, memHandlers);
+        const r = await dispatchTool({ id: t.id, name: t.name, input: t.input }, { extraBuiltins: memHandlers, sessionId: args.sessionId });
         const block: ToolResultBlock = {
           type: 'tool_result',
           tool_use_id: t.id,
-          content: r.content,
+          content: truncateToolResult(r.content),
           is_error: r.is_error,
         };
         toolResults.push(block);

package/lib/chat/tool-dispatcher.ts

@@ -483,6 +483,12 @@ export interface DispatchOptions {
    * therefore don't need an LLM-friendly truncation.
    */
   noTruncation?: boolean;
+  /** Chat session that triggered this call — used to register a watch
+   *  (async tools) bound to the right session for completion callbacks. */
+  sessionId?: string;
+  /** Remaining chain budget for async watch callbacks. Defaults to the
+   *  full depth at the top level; decremented when a watch chains a tool. */
+  chainDepth?: number;
 }
 
 export async function dispatchTool(
@@ -567,30 +573,64 @@ export async function dispatchTool(
   }
 
   try {
+    let result: ToolResult;
     switch (protocol) {
       case 'http':
-        return await runHttp({ tool: located.tool, settings: effectiveSettings, args: argInput, connectorAuth: def.auth, noTruncation: opts.noTruncation });
+        result = await runHttp({ tool: located.tool, settings: effectiveSettings, args: argInput, connectorAuth: def.auth, noTruncation: opts.noTruncation });
+        break;
       case 'shell':
-        return await runShell({ tool: located.tool, settings: effectiveSettings, args: argInput });
+        result = await runShell({ tool: located.tool, settings: effectiveSettings, args: argInput });
+        break;
       case 'ssh':
-        return await runSsh({ tool: located.tool, settings: effectiveSettings, args: argInput });
+        result = await runSsh({ tool: located.tool, settings: effectiveSettings, args: argInput });
+        break;
       case 'browser': {
         // Hand the whole connector + tool spec + input + settings to the
         // extension's runner.ts via the bridge. The extension keeps owning
         // the runner logic (tab acquire, navigate, executeScript).
         const connector = buildConnectorPayload(def, located.entry, effectiveSettings);
-        const result = (await bridgeRpc('connector.run', {
+        const r = (await bridgeRpc('connector.run', {
           pluginId: located.connectorId,       // wire-name kept for extension
           toolName: located.toolName,
           input: argInput,
           connector,
           settings: effectiveSettings,
         }, located.tool.timeout_ms)) as { content?: string; is_error?: boolean } | null;
-        return { content: result?.content ?? '(no content returned)', is_error: !!result?.is_error };
+        result = { content: r?.content ?? '(no content returned)', is_error: !!r?.is_error };
+        break;
       }
       default:
         return { content: `unknown protocol "${protocol}" on tool ${call.name}`, is_error: true };
     }
+
+    // Async (long-task watch): if the tool declared an `async` block and
+    // it ran without error, register a background watch that polls to
+    // completion and reports back to the originating chat session. The
+    // tool's own result is returned to the caller immediately (detach).
+    if (located.tool.async && !result.is_error) {
+      try {
+        const { registerWatch, DEFAULT_CHAIN_DEPTH } = await import('../watch/register');
+        let parsed: unknown = result.content;
+        try { parsed = JSON.parse(result.content); } catch { /* keep string */ }
+        const reg = registerWatch({
+          spec: located.tool.async,
+          connectorId: located.connectorId,
+          toolName: located.toolName,
+          args: argInput,
+          result: parsed,
+          settings: effectiveSettings,
+          sessionId: opts.sessionId ?? null,
+          chainDepth: opts.chainDepth ?? DEFAULT_CHAIN_DEPTH,
+        });
+        const note = reg.ok
+          ? `\n\n[watch ${reg.watch_id} registered — polling in the background; you'll get a chat update on completion.]`
+          : `\n\n[watch not registered: ${reg.reason}]`;
+        result = { ...result, content: result.content + note };
+      } catch (e) {
+        console.warn('[dispatch] registerWatch failed', (e as Error).message);
+      }
+    }
+    return result;
   } catch (e) {
     return { content: `connector tool failed: ${(e as Error).message}`, is_error: true };
   }

package/lib/chat-standalone.ts

@@ -37,6 +37,7 @@ import {
 } from './chat/session-store';
 import { runTurn, type AgentEvent } from './chat/agent-loop';
 import { bridgePush } from './chat/bridge-client';
+import { startWatchRunner } from './watch/watch-runner';
 
 const PORT = Number(process.env.CHAT_PORT) || 8408;
 const startTime = Date.now();
@@ -302,6 +303,17 @@ httpServer.listen(PORT, '127.0.0.1', () => {
     const main = ensureMainSession();
     console.log(`[chat] Main session: ${main.id.slice(0, 8)} "${main.title}"`);
   } catch (e) { console.warn('[chat] ensureMainSession failed:', (e as Error).message); }
+
+  // Background long-task watches: poll to completion, then feed the
+  // result back into the originating session (assistant replies) or push
+  // ambient progress (status chip, not a message).
+  startWatchRunner({
+    onProgress: (sessionId, payload) => fanoutEvent(sessionId, { type: 'watch_status', data: payload }),
+    runChat: (sessionId, text) => {
+      void runTurn({ sessionId, userText: text, callbacks: { onEvent: (e) => fanoutEvent(sessionId, e) } })
+        .catch((err) => console.error('[watch] runChat failed', (err as Error).message));
+    },
+  });
 });
 
 function shutdown(): void {