@trawlme/cli 3.10.0 → 3.12.0

package/README.md

@@ -79,7 +79,7 @@ trawl spec [--json]                            Print a versioned, machine-readab
 - `list`/`get` also show a **health** badge next to the status badge. By default it reads the scrap's own `lastCronOutcome` field (always present, no extra cost): `⚠ cron paused` when the last scheduled tick was skipped for being unhealthy, `—` otherwise — a breadcrumb of the last tick, not a live read (never set on a no-cron scrap, can lag by one tick). `list --unhealthy` asks the API to filter to scraps whose *current* consecutive-failure streak is 3+ (trawl_node's own threshold) and annotates each with the exact `consecutiveFailedRuns`/`unhealthySince` — shown instead of the badge whenever present, and never re-derived client-side from history. `--json` carries whichever fields the request produced: `lastCronOutcome` always, `consecutiveFailedRuns`/`unhealthySince` only when `--unhealthy` was passed. **Old-server safety:** a server predating trawl_node#1952 silently ignores the unknown `--unhealthy` query param and returns everything, unfiltered — the CLI detects that (no returned item carries `consecutiveFailedRuns`) and prints a stderr warning instead of presenting the full list as "your unhealthy scraps".
 - **Long-running calls (`create`, `run`, `data --fresh`, `trigger --wait`):** these hit server-side paths that can legitimately take 30–250s+ (AI generation + scrap creation + a first run for `create`; proxy tier escalation + AI-fix retries for the other three) — the CLI arms a 300s timeout for exactly these four call sites instead of the generic 30s default. `TRAWL_TIMEOUT` (see below) still overrides ALL requests, including these — set it if you need a tighter or looser ceiling than 300s, but note a global override that tight also clamps `create`.
 - **`--watch` is poll-based, not a live stream:** the activities SSE endpoint has no backlog and, for the default async `trigger` (no `--wait`), runs in a separate cron-consumer pod whose events never reach the API pod holding the SSE connection — a naive "await the run, then open SSE" shows nothing. `run --watch` and `trigger --watch` instead poll `GET /api/scraps/:id` (terminal status) and the activities REST list until the run finishes, printing each new activity line as it appears. The watched run's outcome drives the exit code too, in BOTH human and `--json` mode: a genuinely failed terminal run, a poll timeout, or a persistently unreachable API all exit non-zero — a clean successful run is the only exit `0`. A run that never reaches a terminal status within 300s prints an honest timeout notice pointing at `scraps doctor <id>` (human mode) — see the `--json` shape below. `scraps watch <id>` (the standalone command, no trigger) is unchanged — it still opens the live SSE stream directly.
-- `spec --json` prints the CLI's own command tree — `{specVersion, cliVersion, commands[], exitCodes, errorKinds, kindExitCodes}` — DERIVED at runtime by walking the live commander tree (never a hand-maintained file, which would silently drift from reality). Each entry in `commands[]` carries its full path (e.g. `"scraps account session set"`), description, `hidden` (the legacy `scraps <verb>` aliases above), `leaf` (false for a pure namespace/group node like `scraps`/`skills`/`telemetry` — invoking one directly is a guaranteed-failing tool, not a real command), `aliases`, `arguments`, and `options`; `exitCodes`/`errorKinds`/`kindExitCodes` are read from the exact same source `classifyError` uses (see [Exit codes](#exit-codes)) — never a second copy. `kindExitCodes` is the inverse of `exitCodes`: `kind -> exitCode`, since exit code `1` alone is a shared bucket (`api`/`refused`/`unknown`/`in_progress`/`run_failed`/`upgrade_failed`) that `exitCodes`' flat label can't disambiguate. Without `--json`, `spec` prints one short human line (version + visible command count) pointing at `--json`.
+- `spec --json` prints the CLI's own command tree — `{specVersion, cliVersion, commands[], exitCodes, errorKinds, kindExitCodes, docsUrl?, llmsUrl?}` — DERIVED at runtime by walking the live commander tree (never a hand-maintained file, which would silently drift from reality). Each entry in `commands[]` carries its full path (e.g. `"scraps account session set"`), description, `hidden` (the legacy `scraps <verb>` aliases above), `leaf` (false for a pure namespace/group node like `scraps`/`skills`/`telemetry` — invoking one directly is a guaranteed-failing tool, not a real command), `aliases`, `arguments`, `options`, and an optional per-command `docs` deep link (e.g. every `scraps account *` command points at the account-sessions guide); `exitCodes`/`errorKinds`/`kindExitCodes` are read from the exact same source `classifyError` uses (see [Exit codes](#exit-codes)) — never a second copy. `kindExitCodes` is the inverse of `exitCodes`: `kind -> exitCode`, since exit code `1` alone is a shared bucket (`api`/`refused`/`unknown`/`in_progress`/`run_failed`/`upgrade_failed`) that `exitCodes`' flat label can't disambiguate. `docsUrl` is resolved server-first (`externalDocs.url` on the configured API base's own OpenAPI document, when declared), else derived from a KNOWN first-party `trawl.me` API host, else **omitted entirely** — never a guessed URL pointed at the wrong docs host for a self-hosted install. `llmsUrl` and every per-command `docs` link are narrower: they only ever come from that same known-host derivation, NEVER from a server-declared `docsUrl` — this CLI's own guide slugs have no reason to exist on a third party's own docs root, so a self-hosted server that declares `externalDocs.url` gets a correct top-level `docsUrl` but no `llmsUrl` and no per-command `docs` links. A `failureKind:"auth"` run's JSON payload (`doctor`/`data --errors`/`run-info`) carries the same `docs` field, by the same known-host-only rule. Without `--json`, `spec` prints one short human line (version + visible command count) pointing at `--json`.
 - `run --json`/`trigger --json` bypass the spinner and print the raw launch/trigger payload on stdout; combined with `--watch`, every intermediate progress line stays suppressed (stdout stays pure JSON) and, once the watch reaches its outcome, exactly ONE final NDJSON line is emitted: `{"runId","status"}` (the honest terminal status — `success`/`error`/`empty`/`regression`/…), `{"runId","status":"timeout"}` on a poll timeout, or `{"runId","status":"poll_error","error"}` if the API stays unreachable for several consecutive polls — `process.exitCode` is non-zero for all three except a genuine success. `scraps watch --json` emits one raw JSON object per activity line (NDJSON) instead of the formatted `[time] message` text — there's no single final payload to wait for on a live stream.
 
 ### Scrap management
@@ -113,9 +113,12 @@ trawl scraps account delete <id> [--force] [--json]
 trawl scraps account clear-session <id> [--json]
 trawl scraps account status <id> [--json]
 trawl scraps account session set <id> -c <file> [--json]
+trawl scraps account session capture <id> [--chrome <path>] [--json]
 ```
 
-`account session set` uploads a Puppeteer cookie JSON array to bootstrap a logged-in session without storing credentials (BYO-cookies). A missing `-u/--username`/`-p/--password` on `account set` follows the same [non-interactive rule](#non-interactive-rule) as `login`.
+`account session set` uploads a cookie JSON array — or a `{ cookies, origins }` storageState file (see `session capture` below) — to bootstrap a logged-in session without storing credentials (BYO-cookies). A missing `-u/--username`/`-p/--password` on `account set` follows the same [non-interactive rule](#non-interactive-rule) as `login`.
+
+`account session capture <id>` is the one-command version of the same idea: it opens a real, visible Chrome window at the scrap's target URL, waits for **you** to log in there — 2FA included — then reads the resulting session over the Chrome DevTools Protocol (cookies **and** per-origin localStorage) and uploads it. You stay authenticated as yourself; Trawl never sees your credentials, and responsibility for lawful use of the captured session stays with you. The capture is scoped to the scrap's own target host by RFC 6265 domain-matching, the same rule a real browser uses to decide which cookies to send: a cookie explicitly scoped to a domain (e.g. `Domain=example.com`) is in scope for that host and its subdomains, while a host-only cookie (no `Domain` attribute) is in scope only for the exact host it was set on — never a sibling subdomain, even one sharing a parent domain, and never another tenant on the same multi-tenant hosting domain (e.g. a different `*.github.io` site). localStorage follows the same anchor: an origin is in scope only if it's the target host itself or a subdomain of it. The one deliberate gap: a host-only cookie set on a sibling host (e.g. `auth.example.com` while the scrap targets `app.example.com`) is excluded — a real browser would never send it to the target host either, so nothing replay-relevant is lost unless the scrap script itself later navigates to that sibling. Completion is explicit: press Enter in the terminal once you're signed in — the primary path, and the only one guaranteed cross-platform. Closing the Chrome window also completes the capture (cookies only, not localStorage) when Chrome itself stays running after its last window closes, which this CLI verified on macOS; on a platform where closing the last window quits Chrome entirely, the command reports that fact instead of a session, and pressing Enter is the reliable path there. This command needs a local interactive terminal with a real display; it does not work headless, in CI, or over a plain SSH session, and auto-detects a system Chrome/Chromium (override with `--chrome <path>` or `TRAWL_CHROME_PATH`). Never prints the captured session itself, in any mode — `--json` echoes only the server's response plus capture counts.
 
 ### Claude Code skills
 

package/dist/commands/scraps.js

@@ -12,6 +12,36 @@ import { formatDoctor, formatAutofix, fetchRunAndFix, pickRun, pickFix, isAuthWa
 import { renderPinch, pinchEnabled } from '../lib/pinch.js';
 import { maybeShowReferralTip } from '../lib/tips.js';
 import { maybeSuggestSkillsForAuthWall } from '../lib/skillsNudge.js';
+import { captureSession, checkInteractiveEnvironment, LOCALSTORAGE_READ_TIMEOUT_MS, } from '../lib/session-capture.js';
+import { assertSecureTransport } from '../lib/secure-transport.js';
+import { getApiUrl } from '../lib/config.js';
+import { resolveDocsUrls, resolveFailureKindDocsUrl } from '../lib/docs.js';
+/**
+ * #185 — `{ docs }` (spreadable, empty object when nothing to add) for a
+ * run's JSON payload (`doctor`/`data --errors`/`run-info`), keyed on
+ * trawl_node's `failureKind` (NOT errors.ts's unrelated `ErrorEnvelope.kind`
+ * — see docs.ts's `FAILURE_KIND_DOC_PATHS` doc comment for why those two
+ * `'auth'`s must never be conflated). Reuses `isAuthWall` — the SAME
+ * staleness guard `formatDoctor`'s human hint and the #184 skills nudge
+ * already share (a `failureKind:'auth'` stamp survives a later
+ * success/regression patch, see `isAuthWall`'s own doc comment) — so a run
+ * that ultimately succeeded or degraded never carries a stale `docs` link
+ * either, exactly the invariant `detectWallVendor`/`isAuthWall` already
+ * enforce for the human-facing surfaces. One call site, one rule, reused by
+ * all three JSON surfaces below instead of three copies that could drift.
+ */
+function runDocsField(run) {
+    if (!isAuthWall(run))
+        return {};
+    // Whole resolved object (never just `.docsUrl`) so resolveFailureKindDocsUrl
+    // can gate on rung provenance — see docs.ts's DocsUrls.docsUrlIsDerived.
+    // This call never supplies `externalDocsUrl`, so `docsUrl` here can only
+    // ever be rung-2-derived or absent — never a rung-1 server-declared root —
+    // but the gate stays explicit rather than relying on that as an invariant.
+    const resolved = resolveDocsUrls({ apiBaseUrl: getApiUrl() });
+    const docs = resolveFailureKindDocsUrl(run.failureKind, resolved);
+    return docs ? { docs } : {};
+}
 /**
  * Print a usage/validation error consistently: human text to stderr, or a
  * machine envelope on stdout under --json (never both — reportError is the
@@ -1065,8 +1095,12 @@ export function attachDataCommand(parent, attachOpts = {}) {
             // --json is honored for BOTH outcomes (success or failure) — an agent
             // parsing `data --errors --json` must always get the flat run object,
             // never prose gated behind a status check. (#71 finding 13)
+            // #185 — `docs` rides alongside as a sibling field on the same flat
+            // object (never nested under `run`, since there's no wrapper shape
+            // here), present only for a live mapped failureKind (see
+            // runDocsField's doc comment).
             if (opts.json)
-                return json(pickRun(result.run));
+                return json({ ...pickRun(result.run), ...runDocsField(result.run) });
             if (result.run.status === true) {
                 console.log(chalk.green('✓ Last run succeeded. No errors to show.'));
                 return;
@@ -1270,6 +1304,11 @@ export function attachRunInfoCommand(parent, attachOpts = {}) {
             selector: h.errorSnapshot?.selector ?? null,
             emptyContext: h.emptyContext ?? null,
             createdAt: h.createdAt ?? null,
+            // #185 — present only for a live mapped failureKind (see
+            // runDocsField's doc comment); omitted, never `docs: undefined`, for
+            // every other run so this object's own JSON.stringify never grows an
+            // unexpected key on existing consumers.
+            ...runDocsField(h),
         };
         if (opts.json) {
             json(info);
@@ -1558,32 +1597,69 @@ account
 const accountSession = account
     .command('session')
     .description('Manage scrap account session cookies (flavour B BYO-cookies)');
+/** Structural check only (bare origin format, {name,value} shape) — the
+ *  server's sessionShape.js does the authoritative validation. No domain
+ *  scoping here: unlike `capture`'s CDP path, a hand-authored/exported file
+ *  has no "target URL" to scope against, and the human supplying it already
+ *  chose what to include. */
+function isValidOriginsShape(value) {
+    return (Array.isArray(value) &&
+        value.every((o) => o &&
+            typeof o === 'object' &&
+            typeof o.origin === 'string' &&
+            Array.isArray(o.localStorage) &&
+            o.localStorage.every((kv) => kv && typeof kv === 'object' && typeof kv.name === 'string' && typeof kv.value === 'string')));
+}
 // account session set
 accountSession
     .command('set <id>')
-    .description('Upload browser session cookies for a scrap (Puppeteer cookie JSON array)')
-    .requiredOption('-c, --cookies <file>', 'Path to a Puppeteer cookie JSON array file')
+    .description('Upload a browser session for a scrap — a cookie JSON array, or a { cookies, origins } storageState file (see `session capture`)')
+    .requiredOption('-c, --cookies <file>', 'Path to a cookie JSON array, or a storageState file ({ cookies, origins? })')
     .option('--json', 'Output as JSON')
     .action(async (id, opts) => {
     validateObjectId(id);
+    // A session is a bearer-equivalent secret — refuse to PUT it over
+    // plain HTTP (trawl_cli#183 review finding 5). Checked before anything
+    // else in this action, including the file read below.
+    const secureTransportError = assertSecureTransport(getApiUrl());
+    if (secureTransportError) {
+        usageError(secureTransportError, { json: opts.json });
+        return;
+    }
     const { existsSync, readFileSync } = await import('fs');
     if (!existsSync(opts.cookies)) {
         usageError(`File not found: ${opts.cookies}`, { json: opts.json });
         return;
     }
     let cookies;
+    let origins;
     try {
         const raw = readFileSync(opts.cookies, 'utf-8');
-        cookies = JSON.parse(raw);
+        const parsed = JSON.parse(raw);
+        if (Array.isArray(parsed)) {
+            cookies = parsed;
+        }
+        else if (parsed && typeof parsed === 'object' && Array.isArray(parsed.cookies)) {
+            // storageState-shaped file — trawl_cli#183's `session capture` output shape.
+            const shaped = parsed;
+            cookies = shaped.cookies;
+            if (shaped.origins !== undefined) {
+                if (!isValidOriginsShape(shaped.origins)) {
+                    usageError('origins must be an array of { origin: string, localStorage: [{name, value}] }', { json: opts.json });
+                    return;
+                }
+                origins = shaped.origins;
+            }
+        }
+        else {
+            usageError('Cookies file must contain a JSON array, or a { cookies, origins? } storageState object', { json: opts.json });
+            return;
+        }
     }
     catch (e) {
         usageError(`Failed to parse cookies file: ${e.message}`, { json: opts.json });
         return;
     }
-    if (!Array.isArray(cookies)) {
-        usageError('Cookies file must contain a JSON array', { json: opts.json });
-        return;
-    }
     if (cookies.length === 0) {
         usageError('Cookies array must not be empty', { json: opts.json });
         return;
@@ -1592,7 +1668,10 @@ accountSession
         usageError('Each cookie must have a name (string) and value (string)', { json: opts.json });
         return;
     }
-    const call = () => api.put(`/api/scraps/${id}/account/session`, { cookies });
+    const body = { cookies };
+    if (origins)
+        body.origins = origins;
+    const call = () => api.put(`/api/scraps/${id}/account/session`, body);
     if (opts.json) {
         const data = await call();
         json(data);
@@ -1605,6 +1684,164 @@ accountSession
     const acc = data.account;
     console.log(chalk.dim('  Session: ') + (acc.hasSession ? chalk.green('✓ active') : chalk.dim('none')));
 });
+/**
+ * @desc Map a failed `captureSession()` result onto this CLI's existing
+ * exit-code taxonomy (#71/#88/#107). `no_cookies_in_scope` is the one
+ * outcome where everything ran correctly but nothing was found to upload —
+ * a business-level refusal (exit 1, kind:"refused"), not a usage mistake.
+ *
+ * `capture_failed` (trawl_cli#183 R4) is a SECOND exception, for the
+ * opposite reason: launch + handshake already succeeded — Chrome was
+ * reachable over CDP, a page was already open — and the failure is a
+ * genuine bug in this CLI (or an unanticipated non-page-scoped failure),
+ * never a "cannot run in this environment" problem. Routing it through
+ * `usageError`'s exit 2 / "cannot complete in the current environment"
+ * framing would misclassify it exactly the way a bare `launch_failed` used
+ * to before the R4 fix. Reported instead through `reportError` with a bare
+ * `Error` — `classifyError` has no dedicated `kind` for it, so it falls
+ * into the generic `kind:"unknown"`/exit 1 bucket, the SAME code an
+ * unhandled bug reaching index.ts's own top-level catch would already get.
+ * Never `RefusalError` (this is not a business-logic refusal, everything
+ * up to this point may have gone fine) and never `UsageError` (nothing
+ * about the invocation or the environment was wrong).
+ *
+ * Every remaining reason (non_interactive, no_chrome, launch_failed, the CDP
+ * pipe closing before capture, a corrupt CDP frame tearing it down) means
+ * THIS invocation cannot complete in the current environment — exit 2, the
+ * same bucket `confirmDestructive`'s own non-interactive refusal uses.
+ */
+function reportCaptureFailure(result, opts) {
+    if (result.reason === 'no_cookies_in_scope') {
+        process.exitCode = reportError(new RefusalError(result.message), { json: opts.json });
+        return;
+    }
+    if (result.reason === 'capture_failed') {
+        process.exitCode = reportError(new Error(result.message), { json: opts.json });
+        return;
+    }
+    usageError(result.message, { json: opts.json });
+}
+// account session capture — trawl_cli#183
+accountSession
+    .command('capture <id>')
+    .description('Open a headed Chrome to the scrap\'s target URL, wait for you to log in (2FA included), and capture the session over CDP — cookies AND localStorage. Requires a local interactive terminal with a display; does not work headless, in CI, or over a plain SSH session. You stay authenticated as yourself — Trawl never sees your credentials. Responsibility for lawful use of the captured session stays with you.')
+    .option('--chrome <path>', 'Path to a Chrome/Chromium executable (auto-detected if omitted)')
+    .option('--json', 'Output as JSON (counts only — a session is a bearer secret and is never printed, in any mode)')
+    .action(async (id, opts) => {
+    validateObjectId(id);
+    // A captured session is a bearer-equivalent secret — refuse to PUT it
+    // over plain HTTP (trawl_cli#183 review finding 5). Checked before
+    // anything else, including the scrap lookup below (which itself
+    // already sends the auth token over whatever transport is configured).
+    const secureTransportError = assertSecureTransport(getApiUrl());
+    if (secureTransportError) {
+        usageError(secureTransportError, { json: opts.json });
+        return;
+    }
+    const scrap = await api.get(`/api/scraps/${id}`);
+    if (!scrap.url) {
+        usageError(`Scrap ${id} has no target URL configured — nothing to open a browser to. A URL can be set with: trawl scraps update ${id} -u <url>`, { json: opts.json });
+        return;
+    }
+    // Validated here, before captureSession ever creates a temp profile:
+    // `existsSync` alone doesn't mean `spawn()` can run the file — a
+    // non-executable path fails asynchronously deep inside chrome-launch.ts
+    // instead. That failure is still caught there (an 'error' listener is
+    // the safety net for anything not caught by this pre-flight — a race,
+    // a permission change after this check, PUPPETEER_EXECUTABLE_PATH/
+    // CHROME_PATH), but a pre-flight check gives a faster, more specific
+    // message for the common case: an explicit path the user pointed at.
+    if (opts.chrome) {
+        const { existsSync, accessSync, constants } = await import('fs');
+        if (!existsSync(opts.chrome)) {
+            usageError(`Chrome not found at: ${opts.chrome}`, { json: opts.json });
+            return;
+        }
+        try {
+            accessSync(opts.chrome, constants.X_OK);
+        }
+        catch {
+            usageError(`Chrome exists at ${opts.chrome} but is not executable.`, { json: opts.json });
+            return;
+        }
+    }
+    else if (process.env.TRAWL_CHROME_PATH) {
+        // Same check, same reasoning, for the env-var form of an explicit
+        // path — findChrome() gives TRAWL_CHROME_PATH top precedence and
+        // would otherwise hand this same non-executable path straight to
+        // captureSession.
+        const trawlChromePath = process.env.TRAWL_CHROME_PATH;
+        const { existsSync, accessSync, constants } = await import('fs');
+        if (existsSync(trawlChromePath)) {
+            try {
+                accessSync(trawlChromePath, constants.X_OK);
+            }
+            catch {
+                usageError(`Chrome exists at ${trawlChromePath} (TRAWL_CHROME_PATH) but is not executable.`, { json: opts.json });
+                return;
+            }
+        }
+    }
+    // Check the same guard captureSession runs internally BEFORE printing
+    // anything about a Chrome window that may never open — otherwise a
+    // non-interactive run saw "a window will open" immediately followed by
+    // "this cannot work headless" (trawl_cli#183 gap).
+    const nonInteractiveReason = checkInteractiveEnvironment();
+    if (nonInteractiveReason) {
+        reportCaptureFailure({ ok: false, reason: 'non_interactive', message: nonInteractiveReason }, opts);
+        return;
+    }
+    // Progress/prompts are stderr-only, in every mode — stdout under
+    // --json must stay a single parseable document (#88/#107's rule,
+    // restated for this command by trawl_cli#183's own hard rules).
+    console.error(chalk.dim(`Chrome will open at ${scrap.url} for you to log in there (2FA included); capture completes when you press Enter back in this terminal.`));
+    const result = await captureSession(scrap.url, opts.chrome ? { findChrome: () => opts.chrome ?? null } : {});
+    if (!result.ok) {
+        reportCaptureFailure(result, opts);
+        return;
+    }
+    const call = () => api.put(`/api/scraps/${id}/account/session`, {
+        cookies: result.storageState.cookies,
+        origins: result.storageState.origins,
+    });
+    if (opts.json) {
+        const data = await call();
+        // Never the session itself — only what the server echoes back, plus
+        // counts (#183 hard rule: no cookie/localStorage VALUE, ever, under
+        // --json or otherwise).
+        json({ account: data.account, targetDomain: result.targetDomain, capture: result.counts });
+        return;
+    }
+    const data = await spin(call, {
+        text: `Uploading captured session for scrap ${chalk.bold(id)}…`,
+        successText: `Session captured for scrap ${chalk.bold(id)}: ${result.counts.cookiesCaptured} cookie(s), ${result.counts.originsCaptured} origin(s) in scope for ${result.targetDomain}`,
+    });
+    const acc = data.account;
+    console.log(chalk.dim('  Session: ') + (acc.hasSession ? chalk.green('✓ active') : chalk.dim('none')));
+    if (result.counts.cookiesDroppedOutOfScope > 0 || result.counts.originsDroppedOutOfScope > 0) {
+        console.log(chalk.dim(`  Scoped to ${result.targetDomain}: dropped ${result.counts.cookiesDroppedOutOfScope} cookie(s) and ${result.counts.originsDroppedOutOfScope} origin(s) outside that domain.`));
+    }
+    if (result.counts.closedEarly) {
+        console.log(chalk.dim('  The browser window was closed before Enter — localStorage was not captured (cookies only).'));
+    }
+    if (result.counts.originsUnreadable > 0) {
+        // Impossible to overlook (chalk.yellow + ⚠, not the routine chalk.dim
+        // scope-drop note above) — this is a DEGRADED capture: cookies still
+        // uploaded, but at least one in-scope origin's localStorage did not.
+        // Two distinct causes share this one count: the page threw reading
+        // `window.localStorage` (e.g. a SecurityError on partitioned
+        // storage), or the read never got a response within
+        // LOCALSTORAGE_READ_TIMEOUT_MS (the page's renderer was blocked — a
+        // native dialog, a synchronous script, a paused debugger;
+        // trawl_cli#183 review finding 3's own pipe-level timeout fix
+        // reopened finding 4's exact bug class on this one path, since a
+        // timeout does not close the pipe the way a corrupt frame does). The
+        // two are not split apart here: both mean the same actionable fact
+        // ("re-run once nothing is blocking that tab") and neither one names
+        // the page URL/title/exception text — a page controls that content.
+        console.log(chalk.yellow(`  ⚠ ${result.counts.originsUnreadable} origin(s) could not be captured — threw reading localStorage, or gave no response within ${LOCALSTORAGE_READ_TIMEOUT_MS / 1000}s (a blocked tab). Not the same as being empty; cookies were still captured.`));
+    }
+});
 // account status
 account
     .command('status <id>')
@@ -1662,8 +1899,11 @@ scraps
         console.log(chalk.dim('No runs yet.'));
         return;
     }
+    // #185 — `docs` sits alongside `run`/`fix` (present only for a live
+    // mapped failureKind — see runDocsField's doc comment), never nested
+    // inside the allowlisted `run` object.
     if (opts.json)
-        return json({ run: pickRun(result.run), fix: pickFix(result.fix) });
+        return json({ run: pickRun(result.run), fix: pickFix(result.fix), ...runDocsField(result.run) });
     console.log(formatDoctor(result.scrap.title, result.run, result.fix, id));
     if (opts.autofix && result.fix) {
         console.log('\n' + formatAutofix(result.fix));

package/dist/commands/spec.d.ts

@@ -1,4 +1,5 @@
 import { Command } from 'commander';
+import { type DocsUrls } from '../lib/docs.js';
 /**
  * `trawl spec --json` (#170) — a versioned, machine-readable description of
  * the command tree, so an agent can learn the CLI's surface without parsing
@@ -54,6 +55,18 @@ export interface CliSpecCommand {
     aliases: string[];
     arguments: CliSpecArgument[];
     options: CliSpecOption[];
+    /**
+     * #185 — a deep link into a specific guide when one is mapped for this
+     * command (docs.ts's `COMMAND_DOC_PATHS`, e.g. every `scraps account *`
+     * command -> the account-sessions guide). Omitted (never a guessed/generic
+     * link) whenever no guide is mapped, the top-level `docsUrl` itself
+     * couldn't be resolved, OR `docsUrl` resolved from rung 1 (a server-
+     * declared `externalDocs.url`) rather than rung 2 (this CLI's own known-host
+     * derivation) — appending our guide slug onto a third party's own docs root
+     * is exactly the confidently-wrong-404 defect this omission closes. See
+     * `resolveCommandDocsUrl` / `DocsUrls.docsUrlIsDerived`.
+     */
+    docs?: string;
 }
 export interface CliSpec {
     specVersion: 1;
@@ -81,12 +94,30 @@ export interface CliSpec {
      * (errors.ts's KIND_EXIT_CODES) — never a second copy.
      */
     kindExitCodes: Record<string, number>;
+    /**
+     * #185 — where the human-readable guides live, resolved via docs.ts's
+     * ladder (server `externalDocs.url` > derive from the API base's known
+     * `trawl.me` host > omit). Absent (never a guessed URL) when neither rung
+     * resolves — e.g. a self-hosted install with no `externalDocs` declared.
+     */
+    docsUrl?: string;
+    /** #185 — the llms.txt entry point alongside `docsUrl`, when resolvable.
+     *  See docs.ts's module doc comment for why this never rides along with a
+     *  server-declared `docsUrl` it wasn't itself derived from. */
+    llmsUrl?: string;
 }
 /**
  * Build the full spec from a live, already-constructed program (e.g.
  * `createProgram()`'s return value). Includes every node in the tree —
  * hidden legacy aliases (`scraps list`, …) included, flagged via `hidden`,
  * so a consumer that wants only the advertised surface can filter on it.
+ *
+ * #185 — `docs` is an already-RESOLVED `DocsUrls` (docs.ts's ladder run to
+ * completion), never computed in here: `buildSpec` stays pure/synchronous
+ * (no network, no config read) exactly as before, so every existing caller
+ * (the README-vs-spec contract test, the synthetic-tree unit tests) keeps
+ * working unchanged by simply omitting the parameter. Only `spec.ts`'s own
+ * action resolves rung 1 (a bounded fetch) before calling this.
  */
-export declare function buildSpec(program: Command): CliSpec;
+export declare function buildSpec(program: Command, docs?: DocsUrls): CliSpec;
 export declare const spec: Command;

package/dist/commands/spec.js

@@ -1,6 +1,9 @@
 import { Command, Help } from 'commander';
 import { json } from '../lib/format.js';
 import { EXIT_CODE_LABELS, ENVELOPE_KINDS, KIND_EXIT_CODES } from '../lib/errors.js';
+import { getApiUrl } from '../lib/config.js';
+import { api } from '../lib/api.js';
+import { resolveDocsUrls, resolveCommandDocsUrl } from '../lib/docs.js';
 function buildOption(option) {
     const entry = {
         long: option.long ?? '',
@@ -15,8 +18,8 @@ function buildOption(option) {
         entry.default = option.defaultValue;
     return entry;
 }
-function buildCommandEntry(cmd, name, hidden) {
-    return {
+function buildCommandEntry(cmd, name, hidden, docs = {}) {
+    const entry = {
         name,
         description: cmd.description(),
         hidden,
@@ -29,20 +32,29 @@ function buildCommandEntry(cmd, name, hidden) {
         })),
         options: cmd.options.map(buildOption),
     };
+    const docsLink = resolveCommandDocsUrl(name, docs);
+    if (docsLink)
+        entry.docs = docsLink;
+    return entry;
 }
 /**
  * Walk `cmd.commands` recursively, collecting one entry per node under its
  * FULL path name. `hidden` is read via commander's own
  * `Help#visibleCommands()` — the same idiom this codebase already uses
  * (scraps.test.ts's #108 describe block) — rather than reaching for the
- * private, untyped `_hidden` field directly.
+ * private, untyped `_hidden` field directly. The resolved `docs` object,
+ * when present, threads through so every entry can carry its own per-command
+ * `docs` deep link (#185) off the SAME resolved root — never a second lookup
+ * per node, and never flattened to a bare string (see docs.ts's
+ * `DocsUrls.docsUrlIsDerived` — the rung provenance must survive the walk
+ * unchanged so `resolveCommandDocsUrl` can still gate on it at each leaf).
  */
-function walk(cmd, prefix, out) {
+function walk(cmd, prefix, out, docs = {}) {
     const visible = new Set(new Help().visibleCommands(cmd));
     for (const sub of cmd.commands) {
         const name = prefix ? `${prefix} ${sub.name()}` : sub.name();
-        out.push(buildCommandEntry(sub, name, !visible.has(sub)));
-        walk(sub, name, out);
+        out.push(buildCommandEntry(sub, name, !visible.has(sub), docs));
+        walk(sub, name, out, docs);
     }
 }
 /**
@@ -50,11 +62,18 @@ function walk(cmd, prefix, out) {
  * `createProgram()`'s return value). Includes every node in the tree —
  * hidden legacy aliases (`scraps list`, …) included, flagged via `hidden`,
  * so a consumer that wants only the advertised surface can filter on it.
+ *
+ * #185 — `docs` is an already-RESOLVED `DocsUrls` (docs.ts's ladder run to
+ * completion), never computed in here: `buildSpec` stays pure/synchronous
+ * (no network, no config read) exactly as before, so every existing caller
+ * (the README-vs-spec contract test, the synthetic-tree unit tests) keeps
+ * working unchanged by simply omitting the parameter. Only `spec.ts`'s own
+ * action resolves rung 1 (a bounded fetch) before calling this.
  */
-export function buildSpec(program) {
+export function buildSpec(program, docs = {}) {
     const commands = [];
-    walk(program, '', commands);
-    return {
+    walk(program, '', commands, docs);
+    const spec = {
         specVersion: 1,
         cliVersion: program.version() ?? 'unknown',
         commands,
@@ -62,18 +81,71 @@ export function buildSpec(program) {
         errorKinds: [...ENVELOPE_KINDS],
         kindExitCodes: { ...KIND_EXIT_CODES },
     };
+    if (docs.docsUrl)
+        spec.docsUrl = docs.docsUrl;
+    if (docs.llmsUrl)
+        spec.llmsUrl = docs.llmsUrl;
+    return spec;
+}
+/**
+ * #185 — bounds the SOCKET, not just the promise, so a slow/dead server can
+ * never perceptibly slow `spec --json` down. Mirrors lib/tips.ts's
+ * FLAG_CHECK_TIMEOUT_MS for the same reason: this is the one surface allowed
+ * a network call at all, and only because it's a deliberate, once-per-
+ * invocation agent probe.
+ *
+ * This is a REAL wall-clock ceiling, not merely a promise-level one: an
+ * earlier version of this fetch used `api.publicGet` (Node's global
+ * `fetch()` + `AbortSignal.timeout()`), which bounds the promise but not the
+ * OS-level connection — against a routable-but-silently-dropping host (a
+ * dropped SYN, no RST; a realistic firewalled/air-gapped shape, the very
+ * audience this feature cites) the promise rejected at ~2005ms while the
+ * PROCESS didn't exit until ~10.5s later, waiting out Node's own internal
+ * connect-timeout. `api.probeJson` (see its own doc comment in lib/api.ts)
+ * fixes this by managing the raw socket directly — an independent timer
+ * that `req.destroy()`s the connection, never relying on the idle-based
+ * `http.request` `timeout` option. Re-measured after the fix, same
+ * black-holed host, via the exact repro command (`time env
+ * TRAWL_API_URL=https://192.0.2.1 node dist/index.js spec --json`): total
+ * process time ~2.43s/~2.47s across two runs (the 2000ms bound plus Node
+ * startup and spec-tree construction), not ~10.68s.
+ */
+const EXTERNAL_DOCS_FETCH_TIMEOUT_MS = 2_000;
+/**
+ * Rung 1 — read `externalDocs.url` off the OpenAPI document the configured
+ * API base itself serves at `/api/spec.json`. Unset (null) server-side
+ * today, so this resolves to `null` in practice; forward-looking for when
+ * it isn't. ANY failure — network error, timeout, a malformed/unexpected
+ * response shape — is indistinguishable from "not declared" here: resolves
+ * to `null`, never throws (api.probeJson's own contract — see its doc
+ * comment). `spec --json` must stay a pure, reliable, side-effect-free probe
+ * even against an unreachable or ancient server.
+ */
+async function fetchExternalDocsUrl() {
+    const data = await api.probeJson('/api/spec.json', {
+        timeoutMs: EXTERNAL_DOCS_FETCH_TIMEOUT_MS,
+    });
+    const url = data?.externalDocs?.url;
+    return typeof url === 'string' && url ? url : null;
 }
 export const spec = new Command('spec')
     .description('Print a versioned, machine-readable description of the command tree')
     .option('--json', 'Output as JSON')
-    .action((opts) => {
+    .action(async (opts) => {
     // `spec` is registered as a direct child of the program root
     // (index.ts's createProgram), so `.parent` IS that root by the time this
     // action ever runs — commander sets it in `addCommand()`. Falling back
     // to `spec` itself only matters for an isolated unit invocation (e.g. a
     // test driving `spec.parseAsync()` directly, unattached to a program).
     const root = spec.parent ?? spec;
-    const cliSpec = buildSpec(root);
+    const apiBaseUrl = getApiUrl();
+    // #185 — rung 1 (the server fetch) only runs under --json: the
+    // plain-text branch below never reads docsUrl at all, so paying a
+    // network round-trip for it would be pure waste. Rungs 2/3 (sync,
+    // no network) still apply either way via the fallback below.
+    const externalDocsUrl = opts.json ? await fetchExternalDocsUrl() : null;
+    const docs = resolveDocsUrls({ apiBaseUrl, externalDocsUrl });
+    const cliSpec = buildSpec(root, docs);
     if (opts.json) {
         json(cliSpec);
         return;

package/dist/index.d.ts

@@ -32,6 +32,15 @@ export declare function resolveCommandName(actionCommand: Command | undefined):
  */
 export declare function collectCommandNames(root: Command): string[];
 export declare function createProgram(): Command;
+/**
+ * #185 — see the `addHelpText('afterAll', …)` call above for why this is
+ * unconditional (no TTY/--json gate) and why a thrown error must never
+ * escape: commander calls this synchronously while already writing help
+ * output, so a throw here would crash a `--help` invocation, the one code
+ * path this whole feature is not allowed to touch (see docs.ts's own
+ * `docsFooterLine` for the "never a guessed URL" contract this composes).
+ */
+export declare function docsFooterText(): string;
 /**
  * True when this module is the process entrypoint (not merely imported by a test).
  *
@@ -96,6 +105,14 @@ export declare function isBareInvocation(argv: string[]): boolean;
  * overstates its own invariant is worse than no comment, because the next
  * reader trusts it.
  *
+ * #185 — `spec --json` (not plain `spec`) now DOES make its own bounded
+ * (2s), swallowed-on-failure GET for `externalDocs.url` (see spec.ts's
+ * `fetchExternalDocsUrl`) — this is not a regression of the "no mutation"
+ * invariant above (a GET mutates nothing, and any failure — including no
+ * egress at all — degrades to the sync host-derivation fallback, never a
+ * thrown error), just a second, narrower kind of side effect this guard was
+ * never meant to suppress in the first place.
+ *
  * Before this, `trawl spec --json` silently re-synced
  * `~/.claude/skills` (and `./.claude/skills`) and printed `trawl: re-synced
  * skill …` lines to stderr BEFORE the JSON payload — an agent's very first