@trawlme/cli 1.18.2 → 1.18.4

package/README.md

@@ -41,7 +41,7 @@ trawl token                                  Print the stored session JWT (for M
 ### Scraps
 
 ```
-trawl scraps list [--json] [--status <success|failure|never>] [--limit <n>] [--page <n>]
+trawl scraps list [--json] [--status <success|failure|never|running|regression>] [--limit <n>] [--page <n>]
 trawl scraps get <id> [--json]
 trawl scraps create -t <title> [-u <url>] [-r <request>] [-d <description>] [--tier <tier0|tier1|tier2|tier3|tier4>] [--json]
 trawl scraps update <id> [-t <title>] [-u <url>] [-r <request>] [-d <description>] [--cron <expr>|--no-cron] [--alert <email>|--no-alert] [--autofix|--no-autofix] [-p <json>|--params-file <path>] [--tier <tier0|tier1|tier2|tier3|tier4>] [--force-tier <tier0|tier1|tier2|tier3|tier4>] [--json]
@@ -53,18 +53,21 @@ trawl scraps history <id> [--json] [-n <limit>]
 trawl scraps run-info <hid> [--json]
 trawl scraps doctor <id> [--json] [--autofix]
 trawl scraps autofix <id> [--json]
-trawl scraps snapshot <id> [--error] [-o <file>]
+trawl scraps snapshot <id> [--error] [-o <file>] [--json]
 trawl scraps banner <id> -f <file>
 trawl scraps delete <id> [--force]         Alias: rm
 ```
 
-- `--tier` forces a proxy tier; `--force-tier` raises the proxy-tier ceiling past the auto-cap (history-gated: may be refused or cost more). `create --json`/`update --json` print the full scrap object (including the `_tierOverride` outcome) on stdout; a refused tier override exits 1 with a standard `--json` error envelope.
-- `scraps doctor` diagnoses the last run (error, failed selector, block status, page state, autofix outcome); `--autofix` includes the full autofix diff/dry-run/knowledge.
-- `scraps autofix` shows the last auto-fix attempt on its own (decision, diff, dry-run, knowledge).
-- `scraps snapshot --error` fetches the error-path snapshot instead of the normal one; `-o <file>` writes to a file instead of stdout.
+- `--tier` forces a proxy tier; `--force-tier` raises the proxy-tier ceiling past the auto-cap (history-gated: may be refused or cost more). `create --json`/`update --json` print the full scrap object (including the `_tierOverride` outcome) on stdout; a refused tier override exits 1 with a standard `--json` error envelope. When a tier was requested but the server's response carries no `_tierOverride` at all (an older server that can't confirm what actually got applied), a stderr warning is printed either way, and under `--json` the emitted object also carries `"_tierUnconfirmed": true` — the machine-readable counterpart to that warning, since a `--json` caller has no reliable reason to read stderr.
+- `scraps doctor` diagnoses the last run (error, failed selector, block status, page state, autofix outcome); `--autofix` includes the full autofix diff/dry-run/knowledge. A run that is still in flight (`status: null`, server-side) shows a `running` badge, never `failed`; a run whose item count regressed vs baseline (`statusDetail: "regression"`) shows its own amber `regression` badge, never `failed` either (`scraps list`/`get` show the same distinction — a `▼` icon, not the red `✗` a genuine failure gets).
+- `scraps autofix` shows the last auto-fix attempt on its own (decision, diff, dry-run, knowledge). `--json` on a scrap that has **never run** returns `{"status":"no_runs"}` (exit 0) — distinct from `null`, which means a run exists but had no auto-fix attempt.
+- `scraps snapshot --error` fetches the error-path snapshot instead of the normal one; `-o <file>` writes to a file instead of stdout. On a scrap that has **never run**: `--json` returns `{"status":"no_runs"}` (exit 0), matching `doctor`/`autofix`; `-o <file>` (without `--json`) exits 4 (`not_found`) instead of silently exiting 0 with nothing written — a script checking the exit code alone must be able to tell "no file was produced" from success. `--json` takes priority when both are passed.
 - `scraps history` lists past runs (newest first); `scraps run-info <hid>` shows details of a single run from that history.
-- `scraps data` returns the last persisted run payload (no execute quota); `--fresh` runs the scrap live instead (consumes execute quota); `--errors` shows the last run's error detail. `[]` on stdout means a genuine zero-item successful run — a scrap that has never run, whose last run failed, or whose payload aged out of retention returns a `--json` error envelope (exit 4/1/4 respectively) instead.
+- `scraps data` returns the last persisted run payload (no execute quota); `--fresh` runs the scrap live instead (consumes execute quota); `--errors` shows the last run's error detail (`--json` on a never-run scrap returns `{"status":"no_runs"}`, exit 0, matching `doctor --json`). `[]` on stdout means a genuine zero-item successful run — a scrap that has never run, whose last run failed, or whose payload aged out of retention returns a `--json` error envelope (exit 4/1/4 respectively) instead. Two more honest states: a run still **in flight** (`status: null` server-side) returns a `kind:"in_progress"` error envelope (exit 1, "retry shortly" — never suggests `--fresh`, which would just 429 against the run already holding the lock); a run whose item count **regressed** vs baseline (`statusDetail: "regression"`) still returns the real, non-empty items on stdout (exit 0) plus a stderr warning pointing at `scraps doctor <id>` — the data itself is genuine even though the run is flagged.
 - `scraps get` (and anything reading through it, like `scraps data`'s default path) embeds only the newest 100 history rows on the returned scrap object — `scraps run-info` and `scraps doctor` fetch a single run directly and are unaffected by that cap.
+- `scraps banner -f <file>` only accepts `png`/`jpg`/`jpeg`/`webp`; any other extension is a usage error (exit 2) instead of silently uploading the file under a fabricated `image/png` Content-Type.
+- **Long-running scrap execute (`scraps run`, `data --fresh`, `trigger --wait`):** these hit the same server-side scrap-execute path, which can legitimately take 30–250s (proxy tier escalation, AI-fix retries) — the CLI arms a 300s timeout for exactly these three call sites instead of the generic 30s default. `TRAWL_TIMEOUT` (see below) still overrides ALL requests, including these — set it if you need a tighter or looser ceiling than 300s for a long-running scrap.
+- **`--watch` is poll-based, not a live stream:** the activities SSE endpoint has no backlog and, for the default async `trigger` (no `--wait`), runs in a separate cron-consumer pod whose events never reach the API pod holding the SSE connection — a naive "await the run, then open SSE" shows nothing. `scraps run --watch` and `scraps trigger --watch` instead poll `GET /api/scraps/:id` (terminal status) and the activities REST list until the run finishes, printing each new activity line as it appears; a run that never reaches a terminal status within 300s prints an honest timeout notice pointing at `scraps doctor <id>`. `scraps watch <id>` (the standalone command, no trigger) is unchanged — it still opens the live SSE stream directly.
 
 ### Scrap accounts
 
@@ -89,7 +92,7 @@ trawl skills uninstall [<skill>] [--local]            Remove
 trawl skills update [<skill>] [--local] [--force]     Reinstall (force sync with CLI version)
 ```
 
-Skills auto-update when you upgrade the CLI — no need to re-install manually — but it is not silent: it prints an honest `trawl: re-synced skill "<name>" (<scope>) <old> → <new>` line to stderr whenever it rewrites a skill dir, so a rewrite is never invisible. Opt out with `TRAWL_SKILLS_SYNC=0`.
+Skills auto-update when you upgrade the CLI — no need to re-install manually — but it is not silent: it prints an honest `trawl: re-synced skill "<name>" (<scope>) <old> → <new>` line to stderr whenever it rewrites a skill dir, so a rewrite is never invisible. If a bundled skill is renamed or dropped between CLI versions, the old install is swept too — a `trawl: removed orphaned skill "<name>" (<scope>) — no longer bundled with this CLI version` line to stderr, so a stale skill teaching outdated CLI usage never lingers silently. Only marker-owned dirs (installed by trawl itself) are ever touched by either line. Opt out with `TRAWL_SKILLS_SYNC=0`.
 
 A pre-existing skill directory that trawl did not install itself (no `.version` marker) is never touched — `install`/`update` refuse to overwrite it and require `--force` to proceed. This applies to the CLI-upgrade auto-sync too (it silently skips marker-less dirs rather than refusing, since there is no interactive user to show a refusal to).
 
@@ -143,7 +146,7 @@ Every command exits with one of these codes — scripts and agents driving the C
 | Code | Meaning                                                                 |
 |------|--------------------------------------------------------------------------|
 | `0`  | Success                                                                 |
-| `1`  | Unknown/generic error (an unmapped failure — API errors other than 401/404, an unhandled bug) |
+| `1`  | Unknown/generic error (an unmapped failure — API errors other than 401/404, an unhandled bug — or a business-logic refusal like `scraps data`'s `run_failed`/`in_progress` states) |
 | `2`  | Usage error (bad flag/value, invalid ID, missing required argument, unknown option/command) |
 | `3`  | Auth error (not logged in, or the session token is expired/invalid — run `trawl login`) |
 | `4`  | Not found (no such resource, or — for `scraps data` — no persisted payload to read) |
@@ -160,7 +163,7 @@ Under `--json`, a failing command emits a single error envelope on stdout — `{
 | `TRAWL_TELEMETRY`     | Set to `0` to disable telemetry for the current session           |
 | `DO_NOT_TRACK`        | Set to `1` to disable telemetry (cross-vendor convention, https://consoledonottrack.com) — same effect as `TRAWL_TELEMETRY=0` |
 | `TRAWL_CONFIG_DIR`    | Override where the config file (token, API URL, telemetry state) is stored — useful for hermetic CI runs or concurrent `trawl login`s that must not share one on-disk file |
-| `TRAWL_TIMEOUT`       | Override the per-request fetch timeout in milliseconds (default `30000`) |
+| `TRAWL_TIMEOUT`       | Override the per-request fetch timeout in milliseconds (default `30000`; `300000` for `scraps run`/`data --fresh`/`trigger --wait` — this env var always wins over that longer default too) |
 | `TRAWL_SKILLS_SYNC`   | Set to `0` to disable the startup skills auto-sync entirely       |
 
 Session override (no `trawl login` mutation, ideal for CI/QA against another env):

package/dist/commands/doctor.js

@@ -83,13 +83,26 @@ export function pickFix(fix) {
  */
 export function formatDoctor(scrapTitle, run, fix = null, scrapId) {
     const lines = [];
-    // Header + status badge
+    // Header + status badge. status:null means a run is IN FLIGHT (node
+    // persists {status:null, statusDetail:null, inFlight:true} the moment a
+    // run starts) — that must never render as "failed". (#88 item 1)
+    //
+    // #91 LOW — a regression row (status:false, statusDetail:'regression') was
+    // falling into the same red "failed" bucket as a genuine failure, then
+    // showing a contradictory "Regression: X vs Y" detail line right below it.
+    // A regression run's write actually SUCCEEDED (the item count just dropped
+    // vs baseline, detected by an async patch afterward — see #88 item 2 in
+    // scraps.ts) — it needs its own distinct badge, not "failed".
     const ok = run.status === true;
     const badge = ok
         ? chalk.green('● success')
-        : run.statusDetail === 'empty'
-            ? chalk.yellow('● empty')
-            : chalk.red('● failed');
+        : run.status === null
+            ? chalk.cyan('● running')
+            : run.statusDetail === 'empty'
+                ? chalk.yellow('● empty')
+                : run.statusDetail === 'regression'
+                    ? chalk.hex('#FFA500')('● regression')
+                    : chalk.red('● failed');
     lines.push(`${chalk.bold(scrapTitle)}  ${badge}${run.statusDetail ? ` (${run.statusDetail})` : ''}`);
     lines.push(chalk.dim(`  Run ID:  ${run._id}`));
     // Error message — an honest accept-wall string for known-walled scraps

package/dist/commands/scraps.d.ts

@@ -1,2 +1,27 @@
 import { Command } from 'commander';
 export declare const scraps: Command;
+/**
+ * #91 P1 — replaces "await the run to completion, THEN open the activities
+ * SSE stream" (which showed NOTHING: the activities SSE
+ * (GET /api/scraps/:id/activities/stream) is backed by an in-process
+ * EventEmitter with no backlog — trawl_node
+ * modules/activities/services/activities.service.js — so by the time a
+ * synchronous run has already finished there is nothing left to emit; and
+ * the async `trigger` default enqueues the run onto a durable job queue a
+ * SEPARATE cron-consumer pod drains, whose in-process emitter never reaches
+ * the API pod holding the SSE connection at all).
+ *
+ * Instead this polls two REST reads that are BOTH Mongo-backed (not
+ * in-process), so they work no matter which pod actually executed the run:
+ *  - GET /api/scraps/:id/activities?history=<hid>&limit=20 — the SAME
+ *    activities-list endpoint `doctor`'s fetchRunAndFix already calls
+ *    (src/commands/doctor.ts) — prints each new activity line once the new
+ *    run's history id is known.
+ *  - GET /api/scraps/:id — history[0].status/statusDetail, the SAME
+ *    terminal-status signal `lastStatus()` above already trusts (status:null
+ *    === in flight, #88 item 1) to know when the run is done.
+ */
+export declare function pollRunProgress(id: string, beforeHistoryId: string | undefined, opts?: {
+    intervalMs?: number;
+    timeoutMs?: number;
+}): Promise<void>;

package/dist/commands/scraps.js

@@ -1,7 +1,7 @@
 import { Command } from 'commander';
 import chalk from 'chalk';
 import { oraPromise } from 'ora';
-import { api } from '../lib/api.js';
+import { api, LONG_RUN_TIMEOUT_MS } from '../lib/api.js';
 import { table, json } from '../lib/format.js';
 import { promptPassword } from '../lib/prompt.js';
 import { validateObjectId } from '../lib/validate.js';
@@ -20,8 +20,23 @@ function usageError(message, opts = {}) {
 }
 function lastStatus(scrap) {
     const last = scrap.history?.[0];
-    if (!last || last.status === null || last.status === undefined)
+    if (!last)
         return 'never';
+    // status:null means a run is IN FLIGHT (node persists {status:null,
+    // statusDetail:null, inFlight:true} the moment a run starts) — that is NOT
+    // the same thing as "this scrap has never run" (no history row at all).
+    // (#88 item 1)
+    if (last.status === null)
+        return 'running';
+    if (last.status === undefined)
+        return 'never';
+    // #91 LOW — same regression-as-failure bucketing bug as doctor's badge
+    // (formatDoctor): a regression row's write actually succeeded (item count
+    // just dropped vs baseline, flagged by an async patch afterward — #88 item
+    // 2), so it must not collapse into the same 'failure' bucket a genuine
+    // failed run gets.
+    if (last.status === false && last.statusDetail === 'regression')
+        return 'regression';
     return last.status === true ? 'success' : 'failure';
 }
 function lastRun(scrap) {
@@ -41,6 +56,12 @@ function statusIcon(status) {
         return chalk.green('✓');
     if (status === 'failure')
         return chalk.red('✗');
+    if (status === 'running')
+        return chalk.cyan('↻');
+    // #91 LOW — distinct amber icon, never the red ✗ a genuine failure gets
+    // (mirrors doctor's "● regression" badge — see formatDoctor).
+    if (status === 'regression')
+        return chalk.hex('#FFA500')('▼');
     return chalk.dim('—');
 }
 export const scraps = new Command('scraps').description('Manage scraps');
@@ -61,39 +82,151 @@ async function watchActivities(id) {
         }
     }
 }
+/**
+ * #91 P1 — snapshot the current top history entry BEFORE triggering a run,
+ * so pollRunProgress can later detect the NEW run once the server pushes it
+ * (node persists a fresh {status:null, inFlight:true} row the moment a run
+ * starts — see HistorysService.create in trawl_node — so a changed
+ * history[0]._id is an honest "the new run has begun" signal). Best-effort:
+ * a failed lookup just means the poll treats the very first history row it
+ * sees as new, which is still correct for a scrap that has never run.
+ */
+async function captureBeforeHistoryId(id) {
+    try {
+        const scrap = await api.get(`/api/scraps/${id}`);
+        return scrap.history?.[0]?._id;
+    }
+    catch {
+        return undefined;
+    }
+}
+function sleep(ms) {
+    return new Promise((resolve) => setTimeout(resolve, ms));
+}
+const POLL_INTERVAL_MS = 2000;
+// Mirrors LONG_RUN_TIMEOUT_MS (#91 item 1) — the server-side worst case this
+// polls for is the same one that timeout was sized for.
+const POLL_TIMEOUT_MS = LONG_RUN_TIMEOUT_MS;
+/**
+ * #91 P1 — replaces "await the run to completion, THEN open the activities
+ * SSE stream" (which showed NOTHING: the activities SSE
+ * (GET /api/scraps/:id/activities/stream) is backed by an in-process
+ * EventEmitter with no backlog — trawl_node
+ * modules/activities/services/activities.service.js — so by the time a
+ * synchronous run has already finished there is nothing left to emit; and
+ * the async `trigger` default enqueues the run onto a durable job queue a
+ * SEPARATE cron-consumer pod drains, whose in-process emitter never reaches
+ * the API pod holding the SSE connection at all).
+ *
+ * Instead this polls two REST reads that are BOTH Mongo-backed (not
+ * in-process), so they work no matter which pod actually executed the run:
+ *  - GET /api/scraps/:id/activities?history=<hid>&limit=20 — the SAME
+ *    activities-list endpoint `doctor`'s fetchRunAndFix already calls
+ *    (src/commands/doctor.ts) — prints each new activity line once the new
+ *    run's history id is known.
+ *  - GET /api/scraps/:id — history[0].status/statusDetail, the SAME
+ *    terminal-status signal `lastStatus()` above already trusts (status:null
+ *    === in flight, #88 item 1) to know when the run is done.
+ */
+export async function pollRunProgress(id, beforeHistoryId, opts = {}) {
+    const intervalMs = opts.intervalMs ?? POLL_INTERVAL_MS;
+    const timeoutMs = opts.timeoutMs ?? POLL_TIMEOUT_MS;
+    console.log(chalk.dim('Live activity streaming has no signal for this run (async/cross-pod) — polling for progress instead…\n'));
+    const deadline = Date.now() + timeoutMs;
+    const seen = new Set();
+    let first = true;
+    while (Date.now() < deadline) {
+        if (!first)
+            await sleep(intervalMs);
+        first = false;
+        let scrap;
+        try {
+            scrap = await api.get(`/api/scraps/${id}`);
+        }
+        catch {
+            continue; // transient — keep polling rather than aborting the wait
+        }
+        const last = scrap.history?.[0];
+        if (!last?._id || last._id === beforeHistoryId)
+            continue; // new run not recorded yet
+        try {
+            const activities = await api.get(`/api/scraps/${id}/activities?history=${last._id}&limit=20`);
+            // Server returns newest-first — print unseen ones oldest-first.
+            for (const a of [...(activities ?? [])].reverse()) {
+                const key = a._id ?? `${a.createdAt}:${a.message}`;
+                if (seen.has(key))
+                    continue;
+                seen.add(key);
+                const time = new Date(a.createdAt).toLocaleTimeString();
+                console.log(`${chalk.dim(`[${time}]`)} ${a.message}`);
+            }
+        }
+        catch {
+            // best-effort — terminal detection below still works without activity lines
+        }
+        if (last.status !== null) {
+            const outcome = last.statusDetail ?? (last.status ? 'success' : 'failure');
+            console.log(chalk.dim(`Run finished: ${outcome}`));
+            return;
+        }
+    }
+    console.log(chalk.yellow(`⚠ Timed out waiting for the run to finish — check status with: trawl scraps doctor ${id}`));
+}
 // list
 scraps
     .command('list')
     .alias('ls')
     .description('List all scraps')
     .option('--json', 'Output as JSON')
-    .option('--status <status>', 'Filter by last run status (success|failure|never)')
-    .option('--limit <n>', 'Show only the first N results', (v) => parseInt(v, 10))
-    .option('--page <n>', 'Fetch a specific page only (50 per page, no auto-pagination)', (v) => parseInt(v, 10))
+    .option('--status <status>', 'Filter by last run status (success|failure|never|running|regression)')
+    // #88 item 8 — no custom parser here (unlike the old `(v) => parseInt(v,
+    // 10)`): a bad value like "abc" used to silently become NaN, which then
+    // sailed straight through `Number.isInteger`-less checks and into
+    // `.slice(0, NaN)` (silently truncates to 0 rows) or `?page=NaN` (silently
+    // sent to the server) — never a usage error. Keeping the raw string here
+    // lets the validation below mirror `history`'s own --limit check exactly
+    // (~line 660) and report the actual bad input in the error message.
+    .option('--limit <n>', 'Show only the first N results')
+    .option('--page <n>', 'Fetch a specific page only (50 per page, no auto-pagination)')
     .action(async (opts, cmd) => {
     // Guard: --limit and --page are mutually exclusive
     if (opts.limit !== undefined && opts.page !== undefined) {
         usageError('--limit and --page are mutually exclusive. Use one or the other.', { json: opts.json });
         return;
     }
+    let limit;
+    if (opts.limit !== undefined) {
+        limit = Number(opts.limit);
+        if (!Number.isInteger(limit) || limit <= 0) {
+            usageError(`Invalid --limit "${opts.limit}" (expected a positive integer)`, { json: opts.json });
+            return;
+        }
+    }
+    let page;
+    if (opts.page !== undefined) {
+        page = Number(opts.page);
+        if (!Number.isInteger(page) || page <= 0) {
+            usageError(`Invalid --page "${opts.page}" (expected a positive integer)`, { json: opts.json });
+            return;
+        }
+    }
     let data;
     try {
         data = await oraPromise(async () => {
-            if (opts.page !== undefined) {
+            if (page !== undefined) {
                 // Single-page mode: explicit page requested, no loop
-                const pageNum = opts.page;
-                return api.get(`/api/scraps?perPage=50&page=${pageNum}`);
+                return api.get(`/api/scraps?perPage=50&page=${page}`);
             }
             // Fetch-all mode: paginate until a page returns < 200 items
             const perPage = 200;
             let result = [];
-            let page = 1;
+            let pageNum = 1;
             while (true) {
-                const batch = await api.get(`/api/scraps?perPage=${perPage}&page=${page}`);
+                const batch = await api.get(`/api/scraps?perPage=${perPage}&page=${pageNum}`);
                 result = result.concat(batch);
                 if (batch.length < perPage)
                     break;
-                page++;
+                pageNum++;
             }
             return result;
         }, 'Fetching scraps…');
@@ -126,7 +259,6 @@ scraps
     if (opts.status)
         data = data.filter((s) => lastStatus(s) === opts.status);
     const totalMatched = data.length;
-    const limit = opts.limit;
     const rows = limit !== undefined ? data.slice(0, limit) : data;
     if (opts.json)
         return json(rows);
@@ -206,6 +338,23 @@ function warnIfUnconfirmedTier(data, tierWasRequested, id) {
         return;
     console.error(chalk.yellow(`  ⚠ Server did not confirm the tier change (older server) — verify with: trawl scraps get ${id}`));
 }
+/**
+ * #88 item 3 — the --json machine-readable counterpart to
+ * warnIfUnconfirmedTier's stderr warning above. A --json caller (an agent
+ * scripting this CLI) has no reliable reason to read stderr — that channel
+ * is advisory-only everywhere else in this CLI, and stdout must stay the
+ * sole payload. Without this, the ONLY signal that the server never
+ * confirmed the tier change was a string on stderr, invisible to any --json
+ * consumer parsing stdout alone. Adds `_tierUnconfirmed: true` to the
+ * emitted object under EXACTLY the same condition warnIfUnconfirmedTier
+ * warns on (tier requested, response carries no `_tierOverride`) — never
+ * fabricated, never present otherwise.
+ */
+function withTierUnconfirmed(data, tierWasRequested) {
+    if (data._tierOverride || !tierWasRequested)
+        return data;
+    return { ...data, _tierUnconfirmed: true };
+}
 /** #86 finding 5 — the standard error envelope for a refused tier override,
  * routed through the same reportError() central formatting path used
  * everywhere else (exit 1: a business-logic refusal, not a usage error). */
@@ -247,7 +396,7 @@ scraps
             process.exitCode = reportTierRefusal(data, true);
             return;
         }
-        json(data);
+        json(withTierUnconfirmed(data, tierWasRequested));
         return;
     }
     console.log(chalk.dim(`  Title: ${data.title}`));
@@ -360,7 +509,7 @@ scraps
             process.exitCode = reportTierRefusal(data, true);
             return;
         }
-        json(data);
+        json(withTierUnconfirmed(data, tierWasRequested));
         return;
     }
     const shown = data;
@@ -382,15 +531,20 @@ scraps
 scraps
     .command('run <id>')
     .description('Run a scrap')
-    .option('-w, --watch', 'Stream activities after launching')
+    .option('-w, --watch', 'Show progress after launching (polls — see `trawl scraps trigger --watch`, #91)')
     .action(async (id, opts) => {
     validateObjectId(id);
-    await oraPromise(() => api.get(`/api/scraps/load/${id}`), {
+    // #91 P1 — captured BEFORE launching so pollRunProgress can tell "the run
+    // that's about to finish" apart from whatever the last run happened to be.
+    const beforeHistoryId = opts.watch ? await captureBeforeHistoryId(id) : undefined;
+    // #91 P0 — GET /api/scraps/load/:id runs the scrap synchronously
+    // server-side (30-250s); the 30s default was aborting it mid-flight.
+    await oraPromise(() => api.get(`/api/scraps/load/${id}`, { timeoutMs: LONG_RUN_TIMEOUT_MS }), {
         text: 'Launching scrap…',
         successText: 'Scrap launched',
     });
     if (opts.watch) {
-        await watchActivities(id);
+        await pollRunProgress(id, beforeHistoryId);
     }
 });
 // #70 — render an items array either as a table summary or --json. Shared by
@@ -440,8 +594,12 @@ scraps
     if (opts.errors) {
         const result = await fetchRunAndFix(id);
         if (!result) {
+            // #88 item 7 — unified no-runs shape with `doctor --json`: a bare
+            // `null` was indistinguishable from any other absent-payload state
+            // (a scrap CAN legitimately have a null-ish result elsewhere); an
+            // explicit `{status:"no_runs"}` object is unambiguous everywhere.
             if (opts.json) {
-                json(null);
+                json({ status: 'no_runs' });
                 return;
             }
             console.log(chalk.dim('No runs yet.'));
@@ -466,7 +624,9 @@ scraps
     // burning execute quota and 429ing if a run is already in flight. A user
     // or agent "just reading data" must never trigger that by accident.
     if (opts.fresh) {
-        const loaded = await oraPromise(() => api.get(`/api/scraps/load/${id}`), {
+        // #91 P0 — same long-run endpoint as `scraps run` (30-250s server-side);
+        // the 30s default was aborting it mid-flight.
+        const loaded = await oraPromise(() => api.get(`/api/scraps/load/${id}`, { timeoutMs: LONG_RUN_TIMEOUT_MS }), {
             text: 'Launching a fresh scrap run (consumes execute quota)…',
             successText: 'Fresh run complete',
         });
@@ -502,16 +662,33 @@ scraps
         reportDataState(`Scrap ${id} has never run. Run it first (trawl scraps run ${id}) or pass --fresh to launch one now.`, 4, 'not_found', opts.json);
         return;
     }
+    // #88 item 1 — status:null is an IN-FLIGHT run (node persists
+    // {status:null, statusDetail:null, inFlight:true} the moment a run
+    // starts, and only flips status/statusDetail once it finishes). That is
+    // neither "never run" nor "the last run failed" — a caller reading data
+    // mid-run needs an honest "wait" signal. Never suggest --fresh here: a
+    // run already holds the server-side distributed lock, so --fresh would
+    // just 429 against it.
+    if (last.status === null) {
+        reportDataState(`Run in progress for ${id} — retry shortly.`, 1, 'in_progress', opts.json);
+        return;
+    }
     // #86 review — node persists status=false for a GENUINE zero-item run
     // too (historys schema: status boolean|null + statusDetail
     // success/error/empty/regression; a zero-item run is status=false +
     // statusDetail='empty', and the embedded history rows from GET
     // /api/scraps/:id include statusDetail via the repository populate
     // select). An 'empty' run is the one case [] is FOR — only a real
-    // failure (error/regression/unknown detail) gets the run_failed
-    // envelope.
+    // failure (error/unknown detail) gets the run_failed envelope.
+    //
+    // #88 item 2 — statusDetail='regression' is ALSO status=false (an async
+    // patch flips it after item count dropped vs baseline), but the row's
+    // `data` still holds REAL, non-empty items — the write that persisted
+    // them succeeded before the regression was even detected. Treating it as
+    // run_failed would hide genuine data behind a false negative.
     const isEmptyRun = last.status === false && last.statusDetail === 'empty';
-    if (last.status === false && !isEmptyRun) {
+    const isRegression = last.status === false && last.statusDetail === 'regression';
+    if (last.status === false && !isEmptyRun && !isRegression) {
         reportDataState(`Last run failed — see: trawl scraps data ${id} --errors`, 1, 'run_failed', opts.json);
         return;
     }
@@ -533,9 +710,20 @@ scraps
             renderScrapItems([], opts.json);
             return;
         }
-        reportDataState(`No persisted data for the last successful run of ${id} — it aged out of retention. Pass --fresh to launch a new run.`, 4, 'not_found', opts.json);
+        // #88 item 2 — a regression row whose payload aged out of retention has
+        // nothing left to show either; fall through to the SAME honest
+        // aged-out envelope a normal successful row would get (never fabricate
+        // items, never silently succeed).
+        reportDataState(`No persisted data for the last run of ${id} — it aged out of retention. Pass --fresh to launch a new run.`, 4, 'not_found', opts.json);
         return;
     }
+    // #88 item 2 — a regression row's items are REAL (the write succeeded
+    // before the async patch flagged the drop) — return them on stdout
+    // (exit 0, both modes) with an honest stderr warning pointing at the
+    // diagnostic command, instead of hiding genuine data behind run_failed.
+    if (isRegression) {
+        console.error(chalk.yellow(`⚠ Item count regressed vs baseline for the last run of ${id} — see: trawl scraps doctor ${id}`));
+    }
     renderScrapItems(items, opts.json);
 });
 // history — list past runs for a scrap
@@ -633,7 +821,6 @@ scraps
         usageError(`File not found: ${opts.file}`);
         return;
     }
-    const fileBuffer = readFileSync(opts.file);
     const filename = basename(opts.file);
     const ext = filename.split('.').pop()?.toLowerCase() ?? '';
     const mimeMap = {
@@ -642,7 +829,15 @@ scraps
         jpeg: 'image/jpeg',
         webp: 'image/webp',
     };
-    const mimeType = mimeMap[ext] ?? 'image/png';
+    // #91 P2 — an unsupported/unknown extension silently became image/png
+    // (uploading the raw bytes of, say, a .gif or .pdf under an image/png
+    // Content-Type — a lie about the actual file's format). Refuse instead.
+    const mimeType = mimeMap[ext];
+    if (!mimeType) {
+        usageError(`Unsupported image type "${ext ? `.${ext}` : filename}" — use png, jpg, or webp.`);
+        return;
+    }
+    const fileBuffer = readFileSync(opts.file);
     const blob = new Blob([fileBuffer], { type: mimeType });
     const formData = new FormData();
     formData.append('banner', blob, filename);
@@ -663,20 +858,27 @@ scraps
 scraps
     .command('trigger <id>')
     .description('Launch a scrap as a background worker (returns immediately)')
-    .option('-w, --watch', 'Stream activities after triggering')
+    .option('-w, --watch', 'Poll for progress after triggering (#91 — the default async run happens in a separate cron pod; activities SSE never reaches it)')
     .option('--wait', 'Run synchronously and wait for the result (legacy behaviour)')
     .action(async (id, opts) => {
     validateObjectId(id);
+    // #91 P1 — captured BEFORE triggering so pollRunProgress can tell "the
+    // run we just triggered" apart from whatever the last run happened to be.
+    const beforeHistoryId = opts.watch ? await captureBeforeHistoryId(id) : undefined;
     // #50 — default async: the backend (#1313) kicks off the run and returns a
     // 'queued' envelope immediately instead of holding the connection for the
     // whole run. --wait restores the old synchronous round-trip.
     const path = opts.wait ? `/api/scraps/worker/${id}` : `/api/scraps/worker/${id}?wait=false`;
-    await oraPromise(() => api.post(path), {
+    // #91 P0 — the synchronous --wait branch runs the scrap server-side
+    // (30-250s), same as `scraps run`; the 30s default was aborting it
+    // mid-flight. The async (default) POST returns almost immediately, so it
+    // keeps the 30s default.
+    await oraPromise(() => (opts.wait ? api.post(path, undefined, { timeoutMs: LONG_RUN_TIMEOUT_MS }) : api.post(path)), {
         text: opts.wait ? 'Running worker…' : 'Triggering worker…',
         successText: opts.wait ? 'Worker run complete' : 'Worker triggered',
     });
     if (opts.watch)
-        await watchActivities(id);
+        await pollRunProgress(id, beforeHistoryId);
 });
 // account subcommand group
 const account = scraps
@@ -881,14 +1083,19 @@ scraps
     validateObjectId(id);
     const result = await fetchRunAndFix(id);
     if (!result) {
+        // #88 item 7 — unified no-runs shape with `doctor --json` / `data
+        // --errors --json`: a never-run scrap is a distinct, nameable state,
+        // not the same bare `null` a run-with-no-fix-attempt returns below.
         if (opts.json) {
-            json(null);
+            json({ status: 'no_runs' });
             return;
         }
         console.log(chalk.dim('No runs yet.'));
         return;
     }
     if (!result.fix) {
+        // A run DID happen, it just had no autofix attempt — genuinely "no
+        // data", unlike the never-run case above.
         if (opts.json) {
             json(null);
             return;
@@ -906,11 +1113,26 @@ scraps
     .description('Download captured page HTML for the last run of a scrap')
     .option('--error', 'Fetch the error-path snapshot (errorSnapshot.html)')
     .option('-o, --out <file>', 'Write HTML to a file instead of stdout')
+    .option('--json', 'On a never-run scrap, output {"status":"no_runs"} (exit 0) instead of prose')
     .action(async (id, opts) => {
     validateObjectId(id);
     const scrap = await api.get(`/api/scraps/${id}`);
     const hid = scrap.history?.[0]?._id;
     if (!hid) {
+        // #91 P2 — same no-runs contract as `doctor`/`autofix`/`data --errors`
+        // (`{status:"no_runs"}`, exit 0) under --json.
+        if (opts.json) {
+            json({ status: 'no_runs' });
+            return;
+        }
+        // `-o` explicitly asked for a file to be written. Silently exiting 0
+        // with nothing written (and only a dim console line) is indistinguishable
+        // from success to a script checking the exit code alone — give it the
+        // SAME not_found envelope `scraps data`'s own never-run case uses.
+        if (opts.out) {
+            reportDataState(`Scrap ${id} has never run — nothing to write to ${opts.out}.`, 4, 'not_found', false);
+            return;
+        }
         console.log(chalk.dim('No runs yet.'));
         return;
     }

package/dist/commands/skills.js

@@ -31,7 +31,16 @@ skills
     for (const name of bundled) {
         const userInstalled = isSkillInstalled(name, 'user');
         const localInstalled = isSkillInstalled(name, 'local');
-        const installedVersion = getInstalledVersion(name, 'user') ?? getInstalledVersion(name, 'local');
+        // #91 — same EISDIR class as autoUpdateInstalledSkills/installSkill: an
+        // unreadable `.version` marker on one skill must not crash the whole
+        // listing before the other skills are shown.
+        let installedVersion;
+        try {
+            installedVersion = getInstalledVersion(name, 'user') ?? getInstalledVersion(name, 'local');
+        }
+        catch {
+            installedVersion = null;
+        }
         const stale = installedVersion && installedVersion !== version;
         const tag = userInstalled
             ? localInstalled

package/dist/commands/token.js

@@ -1,7 +1,7 @@
 import { Command } from 'commander';
 import chalk from 'chalk';
 import { getToken } from '../lib/config.js';
-import { ApiError, notLoggedInError } from '../lib/api.js';
+import { AuthError, notLoggedInError } from '../lib/api.js';
 import { reportError } from '../lib/errors.js';
 import { decodeExp } from '../lib/jwt.js';
 export const token = new Command('token')
@@ -21,7 +21,10 @@ export const token = new Command('token')
     const exp = decodeExp(stored);
     const nowSeconds = Math.floor(Date.now() / 1000);
     if (exp !== null && exp < nowSeconds) {
-        process.exitCode = reportError(new ApiError(401, 'Session token expired. Run: trawl login to refresh.'));
+        // Decoded entirely client-side (no HTTP call made) — AuthError, not a
+        // fabricated ApiError(401): the server never actually said this. (#88
+        // item 4)
+        process.exitCode = reportError(new AuthError('Session token expired. Run: trawl login to refresh.'));
         return;
     }
     // Print the raw token first (so it can be piped / copied)

package/dist/index.d.ts

@@ -6,6 +6,15 @@ import { Command } from 'commander';
  * --url) don't start with '-' and would otherwise survive an argv filter and
  * leak to PostHog (#67). Falls back to 'unknown' when no command resolved
  * (e.g. an error thrown before any action ran).
+ *
+ * #88 item 9 — walks the FULL parent chain, not just the immediate parent:
+ * the old one-level join (`${parent.name()} ${own.name()}`) resolved a
+ * 4-deep command like `scraps account session set` down to just "session
+ * set", silently dropping "scraps account". The root program node (the
+ * 'trawl' Command itself, which has no `.parent`) is excluded from the
+ * chain — matching the pre-existing convention that a direct child of the
+ * root (e.g. `scraps list`, `telemetry on`) is named relative to its
+ * immediate group, never prefixed with the program name.
  */
 export declare function resolveCommandName(actionCommand: Command | undefined): string;
 /**
@@ -33,6 +42,14 @@ export declare function isHelpOrVersion(argv: string[]): boolean;
  * "argv scan, never trust flag values" caveat as isHelpOrVersion: positional
  * values are never mistaken for `--json` since they don't equal the literal
  * string. (#86 finding 3)
+ *
+ * #88 item 5 — only scans tokens BEFORE the first bare `--`. Commander treats
+ * `--` as "end of options": everything after it is a positional operand, not
+ * a flag, even if the literal text is `--json`. `scraps list -- --json`
+ * passes `--json` as an (excess) positional argument, not the flag — an
+ * unscoped `argv.includes('--json')` would still match it and wrongly emit a
+ * JSON envelope for what is actually a plain usage error with no --json
+ * requested at all.
  */
 export declare function hasJsonFlag(argv: string[]): boolean;
 /**
@@ -50,4 +67,22 @@ export declare function hasJsonFlag(argv: string[]): boolean;
  * walks the whole tree and installs it everywhere. (#86 finding 3)
  */
 export declare function applyExitOverride(cmd: Command): void;
+/**
+ * #88 item 6 — best-effort command-name recovery for a commander parse error
+ * that happened BEFORE any action ran, so `currentCommand` (the preAction
+ * hook's resolved command) is still undefined — e.g. an unknown option on an
+ * otherwise-valid subcommand, or excess arguments. Without this, EVERY such
+ * failure previously collapsed to resolveCommandName(undefined) === 'unknown'
+ * — and since 'unknown' was never itself allowlisted at the
+ * registerAllowedCommands call site (see runCli below), that capture was
+ * silently dropped by captureCommand's allowlist check: a dead branch that
+ * looked like it reported telemetry but never actually did.
+ *
+ * Matches ONLY a name already present in `allowedNames` (the real command
+ * tree, from collectCommandNames) — never invents one from raw argv text.
+ * Checks the two-token form first (`scraps boom`) since most usage errors
+ * happen on a nested leaf command; falls back to the single top-level token,
+ * then to 'unknown' (now itself allowlisted, so that capture fires too).
+ */
+export declare function bestEffortCommandName(argv: string[], allowedNames: readonly string[]): string;
 export declare function runCli(argv?: string[]): Promise<void>;

package/dist/index.js

@@ -19,13 +19,29 @@ const pkg = JSON.parse(readFileSync(join(__dirname, '..', 'package.json'), 'utf8
  * --url) don't start with '-' and would otherwise survive an argv filter and
  * leak to PostHog (#67). Falls back to 'unknown' when no command resolved
  * (e.g. an error thrown before any action ran).
+ *
+ * #88 item 9 — walks the FULL parent chain, not just the immediate parent:
+ * the old one-level join (`${parent.name()} ${own.name()}`) resolved a
+ * 4-deep command like `scraps account session set` down to just "session
+ * set", silently dropping "scraps account". The root program node (the
+ * 'trawl' Command itself, which has no `.parent`) is excluded from the
+ * chain — matching the pre-existing convention that a direct child of the
+ * root (e.g. `scraps list`, `telemetry on`) is named relative to its
+ * immediate group, never prefixed with the program name.
  */
 export function resolveCommandName(actionCommand) {
     if (!actionCommand)
         return 'unknown';
-    return actionCommand.parent
-        ? `${actionCommand.parent.name()} ${actionCommand.name()}`
-        : actionCommand.name();
+    const chain = [];
+    let current = actionCommand;
+    while (current && current.parent) {
+        chain.unshift(current.name());
+        current = current.parent;
+    }
+    // `current` is now the root (no parent) — its name is excluded by design.
+    // If the chain came up empty, actionCommand itself IS the root (no parent
+    // at all) — fall back to its own name so the function never returns ''.
+    return chain.length > 0 ? chain.join(' ') : actionCommand.name();
 }
 /**
  * Walk the full command tree and produce every valid resolveCommandName()
@@ -86,9 +102,19 @@ export function isHelpOrVersion(argv) {
  * "argv scan, never trust flag values" caveat as isHelpOrVersion: positional
  * values are never mistaken for `--json` since they don't equal the literal
  * string. (#86 finding 3)
+ *
+ * #88 item 5 — only scans tokens BEFORE the first bare `--`. Commander treats
+ * `--` as "end of options": everything after it is a positional operand, not
+ * a flag, even if the literal text is `--json`. `scraps list -- --json`
+ * passes `--json` as an (excess) positional argument, not the flag — an
+ * unscoped `argv.includes('--json')` would still match it and wrongly emit a
+ * JSON envelope for what is actually a plain usage error with no --json
+ * requested at all.
  */
 export function hasJsonFlag(argv) {
-    return argv.includes('--json');
+    const dashDashIdx = argv.indexOf('--');
+    const scanned = dashDashIdx === -1 ? argv : argv.slice(0, dashDashIdx);
+    return scanned.includes('--json');
 }
 /**
  * Commander's default (no exitOverride) calls `process.exit()` directly for
@@ -113,6 +139,35 @@ export function applyExitOverride(cmd) {
  * these already printed their own output (to stdout) via commander itself;
  * runCli's catch must treat them as a clean exit, not an error. (#86 finding 3) */
 const HELP_OR_VERSION_CODES = new Set(['commander.helpDisplayed', 'commander.help', 'commander.version']);
+/**
+ * #88 item 6 — best-effort command-name recovery for a commander parse error
+ * that happened BEFORE any action ran, so `currentCommand` (the preAction
+ * hook's resolved command) is still undefined — e.g. an unknown option on an
+ * otherwise-valid subcommand, or excess arguments. Without this, EVERY such
+ * failure previously collapsed to resolveCommandName(undefined) === 'unknown'
+ * — and since 'unknown' was never itself allowlisted at the
+ * registerAllowedCommands call site (see runCli below), that capture was
+ * silently dropped by captureCommand's allowlist check: a dead branch that
+ * looked like it reported telemetry but never actually did.
+ *
+ * Matches ONLY a name already present in `allowedNames` (the real command
+ * tree, from collectCommandNames) — never invents one from raw argv text.
+ * Checks the two-token form first (`scraps boom`) since most usage errors
+ * happen on a nested leaf command; falls back to the single top-level token,
+ * then to 'unknown' (now itself allowlisted, so that capture fires too).
+ */
+export function bestEffortCommandName(argv, allowedNames) {
+    const a2 = argv[2];
+    const a3 = argv[3];
+    if (a2 && a3) {
+        const combined = `${a2} ${a3}`;
+        if (allowedNames.includes(combined))
+            return combined;
+    }
+    if (a2 && allowedNames.includes(a2))
+        return a2;
+    return 'unknown';
+}
 export async function runCli(argv = process.argv) {
     if (!isHelpOrVersion(argv))
         autoUpdateInstalledSkills();
@@ -122,7 +177,13 @@ export async function runCli(argv = process.argv) {
     // including subcommands added via addCommand() that don't otherwise
     // inherit it. (#86 finding 3)
     applyExitOverride(program);
-    registerAllowedCommands(collectCommandNames(program));
+    const commandNames = collectCommandNames(program);
+    // #88 item 6 — 'unknown' is a legitimate resolveCommandName() output (the
+    // fallback for "no command resolved at all"), not free-form user input —
+    // it must be explicitly allowlisted here or every capture that falls back
+    // to it is silently dropped by captureCommand's allowlist check (a dead
+    // branch that looks like it reports telemetry but never does).
+    registerAllowedCommands([...commandNames, 'unknown']);
     // Track start times + the currently-resolved command per instance, so the
     // catch handler below can derive the exact same safe name the success path
     // uses — it must never re-derive anything from argv.
@@ -176,7 +237,15 @@ export async function runCli(argv = process.argv) {
                     console.log(JSON.stringify({ error: { message: err.message, kind: 'usage' } }));
                 }
                 process.exitCode = 2;
-                void captureCommand(resolveCommandName(currentCommand), { exit_code: 2, error: 'CommanderError' });
+                // #88 item 6 — currentCommand is still undefined here whenever the
+                // parse error happened before preAction fired (the common case for a
+                // usage error — commander validates flags/arity before invoking the
+                // action). Try a best-effort match against the real command tree
+                // instead of collapsing straight to 'unknown'.
+                const commandName = currentCommand
+                    ? resolveCommandName(currentCommand)
+                    : bestEffortCommandName(argv, commandNames);
+                void captureCommand(commandName, { exit_code: 2, error: 'CommanderError' });
             }
         }
         else {

package/dist/lib/api.d.ts

@@ -11,25 +11,58 @@ export declare class ApiError extends Error {
 export declare class NetworkError extends Error {
     constructor(message: string);
 }
+/**
+ * A LOCAL auth failure — no token available, or a locally-decoded token
+ * that's provably expired, discovered entirely client-side before any HTTP
+ * call was ever made. Distinguished from ApiError(401) (a real server-issued
+ * 401 response) so the --json envelope never claims `status:401` for
+ * something the server never said — that would be a fabricated fact,
+ * indistinguishable from an actual server round-trip to a machine consumer.
+ * Both classify to the same exit code (3) / kind "auth" in classifyError
+ * (errors.ts); only the envelope's `status` field differs (present for
+ * ApiError, absent here). (#88 item 4)
+ */
+export declare class AuthError extends Error {
+    constructor(message: string);
+}
 /**
  * The single "no token available" error — every call site in this file that
  * needs a token (request/upload/getText/stream) used to throw its own copy
  * of `new Error('Not logged in. Run: trawl login')`, which fell through
  * classifyError's generic branch (exit 1, kind:"unknown") — indistinguishable
- * from an arbitrary bug. Auth-classifying it as an ApiError(401) puts it on
- * the exact same exit-3 / kind:"auth" path a real 401 response already takes.
+ * from an arbitrary bug. Auth-classifying it puts it on the exact same
+ * exit-3 / kind:"auth" path a real 401 response already takes — but as an
+ * AuthError (no HTTP call happened here), never a fabricated ApiError(401).
  * `trawl token` (src/commands/token.ts) reuses this too, so "no token" means
- * the same thing everywhere it can be observed. (#86 findings 1/2)
+ * the same thing everywhere it can be observed. (#86 findings 1/2, #88 item 4)
  */
-export declare function notLoggedInError(): ApiError;
+export declare function notLoggedInError(): AuthError;
+/**
+ * #91 P0 — some endpoints legitimately run 30–250s server-side: a scrap
+ * execute (worker-puppeteer navigation + antibot tier escalation + AI-fix
+ * dry-run retries). The generic 30s default was aborting those mid-flight
+ * and surfacing a fabricated `NetworkError timed out` (exit 5) for a request
+ * that was always going to succeed given enough time. Passed as the per-call
+ * `{timeoutMs}` override at exactly the 3 call sites that hit those
+ * endpoints (src/commands/scraps.ts): `scraps run` / `data --fresh`
+ * (GET /api/scraps/load/:id) and `scraps trigger --wait`
+ * (POST /api/scraps/worker/:id, synchronous branch only — the default async
+ * `?wait=false` POST returns almost immediately and keeps the 30s default).
+ * 300s leaves margin over the ~250s worst case without being unboundedly long.
+ */
+export declare const LONG_RUN_TIMEOUT_MS = 300000;
+export interface RequestOptions {
+    /** Per-call timeout override in ms (e.g. LONG_RUN_TIMEOUT_MS). Ignored — env wins — when TRAWL_TIMEOUT is set; see getTimeoutMs. */
+    timeoutMs?: number;
+}
 export declare const api: {
-    get: <T>(path: string) => Promise<T>;
-    getText: (path: string) => Promise<string>;
-    post: <T>(path: string, body?: unknown) => Promise<T>;
-    put: <T>(path: string, body?: unknown) => Promise<T>;
-    delete: <T>(path: string) => Promise<T>;
-    upload: <T>(path: string, formData: FormData) => Promise<T>;
-    publicPost: <T>(path: string, body?: unknown, baseUrlOverride?: string) => Promise<{
+    get: <T>(path: string, opts?: RequestOptions) => Promise<T>;
+    getText: (path: string, opts?: RequestOptions) => Promise<string>;
+    post: <T>(path: string, body?: unknown, opts?: RequestOptions) => Promise<T>;
+    put: <T>(path: string, body?: unknown, opts?: RequestOptions) => Promise<T>;
+    delete: <T>(path: string, opts?: RequestOptions) => Promise<T>;
+    upload: <T>(path: string, formData: FormData, opts?: RequestOptions) => Promise<T>;
+    publicPost: <T>(path: string, body?: unknown, baseUrlOverride?: string, opts?: RequestOptions) => Promise<{
         data: T;
         headers: Headers;
     }>;

package/dist/lib/api.js

@@ -25,42 +25,94 @@ export class NetworkError extends Error {
         this.name = 'NetworkError';
     }
 }
+/**
+ * A LOCAL auth failure — no token available, or a locally-decoded token
+ * that's provably expired, discovered entirely client-side before any HTTP
+ * call was ever made. Distinguished from ApiError(401) (a real server-issued
+ * 401 response) so the --json envelope never claims `status:401` for
+ * something the server never said — that would be a fabricated fact,
+ * indistinguishable from an actual server round-trip to a machine consumer.
+ * Both classify to the same exit code (3) / kind "auth" in classifyError
+ * (errors.ts); only the envelope's `status` field differs (present for
+ * ApiError, absent here). (#88 item 4)
+ */
+export class AuthError extends Error {
+    constructor(message) {
+        super(message);
+        this.name = 'AuthError';
+    }
+}
 /**
  * The single "no token available" error — every call site in this file that
  * needs a token (request/upload/getText/stream) used to throw its own copy
  * of `new Error('Not logged in. Run: trawl login')`, which fell through
  * classifyError's generic branch (exit 1, kind:"unknown") — indistinguishable
- * from an arbitrary bug. Auth-classifying it as an ApiError(401) puts it on
- * the exact same exit-3 / kind:"auth" path a real 401 response already takes.
+ * from an arbitrary bug. Auth-classifying it puts it on the exact same
+ * exit-3 / kind:"auth" path a real 401 response already takes — but as an
+ * AuthError (no HTTP call happened here), never a fabricated ApiError(401).
  * `trawl token` (src/commands/token.ts) reuses this too, so "no token" means
- * the same thing everywhere it can be observed. (#86 findings 1/2)
+ * the same thing everywhere it can be observed. (#86 findings 1/2, #88 item 4)
  */
 export function notLoggedInError() {
-    return new ApiError(401, 'Not logged in. Run: trawl login');
+    return new AuthError('Not logged in. Run: trawl login');
 }
 const DEFAULT_TIMEOUT_MS = 30_000;
-/** Effective fetch timeout — TRAWL_TIMEOUT env override (ms), default 30s. (#71) */
-function getTimeoutMs() {
+/**
+ * #91 P0 — some endpoints legitimately run 30–250s server-side: a scrap
+ * execute (worker-puppeteer navigation + antibot tier escalation + AI-fix
+ * dry-run retries). The generic 30s default was aborting those mid-flight
+ * and surfacing a fabricated `NetworkError timed out` (exit 5) for a request
+ * that was always going to succeed given enough time. Passed as the per-call
+ * `{timeoutMs}` override at exactly the 3 call sites that hit those
+ * endpoints (src/commands/scraps.ts): `scraps run` / `data --fresh`
+ * (GET /api/scraps/load/:id) and `scraps trigger --wait`
+ * (POST /api/scraps/worker/:id, synchronous branch only — the default async
+ * `?wait=false` POST returns almost immediately and keeps the 30s default).
+ * 300s leaves margin over the ~250s worst case without being unboundedly long.
+ */
+export const LONG_RUN_TIMEOUT_MS = 300_000;
+/**
+ * Effective fetch timeout for one request.
+ *
+ * Precedence (#91): TRAWL_TIMEOUT (env) ALWAYS wins when set to a valid
+ * positive number — an operator/CI override must be able to force a shorter
+ * or longer ceiling globally (e.g. a slow CI network, or a deliberately tight
+ * smoke-test budget) without editing call sites. Only when the env var is
+ * unset/invalid does the caller-supplied per-call `overrideMs` apply (e.g.
+ * LONG_RUN_TIMEOUT_MS for the 3 long-run call sites); otherwise
+ * DEFAULT_TIMEOUT_MS. This preserves #71's original "env always overrides"
+ * contract while layering the new per-call default beneath it, never above.
+ */
+function getTimeoutMs(overrideMs) {
     const raw = process.env['TRAWL_TIMEOUT']?.trim();
-    if (!raw)
-        return DEFAULT_TIMEOUT_MS;
-    const n = Number(raw);
-    return Number.isFinite(n) && n > 0 ? n : DEFAULT_TIMEOUT_MS;
+    if (raw) {
+        const n = Number(raw);
+        if (Number.isFinite(n) && n > 0)
+            return n;
+    }
+    return overrideMs !== undefined && overrideMs > 0 ? overrideMs : DEFAULT_TIMEOUT_MS;
 }
 /**
  * Wrap a `fetch()` call so connection-level failures (ECONNREFUSED, DNS,
  * timeout, …) surface as a NetworkError carrying the effective URL + the
  * unwrapped `err.cause` detail, instead of a bare "fetch failed" with no
  * actionable information. (#71 findings 4/58)
+ *
+ * `effectiveTimeoutMs` is the ACTUAL timeout that was armed for this specific
+ * call (already resolved via getTimeoutMs by the caller) — used only to
+ * report an honest number in the timeout message; #91 — before this it
+ * always re-derived a fresh `getTimeoutMs()` with no override, so a request
+ * armed with LONG_RUN_TIMEOUT_MS that timed out would have lied and reported
+ * "timed out after 30000ms".
  */
-async function safeFetch(url, options) {
+async function safeFetch(url, options, effectiveTimeoutMs) {
     try {
         return await fetch(url, options);
     }
     catch (err) {
         const e = err;
         if (e?.name === 'TimeoutError' || e?.name === 'AbortError') {
-            throw new NetworkError(`Request to ${url} timed out after ${getTimeoutMs()}ms (override with TRAWL_TIMEOUT env var, ms)`);
+            throw new NetworkError(`Request to ${url} timed out after ${effectiveTimeoutMs}ms (override with TRAWL_TIMEOUT env var, ms)`);
         }
         const cause = e?.cause;
         const causeDetail = cause?.code ? ` (${cause.code})` : cause?.message ? ` (${cause.message})` : '';
@@ -176,21 +228,22 @@ async function throwIfError(res, isPublic = false) {
         throw new ApiError(res.status, `${res.status} ${res.statusText}: ${full}`);
     }
 }
-async function request(path, options = {}) {
+async function request(path, options = {}, reqOpts = {}) {
     const token = getToken();
     if (!token)
         throw notLoggedInError();
     const url = `${getApiUrl()}${path}`;
+    const timeoutMs = getTimeoutMs(reqOpts.timeoutMs);
     const res = await safeFetch(url, {
         ...options,
-        signal: AbortSignal.timeout(getTimeoutMs()),
+        signal: AbortSignal.timeout(timeoutMs),
         headers: {
             'Content-Type': 'application/json',
             'User-Agent': USER_AGENT,
             ...options.headers,
             Cookie: `TOKEN=${token}`,
         },
-    });
+    }, timeoutMs);
     await throwIfError(res);
     const text = await res.text();
     try {
@@ -207,21 +260,22 @@ async function request(path, options = {}) {
         throw new Error('Invalid JSON in server response');
     }
 }
-async function upload(path, formData) {
+async function upload(path, formData, reqOpts = {}) {
     const token = getToken();
     if (!token)
         throw notLoggedInError();
     const url = `${getApiUrl()}${path}`;
+    const timeoutMs = getTimeoutMs(reqOpts.timeoutMs);
     // Do NOT set Content-Type — fetch sets it automatically with the correct multipart boundary
     const res = await safeFetch(url, {
         method: 'POST',
         body: formData,
-        signal: AbortSignal.timeout(getTimeoutMs()),
+        signal: AbortSignal.timeout(timeoutMs),
         headers: {
             'User-Agent': USER_AGENT,
             Cookie: `TOKEN=${token}`,
         },
-    });
+    }, timeoutMs);
     await throwIfError(res);
     const text = await res.text();
     try {
@@ -238,14 +292,15 @@ async function upload(path, formData) {
         throw new Error('Invalid JSON in server response');
     }
 }
-async function publicPost(path, body, baseUrlOverride) {
+async function publicPost(path, body, baseUrlOverride, reqOpts = {}) {
     const url = `${baseUrlOverride ?? getApiUrl()}${path}`;
+    const timeoutMs = getTimeoutMs(reqOpts.timeoutMs);
     const res = await safeFetch(url, {
         method: 'POST',
         headers: { 'Content-Type': 'application/json', 'User-Agent': USER_AGENT },
         body: body ? JSON.stringify(body) : undefined,
-        signal: AbortSignal.timeout(getTimeoutMs()),
-    });
+        signal: AbortSignal.timeout(timeoutMs),
+    }, timeoutMs);
     await throwIfError(res, true);
     const text = await res.text();
     try {
@@ -256,35 +311,36 @@ async function publicPost(path, body, baseUrlOverride) {
         throw new Error('Invalid JSON in server response');
     }
 }
-async function getText(path) {
+async function getText(path, reqOpts = {}) {
     const token = getToken();
     if (!token)
         throw notLoggedInError();
     const url = `${getApiUrl()}${path}`;
+    const timeoutMs = getTimeoutMs(reqOpts.timeoutMs);
     const res = await safeFetch(url, {
         headers: {
             'User-Agent': USER_AGENT,
             Cookie: `TOKEN=${token}`,
         },
-        signal: AbortSignal.timeout(getTimeoutMs()),
-    });
+        signal: AbortSignal.timeout(timeoutMs),
+    }, timeoutMs);
     await throwIfError(res);
     return res.text();
 }
 export const api = {
-    get: (path) => request(path),
-    getText: (path) => getText(path),
-    post: (path, body) => request(path, {
+    get: (path, opts) => request(path, {}, opts),
+    getText: (path, opts) => getText(path, opts),
+    post: (path, body, opts) => request(path, {
         method: 'POST',
         body: body ? JSON.stringify(body) : undefined,
-    }),
-    put: (path, body) => request(path, {
+    }, opts),
+    put: (path, body, opts) => request(path, {
         method: 'PUT',
         body: body ? JSON.stringify(body) : undefined,
-    }),
-    delete: (path) => request(path, { method: 'DELETE' }),
-    upload: (path, formData) => upload(path, formData),
-    publicPost: (path, body, baseUrlOverride) => publicPost(path, body, baseUrlOverride),
+    }, opts),
+    delete: (path, opts) => request(path, { method: 'DELETE' }, opts),
+    upload: (path, formData, opts) => upload(path, formData, opts),
+    publicPost: (path, body, baseUrlOverride, opts) => publicPost(path, body, baseUrlOverride, opts),
     stream: async function* (path) {
         const token = getToken();
         if (!token)
@@ -293,13 +349,15 @@ export const api = {
         // No AbortSignal.timeout here — a long-running `watch`/`--watch` stream is
         // expected to sit open indefinitely; only connection-level failures
         // (never a timeout) should surface via safeFetch's cause-unwrapping. (#71)
+        // effectiveTimeoutMs passed to safeFetch here is only ever used to format
+        // a timeout message that can't actually fire (no signal attached).
         const res = await safeFetch(url, {
             headers: {
                 Accept: 'text/event-stream',
                 'User-Agent': USER_AGENT,
                 Cookie: `TOKEN=${token}`,
             },
-        });
+        }, getTimeoutMs());
         if (!res.ok || !res.body) {
             throw new ApiError(res.status, `SSE failed: ${res.status}`);
         }

package/dist/lib/errors.js

@@ -1,5 +1,5 @@
 import chalk from 'chalk';
-import { ApiError, NetworkError } from './api.js';
+import { ApiError, AuthError, NetworkError } from './api.js';
 /**
  * Thrown for CLI usage / input-validation failures (bad flag value, malformed
  * JSON, invalid ObjectId, missing required prompt input, …). Distinguished
@@ -22,6 +22,14 @@ export class UsageError extends Error {
  */
 export function classifyError(err) {
     const message = err instanceof Error ? err.message : String(err);
+    // #88 item 4 — a LOCAL auth failure (no token, or a locally-decoded expired
+    // token) never made an HTTP call, so its envelope must never carry
+    // `status:401` — that would claim a server response that never happened.
+    // Same exit code / kind as a real server 401 (ApiError below); only the
+    // envelope shape differs.
+    if (err instanceof AuthError) {
+        return { exitCode: 3, envelope: { message, kind: 'auth' } };
+    }
     if (err instanceof ApiError) {
         if (err.status === 401)
             return { exitCode: 3, envelope: { message, status: 401, kind: 'auth' } };

package/dist/lib/skills.js

@@ -43,7 +43,20 @@ export function installSkill(name, scope, opts = {}) {
     }
     const dest = join(getSkillsBase(scope), name);
     if (existsSync(dest)) {
-        const owned = getInstalledVersion(name, scope) !== null;
+        // #91 — an unreadable `.version` (e.g. EISDIR from a directory instead of
+        // a file, a corrupted/interrupted install) is NOT proof of ownership
+        // either — treat it exactly like a missing marker (unowned) instead of
+        // letting the raw fs error crash the whole batch (`skills install`/
+        // `update` with no skill argument loops over every bundled skill; one
+        // malformed dest must not block installing the others).
+        let installedVersion;
+        try {
+            installedVersion = getInstalledVersion(name, scope);
+        }
+        catch {
+            installedVersion = null;
+        }
+        const owned = installedVersion !== null;
         if (!owned && !opts.force) {
             throw new Error(`Refusing to overwrite "${dest}" — it was not installed by trawl (no .version marker). ` +
                 `Pass --force to overwrite it anyway.`);
@@ -102,8 +115,19 @@ export function removeOrphanedSkills(scope) {
         }
         if (bundled.has(name))
             continue;
-        // No `.version` marker → not ours → never delete it.
-        if (getInstalledVersion(name, scope) === null)
+        // No `.version` marker → not ours → never delete it. Read is wrapped:
+        // a weird `.version` (e.g. a directory instead of a file → EISDIR on
+        // readFileSync) must skip only THIS entry, not blow up the whole sweep —
+        // a single malformed install must never leave every other entry
+        // unswept. (#88 item 12)
+        let installedVersion;
+        try {
+            installedVersion = getInstalledVersion(name, scope);
+        }
+        catch {
+            continue;
+        }
+        if (installedVersion === null)
             continue;
         rmSync(dir, { recursive: true, force: true });
         process.stderr.write(`trawl: removed orphaned skill "${name}" (${scope}) — no longer bundled with this CLI version\n`);
@@ -136,7 +160,21 @@ export function autoUpdateInstalledSkills() {
             for (const scope of ['user', 'local']) {
                 if (!isSkillInstalled(name, scope))
                     continue;
-                const installed = getInstalledVersion(name, scope);
+                // #91 — mirrors removeOrphanedSkills' guard below: a weird `.version`
+                // marker (e.g. a directory instead of a file, from a corrupted /
+                // interrupted install) throws EISDIR on readFileSync. Without this
+                // per-entry guard, that throw was caught by this function's OUTER
+                // try/catch (below) — which aborts the ENTIRE function, so every
+                // remaining bundled skill silently stopped syncing AND the orphan
+                // sweep (removeOrphanedSkills, called after this loop) never ran
+                // either. Must skip only THIS entry, never abort the whole sweep.
+                let installed;
+                try {
+                    installed = getInstalledVersion(name, scope);
+                }
+                catch {
+                    continue;
+                }
                 // No `.version` marker → not ours → never delete it.
                 if (installed === null)
                     continue;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@trawlme/cli",
-  "version": "1.18.2",
+  "version": "1.18.4",
   "description": "Trawl CLI — manage scraps from the terminal",
   "type": "module",
   "bin": {
@@ -40,7 +40,7 @@
     "url": "https://github.com/comes-io/trawl_cli/issues"
   },
   "dependencies": {
-    "@trawlme/skills": "1.3.1",
+    "@trawlme/skills": "1.3.3",
     "chalk": "^5.6.2",
     "commander": "^14.0.3",
     "conf": "^15.1.0",