@trawlme/cli 1.18.3 → 1.18.5

package/README.md

@@ -41,7 +41,7 @@ trawl token                                  Print the stored session JWT (for M
 ### Scraps
 
 ```
-trawl scraps list [--json] [--status <success|failure|never>] [--limit <n>] [--page <n>]
+trawl scraps list [--json] [--status <success|failure|never|running|regression>] [--limit <n>] [--page <n>]
 trawl scraps get <id> [--json]
 trawl scraps create -t <title> [-u <url>] [-r <request>] [-d <description>] [--tier <tier0|tier1|tier2|tier3|tier4>] [--json]
 trawl scraps update <id> [-t <title>] [-u <url>] [-r <request>] [-d <description>] [--cron <expr>|--no-cron] [--alert <email>|--no-alert] [--autofix|--no-autofix] [-p <json>|--params-file <path>] [--tier <tier0|tier1|tier2|tier3|tier4>] [--force-tier <tier0|tier1|tier2|tier3|tier4>] [--json]
@@ -53,18 +53,21 @@ trawl scraps history <id> [--json] [-n <limit>]
 trawl scraps run-info <hid> [--json]
 trawl scraps doctor <id> [--json] [--autofix]
 trawl scraps autofix <id> [--json]
-trawl scraps snapshot <id> [--error] [-o <file>]
+trawl scraps snapshot <id> [--error] [-o <file>] [--json]
 trawl scraps banner <id> -f <file>
 trawl scraps delete <id> [--force]         Alias: rm
 ```
 
 - `--tier` forces a proxy tier; `--force-tier` raises the proxy-tier ceiling past the auto-cap (history-gated: may be refused or cost more). `create --json`/`update --json` print the full scrap object (including the `_tierOverride` outcome) on stdout; a refused tier override exits 1 with a standard `--json` error envelope. When a tier was requested but the server's response carries no `_tierOverride` at all (an older server that can't confirm what actually got applied), a stderr warning is printed either way, and under `--json` the emitted object also carries `"_tierUnconfirmed": true` — the machine-readable counterpart to that warning, since a `--json` caller has no reliable reason to read stderr.
-- `scraps doctor` diagnoses the last run (error, failed selector, block status, page state, autofix outcome); `--autofix` includes the full autofix diff/dry-run/knowledge. A run that is still in flight (`status: null`, server-side) shows a `running` badge, never `failed`.
+- `scraps doctor` diagnoses the last run (error, failed selector, block status, page state, autofix outcome); `--autofix` includes the full autofix diff/dry-run/knowledge. A run that is still in flight (`status: null`, server-side) shows a `running` badge, never `failed`; a run whose item count regressed vs baseline (`statusDetail: "regression"`) shows its own amber `regression` badge, never `failed` either (`scraps list`/`get` show the same distinction — a `▼` icon, not the red `✗` a genuine failure gets).
 - `scraps autofix` shows the last auto-fix attempt on its own (decision, diff, dry-run, knowledge). `--json` on a scrap that has **never run** returns `{"status":"no_runs"}` (exit 0) — distinct from `null`, which means a run exists but had no auto-fix attempt.
-- `scraps snapshot --error` fetches the error-path snapshot instead of the normal one; `-o <file>` writes to a file instead of stdout.
+- `scraps snapshot --error` fetches the error-path snapshot instead of the normal one; `-o <file>` writes to a file instead of stdout. On a scrap that has **never run**: `--json` returns `{"status":"no_runs"}` (exit 0), matching `doctor`/`autofix`; `-o <file>` (without `--json`) exits 4 (`not_found`) instead of silently exiting 0 with nothing written — a script checking the exit code alone must be able to tell "no file was produced" from success. `--json` takes priority when both are passed.
 - `scraps history` lists past runs (newest first); `scraps run-info <hid>` shows details of a single run from that history.
 - `scraps data` returns the last persisted run payload (no execute quota); `--fresh` runs the scrap live instead (consumes execute quota); `--errors` shows the last run's error detail (`--json` on a never-run scrap returns `{"status":"no_runs"}`, exit 0, matching `doctor --json`). `[]` on stdout means a genuine zero-item successful run — a scrap that has never run, whose last run failed, or whose payload aged out of retention returns a `--json` error envelope (exit 4/1/4 respectively) instead. Two more honest states: a run still **in flight** (`status: null` server-side) returns a `kind:"in_progress"` error envelope (exit 1, "retry shortly" — never suggests `--fresh`, which would just 429 against the run already holding the lock); a run whose item count **regressed** vs baseline (`statusDetail: "regression"`) still returns the real, non-empty items on stdout (exit 0) plus a stderr warning pointing at `scraps doctor <id>` — the data itself is genuine even though the run is flagged.
 - `scraps get` (and anything reading through it, like `scraps data`'s default path) embeds only the newest 100 history rows on the returned scrap object — `scraps run-info` and `scraps doctor` fetch a single run directly and are unaffected by that cap.
+- `scraps banner -f <file>` only accepts `png`/`jpg`/`jpeg`/`webp`; any other extension is a usage error (exit 2) instead of silently uploading the file under a fabricated `image/png` Content-Type.
+- **Long-running scrap execute (`scraps run`, `data --fresh`, `trigger --wait`):** these hit the same server-side scrap-execute path, which can legitimately take 30–250s (proxy tier escalation, AI-fix retries) — the CLI arms a 300s timeout for exactly these three call sites instead of the generic 30s default. `TRAWL_TIMEOUT` (see below) still overrides ALL requests, including these — set it if you need a tighter or looser ceiling than 300s for a long-running scrap.
+- **`--watch` is poll-based, not a live stream:** the activities SSE endpoint has no backlog and, for the default async `trigger` (no `--wait`), runs in a separate cron-consumer pod whose events never reach the API pod holding the SSE connection — a naive "await the run, then open SSE" shows nothing. `scraps run --watch` and `scraps trigger --watch` instead poll `GET /api/scraps/:id` (terminal status) and the activities REST list until the run finishes, printing each new activity line as it appears; a run that never reaches a terminal status within 300s prints an honest timeout notice pointing at `scraps doctor <id>`. `scraps watch <id>` (the standalone command, no trigger) is unchanged — it still opens the live SSE stream directly.
 
 ### Scrap accounts
 
@@ -160,7 +163,7 @@ Under `--json`, a failing command emits a single error envelope on stdout — `{
 | `TRAWL_TELEMETRY`     | Set to `0` to disable telemetry for the current session           |
 | `DO_NOT_TRACK`        | Set to `1` to disable telemetry (cross-vendor convention, https://consoledonottrack.com) — same effect as `TRAWL_TELEMETRY=0` |
 | `TRAWL_CONFIG_DIR`    | Override where the config file (token, API URL, telemetry state) is stored — useful for hermetic CI runs or concurrent `trawl login`s that must not share one on-disk file |
-| `TRAWL_TIMEOUT`       | Override the per-request fetch timeout in milliseconds (default `30000`) |
+| `TRAWL_TIMEOUT`       | Override the per-request fetch timeout in milliseconds (default `30000`; `300000` for `scraps run`/`data --fresh`/`trigger --wait` — this env var always wins over that longer default too) |
 | `TRAWL_SKILLS_SYNC`   | Set to `0` to disable the startup skills auto-sync entirely       |
 
 Session override (no `trawl login` mutation, ideal for CI/QA against another env):

package/dist/commands/doctor.js

@@ -86,6 +86,13 @@ export function formatDoctor(scrapTitle, run, fix = null, scrapId) {
     // Header + status badge. status:null means a run is IN FLIGHT (node
     // persists {status:null, statusDetail:null, inFlight:true} the moment a
     // run starts) — that must never render as "failed". (#88 item 1)
+    //
+    // #91 LOW — a regression row (status:false, statusDetail:'regression') was
+    // falling into the same red "failed" bucket as a genuine failure, then
+    // showing a contradictory "Regression: X vs Y" detail line right below it.
+    // A regression run's write actually SUCCEEDED (the item count just dropped
+    // vs baseline, detected by an async patch afterward — see #88 item 2 in
+    // scraps.ts) — it needs its own distinct badge, not "failed".
     const ok = run.status === true;
     const badge = ok
         ? chalk.green('● success')
@@ -93,7 +100,9 @@ export function formatDoctor(scrapTitle, run, fix = null, scrapId) {
             ? chalk.cyan('● running')
             : run.statusDetail === 'empty'
                 ? chalk.yellow('● empty')
-                : chalk.red('● failed');
+                : run.statusDetail === 'regression'
+                    ? chalk.hex('#FFA500')('● regression')
+                    : chalk.red('● failed');
     lines.push(`${chalk.bold(scrapTitle)}  ${badge}${run.statusDetail ? ` (${run.statusDetail})` : ''}`);
     lines.push(chalk.dim(`  Run ID:  ${run._id}`));
     // Error message — an honest accept-wall string for known-walled scraps

package/dist/commands/scraps.d.ts

@@ -1,2 +1,46 @@
 import { Command } from 'commander';
 export declare const scraps: Command;
+/** The top-of-history snapshot pollRunProgress needs to identify which run
+ * it's watching — see captureBeforeRunState. */
+interface BeforeRunState {
+    id?: string;
+    alreadyInFlight: boolean;
+}
+/**
+ * #91 P1 — replaces "await the run to completion, THEN open the activities
+ * SSE stream" (which showed NOTHING: the activities SSE
+ * (GET /api/scraps/:id/activities/stream) is backed by an in-process
+ * EventEmitter with no backlog — trawl_node
+ * modules/activities/services/activities.service.js — so by the time a
+ * synchronous run has already finished there is nothing left to emit; and
+ * the async `trigger` default enqueues the run onto a durable job queue a
+ * SEPARATE cron-consumer pod drains, whose in-process emitter never reaches
+ * the API pod holding the SSE connection at all).
+ *
+ * Instead this polls two REST reads that are BOTH Mongo-backed (not
+ * in-process), so they work no matter which pod actually executed the run:
+ *  - GET /api/scraps/:id/activities?history=<hid>&limit=20 — the SAME
+ *    activities-list endpoint `doctor`'s fetchRunAndFix already calls
+ *    (src/commands/doctor.ts) — prints each new activity line once the new
+ *    run's history id is known.
+ *  - GET /api/scraps/:id — history[0].status/statusDetail, the SAME
+ *    terminal-status signal `lastStatus()` above already trusts (status:null
+ *    === in flight, #88 item 1) to know when the run is done.
+ *
+ * #93 item 1 — dedup-race fix. "Is this the run we're watching?" used to be
+ * a single check: `history[0]._id !== beforeHistoryId`. That's wrong when
+ * `before.alreadyInFlight` is true (a `trigger` call deduped onto a worker
+ * job that was ALREADY pending/running at capture time): the top row IS the
+ * run we're watching, but its `_id` never changes, so the old guard never
+ * released and the poll ran the full timeout to a false "Timed out". The run
+ * we're watching is now EITHER a brand-new id (fresh trigger, the common
+ * case) OR the same id that was already in-flight (status:null) at capture
+ * (the dedup case) — a same-id row that was already TERMINAL at capture is
+ * neither, and must not be latched onto as "done" (it's just the previous
+ * run, still sitting there until a genuinely new run supersedes it).
+ */
+export declare function pollRunProgress(id: string, before: BeforeRunState | undefined, opts?: {
+    intervalMs?: number;
+    timeoutMs?: number;
+}): Promise<void>;
+export {};

package/dist/commands/scraps.js

@@ -1,7 +1,7 @@
 import { Command } from 'commander';
 import chalk from 'chalk';
 import { oraPromise } from 'ora';
-import { api } from '../lib/api.js';
+import { api, LONG_RUN_TIMEOUT_MS } from '../lib/api.js';
 import { table, json } from '../lib/format.js';
 import { promptPassword } from '../lib/prompt.js';
 import { validateObjectId } from '../lib/validate.js';
@@ -30,6 +30,13 @@ function lastStatus(scrap) {
         return 'running';
     if (last.status === undefined)
         return 'never';
+    // #91 LOW — same regression-as-failure bucketing bug as doctor's badge
+    // (formatDoctor): a regression row's write actually succeeded (item count
+    // just dropped vs baseline, flagged by an async patch afterward — #88 item
+    // 2), so it must not collapse into the same 'failure' bucket a genuine
+    // failed run gets.
+    if (last.status === false && last.statusDetail === 'regression')
+        return 'regression';
     return last.status === true ? 'success' : 'failure';
 }
 function lastRun(scrap) {
@@ -51,6 +58,10 @@ function statusIcon(status) {
         return chalk.red('✗');
     if (status === 'running')
         return chalk.cyan('↻');
+    // #91 LOW — distinct amber icon, never the red ✗ a genuine failure gets
+    // (mirrors doctor's "● regression" badge — see formatDoctor).
+    if (status === 'regression')
+        return chalk.hex('#FFA500')('▼');
     return chalk.dim('—');
 }
 export const scraps = new Command('scraps').description('Manage scraps');
@@ -71,13 +82,130 @@ async function watchActivities(id) {
         }
     }
 }
+/**
+ * #91 P1 / #93 item 1 — snapshot the current top history entry BEFORE
+ * triggering a run, so pollRunProgress can later tell "the run we just
+ * launched" apart from whatever the last run happened to be. Node persists a
+ * fresh {status:null, inFlight:true} row the moment a run starts (see
+ * HistorysService.create in trawl_node), so a changed history[0]._id is
+ * normally an honest "the new run has begun" signal — EXCEPT `trigger`
+ * (method:'worker') dedups onto an already-pending/running worker job
+ * (ScrapJobsService, LIVE_STATUSES) instead of creating a new row: the top
+ * row's `_id` never changes even though this launch IS that run. Recording
+ * `alreadyInFlight` (status===null at capture time) lets pollRunProgress
+ * recognize that dedup case too, instead of waiting forever for an `_id`
+ * that will never arrive (#93 item 1).
+ *
+ * Best-effort: a failed lookup falls back to `{alreadyInFlight:false}`,
+ * which is still correct for a scrap that has never run (id undefined).
+ */
+async function captureBeforeRunState(id) {
+    try {
+        const scrap = await api.get(`/api/scraps/${id}`);
+        const top = scrap.history?.[0];
+        return { id: top?._id, alreadyInFlight: top?.status === null };
+    }
+    catch {
+        return { id: undefined, alreadyInFlight: false };
+    }
+}
+function sleep(ms) {
+    return new Promise((resolve) => setTimeout(resolve, ms));
+}
+const POLL_INTERVAL_MS = 2000;
+// Mirrors LONG_RUN_TIMEOUT_MS (#91 item 1) — the server-side worst case this
+// polls for is the same one that timeout was sized for.
+const POLL_TIMEOUT_MS = LONG_RUN_TIMEOUT_MS;
+/**
+ * #91 P1 — replaces "await the run to completion, THEN open the activities
+ * SSE stream" (which showed NOTHING: the activities SSE
+ * (GET /api/scraps/:id/activities/stream) is backed by an in-process
+ * EventEmitter with no backlog — trawl_node
+ * modules/activities/services/activities.service.js — so by the time a
+ * synchronous run has already finished there is nothing left to emit; and
+ * the async `trigger` default enqueues the run onto a durable job queue a
+ * SEPARATE cron-consumer pod drains, whose in-process emitter never reaches
+ * the API pod holding the SSE connection at all).
+ *
+ * Instead this polls two REST reads that are BOTH Mongo-backed (not
+ * in-process), so they work no matter which pod actually executed the run:
+ *  - GET /api/scraps/:id/activities?history=<hid>&limit=20 — the SAME
+ *    activities-list endpoint `doctor`'s fetchRunAndFix already calls
+ *    (src/commands/doctor.ts) — prints each new activity line once the new
+ *    run's history id is known.
+ *  - GET /api/scraps/:id — history[0].status/statusDetail, the SAME
+ *    terminal-status signal `lastStatus()` above already trusts (status:null
+ *    === in flight, #88 item 1) to know when the run is done.
+ *
+ * #93 item 1 — dedup-race fix. "Is this the run we're watching?" used to be
+ * a single check: `history[0]._id !== beforeHistoryId`. That's wrong when
+ * `before.alreadyInFlight` is true (a `trigger` call deduped onto a worker
+ * job that was ALREADY pending/running at capture time): the top row IS the
+ * run we're watching, but its `_id` never changes, so the old guard never
+ * released and the poll ran the full timeout to a false "Timed out". The run
+ * we're watching is now EITHER a brand-new id (fresh trigger, the common
+ * case) OR the same id that was already in-flight (status:null) at capture
+ * (the dedup case) — a same-id row that was already TERMINAL at capture is
+ * neither, and must not be latched onto as "done" (it's just the previous
+ * run, still sitting there until a genuinely new run supersedes it).
+ */
+export async function pollRunProgress(id, before, opts = {}) {
+    const intervalMs = opts.intervalMs ?? POLL_INTERVAL_MS;
+    const timeoutMs = opts.timeoutMs ?? POLL_TIMEOUT_MS;
+    const beforeId = before?.id;
+    const beforeAlreadyInFlight = before?.alreadyInFlight ?? false;
+    console.log(chalk.dim('Live activity streaming has no signal for this run (async/cross-pod) — polling for progress instead…\n'));
+    const deadline = Date.now() + timeoutMs;
+    const seen = new Set();
+    let first = true;
+    while (Date.now() < deadline) {
+        if (!first)
+            await sleep(intervalMs);
+        first = false;
+        let scrap;
+        try {
+            scrap = await api.get(`/api/scraps/${id}`);
+        }
+        catch {
+            continue; // transient — keep polling rather than aborting the wait
+        }
+        const last = scrap.history?.[0];
+        if (!last?._id)
+            continue; // no history row recorded yet
+        const isNewRun = last._id !== beforeId;
+        const isDedupOntoInFlight = last._id === beforeId && beforeAlreadyInFlight;
+        if (!isNewRun && !isDedupOntoInFlight)
+            continue; // still the stale previous run
+        try {
+            const activities = await api.get(`/api/scraps/${id}/activities?history=${last._id}&limit=20`);
+            // Server returns newest-first — print unseen ones oldest-first.
+            for (const a of [...(activities ?? [])].reverse()) {
+                const key = a._id ?? `${a.createdAt}:${a.message}`;
+                if (seen.has(key))
+                    continue;
+                seen.add(key);
+                const time = new Date(a.createdAt).toLocaleTimeString();
+                console.log(`${chalk.dim(`[${time}]`)} ${a.message}`);
+            }
+        }
+        catch {
+            // best-effort — terminal detection below still works without activity lines
+        }
+        if (last.status !== null) {
+            const outcome = last.statusDetail ?? (last.status ? 'success' : 'failure');
+            console.log(chalk.dim(`Run finished: ${outcome}`));
+            return;
+        }
+    }
+    console.log(chalk.yellow(`⚠ Timed out waiting for the run to finish — check status with: trawl scraps doctor ${id}`));
+}
 // list
 scraps
     .command('list')
     .alias('ls')
     .description('List all scraps')
     .option('--json', 'Output as JSON')
-    .option('--status <status>', 'Filter by last run status (success|failure|never|running)')
+    .option('--status <status>', 'Filter by last run status (success|failure|never|running|regression)')
     // #88 item 8 — no custom parser here (unlike the old `(v) => parseInt(v,
     // 10)`): a bad value like "abc" used to silently become NaN, which then
     // sailed straight through `Number.isInteger`-less checks and into
@@ -430,15 +558,21 @@ scraps
 scraps
     .command('run <id>')
     .description('Run a scrap')
-    .option('-w, --watch', 'Stream activities after launching')
+    .option('-w, --watch', 'Show progress after launching (polls — see `trawl scraps trigger --watch`, #91)')
     .action(async (id, opts) => {
     validateObjectId(id);
-    await oraPromise(() => api.get(`/api/scraps/load/${id}`), {
+    // #91 P1 / #93 item 1 — captured BEFORE launching so pollRunProgress can
+    // tell "the run that's about to finish" apart from whatever the last run
+    // happened to be (including a dedup onto an already-in-flight run).
+    const beforeRun = opts.watch ? await captureBeforeRunState(id) : undefined;
+    // #91 P0 — GET /api/scraps/load/:id runs the scrap synchronously
+    // server-side (30-250s); the 30s default was aborting it mid-flight.
+    await oraPromise(() => api.get(`/api/scraps/load/${id}`, { timeoutMs: LONG_RUN_TIMEOUT_MS }), {
         text: 'Launching scrap…',
         successText: 'Scrap launched',
     });
     if (opts.watch) {
-        await watchActivities(id);
+        await pollRunProgress(id, beforeRun);
     }
 });
 // #70 — render an items array either as a table summary or --json. Shared by
@@ -518,7 +652,9 @@ scraps
     // burning execute quota and 429ing if a run is already in flight. A user
     // or agent "just reading data" must never trigger that by accident.
     if (opts.fresh) {
-        const loaded = await oraPromise(() => api.get(`/api/scraps/load/${id}`), {
+        // #91 P0 — same long-run endpoint as `scraps run` (30-250s server-side);
+        // the 30s default was aborting it mid-flight.
+        const loaded = await oraPromise(() => api.get(`/api/scraps/load/${id}`, { timeoutMs: LONG_RUN_TIMEOUT_MS }), {
             text: 'Launching a fresh scrap run (consumes execute quota)…',
             successText: 'Fresh run complete',
         });
@@ -533,6 +669,27 @@ scraps
             console.log(chalk.dim('No data yet. Run the scrap first.'));
             return;
         }
+        // #93 item 2 — --fresh used to render items without ever checking for a
+        // regression, unlike the persisted `data` path below (#88 item 2). The
+        // load() response's OWN embedded `scrap.history[0]` can't be trusted for
+        // this (see the ScrapLoadResult comment above): it may be the previous
+        // run's row, and even the right row never carries the regression flip.
+        // `--fresh` runs synchronously to completion server-side though — by the
+        // time this call returns, node has already awaited the regression patch
+        // — so a fresh GET /api/scraps/:id (the SAME read the persisted path
+        // below already trusts) reliably observes the finalized DB state.
+        // Best-effort: never fail --fresh's real output over this side check,
+        // and never gate on it — the items returned ARE this run's real,
+        // synchronously-computed data regardless of what this check finds.
+        try {
+            const fresh = await api.get(`/api/scraps/${id}`);
+            if (fresh.history?.[0]?.statusDetail === 'regression') {
+                console.error(chalk.yellow(`⚠ Item count regressed vs baseline for the last run of ${id} — see: trawl scraps doctor ${id}`));
+            }
+        }
+        catch {
+            // best-effort — the fresh run's items are still valid without this check
+        }
         renderScrapItems(items, opts.json);
         return;
     }
@@ -713,7 +870,6 @@ scraps
         usageError(`File not found: ${opts.file}`);
         return;
     }
-    const fileBuffer = readFileSync(opts.file);
     const filename = basename(opts.file);
     const ext = filename.split('.').pop()?.toLowerCase() ?? '';
     const mimeMap = {
@@ -722,7 +878,15 @@ scraps
         jpeg: 'image/jpeg',
         webp: 'image/webp',
     };
-    const mimeType = mimeMap[ext] ?? 'image/png';
+    // #91 P2 — an unsupported/unknown extension silently became image/png
+    // (uploading the raw bytes of, say, a .gif or .pdf under an image/png
+    // Content-Type — a lie about the actual file's format). Refuse instead.
+    const mimeType = mimeMap[ext];
+    if (!mimeType) {
+        usageError(`Unsupported image type "${ext ? `.${ext}` : filename}" — use png, jpg, or webp.`);
+        return;
+    }
+    const fileBuffer = readFileSync(opts.file);
     const blob = new Blob([fileBuffer], { type: mimeType });
     const formData = new FormData();
     formData.append('banner', blob, filename);
@@ -743,20 +907,31 @@ scraps
 scraps
     .command('trigger <id>')
     .description('Launch a scrap as a background worker (returns immediately)')
-    .option('-w, --watch', 'Stream activities after triggering')
+    .option('-w, --watch', 'Poll for progress after triggering (#91 — the default async run happens in a separate cron pod; activities SSE never reaches it)')
     .option('--wait', 'Run synchronously and wait for the result (legacy behaviour)')
     .action(async (id, opts) => {
     validateObjectId(id);
+    // #91 P1 / #93 item 1 — captured BEFORE triggering so pollRunProgress can
+    // tell "the run we just triggered" apart from whatever the last run
+    // happened to be. This is the dedup-prone path: `trigger`'s method:'worker'
+    // collapses onto an already-pending/running worker job for the same scrap
+    // (ScrapJobsService, LIVE_STATUSES) instead of creating a new history row —
+    // captureBeforeRunState records that so pollRunProgress can still track it.
+    const beforeRun = opts.watch ? await captureBeforeRunState(id) : undefined;
     // #50 — default async: the backend (#1313) kicks off the run and returns a
     // 'queued' envelope immediately instead of holding the connection for the
     // whole run. --wait restores the old synchronous round-trip.
     const path = opts.wait ? `/api/scraps/worker/${id}` : `/api/scraps/worker/${id}?wait=false`;
-    await oraPromise(() => api.post(path), {
+    // #91 P0 — the synchronous --wait branch runs the scrap server-side
+    // (30-250s), same as `scraps run`; the 30s default was aborting it
+    // mid-flight. The async (default) POST returns almost immediately, so it
+    // keeps the 30s default.
+    await oraPromise(() => (opts.wait ? api.post(path, undefined, { timeoutMs: LONG_RUN_TIMEOUT_MS }) : api.post(path)), {
         text: opts.wait ? 'Running worker…' : 'Triggering worker…',
         successText: opts.wait ? 'Worker run complete' : 'Worker triggered',
     });
     if (opts.watch)
-        await watchActivities(id);
+        await pollRunProgress(id, beforeRun);
 });
 // account subcommand group
 const account = scraps
@@ -991,11 +1166,26 @@ scraps
     .description('Download captured page HTML for the last run of a scrap')
     .option('--error', 'Fetch the error-path snapshot (errorSnapshot.html)')
     .option('-o, --out <file>', 'Write HTML to a file instead of stdout')
+    .option('--json', 'On a never-run scrap, output {"status":"no_runs"} (exit 0) instead of prose')
     .action(async (id, opts) => {
     validateObjectId(id);
     const scrap = await api.get(`/api/scraps/${id}`);
     const hid = scrap.history?.[0]?._id;
     if (!hid) {
+        // #91 P2 — same no-runs contract as `doctor`/`autofix`/`data --errors`
+        // (`{status:"no_runs"}`, exit 0) under --json.
+        if (opts.json) {
+            json({ status: 'no_runs' });
+            return;
+        }
+        // `-o` explicitly asked for a file to be written. Silently exiting 0
+        // with nothing written (and only a dim console line) is indistinguishable
+        // from success to a script checking the exit code alone — give it the
+        // SAME not_found envelope `scraps data`'s own never-run case uses.
+        if (opts.out) {
+            reportDataState(`Scrap ${id} has never run — nothing to write to ${opts.out}.`, 4, 'not_found', false);
+            return;
+        }
         console.log(chalk.dim('No runs yet.'));
         return;
     }

package/dist/commands/skills.js

@@ -31,7 +31,34 @@ skills
     for (const name of bundled) {
         const userInstalled = isSkillInstalled(name, 'user');
         const localInstalled = isSkillInstalled(name, 'local');
-        const installedVersion = getInstalledVersion(name, 'user') ?? getInstalledVersion(name, 'local');
+        // #91 — same EISDIR class as autoUpdateInstalledSkills/installSkill: an
+        // unreadable `.version` marker on one skill must not crash the whole
+        // listing before the other skills are shown.
+        //
+        // #93 item 3 — the two scope reads must be guarded INDEPENDENTLY. A
+        // single try/catch wrapped around `getInstalledVersion(name, 'user') ??
+        // getInstalledVersion(name, 'local')` still throws the whole expression
+        // the moment the 'user' read throws (e.g. a corrupt/EISDIR `.version`
+        // marker) — `??` never gets a chance to evaluate the 'local' fallback,
+        // so a healthy local install gets masked as "not installed" too. Every
+        // other marker read in this codebase (installSkill, removeOrphanedSkills,
+        // autoUpdateInstalledSkills — all in lib/skills.ts) already guards each
+        // scope on its own; this was the one chained exception.
+        let installedVersion;
+        try {
+            installedVersion = getInstalledVersion(name, 'user');
+        }
+        catch {
+            installedVersion = null;
+        }
+        if (installedVersion === null) {
+            try {
+                installedVersion = getInstalledVersion(name, 'local');
+            }
+            catch {
+                installedVersion = null;
+            }
+        }
         const stale = installedVersion && installedVersion !== version;
         const tag = userInstalled
             ? localInstalled

package/dist/lib/api.d.ts

@@ -37,14 +37,32 @@ export declare class AuthError extends Error {
  * the same thing everywhere it can be observed. (#86 findings 1/2, #88 item 4)
  */
 export declare function notLoggedInError(): AuthError;
+/**
+ * #91 P0 — some endpoints legitimately run 30–250s server-side: a scrap
+ * execute (worker-puppeteer navigation + antibot tier escalation + AI-fix
+ * dry-run retries). The generic 30s default was aborting those mid-flight
+ * and surfacing a fabricated `NetworkError timed out` (exit 5) for a request
+ * that was always going to succeed given enough time. Passed as the per-call
+ * `{timeoutMs}` override at exactly the 3 call sites that hit those
+ * endpoints (src/commands/scraps.ts): `scraps run` / `data --fresh`
+ * (GET /api/scraps/load/:id) and `scraps trigger --wait`
+ * (POST /api/scraps/worker/:id, synchronous branch only — the default async
+ * `?wait=false` POST returns almost immediately and keeps the 30s default).
+ * 300s leaves margin over the ~250s worst case without being unboundedly long.
+ */
+export declare const LONG_RUN_TIMEOUT_MS = 300000;
+export interface RequestOptions {
+    /** Per-call timeout override in ms (e.g. LONG_RUN_TIMEOUT_MS). Ignored — env wins — when TRAWL_TIMEOUT is set; see getTimeoutMs. */
+    timeoutMs?: number;
+}
 export declare const api: {
-    get: <T>(path: string) => Promise<T>;
-    getText: (path: string) => Promise<string>;
-    post: <T>(path: string, body?: unknown) => Promise<T>;
-    put: <T>(path: string, body?: unknown) => Promise<T>;
-    delete: <T>(path: string) => Promise<T>;
-    upload: <T>(path: string, formData: FormData) => Promise<T>;
-    publicPost: <T>(path: string, body?: unknown, baseUrlOverride?: string) => Promise<{
+    get: <T>(path: string, opts?: RequestOptions) => Promise<T>;
+    getText: (path: string, opts?: RequestOptions) => Promise<string>;
+    post: <T>(path: string, body?: unknown, opts?: RequestOptions) => Promise<T>;
+    put: <T>(path: string, body?: unknown, opts?: RequestOptions) => Promise<T>;
+    delete: <T>(path: string, opts?: RequestOptions) => Promise<T>;
+    upload: <T>(path: string, formData: FormData, opts?: RequestOptions) => Promise<T>;
+    publicPost: <T>(path: string, body?: unknown, baseUrlOverride?: string, opts?: RequestOptions) => Promise<{
         data: T;
         headers: Headers;
     }>;

package/dist/lib/api.js

@@ -57,28 +57,62 @@ export function notLoggedInError() {
     return new AuthError('Not logged in. Run: trawl login');
 }
 const DEFAULT_TIMEOUT_MS = 30_000;
-/** Effective fetch timeout — TRAWL_TIMEOUT env override (ms), default 30s. (#71) */
-function getTimeoutMs() {
+/**
+ * #91 P0 — some endpoints legitimately run 30–250s server-side: a scrap
+ * execute (worker-puppeteer navigation + antibot tier escalation + AI-fix
+ * dry-run retries). The generic 30s default was aborting those mid-flight
+ * and surfacing a fabricated `NetworkError timed out` (exit 5) for a request
+ * that was always going to succeed given enough time. Passed as the per-call
+ * `{timeoutMs}` override at exactly the 3 call sites that hit those
+ * endpoints (src/commands/scraps.ts): `scraps run` / `data --fresh`
+ * (GET /api/scraps/load/:id) and `scraps trigger --wait`
+ * (POST /api/scraps/worker/:id, synchronous branch only — the default async
+ * `?wait=false` POST returns almost immediately and keeps the 30s default).
+ * 300s leaves margin over the ~250s worst case without being unboundedly long.
+ */
+export const LONG_RUN_TIMEOUT_MS = 300_000;
+/**
+ * Effective fetch timeout for one request.
+ *
+ * Precedence (#91): TRAWL_TIMEOUT (env) ALWAYS wins when set to a valid
+ * positive number — an operator/CI override must be able to force a shorter
+ * or longer ceiling globally (e.g. a slow CI network, or a deliberately tight
+ * smoke-test budget) without editing call sites. Only when the env var is
+ * unset/invalid does the caller-supplied per-call `overrideMs` apply (e.g.
+ * LONG_RUN_TIMEOUT_MS for the 3 long-run call sites); otherwise
+ * DEFAULT_TIMEOUT_MS. This preserves #71's original "env always overrides"
+ * contract while layering the new per-call default beneath it, never above.
+ */
+function getTimeoutMs(overrideMs) {
     const raw = process.env['TRAWL_TIMEOUT']?.trim();
-    if (!raw)
-        return DEFAULT_TIMEOUT_MS;
-    const n = Number(raw);
-    return Number.isFinite(n) && n > 0 ? n : DEFAULT_TIMEOUT_MS;
+    if (raw) {
+        const n = Number(raw);
+        if (Number.isFinite(n) && n > 0)
+            return n;
+    }
+    return overrideMs !== undefined && overrideMs > 0 ? overrideMs : DEFAULT_TIMEOUT_MS;
 }
 /**
  * Wrap a `fetch()` call so connection-level failures (ECONNREFUSED, DNS,
  * timeout, …) surface as a NetworkError carrying the effective URL + the
  * unwrapped `err.cause` detail, instead of a bare "fetch failed" with no
  * actionable information. (#71 findings 4/58)
+ *
+ * `effectiveTimeoutMs` is the ACTUAL timeout that was armed for this specific
+ * call (already resolved via getTimeoutMs by the caller) — used only to
+ * report an honest number in the timeout message; #91 — before this it
+ * always re-derived a fresh `getTimeoutMs()` with no override, so a request
+ * armed with LONG_RUN_TIMEOUT_MS that timed out would have lied and reported
+ * "timed out after 30000ms".
  */
-async function safeFetch(url, options) {
+async function safeFetch(url, options, effectiveTimeoutMs) {
     try {
         return await fetch(url, options);
     }
     catch (err) {
         const e = err;
         if (e?.name === 'TimeoutError' || e?.name === 'AbortError') {
-            throw new NetworkError(`Request to ${url} timed out after ${getTimeoutMs()}ms (override with TRAWL_TIMEOUT env var, ms)`);
+            throw new NetworkError(`Request to ${url} timed out after ${effectiveTimeoutMs}ms (override with TRAWL_TIMEOUT env var, ms)`);
         }
         const cause = e?.cause;
         const causeDetail = cause?.code ? ` (${cause.code})` : cause?.message ? ` (${cause.message})` : '';
@@ -194,21 +228,22 @@ async function throwIfError(res, isPublic = false) {
         throw new ApiError(res.status, `${res.status} ${res.statusText}: ${full}`);
     }
 }
-async function request(path, options = {}) {
+async function request(path, options = {}, reqOpts = {}) {
     const token = getToken();
     if (!token)
         throw notLoggedInError();
     const url = `${getApiUrl()}${path}`;
+    const timeoutMs = getTimeoutMs(reqOpts.timeoutMs);
     const res = await safeFetch(url, {
         ...options,
-        signal: AbortSignal.timeout(getTimeoutMs()),
+        signal: AbortSignal.timeout(timeoutMs),
         headers: {
             'Content-Type': 'application/json',
             'User-Agent': USER_AGENT,
             ...options.headers,
             Cookie: `TOKEN=${token}`,
         },
-    });
+    }, timeoutMs);
     await throwIfError(res);
     const text = await res.text();
     try {
@@ -225,21 +260,22 @@ async function request(path, options = {}) {
         throw new Error('Invalid JSON in server response');
     }
 }
-async function upload(path, formData) {
+async function upload(path, formData, reqOpts = {}) {
     const token = getToken();
     if (!token)
         throw notLoggedInError();
     const url = `${getApiUrl()}${path}`;
+    const timeoutMs = getTimeoutMs(reqOpts.timeoutMs);
     // Do NOT set Content-Type — fetch sets it automatically with the correct multipart boundary
     const res = await safeFetch(url, {
         method: 'POST',
         body: formData,
-        signal: AbortSignal.timeout(getTimeoutMs()),
+        signal: AbortSignal.timeout(timeoutMs),
         headers: {
             'User-Agent': USER_AGENT,
             Cookie: `TOKEN=${token}`,
         },
-    });
+    }, timeoutMs);
     await throwIfError(res);
     const text = await res.text();
     try {
@@ -256,14 +292,15 @@ async function upload(path, formData) {
         throw new Error('Invalid JSON in server response');
     }
 }
-async function publicPost(path, body, baseUrlOverride) {
+async function publicPost(path, body, baseUrlOverride, reqOpts = {}) {
     const url = `${baseUrlOverride ?? getApiUrl()}${path}`;
+    const timeoutMs = getTimeoutMs(reqOpts.timeoutMs);
     const res = await safeFetch(url, {
         method: 'POST',
         headers: { 'Content-Type': 'application/json', 'User-Agent': USER_AGENT },
         body: body ? JSON.stringify(body) : undefined,
-        signal: AbortSignal.timeout(getTimeoutMs()),
-    });
+        signal: AbortSignal.timeout(timeoutMs),
+    }, timeoutMs);
     await throwIfError(res, true);
     const text = await res.text();
     try {
@@ -274,35 +311,36 @@ async function publicPost(path, body, baseUrlOverride) {
         throw new Error('Invalid JSON in server response');
     }
 }
-async function getText(path) {
+async function getText(path, reqOpts = {}) {
     const token = getToken();
     if (!token)
         throw notLoggedInError();
     const url = `${getApiUrl()}${path}`;
+    const timeoutMs = getTimeoutMs(reqOpts.timeoutMs);
     const res = await safeFetch(url, {
         headers: {
             'User-Agent': USER_AGENT,
             Cookie: `TOKEN=${token}`,
         },
-        signal: AbortSignal.timeout(getTimeoutMs()),
-    });
+        signal: AbortSignal.timeout(timeoutMs),
+    }, timeoutMs);
     await throwIfError(res);
     return res.text();
 }
 export const api = {
-    get: (path) => request(path),
-    getText: (path) => getText(path),
-    post: (path, body) => request(path, {
+    get: (path, opts) => request(path, {}, opts),
+    getText: (path, opts) => getText(path, opts),
+    post: (path, body, opts) => request(path, {
         method: 'POST',
         body: body ? JSON.stringify(body) : undefined,
-    }),
-    put: (path, body) => request(path, {
+    }, opts),
+    put: (path, body, opts) => request(path, {
         method: 'PUT',
         body: body ? JSON.stringify(body) : undefined,
-    }),
-    delete: (path) => request(path, { method: 'DELETE' }),
-    upload: (path, formData) => upload(path, formData),
-    publicPost: (path, body, baseUrlOverride) => publicPost(path, body, baseUrlOverride),
+    }, opts),
+    delete: (path, opts) => request(path, { method: 'DELETE' }, opts),
+    upload: (path, formData, opts) => upload(path, formData, opts),
+    publicPost: (path, body, baseUrlOverride, opts) => publicPost(path, body, baseUrlOverride, opts),
     stream: async function* (path) {
         const token = getToken();
         if (!token)
@@ -311,13 +349,15 @@ export const api = {
         // No AbortSignal.timeout here — a long-running `watch`/`--watch` stream is
         // expected to sit open indefinitely; only connection-level failures
         // (never a timeout) should surface via safeFetch's cause-unwrapping. (#71)
+        // effectiveTimeoutMs passed to safeFetch here is only ever used to format
+        // a timeout message that can't actually fire (no signal attached).
         const res = await safeFetch(url, {
             headers: {
                 Accept: 'text/event-stream',
                 'User-Agent': USER_AGENT,
                 Cookie: `TOKEN=${token}`,
             },
-        });
+        }, getTimeoutMs());
         if (!res.ok || !res.body) {
             throw new ApiError(res.status, `SSE failed: ${res.status}`);
         }

package/dist/lib/skills.js

@@ -43,7 +43,20 @@ export function installSkill(name, scope, opts = {}) {
     }
     const dest = join(getSkillsBase(scope), name);
     if (existsSync(dest)) {
-        const owned = getInstalledVersion(name, scope) !== null;
+        // #91 — an unreadable `.version` (e.g. EISDIR from a directory instead of
+        // a file, a corrupted/interrupted install) is NOT proof of ownership
+        // either — treat it exactly like a missing marker (unowned) instead of
+        // letting the raw fs error crash the whole batch (`skills install`/
+        // `update` with no skill argument loops over every bundled skill; one
+        // malformed dest must not block installing the others).
+        let installedVersion;
+        try {
+            installedVersion = getInstalledVersion(name, scope);
+        }
+        catch {
+            installedVersion = null;
+        }
+        const owned = installedVersion !== null;
         if (!owned && !opts.force) {
             throw new Error(`Refusing to overwrite "${dest}" — it was not installed by trawl (no .version marker). ` +
                 `Pass --force to overwrite it anyway.`);
@@ -147,7 +160,21 @@ export function autoUpdateInstalledSkills() {
             for (const scope of ['user', 'local']) {
                 if (!isSkillInstalled(name, scope))
                     continue;
-                const installed = getInstalledVersion(name, scope);
+                // #91 — mirrors removeOrphanedSkills' guard below: a weird `.version`
+                // marker (e.g. a directory instead of a file, from a corrupted /
+                // interrupted install) throws EISDIR on readFileSync. Without this
+                // per-entry guard, that throw was caught by this function's OUTER
+                // try/catch (below) — which aborts the ENTIRE function, so every
+                // remaining bundled skill silently stopped syncing AND the orphan
+                // sweep (removeOrphanedSkills, called after this loop) never ran
+                // either. Must skip only THIS entry, never abort the whole sweep.
+                let installed;
+                try {
+                    installed = getInstalledVersion(name, scope);
+                }
+                catch {
+                    continue;
+                }
                 // No `.version` marker → not ours → never delete it.
                 if (installed === null)
                     continue;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@trawlme/cli",
-  "version": "1.18.3",
+  "version": "1.18.5",
   "description": "Trawl CLI — manage scraps from the terminal",
   "type": "module",
   "bin": {
@@ -40,7 +40,7 @@
     "url": "https://github.com/comes-io/trawl_cli/issues"
   },
   "dependencies": {
-    "@trawlme/skills": "1.3.2",
+    "@trawlme/skills": "1.3.3",
     "chalk": "^5.6.2",
     "commander": "^14.0.3",
     "conf": "^15.1.0",