@trawlme/cli 3.12.5 → 3.12.6

package/README.md

@@ -68,18 +68,20 @@ trawl spec [--json]                            Print a versioned, machine-readab
 
 > **No breaking change:** every verb above is also still reachable under its pre-reorg path, `trawl scraps <verb>` (e.g. `trawl scraps list`, `trawl scraps run <id>`) — kept as a hidden alias so scripts written before the surface reorg keep working. `trawl --help` only shows the top-level form above; `trawl scraps --help` only shows the remaining scrap-management commands below.
 
-`create`/`whoami`/`ping` are fully non-interactive — all three read auth only from `TRAWL_API_KEY`/`TRAWL_TOKEN`/the stored login token, never prompt (`whoami` is JWT-only — see [Authentication](#authentication) — so it still refuses honestly under a key instead of prompting). `trawl create` runs the AI wizard server-side (`POST /api/ai/wizard`): generate scrap code from `--prompt` via LLM, persist the scrap, trigger its FIRST run, and auto-fix on failure (default on — `--no-autofix` disables it, sending `autoFix:false`). On a successful first run (human mode) it prints a small **data sample** (item count + first-item fields + one truncated value) as proof of value — best-effort, silent if the sample can't be fetched — and points `Next step` at `trawl data <id>` (the data), with `trawl get <id>` as the secondary detail view. `--json` skips the sample fetch and prints the raw wizard payload verbatim. `success` is an honest outcome of that first run, not "did the HTTP call succeed" — a failed first run is still a 200 response (the scrap was still created; auto-fix, when enabled, retries in the background), and the CLI exits 1 in that case (both human and `--json` modes) even though `--json` always prints the raw payload verbatim. The call legitimately takes many minutes server-side (AI generation + a proxy-tier probe ladder + an iteration loop + a real run), so it arms a 1500s ceiling of its own — longer than the 300s `run`/`data --fresh`/`trigger --wait` use below, and over a transport of its own (Node's global `fetch` cannot wait past ~300s whatever ceiling is asked for). `trawl whoami`/`trawl ping` mirror the MCP `trawl_whoami`/`trawl_health_ping` tools as closely as the REST surface allows (`GET /api/users/me` / `GET /api/health`) — `ping`'s `--json` payload is admin-enriched (version/uptime/db) and just `{"status":"ok"}` for anyone else.
+`create`/`whoami`/`ping` are fully non-interactive — all three read auth only from `TRAWL_API_KEY`/`TRAWL_TOKEN`/the stored login token, never prompt (`whoami` is JWT-only — see [Authentication](#authentication) — so it still refuses honestly under a key instead of prompting). `trawl create` runs the AI wizard server-side (`POST /api/ai/wizard`): generate scrap code from `--prompt` via LLM, persist the scrap, trigger its FIRST run, and auto-fix on failure (default on — `--no-autofix` disables it, sending `autoFix:false`). On a successful first run (human mode) it prints a small **data sample** (item count + first-item fields + one truncated value) as proof of value — best-effort, silent if the sample can't be fetched — and points `Next step` at `trawl data <id>` (the data), with `trawl get <id>` as the secondary detail view. `--json` skips the sample fetch and prints the raw wizard payload verbatim. `success` is an honest outcome of that first run, not "did the HTTP call succeed" — a failed first run is still a 200 response (the scrap was still created; auto-fix, when enabled, retries in the background), and the CLI exits 1 in that case (both human and `--json` modes) even though `--json` always prints the raw payload verbatim. The call legitimately takes many minutes server-side (AI generation + a proxy-tier probe ladder + an iteration loop + a real run), so it reads the wizard's NDJSON progress stream (`Accept: application/x-ndjson`) — human mode prints one short line per stage on stderr, `--json` prints only the final payload, same shape as always — and arms a 3600s safety ceiling of its own, longer than the 660s `run`/`data --fresh`/`trigger --wait` use below, over a transport of its own (Node's global `fetch` cannot wait past ~300s whatever ceiling is asked for). `trawl whoami`/`trawl ping` mirror the MCP `trawl_whoami`/`trawl_health_ping` tools as closely as the REST surface allows (`GET /api/users/me` / `GET /api/health`) — `ping`'s `--json` payload is admin-enriched (version/uptime/db) and just `{"status":"ok"}` for anyone else.
 
-> **`create` is NOT idempotent, and every wizard-created scrap runs on a DAILY cron by default.** A client-side transport failure (exit `5`, a `NetworkError`) — a timeout, or a connection dropped after the API already answered — does not mean the wizard failed server-side: scrap creation + the first run keep going after the CLI gives up waiting, so the scrap may already exist. In human mode the CLI prints a stderr warning whenever the request reached the API without returning a complete answer; under `--json` there is no warning and no extra envelope field — a machine caller sees exit `5` and must check `trawl list` itself. Run `trawl list` and look for a matching URL/title **before** retrying — a blind retry creates a DUPLICATE scrap and burns AI-generation quota a second time for the same goal. Separately, the scrap the wizard creates is scheduled to re-run every day at 07:00 UTC (`cron: "0 7 * * *"`, hardcoded server-side, unrelated to `--no-autofix`) — each of those recurring runs consumes execute quota like any other run. Review the generated scrap, then change or disable the schedule with `trawl scraps update <id> --cron <expr>` (or `--no-cron` to disable it). Because the call can legitimately run many minutes, also confirm `TRAWL_TIMEOUT` isn't set to something tighter than `create` needs — the env var always wins over `create`'s own 1500s default (see [Environment variables](#environment-variables)), so a value set for another purpose (e.g. a tight CI smoke-test budget) silently clamps `create` too; unset it or raise it before running `create`.
+> **`create` is NOT idempotent, and every wizard-created scrap runs on a WEEKLY cron by default.** A transport failure — a client-side timeout, a progress stream cut or ended without a result, a connection dropped after the API already answered (exit `5`, a `NetworkError`), or a gateway `502`/`504` — does not mean the wizard failed server-side: scrap creation + the first run keep going after the CLI gives up waiting, so the scrap may already exist. In human mode the CLI prints a stderr warning whenever the request reached the API without returning a complete answer; under `--json` there is no warning and no extra envelope field — a machine caller sees exit `5` (or, for a gateway `502`/`504`, exit `1` with `status` 502/504) and must check `trawl list` itself. Run `trawl list` and look for a matching URL/title **before** retrying — a blind retry creates a DUPLICATE scrap and burns AI-generation quota a second time for the same goal. Separately, the scrap the wizard creates is scheduled to re-run every week, on Monday at 07:00 UTC (`cron: "0 7 * * 1"`, hardcoded server-side, unrelated to `--no-autofix`) — each of those recurring runs consumes execute quota like any other run. Review the generated scrap, then change or disable the schedule with `trawl scraps update <id> --cron <expr>` (or `--no-cron` to disable it). Because the call can legitimately run many minutes, also confirm `TRAWL_TIMEOUT` isn't set to something tighter than `create` needs — the env var always wins over `create`'s own 3600s default (see [Environment variables](#environment-variables)), so a value set for another purpose (e.g. a tight CI smoke-test budget) silently clamps `create` too; unset it or raise it before running `create`.
+
+> **A failure the wizard reports itself** — the stream's terminal `error` event — arrives after the HTTP `200` was already sent, so there is no status to report: `create` exits `1` as `kind:"refused"` with **no `status` field** (`{"error":{"message":"…","kind":"refused","retryable":false}}` under `--json`, the server's message verbatim; `✗ <message>` on stderr otherwise, stripped of control characters and capped at 200 characters). Branch on `kind`, not on a status.
 
 - `list` has a short alias, `ls` (matches `trawl --help`'s `list|ls`).
 - `history` lists past runs (newest first); `run-info <hid>` shows details of a single run from that history.
 - `data` returns the last persisted run payload (no execute quota); `--fresh` runs the scrap live instead (consumes execute quota); `--errors` shows the last run's error detail (`--json` on a never-run scrap returns `{"status":"no_runs"}`, exit 0, matching `scraps doctor --json`). `[]` on stdout means a genuine zero-item successful run — a scrap that has never run, whose last run failed, or whose payload aged out of retention returns a `--json` error envelope (exit 4/1/4 respectively) instead. Two more honest states: a run still **in flight** (`status: null` server-side) returns a `kind:"in_progress"` error envelope (exit 1, "retry shortly" — never suggests `--fresh`, which would just 429 against the run already holding the lock); a run whose item count **regressed** vs baseline (`statusDetail: "regression"`) still returns the real, non-empty items on stdout (exit 0) plus a stderr warning pointing at `scraps doctor <id>` — the data itself is genuine even though the run is flagged.
 - `get` (and anything reading through it, like `data`'s default path) embeds only the newest 100 history rows on the returned scrap object — `run-info` and `scraps doctor` fetch a single run directly and are unaffected by that cap. `list`/`get` show a distinct amber `▼` "regression" badge, never the red `✗` a genuine failure gets (matches `scraps doctor`'s own badge).
 - `list`/`get` also show a **health** badge next to the status badge. By default it reads the scrap's own `lastCronOutcome` field (always present, no extra cost): `⚠ cron paused` when the last scheduled tick was skipped for being unhealthy, `—` otherwise — a breadcrumb of the last tick, not a live read (never set on a no-cron scrap, can lag by one tick). `list --unhealthy` asks the API to filter to scraps whose *current* consecutive-failure streak is 3+ (the server's own threshold) and annotates each with the exact `consecutiveFailedRuns`/`unhealthySince` — shown instead of the badge whenever present, and never re-derived client-side from history. `--json` carries whichever fields the request produced: `lastCronOutcome` always, `consecutiveFailedRuns`/`unhealthySince` only when `--unhealthy` was passed. **Old-server safety:** an older server version that predates this filter silently ignores the unknown `--unhealthy` query param and returns everything, unfiltered — the CLI detects that (no returned item carries `consecutiveFailedRuns`) and prints a stderr warning instead of presenting the full list as "your unhealthy scraps".
-- **Long-running calls (`run`, `data --fresh`, `trigger --wait`):** these hit server-side paths that can legitimately take 30–250s+ (proxy tier escalation + AI-fix retries) — the CLI arms a 300s timeout for exactly these three call sites instead of the generic 30s default.
-- **`create` is longer still:** the wizard adds a proxy-tier probe ladder and an iteration loop on top of generation + creation + a first run, so it arms **1500s** for that one call site (600s probe ladder + 480s iteration loop + 250s worst-case first run + margin). Raising the 300s above to cover it would make the three calls above wait 25 minutes on an unreachable API, so the two ceilings are deliberately separate. `create` is also the one call that does not go through Node's global `fetch`, which caps any wait for response headers at ~300s regardless of the ceiling asked for. `TRAWL_TIMEOUT` (see below) still overrides ALL requests, including both — set it if you need a tighter or looser ceiling, but note a global override sized for a quick call also clamps `create`.
-- **`--watch` is poll-based, not a live stream:** the activities SSE endpoint has no backlog and, for the default async `trigger` (no `--wait`), runs in a separate cron-consumer pod whose events never reach the API pod holding the SSE connection — a naive "await the run, then open SSE" shows nothing. `run --watch` and `trigger --watch` instead poll `GET /api/scraps/:id` (terminal status) and the activities REST list until the run finishes, printing each new activity line as it appears. The watched run's outcome drives the exit code too, in BOTH human and `--json` mode: a genuinely failed terminal run, a poll timeout, or a persistently unreachable API all exit non-zero — a clean successful run is the only exit `0`. A run that never reaches a terminal status within 300s prints an honest timeout notice pointing at `scraps doctor <id>` (human mode) — see the `--json` shape below. `scraps watch <id>` (the standalone command, no trigger) is unchanged — it still opens the live SSE stream directly.
+- **Long-running calls (`run`, `data --fresh`, `trigger --wait`):** these hit server-side paths that can legitimately take minutes (proxy tier escalation + AI-fix retries, up to the server's 600s per-run budget) — the CLI arms a 660s timeout (600s + 60s) for exactly these three call sites instead of the generic 30s default, over the same raw `node:http(s)` transport `create` uses: Node's global `fetch` kills any call at ~300s whatever ceiling is asked for.
+- **`create` is longer still:** the wizard adds a proxy-tier probe ladder and an iteration loop on top of generation + creation + a first run, so it arms **3600s** for that one call site (600s probe ladder + 900s iteration loop + 600s for a dry-run still in flight + 600s first run + margin) — a safety ceiling only, since `create` reads the wizard's live NDJSON progress stream. Raising the 660s above to cover it would make the three calls above wait an hour on an unreachable API, so the two ceilings are deliberately separate. `TRAWL_TIMEOUT` (see below) still overrides ALL requests, including both — set it if you need a tighter or looser ceiling, but note a global override sized for a quick call also clamps `create`.
+- **`--watch` is poll-based, not a live stream:** the activities SSE endpoint has no backlog and, for the default async `trigger` (no `--wait`), runs in a separate cron-consumer pod whose events never reach the API pod holding the SSE connection — a naive "await the run, then open SSE" shows nothing. `run --watch` and `trigger --watch` instead poll `GET /api/scraps/:id` (terminal status) and the activities REST list until the run finishes, printing each new activity line as it appears. The watched run's outcome drives the exit code too, in BOTH human and `--json` mode: a genuinely failed terminal run, a poll timeout, or a persistently unreachable API all exit non-zero — a clean successful run is the only exit `0`. A run that never reaches a terminal status within 660s prints an honest timeout notice pointing at `scraps doctor <id>` (human mode) — see the `--json` shape below. `scraps watch <id>` (the standalone command, no trigger) is unchanged — it still opens the live SSE stream directly.
 - `spec --json` prints the CLI's own command tree — `{specVersion, cliVersion, commands[], exitCodes, errorKinds, kindExitCodes, docsUrl?, llmsUrl?}` — DERIVED at runtime by walking the live commander tree (never a hand-maintained file, which would silently drift from reality). Each entry in `commands[]` carries its full path (e.g. `"scraps account session set"`), description, `hidden` (the legacy `scraps <verb>` aliases above), `leaf` (false for a pure namespace/group node like `scraps`/`skills`/`telemetry` — invoking one directly is a guaranteed-failing tool, not a real command), `aliases`, `arguments`, `options`, and an optional per-command `docs` deep link (e.g. every `scraps account *` command points at the account-sessions guide); `exitCodes`/`errorKinds`/`kindExitCodes` are read from the exact same source `classifyError` uses (see [Exit codes](#exit-codes)) — never a second copy. `kindExitCodes` is the inverse of `exitCodes`: `kind -> exitCode`, since exit code `1` alone is a shared bucket (`api`/`refused`/`unknown`/`in_progress`/`run_failed`/`upgrade_failed`) that `exitCodes`' flat label can't disambiguate. `docsUrl` is resolved server-first (`externalDocs.url` on the configured API base's own OpenAPI document, when declared), else derived from a KNOWN first-party `trawl.me` API host, else **omitted entirely** — never a guessed URL pointed at the wrong docs host for a self-hosted install. `llmsUrl` and every per-command `docs` link are narrower: they only ever come from that same known-host derivation, NEVER from a server-declared `docsUrl` — this CLI's own guide slugs have no reason to exist on a third party's own docs root, so a self-hosted server that declares `externalDocs.url` gets a correct top-level `docsUrl` but no `llmsUrl` and no per-command `docs` links. A `failureKind:"auth"` run's JSON payload (`doctor`/`data --errors`/`run-info`) carries the same `docs` field, by the same known-host-only rule. Without `--json`, `spec` prints one short human line (version + visible command count) pointing at `--json`.
 - `run --json`/`trigger --json` bypass the spinner and print the raw launch/trigger payload on stdout; combined with `--watch`, every intermediate progress line stays suppressed (stdout stays pure JSON) and, once the watch reaches its outcome, exactly ONE final NDJSON line is emitted: `{"runId","status"}` (the honest terminal status — `success`/`error`/`empty`/`regression`/…), `{"runId","status":"timeout"}` on a poll timeout, or `{"runId","status":"poll_error","error"}` if the API stays unreachable for several consecutive polls — `process.exitCode` is non-zero for all three except a genuine success. `scraps watch --json` emits one raw JSON object per activity line (NDJSON) instead of the formatted `[time] message` text — there's no single final payload to wait for on a live stream.
 
@@ -238,7 +240,7 @@ Under `--json`, a failing command emits a single error envelope on stdout — `{
 | `TRAWL_TELEMETRY`     | Set to `0` to disable telemetry for the current session           |
 | `DO_NOT_TRACK`        | Set to `1` to disable telemetry (cross-vendor convention, https://consoledonottrack.com) — same effect as `TRAWL_TELEMETRY=0` |
 | `TRAWL_CONFIG_DIR`    | Override where the config file (token, API URL, telemetry state) is stored — useful for hermetic CI runs or concurrent `trawl login`s that must not share one on-disk file |
-| `TRAWL_TIMEOUT`       | Override the per-request fetch timeout in milliseconds (default `30000`; `300000` for `run`/`data --fresh`/`trigger --wait`, `1500000` for `create` — this env var always wins over those longer defaults too) |
+| `TRAWL_TIMEOUT`       | Override the per-request fetch timeout in milliseconds (default `30000`; `660000` for `run`/`data --fresh`/`trigger --wait`, `3600000` for `create` — this env var always wins over those longer defaults too) |
 | `TRAWL_SKILLS_SYNC`   | Set to `0` to disable the startup skills auto-sync entirely       |
 
 Session override (no `trawl login` mutation, ideal for CI/QA against another env):

package/dist/commands/create.js

@@ -1,11 +1,11 @@
 import { Command } from 'commander';
 import chalk from 'chalk';
 import { spin } from '../lib/spinner.js';
-import { api, WIZARD_TIMEOUT_MS, NetworkError } from '../lib/api.js';
-import { json } from '../lib/format.js';
+import { api, WIZARD_TIMEOUT_MS, ApiError, NetworkError, StreamError } from '../lib/api.js';
+import { json, printable } from '../lib/format.js';
 import { parseServerJson } from '../lib/json.js';
 import { requireUrl, requireString } from '../lib/validate.js';
-import { UsageError } from '../lib/errors.js';
+import { UsageError, RefusalError, HUMAN_ERROR_MAX } from '../lib/errors.js';
 import { renderPinch, pinchEnabled } from '../lib/pinch.js';
 import { startPinchAnimation } from '../lib/pinchAnimation.js';
 function firstRunLabel(data, autoFixEnabled) {
@@ -17,21 +17,24 @@ function firstRunLabel(data, autoFixEnabled) {
         return 'failed (auto-fix retrying in the background)';
     return 'failed';
 }
+const WEEKDAYS = ['Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday'];
 function describeCron(cron) {
-    const match = /^(\d{1,2})\s+(\d{1,2})\s+\*\s+\*\s+\*$/.exec(cron.trim());
+    const match = /^(\d{1,2})\s+(\d{1,2})\s+\*\s+\*\s+(\*|[0-7])$/.exec(cron.trim());
     if (!match)
         return cron;
-    const [, min, hour] = match;
-    return `daily ${hour.padStart(2, '0')}:${min.padStart(2, '0')}`;
+    const [, min, hour, dow] = match;
+    const time = `${hour.padStart(2, '0')}:${min.padStart(2, '0')}`;
+    return dow === '*' ? `daily ${time}` : `weekly ${WEEKDAYS[Number(dow) % 7]} ${time}`;
 }
 function scheduleLabel(scrap) {
     if (!scrap)
         return null;
-    if (typeof scrap.cron === 'string' && scrap.cron.trim() !== '') {
-        const tz = typeof scrap.cronTimezone === 'string' && scrap.cronTimezone ? scrap.cronTimezone : 'UTC';
-        return `${describeCron(scrap.cron)} ${tz} (cron ${scrap.cron})`;
+    const cron = typeof scrap.cron === 'string' ? printable(scrap.cron) : '';
+    if (cron !== '') {
+        const tz = (typeof scrap.cronTimezone === 'string' ? printable(scrap.cronTimezone) : '') || 'UTC';
+        return `${describeCron(cron)} ${tz} (cron ${cron})`;
     }
-    return 'scheduled daily by default';
+    return 'scheduled weekly by default (Monday 07:00 UTC)';
 }
 async function printDataSample(historyId) {
     try {
@@ -45,18 +48,36 @@ async function printDataSample(historyId) {
         const first = items[0];
         if (first && typeof first === 'object') {
             const keys = Object.keys(first);
-            console.log(chalk.dim(`  Fields:     `) + keys.join(', '));
+            console.log(chalk.dim(`  Fields:     `) + printable(keys.join(', '), HUMAN_ERROR_MAX));
             const firstKey = keys[0];
             if (firstKey) {
-                const raw = String(first[firstKey] ?? '');
-                const val = raw.length > 60 ? `${raw.slice(0, 60)}…` : raw;
-                console.log(chalk.dim(`  ${firstKey.slice(0, 10).padEnd(10)}: `) + val);
+                const val = printable(first[firstKey], 61);
+                console.log(chalk.dim(`  ${printable(firstKey).slice(0, 10).padEnd(10)}: `) + val);
             }
         }
     }
     catch {
     }
 }
+function stageLine(event) {
+    if (event.type !== 'stage')
+        return null;
+    const label = typeof event['label'] === 'string' ? printable(event['label']) : '';
+    if (label !== '')
+        return label;
+    const stage = typeof event['stage'] === 'string' ? printable(event['stage']) : '';
+    if (stage === '')
+        return null;
+    if (stage === 'probing') {
+        const tier = event['tier'];
+        const shown = typeof tier === 'string' || typeof tier === 'number' ? printable(String(tier)) : '';
+        return shown !== '' ? `Probing the site (tier ${shown})` : 'Probing the site';
+    }
+    return stage.charAt(0).toUpperCase() + stage.slice(1);
+}
+function mayHaveCompleted(err) {
+    return (err instanceof NetworkError || err instanceof ApiError) && err.mayHaveCompleted;
+}
 export const create = new Command('create')
     .description('Create a persistent, self-healing scrap from a URL + a goal (AI-generated) — distinct from `trawl scraps create`, which is raw manual entry')
     .argument('[url]', 'Target public URL (http/https)')
@@ -78,24 +99,41 @@ export const create = new Command('create')
         goal,
         ...(opts.autofix === false && { autoFix: false }),
     };
-    const call = () => api.postLongRunning('/api/ai/wizard', body, { timeoutMs: WIZARD_TIMEOUT_MS });
+    const call = (onEvent) => api
+        .postLongRunning('/api/ai/wizard', body, { timeoutMs: WIZARD_TIMEOUT_MS, onEvent })
+        .catch((err) => {
+        if (!(err instanceof StreamError))
+            throw err;
+        throw new RefusalError(opts.json ? err.message : printable(err.message) || 'the server reported an error');
+    });
     let data;
     if (opts.json) {
-        data = await call();
+        data = await call(() => { });
     }
     else {
         const anim = pinchEnabled() ? startPinchAnimation(`Creating a scrap from ${url}…`) : null;
+        let lastStage = '';
+        const showStage = (event) => {
+            const line = stageLine(event);
+            if (line === null || line === lastStage)
+                return;
+            lastStage = line;
+            if (anim)
+                anim.setStatus(line);
+            else
+                console.error(`${process.stderr.isTTY ? '\r\x1b[2K' : ''}${chalk.dim(`  ${line}`)}`);
+        };
         try {
             data = anim
-                ? await call()
-                : await spin(call, {
+                ? await call(showStage)
+                : await spin(() => call(showStage), {
                     text: `Creating a scrap from ${url}…`,
                     successText: 'Request complete',
                 });
         }
         catch (err) {
             anim?.stop();
-            if (err instanceof NetworkError && err.mayHaveCompleted) {
+            if (mayHaveCompleted(err)) {
                 console.error(chalk.yellow('⚠ The request did not complete client-side, but the scrap may STILL have been created server-side — run `trawl list` before retrying (a retry creates a DUPLICATE scrap + burns quota).'));
             }
             throw err;
@@ -106,14 +144,15 @@ export const create = new Command('create')
         json(data);
     }
     else {
-        const scrapId = data.scrap?._id;
-        const title = data.scrap?.title ?? url;
+        const scrapId = printable(data.scrap?._id);
+        const historyId = printable(data.historyId);
+        const title = printable(data.scrap?.title ?? url);
         const autoFixEnabled = opts.autofix !== false;
         const schedule = scheduleLabel(data.scrap);
         console.log(`${data.success ? chalk.green('✓') : chalk.red('✗')} ${chalk.bold(title)}`);
-        console.log(chalk.dim(`  Scrap ID:   `) + (scrapId ?? '—'));
-        if (data.historyId)
-            console.log(chalk.dim(`  History ID: `) + data.historyId);
+        console.log(chalk.dim(`  Scrap ID:   `) + (scrapId || '—'));
+        if (historyId)
+            console.log(chalk.dim(`  History ID: `) + historyId);
         console.log(chalk.dim(`  First run:  `) + firstRunLabel(data, autoFixEnabled));
         if (schedule)
             console.log(chalk.dim(`  Schedule:   `) + schedule);
@@ -123,8 +162,8 @@ export const create = new Command('create')
             console.log(chalk.dim(`  Next step:  `) + `trawl data ${scrapId}` + chalk.dim(`  (details: trawl get ${scrapId})`));
         }
         if (!data.success && data.scrap && autoFixEnabled) {
-            const pollTarget = data.historyId
-                ? `\`trawl data ${scrapId}\` or \`trawl run-info ${data.historyId}\``
+            const pollTarget = historyId
+                ? `\`trawl data ${scrapId}\` or \`trawl run-info ${historyId}\``
                 : `\`trawl data ${scrapId}\``;
             console.log(chalk.yellow(`  Note:       `) +
                 `Auto-fix is retrying in the background — do NOT re-run create; poll ${pollTarget}.`);

package/dist/commands/scraps.js

@@ -2,11 +2,11 @@ import { Command } from 'commander';
 import chalk from 'chalk';
 import { spin, confirmNonTTY } from '../lib/spinner.js';
 import { api, LONG_RUN_TIMEOUT_MS } from '../lib/api.js';
-import { table, json, formatDate } from '../lib/format.js';
+import { table, json, jsonText, formatDate, printable } from '../lib/format.js';
 import { parseServerJson } from '../lib/json.js';
 import { promptPassword } from '../lib/prompt.js';
 import { validateObjectId, requireUrl } from '../lib/validate.js';
-import { classifyError, reportError, retryFieldsFor, UsageError, RefusalError } from '../lib/errors.js';
+import { classifyError, reportError, retryFieldsFor, UsageError, RefusalError, HUMAN_ERROR_MAX } from '../lib/errors.js';
 import { confirmDestructive, isInteractive } from '../lib/confirm.js';
 import { formatDoctor, formatAutofix, fetchRunAndFix, pickRun, pickFix, isAuthWall } from './doctor.js';
 import { renderPinch, pinchEnabled } from '../lib/pinch.js';
@@ -127,11 +127,11 @@ async function watchActivities(id, asJson) {
         try {
             const activity = parseServerJson(event);
             if (asJson) {
-                console.log(JSON.stringify(activity));
+                console.log(jsonText(activity));
                 continue;
             }
             const time = new Date(activity.createdAt).toLocaleTimeString();
-            console.log(`${chalk.dim(`[${time}]`)} ${activity.message}`);
+            console.log(`${chalk.dim(`[${time}]`)} ${printable(activity.message, HUMAN_ERROR_MAX)}`);
         }
         catch {
             if (process.env['DEBUG'] && event.trim()) {
@@ -185,10 +185,10 @@ export async function pollRunProgress(id, before, opts = {}) {
                 const message = err instanceof Error ? err.message : String(err);
                 if (asJson) {
                     const outcome = { runId: beforeId, status: 'poll_error', error: message };
-                    console.log(JSON.stringify(outcome));
+                    console.log(jsonText(outcome));
                 }
                 else {
-                    console.log(chalk.red(`✗ Polling failed repeatedly (${message}) — check status with: trawl scraps doctor ${id}`));
+                    console.log(chalk.red(`✗ Polling failed repeatedly (${printable(message, HUMAN_ERROR_MAX)}) — check status with: trawl scraps doctor ${id}`));
                 }
                 process.exitCode = 1;
                 return;
@@ -217,7 +217,7 @@ export async function pollRunProgress(id, before, opts = {}) {
                 if (asJson)
                     continue;
                 const time = new Date(a.createdAt).toLocaleTimeString();
-                console.log(`${chalk.dim(`[${time}]`)} ${a.message}`);
+                console.log(`${chalk.dim(`[${time}]`)} ${printable(a.message, HUMAN_ERROR_MAX)}`);
             }
         }
         catch {
@@ -227,10 +227,10 @@ export async function pollRunProgress(id, before, opts = {}) {
             const isGenuineFailure = last.status === false && outcome !== 'empty' && outcome !== 'regression';
             if (asJson) {
                 const payload = { runId: last._id, status: outcome };
-                console.log(JSON.stringify(payload));
+                console.log(jsonText(payload));
             }
             else {
-                console.log(chalk.dim(`Run finished: ${outcome}`));
+                console.log(chalk.dim(`Run finished: ${printable(outcome)}`));
                 if (last.status === true && pinchEnabled()) {
                     console.log(renderPinch('celebrating'));
                 }
@@ -241,7 +241,7 @@ export async function pollRunProgress(id, before, opts = {}) {
     }
     if (asJson) {
         const outcome = { runId: beforeId, status: 'timeout' };
-        console.log(JSON.stringify(outcome));
+        console.log(jsonText(outcome));
     }
     else {
         console.log(chalk.yellow(`⚠ Timed out waiting for the run to finish — check status with: trawl scraps doctor ${id}`));
@@ -308,14 +308,14 @@ export function attachListCommand(parent, attachOpts = {}) {
         catch (err) {
             const isDebug = Boolean(cmd.optsWithGlobals().debug || process.env['DEBUG']);
             const { exitCode, envelope } = classifyError(err);
-            const message = `Failed to fetch scraps: ${envelope.message}`;
+            const prefix = 'Failed to fetch scraps: ';
             if (isDebug)
                 console.error(err);
             if (opts.json) {
-                console.log(JSON.stringify({ error: { ...envelope, message } }));
+                console.log(jsonText({ error: { ...envelope, message: prefix + envelope.message } }));
             }
             else if (!isDebug) {
-                console.error(chalk.red(`✗ ${message}`));
+                console.error(chalk.red(`✗ ${prefix}${printable(envelope.message, HUMAN_ERROR_MAX)}`));
             }
             process.exitCode = exitCode;
             return;
@@ -336,7 +336,9 @@ export function attachListCommand(parent, attachOpts = {}) {
             'last run': lastRun(s),
             updated: formatDate(s.updatedAt),
         }));
-        table(tableRows, ['id', 'title', 'cron', 'status', 'health', 'last run', 'updated']);
+        table(tableRows, ['id', 'title', 'cron', 'status', 'health', 'last run', 'updated'], {
+            styled: ['status', 'health'],
+        });
         if (limit !== undefined && rows.length < totalMatched) {
             console.log(chalk.dim(`Showing ${rows.length} of ${totalMatched} — omit --limit to see all`));
         }
@@ -353,9 +355,9 @@ export function attachGetCommand(parent, attachOpts = {}) {
         const data = await api.get(`/api/scraps/${id}`);
         if (opts.json)
             return json(data);
-        console.log(chalk.bold(data.title || '(untitled)'));
-        console.log(chalk.dim(`  ID:       `) + data._id);
-        console.log(chalk.dim(`  Cron:     `) + (data.cron || '—'));
+        console.log(chalk.bold(printable(data.title) || '(untitled)'));
+        console.log(chalk.dim(`  ID:       `) + printable(data._id));
+        console.log(chalk.dim(`  Cron:     `) + (printable(data.cron) || '—'));
         console.log(chalk.dim(`  Status:   `) + statusIcon(lastStatus(data)));
         console.log(chalk.dim(`  Health:   `) + healthBadge(data));
         console.log(chalk.dim(`  Last run: `) + lastRun(data));
@@ -369,22 +371,23 @@ function renderTierOverrideHuman(data) {
     if (!ov)
         return;
     if (ov.refused) {
-        console.error(chalk.red(`  ✗ tier ceiling override refused: ${ov.reason ?? 'unknown'}`)
-            + chalk.dim(` (requested ${ov.requestedMaxTier ?? '—'}; kept the registry cap)`));
+        console.error(chalk.red(`  ✗ tier ceiling override refused: ${printable(ov.reason, HUMAN_ERROR_MAX) || 'unknown'}`)
+            + chalk.dim(` (requested ${printable(ov.requestedMaxTier) || '—'}; kept the registry cap)`));
     }
     else if (ov.effectiveMaxTier) {
-        console.log(chalk.green(`  ✓ tier ceiling: ${ov.effectiveMaxTier}`)
-            + chalk.dim(` (${ov.reason ?? ''}${ov.provider ? `, ${ov.provider}` : ''})`));
+        const provider = printable(ov.provider);
+        console.log(chalk.green(`  ✓ tier ceiling: ${printable(ov.effectiveMaxTier)}`)
+            + chalk.dim(` (${printable(ov.reason, HUMAN_ERROR_MAX)}${provider ? `, ${provider}` : ''})`));
         if (ov.warning)
-            console.log(chalk.yellow(`    ⚠ ${ov.warning}`));
+            console.log(chalk.yellow(`    ⚠ ${printable(ov.warning, HUMAN_ERROR_MAX)}`));
     }
     if (ov.proxyTier) {
         if (ov.proxyTier.clamped) {
-            console.log(chalk.yellow(`  ⚠ proxyTier requested ${ov.proxyTier.requested} → applied ${ov.proxyTier.effective}`)
-                + chalk.dim(` (${ov.proxyTier.reason ?? 'capped'})`));
+            console.log(chalk.yellow(`  ⚠ proxyTier requested ${printable(ov.proxyTier.requested)} → applied ${printable(ov.proxyTier.effective)}`)
+                + chalk.dim(` (${printable(ov.proxyTier.reason, HUMAN_ERROR_MAX) || 'capped'})`));
         }
         else {
-            console.log(chalk.dim(`  proxyTier: `) + ov.proxyTier.effective);
+            console.log(chalk.dim(`  proxyTier: `) + printable(ov.proxyTier.effective));
         }
     }
 }
@@ -426,7 +429,7 @@ scraps
         request: opts.request || '',
         ...(opts.description !== undefined && { description: opts.description }),
         ...(opts.tier !== undefined && { proxyTier: opts.tier }),
-    }), { text: 'Creating scrap…', successText: (d) => `Scrap created: ${chalk.bold(d._id)}` });
+    }), { text: 'Creating scrap…', successText: (d) => `Scrap created: ${chalk.bold(printable(d._id))}` });
     const tierWasRequested = opts.tier !== undefined;
     warnIfUnconfirmedTier(data, tierWasRequested, data._id);
     const refused = Boolean(data._tierOverride?.refused);
@@ -438,7 +441,7 @@ scraps
         json(withTierUnconfirmed(data, tierWasRequested));
         return;
     }
-    console.log(chalk.dim(`  Title: ${data.title}`));
+    console.log(chalk.dim(`  Title: ${printable(data.title)}`));
     renderTierOverrideHuman(data);
     if (refused)
         process.exitCode = 1;
@@ -536,7 +539,7 @@ scraps
     }
     const data = await spin(() => api.put(`/api/scraps/${id}`, body), {
         text: 'Updating scrap…',
-        successText: (d) => `Scrap updated: ${chalk.bold(d._id)}`,
+        successText: (d) => `Scrap updated: ${chalk.bold(printable(d._id))}`,
     });
     const tierWasRequested = opts.tier !== undefined || opts.forceTier !== undefined;
     warnIfUnconfirmedTier(data, tierWasRequested, id);
@@ -554,7 +557,7 @@ scraps
         if (key === 'proxyTier' || key === 'proxyMaxTier')
             continue;
         const src = key in shown ? shown[key] : body[key];
-        console.log(chalk.dim(`  ${key}: `) + String(src ?? '—'));
+        console.log(chalk.dim(`  ${key}: `) + (printable(src, HUMAN_ERROR_MAX) || '—'));
     }
     renderTierOverrideHuman(data);
     if (refused)
@@ -569,7 +572,7 @@ export function attachRunCommand(parent, attachOpts = {}) {
         .action(async (id, opts) => {
         validateObjectId(id);
         const beforeRun = opts.watch ? await captureBeforeRunState(id) : undefined;
-        const call = () => api.get(`/api/scraps/load/${id}`, { timeoutMs: LONG_RUN_TIMEOUT_MS });
+        const call = () => api.getLongRunning(`/api/scraps/load/${id}`, { timeoutMs: LONG_RUN_TIMEOUT_MS });
         const data = opts.json
             ? await call()
             : await spin(call, { text: 'Launching scrap…', successText: 'Scrap launched' });
@@ -595,7 +598,7 @@ function renderScrapItems(items, asJson) {
     console.log(chalk.bold('Last run data:'));
     console.log(chalk.dim(`  Items: ${items.length}`));
     if (items.length > 0 && typeof items[0] === 'object' && items[0] !== null) {
-        console.log(chalk.dim(`  First item keys: ${Object.keys(items[0]).join(', ')}`));
+        console.log(chalk.dim(`  First item keys: ${printable(Object.keys(items[0]).join(', '), HUMAN_ERROR_MAX)}`));
     }
     console.log(chalk.dim('  Use --json for full output.'));
 }
@@ -604,7 +607,7 @@ function reportDataState(message, exitCode, kind, wantsJson, retryOverride) {
     if (wantsJson) {
         const { retryable, next } = retryOverride ?? retryFieldsFor(kind);
         const envelope = { message, kind, retryable, ...(next ? { next } : {}) };
-        console.log(JSON.stringify({ error: envelope }));
+        console.log(jsonText({ error: envelope }));
     }
     process.exitCode = exitCode;
 }
@@ -640,7 +643,7 @@ export function attachDataCommand(parent, attachOpts = {}) {
             return;
         }
         if (opts.fresh) {
-            const loaded = await spin(() => api.get(`/api/scraps/load/${id}`, { timeoutMs: LONG_RUN_TIMEOUT_MS }), {
+            const loaded = await spin(() => api.getLongRunning(`/api/scraps/load/${id}`, { timeoutMs: LONG_RUN_TIMEOUT_MS }), {
                 text: 'Launching a fresh scrap run (consumes execute quota)…',
                 successText: 'Fresh run complete',
             });
@@ -857,7 +860,7 @@ export function attachTriggerCommand(parent, attachOpts = {}) {
         validateObjectId(id);
         const beforeRun = opts.watch ? await captureBeforeRunState(id) : undefined;
         const path = opts.wait ? `/api/scraps/worker/${id}` : `/api/scraps/worker/${id}?wait=false`;
-        const call = () => (opts.wait ? api.post(path, undefined, { timeoutMs: LONG_RUN_TIMEOUT_MS }) : api.post(path));
+        const call = () => (opts.wait ? api.postLongRunning(path, undefined, { timeoutMs: LONG_RUN_TIMEOUT_MS }) : api.post(path));
         const data = opts.json
             ? await call()
             : await spin(call, {
@@ -1121,7 +1124,7 @@ accountSession
         reportCaptureFailure({ ok: false, reason: 'non_interactive', message: nonInteractiveReason }, opts);
         return;
     }
-    console.error(chalk.dim(`Chrome will open at ${scrap.url} for you to log in there (2FA included); capture completes when you press Enter back in this terminal.`));
+    console.error(chalk.dim(`Chrome will open at ${printable(scrap.url)} for you to log in there (2FA included); capture completes when you press Enter back in this terminal.`));
     const result = await captureSession(scrap.url, opts.chrome ? { findChrome: () => opts.chrome ?? null } : {});
     if (!result.ok) {
         reportCaptureFailure(result, opts);

package/dist/lib/api.d.ts

@@ -2,6 +2,10 @@ export declare class ApiError extends Error {
     status: number;
     next?: string[] | undefined;
     constructor(status: number, message: string, next?: string[] | undefined);
+    mayHaveCompleted: boolean;
+}
+export declare class StreamError extends Error {
+    constructor(message: string);
 }
 export declare class NetworkError extends Error {
     readonly mayHaveCompleted: boolean;
@@ -13,13 +17,20 @@ export declare class AuthError extends Error {
 }
 export declare function notLoggedInError(): AuthError;
 export declare function apiKeyUnsupportedError(command: string): AuthError;
-export declare const LONG_RUN_TIMEOUT_MS = 300000;
-export declare const WIZARD_TIMEOUT_MS = 1500000;
+export declare const LONG_RUN_TIMEOUT_MS = 660000;
+export declare const WIZARD_TIMEOUT_MS = 3600000;
 export interface RequestOptions {
     timeoutMs?: number;
 }
 export declare function effectivePort(u: URL): string;
 export declare function sameEffectivePort(from: URL, next: URL): boolean;
+export interface StreamEvent {
+    type: string;
+    [key: string]: unknown;
+}
+export interface LongRunningOptions extends RequestOptions {
+    onEvent?: (event: StreamEvent) => void;
+}
 export declare const api: {
     get: <T>(path: string, opts?: RequestOptions) => Promise<T>;
     publicGet: <T>(path: string, opts?: RequestOptions) => Promise<T>;
@@ -28,7 +39,8 @@ export declare const api: {
     }) => Promise<T | null>;
     getText: (path: string, opts?: RequestOptions) => Promise<string>;
     post: <T>(path: string, body?: unknown, opts?: RequestOptions) => Promise<T>;
-    postLongRunning: <T>(path: string, body?: unknown, opts?: RequestOptions) => Promise<T>;
+    postLongRunning: <T>(path: string, body?: unknown, opts?: LongRunningOptions) => Promise<T>;
+    getLongRunning: <T>(path: string, opts?: RequestOptions) => Promise<T>;
     put: <T>(path: string, body?: unknown, opts?: RequestOptions) => Promise<T>;
     delete: <T>(path: string, opts?: RequestOptions) => Promise<T>;
     upload: <T>(path: string, formData: FormData, opts?: RequestOptions) => Promise<T>;

package/dist/lib/api.js

@@ -4,7 +4,9 @@ import { dirname, resolve } from 'node:path';
 import { request as httpRequest, } from 'node:http';
 import { request as httpsRequest } from 'node:https';
 import { brotliDecompressSync, gunzipSync, inflateRawSync, inflateSync } from 'node:zlib';
+import { StringDecoder } from 'node:string_decoder';
 import { getApiUrl, getToken, getAuthMode, getLiveAuthEnvVar } from './config.js';
+import { printable } from './format.js';
 import { parseServerJson } from './json.js';
 const __dirname = dirname(fileURLToPath(import.meta.url));
 const pkg = JSON.parse(readFileSync(resolve(__dirname, '../../package.json'), 'utf8'));
@@ -18,6 +20,13 @@ export class ApiError extends Error {
         this.next = next;
         this.name = 'ApiError';
     }
+    mayHaveCompleted = false;
+}
+export class StreamError extends Error {
+    constructor(message) {
+        super(message);
+        this.name = 'StreamError';
+    }
 }
 export class NetworkError extends Error {
     mayHaveCompleted;
@@ -64,8 +73,8 @@ function authHeaders(token) {
     return getAuthMode(token) === 'apiKey' ? { Authorization: `Bearer ${token}` } : { Cookie: `TOKEN=${token}` };
 }
 const DEFAULT_TIMEOUT_MS = 30_000;
-export const LONG_RUN_TIMEOUT_MS = 300_000;
-export const WIZARD_TIMEOUT_MS = 1_500_000;
+export const LONG_RUN_TIMEOUT_MS = 660_000;
+export const WIZARD_TIMEOUT_MS = 3_600_000;
 function getTimeoutMs(overrideMs) {
     const raw = process.env['TRAWL_TIMEOUT']?.trim();
     if (raw) {
@@ -373,6 +382,8 @@ const MAX_REDIRECTS = 3;
 const REDIRECT_KEEPS_METHOD = new Set([307, 308]);
 const REDIRECT_DROPS_BODY = new Set([301, 302, 303]);
 const MAX_DECODED_BYTES = 64 * 1024 * 1024;
+const MAX_BODY_BYTES = MAX_DECODED_BYTES;
+const MAX_STREAM_LINE_CHARS = 32 * 1024 * 1024;
 const DECODE_LIMIT = { maxOutputLength: MAX_DECODED_BYTES };
 function isOutputOverflow(err) {
     return err?.code === 'ERR_BUFFER_TOO_LARGE';
@@ -393,6 +404,7 @@ const CONTENT_DECODERS = Object.freeze({
     deflate: inflateEitherForm,
     br: (body) => brotliDecompressSync(body, DECODE_LIMIT),
 });
+const ACCEPT_ENCODING = 'gzip, deflate, br';
 function decodeBody(body, contentEncoding) {
     const encoding = (contentEncoding ?? '').trim().toLowerCase();
     if (encoding === '' || encoding === 'identity')
@@ -412,7 +424,35 @@ function decodeBody(body, contentEncoding) {
         return { reason: `Content-Encoding "${encoding}" failed to decode (${err.message})` };
     }
 }
-function postRaw(url, payload, headers, timeoutMs) {
+function createLineReader(onLine) {
+    let pending = '';
+    const emit = (line) => {
+        if (line.length > MAX_STREAM_LINE_CHARS)
+            return 'overflow';
+        return line.trim() !== '' && onLine(line) ? 'stop' : 'more';
+    };
+    return {
+        push(text) {
+            let start = 0;
+            for (let newline = text.indexOf('\n'); newline !== -1; newline = text.indexOf('\n', start)) {
+                const line = pending + text.slice(start, newline);
+                pending = '';
+                start = newline + 1;
+                const verdict = emit(line);
+                if (verdict !== 'more')
+                    return verdict;
+            }
+            pending += text.slice(start);
+            return pending.length > MAX_STREAM_LINE_CHARS ? 'overflow' : 'more';
+        },
+        end() {
+            const line = pending;
+            pending = '';
+            return emit(line);
+        },
+    };
+}
+function rawRequest(method, url, payload, headers, timeoutMs, onStreamLine) {
     return new Promise((resolve, reject) => {
         let target;
         try {
@@ -426,17 +466,23 @@ function postRaw(url, payload, headers, timeoutMs) {
             reject(connectionNetworkError(url, Object.assign(new Error(`unsupported protocol "${target.protocol}"`), { code: 'ERR_INVALID_PROTOCOL' })));
             return;
         }
+        const streaming = onStreamLine !== undefined;
         let settled = false;
         let timedOut = false;
         let mayHaveCompleted = false;
         let timer;
         let active;
+        const hops = [];
+        const release = () => {
+            for (const hop of hops)
+                hop.destroy();
+        };
         const settleWith = (err) => {
             if (settled)
                 return;
             settled = true;
             clearTimeout(timer);
-            active?.destroy();
+            release();
             reject(err);
         };
         const fail = (err) => {
@@ -447,58 +493,116 @@ function postRaw(url, payload, headers, timeoutMs) {
                 return;
             settled = true;
             clearTimeout(timer);
+            release();
             resolve(value);
         };
-        const onResponse = (res, from, hopsLeft, failThisHop) => {
+        const lineTooLong = () => new NetworkError(`The response stream from ${url} sent a line longer than ${MAX_STREAM_LINE_CHARS / (1024 * 1024)} MB — refusing to buffer it`, mayHaveCompleted);
+        const onResponse = (res, from, hopsLeft, failThisHop, hop) => {
             const status = res.statusCode ?? 0;
             const statusText = res.statusMessage ?? '';
             res.on('error', failThisHop);
-            if (REDIRECT_KEEPS_METHOD.has(status)) {
-                res.resume();
+            if (REDIRECT_KEEPS_METHOD.has(status) || (method === 'GET' && REDIRECT_DROPS_BODY.has(status))) {
                 const location = res.headers.location;
                 const next = location ? safeUrl(location, from) : null;
                 const refusal = !next
                     ? 'it carries no usable Location header'
                     : next.protocol !== 'http:' && next.protocol !== 'https:'
                         ? `its target uses an unsupported protocol ("${next.protocol}")`
-                        : next.hostname !== from.hostname
-                            ? `its target is a different host (${next.hostname}) — this request carries your credential`
-                            : !sameEffectivePort(from, next)
-                                ? `its target is a different port (${effectivePort(next)}) — this request carries your credential`
-                                : from.protocol === 'https:' && next.protocol === 'http:'
-                                    ? 'it downgrades https to http — this request carries your credential'
-                                    : hopsLeft <= 0
-                                        ? `it exceeds ${MAX_REDIRECTS} redirects`
-                                        : null;
+                        : introducesUserinfo(from, next)
+                            ? 'its target embeds credentials in the URL (user:password@) — this request carries your credential'
+                            : next.hostname !== from.hostname
+                                ? `its target is a different host (${next.hostname}) — this request carries your credential`
+                                : !sameEffectivePort(from, next)
+                                    ? `its target is a different port (${effectivePort(next)}) — this request carries your credential`
+                                    : from.protocol === 'https:' && next.protocol === 'http:'
+                                        ? 'it downgrades https to http — this request carries your credential'
+                                        : hopsLeft <= 0
+                                            ? `it exceeds ${MAX_REDIRECTS} redirects`
+                                            : null;
                 if (refusal || !next) {
-                    settleWith(new ApiError(status, `${status} ${statusText}: refused to follow this redirect because ${refusal}. Point TRAWL_API_URL at the final URL.`));
+                    settleWith(new ApiError(status, `${status} ${printable(statusText)}: refused to follow this redirect because ${refusal}. Point TRAWL_API_URL at the final URL.`));
                     return;
                 }
                 send(next, hopsLeft - 1);
+                hop.destroy();
                 return;
             }
             if (REDIRECT_DROPS_BODY.has(status)) {
-                res.resume();
-                settleWith(new ApiError(status, `${status} ${statusText}: this endpoint redirected the POST to ${res.headers.location ?? 'an unspecified location'}, and that status rewrites it to a bodyless GET — only 307/308 are followed. Point TRAWL_API_URL at the final URL.`));
+                settleWith(new ApiError(status, `${status} ${printable(statusText)}: this endpoint redirected the POST to ${printable(res.headers.location ?? 'an unspecified location')}, and that status rewrites it to a bodyless GET — only 307/308 are followed. Point TRAWL_API_URL at the final URL.`));
                 return;
             }
             mayHaveCompleted = status >= 200 && status < 300;
-            const chunks = [];
-            res.on('data', (chunk) => chunks.push(chunk));
-            res.on('end', () => {
-                const decoded = decodeBody(Buffer.concat(chunks), res.headers['content-encoding']);
-                if (!Buffer.isBuffer(decoded)) {
-                    settleWith(new NetworkError(`Unreadable response from ${url} (HTTP ${status}): ${decoded.reason} — the server ignored "Accept-Encoding: identity"`, mayHaveCompleted));
-                    return;
-                }
-                done({ status, statusText, headers: toHeaders(res.headers), text: decoded.toString('utf8') });
-            });
+            const lineSink = mayHaveCompleted && /ndjson/i.test(String(res.headers['content-type'] ?? '')) ? onStreamLine : undefined;
+            const encoding = String(res.headers['content-encoding'] ?? '').trim().toLowerCase();
+            if (lineSink && (encoding === '' || encoding === 'identity')) {
+                const decoder = new StringDecoder('utf8');
+                const reader = createLineReader(lineSink);
+                const finish = (verdict) => {
+                    if (verdict === 'overflow')
+                        settleWith(lineTooLong());
+                    else
+                        done({ status, statusText, headers: toHeaders(res.headers), text: '', streamed: true });
+                };
+                res.on('data', (chunk) => {
+                    if (settled)
+                        return;
+                    const verdict = reader.push(decoder.write(chunk));
+                    if (verdict !== 'more')
+                        finish(verdict);
+                });
+                res.on('end', () => {
+                    if (settled)
+                        return;
+                    let verdict = reader.push(decoder.end());
+                    if (verdict === 'more')
+                        verdict = reader.end();
+                    finish(verdict);
+                });
+            }
+            else {
+                const chunks = [];
+                let received = 0;
+                res.on('data', (chunk) => {
+                    if (settled)
+                        return;
+                    received += chunk.length;
+                    if (received > MAX_BODY_BYTES) {
+                        settleWith(new NetworkError(`Response from ${url} (HTTP ${status}) is larger than the ${MAX_BODY_BYTES / (1024 * 1024)} MB limit — refusing to buffer it`, mayHaveCompleted));
+                        return;
+                    }
+                    chunks.push(chunk);
+                });
+                res.on('end', () => {
+                    if (settled)
+                        return;
+                    const decoded = decodeBody(Buffer.concat(chunks), res.headers['content-encoding']);
+                    if (!Buffer.isBuffer(decoded)) {
+                        settleWith(new NetworkError(`Unreadable response from ${url} (HTTP ${status}): ${printable(decoded.reason)}${streaming ? ' — the server ignored "Accept-Encoding: identity"' : ''}`, mayHaveCompleted));
+                        return;
+                    }
+                    const text = decoded.toString('utf8');
+                    if (lineSink) {
+                        const reader = createLineReader(lineSink);
+                        let verdict = reader.push(text);
+                        if (verdict === 'more')
+                            verdict = reader.end();
+                        if (verdict === 'overflow')
+                            settleWith(lineTooLong());
+                        else
+                            done({ status, statusText, headers: toHeaders(res.headers), text: '', streamed: true });
+                        return;
+                    }
+                    done({ status, statusText, headers: toHeaders(res.headers), text });
+                });
+            }
             res.on('close', () => {
                 if (!res.complete)
                     failThisHop(new Error('response aborted before it finished'));
             });
         };
         const send = (to, hopsLeft) => {
+            if (settled)
+                return;
             mayHaveCompleted = false;
             const transport = to.protocol === 'http:' ? httpRequest : httpsRequest;
             let req;
@@ -508,13 +612,14 @@ function postRaw(url, payload, headers, timeoutMs) {
                 fail(err);
             };
             try {
-                req = transport(to, { method: 'POST', headers: { 'Accept-Encoding': 'identity', ...headers } }, (res) => onResponse(res, to, hopsLeft, failThisHop));
+                req = transport(to, { method, headers: { 'Accept-Encoding': streaming ? 'identity' : ACCEPT_ENCODING, ...headers } }, (res) => onResponse(res, to, hopsLeft, failThisHop, req));
             }
             catch (err) {
                 fail(err);
                 return;
             }
             active = req;
+            hops.push(req);
             req.on('error', failThisHop);
             if (payload !== undefined)
                 req.write(payload);
@@ -535,6 +640,10 @@ export function sameEffectivePort(from, next) {
         return true;
     return effectivePort(from) === effectivePort(next);
 }
+function introducesUserinfo(from, next) {
+    return ((next.username !== '' || next.password !== '') &&
+        (next.username !== from.username || next.password !== from.password));
+}
 function safeUrl(raw, base) {
     try {
         return new URL(raw, base);
@@ -543,20 +652,71 @@ function safeUrl(raw, base) {
         return null;
     }
 }
-async function postLongRunning(path, body, reqOpts = {}) {
+const GATEWAY_CUT_STATUSES = new Set([502, 504]);
+async function longRunning(method, path, body, reqOpts = {}) {
     const token = getToken();
     if (!token)
         throw notLoggedInError();
     const url = `${getApiUrl()}${path}`;
     const timeoutMs = getTimeoutMs(reqOpts.timeoutMs);
     const payload = body ? JSON.stringify(body) : undefined;
-    const raw = await postRaw(url, payload, {
+    const { onEvent } = reqOpts;
+    let terminal;
+    const onStreamLine = onEvent
+        ? (line) => {
+            if (terminal)
+                return true;
+            if (!line.trimStart().startsWith('{'))
+                return false;
+            let event;
+            try {
+                event = parseServerJson(line);
+            }
+            catch {
+                return false;
+            }
+            if (event === null || typeof event !== 'object' || typeof event.type !== 'string')
+                return false;
+            const { type, ...rest } = event;
+            if (type === 'done') {
+                terminal = { event: 'done', payload: rest };
+                return true;
+            }
+            if (type === 'error') {
+                const message = rest['message'];
+                terminal = { event: 'error', message: typeof message === 'string' && message ? message : 'the server reported an error' };
+                return true;
+            }
+            try {
+                onEvent({ type, ...rest });
+            }
+            catch {
+            }
+            return false;
+        }
+        : undefined;
+    const raw = await rawRequest(method, url, payload, {
         'Content-Type': 'application/json',
         'User-Agent': USER_AGENT,
+        ...(onEvent ? { Accept: 'application/x-ndjson' } : {}),
         ...(payload !== undefined ? { 'Content-Length': String(Buffer.byteLength(payload)) } : {}),
         ...authHeaders(token),
-    }, timeoutMs);
-    await throwIfError(toResponse(raw), false, getAuthMode(token));
+    }, timeoutMs, onStreamLine);
+    try {
+        await throwIfError(toResponse(raw), false, getAuthMode(token));
+    }
+    catch (err) {
+        if (err instanceof ApiError && GATEWAY_CUT_STATUSES.has(err.status))
+            err.mayHaveCompleted = true;
+        throw err;
+    }
+    if (raw.streamed) {
+        if (terminal?.event === 'done')
+            return terminal.payload;
+        if (terminal?.event === 'error')
+            throw new StreamError(terminal.message);
+        throw new NetworkError(`The response stream from ${url} ended before a result arrived (no "done" or "error" event)`, true);
+    }
     try {
         if (!raw.text)
             return {};
@@ -567,6 +727,8 @@ async function postLongRunning(path, body, reqOpts = {}) {
         return parsed;
     }
     catch {
+        if (onEvent === undefined)
+            throw new Error('Invalid JSON in server response');
         throw new NetworkError('Invalid JSON in server response', true);
     }
 }
@@ -595,7 +757,8 @@ export const api = {
         method: 'POST',
         body: body ? JSON.stringify(body) : undefined,
     }, opts),
-    postLongRunning: (path, body, opts) => postLongRunning(path, body, opts),
+    postLongRunning: (path, body, opts) => longRunning('POST', path, body, opts),
+    getLongRunning: (path, opts) => longRunning('GET', path, undefined, opts),
     put: (path, body, opts) => request(path, {
         method: 'PUT',
         body: body ? JSON.stringify(body) : undefined,

package/dist/lib/errors.d.ts

@@ -1,3 +1,4 @@
+import { HUMAN_ERROR_MAX } from './format.js';
 export declare class UsageError extends Error {
     constructor(message: string);
 }
@@ -37,6 +38,7 @@ export declare function retryFieldsFor(kind: string): {
     next?: string[];
 };
 export declare function classifyError(err: unknown): ClassifiedError;
+export { HUMAN_ERROR_MAX };
 export declare function reportError(err: unknown, opts?: {
     json?: boolean;
     quiet?: boolean;

package/dist/lib/errors.js

@@ -1,5 +1,6 @@
 import chalk from 'chalk';
 import { ApiError, AuthError, NetworkError } from './api.js';
+import { printable, jsonText, HUMAN_ERROR_MAX } from './format.js';
 export class UsageError extends Error {
     constructor(message) {
         super(message);
@@ -103,13 +104,14 @@ export function classifyError(err) {
     }
     return { exitCode: EXIT_CODES.UNKNOWN, envelope: { message, kind: 'unknown', ...retryFieldsFor('unknown') } };
 }
+export { HUMAN_ERROR_MAX };
 export function reportError(err, opts = {}) {
     const { exitCode, envelope } = classifyError(err);
     if (opts.json) {
-        console.log(JSON.stringify({ error: envelope }));
+        console.log(jsonText({ error: envelope }));
     }
     else if (!opts.quiet) {
-        console.error(chalk.red('✗ ' + envelope.message));
+        console.error(chalk.red('✗ ' + printable(envelope.message, HUMAN_ERROR_MAX)));
     }
     return exitCode;
 }

package/dist/lib/format.d.ts

@@ -1,3 +1,8 @@
-export declare function table(rows: Record<string, unknown>[], columns: string[]): void;
+export declare const HUMAN_ERROR_MAX = 1000;
+export declare function table(rows: Record<string, unknown>[], columns: string[], opts?: {
+    styled?: readonly string[];
+}): void;
+export declare function jsonText(data: unknown, space?: number): string;
 export declare function json(data: unknown): void;
+export declare function printable(text: unknown, max?: number): string;
 export declare function formatDate(value: string | number | Date | null | undefined): string;

package/dist/lib/format.js

@@ -1,27 +1,47 @@
 import chalk from 'chalk';
 import stripAnsi from 'strip-ansi';
-export function table(rows, columns) {
+export const HUMAN_ERROR_MAX = 1000;
+export function table(rows, columns, opts = {}) {
     if (!rows.length) {
         console.log(chalk.dim('No results.'));
         return;
     }
-    const widths = columns.map((col) => Math.max(col.length, ...rows.map((r) => stripAnsi(String(r[col] ?? '')).length)));
+    const styled = new Set(opts.styled);
+    const cells = rows.map((row) => columns.map((col) => (styled.has(col) ? String(row[col] ?? '') : printable(row[col], HUMAN_ERROR_MAX))));
+    const visibleLength = (cell) => stripAnsi(cell).length;
+    const widths = columns.map((col, i) => Math.max(col.length, ...cells.map((row) => visibleLength(row[i]))));
     const header = columns.map((col, i) => chalk.bold(col.padEnd(widths[i]))).join('  ');
     console.log(header);
     console.log(chalk.dim('─'.repeat(header.replace(/\x1b\[[0-9;]*m/g, '').length)));
-    for (const row of rows) {
-        const line = columns
-            .map((col, i) => {
-            const raw = String(row[col] ?? '');
-            const visLen = stripAnsi(raw).length;
-            return raw + ' '.repeat(Math.max(0, widths[i] - visLen));
-        })
+    for (const row of cells) {
+        const line = row
+            .map((cell, i) => cell + ' '.repeat(Math.max(0, widths[i] - visibleLength(cell))))
             .join('  ');
         console.log(line);
     }
 }
+const RAW_CONTROL_CHARS = /[\u007f-\u009f]/g;
+export function jsonText(data, space) {
+    const text = JSON.stringify(data, null, space);
+    if (typeof text !== 'string')
+        return text;
+    return text.replace(RAW_CONTROL_CHARS, (c) => `\\u${c.charCodeAt(0).toString(16).padStart(4, '0')}`);
+}
 export function json(data) {
-    console.log(JSON.stringify(data, null, 2));
+    console.log(jsonText(data, 2));
+}
+const INPUT_SPAN = 8;
+export function printable(text, max = 200) {
+    const value = typeof text === 'string' ? text : text == null ? '' : String(text);
+    const clean = stripAnsi(value.slice(0, max * INPUT_SPAN))
+        .replace(/[\t\n\v\f\r\u0085]+/g, ' ')
+        .replace(/[\u0000-\u001f\u007f-\u009f]/g, '')
+        .trim();
+    if (clean.length <= max)
+        return clean;
+    const cut = clean.slice(0, Math.max(0, max - 1));
+    const last = cut.charCodeAt(cut.length - 1);
+    return `${last >= 0xd800 && last <= 0xdbff ? cut.slice(0, -1) : cut}…`;
 }
 export function formatDate(value) {
     if (value === null || value === undefined)

package/dist/lib/json.js

@@ -5,49 +5,54 @@ const NAMED_CONTROL_CHAR_ESCAPES = {
     '\b': '\\b',
     '\f': '\\f',
 };
+const CONTROL_CHAR_ESCAPES = Array.from({ length: 0x20 }, (_, code) => NAMED_CONTROL_CHAR_ESCAPES[String.fromCharCode(code)] ?? `\\u${code.toString(16).padStart(4, '0')}`);
 export function escapeRawControlCharsInJsonStrings(text) {
-    let out = '';
+    const parts = [];
+    let copied = 0;
     let inString = false;
     let escaped = false;
     for (let i = 0; i < text.length; i += 1) {
-        const ch = text[i];
         const code = text.charCodeAt(i);
         if (!inString) {
-            if (ch === '"')
+            if (code === 0x22)
                 inString = true;
-            out += ch;
             continue;
         }
         if (escaped) {
-            out += ch;
             escaped = false;
             continue;
         }
-        if (ch === '\\') {
-            out += ch;
+        if (code === 0x5c) {
             escaped = true;
             continue;
         }
-        if (ch === '"') {
+        if (code === 0x22) {
             inString = false;
-            out += ch;
             continue;
         }
         if (code <= 0x1f) {
-            out += NAMED_CONTROL_CHAR_ESCAPES[ch] ?? `\\u${code.toString(16).padStart(4, '0')}`;
-            continue;
+            if (i > copied)
+                parts.push(text.slice(copied, i));
+            parts.push(CONTROL_CHAR_ESCAPES[code]);
+            copied = i + 1;
         }
-        out += ch;
     }
-    return out;
+    if (parts.length === 0)
+        return text;
+    if (copied < text.length)
+        parts.push(text.slice(copied));
+    return parts.join('');
 }
 export function parseServerJson(text) {
     try {
         return JSON.parse(text);
     }
     catch (firstErr) {
+        const repaired = escapeRawControlCharsInJsonStrings(text);
+        if (repaired === text)
+            throw firstErr;
         try {
-            return JSON.parse(escapeRawControlCharsInJsonStrings(text));
+            return JSON.parse(repaired);
         }
         catch {
             throw firstErr;

package/dist/lib/pinchAnimation.d.ts

@@ -1,4 +1,5 @@
 export interface PinchAnimation {
     stop(): void;
+    setStatus(text: string): void;
 }
-export declare function startPinchAnimation(statusText: string): PinchAnimation;
+export declare function startPinchAnimation(initialStatus: string): PinchAnimation;

package/dist/lib/pinchAnimation.js

@@ -1,7 +1,9 @@
 import chalk from 'chalk';
+import { printable } from './format.js';
 import { renderPinchArt } from './pinch.js';
 const FRAME_MS = 600;
-export function startPinchAnimation(statusText) {
+export function startPinchAnimation(initialStatus) {
+    let statusText = initialStatus;
     const frames = [
         renderPinchArt('working', { compact: true, frame: 0 }),
         renderPinchArt('working', { compact: true, frame: 1 }),
@@ -30,6 +32,9 @@ export function startPinchAnimation(statusText) {
         timer.unref();
     let stopped = false;
     return {
+        setStatus(text) {
+            statusText = printable(text);
+        },
         stop() {
             if (stopped)
                 return;

package/docs/agent-quickstart.md

@@ -103,30 +103,40 @@ succeed" — a failed first run is still a 200 response (the scrap was still
 created), and the CLI exits `1` in that case even though `--json` always
 prints the raw payload verbatim. The call legitimately runs many minutes
 server-side (AI generation + a proxy-tier probe ladder + an iteration loop +
-a real run) — the CLI arms a 1500s ceiling for this endpoint alone, above the
-300s `run`/`data --fresh`/`trigger --wait` use and well above the generic 30s
-default. That one call also uses a transport of its own: Node's global
-`fetch` cannot wait longer than ~300s no matter what ceiling is asked for.
-
-> **Not idempotent, and not a one-off run.** A client-side transport failure
-> (exit `5`, a `NetworkError`) — a timeout, or a connection dropped after the
-> API already answered — does not mean the wizard failed server-side: scrap
+a real run), so the CLI reads the wizard's NDJSON progress stream: human mode
+prints one short line per stage on stderr, `--json` prints only the final
+payload (same shape as always). It arms a 3600s safety ceiling for this
+endpoint alone, above the 660s `run`/`data --fresh`/`trigger --wait` use and
+well above the generic 30s default. Both use a transport of their own: Node's
+global `fetch` cannot wait longer than ~300s no matter what ceiling is asked
+for.
+
+> **Not idempotent, and not a one-off run.** A transport failure — a
+> client-side timeout, a progress stream cut or ended without a result, a
+> connection dropped after the API already answered (exit `5`, a
+> `NetworkError`), or a gateway `502`/`504` — does not mean the wizard failed server-side: scrap
 > creation + the first run keep going after the CLI gives up waiting, so the
 > scrap may already exist. ⚠ The stderr warning about this is HUMAN-mode
 > only: under `--json` you get exit `5` and nothing else — no warning line,
-> no extra envelope field — so treat ANY exit `5` from `create` as "maybe
-> created". Check `trawl list --json` for a matching URL/title
+> no extra envelope field — so treat ANY exit `5` from `create`, and an
+> exit `1` with `status` 502/504, as "maybe created". Check `trawl list --json` for a matching URL/title
 > **before** retrying — a blind retry creates a duplicate scrap and burns
 > quota a second time for the same goal. Separately, the created scrap is
-> scheduled to re-run every day at 07:00 UTC by default
-> (`cron: "0 7 * * *"`, unrelated to `--no-autofix`) — each recurring run
+> scheduled to re-run every week, Monday at 07:00 UTC, by default
+> (`cron: "0 7 * * 1"`, unrelated to `--no-autofix`) — each recurring run
 > consumes execute quota. Disable or change it once you've reviewed the
 > scrap: `trawl scraps update <id> --no-cron` (or `--cron <expr>`). Finally,
-> if `TRAWL_TIMEOUT` is set globally for a tighter budget than 1500s, it
+> if `TRAWL_TIMEOUT` is set globally for a tighter budget than 3600s, it
 > clamps `create`'s ceiling too (env always wins) — so a value picked for a
 > quick call, e.g. 600s, aborts `create` mid-wizard. Unset it or raise it
 > before calling `create`.
 
+> **A failure the wizard reports itself** — the stream's terminal `error`
+> event — arrives after the HTTP `200` was already sent, so it has no
+> `status`: `create` exits `1` with
+> `{"error":{"message":"…","kind":"refused","retryable":false}}` (no `status`
+> field). Branch on `kind`, not on a status.
+
 > **No breaking change:** every verb above is also still reachable under its
 > pre-reorg path, `trawl scraps <verb>` (e.g. `trawl scraps list`) — kept as
 > a hidden alias. Prefer the bare top-level form above; it's what

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@trawlme/cli",
-  "version": "3.12.5",
+  "version": "3.12.6",
   "description": "Trawl CLI — manage scraps from the terminal",
   "type": "module",
   "bin": {