@trawlme/cli 3.12.3 → 3.12.5

package/README.md

@@ -68,16 +68,17 @@ trawl spec [--json]                            Print a versioned, machine-readab
 
 > **No breaking change:** every verb above is also still reachable under its pre-reorg path, `trawl scraps <verb>` (e.g. `trawl scraps list`, `trawl scraps run <id>`) — kept as a hidden alias so scripts written before the surface reorg keep working. `trawl --help` only shows the top-level form above; `trawl scraps --help` only shows the remaining scrap-management commands below.
 
-`create`/`whoami`/`ping` are fully non-interactive — all three read auth only from `TRAWL_API_KEY`/`TRAWL_TOKEN`/the stored login token, never prompt (`whoami` is JWT-only — see [Authentication](#authentication) — so it still refuses honestly under a key instead of prompting). `trawl create` runs the AI wizard server-side (`POST /api/ai/wizard`): generate scrap code from `--prompt` via LLM, persist the scrap, trigger its FIRST run, and auto-fix on failure (default on — `--no-autofix` disables it, sending `autoFix:false`). On a successful first run (human mode) it prints a small **data sample** (item count + first-item fields + one truncated value) as proof of value — best-effort, silent if the sample can't be fetched — and points `Next step` at `trawl data <id>` (the data), with `trawl get <id>` as the secondary detail view. `--json` skips the sample fetch and prints the raw wizard payload verbatim. `success` is an honest outcome of that first run, not "did the HTTP call succeed" — a failed first run is still a 200 response (the scrap was still created; auto-fix, when enabled, retries in the background), and the CLI exits 1 in that case (both human and `--json` modes) even though `--json` always prints the raw payload verbatim. The call legitimately takes 30–250s+ server-side (AI generation + a real run), same long-run timeout as `run`/`data --fresh`/`trigger --wait` below. `trawl whoami`/`trawl ping` mirror the MCP `trawl_whoami`/`trawl_health_ping` tools as closely as the REST surface allows (`GET /api/users/me` / `GET /api/health`) — `ping`'s `--json` payload is admin-enriched (version/uptime/db) and just `{"status":"ok"}` for anyone else.
+`create`/`whoami`/`ping` are fully non-interactive — all three read auth only from `TRAWL_API_KEY`/`TRAWL_TOKEN`/the stored login token, never prompt (`whoami` is JWT-only — see [Authentication](#authentication) — so it still refuses honestly under a key instead of prompting). `trawl create` runs the AI wizard server-side (`POST /api/ai/wizard`): generate scrap code from `--prompt` via LLM, persist the scrap, trigger its FIRST run, and auto-fix on failure (default on — `--no-autofix` disables it, sending `autoFix:false`). On a successful first run (human mode) it prints a small **data sample** (item count + first-item fields + one truncated value) as proof of value — best-effort, silent if the sample can't be fetched — and points `Next step` at `trawl data <id>` (the data), with `trawl get <id>` as the secondary detail view. `--json` skips the sample fetch and prints the raw wizard payload verbatim. `success` is an honest outcome of that first run, not "did the HTTP call succeed" — a failed first run is still a 200 response (the scrap was still created; auto-fix, when enabled, retries in the background), and the CLI exits 1 in that case (both human and `--json` modes) even though `--json` always prints the raw payload verbatim. The call legitimately takes many minutes server-side (AI generation + a proxy-tier probe ladder + an iteration loop + a real run), so it arms a 1500s ceiling of its own — longer than the 300s `run`/`data --fresh`/`trigger --wait` use below, and over a transport of its own (Node's global `fetch` cannot wait past ~300s whatever ceiling is asked for). `trawl whoami`/`trawl ping` mirror the MCP `trawl_whoami`/`trawl_health_ping` tools as closely as the REST surface allows (`GET /api/users/me` / `GET /api/health`) — `ping`'s `--json` payload is admin-enriched (version/uptime/db) and just `{"status":"ok"}` for anyone else.
 
-> **`create` is NOT idempotent, and every wizard-created scrap runs on a DAILY cron by default.** A client-side timeout (exit `5`, a `NetworkError`) does not mean the wizard failed server-side — scrap creation + the first run keep going after the CLI gives up waiting, so the scrap may already exist. Run `trawl list` and look for a matching URL/title **before** retrying — a blind retry creates a DUPLICATE scrap and burns AI-generation quota a second time for the same goal. Separately, the scrap the wizard creates is scheduled to re-run every day at 07:00 UTC (`cron: "0 7 * * *"`, hardcoded server-side, unrelated to `--no-autofix`) — each of those recurring runs consumes execute quota like any other run. Review the generated scrap, then change or disable the schedule with `trawl scraps update <id> --cron <expr>` (or `--no-cron` to disable it). Because the call can legitimately run 250s+, also confirm `TRAWL_TIMEOUT` isn't set to something tighter than `create` needs — the env var always wins over `create`'s own 300s default (see [Environment variables](#environment-variables)), so a value set for another purpose (e.g. a tight CI smoke-test budget) silently clamps `create` too; unset it or raise it before running `create`.
+> **`create` is NOT idempotent, and every wizard-created scrap runs on a DAILY cron by default.** A client-side transport failure (exit `5`, a `NetworkError`) — a timeout, or a connection dropped after the API already answered — does not mean the wizard failed server-side: scrap creation + the first run keep going after the CLI gives up waiting, so the scrap may already exist. In human mode the CLI prints a stderr warning whenever the request reached the API without returning a complete answer; under `--json` there is no warning and no extra envelope field — a machine caller sees exit `5` and must check `trawl list` itself. Run `trawl list` and look for a matching URL/title **before** retrying — a blind retry creates a DUPLICATE scrap and burns AI-generation quota a second time for the same goal. Separately, the scrap the wizard creates is scheduled to re-run every day at 07:00 UTC (`cron: "0 7 * * *"`, hardcoded server-side, unrelated to `--no-autofix`) — each of those recurring runs consumes execute quota like any other run. Review the generated scrap, then change or disable the schedule with `trawl scraps update <id> --cron <expr>` (or `--no-cron` to disable it). Because the call can legitimately run many minutes, also confirm `TRAWL_TIMEOUT` isn't set to something tighter than `create` needs — the env var always wins over `create`'s own 1500s default (see [Environment variables](#environment-variables)), so a value set for another purpose (e.g. a tight CI smoke-test budget) silently clamps `create` too; unset it or raise it before running `create`.
 
 - `list` has a short alias, `ls` (matches `trawl --help`'s `list|ls`).
 - `history` lists past runs (newest first); `run-info <hid>` shows details of a single run from that history.
 - `data` returns the last persisted run payload (no execute quota); `--fresh` runs the scrap live instead (consumes execute quota); `--errors` shows the last run's error detail (`--json` on a never-run scrap returns `{"status":"no_runs"}`, exit 0, matching `scraps doctor --json`). `[]` on stdout means a genuine zero-item successful run — a scrap that has never run, whose last run failed, or whose payload aged out of retention returns a `--json` error envelope (exit 4/1/4 respectively) instead. Two more honest states: a run still **in flight** (`status: null` server-side) returns a `kind:"in_progress"` error envelope (exit 1, "retry shortly" — never suggests `--fresh`, which would just 429 against the run already holding the lock); a run whose item count **regressed** vs baseline (`statusDetail: "regression"`) still returns the real, non-empty items on stdout (exit 0) plus a stderr warning pointing at `scraps doctor <id>` — the data itself is genuine even though the run is flagged.
 - `get` (and anything reading through it, like `data`'s default path) embeds only the newest 100 history rows on the returned scrap object — `run-info` and `scraps doctor` fetch a single run directly and are unaffected by that cap. `list`/`get` show a distinct amber `▼` "regression" badge, never the red `✗` a genuine failure gets (matches `scraps doctor`'s own badge).
 - `list`/`get` also show a **health** badge next to the status badge. By default it reads the scrap's own `lastCronOutcome` field (always present, no extra cost): `⚠ cron paused` when the last scheduled tick was skipped for being unhealthy, `—` otherwise — a breadcrumb of the last tick, not a live read (never set on a no-cron scrap, can lag by one tick). `list --unhealthy` asks the API to filter to scraps whose *current* consecutive-failure streak is 3+ (the server's own threshold) and annotates each with the exact `consecutiveFailedRuns`/`unhealthySince` — shown instead of the badge whenever present, and never re-derived client-side from history. `--json` carries whichever fields the request produced: `lastCronOutcome` always, `consecutiveFailedRuns`/`unhealthySince` only when `--unhealthy` was passed. **Old-server safety:** an older server version that predates this filter silently ignores the unknown `--unhealthy` query param and returns everything, unfiltered — the CLI detects that (no returned item carries `consecutiveFailedRuns`) and prints a stderr warning instead of presenting the full list as "your unhealthy scraps".
-- **Long-running calls (`create`, `run`, `data --fresh`, `trigger --wait`):** these hit server-side paths that can legitimately take 30–250s+ (AI generation + scrap creation + a first run for `create`; proxy tier escalation + AI-fix retries for the other three) — the CLI arms a 300s timeout for exactly these four call sites instead of the generic 30s default. `TRAWL_TIMEOUT` (see below) still overrides ALL requests, including these — set it if you need a tighter or looser ceiling than 300s, but note a global override that tight also clamps `create`.
+- **Long-running calls (`run`, `data --fresh`, `trigger --wait`):** these hit server-side paths that can legitimately take 30–250s+ (proxy tier escalation + AI-fix retries) — the CLI arms a 300s timeout for exactly these three call sites instead of the generic 30s default.
+- **`create` is longer still:** the wizard adds a proxy-tier probe ladder and an iteration loop on top of generation + creation + a first run, so it arms **1500s** for that one call site (600s probe ladder + 480s iteration loop + 250s worst-case first run + margin). Raising the 300s above to cover it would make the three calls above wait 25 minutes on an unreachable API, so the two ceilings are deliberately separate. `create` is also the one call that does not go through Node's global `fetch`, which caps any wait for response headers at ~300s regardless of the ceiling asked for. `TRAWL_TIMEOUT` (see below) still overrides ALL requests, including both — set it if you need a tighter or looser ceiling, but note a global override sized for a quick call also clamps `create`.
 - **`--watch` is poll-based, not a live stream:** the activities SSE endpoint has no backlog and, for the default async `trigger` (no `--wait`), runs in a separate cron-consumer pod whose events never reach the API pod holding the SSE connection — a naive "await the run, then open SSE" shows nothing. `run --watch` and `trigger --watch` instead poll `GET /api/scraps/:id` (terminal status) and the activities REST list until the run finishes, printing each new activity line as it appears. The watched run's outcome drives the exit code too, in BOTH human and `--json` mode: a genuinely failed terminal run, a poll timeout, or a persistently unreachable API all exit non-zero — a clean successful run is the only exit `0`. A run that never reaches a terminal status within 300s prints an honest timeout notice pointing at `scraps doctor <id>` (human mode) — see the `--json` shape below. `scraps watch <id>` (the standalone command, no trigger) is unchanged — it still opens the live SSE stream directly.
 - `spec --json` prints the CLI's own command tree — `{specVersion, cliVersion, commands[], exitCodes, errorKinds, kindExitCodes, docsUrl?, llmsUrl?}` — DERIVED at runtime by walking the live commander tree (never a hand-maintained file, which would silently drift from reality). Each entry in `commands[]` carries its full path (e.g. `"scraps account session set"`), description, `hidden` (the legacy `scraps <verb>` aliases above), `leaf` (false for a pure namespace/group node like `scraps`/`skills`/`telemetry` — invoking one directly is a guaranteed-failing tool, not a real command), `aliases`, `arguments`, `options`, and an optional per-command `docs` deep link (e.g. every `scraps account *` command points at the account-sessions guide); `exitCodes`/`errorKinds`/`kindExitCodes` are read from the exact same source `classifyError` uses (see [Exit codes](#exit-codes)) — never a second copy. `kindExitCodes` is the inverse of `exitCodes`: `kind -> exitCode`, since exit code `1` alone is a shared bucket (`api`/`refused`/`unknown`/`in_progress`/`run_failed`/`upgrade_failed`) that `exitCodes`' flat label can't disambiguate. `docsUrl` is resolved server-first (`externalDocs.url` on the configured API base's own OpenAPI document, when declared), else derived from a KNOWN first-party `trawl.me` API host, else **omitted entirely** — never a guessed URL pointed at the wrong docs host for a self-hosted install. `llmsUrl` and every per-command `docs` link are narrower: they only ever come from that same known-host derivation, NEVER from a server-declared `docsUrl` — this CLI's own guide slugs have no reason to exist on a third party's own docs root, so a self-hosted server that declares `externalDocs.url` gets a correct top-level `docsUrl` but no `llmsUrl` and no per-command `docs` links. A `failureKind:"auth"` run's JSON payload (`doctor`/`data --errors`/`run-info`) carries the same `docs` field, by the same known-host-only rule. Without `--json`, `spec` prints one short human line (version + visible command count) pointing at `--json`.
 - `run --json`/`trigger --json` bypass the spinner and print the raw launch/trigger payload on stdout; combined with `--watch`, every intermediate progress line stays suppressed (stdout stays pure JSON) and, once the watch reaches its outcome, exactly ONE final NDJSON line is emitted: `{"runId","status"}` (the honest terminal status — `success`/`error`/`empty`/`regression`/…), `{"runId","status":"timeout"}` on a poll timeout, or `{"runId","status":"poll_error","error"}` if the API stays unreachable for several consecutive polls — `process.exitCode` is non-zero for all three except a genuine success. `scraps watch --json` emits one raw JSON object per activity line (NDJSON) instead of the formatted `[time] message` text — there's no single final payload to wait for on a live stream.
@@ -237,7 +238,7 @@ Under `--json`, a failing command emits a single error envelope on stdout — `{
 | `TRAWL_TELEMETRY`     | Set to `0` to disable telemetry for the current session           |
 | `DO_NOT_TRACK`        | Set to `1` to disable telemetry (cross-vendor convention, https://consoledonottrack.com) — same effect as `TRAWL_TELEMETRY=0` |
 | `TRAWL_CONFIG_DIR`    | Override where the config file (token, API URL, telemetry state) is stored — useful for hermetic CI runs or concurrent `trawl login`s that must not share one on-disk file |
-| `TRAWL_TIMEOUT`       | Override the per-request fetch timeout in milliseconds (default `30000`; `300000` for `create`/`run`/`data --fresh`/`trigger --wait` — this env var always wins over that longer default too) |
+| `TRAWL_TIMEOUT`       | Override the per-request fetch timeout in milliseconds (default `30000`; `300000` for `run`/`data --fresh`/`trigger --wait`, `1500000` for `create` — this env var always wins over those longer defaults too) |
 | `TRAWL_SKILLS_SYNC`   | Set to `0` to disable the startup skills auto-sync entirely       |
 
 Session override (no `trawl login` mutation, ideal for CI/QA against another env):

package/dist/commands/create.js

@@ -1,7 +1,7 @@
 import { Command } from 'commander';
 import chalk from 'chalk';
 import { spin } from '../lib/spinner.js';
-import { api, LONG_RUN_TIMEOUT_MS, NetworkError } from '../lib/api.js';
+import { api, WIZARD_TIMEOUT_MS, NetworkError } from '../lib/api.js';
 import { json } from '../lib/format.js';
 import { parseServerJson } from '../lib/json.js';
 import { requireUrl, requireString } from '../lib/validate.js';
@@ -78,7 +78,7 @@ export const create = new Command('create')
         goal,
         ...(opts.autofix === false && { autoFix: false }),
     };
-    const call = () => api.post('/api/ai/wizard', body, { timeoutMs: LONG_RUN_TIMEOUT_MS });
+    const call = () => api.postLongRunning('/api/ai/wizard', body, { timeoutMs: WIZARD_TIMEOUT_MS });
     let data;
     if (opts.json) {
         data = await call();
@@ -95,8 +95,8 @@ export const create = new Command('create')
         }
         catch (err) {
             anim?.stop();
-            if (err instanceof NetworkError && /timed out/i.test(err.message)) {
-                console.error(chalk.yellow('⚠ The request timed out client-side, but the scrap may STILL have been created server-side — run `trawl list` before retrying (a retry creates a DUPLICATE scrap + burns quota).'));
+            if (err instanceof NetworkError && err.mayHaveCompleted) {
+                console.error(chalk.yellow('⚠ The request did not complete client-side, but the scrap may STILL have been created server-side — run `trawl list` before retrying (a retry creates a DUPLICATE scrap + burns quota).'));
             }
             throw err;
         }

package/dist/lib/api.d.ts

@@ -4,7 +4,8 @@ export declare class ApiError extends Error {
     constructor(status: number, message: string, next?: string[] | undefined);
 }
 export declare class NetworkError extends Error {
-    constructor(message: string);
+    readonly mayHaveCompleted: boolean;
+    constructor(message: string, mayHaveCompleted?: boolean);
 }
 export declare class AuthError extends Error {
     next?: string[] | undefined;
@@ -13,9 +14,12 @@ export declare class AuthError extends Error {
 export declare function notLoggedInError(): AuthError;
 export declare function apiKeyUnsupportedError(command: string): AuthError;
 export declare const LONG_RUN_TIMEOUT_MS = 300000;
+export declare const WIZARD_TIMEOUT_MS = 1500000;
 export interface RequestOptions {
     timeoutMs?: number;
 }
+export declare function effectivePort(u: URL): string;
+export declare function sameEffectivePort(from: URL, next: URL): boolean;
 export declare const api: {
     get: <T>(path: string, opts?: RequestOptions) => Promise<T>;
     publicGet: <T>(path: string, opts?: RequestOptions) => Promise<T>;
@@ -24,6 +28,7 @@ export declare const api: {
     }) => Promise<T | null>;
     getText: (path: string, opts?: RequestOptions) => Promise<string>;
     post: <T>(path: string, body?: unknown, opts?: RequestOptions) => Promise<T>;
+    postLongRunning: <T>(path: string, body?: unknown, opts?: RequestOptions) => Promise<T>;
     put: <T>(path: string, body?: unknown, opts?: RequestOptions) => Promise<T>;
     delete: <T>(path: string, opts?: RequestOptions) => Promise<T>;
     upload: <T>(path: string, formData: FormData, opts?: RequestOptions) => Promise<T>;

package/dist/lib/api.js

@@ -1,8 +1,9 @@
 import { readFileSync } from 'node:fs';
 import { fileURLToPath } from 'node:url';
 import { dirname, resolve } from 'node:path';
-import { request as httpRequest } from 'node:http';
+import { request as httpRequest, } from 'node:http';
 import { request as httpsRequest } from 'node:https';
+import { brotliDecompressSync, gunzipSync, inflateRawSync, inflateSync } from 'node:zlib';
 import { getApiUrl, getToken, getAuthMode, getLiveAuthEnvVar } from './config.js';
 import { parseServerJson } from './json.js';
 const __dirname = dirname(fileURLToPath(import.meta.url));
@@ -19,8 +20,10 @@ export class ApiError extends Error {
     }
 }
 export class NetworkError extends Error {
-    constructor(message) {
+    mayHaveCompleted;
+    constructor(message, mayHaveCompleted = false) {
         super(message);
+        this.mayHaveCompleted = mayHaveCompleted;
         this.name = 'NetworkError';
     }
 }
@@ -62,6 +65,7 @@ function authHeaders(token) {
 }
 const DEFAULT_TIMEOUT_MS = 30_000;
 export const LONG_RUN_TIMEOUT_MS = 300_000;
+export const WIZARD_TIMEOUT_MS = 1_500_000;
 function getTimeoutMs(overrideMs) {
     const raw = process.env['TRAWL_TIMEOUT']?.trim();
     if (raw) {
@@ -71,6 +75,16 @@ function getTimeoutMs(overrideMs) {
     }
     return overrideMs !== undefined && overrideMs > 0 ? overrideMs : DEFAULT_TIMEOUT_MS;
 }
+function timeoutNetworkError(url, effectiveTimeoutMs) {
+    return new NetworkError(`Request to ${url} timed out after ${effectiveTimeoutMs}ms (override with TRAWL_TIMEOUT env var, ms)`, true);
+}
+function connectionNetworkError(url, err, mayHaveCompleted = false) {
+    const e = err;
+    const cause = e?.cause;
+    const code = e?.code ?? cause?.code;
+    const detail = code ? ` (${code})` : cause?.message ? ` (${cause.message})` : '';
+    return new NetworkError(`Network error reaching ${url}${detail}: ${e?.message ?? String(err)}`, mayHaveCompleted);
+}
 async function safeFetch(url, options, effectiveTimeoutMs) {
     try {
         return await fetch(url, options);
@@ -78,11 +92,9 @@ async function safeFetch(url, options, effectiveTimeoutMs) {
     catch (err) {
         const e = err;
         if (e?.name === 'TimeoutError' || e?.name === 'AbortError') {
-            throw new NetworkError(`Request to ${url} timed out after ${effectiveTimeoutMs}ms (override with TRAWL_TIMEOUT env var, ms)`);
+            throw timeoutNetworkError(url, effectiveTimeoutMs);
         }
-        const cause = e?.cause;
-        const causeDetail = cause?.code ? ` (${cause.code})` : cause?.message ? ` (${cause.message})` : '';
-        throw new NetworkError(`Network error reaching ${url}${causeDetail}: ${e?.message ?? String(err)}`);
+        throw connectionNetworkError(url, err);
     }
 }
 function extractErrorMessage(raw, statusText) {
@@ -331,6 +343,233 @@ function probeJson(path, opts) {
         req.end();
     });
 }
+function toHeaders(raw) {
+    const headers = new Headers();
+    for (const [key, value] of Object.entries(raw)) {
+        try {
+            if (Array.isArray(value))
+                for (const one of value)
+                    headers.append(key, one);
+            else if (value !== undefined)
+                headers.append(key, value);
+        }
+        catch {
+        }
+    }
+    return headers;
+}
+const NULL_BODY_STATUS = new Set([204, 205, 304]);
+function toResponse({ status, statusText, headers, text }) {
+    const body = NULL_BODY_STATUS.has(status) ? null : text;
+    const safeStatus = status >= 200 && status <= 599 ? status : 502;
+    try {
+        return new Response(body, { status: safeStatus, statusText, headers });
+    }
+    catch {
+        return new Response(body, { status: safeStatus, headers });
+    }
+}
+const MAX_REDIRECTS = 3;
+const REDIRECT_KEEPS_METHOD = new Set([307, 308]);
+const REDIRECT_DROPS_BODY = new Set([301, 302, 303]);
+const MAX_DECODED_BYTES = 64 * 1024 * 1024;
+const DECODE_LIMIT = { maxOutputLength: MAX_DECODED_BYTES };
+function isOutputOverflow(err) {
+    return err?.code === 'ERR_BUFFER_TOO_LARGE';
+}
+function inflateEitherForm(body) {
+    try {
+        return inflateSync(body, DECODE_LIMIT);
+    }
+    catch (err) {
+        if (isOutputOverflow(err))
+            throw err;
+        return inflateRawSync(body, DECODE_LIMIT);
+    }
+}
+const CONTENT_DECODERS = Object.freeze({
+    gzip: (body) => gunzipSync(body, DECODE_LIMIT),
+    'x-gzip': (body) => gunzipSync(body, DECODE_LIMIT),
+    deflate: inflateEitherForm,
+    br: (body) => brotliDecompressSync(body, DECODE_LIMIT),
+});
+function decodeBody(body, contentEncoding) {
+    const encoding = (contentEncoding ?? '').trim().toLowerCase();
+    if (encoding === '' || encoding === 'identity')
+        return body;
+    const decoder = CONTENT_DECODERS[encoding];
+    if (!decoder)
+        return { reason: `unsupported Content-Encoding "${encoding}"` };
+    try {
+        return decoder(body);
+    }
+    catch (err) {
+        if (isOutputOverflow(err)) {
+            return {
+                reason: `Content-Encoding "${encoding}" expanded past the ${MAX_DECODED_BYTES / (1024 * 1024)} MB decode ceiling`,
+            };
+        }
+        return { reason: `Content-Encoding "${encoding}" failed to decode (${err.message})` };
+    }
+}
+function postRaw(url, payload, headers, timeoutMs) {
+    return new Promise((resolve, reject) => {
+        let target;
+        try {
+            target = new URL(url);
+        }
+        catch {
+            reject(connectionNetworkError(url, Object.assign(new Error('invalid API base URL'), { code: 'ERR_INVALID_URL' })));
+            return;
+        }
+        if (target.protocol !== 'http:' && target.protocol !== 'https:') {
+            reject(connectionNetworkError(url, Object.assign(new Error(`unsupported protocol "${target.protocol}"`), { code: 'ERR_INVALID_PROTOCOL' })));
+            return;
+        }
+        let settled = false;
+        let timedOut = false;
+        let mayHaveCompleted = false;
+        let timer;
+        let active;
+        const settleWith = (err) => {
+            if (settled)
+                return;
+            settled = true;
+            clearTimeout(timer);
+            active?.destroy();
+            reject(err);
+        };
+        const fail = (err) => {
+            settleWith(timedOut ? timeoutNetworkError(url, timeoutMs) : connectionNetworkError(url, err, mayHaveCompleted));
+        };
+        const done = (value) => {
+            if (settled)
+                return;
+            settled = true;
+            clearTimeout(timer);
+            resolve(value);
+        };
+        const onResponse = (res, from, hopsLeft, failThisHop) => {
+            const status = res.statusCode ?? 0;
+            const statusText = res.statusMessage ?? '';
+            res.on('error', failThisHop);
+            if (REDIRECT_KEEPS_METHOD.has(status)) {
+                res.resume();
+                const location = res.headers.location;
+                const next = location ? safeUrl(location, from) : null;
+                const refusal = !next
+                    ? 'it carries no usable Location header'
+                    : next.protocol !== 'http:' && next.protocol !== 'https:'
+                        ? `its target uses an unsupported protocol ("${next.protocol}")`
+                        : next.hostname !== from.hostname
+                            ? `its target is a different host (${next.hostname}) — this request carries your credential`
+                            : !sameEffectivePort(from, next)
+                                ? `its target is a different port (${effectivePort(next)}) — this request carries your credential`
+                                : from.protocol === 'https:' && next.protocol === 'http:'
+                                    ? 'it downgrades https to http — this request carries your credential'
+                                    : hopsLeft <= 0
+                                        ? `it exceeds ${MAX_REDIRECTS} redirects`
+                                        : null;
+                if (refusal || !next) {
+                    settleWith(new ApiError(status, `${status} ${statusText}: refused to follow this redirect because ${refusal}. Point TRAWL_API_URL at the final URL.`));
+                    return;
+                }
+                send(next, hopsLeft - 1);
+                return;
+            }
+            if (REDIRECT_DROPS_BODY.has(status)) {
+                res.resume();
+                settleWith(new ApiError(status, `${status} ${statusText}: this endpoint redirected the POST to ${res.headers.location ?? 'an unspecified location'}, and that status rewrites it to a bodyless GET — only 307/308 are followed. Point TRAWL_API_URL at the final URL.`));
+                return;
+            }
+            mayHaveCompleted = status >= 200 && status < 300;
+            const chunks = [];
+            res.on('data', (chunk) => chunks.push(chunk));
+            res.on('end', () => {
+                const decoded = decodeBody(Buffer.concat(chunks), res.headers['content-encoding']);
+                if (!Buffer.isBuffer(decoded)) {
+                    settleWith(new NetworkError(`Unreadable response from ${url} (HTTP ${status}): ${decoded.reason} — the server ignored "Accept-Encoding: identity"`, mayHaveCompleted));
+                    return;
+                }
+                done({ status, statusText, headers: toHeaders(res.headers), text: decoded.toString('utf8') });
+            });
+            res.on('close', () => {
+                if (!res.complete)
+                    failThisHop(new Error('response aborted before it finished'));
+            });
+        };
+        const send = (to, hopsLeft) => {
+            mayHaveCompleted = false;
+            const transport = to.protocol === 'http:' ? httpRequest : httpsRequest;
+            let req;
+            const failThisHop = (err) => {
+                if (active !== req)
+                    return;
+                fail(err);
+            };
+            try {
+                req = transport(to, { method: 'POST', headers: { 'Accept-Encoding': 'identity', ...headers } }, (res) => onResponse(res, to, hopsLeft, failThisHop));
+            }
+            catch (err) {
+                fail(err);
+                return;
+            }
+            active = req;
+            req.on('error', failThisHop);
+            if (payload !== undefined)
+                req.write(payload);
+            req.end();
+        };
+        timer = setTimeout(() => {
+            timedOut = true;
+            active?.destroy(new Error(`request to ${url} timed out after ${timeoutMs}ms`));
+        }, timeoutMs);
+        send(target, MAX_REDIRECTS);
+    });
+}
+export function effectivePort(u) {
+    return u.port !== '' ? u.port : u.protocol === 'https:' ? '443' : '80';
+}
+export function sameEffectivePort(from, next) {
+    if (from.port === '' && next.port === '')
+        return true;
+    return effectivePort(from) === effectivePort(next);
+}
+function safeUrl(raw, base) {
+    try {
+        return new URL(raw, base);
+    }
+    catch {
+        return null;
+    }
+}
+async function postLongRunning(path, body, reqOpts = {}) {
+    const token = getToken();
+    if (!token)
+        throw notLoggedInError();
+    const url = `${getApiUrl()}${path}`;
+    const timeoutMs = getTimeoutMs(reqOpts.timeoutMs);
+    const payload = body ? JSON.stringify(body) : undefined;
+    const raw = await postRaw(url, payload, {
+        'Content-Type': 'application/json',
+        'User-Agent': USER_AGENT,
+        ...(payload !== undefined ? { 'Content-Length': String(Buffer.byteLength(payload)) } : {}),
+        ...authHeaders(token),
+    }, timeoutMs);
+    await throwIfError(toResponse(raw), false, getAuthMode(token));
+    try {
+        if (!raw.text)
+            return {};
+        const parsed = parseServerJson(raw.text);
+        if (parsed !== null && typeof parsed === 'object' && 'data' in parsed) {
+            return parsed.data;
+        }
+        return parsed;
+    }
+    catch {
+        throw new NetworkError('Invalid JSON in server response', true);
+    }
+}
 async function getText(path, reqOpts = {}) {
     const token = getToken();
     if (!token)
@@ -356,6 +595,7 @@ export const api = {
         method: 'POST',
         body: body ? JSON.stringify(body) : undefined,
     }, opts),
+    postLongRunning: (path, body, opts) => postLongRunning(path, body, opts),
     put: (path, body, opts) => request(path, {
         method: 'PUT',
         body: body ? JSON.stringify(body) : undefined,

package/dist/lib/config.js

@@ -1,8 +1,10 @@
+import { chmodSync, statSync } from 'node:fs';
 import Conf from 'conf';
 const configDir = process.env['TRAWL_CONFIG_DIR']?.trim();
 const config = new Conf({
     projectName: 'trawl-cli',
     ...(configDir ? { cwd: configDir } : {}),
+    configFileMode: 0o600,
     defaults: {
         apiUrl: 'https://api.trawl.me',
         token: '',
@@ -12,6 +14,16 @@ const config = new Conf({
         skillsNudgeShownAt: 0,
     },
 });
+if (process.platform !== 'win32') {
+    try {
+        const stat = statSync(config.path, { throwIfNoEntry: false });
+        if (stat && (stat.mode & 0o077) !== 0) {
+            chmodSync(config.path, 0o600);
+        }
+    }
+    catch {
+    }
+}
 export function getApiUrl() {
     const override = process.env['TRAWL_API_URL']?.trim();
     return override ? override : config.get('apiUrl');

package/docs/agent-quickstart.md

@@ -101,23 +101,30 @@ first run, and auto-fix on failure (default on; `--no-autofix` disables it).
 `success` is an honest outcome of that first run, not "did the HTTP call
 succeed" — a failed first run is still a 200 response (the scrap was still
 created), and the CLI exits `1` in that case even though `--json` always
-prints the raw payload verbatim. The call legitimately runs 30–250s+
-server-side (AI generation + a real run) — the CLI arms the same long-run
-timeout `run`/`data --fresh`/`trigger --wait` use instead of the generic 30s
-default.
-
-> **Not idempotent, and not a one-off run.** A client-side timeout (exit `5`,
-> a `NetworkError`) does not mean the wizard failed server-side — scrap
+prints the raw payload verbatim. The call legitimately runs many minutes
+server-side (AI generation + a proxy-tier probe ladder + an iteration loop +
+a real run) — the CLI arms a 1500s ceiling for this endpoint alone, above the
+300s `run`/`data --fresh`/`trigger --wait` use and well above the generic 30s
+default. That one call also uses a transport of its own: Node's global
+`fetch` cannot wait longer than ~300s no matter what ceiling is asked for.
+
+> **Not idempotent, and not a one-off run.** A client-side transport failure
+> (exit `5`, a `NetworkError`) — a timeout, or a connection dropped after the
+> API already answered — does not mean the wizard failed server-side: scrap
 > creation + the first run keep going after the CLI gives up waiting, so the
-> scrap may already exist. Check `trawl list --json` for a matching URL/title
+> scrap may already exist. ⚠ The stderr warning about this is HUMAN-mode
+> only: under `--json` you get exit `5` and nothing else — no warning line,
+> no extra envelope field — so treat ANY exit `5` from `create` as "maybe
+> created". Check `trawl list --json` for a matching URL/title
 > **before** retrying — a blind retry creates a duplicate scrap and burns
 > quota a second time for the same goal. Separately, the created scrap is
 > scheduled to re-run every day at 07:00 UTC by default
 > (`cron: "0 7 * * *"`, unrelated to `--no-autofix`) — each recurring run
 > consumes execute quota. Disable or change it once you've reviewed the
 > scrap: `trawl scraps update <id> --no-cron` (or `--cron <expr>`). Finally,
-> if `TRAWL_TIMEOUT` is set globally for a tighter budget than 300s, it
-> clamps `create`'s ceiling too (env always wins) — unset it or raise it
+> if `TRAWL_TIMEOUT` is set globally for a tighter budget than 1500s, it
+> clamps `create`'s ceiling too (env always wins) — so a value picked for a
+> quick call, e.g. 600s, aborts `create` mid-wizard. Unset it or raise it
 > before calling `create`.
 
 > **No breaking change:** every verb above is also still reachable under its

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@trawlme/cli",
-  "version": "3.12.3",
+  "version": "3.12.5",
   "description": "Trawl CLI — manage scraps from the terminal",
   "type": "module",
   "bin": {