@trawlme/cli 1.22.0 → 2.1.0

package/README.md

@@ -8,7 +8,7 @@ Command-line client for [Trawl](https://trawl.me) — manage your scraps from th
 
 > ⚠️ **Early stage.** APIs may change before `1.0`. Pin a version if you depend on this in CI.
 
-> 🤖 **Driving this CLI from an AI agent?** See [docs/agent-quickstart.md](docs/agent-quickstart.md) for the minimal surface (`fetch`, `--json`, `TRAWL_TOKEN`, exit codes) — this README is the full human/CI guide.
+> 🤖 **Driving this CLI from an AI agent?** See [docs/agent-quickstart.md](docs/agent-quickstart.md) for the minimal surface (`create`, `--json`, `TRAWL_TOKEN`, exit codes) — this README is the full human/CI guide.
 
 ## Install
 
@@ -37,7 +37,8 @@ All commands accept a global `--debug` flag to show full error stack traces on f
 ### Core commands (agent + human)
 
 ```
-trawl fetch <url> [--json] [--reason <text>]   One-shot fetch + extract readable content from a public URL (no scrap needed)
+trawl create <url> --prompt <goal> [--no-autofix] [--json]   # url also accepted as --url <url> (#116)
+                                                              Create a persistent, self-healing scrap from a URL + a goal (AI-generated)
 trawl run <id> [--watch] [--json]              Run a scrap
 trawl list|ls [--json] [--status <success|failure|never|running|regression>] [--limit <n>] [--page <n>]
 trawl get <id> [--json]                        Get scrap details
@@ -51,13 +52,15 @@ trawl ping [--json]                            Health/version handshake against
 
 > **No breaking change:** every verb above is also still reachable under its pre-reorg path, `trawl scraps <verb>` (e.g. `trawl scraps list`, `trawl scraps run <id>`) — kept as a hidden alias so scripts written before the surface reorg keep working. `trawl --help` only shows the top-level form above; `trawl scraps --help` only shows the remaining scrap-management commands below.
 
-`fetch`/`whoami`/`ping` are fully non-interactive — all three read auth only from `TRAWL_TOKEN`/the stored login token, never prompt. `trawl fetch` is the REST counterpart of the MCP `trawl_fetch_url` tool (same shared engine, `POST /api/scraps/fetch-url`): `status` is an honest outcome (`completed`/`failed`/`empty`/`blocked`), not "did the HTTP call succeed" — a failed fetch is still a 200 response with `status:'failed'` + `error`, and the CLI exits 1 in that case (both human and `--json` modes) even though `--json` always prints the raw payload verbatim. `truncated:true` means the 100KB response cap tripped; `result` is dropped and `url` repoints at the full history row instead. `trawl whoami`/`trawl ping` mirror the MCP `trawl_whoami`/`trawl_health_ping` tools as closely as the REST surface allows (`GET /api/users/me` / `GET /api/health`) — `ping`'s `--json` payload is admin-enriched (version/uptime/db) and just `{"status":"ok"}` for anyone else.
+`create`/`whoami`/`ping` are fully non-interactive — all three read auth only from `TRAWL_TOKEN`/the stored login token, never prompt. `trawl create` runs the AI wizard server-side (`POST /api/ai/wizard`): generate scrap code from `--prompt` via LLM, persist the scrap, trigger its FIRST run, and auto-fix on failure (default on — `--no-autofix` disables it, sending `autoFix:false`). `success` is an honest outcome of that first run, not "did the HTTP call succeed" — a failed first run is still a 200 response (the scrap was still created; auto-fix, when enabled, retries in the background), and the CLI exits 1 in that case (both human and `--json` modes) even though `--json` always prints the raw payload verbatim. The call legitimately takes 30–250s+ server-side (AI generation + a real run), same long-run timeout as `run`/`data --fresh`/`trigger --wait` below. `trawl whoami`/`trawl ping` mirror the MCP `trawl_whoami`/`trawl_health_ping` tools as closely as the REST surface allows (`GET /api/users/me` / `GET /api/health`) — `ping`'s `--json` payload is admin-enriched (version/uptime/db) and just `{"status":"ok"}` for anyone else.
+
+> **`create` is NOT idempotent, and every wizard-created scrap runs on a DAILY cron by default.** A client-side timeout (exit `5`, a `NetworkError`) does not mean the wizard failed server-side — scrap creation + the first run keep going after the CLI gives up waiting, so the scrap may already exist. Run `trawl list` and look for a matching URL/title **before** retrying — a blind retry creates a DUPLICATE scrap and burns AI-generation quota a second time for the same goal. Separately, the scrap the wizard creates is scheduled to re-run every day at 07:00 UTC (`cron: "0 7 * * *"`, hardcoded server-side, unrelated to `--no-autofix`) — each of those recurring runs consumes execute quota like any other run. Review the generated scrap, then change or disable the schedule with `trawl scraps update <id> --cron <expr>` (or `--no-cron` to disable it). Because the call can legitimately run 250s+, also confirm `TRAWL_TIMEOUT` isn't set to something tighter than `create` needs — the env var always wins over `create`'s own 300s default (see [Environment variables](#environment-variables)), so a value set for another purpose (e.g. a tight CI smoke-test budget) silently clamps `create` too; unset it or raise it before running `create`.
 
 - `list` has a short alias, `ls` (matches `trawl --help`'s `list|ls`).
 - `history` lists past runs (newest first); `run-info <hid>` shows details of a single run from that history.
 - `data` returns the last persisted run payload (no execute quota); `--fresh` runs the scrap live instead (consumes execute quota); `--errors` shows the last run's error detail (`--json` on a never-run scrap returns `{"status":"no_runs"}`, exit 0, matching `scraps doctor --json`). `[]` on stdout means a genuine zero-item successful run — a scrap that has never run, whose last run failed, or whose payload aged out of retention returns a `--json` error envelope (exit 4/1/4 respectively) instead. Two more honest states: a run still **in flight** (`status: null` server-side) returns a `kind:"in_progress"` error envelope (exit 1, "retry shortly" — never suggests `--fresh`, which would just 429 against the run already holding the lock); a run whose item count **regressed** vs baseline (`statusDetail: "regression"`) still returns the real, non-empty items on stdout (exit 0) plus a stderr warning pointing at `scraps doctor <id>` — the data itself is genuine even though the run is flagged.
 - `get` (and anything reading through it, like `data`'s default path) embeds only the newest 100 history rows on the returned scrap object — `run-info` and `scraps doctor` fetch a single run directly and are unaffected by that cap. `list`/`get` show a distinct amber `▼` "regression" badge, never the red `✗` a genuine failure gets (matches `scraps doctor`'s own badge).
-- **Long-running scrap execute (`run`, `data --fresh`, `trigger --wait`):** these hit the same server-side scrap-execute path, which can legitimately take 30–250s (proxy tier escalation, AI-fix retries) — the CLI arms a 300s timeout for exactly these three call sites instead of the generic 30s default. `TRAWL_TIMEOUT` (see below) still overrides ALL requests, including these — set it if you need a tighter or looser ceiling than 300s for a long-running scrap.
+- **Long-running calls (`create`, `run`, `data --fresh`, `trigger --wait`):** these hit server-side paths that can legitimately take 30–250s+ (AI generation + scrap creation + a first run for `create`; proxy tier escalation + AI-fix retries for the other three) — the CLI arms a 300s timeout for exactly these four call sites instead of the generic 30s default. `TRAWL_TIMEOUT` (see below) still overrides ALL requests, including these — set it if you need a tighter or looser ceiling than 300s, but note a global override that tight also clamps `create`.
 - **`--watch` is poll-based, not a live stream:** the activities SSE endpoint has no backlog and, for the default async `trigger` (no `--wait`), runs in a separate cron-consumer pod whose events never reach the API pod holding the SSE connection — a naive "await the run, then open SSE" shows nothing. `run --watch` and `trigger --watch` instead poll `GET /api/scraps/:id` (terminal status) and the activities REST list until the run finishes, printing each new activity line as it appears. The watched run's outcome drives the exit code too, in BOTH human and `--json` mode: a genuinely failed terminal run, a poll timeout, or a persistently unreachable API all exit non-zero — a clean successful run is the only exit `0`. A run that never reaches a terminal status within 300s prints an honest timeout notice pointing at `scraps doctor <id>` (human mode) — see the `--json` shape below. `scraps watch <id>` (the standalone command, no trigger) is unchanged — it still opens the live SSE stream directly.
 - `run --json`/`trigger --json` bypass the spinner and print the raw launch/trigger payload on stdout; combined with `--watch`, every intermediate progress line stays suppressed (stdout stays pure JSON) and, once the watch reaches its outcome, exactly ONE final NDJSON line is emitted: `{"runId","status"}` (the honest terminal status — `success`/`error`/`empty`/`regression`/…), `{"runId","status":"timeout"}` on a poll timeout, or `{"runId","status":"poll_error","error"}` if the API stays unreachable for several consecutive polls — `process.exitCode` is non-zero for all three except a genuine success. `scraps watch --json` emits one raw JSON object per activity line (NDJSON) instead of the formatted `[time] message` text — there's no single final payload to wait for on a live stream.
 
@@ -179,7 +182,7 @@ Every command exits with one of these codes — scripts and agents driving the C
 | Code | Meaning                                                                 |
 |------|--------------------------------------------------------------------------|
 | `0`  | Success                                                                 |
-| `1`  | Unknown/generic error (an unmapped failure — API errors other than 401/404, an unhandled bug — or a business-logic refusal like `data`'s `run_failed`/`in_progress` states, or `trawl fetch`'s honest `status:'failed'`/`'blocked'` outcome). **Known overload:** `fetch`'s domain-level failure and an arbitrary unmapped bug both land on `1` — a script needs to read the `--json` payload's own `status`/`error` field (for `fetch`) to tell them apart; this is intentional (the REST contract's outcome states are a separate axis from the CLI's transport-error taxonomy) and documented here rather than "resolved" by inventing a new code that would only apply to one command. |
+| `1`  | Unknown/generic error (an unmapped failure — API errors other than 401/404, an unhandled bug — or a business-logic refusal like `data`'s `run_failed`/`in_progress` states, or `trawl create`'s honest `success:false` first-run outcome). **Known overload:** `create`'s domain-level failure and an arbitrary unmapped bug both land on `1` — a script needs to read the `--json` payload's own `success` field (for `create`) to tell them apart; this is intentional (the REST contract's outcome states are a separate axis from the CLI's transport-error taxonomy) and documented here rather than "resolved" by inventing a new code that would only apply to one command. |
 | `2`  | Usage error (bad flag/value, invalid ID, missing required argument, unknown option/command — including the [non-interactive rule](#non-interactive-rule) refusing to prompt) |
 | `3`  | Auth error (not logged in, or the session token is expired/invalid — run `trawl login`) |
 | `4`  | Not found (no such resource, or — for `data` — no persisted payload to read) |
@@ -196,7 +199,7 @@ Under `--json`, a failing command emits a single error envelope on stdout — `{
 | `TRAWL_TELEMETRY`     | Set to `0` to disable telemetry for the current session           |
 | `DO_NOT_TRACK`        | Set to `1` to disable telemetry (cross-vendor convention, https://consoledonottrack.com) — same effect as `TRAWL_TELEMETRY=0` |
 | `TRAWL_CONFIG_DIR`    | Override where the config file (token, API URL, telemetry state) is stored — useful for hermetic CI runs or concurrent `trawl login`s that must not share one on-disk file |
-| `TRAWL_TIMEOUT`       | Override the per-request fetch timeout in milliseconds (default `30000`; `300000` for `run`/`data --fresh`/`trigger --wait` — this env var always wins over that longer default too) |
+| `TRAWL_TIMEOUT`       | Override the per-request fetch timeout in milliseconds (default `30000`; `300000` for `create`/`run`/`data --fresh`/`trigger --wait` — this env var always wins over that longer default too) |
 | `TRAWL_SKILLS_SYNC`   | Set to `0` to disable the startup skills auto-sync entirely       |
 
 Session override (no `trawl login` mutation, ideal for CI/QA against another env):

package/dist/commands/create.d.ts

@@ -0,0 +1,41 @@
+import { Command } from 'commander';
+/**
+ * `POST /api/ai/wizard` response contract (#114, trawl_node —
+ * ai.wizard.service.js#runWizard, shipped S1, contract LOCKED). The wizard
+ * chains, entirely server-side: AI code generation -> scrap creation ->
+ * the FIRST run (ScrapsService.load) -> auto-fix on failure (autoFix
+ * defaults true unless `--no-autofix` maps to `autoFix:false`).
+ *
+ *  - `success` is an HONEST outcome of that first run, not "did the HTTP
+ *    call succeed" — a failed first run is still a 200 response (the scrap
+ *    itself was created either way; auto-fix, when enabled, retries in the
+ *    background). Callers must branch on `success`, never assume 2xx means
+ *    "the scrap works".
+ *  - `scrap` is the full created Scrap object (present whenever creation got
+ *    far enough to persist it — a hard failure before that point surfaces as
+ *    a real HTTP error instead, handled by the shared error path).
+ *  - `historyId` is best-effort (a failed server-side lookup leaves it
+ *    `null`, never breaks the response) — it points at the first run just
+ *    executed.
+ */
+export interface WizardResponse {
+    success: boolean;
+    scrap?: {
+        _id: string;
+        title: string;
+        /**
+         * Cron expression the wizard schedules this scrap on. As of #114/S1
+         * (trawl_node ai.wizard.service.js#runWizard's `scrapBody`) this is
+         * hardcoded server-side to a DAILY run — `'0 7 * * *'` / `cronTimezone:
+         * 'UTC'` — unconditionally, regardless of `--prompt`/`--no-autofix`.
+         * Present on the scrap object returned here (the wizard controller
+         * passes the created scrap straight through, no stripping) — change or
+         * disable it with `trawl scraps update <id> --cron <expr>` / `--no-cron`.
+         */
+        cron?: string | null;
+        cronTimezone?: string;
+        [key: string]: unknown;
+    };
+    historyId?: string | null;
+}
+export declare const create: Command;

package/dist/commands/create.js

@@ -0,0 +1,160 @@
+import { Command } from 'commander';
+import chalk from 'chalk';
+import { oraPromise } from 'ora';
+import { api, LONG_RUN_TIMEOUT_MS, NetworkError } from '../lib/api.js';
+import { json } from '../lib/format.js';
+import { requireUrl, requireString } from '../lib/validate.js';
+import { UsageError } from '../lib/errors.js';
+/** Best-effort, honest first-run summary — never claims a background retry
+ * happened when auto-fix was disabled for this call, and never claims a
+ * scrap was persisted when the response carries none (#114-F3 — a hard
+ * failure before persistence still comes back as `success:false` with no
+ * `scrap` at all; claiming "auto-fix retrying in the background" then would
+ * be fabricated — there is nothing to retry). */
+function firstRunLabel(data, autoFixEnabled) {
+    if (data.success)
+        return 'succeeded';
+    if (!data.scrap)
+        return 'failed (no scrap was persisted)';
+    if (autoFixEnabled)
+        return 'failed (auto-fix retrying in the background)';
+    return 'failed';
+}
+/** Best-effort human description of a daily cron (`M H * * *`) — the only
+ * shape the wizard's server-side default currently produces. Falls back to
+ * printing the raw expression for anything else rather than guessing at a
+ * schedule the CLI can't actually parse. */
+function describeCron(cron) {
+    const match = /^(\d{1,2})\s+(\d{1,2})\s+\*\s+\*\s+\*$/.exec(cron.trim());
+    if (!match)
+        return cron;
+    const [, min, hour] = match;
+    return `daily ${hour.padStart(2, '0')}:${min.padStart(2, '0')}`;
+}
+/**
+ * #114-F2 — a wizard-created scrap runs on a DAILY cron by default
+ * server-side; surface that up front rather than leaving it to be
+ * discovered later as an unexpected recurring quota charge. Reads the real
+ * `cron`/`cronTimezone` field off the response scrap when present; falls
+ * back to the known wizard default wording ONLY when the field is missing
+ * from the response (an older server, or a future rename) — never invents a
+ * schedule value that might not match what the server actually applied.
+ * Returns null when there's no scrap to schedule at all.
+ */
+function scheduleLabel(scrap) {
+    if (!scrap)
+        return null;
+    if (typeof scrap.cron === 'string' && scrap.cron.trim() !== '') {
+        const tz = typeof scrap.cronTimezone === 'string' && scrap.cronTimezone ? scrap.cronTimezone : 'UTC';
+        return `${describeCron(scrap.cron)} ${tz} (cron ${scrap.cron})`;
+    }
+    return 'scheduled daily by default';
+}
+export const create = new Command('create')
+    .description('Create a persistent, self-healing scrap from a URL + a goal (AI-generated)')
+    .argument('[url]', 'Target public URL (http/https)')
+    .requiredOption('--prompt <goal>', 'What to extract/scrape, in plain language')
+    .option('--url <url>', 'Target public URL — alias of the positional argument (#116)')
+    .option('--no-autofix', 'Disable AI auto-fix on first-run failure (default: on)')
+    .option('--json', 'Output the raw API payload')
+    .action(async (rawUrl, opts) => {
+    // Fast, local usage-errors (exit 2) — never a round-trip to the server
+    // for something we can already tell is bad. Same fail-fast pattern the
+    // former `fetch` command used for its URL argument.
+    //
+    // #116 — the url is accepted BOTH as the positional argument (agent
+    // one-liner) and as `--url` (muscle memory from `scraps create` and the
+    // API body {url, goal}). Exactly one is required; both are fine only
+    // when identical — two DIFFERENT urls is ambiguous, refuse loudly
+    // rather than silently picking one.
+    if (rawUrl === undefined && opts.url === undefined) {
+        throw new UsageError("missing required argument 'url' (positional, or --url <url>)");
+    }
+    if (rawUrl !== undefined && opts.url !== undefined && rawUrl !== opts.url) {
+        throw new UsageError(`conflicting urls: positional "${rawUrl}" vs --url "${opts.url}" — pass only one`);
+    }
+    const url = requireUrl(rawUrl ?? opts.url, 'url');
+    const goal = requireString(opts.prompt, '--prompt');
+    const body = {
+        url,
+        goal,
+        ...(opts.autofix === false && { autoFix: false }),
+    };
+    // #91/#106-F1 long-run pattern — the wizard runs AI generation + scrap
+    // creation + a real FIRST run (+ autofix retries) entirely server-side,
+    // legitimately 30-250s+. The 30s DEFAULT_TIMEOUT_MS would abort it
+    // mid-flight and fabricate a NetworkError timeout for a request that was
+    // always going to succeed.
+    const call = () => api.post('/api/ai/wizard', body, { timeoutMs: LONG_RUN_TIMEOUT_MS });
+    // #106-F2 pattern carried over from `fetch` — under --json stdout must
+    // be provably pure: no spinner channel at all. Only the human path gets
+    // the ora progress indicator; --json calls the API directly.
+    let data;
+    if (opts.json) {
+        data = await call();
+    }
+    else {
+        try {
+            data = await oraPromise(call, {
+                text: `Creating a scrap from ${url}…`,
+                // No verdict symbol here (#106-F3) — ora's success only means "the
+                // HTTP call didn't throw", not "the first run succeeded". The real
+                // outcome renders below via the icon + the First run: line.
+                successText: 'Request complete',
+            });
+        }
+        catch (err) {
+            // #114-F1 — a client-side timeout (NetworkError, "timed out after
+            // …ms" per api.ts's safeFetch) does NOT mean the wizard failed
+            // server-side: the scrap creation + first run keep going on the
+            // server after the CLI gives up waiting, so the scrap may already
+            // exist (or land moments later). Warn BEFORE rethrowing so a retry
+            // isn't the first instinct — a blind retry creates a duplicate scrap
+            // and burns AI-generation quota a second time for the same goal.
+            // Only a NetworkError whose message identifies it as the timeout
+            // branch qualifies — a DNS/connection-refused NetworkError never
+            // reached the server at all, so there's nothing to warn about here.
+            if (err instanceof NetworkError && /timed out/i.test(err.message)) {
+                console.error(chalk.yellow('⚠ The request timed out client-side, but the scrap may STILL have been created server-side — run `trawl list` before retrying (a retry creates a DUPLICATE scrap + burns quota).'));
+            }
+            // Rethrow unchanged so index.ts's classifyError/exit-code taxonomy
+            // stays intact (this stays a NetworkError -> exit 5, same as before).
+            throw err;
+        }
+    }
+    if (opts.json) {
+        json(data);
+    }
+    else {
+        const scrapId = data.scrap?._id;
+        const title = data.scrap?.title ?? url;
+        const autoFixEnabled = opts.autofix !== false;
+        const schedule = scheduleLabel(data.scrap);
+        console.log(`${data.success ? chalk.green('✓') : chalk.red('✗')} ${chalk.bold(title)}`);
+        console.log(chalk.dim(`  Scrap ID:   `) + (scrapId ?? '—'));
+        if (data.historyId)
+            console.log(chalk.dim(`  History ID: `) + data.historyId);
+        console.log(chalk.dim(`  First run:  `) + firstRunLabel(data, autoFixEnabled));
+        if (schedule)
+            console.log(chalk.dim(`  Schedule:   `) + schedule);
+        if (scrapId)
+            console.log(chalk.dim(`  Next step:  `) + `trawl get ${scrapId}`);
+        // #114-F3 — only claim a background retry is happening when a scrap
+        // actually exists to retry (never fabricate progress that isn't real);
+        // `firstRunLabel` above already covers the !scrap / autofix-disabled
+        // wording, this adds the actionable poll target on top.
+        if (!data.success && data.scrap && autoFixEnabled) {
+            const pollTarget = data.historyId
+                ? `\`trawl data ${scrapId}\` or \`trawl run-info ${data.historyId}\``
+                : `\`trawl data ${scrapId}\``;
+            console.log(chalk.yellow(`  Note:       `) +
+                `Auto-fix is retrying in the background — do NOT re-run create; poll ${pollTarget}.`);
+        }
+    }
+    // Honest exit code alongside the honest payload — a --json caller gets
+    // the raw body regardless (never wrapped/altered), but a script checking
+    // the exit code alone must be able to tell "first run failed" from "ran
+    // fine" without parsing. Same contract the former `fetch` command used.
+    if (!data.success)
+        process.exitCode = 1;
+});

package/dist/commands/scraps.d.ts

@@ -2,7 +2,8 @@ import { Command } from 'commander';
 /**
  * #108 — surface reorg. `run`/`list`/`get`/`data`/`history`/`run-info`/
  * `trigger` are promoted to top-level verbs (see index.ts's `createProgram`)
- * alongside `fetch`/`whoami`/`ping`. Each is built by an exported
+ * alongside `create`/`whoami`/`ping` (#114 — `create` replaced `fetch` in
+ * that group). Each is built by an exported
  * `attachXCommand(parent, attachOpts)` factory instead of a fixed
  * `scraps.command(...)` chain, so it can be attached TWICE with a single
  * source-of-truth definition: once to `program` (the new canonical

package/dist/commands/scraps.js

@@ -732,7 +732,7 @@ export function attachRunCommand(parent, attachOpts = {}) {
         // server-side (30-250s); the 30s default was aborting it mid-flight.
         const call = () => api.get(`/api/scraps/load/${id}`, { timeoutMs: LONG_RUN_TIMEOUT_MS });
         // #107 — under --json the stdout path stays pure: no spinner channel at
-        // all, mirroring `trawl fetch`'s own --json handling.
+        // all, mirroring `trawl create`'s own --json handling.
         const data = opts.json
             ? await call()
             : await oraPromise(call, { text: 'Launching scrap…', successText: 'Scrap launched' });

package/dist/index.js

@@ -8,7 +8,7 @@ import { scraps, attachListCommand, attachGetCommand, attachRunCommand, attachDa
 import { skills } from './commands/skills.js';
 import { telemetry } from './commands/telemetry.js';
 import { token } from './commands/token.js';
-import { fetchUrl } from './commands/fetch.js';
+import { create } from './commands/create.js';
 import { whoami } from './commands/whoami.js';
 import { ping } from './commands/ping.js';
 import { autoUpdateInstalledSkills } from './lib/skills.js';
@@ -89,14 +89,18 @@ export function createProgram() {
         .description('Trawl CLI — manage scraps from the terminal')
         .version(pkg.version)
         .option('--debug', 'Show full error stack traces');
-    // Core verbs (#108) — promoted/listed first: fetch, run, list, get, data,
+    // Core verbs (#108) — promoted/listed first: create, run, list, get, data,
     // history, run-info, trigger, whoami, ping. `list`/`get`/`run`/`data`/
     // `history`/`run-info`/`trigger` are built via scraps.ts's exported
     // attachXCommand() factories — the SAME definition also stays wired
     // (hidden) under `scraps` there, so every pre-#108 `trawl scraps <verb>`
     // invocation keeps resolving (no breaking change).
+    //
+    // #114 — `create` replaced `fetch` in this slot: `POST /api/ai/wizard`
+    // (AI-generate + persist + first-run + autofix), a distinct command from
+    // the still-untouched `trawl scraps create` (raw-script management verb).
     program.commandsGroup(CORE_GROUP);
-    program.addCommand(fetchUrl);
+    program.addCommand(create);
     attachRunCommand(program);
     attachListCommand(program);
     attachGetCommand(program);

package/dist/lib/api.d.ts

@@ -43,12 +43,15 @@ export declare function notLoggedInError(): AuthError;
  * dry-run retries). The generic 30s default was aborting those mid-flight
  * and surfacing a fabricated `NetworkError timed out` (exit 5) for a request
  * that was always going to succeed given enough time. Passed as the per-call
- * `{timeoutMs}` override at exactly the 3 call sites that hit those
- * endpoints (src/commands/scraps.ts): `scraps run` / `data --fresh`
- * (GET /api/scraps/load/:id) and `scraps trigger --wait`
- * (POST /api/scraps/worker/:id, synchronous branch only — the default async
- * `?wait=false` POST returns almost immediately and keeps the 30s default).
- * 300s leaves margin over the ~250s worst case without being unboundedly long.
+ * `{timeoutMs}` override at exactly the 4 call sites that hit those
+ * endpoints: `scraps run` / `data --fresh` (GET /api/scraps/load/:id) and
+ * `scraps trigger --wait` (POST /api/scraps/worker/:id, synchronous branch
+ * only — the default async `?wait=false` POST returns almost immediately and
+ * keeps the 30s default) — all three in src/commands/scraps.ts — plus (#114)
+ * `create` (POST /api/ai/wizard, src/commands/create.ts), whose AI-generation
+ * + scrap creation + first run + autofix pipeline runs the same 30–250s+
+ * server-side. 300s leaves margin over the ~250s worst case without being
+ * unboundedly long.
  */
 export declare const LONG_RUN_TIMEOUT_MS = 300000;
 export interface RequestOptions {

package/dist/lib/api.js

@@ -63,12 +63,15 @@ const DEFAULT_TIMEOUT_MS = 30_000;
  * dry-run retries). The generic 30s default was aborting those mid-flight
  * and surfacing a fabricated `NetworkError timed out` (exit 5) for a request
  * that was always going to succeed given enough time. Passed as the per-call
- * `{timeoutMs}` override at exactly the 3 call sites that hit those
- * endpoints (src/commands/scraps.ts): `scraps run` / `data --fresh`
- * (GET /api/scraps/load/:id) and `scraps trigger --wait`
- * (POST /api/scraps/worker/:id, synchronous branch only — the default async
- * `?wait=false` POST returns almost immediately and keeps the 30s default).
- * 300s leaves margin over the ~250s worst case without being unboundedly long.
+ * `{timeoutMs}` override at exactly the 4 call sites that hit those
+ * endpoints: `scraps run` / `data --fresh` (GET /api/scraps/load/:id) and
+ * `scraps trigger --wait` (POST /api/scraps/worker/:id, synchronous branch
+ * only — the default async `?wait=false` POST returns almost immediately and
+ * keeps the 30s default) — all three in src/commands/scraps.ts — plus (#114)
+ * `create` (POST /api/ai/wizard, src/commands/create.ts), whose AI-generation
+ * + scrap creation + first run + autofix pipeline runs the same 30–250s+
+ * server-side. 300s leaves margin over the ~250s worst case without being
+ * unboundedly long.
  */
 export const LONG_RUN_TIMEOUT_MS = 300_000;
 /**

package/docs/agent-quickstart.md

@@ -25,7 +25,8 @@ first-class on every one, and none of them ever blocks on a prompt (see
 [Non-interactive contract](#non-interactive-contract) below):
 
 ```
-trawl fetch <url> [--json] [--reason <text>]   One-shot fetch + extract readable content from a public URL (no scrap needed)
+trawl create <url> --prompt <goal> [--no-autofix] [--json]   # url also accepted as --url <url> (#116)
+                                                              Create a persistent, self-healing scrap from a URL + a goal (AI-generated)
 trawl run <id> [--watch] [--json]              Run a scrap
 trawl list|ls [--json] [--status <s>] [--limit <n>] [--page <n>]   List all scraps
 trawl get <id> [--json]                        Get scrap details
@@ -37,21 +38,39 @@ trawl whoami [--json]                          Show the authenticated user's ide
 trawl ping [--json]                            Health/version handshake against the Trawl API
 ```
 
-`trawl fetch` is the one command with no persisted scrap behind it — a
-one-shot fetch + extract for any public URL, the closest primitive to
-"just get me this page's content." It's the REST counterpart of the MCP
-`trawl_fetch_url` tool (same shared engine): `status` is an honest outcome
-(`completed`/`failed`/`empty`/`blocked`), not "did the HTTP call succeed" — a
-failed fetch is still a 200 response with `status:'failed'` + `error`, and
-the CLI exits `1` in that case even though `--json` always prints the raw
-payload verbatim.
+`trawl create` is the closest primitive to "turn this URL + goal into a
+working scrap" — it runs the AI wizard server-side (`POST /api/ai/wizard`):
+generate scrap code from `--prompt` via LLM, persist the scrap, trigger its
+first run, and auto-fix on failure (default on; `--no-autofix` disables it).
+`success` is an honest outcome of that first run, not "did the HTTP call
+succeed" — a failed first run is still a 200 response (the scrap was still
+created), and the CLI exits `1` in that case even though `--json` always
+prints the raw payload verbatim. The call legitimately runs 30–250s+
+server-side (AI generation + a real run) — the CLI arms the same long-run
+timeout `run`/`data --fresh`/`trigger --wait` use instead of the generic 30s
+default.
+
+> **Not idempotent, and not a one-off run.** A client-side timeout (exit `5`,
+> a `NetworkError`) does not mean the wizard failed server-side — scrap
+> creation + the first run keep going after the CLI gives up waiting, so the
+> scrap may already exist. Check `trawl list --json` for a matching URL/title
+> **before** retrying — a blind retry creates a duplicate scrap and burns
+> quota a second time for the same goal. Separately, the created scrap is
+> scheduled to re-run every day at 07:00 UTC by default
+> (`cron: "0 7 * * *"`, unrelated to `--no-autofix`) — each recurring run
+> consumes execute quota. Disable or change it once you've reviewed the
+> scrap: `trawl scraps update <id> --no-cron` (or `--cron <expr>`). Finally,
+> if `TRAWL_TIMEOUT` is set globally for a tighter budget than 300s, it
+> clamps `create`'s ceiling too (env always wins) — unset it or raise it
+> before calling `create`.
 
 > **No breaking change:** every verb above is also still reachable under its
 > pre-reorg path, `trawl scraps <verb>` (e.g. `trawl scraps list`) — kept as
 > a hidden alias. Prefer the bare top-level form above; it's what
 > `trawl --help` now shows.
 
-For the full flag reference (tier overrides on `create`/`update`, the
+For the full flag reference (tier overrides on `scraps create`/`scraps
+update` — the core `create` verb above has no `--tier` of its own, the
 `--watch` polling mechanics, retention/regression semantics on `data`, …)
 see the README's [Core commands](../README.md#core-commands-agent--human) section
 — this doc intentionally stays minimal.
@@ -85,19 +104,19 @@ hanging. Full rule + rationale: README's
 | Code | Meaning |
 |------|---------|
 | `0`  | Success |
-| `1`  | Unknown/generic error, or a business-logic outcome (e.g. `fetch`'s honest `status:'failed'`/`'blocked'`, `data`'s `run_failed`/`in_progress`) |
+| `1`  | Unknown/generic error, or a business-logic outcome (e.g. `create`'s honest `success:false` first-run outcome, `data`'s `run_failed`/`in_progress`) |
 | `2`  | Usage error (bad flag/value, invalid ID, missing required argument, or the non-interactive guard refusing to prompt) |
 | `3`  | Auth error (not logged in, or the session token is expired/invalid) |
 | `4`  | Not found (no such resource, or no persisted payload to read) |
 | `5`  | Network error (API host unreachable, DNS/connection/TLS failure, or timeout) |
 
 This table is the stable contract; per-command nuance and overloads (e.g.
-`fetch`'s domain-level failure sharing exit `1` with an unmapped bug) are
+`create`'s domain-level failure sharing exit `1` with an unmapped bug) are
 documented once, in the README's [Exit codes](../README.md#exit-codes)
 section — treat that as canonical if the two ever seem to disagree.
 
 ## Minimal example
 
 ```bash
-TRAWL_TOKEN=<jwt> trawl fetch https://example.com --json
+TRAWL_TOKEN=<jwt> trawl create https://example.com --prompt "Extract the article title and body text" --json
 ```

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@trawlme/cli",
-  "version": "1.22.0",
+  "version": "2.1.0",
   "description": "Trawl CLI — manage scraps from the terminal",
   "type": "module",
   "bin": {

package/dist/commands/fetch.d.ts

@@ -1,31 +0,0 @@
-import { Command } from 'commander';
-/**
- * `POST /api/scraps/fetch-url` response contract (#1697, trawl_node —
- * shipped, contract LOCKED). REST counterpart of the MCP `trawl_fetch_url`
- * tool: both run through the same shared engine
- * (scraps.fetchUrl.service.js#runEphemeralUrlFetch), so the outcome shape is
- * identical regardless of transport.
- *
- *  - `status`: 'completed' | 'failed' | 'empty' | 'blocked' — an HONEST
- *    outcome, not a bare "it ran". A failed/blocked/empty run is still a
- *    200 response (this is not an HTTP error), so callers must branch on
- *    `status`, never assume 2xx means "got data".
- *  - `error` is present ONLY when `status === 'failed'`.
- *  - `truncated: true` when the 100KB response-payload cap tripped — the
- *    server drops `result` in that case and repoints `url` at the full
- *    history row (`/api/historys/:runId`) instead of echoing the fetched
- *    target.
- */
-export interface FetchUrlResponse {
-    url: string;
-    status: 'completed' | 'failed' | 'empty' | 'blocked';
-    runId?: string | null;
-    statusDetail?: string | null;
-    blocked?: boolean;
-    length?: number | null;
-    result?: unknown;
-    truncated?: boolean;
-    reason?: string;
-    error?: string;
-}
-export declare const fetchUrl: Command;

package/dist/commands/fetch.js

@@ -1,95 +0,0 @@
-import { Command } from 'commander';
-import chalk from 'chalk';
-import { oraPromise } from 'ora';
-import { api, LONG_RUN_TIMEOUT_MS } from '../lib/api.js';
-import { json } from '../lib/format.js';
-import { requireUrl } from '../lib/validate.js';
-function statusIcon(status) {
-    if (status === 'completed')
-        return chalk.green('✓');
-    if (status === 'failed')
-        return chalk.red('✗');
-    if (status === 'blocked')
-        return chalk.yellow('⚠');
-    return chalk.dim('•'); // empty
-}
-/** Best-effort human summary of `result` — same "count + first-item keys"
- * shape scraps.ts's renderScrapItems uses, never a full dump (that's what
- * --json is for). */
-function renderResultSummary(result) {
-    if (!Array.isArray(result))
-        return;
-    console.log(chalk.dim(`  Items:      `) + result.length);
-    const first = result[0];
-    if (result.length > 0 && first && typeof first === 'object') {
-        console.log(chalk.dim(`  First keys: `) + Object.keys(first).join(', '));
-    }
-}
-export const fetchUrl = new Command('fetch')
-    .description('One-shot fetch + extract readable content from a public URL (no scrap needed)')
-    .argument('<url>', 'Target public HTTPS URL')
-    .option('--json', 'Output the raw API payload')
-    .option('--reason <reason>', 'Audit-trail reason for this fetch (logged server-side, max 500 chars)')
-    .action(async (rawUrl, opts) => {
-    // Fast, local usage-error (exit 2) on an obviously malformed URL — never
-    // a round-trip to the server for something we can already tell is bad.
-    // The server still re-validates (SSRF guard) — this is a UX fast-path,
-    // not a security boundary.
-    const url = requireUrl(rawUrl, 'url');
-    const body = {
-        url,
-        ...(opts.reason !== undefined && { reason: opts.reason }),
-    };
-    // #106 review F1 — this endpoint runs the FULL worker pipeline (browser
-    // `goto` networkidle2 + extraction + persistence), legitimately 30-250s
-    // server-side — same #91 P0 pattern as `scraps run` / `data --fresh` /
-    // `trigger --wait` (src/commands/scraps.ts). The 30s DEFAULT_TIMEOUT_MS
-    // was aborting it mid-flight and would have surfaced a fabricated
-    // NetworkError timeout for a request that was always going to succeed.
-    const call = () => api.post('/api/scraps/fetch-url', body, { timeoutMs: LONG_RUN_TIMEOUT_MS });
-    // #106 review F2 — under --json the stdout path must be provably pure:
-    // no spinner channel at all. Only the human path gets the ora progress
-    // indicator; --json calls the API directly.
-    const data = opts.json
-        ? await call()
-        : await oraPromise(call, {
-            text: `Fetching ${url}…`,
-            // #106 review F3 — no successText verdict here. ora's success
-            // symbol only means "the HTTP call didn't throw", not "the fetch
-            // succeeded" — a failed/blocked domain `status` is still a 200
-            // response. The real outcome is rendered below via statusIcon +
-            // the Status: line; a green check here would contradict a
-            // red/yellow icon printed right after it.
-            successText: 'Request complete',
-        });
-    if (opts.json) {
-        json(data);
-    }
-    else {
-        console.log(`${statusIcon(data.status)} ${chalk.bold(data.url)}`);
-        console.log(chalk.dim(`  Status:     `) + data.status);
-        if (data.runId)
-            console.log(chalk.dim(`  Run ID:     `) + data.runId);
-        if (data.statusDetail)
-            console.log(chalk.dim(`  Detail:     `) + data.statusDetail);
-        if (data.length != null)
-            console.log(chalk.dim(`  Length:     `) + data.length);
-        if (data.truncated) {
-            console.log(chalk.yellow(`  ⚠ Truncated (payload too large) — full result: ${data.url}`));
-        }
-        if (data.status === 'failed' && data.error) {
-            console.log(chalk.red(`  Error:      `) + data.error);
-        }
-        renderResultSummary(data.result);
-    }
-    // Honest exit code alongside the honest payload — a --json caller gets
-    // the raw body regardless (never wrapped/altered), but a script checking
-    // the exit code alone must be able to tell "no usable data" from "ran
-    // fine" without parsing. `blocked` (#106 review F4 — an antibot wall) is
-    // a failure to get data exactly like `failed`: an agent scripting
-    // `trawl fetch ... || handle` must see non-zero for either. `empty`
-    // stays 0 on purpose — the fetch genuinely ran to completion, there was
-    // just nothing extractable at that URL; that's not an error.
-    if (data.status === 'failed' || data.status === 'blocked')
-        process.exitCode = 1;
-});