@trawlme/cli 1.21.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +45 -34
- package/dist/commands/create.d.ts +41 -0
- package/dist/commands/create.js +146 -0
- package/dist/commands/scraps.d.ts +30 -0
- package/dist/commands/scraps.js +413 -392
- package/dist/index.d.ts +8 -0
- package/dist/index.js +50 -8
- package/dist/lib/api.d.ts +9 -6
- package/dist/lib/api.js +9 -6
- package/docs/agent-quickstart.md +121 -0
- package/package.json +2 -1
- package/dist/commands/fetch.d.ts +0 -31
- package/dist/commands/fetch.js +0 -95
package/README.md
CHANGED
|
@@ -8,6 +8,8 @@ Command-line client for [Trawl](https://trawl.me) — manage your scraps from th
|
|
|
8
8
|
|
|
9
9
|
> ⚠️ **Early stage.** APIs may change before `1.0`. Pin a version if you depend on this in CI.
|
|
10
10
|
|
|
11
|
+
> 🤖 **Driving this CLI from an AI agent?** See [docs/agent-quickstart.md](docs/agent-quickstart.md) for the minimal surface (`create`, `--json`, `TRAWL_TOKEN`, exit codes) — this README is the full human/CI guide.
|
|
12
|
+
|
|
11
13
|
## Install
|
|
12
14
|
|
|
13
15
|
Requires Node.js >= 20.
|
|
@@ -21,48 +23,52 @@ npm install -g @trawlme/cli
|
|
|
21
23
|
Three methods:
|
|
22
24
|
|
|
23
25
|
1. **Interactive** — `trawl login` (prompts for email and password)
|
|
24
|
-
2. **Env var** — `TRAWL_TOKEN=<jwt> trawl
|
|
26
|
+
2. **Env var** — `TRAWL_TOKEN=<jwt> trawl list` (CI/CD, bypasses prompt)
|
|
25
27
|
3. **Token flag** — `trawl login --token <jwt>` (CI/CD, direct JWT)
|
|
26
28
|
|
|
27
29
|
Custom API URL: `trawl login --url https://self-hosted.example.com`
|
|
28
30
|
|
|
29
31
|
## Commands
|
|
30
32
|
|
|
33
|
+
`trawl --help` groups commands into two labeled tiers, in this exact wording: **Core commands (agent + human)** (listed first — `--json` first-class, never interactive) and **Management commands (human/CI)** (grouped below — the full scrap-admin/account/auth/skills/telemetry surface). Commander also prints a small ungrouped `help [command]` entry at the bottom, so "two tiers" describes the labeled sections, not literally every line of `--help` output. This section documents both tiers, in that order; [docs/agent-quickstart.md](docs/agent-quickstart.md) covers just the Core tier for an agent integration.
|
|
34
|
+
|
|
31
35
|
All commands accept a global `--debug` flag to show full error stack traces on failure. Every command below supports `--json` (a raw payload / structured result on stdout, nothing else) and never blocks on an interactive prompt when `--json` is set or stdin/stdout isn't a real TTY — see [Non-interactive rule](#non-interactive-rule) and [Exit codes](#exit-codes). Two documented exceptions to "single payload": `scraps watch --json` is a live stream, so it emits NDJSON (one JSON object per line) instead; `scraps snapshot --json` only changes behavior on a never-run scrap (`{"status":"no_runs"}`) — when a run exists it still writes the raw HTML snapshot regardless of `--json` (there's no JSON-encoded form of an HTML page to emit).
|
|
32
36
|
|
|
33
|
-
###
|
|
37
|
+
### Core commands (agent + human)
|
|
34
38
|
|
|
35
39
|
```
|
|
36
|
-
trawl
|
|
37
|
-
trawl
|
|
38
|
-
trawl
|
|
40
|
+
trawl create <url> --prompt <goal> [--no-autofix] [--json] Create a persistent, self-healing scrap from a URL + a goal (AI-generated)
|
|
41
|
+
trawl run <id> [--watch] [--json] Run a scrap
|
|
42
|
+
trawl list|ls [--json] [--status <success|failure|never|running|regression>] [--limit <n>] [--page <n>]
|
|
43
|
+
trawl get <id> [--json] Get scrap details
|
|
44
|
+
trawl data <id> [--json] [--fresh] [--errors] Get scrap data (last persisted run — read-only, no quota; --fresh to launch one)
|
|
45
|
+
trawl history <id> [--json] [-n <limit>] List past runs for a scrap (newest first)
|
|
46
|
+
trawl run-info <hid> [--json] Show details of a single run
|
|
47
|
+
trawl trigger <id> [--watch] [--wait] [--json] Launch a scrap as a background worker (returns immediately)
|
|
48
|
+
trawl whoami [--json] Show the authenticated user's identity
|
|
49
|
+
trawl ping [--json] Health/version handshake against the Trawl API
|
|
39
50
|
```
|
|
40
51
|
|
|
41
|
-
|
|
52
|
+
> **No breaking change:** every verb above is also still reachable under its pre-reorg path, `trawl scraps <verb>` (e.g. `trawl scraps list`, `trawl scraps run <id>`) — kept as a hidden alias so scripts written before the surface reorg keep working. `trawl --help` only shows the top-level form above; `trawl scraps --help` only shows the remaining scrap-management commands below.
|
|
42
53
|
|
|
43
|
-
|
|
54
|
+
`create`/`whoami`/`ping` are fully non-interactive — all three read auth only from `TRAWL_TOKEN`/the stored login token, never prompt. `trawl create` runs the AI wizard server-side (`POST /api/ai/wizard`): generate scrap code from `--prompt` via LLM, persist the scrap, trigger its FIRST run, and auto-fix on failure (default on — `--no-autofix` disables it, sending `autoFix:false`). `success` is an honest outcome of that first run, not "did the HTTP call succeed" — a failed first run is still a 200 response (the scrap was still created; auto-fix, when enabled, retries in the background), and the CLI exits 1 in that case (both human and `--json` modes) even though `--json` always prints the raw payload verbatim. The call legitimately takes 30–250s+ server-side (AI generation + a real run), same long-run timeout as `run`/`data --fresh`/`trigger --wait` below. `trawl whoami`/`trawl ping` mirror the MCP `trawl_whoami`/`trawl_health_ping` tools as closely as the REST surface allows (`GET /api/users/me` / `GET /api/health`) — `ping`'s `--json` payload is admin-enriched (version/uptime/db) and just `{"status":"ok"}` for anyone else.
|
|
44
55
|
|
|
45
|
-
|
|
46
|
-
trawl fetch <url> [--json] [--reason <text>] One-shot fetch + extract readable content from a public URL (no scrap needed)
|
|
47
|
-
trawl whoami [--json] Show the authenticated user's identity
|
|
48
|
-
trawl ping [--json] Health/version handshake against the Trawl API
|
|
49
|
-
```
|
|
56
|
+
> **`create` is NOT idempotent, and every wizard-created scrap runs on a DAILY cron by default.** A client-side timeout (exit `5`, a `NetworkError`) does not mean the wizard failed server-side — scrap creation + the first run keep going after the CLI gives up waiting, so the scrap may already exist. Run `trawl list` and look for a matching URL/title **before** retrying — a blind retry creates a DUPLICATE scrap and burns AI-generation quota a second time for the same goal. Separately, the scrap the wizard creates is scheduled to re-run every day at 07:00 UTC (`cron: "0 7 * * *"`, hardcoded server-side, unrelated to `--no-autofix`) — each of those recurring runs consumes execute quota like any other run. Review the generated scrap, then change or disable the schedule with `trawl scraps update <id> --cron <expr>` (or `--no-cron` to disable it). Because the call can legitimately run 250s+, also confirm `TRAWL_TIMEOUT` isn't set to something tighter than `create` needs — the env var always wins over `create`'s own 300s default (see [Environment variables](#environment-variables)), so a value set for another purpose (e.g. a tight CI smoke-test budget) silently clamps `create` too; unset it or raise it before running `create`.
|
|
50
57
|
|
|
51
|
-
|
|
58
|
+
- `list` has a short alias, `ls` (matches `trawl --help`'s `list|ls`).
|
|
59
|
+
- `history` lists past runs (newest first); `run-info <hid>` shows details of a single run from that history.
|
|
60
|
+
- `data` returns the last persisted run payload (no execute quota); `--fresh` runs the scrap live instead (consumes execute quota); `--errors` shows the last run's error detail (`--json` on a never-run scrap returns `{"status":"no_runs"}`, exit 0, matching `scraps doctor --json`). `[]` on stdout means a genuine zero-item successful run — a scrap that has never run, whose last run failed, or whose payload aged out of retention returns a `--json` error envelope (exit 4/1/4 respectively) instead. Two more honest states: a run still **in flight** (`status: null` server-side) returns a `kind:"in_progress"` error envelope (exit 1, "retry shortly" — never suggests `--fresh`, which would just 429 against the run already holding the lock); a run whose item count **regressed** vs baseline (`statusDetail: "regression"`) still returns the real, non-empty items on stdout (exit 0) plus a stderr warning pointing at `scraps doctor <id>` — the data itself is genuine even though the run is flagged.
|
|
61
|
+
- `get` (and anything reading through it, like `data`'s default path) embeds only the newest 100 history rows on the returned scrap object — `run-info` and `scraps doctor` fetch a single run directly and are unaffected by that cap. `list`/`get` show a distinct amber `▼` "regression" badge, never the red `✗` a genuine failure gets (matches `scraps doctor`'s own badge).
|
|
62
|
+
- **Long-running calls (`create`, `run`, `data --fresh`, `trigger --wait`):** these hit server-side paths that can legitimately take 30–250s+ (AI generation + scrap creation + a first run for `create`; proxy tier escalation + AI-fix retries for the other three) — the CLI arms a 300s timeout for exactly these four call sites instead of the generic 30s default. `TRAWL_TIMEOUT` (see below) still overrides ALL requests, including these — set it if you need a tighter or looser ceiling than 300s, but note a global override that tight also clamps `create`.
|
|
63
|
+
- **`--watch` is poll-based, not a live stream:** the activities SSE endpoint has no backlog and, for the default async `trigger` (no `--wait`), runs in a separate cron-consumer pod whose events never reach the API pod holding the SSE connection — a naive "await the run, then open SSE" shows nothing. `run --watch` and `trigger --watch` instead poll `GET /api/scraps/:id` (terminal status) and the activities REST list until the run finishes, printing each new activity line as it appears. The watched run's outcome drives the exit code too, in BOTH human and `--json` mode: a genuinely failed terminal run, a poll timeout, or a persistently unreachable API all exit non-zero — a clean successful run is the only exit `0`. A run that never reaches a terminal status within 300s prints an honest timeout notice pointing at `scraps doctor <id>` (human mode) — see the `--json` shape below. `scraps watch <id>` (the standalone command, no trigger) is unchanged — it still opens the live SSE stream directly.
|
|
64
|
+
- `run --json`/`trigger --json` bypass the spinner and print the raw launch/trigger payload on stdout; combined with `--watch`, every intermediate progress line stays suppressed (stdout stays pure JSON) and, once the watch reaches its outcome, exactly ONE final NDJSON line is emitted: `{"runId","status"}` (the honest terminal status — `success`/`error`/`empty`/`regression`/…), `{"runId","status":"timeout"}` on a poll timeout, or `{"runId","status":"poll_error","error"}` if the API stays unreachable for several consecutive polls — `process.exitCode` is non-zero for all three except a genuine success. `scraps watch --json` emits one raw JSON object per activity line (NDJSON) instead of the formatted `[time] message` text — there's no single final payload to wait for on a live stream.
|
|
52
65
|
|
|
53
|
-
###
|
|
66
|
+
### Scrap management
|
|
54
67
|
|
|
55
68
|
```
|
|
56
|
-
trawl scraps list [--json] [--status <success|failure|never|running|regression>] [--limit <n>] [--page <n>]
|
|
57
|
-
trawl scraps get <id> [--json]
|
|
58
69
|
trawl scraps create -t <title> [-u <url>] [-r <request>] [-d <description>] [--tier <tier0|tier1|tier2|tier3|tier4>] [--json]
|
|
59
70
|
trawl scraps update <id> [-t <title>] [-u <url>] [-r <request>] [-d <description>] [--cron <expr>|--no-cron] [--alert <email>|--no-alert] [--autofix|--no-autofix] [-p <json>|--params-file <path>] [--tier <tier0|tier1|tier2|tier3|tier4>] [--force-tier <tier0|tier1|tier2|tier3|tier4>] [--json]
|
|
60
|
-
trawl scraps run <id> [--watch] [--json]
|
|
61
|
-
trawl scraps trigger <id> [--watch] [--wait] [--json]
|
|
62
71
|
trawl scraps watch <id> [--json]
|
|
63
|
-
trawl scraps data <id> [--json] [--fresh] [--errors]
|
|
64
|
-
trawl scraps history <id> [--json] [-n <limit>]
|
|
65
|
-
trawl scraps run-info <hid> [--json]
|
|
66
72
|
trawl scraps doctor <id> [--json] [--autofix]
|
|
67
73
|
trawl scraps autofix <id> [--json]
|
|
68
74
|
trawl scraps snapshot <id> [--error] [-o <file>] [--json]
|
|
@@ -71,16 +77,11 @@ trawl scraps delete <id> [--force] [--json] Alias: rm
|
|
|
71
77
|
```
|
|
72
78
|
|
|
73
79
|
- `--tier` forces a proxy tier; `--force-tier` raises the proxy-tier ceiling past the auto-cap (history-gated: may be refused or cost more). `create --json`/`update --json` print the full scrap object (including the `_tierOverride` outcome) on stdout; a refused tier override exits 1 with a standard `--json` error envelope (`kind:"refused"` — distinct from `"unknown"`, so a script can branch on "the server said no"). When a tier was requested but the server's response carries no `_tierOverride` at all (an older server that can't confirm what actually got applied), a stderr warning is printed either way, and under `--json` the emitted object also carries `"_tierUnconfirmed": true` — the machine-readable counterpart to that warning, since a `--json` caller has no reliable reason to read stderr.
|
|
74
|
-
- `scraps doctor` diagnoses the last run (error, failed selector, block status, page state, autofix outcome); `--autofix` includes the full autofix diff/dry-run/knowledge. A run that is still in flight (`status: null`, server-side) shows a `running` badge, never `failed`; a run whose item count regressed vs baseline (`statusDetail: "regression"`) shows its own amber `regression` badge, never `failed` either
|
|
80
|
+
- `scraps doctor` diagnoses the last run (error, failed selector, block status, page state, autofix outcome); `--autofix` includes the full autofix diff/dry-run/knowledge. A run that is still in flight (`status: null`, server-side) shows a `running` badge, never `failed`; a run whose item count regressed vs baseline (`statusDetail: "regression"`) shows its own amber `regression` badge, never `failed` either.
|
|
75
81
|
- `scraps autofix` shows the last auto-fix attempt on its own (decision, diff, dry-run, knowledge). `--json` on a scrap that has **never run** returns `{"status":"no_runs"}` (exit 0) — distinct from `null`, which means a run exists but had no auto-fix attempt.
|
|
76
82
|
- `scraps snapshot --error` fetches the error-path snapshot instead of the normal one; `-o <file>` writes to a file instead of stdout. On a scrap that has **never run**: `--json` returns `{"status":"no_runs"}` (exit 0), matching `doctor`/`autofix`; `-o <file>` (without `--json`) exits 4 (`not_found`) instead of silently exiting 0 with nothing written — a script checking the exit code alone must be able to tell "no file was produced" from success. `--json` takes priority when both are passed.
|
|
77
|
-
- `scraps history` lists past runs (newest first); `scraps run-info <hid>` shows details of a single run from that history.
|
|
78
|
-
- `scraps data` returns the last persisted run payload (no execute quota); `--fresh` runs the scrap live instead (consumes execute quota); `--errors` shows the last run's error detail (`--json` on a never-run scrap returns `{"status":"no_runs"}`, exit 0, matching `doctor --json`). `[]` on stdout means a genuine zero-item successful run — a scrap that has never run, whose last run failed, or whose payload aged out of retention returns a `--json` error envelope (exit 4/1/4 respectively) instead. Two more honest states: a run still **in flight** (`status: null` server-side) returns a `kind:"in_progress"` error envelope (exit 1, "retry shortly" — never suggests `--fresh`, which would just 429 against the run already holding the lock); a run whose item count **regressed** vs baseline (`statusDetail: "regression"`) still returns the real, non-empty items on stdout (exit 0) plus a stderr warning pointing at `scraps doctor <id>` — the data itself is genuine even though the run is flagged.
|
|
79
|
-
- `scraps get` (and anything reading through it, like `scraps data`'s default path) embeds only the newest 100 history rows on the returned scrap object — `scraps run-info` and `scraps doctor` fetch a single run directly and are unaffected by that cap.
|
|
80
83
|
- `scraps banner -f <file>` only accepts `png`/`jpg`/`jpeg`/`webp`; any other extension is a usage error (exit 2) instead of silently uploading the file under a fabricated `image/png` Content-Type.
|
|
81
|
-
-
|
|
82
|
-
- **`--watch` is poll-based, not a live stream:** the activities SSE endpoint has no backlog and, for the default async `trigger` (no `--wait`), runs in a separate cron-consumer pod whose events never reach the API pod holding the SSE connection — a naive "await the run, then open SSE" shows nothing. `scraps run --watch` and `scraps trigger --watch` instead poll `GET /api/scraps/:id` (terminal status) and the activities REST list until the run finishes, printing each new activity line as it appears. The watched run's outcome drives the exit code too, in BOTH human and `--json` mode: a genuinely failed terminal run, a poll timeout, or a persistently unreachable API all exit non-zero — a clean successful run is the only exit `0`. A run that never reaches a terminal status within 300s prints an honest timeout notice pointing at `scraps doctor <id>` (human mode) — see the `--json` shape below. `scraps watch <id>` (the standalone command, no trigger) is unchanged — it still opens the live SSE stream directly.
|
|
83
|
-
- `scraps run --json`/`trigger --json` bypass the spinner and print the raw launch/trigger payload on stdout; combined with `--watch`, every intermediate progress line stays suppressed (stdout stays pure JSON) and, once the watch reaches its outcome, exactly ONE final NDJSON line is emitted: `{"runId","status"}` (the honest terminal status — `success`/`error`/`empty`/`regression`/…), `{"runId","status":"timeout"}` on a poll timeout, or `{"runId","status":"poll_error","error"}` if the API stays unreachable for several consecutive polls — `process.exitCode` is non-zero for all three except a genuine success. `scraps watch --json` emits one raw JSON object per activity line (NDJSON) instead of the formatted `[time] message` text — there's no single final payload to wait for on a live stream. `scraps delete`/`banner --json` print `{"deleted":true,"id"}` / the raw upload response.
|
|
84
|
+
- `scraps delete`/`banner --json` print `{"deleted":true,"id"}` / the raw upload response.
|
|
84
85
|
|
|
85
86
|
### Scrap accounts
|
|
86
87
|
|
|
@@ -113,6 +114,16 @@ A pre-existing skill directory that trawl did not install itself (no `.version`
|
|
|
113
114
|
|
|
114
115
|
You can also install skills standalone (without the CLI): `npx @trawlme/skills install`.
|
|
115
116
|
|
|
117
|
+
### Auth
|
|
118
|
+
|
|
119
|
+
```
|
|
120
|
+
trawl login [--url <url>] [--token <jwt>] [--email <email>] [--password <pass>] [--json]
|
|
121
|
+
trawl logout [--json]
|
|
122
|
+
trawl token [--json] Print the stored session JWT (for MCP Bearer auth)
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
`login --json` prints `{"ok":true,"apiUrl","config",email?}` on success — never the raw token (that's `token`'s job). `token --json` prints `{"token","exp","expiresAt"}` instead of the bare JWT + stderr advisories. See [Non-interactive rule](#non-interactive-rule) below: a missing `--email`/`--password` refuses immediately (usage error, exit 2) instead of prompting when `--json` is set or stdin/stdout isn't a real TTY.
|
|
126
|
+
|
|
116
127
|
### Telemetry
|
|
117
128
|
|
|
118
129
|
```
|
|
@@ -126,7 +137,7 @@ trawl telemetry status [--json] Show current state, telemetry ID, and opt-out
|
|
|
126
137
|
`@trawlme/cli` collects anonymous usage data to help us improve the CLI based on real usage patterns. Telemetry is enabled by default.
|
|
127
138
|
|
|
128
139
|
**What we collect:**
|
|
129
|
-
- Command name (e.g. `scraps
|
|
140
|
+
- Command name (e.g. `list` — the #108 surface reorg renamed a promoted verb's telemetry event from `scraps <verb>` to `<verb>`; invoking the legacy `trawl scraps <verb>` path still emits the pre-reorg `scraps <verb>` name)
|
|
130
141
|
- CLI version
|
|
131
142
|
- Node.js version
|
|
132
143
|
- Platform (e.g. `darwin`, `linux`, `win32`)
|
|
@@ -149,7 +160,7 @@ trawl telemetry status [--json] Show current state, telemetry ID, and opt-out
|
|
|
149
160
|
trawl telemetry off
|
|
150
161
|
|
|
151
162
|
# Session-level opt-out (env var, takes precedence)
|
|
152
|
-
TRAWL_TELEMETRY=0 trawl
|
|
163
|
+
TRAWL_TELEMETRY=0 trawl list
|
|
153
164
|
```
|
|
154
165
|
|
|
155
166
|
**Why:** usage data helps us prioritise CLI features and catch silent errors before users report them.
|
|
@@ -170,10 +181,10 @@ Every command exits with one of these codes — scripts and agents driving the C
|
|
|
170
181
|
| Code | Meaning |
|
|
171
182
|
|------|--------------------------------------------------------------------------|
|
|
172
183
|
| `0` | Success |
|
|
173
|
-
| `1` | Unknown/generic error (an unmapped failure — API errors other than 401/404, an unhandled bug — or a business-logic refusal like `
|
|
184
|
+
| `1` | Unknown/generic error (an unmapped failure — API errors other than 401/404, an unhandled bug — or a business-logic refusal like `data`'s `run_failed`/`in_progress` states, or `trawl create`'s honest `success:false` first-run outcome). **Known overload:** `create`'s domain-level failure and an arbitrary unmapped bug both land on `1` — a script needs to read the `--json` payload's own `success` field (for `create`) to tell them apart; this is intentional (the REST contract's outcome states are a separate axis from the CLI's transport-error taxonomy) and documented here rather than "resolved" by inventing a new code that would only apply to one command. |
|
|
174
185
|
| `2` | Usage error (bad flag/value, invalid ID, missing required argument, unknown option/command — including the [non-interactive rule](#non-interactive-rule) refusing to prompt) |
|
|
175
186
|
| `3` | Auth error (not logged in, or the session token is expired/invalid — run `trawl login`) |
|
|
176
|
-
| `4` | Not found (no such resource, or — for `
|
|
187
|
+
| `4` | Not found (no such resource, or — for `data` — no persisted payload to read) |
|
|
177
188
|
| `5` | Network error (the API host is unreachable, DNS/connection/TLS failure, or the request timed out) |
|
|
178
189
|
|
|
179
190
|
Under `--json`, a failing command emits a single error envelope on stdout — `{"error":{"message","status?","kind"}}` — instead of prose; the human-readable line always goes to stderr, never stdout. `kind` is the machine-readable discriminant (`"usage"`/`"auth"`/`"not_found"`/`"network"`/`"api"`/`"refused"`/`"unknown"`/…) — the short string a script should switch on (`"refused"` is a business-logic refusal the server explicitly reported back, e.g. `scraps create|update`'s tier-ceiling override rejection — distinct from `"unknown"`, which stays reserved for an unmapped bug); `status` is present only when a real HTTP response carried one (never fabricated for a local failure like an expired-locally JWT or a refused confirmation prompt).
|
|
@@ -187,13 +198,13 @@ Under `--json`, a failing command emits a single error envelope on stdout — `{
|
|
|
187
198
|
| `TRAWL_TELEMETRY` | Set to `0` to disable telemetry for the current session |
|
|
188
199
|
| `DO_NOT_TRACK` | Set to `1` to disable telemetry (cross-vendor convention, https://consoledonottrack.com) — same effect as `TRAWL_TELEMETRY=0` |
|
|
189
200
|
| `TRAWL_CONFIG_DIR` | Override where the config file (token, API URL, telemetry state) is stored — useful for hermetic CI runs or concurrent `trawl login`s that must not share one on-disk file |
|
|
190
|
-
| `TRAWL_TIMEOUT` | Override the per-request fetch timeout in milliseconds (default `30000`; `300000` for `
|
|
201
|
+
| `TRAWL_TIMEOUT` | Override the per-request fetch timeout in milliseconds (default `30000`; `300000` for `create`/`run`/`data --fresh`/`trigger --wait` — this env var always wins over that longer default too) |
|
|
191
202
|
| `TRAWL_SKILLS_SYNC` | Set to `0` to disable the startup skills auto-sync entirely |
|
|
192
203
|
|
|
193
204
|
Session override (no `trawl login` mutation, ideal for CI/QA against another env):
|
|
194
205
|
|
|
195
206
|
```bash
|
|
196
|
-
TRAWL_API_URL=https://dev.trawl.me TRAWL_TOKEN=<jwt> trawl
|
|
207
|
+
TRAWL_API_URL=https://dev.trawl.me TRAWL_TOKEN=<jwt> trawl list --json
|
|
197
208
|
```
|
|
198
209
|
|
|
199
210
|
## API documentation
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
import { Command } from 'commander';
|
|
2
|
+
/**
|
|
3
|
+
* `POST /api/ai/wizard` response contract (#114, trawl_node —
|
|
4
|
+
* ai.wizard.service.js#runWizard, shipped S1, contract LOCKED). The wizard
|
|
5
|
+
* chains, entirely server-side: AI code generation -> scrap creation ->
|
|
6
|
+
* the FIRST run (ScrapsService.load) -> auto-fix on failure (autoFix
|
|
7
|
+
* defaults true unless `--no-autofix` maps to `autoFix:false`).
|
|
8
|
+
*
|
|
9
|
+
* - `success` is an HONEST outcome of that first run, not "did the HTTP
|
|
10
|
+
* call succeed" — a failed first run is still a 200 response (the scrap
|
|
11
|
+
* itself was created either way; auto-fix, when enabled, retries in the
|
|
12
|
+
* background). Callers must branch on `success`, never assume 2xx means
|
|
13
|
+
* "the scrap works".
|
|
14
|
+
* - `scrap` is the full created Scrap object (present whenever creation got
|
|
15
|
+
* far enough to persist it — a hard failure before that point surfaces as
|
|
16
|
+
* a real HTTP error instead, handled by the shared error path).
|
|
17
|
+
* - `historyId` is best-effort (a failed server-side lookup leaves it
|
|
18
|
+
* `null`, never breaks the response) — it points at the first run just
|
|
19
|
+
* executed.
|
|
20
|
+
*/
|
|
21
|
+
export interface WizardResponse {
|
|
22
|
+
success: boolean;
|
|
23
|
+
scrap?: {
|
|
24
|
+
_id: string;
|
|
25
|
+
title: string;
|
|
26
|
+
/**
|
|
27
|
+
* Cron expression the wizard schedules this scrap on. As of #114/S1
|
|
28
|
+
* (trawl_node ai.wizard.service.js#runWizard's `scrapBody`) this is
|
|
29
|
+
* hardcoded server-side to a DAILY run — `'0 7 * * *'` / `cronTimezone:
|
|
30
|
+
* 'UTC'` — unconditionally, regardless of `--prompt`/`--no-autofix`.
|
|
31
|
+
* Present on the scrap object returned here (the wizard controller
|
|
32
|
+
* passes the created scrap straight through, no stripping) — change or
|
|
33
|
+
* disable it with `trawl scraps update <id> --cron <expr>` / `--no-cron`.
|
|
34
|
+
*/
|
|
35
|
+
cron?: string | null;
|
|
36
|
+
cronTimezone?: string;
|
|
37
|
+
[key: string]: unknown;
|
|
38
|
+
};
|
|
39
|
+
historyId?: string | null;
|
|
40
|
+
}
|
|
41
|
+
export declare const create: Command;
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
import { Command } from 'commander';
|
|
2
|
+
import chalk from 'chalk';
|
|
3
|
+
import { oraPromise } from 'ora';
|
|
4
|
+
import { api, LONG_RUN_TIMEOUT_MS, NetworkError } from '../lib/api.js';
|
|
5
|
+
import { json } from '../lib/format.js';
|
|
6
|
+
import { requireUrl, requireString } from '../lib/validate.js';
|
|
7
|
+
/** Best-effort, honest first-run summary — never claims a background retry
|
|
8
|
+
* happened when auto-fix was disabled for this call, and never claims a
|
|
9
|
+
* scrap was persisted when the response carries none (#114-F3 — a hard
|
|
10
|
+
* failure before persistence still comes back as `success:false` with no
|
|
11
|
+
* `scrap` at all; claiming "auto-fix retrying in the background" then would
|
|
12
|
+
* be fabricated — there is nothing to retry). */
|
|
13
|
+
function firstRunLabel(data, autoFixEnabled) {
|
|
14
|
+
if (data.success)
|
|
15
|
+
return 'succeeded';
|
|
16
|
+
if (!data.scrap)
|
|
17
|
+
return 'failed (no scrap was persisted)';
|
|
18
|
+
if (autoFixEnabled)
|
|
19
|
+
return 'failed (auto-fix retrying in the background)';
|
|
20
|
+
return 'failed';
|
|
21
|
+
}
|
|
22
|
+
/** Best-effort human description of a daily cron (`M H * * *`) — the only
|
|
23
|
+
* shape the wizard's server-side default currently produces. Falls back to
|
|
24
|
+
* printing the raw expression for anything else rather than guessing at a
|
|
25
|
+
* schedule the CLI can't actually parse. */
|
|
26
|
+
function describeCron(cron) {
|
|
27
|
+
const match = /^(\d{1,2})\s+(\d{1,2})\s+\*\s+\*\s+\*$/.exec(cron.trim());
|
|
28
|
+
if (!match)
|
|
29
|
+
return cron;
|
|
30
|
+
const [, min, hour] = match;
|
|
31
|
+
return `daily ${hour.padStart(2, '0')}:${min.padStart(2, '0')}`;
|
|
32
|
+
}
|
|
33
|
+
/**
|
|
34
|
+
* #114-F2 — a wizard-created scrap runs on a DAILY cron by default
|
|
35
|
+
* server-side; surface that up front rather than leaving it to be
|
|
36
|
+
* discovered later as an unexpected recurring quota charge. Reads the real
|
|
37
|
+
* `cron`/`cronTimezone` field off the response scrap when present; falls
|
|
38
|
+
* back to the known wizard default wording ONLY when the field is missing
|
|
39
|
+
* from the response (an older server, or a future rename) — never invents a
|
|
40
|
+
* schedule value that might not match what the server actually applied.
|
|
41
|
+
* Returns null when there's no scrap to schedule at all.
|
|
42
|
+
*/
|
|
43
|
+
function scheduleLabel(scrap) {
|
|
44
|
+
if (!scrap)
|
|
45
|
+
return null;
|
|
46
|
+
if (typeof scrap.cron === 'string' && scrap.cron.trim() !== '') {
|
|
47
|
+
const tz = typeof scrap.cronTimezone === 'string' && scrap.cronTimezone ? scrap.cronTimezone : 'UTC';
|
|
48
|
+
return `${describeCron(scrap.cron)} ${tz} (cron ${scrap.cron})`;
|
|
49
|
+
}
|
|
50
|
+
return 'scheduled daily by default';
|
|
51
|
+
}
|
|
52
|
+
export const create = new Command('create')
|
|
53
|
+
.description('Create a persistent, self-healing scrap from a URL + a goal (AI-generated)')
|
|
54
|
+
.argument('<url>', 'Target public URL (http/https)')
|
|
55
|
+
.requiredOption('--prompt <goal>', 'What to extract/scrape, in plain language')
|
|
56
|
+
.option('--no-autofix', 'Disable AI auto-fix on first-run failure (default: on)')
|
|
57
|
+
.option('--json', 'Output the raw API payload')
|
|
58
|
+
.action(async (rawUrl, opts) => {
|
|
59
|
+
// Fast, local usage-errors (exit 2) — never a round-trip to the server
|
|
60
|
+
// for something we can already tell is bad. Same fail-fast pattern the
|
|
61
|
+
// former `fetch` command used for its URL argument.
|
|
62
|
+
const url = requireUrl(rawUrl, 'url');
|
|
63
|
+
const goal = requireString(opts.prompt, '--prompt');
|
|
64
|
+
const body = {
|
|
65
|
+
url,
|
|
66
|
+
goal,
|
|
67
|
+
...(opts.autofix === false && { autoFix: false }),
|
|
68
|
+
};
|
|
69
|
+
// #91/#106-F1 long-run pattern — the wizard runs AI generation + scrap
|
|
70
|
+
// creation + a real FIRST run (+ autofix retries) entirely server-side,
|
|
71
|
+
// legitimately 30-250s+. The 30s DEFAULT_TIMEOUT_MS would abort it
|
|
72
|
+
// mid-flight and fabricate a NetworkError timeout for a request that was
|
|
73
|
+
// always going to succeed.
|
|
74
|
+
const call = () => api.post('/api/ai/wizard', body, { timeoutMs: LONG_RUN_TIMEOUT_MS });
|
|
75
|
+
// #106-F2 pattern carried over from `fetch` — under --json stdout must
|
|
76
|
+
// be provably pure: no spinner channel at all. Only the human path gets
|
|
77
|
+
// the ora progress indicator; --json calls the API directly.
|
|
78
|
+
let data;
|
|
79
|
+
if (opts.json) {
|
|
80
|
+
data = await call();
|
|
81
|
+
}
|
|
82
|
+
else {
|
|
83
|
+
try {
|
|
84
|
+
data = await oraPromise(call, {
|
|
85
|
+
text: `Creating a scrap from ${url}…`,
|
|
86
|
+
// No verdict symbol here (#106-F3) — ora's success only means "the
|
|
87
|
+
// HTTP call didn't throw", not "the first run succeeded". The real
|
|
88
|
+
// outcome renders below via the icon + the First run: line.
|
|
89
|
+
successText: 'Request complete',
|
|
90
|
+
});
|
|
91
|
+
}
|
|
92
|
+
catch (err) {
|
|
93
|
+
// #114-F1 — a client-side timeout (NetworkError, "timed out after
|
|
94
|
+
// …ms" per api.ts's safeFetch) does NOT mean the wizard failed
|
|
95
|
+
// server-side: the scrap creation + first run keep going on the
|
|
96
|
+
// server after the CLI gives up waiting, so the scrap may already
|
|
97
|
+
// exist (or land moments later). Warn BEFORE rethrowing so a retry
|
|
98
|
+
// isn't the first instinct — a blind retry creates a duplicate scrap
|
|
99
|
+
// and burns AI-generation quota a second time for the same goal.
|
|
100
|
+
// Only a NetworkError whose message identifies it as the timeout
|
|
101
|
+
// branch qualifies — a DNS/connection-refused NetworkError never
|
|
102
|
+
// reached the server at all, so there's nothing to warn about here.
|
|
103
|
+
if (err instanceof NetworkError && /timed out/i.test(err.message)) {
|
|
104
|
+
console.error(chalk.yellow('⚠ The request timed out client-side, but the scrap may STILL have been created server-side — run `trawl list` before retrying (a retry creates a DUPLICATE scrap + burns quota).'));
|
|
105
|
+
}
|
|
106
|
+
// Rethrow unchanged so index.ts's classifyError/exit-code taxonomy
|
|
107
|
+
// stays intact (this stays a NetworkError -> exit 5, same as before).
|
|
108
|
+
throw err;
|
|
109
|
+
}
|
|
110
|
+
}
|
|
111
|
+
if (opts.json) {
|
|
112
|
+
json(data);
|
|
113
|
+
}
|
|
114
|
+
else {
|
|
115
|
+
const scrapId = data.scrap?._id;
|
|
116
|
+
const title = data.scrap?.title ?? url;
|
|
117
|
+
const autoFixEnabled = opts.autofix !== false;
|
|
118
|
+
const schedule = scheduleLabel(data.scrap);
|
|
119
|
+
console.log(`${data.success ? chalk.green('✓') : chalk.red('✗')} ${chalk.bold(title)}`);
|
|
120
|
+
console.log(chalk.dim(` Scrap ID: `) + (scrapId ?? '—'));
|
|
121
|
+
if (data.historyId)
|
|
122
|
+
console.log(chalk.dim(` History ID: `) + data.historyId);
|
|
123
|
+
console.log(chalk.dim(` First run: `) + firstRunLabel(data, autoFixEnabled));
|
|
124
|
+
if (schedule)
|
|
125
|
+
console.log(chalk.dim(` Schedule: `) + schedule);
|
|
126
|
+
if (scrapId)
|
|
127
|
+
console.log(chalk.dim(` Next step: `) + `trawl get ${scrapId}`);
|
|
128
|
+
// #114-F3 — only claim a background retry is happening when a scrap
|
|
129
|
+
// actually exists to retry (never fabricate progress that isn't real);
|
|
130
|
+
// `firstRunLabel` above already covers the !scrap / autofix-disabled
|
|
131
|
+
// wording, this adds the actionable poll target on top.
|
|
132
|
+
if (!data.success && data.scrap && autoFixEnabled) {
|
|
133
|
+
const pollTarget = data.historyId
|
|
134
|
+
? `\`trawl data ${scrapId}\` or \`trawl run-info ${data.historyId}\``
|
|
135
|
+
: `\`trawl data ${scrapId}\``;
|
|
136
|
+
console.log(chalk.yellow(` Note: `) +
|
|
137
|
+
`Auto-fix is retrying in the background — do NOT re-run create; poll ${pollTarget}.`);
|
|
138
|
+
}
|
|
139
|
+
}
|
|
140
|
+
// Honest exit code alongside the honest payload — a --json caller gets
|
|
141
|
+
// the raw body regardless (never wrapped/altered), but a script checking
|
|
142
|
+
// the exit code alone must be able to tell "first run failed" from "ran
|
|
143
|
+
// fine" without parsing. Same contract the former `fetch` command used.
|
|
144
|
+
if (!data.success)
|
|
145
|
+
process.exitCode = 1;
|
|
146
|
+
});
|
|
@@ -1,4 +1,27 @@
|
|
|
1
1
|
import { Command } from 'commander';
|
|
2
|
+
/**
|
|
3
|
+
* #108 — surface reorg. `run`/`list`/`get`/`data`/`history`/`run-info`/
|
|
4
|
+
* `trigger` are promoted to top-level verbs (see index.ts's `createProgram`)
|
|
5
|
+
* alongside `create`/`whoami`/`ping` (#114 — `create` replaced `fetch` in
|
|
6
|
+
* that group). Each is built by an exported
|
|
7
|
+
* `attachXCommand(parent, attachOpts)` factory instead of a fixed
|
|
8
|
+
* `scraps.command(...)` chain, so it can be attached TWICE with a single
|
|
9
|
+
* source-of-truth definition: once to `program` (the new canonical
|
|
10
|
+
* top-level path) and once more, hidden, right back onto `scraps` — so every
|
|
11
|
+
* pre-#108 `trawl scraps <verb>` invocation keeps resolving unchanged
|
|
12
|
+
* (`{ hidden: true }` only affects help visibility, never resolution).
|
|
13
|
+
*
|
|
14
|
+
* Future-drift guard: ALL `.option()`/`.argument()`/`.action()` wiring for a
|
|
15
|
+
* promoted verb MUST live INSIDE its `attachXCommand` factory body, never
|
|
16
|
+
* bolted onto one of its two call sites (index.ts's `createProgram` for the
|
|
17
|
+
* top-level attach, this file's own `attachXCommand(scraps, { hidden: true
|
|
18
|
+
* })` call for the legacy one). That's the only thing keeping `trawl run
|
|
19
|
+
* <id>` and `trawl scraps run <id>` identical — a call-site-only tweak to
|
|
20
|
+
* one attachment would silently diverge the two.
|
|
21
|
+
*/
|
|
22
|
+
type AttachOptions = {
|
|
23
|
+
hidden?: boolean;
|
|
24
|
+
};
|
|
2
25
|
export declare const scraps: Command;
|
|
3
26
|
/** The top-of-history snapshot pollRunProgress needs to identify which run
|
|
4
27
|
* it's watching — see captureBeforeRunState.
|
|
@@ -115,4 +138,11 @@ export declare function pollRunProgress(id: string, before: BeforeRunState | und
|
|
|
115
138
|
timeoutMs?: number;
|
|
116
139
|
json?: boolean;
|
|
117
140
|
}): Promise<void>;
|
|
141
|
+
export declare function attachListCommand(parent: Command, attachOpts?: AttachOptions): Command;
|
|
142
|
+
export declare function attachGetCommand(parent: Command, attachOpts?: AttachOptions): Command;
|
|
143
|
+
export declare function attachRunCommand(parent: Command, attachOpts?: AttachOptions): Command;
|
|
144
|
+
export declare function attachDataCommand(parent: Command, attachOpts?: AttachOptions): Command;
|
|
145
|
+
export declare function attachHistoryCommand(parent: Command, attachOpts?: AttachOptions): Command;
|
|
146
|
+
export declare function attachRunInfoCommand(parent: Command, attachOpts?: AttachOptions): Command;
|
|
147
|
+
export declare function attachTriggerCommand(parent: Command, attachOpts?: AttachOptions): Command;
|
|
118
148
|
export {};
|