@trawlme/cli 1.19.1 → 1.21.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +50 -26
- package/dist/commands/fetch.d.ts +31 -0
- package/dist/commands/fetch.js +95 -0
- package/dist/commands/login.js +35 -11
- package/dist/commands/ping.d.ts +24 -0
- package/dist/commands/ping.js +29 -0
- package/dist/commands/scraps.d.ts +32 -0
- package/dist/commands/scraps.js +256 -79
- package/dist/commands/skills.js +52 -6
- package/dist/commands/telemetry.js +23 -3
- package/dist/commands/token.js +9 -3
- package/dist/commands/whoami.d.ts +27 -0
- package/dist/commands/whoami.js +24 -0
- package/dist/index.js +6 -0
- package/dist/lib/confirm.d.ts +70 -0
- package/dist/lib/confirm.js +79 -0
- package/dist/lib/errors.d.ts +13 -0
- package/dist/lib/errors.js +19 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -28,16 +28,28 @@ Custom API URL: `trawl login --url https://self-hosted.example.com`
|
|
|
28
28
|
|
|
29
29
|
## Commands
|
|
30
30
|
|
|
31
|
-
All commands accept a global `--debug` flag to show full error stack traces on failure.
|
|
31
|
+
All commands accept a global `--debug` flag to show full error stack traces on failure. Every command below supports `--json` (a raw payload / structured result on stdout, nothing else) and never blocks on an interactive prompt when `--json` is set or stdin/stdout isn't a real TTY — see [Non-interactive rule](#non-interactive-rule) and [Exit codes](#exit-codes). Two documented exceptions to "single payload": `scraps watch --json` is a live stream, so it emits NDJSON (one JSON object per line) instead; `scraps snapshot --json` only changes behavior on a never-run scrap (`{"status":"no_runs"}`) — when a run exists it still writes the raw HTML snapshot regardless of `--json` (there's no JSON-encoded form of an HTML page to emit).
|
|
32
32
|
|
|
33
33
|
### Auth
|
|
34
34
|
|
|
35
35
|
```
|
|
36
|
-
trawl login [--url <url>] [--token <jwt>] [--email <email>] [--password <pass>]
|
|
37
|
-
trawl logout
|
|
38
|
-
trawl token
|
|
36
|
+
trawl login [--url <url>] [--token <jwt>] [--email <email>] [--password <pass>] [--json]
|
|
37
|
+
trawl logout [--json]
|
|
38
|
+
trawl token [--json] Print the stored session JWT (for MCP Bearer auth)
|
|
39
39
|
```
|
|
40
40
|
|
|
41
|
+
`login --json` prints `{"ok":true,"apiUrl","config",email?}` on success — never the raw token (that's `token`'s job). `token --json` prints `{"token","exp","expiresAt"}` instead of the bare JWT + stderr advisories. See [Non-interactive rule](#non-interactive-rule) below: a missing `--email`/`--password` refuses immediately (usage error, exit 2) instead of prompting when `--json` is set or stdin/stdout isn't a real TTY.
|
|
42
|
+
|
|
43
|
+
### Agent core verbs
|
|
44
|
+
|
|
45
|
+
```
|
|
46
|
+
trawl fetch <url> [--json] [--reason <text>] One-shot fetch + extract readable content from a public URL (no scrap needed)
|
|
47
|
+
trawl whoami [--json] Show the authenticated user's identity
|
|
48
|
+
trawl ping [--json] Health/version handshake against the Trawl API
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
Fully non-interactive — all three read auth only from `TRAWL_TOKEN`/the stored login token, never prompt. `trawl fetch` is the REST counterpart of the MCP `trawl_fetch_url` tool (same shared engine, `POST /api/scraps/fetch-url`): `status` is an honest outcome (`completed`/`failed`/`empty`/`blocked`), not "did the HTTP call succeed" — a failed fetch is still a 200 response with `status:'failed'` + `error`, and the CLI exits 1 in that case (both human and `--json` modes) even though `--json` always prints the raw payload verbatim. `truncated:true` means the 100KB response cap tripped; `result` is dropped and `url` repoints at the full history row instead. `trawl whoami`/`trawl ping` mirror the MCP `trawl_whoami`/`trawl_health_ping` tools as closely as the REST surface allows (`GET /api/users/me` / `GET /api/health`) — `ping`'s `--json` payload is admin-enriched (version/uptime/db) and just `{"status":"ok"}` for anyone else.
|
|
52
|
+
|
|
41
53
|
### Scraps
|
|
42
54
|
|
|
43
55
|
```
|
|
@@ -45,20 +57,20 @@ trawl scraps list [--json] [--status <success|failure|never|running|regression>]
|
|
|
45
57
|
trawl scraps get <id> [--json]
|
|
46
58
|
trawl scraps create -t <title> [-u <url>] [-r <request>] [-d <description>] [--tier <tier0|tier1|tier2|tier3|tier4>] [--json]
|
|
47
59
|
trawl scraps update <id> [-t <title>] [-u <url>] [-r <request>] [-d <description>] [--cron <expr>|--no-cron] [--alert <email>|--no-alert] [--autofix|--no-autofix] [-p <json>|--params-file <path>] [--tier <tier0|tier1|tier2|tier3|tier4>] [--force-tier <tier0|tier1|tier2|tier3|tier4>] [--json]
|
|
48
|
-
trawl scraps run <id> [--watch]
|
|
49
|
-
trawl scraps trigger <id> [--watch] [--wait]
|
|
50
|
-
trawl scraps watch <id>
|
|
60
|
+
trawl scraps run <id> [--watch] [--json]
|
|
61
|
+
trawl scraps trigger <id> [--watch] [--wait] [--json]
|
|
62
|
+
trawl scraps watch <id> [--json]
|
|
51
63
|
trawl scraps data <id> [--json] [--fresh] [--errors]
|
|
52
64
|
trawl scraps history <id> [--json] [-n <limit>]
|
|
53
65
|
trawl scraps run-info <hid> [--json]
|
|
54
66
|
trawl scraps doctor <id> [--json] [--autofix]
|
|
55
67
|
trawl scraps autofix <id> [--json]
|
|
56
68
|
trawl scraps snapshot <id> [--error] [-o <file>] [--json]
|
|
57
|
-
trawl scraps banner <id> -f <file>
|
|
58
|
-
trawl scraps delete <id> [--force] Alias: rm
|
|
69
|
+
trawl scraps banner <id> -f <file> [--json]
|
|
70
|
+
trawl scraps delete <id> [--force] [--json] Alias: rm
|
|
59
71
|
```
|
|
60
72
|
|
|
61
|
-
- `--tier` forces a proxy tier; `--force-tier` raises the proxy-tier ceiling past the auto-cap (history-gated: may be refused or cost more). `create --json`/`update --json` print the full scrap object (including the `_tierOverride` outcome) on stdout; a refused tier override exits 1 with a standard `--json` error envelope. When a tier was requested but the server's response carries no `_tierOverride` at all (an older server that can't confirm what actually got applied), a stderr warning is printed either way, and under `--json` the emitted object also carries `"_tierUnconfirmed": true` — the machine-readable counterpart to that warning, since a `--json` caller has no reliable reason to read stderr.
|
|
73
|
+
- `--tier` forces a proxy tier; `--force-tier` raises the proxy-tier ceiling past the auto-cap (history-gated: may be refused or cost more). `create --json`/`update --json` print the full scrap object (including the `_tierOverride` outcome) on stdout; a refused tier override exits 1 with a standard `--json` error envelope (`kind:"refused"` — distinct from `"unknown"`, so a script can branch on "the server said no"). When a tier was requested but the server's response carries no `_tierOverride` at all (an older server that can't confirm what actually got applied), a stderr warning is printed either way, and under `--json` the emitted object also carries `"_tierUnconfirmed": true` — the machine-readable counterpart to that warning, since a `--json` caller has no reliable reason to read stderr.
|
|
62
74
|
- `scraps doctor` diagnoses the last run (error, failed selector, block status, page state, autofix outcome); `--autofix` includes the full autofix diff/dry-run/knowledge. A run that is still in flight (`status: null`, server-side) shows a `running` badge, never `failed`; a run whose item count regressed vs baseline (`statusDetail: "regression"`) shows its own amber `regression` badge, never `failed` either (`scraps list`/`get` show the same distinction — a `▼` icon, not the red `✗` a genuine failure gets).
|
|
63
75
|
- `scraps autofix` shows the last auto-fix attempt on its own (decision, diff, dry-run, knowledge). `--json` on a scrap that has **never run** returns `{"status":"no_runs"}` (exit 0) — distinct from `null`, which means a run exists but had no auto-fix attempt.
|
|
64
76
|
- `scraps snapshot --error` fetches the error-path snapshot instead of the normal one; `-o <file>` writes to a file instead of stdout. On a scrap that has **never run**: `--json` returns `{"status":"no_runs"}` (exit 0), matching `doctor`/`autofix`; `-o <file>` (without `--json`) exits 4 (`not_found`) instead of silently exiting 0 with nothing written — a script checking the exit code alone must be able to tell "no file was produced" from success. `--json` takes priority when both are passed.
|
|
@@ -67,31 +79,34 @@ trawl scraps delete <id> [--force] Alias: rm
|
|
|
67
79
|
- `scraps get` (and anything reading through it, like `scraps data`'s default path) embeds only the newest 100 history rows on the returned scrap object — `scraps run-info` and `scraps doctor` fetch a single run directly and are unaffected by that cap.
|
|
68
80
|
- `scraps banner -f <file>` only accepts `png`/`jpg`/`jpeg`/`webp`; any other extension is a usage error (exit 2) instead of silently uploading the file under a fabricated `image/png` Content-Type.
|
|
69
81
|
- **Long-running scrap execute (`scraps run`, `data --fresh`, `trigger --wait`):** these hit the same server-side scrap-execute path, which can legitimately take 30–250s (proxy tier escalation, AI-fix retries) — the CLI arms a 300s timeout for exactly these three call sites instead of the generic 30s default. `TRAWL_TIMEOUT` (see below) still overrides ALL requests, including these — set it if you need a tighter or looser ceiling than 300s for a long-running scrap.
|
|
70
|
-
- **`--watch` is poll-based, not a live stream:** the activities SSE endpoint has no backlog and, for the default async `trigger` (no `--wait`), runs in a separate cron-consumer pod whose events never reach the API pod holding the SSE connection — a naive "await the run, then open SSE" shows nothing. `scraps run --watch` and `scraps trigger --watch` instead poll `GET /api/scraps/:id` (terminal status) and the activities REST list until the run finishes, printing each new activity line as it appears
|
|
82
|
+
- **`--watch` is poll-based, not a live stream:** the activities SSE endpoint has no backlog and, for the default async `trigger` (no `--wait`), runs in a separate cron-consumer pod whose events never reach the API pod holding the SSE connection — a naive "await the run, then open SSE" shows nothing. `scraps run --watch` and `scraps trigger --watch` instead poll `GET /api/scraps/:id` (terminal status) and the activities REST list until the run finishes, printing each new activity line as it appears. The watched run's outcome drives the exit code too, in BOTH human and `--json` mode: a genuinely failed terminal run, a poll timeout, or a persistently unreachable API all exit non-zero — a clean successful run is the only exit `0`. A run that never reaches a terminal status within 300s prints an honest timeout notice pointing at `scraps doctor <id>` (human mode) — see the `--json` shape below. `scraps watch <id>` (the standalone command, no trigger) is unchanged — it still opens the live SSE stream directly.
|
|
83
|
+
- `scraps run --json`/`trigger --json` bypass the spinner and print the raw launch/trigger payload on stdout; combined with `--watch`, every intermediate progress line stays suppressed (stdout stays pure JSON) and, once the watch reaches its outcome, exactly ONE final NDJSON line is emitted: `{"runId","status"}` (the honest terminal status — `success`/`error`/`empty`/`regression`/…), `{"runId","status":"timeout"}` on a poll timeout, or `{"runId","status":"poll_error","error"}` if the API stays unreachable for several consecutive polls — `process.exitCode` is non-zero for all three except a genuine success. `scraps watch --json` emits one raw JSON object per activity line (NDJSON) instead of the formatted `[time] message` text — there's no single final payload to wait for on a live stream. `scraps delete`/`banner --json` print `{"deleted":true,"id"}` / the raw upload response.
|
|
71
84
|
|
|
72
85
|
### Scrap accounts
|
|
73
86
|
|
|
74
87
|
```
|
|
75
|
-
trawl scraps account set <id> [-u <username>] [-p <password>]
|
|
76
|
-
trawl scraps account delete <id> [--force]
|
|
77
|
-
trawl scraps account clear-session <id>
|
|
88
|
+
trawl scraps account set <id> [-u <username>] [-p <password>] [--json]
|
|
89
|
+
trawl scraps account delete <id> [--force] [--json]
|
|
90
|
+
trawl scraps account clear-session <id> [--json]
|
|
78
91
|
trawl scraps account status <id> [--json]
|
|
79
|
-
trawl scraps account session set <id> -c <file>
|
|
92
|
+
trawl scraps account session set <id> -c <file> [--json]
|
|
80
93
|
```
|
|
81
94
|
|
|
82
|
-
`account session set` uploads a Puppeteer cookie JSON array to bootstrap a logged-in session without storing credentials (BYO-cookies).
|
|
95
|
+
`account session set` uploads a Puppeteer cookie JSON array to bootstrap a logged-in session without storing credentials (BYO-cookies). A missing `-u/--username`/`-p/--password` on `account set` follows the same [non-interactive rule](#non-interactive-rule) as `login`.
|
|
83
96
|
|
|
84
97
|
### Claude Code skills
|
|
85
98
|
|
|
86
99
|
The CLI bundles 5 Claude Code skills that teach Claude how to use `trawl`. Once installed, Claude can manage scraps for you via prompts.
|
|
87
100
|
|
|
88
101
|
```
|
|
89
|
-
trawl skills list List bundled skills and install status
|
|
90
|
-
trawl skills install [<skill>] [--local] [--force] Install all (or one). Default: ~/.claude/skills/
|
|
91
|
-
trawl skills uninstall [<skill>] [--local] Remove
|
|
92
|
-
trawl skills update [<skill>] [--local] [--force] Reinstall (force sync with CLI version)
|
|
102
|
+
trawl skills list [--json] List bundled skills and install status
|
|
103
|
+
trawl skills install [<skill>] [--local] [--force] [--json] Install all (or one). Default: ~/.claude/skills/
|
|
104
|
+
trawl skills uninstall [<skill>] [--local] [--json] Remove
|
|
105
|
+
trawl skills update [<skill>] [--local] [--force] [--json] Reinstall (force sync with CLI version)
|
|
93
106
|
```
|
|
94
107
|
|
|
108
|
+
`--json` prints a structured result (`{"version","skills":[...]}` for `list`; `{"installed"/"uninstalled"/"updated":[{"name","scope","dest"}]}` for the others) instead of the `✓`-prefixed lines — the orphan-sweep/re-sync stderr lines are unaffected either way (stdout stays pure).
|
|
109
|
+
|
|
95
110
|
Skills auto-update when you upgrade the CLI — no need to re-install manually — but it is not silent: it prints an honest `trawl: re-synced skill "<name>" (<scope>) <old> → <new>` line to stderr whenever it rewrites a skill dir, so a rewrite is never invisible. If a bundled skill is renamed or dropped between CLI versions, the old install is swept too — a `trawl: removed orphaned skill "<name>" (<scope>) — no longer bundled with this CLI version` line to stderr, so a stale skill teaching outdated CLI usage never lingers silently. Only marker-owned dirs (installed by trawl itself) are ever touched by either line. Opt out with `TRAWL_SKILLS_SYNC=0`.
|
|
96
111
|
|
|
97
112
|
A pre-existing skill directory that trawl did not install itself (no `.version` marker) is never touched — `install`/`update` refuse to overwrite it and require `--force` to proceed. This applies to the CLI-upgrade auto-sync too (it silently skips marker-less dirs rather than refusing, since there is no interactive user to show a refusal to).
|
|
@@ -101,9 +116,9 @@ You can also install skills standalone (without the CLI): `npx @trawlme/skills i
|
|
|
101
116
|
### Telemetry
|
|
102
117
|
|
|
103
118
|
```
|
|
104
|
-
trawl telemetry on Enable usage telemetry (default)
|
|
105
|
-
trawl telemetry off Disable usage telemetry
|
|
106
|
-
trawl telemetry status Show current state, telemetry ID, and opt-out instructions
|
|
119
|
+
trawl telemetry on [--json] Enable usage telemetry (default)
|
|
120
|
+
trawl telemetry off [--json] Disable usage telemetry
|
|
121
|
+
trawl telemetry status [--json] Show current state, telemetry ID, and opt-out instructions
|
|
107
122
|
```
|
|
108
123
|
|
|
109
124
|
## Telemetry
|
|
@@ -139,6 +154,15 @@ TRAWL_TELEMETRY=0 trawl scraps list
|
|
|
139
154
|
|
|
140
155
|
**Why:** usage data helps us prioritise CLI features and catch silent errors before users report them.
|
|
141
156
|
|
|
157
|
+
## Non-interactive rule
|
|
158
|
+
|
|
159
|
+
Every command that would otherwise block on a `readline` prompt (a destructive-action `[y/N]` confirmation, or a required value like `login`'s email/password) follows the same rule: when `--json` is set, **or** stdin/stdout isn't a real TTY (a pipe, a CI runner, an agent driving this CLI as a subprocess), it **never prompts** — it reports a structured usage error (exit `2`) instead of hanging forever waiting for an answer that can't arrive.
|
|
160
|
+
|
|
161
|
+
- **Confirmations** (`scraps delete`/`rm`, `scraps account delete`): pass `-f`/`--force` to pre-confirm and skip the prompt entirely, interactive or not.
|
|
162
|
+
- **Required values with no flag equivalent for "skip"** (`login`'s email/password, `scraps account set`'s username/password): pass the flag (`-e`/`-p`, `-u`/`-p`) or set `TRAWL_TOKEN`, instead of relying on the prompt.
|
|
163
|
+
- The guard is a hard gate, not a timeout — an agent-spawned subprocess with an open-but-idle stdin (common when a harness doesn't explicitly close it) would otherwise hang indefinitely; this refuses immediately instead.
|
|
164
|
+
- **Behavior change:** a piped `y` (e.g. `echo y | trawl scraps delete <id>`) no longer confirms the deletion. Piped stdin isn't a real TTY, so this now hits the same non-interactive refusal as any other scripted invocation (exit `2`, fail-closed) instead of silently proceeding on whatever text happened to be piped in. Existing scripts that relied on `echo y | …` must pass `-f`/`--force` instead.
|
|
165
|
+
|
|
142
166
|
## Exit codes
|
|
143
167
|
|
|
144
168
|
Every command exits with one of these codes — scripts and agents driving the CLI unattended can branch on the exact failure kind instead of a uniform pass/fail:
|
|
@@ -146,13 +170,13 @@ Every command exits with one of these codes — scripts and agents driving the C
|
|
|
146
170
|
| Code | Meaning |
|
|
147
171
|
|------|--------------------------------------------------------------------------|
|
|
148
172
|
| `0` | Success |
|
|
149
|
-
| `1` | Unknown/generic error (an unmapped failure — API errors other than 401/404, an unhandled bug — or a business-logic refusal like `scraps data`'s `run_failed`/`in_progress` states) |
|
|
150
|
-
| `2` | Usage error (bad flag/value, invalid ID, missing required argument, unknown option/command) |
|
|
173
|
+
| `1` | Unknown/generic error (an unmapped failure — API errors other than 401/404, an unhandled bug — or a business-logic refusal like `scraps data`'s `run_failed`/`in_progress` states, or `trawl fetch`'s honest `status:'failed'`/`'blocked'` outcome). **Known overload:** `fetch`'s domain-level failure and an arbitrary unmapped bug both land on `1` — a script needs to read the `--json` payload's own `status`/`error` field (for `fetch`) to tell them apart; this is intentional (the REST contract's outcome states are a separate axis from the CLI's transport-error taxonomy) and documented here rather than "resolved" by inventing a new code that would only apply to one command. |
|
|
174
|
+
| `2` | Usage error (bad flag/value, invalid ID, missing required argument, unknown option/command — including the [non-interactive rule](#non-interactive-rule) refusing to prompt) |
|
|
151
175
|
| `3` | Auth error (not logged in, or the session token is expired/invalid — run `trawl login`) |
|
|
152
176
|
| `4` | Not found (no such resource, or — for `scraps data` — no persisted payload to read) |
|
|
153
177
|
| `5` | Network error (the API host is unreachable, DNS/connection/TLS failure, or the request timed out) |
|
|
154
178
|
|
|
155
|
-
Under `--json`, a failing command emits a single error envelope on stdout — `{"error":{"message","status?","kind"}}` — instead of prose; the human-readable line always goes to stderr, never stdout.
|
|
179
|
+
Under `--json`, a failing command emits a single error envelope on stdout — `{"error":{"message","status?","kind"}}` — instead of prose; the human-readable line always goes to stderr, never stdout. `kind` is the machine-readable discriminant (`"usage"`/`"auth"`/`"not_found"`/`"network"`/`"api"`/`"refused"`/`"unknown"`/…) — the short string a script should switch on (`"refused"` is a business-logic refusal the server explicitly reported back, e.g. `scraps create|update`'s tier-ceiling override rejection — distinct from `"unknown"`, which stays reserved for an unmapped bug); `status` is present only when a real HTTP response carried one (never fabricated for a local failure like an expired-locally JWT or a refused confirmation prompt).
|
|
156
180
|
|
|
157
181
|
## Environment variables
|
|
158
182
|
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
import { Command } from 'commander';
|
|
2
|
+
/**
|
|
3
|
+
* `POST /api/scraps/fetch-url` response contract (#1697, trawl_node —
|
|
4
|
+
* shipped, contract LOCKED). REST counterpart of the MCP `trawl_fetch_url`
|
|
5
|
+
* tool: both run through the same shared engine
|
|
6
|
+
* (scraps.fetchUrl.service.js#runEphemeralUrlFetch), so the outcome shape is
|
|
7
|
+
* identical regardless of transport.
|
|
8
|
+
*
|
|
9
|
+
* - `status`: 'completed' | 'failed' | 'empty' | 'blocked' — an HONEST
|
|
10
|
+
* outcome, not a bare "it ran". A failed/blocked/empty run is still a
|
|
11
|
+
* 200 response (this is not an HTTP error), so callers must branch on
|
|
12
|
+
* `status`, never assume 2xx means "got data".
|
|
13
|
+
* - `error` is present ONLY when `status === 'failed'`.
|
|
14
|
+
* - `truncated: true` when the 100KB response-payload cap tripped — the
|
|
15
|
+
* server drops `result` in that case and repoints `url` at the full
|
|
16
|
+
* history row (`/api/historys/:runId`) instead of echoing the fetched
|
|
17
|
+
* target.
|
|
18
|
+
*/
|
|
19
|
+
export interface FetchUrlResponse {
|
|
20
|
+
url: string;
|
|
21
|
+
status: 'completed' | 'failed' | 'empty' | 'blocked';
|
|
22
|
+
runId?: string | null;
|
|
23
|
+
statusDetail?: string | null;
|
|
24
|
+
blocked?: boolean;
|
|
25
|
+
length?: number | null;
|
|
26
|
+
result?: unknown;
|
|
27
|
+
truncated?: boolean;
|
|
28
|
+
reason?: string;
|
|
29
|
+
error?: string;
|
|
30
|
+
}
|
|
31
|
+
export declare const fetchUrl: Command;
|
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
import { Command } from 'commander';
|
|
2
|
+
import chalk from 'chalk';
|
|
3
|
+
import { oraPromise } from 'ora';
|
|
4
|
+
import { api, LONG_RUN_TIMEOUT_MS } from '../lib/api.js';
|
|
5
|
+
import { json } from '../lib/format.js';
|
|
6
|
+
import { requireUrl } from '../lib/validate.js';
|
|
7
|
+
function statusIcon(status) {
|
|
8
|
+
if (status === 'completed')
|
|
9
|
+
return chalk.green('✓');
|
|
10
|
+
if (status === 'failed')
|
|
11
|
+
return chalk.red('✗');
|
|
12
|
+
if (status === 'blocked')
|
|
13
|
+
return chalk.yellow('⚠');
|
|
14
|
+
return chalk.dim('•'); // empty
|
|
15
|
+
}
|
|
16
|
+
/** Best-effort human summary of `result` — same "count + first-item keys"
|
|
17
|
+
* shape scraps.ts's renderScrapItems uses, never a full dump (that's what
|
|
18
|
+
* --json is for). */
|
|
19
|
+
function renderResultSummary(result) {
|
|
20
|
+
if (!Array.isArray(result))
|
|
21
|
+
return;
|
|
22
|
+
console.log(chalk.dim(` Items: `) + result.length);
|
|
23
|
+
const first = result[0];
|
|
24
|
+
if (result.length > 0 && first && typeof first === 'object') {
|
|
25
|
+
console.log(chalk.dim(` First keys: `) + Object.keys(first).join(', '));
|
|
26
|
+
}
|
|
27
|
+
}
|
|
28
|
+
export const fetchUrl = new Command('fetch')
|
|
29
|
+
.description('One-shot fetch + extract readable content from a public URL (no scrap needed)')
|
|
30
|
+
.argument('<url>', 'Target public HTTPS URL')
|
|
31
|
+
.option('--json', 'Output the raw API payload')
|
|
32
|
+
.option('--reason <reason>', 'Audit-trail reason for this fetch (logged server-side, max 500 chars)')
|
|
33
|
+
.action(async (rawUrl, opts) => {
|
|
34
|
+
// Fast, local usage-error (exit 2) on an obviously malformed URL — never
|
|
35
|
+
// a round-trip to the server for something we can already tell is bad.
|
|
36
|
+
// The server still re-validates (SSRF guard) — this is a UX fast-path,
|
|
37
|
+
// not a security boundary.
|
|
38
|
+
const url = requireUrl(rawUrl, 'url');
|
|
39
|
+
const body = {
|
|
40
|
+
url,
|
|
41
|
+
...(opts.reason !== undefined && { reason: opts.reason }),
|
|
42
|
+
};
|
|
43
|
+
// #106 review F1 — this endpoint runs the FULL worker pipeline (browser
|
|
44
|
+
// `goto` networkidle2 + extraction + persistence), legitimately 30-250s
|
|
45
|
+
// server-side — same #91 P0 pattern as `scraps run` / `data --fresh` /
|
|
46
|
+
// `trigger --wait` (src/commands/scraps.ts). The 30s DEFAULT_TIMEOUT_MS
|
|
47
|
+
// was aborting it mid-flight and would have surfaced a fabricated
|
|
48
|
+
// NetworkError timeout for a request that was always going to succeed.
|
|
49
|
+
const call = () => api.post('/api/scraps/fetch-url', body, { timeoutMs: LONG_RUN_TIMEOUT_MS });
|
|
50
|
+
// #106 review F2 — under --json the stdout path must be provably pure:
|
|
51
|
+
// no spinner channel at all. Only the human path gets the ora progress
|
|
52
|
+
// indicator; --json calls the API directly.
|
|
53
|
+
const data = opts.json
|
|
54
|
+
? await call()
|
|
55
|
+
: await oraPromise(call, {
|
|
56
|
+
text: `Fetching ${url}…`,
|
|
57
|
+
// #106 review F3 — no successText verdict here. ora's success
|
|
58
|
+
// symbol only means "the HTTP call didn't throw", not "the fetch
|
|
59
|
+
// succeeded" — a failed/blocked domain `status` is still a 200
|
|
60
|
+
// response. The real outcome is rendered below via statusIcon +
|
|
61
|
+
// the Status: line; a green check here would contradict a
|
|
62
|
+
// red/yellow icon printed right after it.
|
|
63
|
+
successText: 'Request complete',
|
|
64
|
+
});
|
|
65
|
+
if (opts.json) {
|
|
66
|
+
json(data);
|
|
67
|
+
}
|
|
68
|
+
else {
|
|
69
|
+
console.log(`${statusIcon(data.status)} ${chalk.bold(data.url)}`);
|
|
70
|
+
console.log(chalk.dim(` Status: `) + data.status);
|
|
71
|
+
if (data.runId)
|
|
72
|
+
console.log(chalk.dim(` Run ID: `) + data.runId);
|
|
73
|
+
if (data.statusDetail)
|
|
74
|
+
console.log(chalk.dim(` Detail: `) + data.statusDetail);
|
|
75
|
+
if (data.length != null)
|
|
76
|
+
console.log(chalk.dim(` Length: `) + data.length);
|
|
77
|
+
if (data.truncated) {
|
|
78
|
+
console.log(chalk.yellow(` ⚠ Truncated (payload too large) — full result: ${data.url}`));
|
|
79
|
+
}
|
|
80
|
+
if (data.status === 'failed' && data.error) {
|
|
81
|
+
console.log(chalk.red(` Error: `) + data.error);
|
|
82
|
+
}
|
|
83
|
+
renderResultSummary(data.result);
|
|
84
|
+
}
|
|
85
|
+
// Honest exit code alongside the honest payload — a --json caller gets
|
|
86
|
+
// the raw body regardless (never wrapped/altered), but a script checking
|
|
87
|
+
// the exit code alone must be able to tell "no usable data" from "ran
|
|
88
|
+
// fine" without parsing. `blocked` (#106 review F4 — an antibot wall) is
|
|
89
|
+
// a failure to get data exactly like `failed`: an agent scripting
|
|
90
|
+
// `trawl fetch ... || handle` must see non-zero for either. `empty`
|
|
91
|
+
// stays 0 on purpose — the fetch genuinely ran to completion, there was
|
|
92
|
+
// just nothing extractable at that URL; that's not an error.
|
|
93
|
+
if (data.status === 'failed' || data.status === 'blocked')
|
|
94
|
+
process.exitCode = 1;
|
|
95
|
+
});
|
package/dist/commands/login.js
CHANGED
|
@@ -4,6 +4,8 @@ import config, { getApiUrl } from '../lib/config.js';
|
|
|
4
4
|
import { api } from '../lib/api.js';
|
|
5
5
|
import { requireFreshJwt, requireUrl } from '../lib/validate.js';
|
|
6
6
|
import { promptPassword } from '../lib/prompt.js';
|
|
7
|
+
import { requireInteractive } from '../lib/confirm.js';
|
|
8
|
+
import { json } from '../lib/format.js';
|
|
7
9
|
async function promptEmail() {
|
|
8
10
|
const { createInterface } = await import('readline');
|
|
9
11
|
const rl = createInterface({ input: process.stdin, output: process.stdout });
|
|
@@ -25,12 +27,26 @@ async function promptEmail() {
|
|
|
25
27
|
rl.close();
|
|
26
28
|
}
|
|
27
29
|
}
|
|
30
|
+
/** #107 — the shared success-reporting tail for all three login paths (env
|
|
31
|
+
* token, --token flag, email/password). Under --json, prints a single
|
|
32
|
+
* structured result on stdout instead of the chalk lines — never the raw
|
|
33
|
+
* token itself (that's what `trawl token` is for). */
|
|
34
|
+
function reportLoginSuccess(opts, email) {
|
|
35
|
+
if (opts.json) {
|
|
36
|
+
json({ ok: true, apiUrl: getApiUrl(), config: config.path, ...(email && { email }) });
|
|
37
|
+
return;
|
|
38
|
+
}
|
|
39
|
+
console.log(chalk.green(email ? `✓ Logged in as ${email}` : '✓ Logged in'));
|
|
40
|
+
console.log(chalk.dim(` API: ${getApiUrl()}`));
|
|
41
|
+
console.log(chalk.dim(` Config: ${config.path}`));
|
|
42
|
+
}
|
|
28
43
|
export const login = new Command('login')
|
|
29
44
|
.description('Authenticate with the Trawl API')
|
|
30
45
|
.option('-u, --url <url>', 'API base URL')
|
|
31
46
|
.option('-t, --token <token>', 'JWT token (for CI use only)')
|
|
32
47
|
.option('-e, --email <email>', 'Email address')
|
|
33
48
|
.option('-p, --password <password>', 'Password (CI only — visible in process list and shell history)')
|
|
49
|
+
.option('--json', 'Output as JSON')
|
|
34
50
|
.action(async (opts) => {
|
|
35
51
|
// Validate --url but do NOT persist it yet. A failed signin must not
|
|
36
52
|
// brick the config by pointing it at an unreachable/wrong host while the
|
|
@@ -47,23 +63,28 @@ export const login = new Command('login')
|
|
|
47
63
|
if (envToken) {
|
|
48
64
|
config.set('token', requireFreshJwt(envToken, 'TRAWL_TOKEN'));
|
|
49
65
|
persistUrlIfPending();
|
|
50
|
-
|
|
51
|
-
console.log(chalk.dim(` API: ${getApiUrl()}`));
|
|
52
|
-
console.log(chalk.dim(` Config: ${config.path}`));
|
|
66
|
+
reportLoginSuccess(opts);
|
|
53
67
|
return;
|
|
54
68
|
}
|
|
55
69
|
// --token flag: direct JWT (CI / retrocompat)
|
|
56
70
|
if (opts.token) {
|
|
57
71
|
config.set('token', requireFreshJwt(opts.token, '--token'));
|
|
58
72
|
persistUrlIfPending();
|
|
59
|
-
|
|
60
|
-
console.log(chalk.dim(` API: ${getApiUrl()}`));
|
|
61
|
-
console.log(chalk.dim(` Config: ${config.path}`));
|
|
73
|
+
reportLoginSuccess(opts);
|
|
62
74
|
return;
|
|
63
75
|
}
|
|
64
76
|
// Email/password flow
|
|
65
77
|
if (opts.password) {
|
|
66
|
-
console.
|
|
78
|
+
console.error(chalk.yellow('Warning: passing --password on the command line is insecure.'));
|
|
79
|
+
}
|
|
80
|
+
// #107 — never blocks on a readline prompt under --json or a non-TTY
|
|
81
|
+
// invocation (agent/CI subprocess); refuses with a clear, structured
|
|
82
|
+
// usage error instead, naming the flag (or TRAWL_TOKEN) to pass.
|
|
83
|
+
if (!opts.email) {
|
|
84
|
+
requireInteractive('Email is required — pass -e/--email, --token, or set TRAWL_TOKEN (refusing to block on a prompt, non-interactive).', { json: opts.json });
|
|
85
|
+
}
|
|
86
|
+
if (!opts.password) {
|
|
87
|
+
requireInteractive('Password is required — pass -p/--password, --token, or set TRAWL_TOKEN (refusing to block on a prompt, non-interactive).', { json: opts.json });
|
|
67
88
|
}
|
|
68
89
|
const email = opts.email ?? (await promptEmail());
|
|
69
90
|
const password = opts.password ?? (await promptPassword('Password: '));
|
|
@@ -82,13 +103,16 @@ export const login = new Command('login')
|
|
|
82
103
|
const token = requireFreshJwt(raw, 'token');
|
|
83
104
|
config.set('token', token);
|
|
84
105
|
persistUrlIfPending();
|
|
85
|
-
|
|
86
|
-
console.log(chalk.dim(` API: ${getApiUrl()}`));
|
|
87
|
-
console.log(chalk.dim(` Config: ${config.path}`));
|
|
106
|
+
reportLoginSuccess(opts, email);
|
|
88
107
|
});
|
|
89
108
|
export const logout = new Command('logout')
|
|
90
109
|
.description('Clear stored credentials')
|
|
91
|
-
.
|
|
110
|
+
.option('--json', 'Output as JSON')
|
|
111
|
+
.action((opts) => {
|
|
92
112
|
config.set('token', '');
|
|
113
|
+
if (opts.json) {
|
|
114
|
+
json({ ok: true });
|
|
115
|
+
return;
|
|
116
|
+
}
|
|
93
117
|
console.log(chalk.green('✓ Logged out'));
|
|
94
118
|
});
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
import { Command } from 'commander';
|
|
2
|
+
/**
|
|
3
|
+
* `GET /api/health` response shape (home.controller.js#health,
|
|
4
|
+
* home.service.js#getHealthStatus) — MCP `trawl_health_ping` parity
|
|
5
|
+
* (`{ok, version, uptime}`). The REST route enriches the payload with
|
|
6
|
+
* `version`/`uptime`/`db`/`memory` ONLY for an admin caller
|
|
7
|
+
* (home.controller.js's `isAdmin` gate); a non-admin JWT gets `{status:'ok'}`
|
|
8
|
+
* alone on a healthy server. A non-2xx response (degraded, `status:503`)
|
|
9
|
+
* throws an ApiError like any other failed request — but a 2xx response is
|
|
10
|
+
* NOT a guarantee that `status === 'ok'` (#106 review F5): a soft-degraded
|
|
11
|
+
* signal (e.g. a partial dependency outage) could still come back as a 200
|
|
12
|
+
* with a non-"ok" `status`. The HTTP call not throwing only means the
|
|
13
|
+
* transport succeeded, never that the reported health is good — so the
|
|
14
|
+
* human-mode `✓ OK` line is gated on the actual field, not assumed from a
|
|
15
|
+
* successful round-trip.
|
|
16
|
+
*/
|
|
17
|
+
export interface PingResponse {
|
|
18
|
+
status: string;
|
|
19
|
+
db?: string;
|
|
20
|
+
uptime?: number;
|
|
21
|
+
version?: string;
|
|
22
|
+
memory?: unknown;
|
|
23
|
+
}
|
|
24
|
+
export declare const ping: Command;
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
import { Command } from 'commander';
|
|
2
|
+
import chalk from 'chalk';
|
|
3
|
+
import { api } from '../lib/api.js';
|
|
4
|
+
import { json } from '../lib/format.js';
|
|
5
|
+
export const ping = new Command('ping')
|
|
6
|
+
.description('Health/version handshake against the Trawl API')
|
|
7
|
+
.option('--json', 'Output as JSON')
|
|
8
|
+
.action(async (opts) => {
|
|
9
|
+
const data = await api.get('/api/health');
|
|
10
|
+
// #106 review (kimi) — honest exit code: a soft-degraded 200 (status !== 'ok')
|
|
11
|
+
// must exit non-zero so `trawl ping || handle_degraded` doesn't treat a
|
|
12
|
+
// degraded API as healthy. Applies in BOTH --json and human modes, alongside
|
|
13
|
+
// the raw/decorated output below.
|
|
14
|
+
if (data.status !== 'ok')
|
|
15
|
+
process.exitCode = 1;
|
|
16
|
+
if (opts.json) {
|
|
17
|
+
json(data);
|
|
18
|
+
return;
|
|
19
|
+
}
|
|
20
|
+
const versionSuffix = data.version ? ` (v${data.version})` : '';
|
|
21
|
+
if (data.status === 'ok') {
|
|
22
|
+
console.log(chalk.green('✓ OK') + versionSuffix);
|
|
23
|
+
}
|
|
24
|
+
else {
|
|
25
|
+
// #106 review F5 — never green-light a degraded API just because the
|
|
26
|
+
// HTTP call didn't throw; show the real reported status instead.
|
|
27
|
+
console.log(chalk.yellow(`⚠ ${data.status}`) + versionSuffix);
|
|
28
|
+
}
|
|
29
|
+
});
|
|
@@ -17,6 +17,17 @@ interface BeforeRunState {
|
|
|
17
17
|
alreadyInFlight: boolean;
|
|
18
18
|
captured: boolean;
|
|
19
19
|
}
|
|
20
|
+
/** The machine-readable outcome `pollRunProgress` prints as the single final
|
|
21
|
+
* NDJSON line under `--json` (#107 review F1). `status` is the same honest
|
|
22
|
+
* terminal string the human-mode "Run finished: <status>" line already
|
|
23
|
+
* shows (`success`/`error`/`empty`/`regression`/…), or `'timeout'` /
|
|
24
|
+
* `'poll_error'` for the two non-terminal exits. */
|
|
25
|
+
export interface PollOutcome {
|
|
26
|
+
runId?: string;
|
|
27
|
+
status: string;
|
|
28
|
+
/** Present only for `status:'poll_error'` — the last poll failure's message. */
|
|
29
|
+
error?: string;
|
|
30
|
+
}
|
|
20
31
|
/**
|
|
21
32
|
* #91 P1 — replaces "await the run to completion, THEN open the activities
|
|
22
33
|
* SSE stream" (which showed NOTHING: the activities SSE
|
|
@@ -78,9 +89,30 @@ interface BeforeRunState {
|
|
|
78
89
|
* this function has, so it resolves to the same honest timeout rather than
|
|
79
90
|
* risk reporting a possibly-wrong outcome (never a lie, at worst a timeout
|
|
80
91
|
* telling the caller to check `doctor`).
|
|
92
|
+
*
|
|
93
|
+
* #107 review F1 — before this fix, `run|trigger --json --watch` was
|
|
94
|
+
* outcome-blind: `quiet` suppressed ALL output (including "Run finished:
|
|
95
|
+
* failure" and the timeout notice), a transient poll error was caught and
|
|
96
|
+
* silently retried FOREVER within the deadline, and the process always
|
|
97
|
+
* exited 0 after the poll loop regardless of what the watched run actually
|
|
98
|
+
* did — dead air, then a clean exit code, even for a failed or timed-out
|
|
99
|
+
* run. An agent scripting this CLI had no way to tell success from failure
|
|
100
|
+
* from "we gave up". Fixed by:
|
|
101
|
+
* - emitting exactly ONE final NDJSON line on stdout under `--json` once
|
|
102
|
+
* the watch reaches ANY of its three exits (terminal status, timeout, or
|
|
103
|
+
* a persistent poll error) — `{runId,status}` (+`error` for a poll
|
|
104
|
+
* error) — while every intermediate progress line stays suppressed
|
|
105
|
+
* (unchanged from before);
|
|
106
|
+
* - setting `process.exitCode` non-zero on a genuine run failure, a
|
|
107
|
+
* timeout, or a persistent poll error, and `0` on a real success — in
|
|
108
|
+
* BOTH `--json` and human `--watch` modes (human mode used to exit 0
|
|
109
|
+
* unconditionally, the same bug, just silent instead of dishonest);
|
|
110
|
+
* - giving up after `MAX_CONSECUTIVE_POLL_ERRORS` consecutive failed reads
|
|
111
|
+
* instead of retrying the same dead endpoint for the full 300s.
|
|
81
112
|
*/
|
|
82
113
|
export declare function pollRunProgress(id: string, before: BeforeRunState | undefined, opts?: {
|
|
83
114
|
intervalMs?: number;
|
|
84
115
|
timeoutMs?: number;
|
|
116
|
+
json?: boolean;
|
|
85
117
|
}): Promise<void>;
|
|
86
118
|
export {};
|