@trawlme/cli 3.7.5 → 3.8.1

package/README.md

@@ -29,11 +29,14 @@ trawl upgrade --check  # only report if an update is available (exit 1 if so)
 
 ## Authentication
 
-Three methods:
+Four methods:
 
 1. **Interactive** — `trawl login` (prompts for email and password)
 2. **Env var** — `TRAWL_TOKEN=<jwt> trawl list` (CI/CD, bypasses prompt)
 3. **Token flag** — `trawl login --token <jwt>` (CI/CD, direct JWT)
+4. **API key** — `TRAWL_API_KEY=trawl_xxx trawl list` (scoped, revocable one-by-one — the recommended credential for an agent driving this CLI; see [docs/agent-quickstart.md](docs/agent-quickstart.md))
+
+A `TRAWL_API_KEY` is a `trawl_*`-prefixed credential (create/revoke one in the Trawl dashboard) sent as `Authorization: Bearer` instead of the session `Cookie: TOKEN=` a JWT uses — `trawl` picks the right one automatically based on the credential's own shape, never a flag. `TRAWL_API_KEY` wins over `TRAWL_TOKEN` when both happen to be set. It is scoped server-side — in practice to a set of scraps, since the dashboard's key-create form sets only the scrap allow-list; narrowing which *actions* a key may perform needs an explicit `scopes` array at creation over the API, and a key without one has every action granted. It works on most of the Core tier — `create`/`list`/`get`/`data`/`history`/`run-info`/`run`/`trigger`/`ping`, including the `--watch` **polling flag** on `run`/`trigger` — plus, outside Core, `scraps account status`/`scraps doctor`/`scraps autofix` (all three only ever read routes trawl_node opened to keys). It does **not** work on `whoami`, `scraps update`/`delete`, `scraps account set`/`delete`/`clear-session`/`session set`, `scraps banner`, `scraps snapshot` (the scrap lookup it starts from is dual-auth, but the `html-snapshot` route it downloads from isn't), or the standalone SSE **command** `scraps watch` (do not conflate the two: `--watch` is a flag on `run`/`trigger` and works under a key; `scraps watch` is a separate command and is JWT-only) — those stay JWT-only and fail with a `kind:"auth"` envelope (exit `3`) pointing at `trawl login` under a key. This list mirrors trawl_node's route wiring as of this writing, not a frozen guarantee — for anything not named here, trust the real `--json` envelope over this paragraph. Never send both a key and a JWT on the same request — the CLI only ever attaches one.
 
 Custom API URL: `trawl login --url https://self-hosted.example.com`
 
@@ -56,15 +59,16 @@ trawl data <id> [--json] [--fresh] [--errors]  Get scrap data (last persisted ru
 trawl history <id> [--json] [-n <limit>]       List past runs for a scrap (newest first)
 trawl run-info <hid> [--json]                  Show details of a single run
 trawl whoami [--json]                          Show the authenticated user's identity
-trawl token [--json]                           Print the stored session JWT (for MCP Bearer auth)
+trawl token [--json]                           Print the stored credential — API key or session JWT (for MCP Bearer auth)
 trawl ping [--json]                            Health/version handshake against the Trawl API
+trawl spec [--json]                            Print a versioned, machine-readable description of the command tree
 ```
 
 (`trawl create` AI-generates and persists a new scrap from a URL + a goal — distinct from `trawl scraps create`, which is raw manual scrap entry.)
 
 > **No breaking change:** every verb above is also still reachable under its pre-reorg path, `trawl scraps <verb>` (e.g. `trawl scraps list`, `trawl scraps run <id>`) — kept as a hidden alias so scripts written before the surface reorg keep working. `trawl --help` only shows the top-level form above; `trawl scraps --help` only shows the remaining scrap-management commands below.
 
-`create`/`whoami`/`ping` are fully non-interactive — all three read auth only from `TRAWL_TOKEN`/the stored login token, never prompt. `trawl create` runs the AI wizard server-side (`POST /api/ai/wizard`): generate scrap code from `--prompt` via LLM, persist the scrap, trigger its FIRST run, and auto-fix on failure (default on — `--no-autofix` disables it, sending `autoFix:false`). On a successful first run (human mode) it prints a small **data sample** (item count + first-item fields + one truncated value) as proof of value — best-effort, silent if the sample can't be fetched — and points `Next step` at `trawl data <id>` (the data), with `trawl get <id>` as the secondary detail view. `--json` skips the sample fetch and prints the raw wizard payload verbatim. `success` is an honest outcome of that first run, not "did the HTTP call succeed" — a failed first run is still a 200 response (the scrap was still created; auto-fix, when enabled, retries in the background), and the CLI exits 1 in that case (both human and `--json` modes) even though `--json` always prints the raw payload verbatim. The call legitimately takes 30–250s+ server-side (AI generation + a real run), same long-run timeout as `run`/`data --fresh`/`trigger --wait` below. `trawl whoami`/`trawl ping` mirror the MCP `trawl_whoami`/`trawl_health_ping` tools as closely as the REST surface allows (`GET /api/users/me` / `GET /api/health`) — `ping`'s `--json` payload is admin-enriched (version/uptime/db) and just `{"status":"ok"}` for anyone else.
+`create`/`whoami`/`ping` are fully non-interactive — all three read auth only from `TRAWL_API_KEY`/`TRAWL_TOKEN`/the stored login token, never prompt (`whoami` is JWT-only — see [Authentication](#authentication) — so it still refuses honestly under a key instead of prompting). `trawl create` runs the AI wizard server-side (`POST /api/ai/wizard`): generate scrap code from `--prompt` via LLM, persist the scrap, trigger its FIRST run, and auto-fix on failure (default on — `--no-autofix` disables it, sending `autoFix:false`). On a successful first run (human mode) it prints a small **data sample** (item count + first-item fields + one truncated value) as proof of value — best-effort, silent if the sample can't be fetched — and points `Next step` at `trawl data <id>` (the data), with `trawl get <id>` as the secondary detail view. `--json` skips the sample fetch and prints the raw wizard payload verbatim. `success` is an honest outcome of that first run, not "did the HTTP call succeed" — a failed first run is still a 200 response (the scrap was still created; auto-fix, when enabled, retries in the background), and the CLI exits 1 in that case (both human and `--json` modes) even though `--json` always prints the raw payload verbatim. The call legitimately takes 30–250s+ server-side (AI generation + a real run), same long-run timeout as `run`/`data --fresh`/`trigger --wait` below. `trawl whoami`/`trawl ping` mirror the MCP `trawl_whoami`/`trawl_health_ping` tools as closely as the REST surface allows (`GET /api/users/me` / `GET /api/health`) — `ping`'s `--json` payload is admin-enriched (version/uptime/db) and just `{"status":"ok"}` for anyone else.
 
 > **`create` is NOT idempotent, and every wizard-created scrap runs on a DAILY cron by default.** A client-side timeout (exit `5`, a `NetworkError`) does not mean the wizard failed server-side — scrap creation + the first run keep going after the CLI gives up waiting, so the scrap may already exist. Run `trawl list` and look for a matching URL/title **before** retrying — a blind retry creates a DUPLICATE scrap and burns AI-generation quota a second time for the same goal. Separately, the scrap the wizard creates is scheduled to re-run every day at 07:00 UTC (`cron: "0 7 * * *"`, hardcoded server-side, unrelated to `--no-autofix`) — each of those recurring runs consumes execute quota like any other run. Review the generated scrap, then change or disable the schedule with `trawl scraps update <id> --cron <expr>` (or `--no-cron` to disable it). Because the call can legitimately run 250s+, also confirm `TRAWL_TIMEOUT` isn't set to something tighter than `create` needs — the env var always wins over `create`'s own 300s default (see [Environment variables](#environment-variables)), so a value set for another purpose (e.g. a tight CI smoke-test budget) silently clamps `create` too; unset it or raise it before running `create`.
 
@@ -74,6 +78,7 @@ trawl ping [--json]                            Health/version handshake against
 - `get` (and anything reading through it, like `data`'s default path) embeds only the newest 100 history rows on the returned scrap object — `run-info` and `scraps doctor` fetch a single run directly and are unaffected by that cap. `list`/`get` show a distinct amber `▼` "regression" badge, never the red `✗` a genuine failure gets (matches `scraps doctor`'s own badge).
 - **Long-running calls (`create`, `run`, `data --fresh`, `trigger --wait`):** these hit server-side paths that can legitimately take 30–250s+ (AI generation + scrap creation + a first run for `create`; proxy tier escalation + AI-fix retries for the other three) — the CLI arms a 300s timeout for exactly these four call sites instead of the generic 30s default. `TRAWL_TIMEOUT` (see below) still overrides ALL requests, including these — set it if you need a tighter or looser ceiling than 300s, but note a global override that tight also clamps `create`.
 - **`--watch` is poll-based, not a live stream:** the activities SSE endpoint has no backlog and, for the default async `trigger` (no `--wait`), runs in a separate cron-consumer pod whose events never reach the API pod holding the SSE connection — a naive "await the run, then open SSE" shows nothing. `run --watch` and `trigger --watch` instead poll `GET /api/scraps/:id` (terminal status) and the activities REST list until the run finishes, printing each new activity line as it appears. The watched run's outcome drives the exit code too, in BOTH human and `--json` mode: a genuinely failed terminal run, a poll timeout, or a persistently unreachable API all exit non-zero — a clean successful run is the only exit `0`. A run that never reaches a terminal status within 300s prints an honest timeout notice pointing at `scraps doctor <id>` (human mode) — see the `--json` shape below. `scraps watch <id>` (the standalone command, no trigger) is unchanged — it still opens the live SSE stream directly.
+- `spec --json` prints the CLI's own command tree — `{specVersion, cliVersion, commands[], exitCodes, errorKinds, kindExitCodes}` — DERIVED at runtime by walking the live commander tree (never a hand-maintained file, which would silently drift from reality). Each entry in `commands[]` carries its full path (e.g. `"scraps account session set"`), description, `hidden` (the legacy `scraps <verb>` aliases above), `leaf` (false for a pure namespace/group node like `scraps`/`skills`/`telemetry` — invoking one directly is a guaranteed-failing tool, not a real command), `aliases`, `arguments`, and `options`; `exitCodes`/`errorKinds`/`kindExitCodes` are read from the exact same source `classifyError` uses (see [Exit codes](#exit-codes)) — never a second copy. `kindExitCodes` is the inverse of `exitCodes`: `kind -> exitCode`, since exit code `1` alone is a shared bucket (`api`/`refused`/`unknown`/`in_progress`/`run_failed`/`upgrade_failed`) that `exitCodes`' flat label can't disambiguate. Without `--json`, `spec` prints one short human line (version + visible command count) pointing at `--json`.
 - `run --json`/`trigger --json` bypass the spinner and print the raw launch/trigger payload on stdout; combined with `--watch`, every intermediate progress line stays suppressed (stdout stays pure JSON) and, once the watch reaches its outcome, exactly ONE final NDJSON line is emitted: `{"runId","status"}` (the honest terminal status — `success`/`error`/`empty`/`regression`/…), `{"runId","status":"timeout"}` on a poll timeout, or `{"runId","status":"poll_error","error"}` if the API stays unreachable for several consecutive polls — `process.exitCode` is non-zero for all three except a genuine success. `scraps watch --json` emits one raw JSON object per activity line (NDJSON) instead of the formatted `[time] message` text — there's no single final payload to wait for on a live stream.
 
 ### Scrap management
@@ -135,7 +140,9 @@ trawl login [--url <url>] [--token <jwt>] [--email <email>] [--password <pass>]
 trawl logout [--json]
 ```
 
-`login --json` prints `{"ok":true,"apiUrl","config",email?}` on success — never the raw token (that's `trawl token`'s job, in the [Core commands](#core-commands-agent--human) section above — non-interactive/agent-consumable, unlike `login`/`logout`). `token --json` prints `{"token","exp","expiresAt"}` instead of the bare JWT + stderr advisories. See [Non-interactive rule](#non-interactive-rule) below: a missing `--email`/`--password` refuses immediately (usage error, exit 2) instead of prompting when `--json` is set or stdin/stdout isn't a real TTY.
+`login --json` prints `{"ok":true,"apiUrl","config",email?}` on success — never the raw token (that's `trawl token`'s job, in the [Core commands](#core-commands-agent--human) section above — non-interactive/agent-consumable, unlike `login`/`logout`). `token --json` prints `{"token","exp","expiresAt","mode"}` — `mode` is `"apiKey"` or `"jwt"`, so a machine caller can tell which credential it just got without re-deriving the `trawl_*` prefix itself; `exp`/`expiresAt` are `null` for an API key (it has no expiry to report). Human mode prints the credential on stdout plus a stderr advisory: an expiry countdown for a JWT, or a one-line note for an API key that it does not expire on its own. See [Non-interactive rule](#non-interactive-rule) below: a missing `--email`/`--password` refuses immediately (usage error, exit 2) instead of prompting when `--json` is set or stdin/stdout isn't a real TTY.
+
+If `TRAWL_API_KEY` or `TRAWL_TOKEN` is set in the environment, both `login` and `logout` print a stderr warning naming that variable — it outranks the config-stored token (see [Authentication](#authentication)) for every subsequent command regardless of what `login`/`logout` just did. On `logout` specifically this means access is **not actually revoked** while that variable stays set.
 
 ### Telemetry
 
@@ -204,16 +211,19 @@ Every command exits with one of these codes — scripts and agents driving the C
 | `0`  | Success                                                                 |
 | `1`  | Unknown/generic error (an unmapped failure — API errors other than 401/404, an unhandled bug — or a business-logic refusal like `data`'s `run_failed`/`in_progress` states, or `trawl create`'s honest `success:false` first-run outcome). **Known overload:** `create`'s domain-level failure and an arbitrary unmapped bug both land on `1` — a script needs to read the `--json` payload's own `success` field (for `create`) to tell them apart; this is intentional (the REST contract's outcome states are a separate axis from the CLI's transport-error taxonomy) and documented here rather than "resolved" by inventing a new code that would only apply to one command. |
 | `2`  | Usage error (bad flag/value, invalid ID, missing required argument, unknown option/command — including the [non-interactive rule](#non-interactive-rule) refusing to prompt) |
-| `3`  | Auth error (not logged in, or the session token is expired/invalid — run `trawl login`) |
+| `3`  | Auth error (not logged in, the session token is expired/invalid, or a JWT-only command was run under an API key — see [Authentication](#authentication)) — run `trawl login` |
 | `4`  | Not found (no such resource, or — for `data` — no persisted payload to read) |
 | `5`  | Network error (the API host is unreachable, DNS/connection/TLS failure, or the request timed out) |
 
-Under `--json`, a failing command emits a single error envelope on stdout — `{"error":{"message","status?","kind"}}` — instead of prose; the human-readable line always goes to stderr, never stdout. `kind` is the machine-readable discriminant (`"usage"`/`"auth"`/`"not_found"`/`"network"`/`"api"`/`"refused"`/`"unknown"`/…) — the short string a script should switch on (`"refused"` is a business-logic refusal the server explicitly reported back, e.g. `scraps create|update`'s tier-ceiling override rejection — distinct from `"unknown"`, which stays reserved for an unmapped bug); `status` is present only when a real HTTP response carried one (never fabricated for a local failure like an expired-locally JWT or a refused confirmation prompt).
+Under `--json`, a failing command emits a single error envelope on stdout — `{"error":{"message","status?","kind","retryable","next?"}}` — instead of prose; the human-readable line always goes to stderr, never stdout. `kind` **stays the primary discriminant** — the short string a script should switch on (`"usage"`/`"auth"`/`"not_found"`/`"network"`/`"api"`/`"refused"`/`"unknown"`/`"in_progress"`/`"run_failed"`/`"upgrade_failed"`; `"refused"` is a business-logic refusal the server explicitly reported back, e.g. `scraps create|update`'s tier-ceiling override rejection — distinct from `"unknown"`, which stays reserved for an unmapped bug); `status` is present only when a real HTTP response carried one (never fabricated for a local failure like an expired-locally JWT or a refused confirmation prompt).
+
+`retryable` and `next` are additive — a retry loop branches on `retryable`, not on parsing `message` (e.g. the pre-existing "retry shortly" text is English prose, not a machine contract). `retryable` answers one question: is retrying the SAME command, unchanged, worth it? A network failure is (`retryable:true`); a bad flag, an expired token, or a 404 isn't — retrying without changing anything just repeats the same failure. `next`, when present, lists commands worth running next, most useful first (e.g. `["trawl login --token <jwt>"]` on an auth failure — the one form of `login` that never blocks on an interactive prompt) — it is empty/absent whenever there is nothing honest to suggest, never filled in just to look helpful. Two cases where `kind` alone doesn't tell the whole retry story: a `429` response (`kind:"api"`) is retryable even though `"api"` isn't by default; `data`'s "run in progress" refusal is retryable with `next:["trawl data <id>"]` — it never suggests `--fresh`, which would just 429 against the run already holding the lock.
 
 ## Environment variables
 
 | Variable             | Description                                                       |
 |-----------------------|--------------------------------------------------------------------|
+| `TRAWL_API_KEY`       | Scoped `trawl_*` API key — takes precedence over `TRAWL_TOKEN`; recommended for an agent driving the CLI (see [Authentication](#authentication)) |
 | `TRAWL_TOKEN`         | JWT token — bypasses login prompt, useful for CI/CD               |
 | `TRAWL_API_URL`       | Override the API base URL for the session (takes precedence over `trawl login --url`) |
 | `TRAWL_TELEMETRY`     | Set to `0` to disable telemetry for the current session           |

package/dist/commands/login.js

@@ -1,6 +1,6 @@
 import { Command } from 'commander';
 import chalk from 'chalk';
-import config, { getApiUrl } from '../lib/config.js';
+import config, { getApiUrl, getLiveAuthEnvVar } from '../lib/config.js';
 import { api } from '../lib/api.js';
 import { requireFreshJwt, requireUrl } from '../lib/validate.js';
 import { promptPassword } from '../lib/prompt.js';
@@ -32,6 +32,14 @@ async function promptEmail() {
  * structured result on stdout instead of the chalk lines — never the raw
  * token itself (that's what `trawl token` is for). */
 function reportLoginSuccess(opts, email) {
+    // #169 review finding 3 — stderr, both under --json and plain output: this
+    // is a warning about what happens AFTER login succeeds, not part of either
+    // payload shape, so it must never be silently dropped just because --json
+    // is set.
+    const overrideVar = getLiveAuthEnvVar();
+    if (overrideVar) {
+        console.error(chalk.yellow(`⚠ ${overrideVar} is set in your environment — it overrides the token just stored here for every subsequent command until you unset it.`));
+    }
     if (opts.json) {
         json({ ok: true, apiUrl: getApiUrl(), config: config.path, ...(email && { email }) });
         return;
@@ -110,6 +118,16 @@ export const logout = new Command('logout')
     .option('--json', 'Output as JSON')
     .action((opts) => {
     config.set('token', '');
+    // #169 review finding 3 — this is the sharpest version of the gap: an
+    // operator runs `logout` SPECIFICALLY to kill access, and if
+    // TRAWL_API_KEY/TRAWL_TOKEN is set, access is NOT killed — every
+    // subsequent command keeps authenticating with the env credential this
+    // command cannot touch. The wording below must not read as a variant of
+    // "logged out"; it says plainly that access is still live.
+    const overrideVar = getLiveAuthEnvVar();
+    if (overrideVar) {
+        console.error(chalk.yellow(`⚠ ${overrideVar} is still set in your environment — access is NOT revoked. Every subsequent command will keep authenticating with it until you unset it (or revoke the key in the dashboard).`));
+    }
     if (opts.json) {
         json({ ok: true });
         return;

package/dist/commands/scraps.js

@@ -6,7 +6,7 @@ import { table, json, formatDate } from '../lib/format.js';
 import { parseServerJson } from '../lib/json.js';
 import { promptPassword } from '../lib/prompt.js';
 import { validateObjectId, requireUrl } from '../lib/validate.js';
-import { classifyError, reportError, UsageError, RefusalError } from '../lib/errors.js';
+import { classifyError, reportError, retryFieldsFor, UsageError, RefusalError } from '../lib/errors.js';
 import { confirmDestructive, isInteractive } from '../lib/confirm.js';
 import { formatDoctor, formatAutofix, fetchRunAndFix, pickRun, pickFix } from './doctor.js';
 import { renderPinch, pinchEnabled } from '../lib/pinch.js';
@@ -47,6 +47,58 @@ function tryValidateUrl(value, opts) {
         throw err;
     }
 }
+/**
+ * One-line rendering of an emptyContext blob for the run-info table. Two
+ * buckets are actionable and must stay distinct: selectors that matched zero
+ * nodes (the page changed) and selectors the browser rejected outright
+ * (`-1`, a bug in the scrap script). Folding the second into "all matched"
+ * would report a broken selector as healthy. `--json` still emits the full
+ * object, `page.topClasses` / `topAnchorPaths` included.
+ */
+function summarizeEmptyContext(ctx) {
+    if (!ctx || typeof ctx !== 'object')
+        return null;
+    const parts = [];
+    const selectors = ctx.selectors && typeof ctx.selectors === 'object' ? ctx.selectors : {};
+    const names = Object.keys(selectors);
+    // Selectors may themselves contain a comma (`div, span`), so quote each one
+    // — a raw join renders one grouping selector as if it were two.
+    const list = (ns) => `${ns.slice(0, 3).map((s) => `\`${s}\``).join(', ')}${ns.length > 3 ? ` (+${ns.length - 3})` : ''}`;
+    // Four disjoint, exhaustive buckets — every selector lands in exactly one, so
+    // no count is ever dropped and none is summarised as something it is not.
+    // `selectors` comes from a Mongoose Mixed field, so a count may be `null` or
+    // a string: that is `unreadable`, which is neither healthy nor zero-match and
+    // must be named as its own thing next to the others, not collapsed into them.
+    const isCount = (v) => typeof v === 'number' && Number.isFinite(v);
+    const zero = names.filter((s) => selectors[s] === 0);
+    const invalid = names.filter((s) => selectors[s] === -1);
+    const matched = names.filter((s) => isCount(selectors[s]) && selectors[s] > 0);
+    const unreadable = names.filter((s) => !zero.includes(s) && !invalid.includes(s) && !matched.includes(s));
+    const mixed = zero.length > 0 || invalid.length > 0 || unreadable.length > 0;
+    if (zero.length)
+        parts.push(`0-match: ${list(zero)}`);
+    if (invalid.length)
+        parts.push(`invalid: ${list(invalid)}`);
+    if (unreadable.length)
+        parts.push(`unreadable: ${list(unreadable)}`);
+    if (matched.length) {
+        // Named even in a mixed set: "3 of the selectors still work" is the
+        // difference between a page that moved and a page that vanished.
+        parts.push(mixed ? `${matched.length} matched` : `${matched.length} selector(s), all matched`);
+    }
+    // Same Mixed-field caution on the title: coerce before calling a string
+    // method (a numeric title would otherwise crash `run-info` on the very run
+    // it exists to explain), and test against null/undefined rather than
+    // truthiness so a falsy-but-real title like `0` still shows. Inner double
+    // quotes would close the wrapper early and corrupt the line.
+    const title = ctx.page?.title;
+    if (title !== undefined && title !== null && String(title) !== '') {
+        parts.push(`title="${String(title).replace(/"/g, "'")}"`);
+    }
+    if (typeof ctx.page?.totalAnchors === 'number')
+        parts.push(`${ctx.page.totalAnchors} anchors`);
+    return parts.length ? parts.join(' | ') : null;
+}
 function lastStatus(scrap) {
     const last = scrap.history?.[0];
     if (!last)
@@ -590,7 +642,17 @@ scraps
     .command('create')
     .description('Create a new scrap')
     .requiredOption('-t, --title <title>', 'Scrap title')
-    .option('-u, --url <url>', 'Target URL — the site the scrap targets, not necessarily the exact URL fetched at run time (a request script can compute that, e.g. a templated search query). Required.')
+    // #170 review F6 — promoted from a plain `.option()` to `.requiredOption()`:
+    // this is unconditionally required (no positional alternative exists on
+    // THIS command, unlike the top-level `trawl create <url>`, which accepts
+    // `[url]` OR `--url` — deliberately left untouched, that either/or needs a
+    // human product decision, tracked separately). Before this, the flag's
+    // description carried the only "Required." signal, in English prose — the
+    // exact thing docs/agent-quickstart.md tells an agent not to parse.
+    // Commander itself now enforces it, so `trawl spec --json`'s published
+    // `mandatory:true` is honest instead of a second, driftable copy of this
+    // same fact.
+    .requiredOption('-u, --url <url>', 'Target URL — the site the scrap targets, not necessarily the exact URL fetched at run time (a request script can compute that, e.g. a templated search query).')
     .option('-r, --request <request>', 'Request/query')
     .option('-d, --description <text>', 'Scrap description')
     .option('--tier <tier>', `Force proxy tier (${VALID_TIERS.join('|')})`)
@@ -856,10 +918,27 @@ function renderScrapItems(items, asJson) {
  * all "usage" (2) — never-run / aged-out-of-retention are not_found (4), a
  * failed last run is a business-logic failure (1).
  */
-function reportDataState(message, exitCode, kind, wantsJson) {
+/**
+ * #170 — `retryable`/`next` follow the SAME frozen `RETRY_POLICY` map
+ * `classifyError` reads (errors.ts), keyed by `kind` — this path never
+ * builds its own copy. `retryOverride` is for the one case here where the
+ * kind alone is genuinely wrong: `in_progress`'s caller passes
+ * `{retryable:true, next:['trawl data <id>']}` since retrying `data` (not
+ * `--fresh`, which would 429 against the lock already held) IS worth doing
+ * once the run finishes. Every other kind here (`not_found`, `run_failed`)
+ * isn't in `RETRY_POLICY` at all — `retryFieldsFor` is total, so those fall
+ * back to its conservative "not retryable, nothing to suggest" default.
+ */
+function reportDataState(message, exitCode, kind, wantsJson, retryOverride) {
     console.error(chalk.red(`✗ ${message}`));
     if (wantsJson) {
-        console.log(JSON.stringify({ error: { message, kind } }));
+        const { retryable, next } = retryOverride ?? retryFieldsFor(kind);
+        // #170 review F3 — typed as ErrorEnvelope so the compiler enforces
+        // `retryable` here too (this was one of two emitters a mutation test
+        // proved `tsc --noEmit` never actually guarded before this annotation —
+        // the untyped literal let `retryable` be omitted silently).
+        const envelope = { message, kind, retryable, ...(next ? { next } : {}) };
+        console.log(JSON.stringify({ error: envelope }));
     }
     process.exitCode = exitCode;
 }
@@ -975,7 +1054,10 @@ export function attachDataCommand(parent, attachOpts = {}) {
         // run already holds the server-side distributed lock, so --fresh would
         // just 429 against it.
         if (last.status === null) {
-            reportDataState(`Run in progress for ${id} — retry shortly.`, 1, 'in_progress', opts.json);
+            // #170 — retrying `trawl data <id>` (unchanged) once the run
+            // finishes IS worth it; `--fresh` never is (it would 429 against the
+            // lock this very run already holds) — never suggested here.
+            reportDataState(`Run in progress for ${id} — retry shortly.`, 1, 'in_progress', opts.json, { retryable: true, next: [`trawl data ${id}`] });
             return;
         }
         // #86 review — node persists status=false for a GENUINE zero-item run
@@ -1088,14 +1170,15 @@ export function attachRunInfoCommand(parent, attachOpts = {}) {
             blockType: h.blockType ?? null,
             errorMessage: h.errorSnapshot?.errorMessage ?? null,
             selector: h.errorSnapshot?.selector ?? null,
-            emptyContext: h.errorSnapshot?.emptyContext ?? null,
+            emptyContext: h.emptyContext ?? null,
             createdAt: h.createdAt ?? null,
         };
         if (opts.json) {
             json(info);
             return;
         }
-        table([info], ['hid', 'status', 'time', 'tier', 'failureKind', 'blockType', 'errorMessage', 'selector', 'emptyContext', 'createdAt']);
+        // The blob is an object; the table needs one line. --json keeps it whole.
+        table([{ ...info, emptyContext: summarizeEmptyContext(info.emptyContext) }], ['hid', 'status', 'time', 'tier', 'failureKind', 'blockType', 'errorMessage', 'selector', 'emptyContext', 'createdAt']);
     });
 }
 attachRunInfoCommand(scraps, { hidden: true });

package/dist/commands/spec.d.ts

@@ -0,0 +1,92 @@
+import { Command } from 'commander';
+/**
+ * `trawl spec --json` (#170) — a versioned, machine-readable description of
+ * the command tree, so an agent can learn the CLI's surface without parsing
+ * ~250 lines of README prose. Everything below is DERIVED at runtime from
+ * the live commander `Command` tree (`buildSpec`) — never a hand-maintained
+ * file, which would be a second source of truth that silently drifts. See
+ * `tests/contracts/` for the README-vs-spec cross-check that exists
+ * specifically to catch that drift.
+ */
+export interface CliSpecArgument {
+    name: string;
+    required: boolean;
+    variadic: boolean;
+}
+export interface CliSpecOption {
+    long: string;
+    short?: string;
+    description: string;
+    /**
+     * commander's `Option#required`/`Option#mandatory` are a false-friend pair
+     * that an earlier draft of this file collapsed into one `required` key —
+     * DON'T "simplify" it back. `Option#required` means "if this option is
+     * used, a value must follow" (true for any `<value>` flag, e.g. `--limit
+     * <n>`, whether or not the option itself is optional to pass). `mandatory`
+     * means "the user MUST pass this option at all" (only true for
+     * `.requiredOption()`). Naming the value-arity field `required` (as this
+     * did before) reads as "you must specify this option" — wrong for ~30 of
+     * the ~34 options that tripped it. See commander's option.js:15/19.
+     */
+    mandatory: boolean;
+    valueRequired: boolean;
+    negatable: boolean;
+    default?: unknown;
+}
+export interface CliSpecCommand {
+    /** Full path from the program root, e.g. "scraps account session set". */
+    name: string;
+    description: string;
+    hidden: boolean;
+    /**
+     * True when this node has no subcommands of its own — i.e. it is an
+     * executable leaf (`.action()` actually runs something), not a pure
+     * namespace/group node like `scraps`/`skills`/`telemetry`/`scraps account`.
+     * An agent that emits one tool per spec entry needs this to skip group
+     * nodes: invoking one directly (e.g. `trawl scraps`) exits non-zero with
+     * usage text on stderr and ZERO bytes on stdout — a guaranteed-failing
+     * tool, not a --json envelope. Derived the same way
+     * `tests/contracts/readme-commands.test.ts` independently re-derives it
+     * (its own `leafNames()` walk existed ONLY because this field didn't) —
+     * that test now consumes THIS field instead of re-walking the tree itself.
+     */
+    leaf: boolean;
+    aliases: string[];
+    arguments: CliSpecArgument[];
+    options: CliSpecOption[];
+}
+export interface CliSpec {
+    specVersion: 1;
+    cliVersion: string;
+    commands: CliSpecCommand[];
+    /** "0".."5" → a short kind label. Same constant `classifyError` reads
+     *  (errors.ts's EXIT_CODE_LABELS) — never a second copy. Code `1` is a
+     *  SHARED bucket (several `kind`s land there, see `kindExitCodes` below
+     *  for the honest reverse mapping) — this label names the generic/unmapped
+     *  case, not an exhaustive claim that `1` means only that. */
+    exitCodes: Record<string, string>;
+    /** Every `kind` a --json error envelope can carry — the classifyError
+     *  kinds PLUS the hand-built ones (`in_progress`/`run_failed`/
+     *  `upgrade_failed`). Same constant (errors.ts's ENVELOPE_KINDS) — never a
+     *  second copy, and never `ERROR_KINDS` (that one omits the hand-built
+     *  kinds by design — see its doc comment). */
+    errorKinds: string[];
+    /**
+     * #170 review F7 — the inverse of `exitCodes`: `kind -> exitCode`, one
+     * entry per `errorKinds` member. `exitCodes` alone can't tell an agent
+     * which of the several kinds sharing code `1` (`api`/`refused`/`unknown`/
+     * `in_progress`/`run_failed`/`upgrade_failed`) it actually got — this
+     * field answers that directly instead of requiring the reverse lookup to
+     * be reconstructed by hand. Same source constant as `errorKinds`
+     * (errors.ts's KIND_EXIT_CODES) — never a second copy.
+     */
+    kindExitCodes: Record<string, number>;
+}
+/**
+ * Build the full spec from a live, already-constructed program (e.g.
+ * `createProgram()`'s return value). Includes every node in the tree —
+ * hidden legacy aliases (`scraps list`, …) included, flagged via `hidden`,
+ * so a consumer that wants only the advertised surface can filter on it.
+ */
+export declare function buildSpec(program: Command): CliSpec;
+export declare const spec: Command;

package/dist/commands/spec.js

@@ -0,0 +1,83 @@
+import { Command, Help } from 'commander';
+import { json } from '../lib/format.js';
+import { EXIT_CODE_LABELS, ENVELOPE_KINDS, KIND_EXIT_CODES } from '../lib/errors.js';
+function buildOption(option) {
+    const entry = {
+        long: option.long ?? '',
+        description: option.description,
+        mandatory: option.mandatory,
+        valueRequired: option.required,
+        negatable: option.negate,
+    };
+    if (option.short)
+        entry.short = option.short;
+    if (option.defaultValue !== undefined)
+        entry.default = option.defaultValue;
+    return entry;
+}
+function buildCommandEntry(cmd, name, hidden) {
+    return {
+        name,
+        description: cmd.description(),
+        hidden,
+        leaf: cmd.commands.length === 0,
+        aliases: [...cmd.aliases()],
+        arguments: cmd.registeredArguments.map((arg) => ({
+            name: arg.name(),
+            required: arg.required,
+            variadic: arg.variadic,
+        })),
+        options: cmd.options.map(buildOption),
+    };
+}
+/**
+ * Walk `cmd.commands` recursively, collecting one entry per node under its
+ * FULL path name. `hidden` is read via commander's own
+ * `Help#visibleCommands()` — the same idiom this codebase already uses
+ * (scraps.test.ts's #108 describe block) — rather than reaching for the
+ * private, untyped `_hidden` field directly.
+ */
+function walk(cmd, prefix, out) {
+    const visible = new Set(new Help().visibleCommands(cmd));
+    for (const sub of cmd.commands) {
+        const name = prefix ? `${prefix} ${sub.name()}` : sub.name();
+        out.push(buildCommandEntry(sub, name, !visible.has(sub)));
+        walk(sub, name, out);
+    }
+}
+/**
+ * Build the full spec from a live, already-constructed program (e.g.
+ * `createProgram()`'s return value). Includes every node in the tree —
+ * hidden legacy aliases (`scraps list`, …) included, flagged via `hidden`,
+ * so a consumer that wants only the advertised surface can filter on it.
+ */
+export function buildSpec(program) {
+    const commands = [];
+    walk(program, '', commands);
+    return {
+        specVersion: 1,
+        cliVersion: program.version() ?? 'unknown',
+        commands,
+        exitCodes: { ...EXIT_CODE_LABELS },
+        errorKinds: [...ENVELOPE_KINDS],
+        kindExitCodes: { ...KIND_EXIT_CODES },
+    };
+}
+export const spec = new Command('spec')
+    .description('Print a versioned, machine-readable description of the command tree')
+    .option('--json', 'Output as JSON')
+    .action((opts) => {
+    // `spec` is registered as a direct child of the program root
+    // (index.ts's createProgram), so `.parent` IS that root by the time this
+    // action ever runs — commander sets it in `addCommand()`. Falling back
+    // to `spec` itself only matters for an isolated unit invocation (e.g. a
+    // test driving `spec.parseAsync()` directly, unattached to a program).
+    const root = spec.parent ?? spec;
+    const cliSpec = buildSpec(root);
+    if (opts.json) {
+        json(cliSpec);
+        return;
+    }
+    const visibleCount = cliSpec.commands.filter((c) => !c.hidden).length;
+    console.log(`trawl ${cliSpec.cliVersion} — ${visibleCount} commands. Use --json for the full machine-readable spec.`);
+});

package/dist/commands/token.js

@@ -1,17 +1,18 @@
 import { Command } from 'commander';
 import chalk from 'chalk';
-import { getToken } from '../lib/config.js';
+import { getToken, getAuthMode } from '../lib/config.js';
 import { AuthError, notLoggedInError } from '../lib/api.js';
 import { reportError } from '../lib/errors.js';
 import { decodeExp } from '../lib/jwt.js';
 import { json } from '../lib/format.js';
 export const token = new Command('token')
-    .description('Print the stored session JWT (for MCP Bearer auth)')
-    .option('--json', 'Output as JSON ({token, exp, expiresAt}) instead of the raw JWT')
+    .description('Print the stored credential — a scoped API key or a session JWT (for MCP Bearer auth)')
+    .option('--json', 'Output as JSON ({token, exp, expiresAt, mode}) instead of the raw credential')
     .action((opts) => {
-    // getToken() resolves TRAWL_TOKEN env first, then the stored config
-    // token (see config.ts:47-51) — matching every other token consumer in
-    // the CLI instead of reading the config store directly. (#86 finding 1)
+    // getToken() resolves TRAWL_API_KEY env first, then TRAWL_TOKEN env, then
+    // the stored config token (see config.ts) — matching every other token
+    // consumer in the CLI instead of reading the config store directly. (#86
+    // finding 1, #169)
     const stored = getToken();
     if (!stored) {
         // Auth-classified (ApiError 401 → exit 3, kind:"auth"), not a generic
@@ -20,6 +21,27 @@ export const token = new Command('token')
         process.exitCode = reportError(notLoggedInError(), { json: opts.json });
         return;
     }
+    // #169 review finding 2 — a scoped API key is not a JWT: it has no `exp`
+    // claim to decode (decodeExp() would return null for one anyway, since
+    // it isn't dot-segmented), and unlike a session JWT it does NOT expire
+    // on its own — it stays live until revoked in the dashboard. Branching
+    // on getAuthMode() explicitly (rather than silently relying on
+    // decodeExp()'s null-for-non-JWT behaviour) means the distinct advisory
+    // below is a deliberate case, not an accident of what decodeExp() happens
+    // to return — and it replaces the old "(could not decode expiry — verify
+    // the token manually)" fallback, which was written for a malformed JWT
+    // and, read against a key, wrongly implied something was wrong with it.
+    if (getAuthMode(stored) === 'apiKey') {
+        if (opts.json) {
+            json({ token: stored, exp: null, expiresAt: null, mode: 'apiKey' });
+            return;
+        }
+        // Print the raw key first (so it can be piped / copied) — same
+        // stdout-only-the-credential contract `$(trawl token)` relies on.
+        console.log(stored);
+        console.error(chalk.dim('  This is a scoped API key (trawl_*) — it does not expire on its own. Revoke it in the dashboard to end its access.'));
+        return;
+    }
     const exp = decodeExp(stored);
     const nowSeconds = Math.floor(Date.now() / 1000);
     if (exp !== null && exp < nowSeconds) {
@@ -30,7 +52,12 @@ export const token = new Command('token')
         return;
     }
     if (opts.json) {
-        json({ token: stored, exp, expiresAt: exp !== null ? new Date(exp * 1000).toISOString() : null });
+        json({
+            token: stored,
+            exp,
+            expiresAt: exp !== null ? new Date(exp * 1000).toISOString() : null,
+            mode: 'jwt',
+        });
         return;
     }
     // Print the raw token first (so it can be piped / copied)

package/dist/commands/upgrade.js

@@ -3,6 +3,7 @@ import chalk from 'chalk';
 import { execFile } from 'node:child_process';
 import { promisify } from 'node:util';
 import { json } from '../lib/format.js';
+import { retryFieldsFor } from '../lib/errors.js';
 import { PKG_NAME, currentVersion, fetchLatestVersion } from '../lib/version.js';
 const execFileP = promisify(execFile);
 export const upgrade = new Command('upgrade')
@@ -20,11 +21,16 @@ export const upgrade = new Command('upgrade')
         return;
     }
     if (opts.check) {
+        // #170 review F10 — set BEFORE the --json early return. Before this,
+        // `trawl upgrade --check` exited 1 but `trawl upgrade --check --json`
+        // (identical state) exited 0 — the exit code was set below the --json
+        // `return`, so a --json caller never saw it. README documents this gate
+        // as "exit 1 if so" with no --json carve-out; a CI step gating on the
+        // exit code alone silently never fired under --json.
+        process.exitCode = 1;
         if (opts.json)
             return json({ package: PKG_NAME, current, latest, upToDate: false, upgraded: false });
         console.log(`${chalk.yellow('↑')} Update available: ${chalk.bold(current)} → ${chalk.bold(latest)}. Run ${chalk.cyan('trawl upgrade')} to install.`);
-        // Honest exit code so a script can gate on "is an update available".
-        process.exitCode = 1;
         return;
     }
     // Install the latest globally. This shells out to the same npm the user
@@ -43,7 +49,25 @@ export const upgrade = new Command('upgrade')
                 ? 'permission denied on the global npm prefix — retry with sudo, or use a Node version manager.'
                 : (e.stderr?.trim() || e.message);
         if (opts.json) {
-            console.log(JSON.stringify({ error: { message: `upgrade failed: ${hint}`, kind: 'upgrade_failed' } }));
+            // `upgrade_failed` is intentionally NOT in errors.ts's RETRY_POLICY
+            // (that map is the exhaustive set of kinds classifyError can
+            // produce — ERROR_KINDS derives from its keys) — retryFieldsFor is
+            // total, so an unmapped kind still gets the honest conservative
+            // default (`retryable:false`) instead of silently omitting the
+            // field. It IS registered in ENVELOPE_KINDS, the superset
+            // `spec --json`'s errorKinds actually publishes — see that
+            // constant's doc comment.
+            //
+            // #170 review F3 — typed as ErrorEnvelope so the compiler enforces
+            // `retryable` here too (this was one of the two emitters a
+            // mutation-test proved `tsc --noEmit` never actually guarded before
+            // this annotation).
+            const envelope = {
+                message: `upgrade failed: ${hint}`,
+                kind: 'upgrade_failed',
+                ...retryFieldsFor('upgrade_failed'),
+            };
+            console.log(JSON.stringify({ error: envelope }));
         }
         else {
             console.error(chalk.red(`✗ Upgrade failed: ${hint}`));

package/dist/commands/whoami.js

@@ -1,11 +1,21 @@
 import { Command } from 'commander';
 import chalk from 'chalk';
-import { api } from '../lib/api.js';
+import { api, apiKeyUnsupportedError } from '../lib/api.js';
+import { getAuthMode } from '../lib/config.js';
 import { json } from '../lib/format.js';
 export const whoami = new Command('whoami')
     .description("Show the authenticated user's identity")
     .option('--json', 'Output as JSON')
     .action(async (opts) => {
+    // #169 — JWT-only route (see WhoamiResponse's own doc comment above for
+    // why it stays that way). Under a scoped API key, a real request here
+    // would 401 and surface the generic "Session expired or invalid" text —
+    // wrong twice over: this route never accepts keys at all, and no
+    // session ever "expired". Detect it client-side, before any HTTP call,
+    // and say so plainly instead.
+    if (getAuthMode() === 'apiKey') {
+        throw apiKeyUnsupportedError('trawl whoami');
+    }
     const data = await api.get('/api/users/me');
     if (opts.json) {
         json(data);