@trawlme/cli 3.8.2 → 3.9.0

package/README.md

@@ -36,7 +36,7 @@ Four methods:
 3. **Token flag** — `trawl login --token <jwt>` (CI/CD, direct JWT)
 4. **API key** — `TRAWL_API_KEY=trawl_xxx trawl list` (scoped, revocable one-by-one — the recommended credential for an agent driving this CLI; see [docs/agent-quickstart.md](docs/agent-quickstart.md))
 
-A `TRAWL_API_KEY` is a `trawl_*`-prefixed credential (create/revoke one in the Trawl dashboard) sent as `Authorization: Bearer` instead of the session `Cookie: TOKEN=` a JWT uses — `trawl` picks the right one automatically based on the credential's own shape, never a flag. `TRAWL_API_KEY` wins over `TRAWL_TOKEN` when both happen to be set. It is scoped server-side — in practice to a set of scraps, since the dashboard's key-create form sets only the scrap allow-list; narrowing which *actions* a key may perform needs an explicit `scopes` array at creation over the API, and a key without one has every action granted. It works on most of the Core tier — `create`/`list`/`get`/`data`/`history`/`run-info`/`run`/`trigger`/`ping`, including the `--watch` **polling flag** on `run`/`trigger` — plus, outside Core, `scraps account status`/`scraps doctor`/`scraps autofix` (all three only ever read routes trawl_node opened to keys). It does **not** work on `whoami`, `scraps update`/`delete`, `scraps account set`/`delete`/`clear-session`/`session set`, `scraps banner`, `scraps snapshot` (the scrap lookup it starts from is dual-auth, but the `html-snapshot` route it downloads from isn't), or the standalone SSE **command** `scraps watch` (do not conflate the two: `--watch` is a flag on `run`/`trigger` and works under a key; `scraps watch` is a separate command and is JWT-only) — those stay JWT-only and fail with a `kind:"auth"` envelope (exit `3`) pointing at `trawl login` under a key. This list mirrors trawl_node's route wiring as of this writing, not a frozen guarantee — for anything not named here, trust the real `--json` envelope over this paragraph. Never send both a key and a JWT on the same request — the CLI only ever attaches one.
+A `TRAWL_API_KEY` is a `trawl_*`-prefixed credential (create/revoke one in the Trawl dashboard) sent as `Authorization: Bearer` instead of the session `Cookie: TOKEN=` a JWT uses — `trawl` picks the right one automatically based on the credential's own shape, never a flag. `TRAWL_API_KEY` wins over `TRAWL_TOKEN` when both happen to be set. It is scoped server-side — in practice to a set of scraps, since the dashboard's key-create form sets only the scrap allow-list; narrowing which *actions* a key may perform needs an explicit `scopes` array at creation over the API, and a key without one has every action granted. It works on most of the Core tier — `create`/`list`/`get`/`data`/`history`/`run-info`/`run`/`trigger`/`ping`, including the `--watch` **polling flag** on `run`/`trigger` — plus, outside Core, `scraps account status`/`scraps doctor`/`scraps autofix` (all three only ever read routes trawl_node opened to keys). It does **not** work on `whoami`, `scraps update`/`delete`, `scraps account set`/`delete`/`clear-session`/`session set`, `scraps banner`, `scraps snapshot` (the scrap lookup it starts from is dual-auth, but the `html-snapshot` route it downloads from isn't), or the standalone SSE **command** `scraps watch` (do not conflate the two: `--watch` is a flag on `run`/`trigger` and works under a key; `scraps watch` is a separate command and is JWT-only) — those stay JWT-only and fail with a `kind:"auth"` envelope (exit `3`) pointing at `trawl login` under a key. This list mirrors trawl_node's route wiring as of this writing, not a frozen guarantee — for anything not named here, trust the real `--json` envelope over this paragraph. Never send both a key and a JWT on the same request — the CLI only ever attaches one. `list --unhealthy` (and the plain health badge on `list`/`get`) rides the same `GET /api/scraps`/`GET /api/scraps/:id` routes as the rest of the group — no separate auth path — so it works under a key exactly like plain `list`/`get`, and a scrap-scoped key still only ever sees its own allow-listed scraps through the filter.
 
 Custom API URL: `trawl login --url https://self-hosted.example.com`
 
@@ -51,7 +51,7 @@ All commands accept a global `--debug` flag to show full error stack traces on f
 ```
 trawl create <url> --prompt <goal> [--no-autofix] [--json]   # url also accepted as --url <url> (#116)
                                                               Create a persistent, self-healing scrap from a URL + a goal (AI-generated)
-trawl list|ls [--json] [--status <success|failure|never|running|regression>] [--limit <n>] [--page <n>]
+trawl list|ls [--json] [--status <success|failure|never|running|regression>] [--unhealthy] [--limit <n>] [--page <n>]
 trawl get <id> [--json]                        Get scrap details
 trawl run <id> [--watch] [--json]              Run a scrap (blocks until finished)
 trawl trigger <id> [--watch] [--wait] [--json] Launch a scrap as a background worker (returns immediately)
@@ -76,6 +76,7 @@ trawl spec [--json]                            Print a versioned, machine-readab
 - `history` lists past runs (newest first); `run-info <hid>` shows details of a single run from that history.
 - `data` returns the last persisted run payload (no execute quota); `--fresh` runs the scrap live instead (consumes execute quota); `--errors` shows the last run's error detail (`--json` on a never-run scrap returns `{"status":"no_runs"}`, exit 0, matching `scraps doctor --json`). `[]` on stdout means a genuine zero-item successful run — a scrap that has never run, whose last run failed, or whose payload aged out of retention returns a `--json` error envelope (exit 4/1/4 respectively) instead. Two more honest states: a run still **in flight** (`status: null` server-side) returns a `kind:"in_progress"` error envelope (exit 1, "retry shortly" — never suggests `--fresh`, which would just 429 against the run already holding the lock); a run whose item count **regressed** vs baseline (`statusDetail: "regression"`) still returns the real, non-empty items on stdout (exit 0) plus a stderr warning pointing at `scraps doctor <id>` — the data itself is genuine even though the run is flagged.
 - `get` (and anything reading through it, like `data`'s default path) embeds only the newest 100 history rows on the returned scrap object — `run-info` and `scraps doctor` fetch a single run directly and are unaffected by that cap. `list`/`get` show a distinct amber `▼` "regression" badge, never the red `✗` a genuine failure gets (matches `scraps doctor`'s own badge).
+- `list`/`get` also show a **health** badge next to the status badge. By default it reads the scrap's own `lastCronOutcome` field (always present, no extra cost): `⚠ cron paused` when the last scheduled tick was skipped for being unhealthy, `—` otherwise — a breadcrumb of the last tick, not a live read (never set on a no-cron scrap, can lag by one tick). `list --unhealthy` asks the API to filter to scraps whose *current* consecutive-failure streak is 3+ (trawl_node's own threshold) and annotates each with the exact `consecutiveFailedRuns`/`unhealthySince` — shown instead of the badge whenever present, and never re-derived client-side from history. `--json` carries whichever fields the request produced: `lastCronOutcome` always, `consecutiveFailedRuns`/`unhealthySince` only when `--unhealthy` was passed. **Old-server safety:** a server predating trawl_node#1952 silently ignores the unknown `--unhealthy` query param and returns everything, unfiltered — the CLI detects that (no returned item carries `consecutiveFailedRuns`) and prints a stderr warning instead of presenting the full list as "your unhealthy scraps".
 - **Long-running calls (`create`, `run`, `data --fresh`, `trigger --wait`):** these hit server-side paths that can legitimately take 30–250s+ (AI generation + scrap creation + a first run for `create`; proxy tier escalation + AI-fix retries for the other three) — the CLI arms a 300s timeout for exactly these four call sites instead of the generic 30s default. `TRAWL_TIMEOUT` (see below) still overrides ALL requests, including these — set it if you need a tighter or looser ceiling than 300s, but note a global override that tight also clamps `create`.
 - **`--watch` is poll-based, not a live stream:** the activities SSE endpoint has no backlog and, for the default async `trigger` (no `--wait`), runs in a separate cron-consumer pod whose events never reach the API pod holding the SSE connection — a naive "await the run, then open SSE" shows nothing. `run --watch` and `trigger --watch` instead poll `GET /api/scraps/:id` (terminal status) and the activities REST list until the run finishes, printing each new activity line as it appears. The watched run's outcome drives the exit code too, in BOTH human and `--json` mode: a genuinely failed terminal run, a poll timeout, or a persistently unreachable API all exit non-zero — a clean successful run is the only exit `0`. A run that never reaches a terminal status within 300s prints an honest timeout notice pointing at `scraps doctor <id>` (human mode) — see the `--json` shape below. `scraps watch <id>` (the standalone command, no trigger) is unchanged — it still opens the live SSE stream directly.
 - `spec --json` prints the CLI's own command tree — `{specVersion, cliVersion, commands[], exitCodes, errorKinds, kindExitCodes}` — DERIVED at runtime by walking the live commander tree (never a hand-maintained file, which would silently drift from reality). Each entry in `commands[]` carries its full path (e.g. `"scraps account session set"`), description, `hidden` (the legacy `scraps <verb>` aliases above), `leaf` (false for a pure namespace/group node like `scraps`/`skills`/`telemetry` — invoking one directly is a guaranteed-failing tool, not a real command), `aliases`, `arguments`, and `options`; `exitCodes`/`errorKinds`/`kindExitCodes` are read from the exact same source `classifyError` uses (see [Exit codes](#exit-codes)) — never a second copy. `kindExitCodes` is the inverse of `exitCodes`: `kind -> exitCode`, since exit code `1` alone is a shared bucket (`api`/`refused`/`unknown`/`in_progress`/`run_failed`/`upgrade_failed`) that `exitCodes`' flat label can't disambiguate. Without `--json`, `spec` prints one short human line (version + visible command count) pointing at `--json`.
@@ -96,6 +97,8 @@ trawl scraps delete <id> [--force] [--json]         Alias: rm
 
 - `-u/--url` is **the site the scrap targets**, not necessarily the exact URL fetched at run time — a scrap whose real target is computed inside its `-r/--request` script (a templated search query, an interpolated item id, e.g. `https://www.ebay.com/sch/i.html?_nkw=${query}`) should still supply a base/site URL. Every domain-level policy (proxy tier ceiling, provider routing, per-domain memoize) keys off this field, so a blank value silently disables all of it. `create` requires it — validated client-side (must parse as an `http`/`https` URL) so a missing or malformed value fails fast with a readable usage error (exit 2) instead of a raw server 400/422; `update` keeps it optional (omitting it is still a no-op, matching the API) but a value that **is** passed is validated the same way.
 - `--tier` forces a proxy tier; `--force-tier` raises the proxy-tier ceiling past the auto-cap (history-gated: may be refused or cost more). `create --json`/`update --json` print the full scrap object (including the `_tierOverride` outcome) on stdout; a refused tier override exits 1 with a standard `--json` error envelope (`kind:"refused"` — distinct from `"unknown"`, so a script can branch on "the server said no"). When a tier was requested but the server's response carries no `_tierOverride` at all (an older server that can't confirm what actually got applied), a stderr warning is printed either way, and under `--json` the emitted object also carries `"_tierUnconfirmed": true` — the machine-readable counterpart to that warning, since a `--json` caller has no reliable reason to read stderr.
+
+`list --unhealthy` has the same shape of guard: against a server that predates the filter, the command warns on stderr and, under `--json`, wraps the payload as `{ "scraps": [...], "_healthFilterUnconfirmed": true }` instead of emitting a bare array — so a script can tell an unfiltered list from a genuinely-unhealthy one. A confirming server returns the bare array unchanged.
 - `scraps doctor` diagnoses the last run (error, failed selector, block status, page state, autofix outcome); `--autofix` includes the full autofix diff/dry-run/knowledge. A run that is still in flight (`status: null`, server-side) shows a `running` badge, never `failed`; a run whose item count regressed vs baseline (`statusDetail: "regression"`) shows its own amber `regression` badge, never `failed` either.
 - `scraps autofix` shows the last auto-fix attempt on its own (decision, diff, dry-run, knowledge). `--json` on a scrap that has **never run** returns `{"status":"no_runs"}` (exit 0) — distinct from `null`, which means a run exists but had no auto-fix attempt.
 - `scraps snapshot --error` fetches the error-path snapshot instead of the normal one; `-o <file>` writes to a file instead of stdout. On a scrap that has **never run**: `--json` returns `{"status":"no_runs"}` (exit 0), matching `doctor`/`autofix`; `-o <file>` (without `--json`) exits 4 (`not_found`) instead of silently exiting 0 with nothing written — a script checking the exit code alone must be able to tell "no file was produced" from success. `--json` takes priority when both are passed.

package/dist/commands/scraps.js

@@ -136,6 +136,67 @@ function statusIcon(status) {
         return chalk.hex('#FFA500')('▼');
     return chalk.dim('—');
 }
+/**
+ * #179 — health badge, next to the existing status icon. Two distinct
+ * sources, deliberately never blended into one derived verdict:
+ *   1. `consecutiveFailedRuns`/`unhealthySince` — present ONLY when `--unhealthy`
+ *      asked the API to attach them (see attachListCommand). This is the
+ *      live, exact server verdict — shown whenever available, since it is
+ *      strictly more informative than the breadcrumb below.
+ *   2. `lastCronOutcome === 'skipped_unhealthy'` — always present (a normal
+ *      scrap field, no extra query cost), but a STALE breadcrumb of the last
+ *      scheduled tick, not a live read: never set on a no-cron scrap, lags
+ *      by up to one tick, and flips back to 'ok'/'error' once the recovery
+ *      probe fires. Labeled as what it is ("cron paused"), never upgraded to
+ *      "unhealthy" — that word is reserved for the live verdict above.
+ * Never re-derives either signal from `history[]` — both come straight off
+ * the server (#1952 in trawl_node).
+ */
+function healthBadge(scrap) {
+    if (typeof scrap.consecutiveFailedRuns === 'number' && scrap.consecutiveFailedRuns > 0) {
+        const n = scrap.consecutiveFailedRuns;
+        return chalk.yellow(`⚠ failing ${n} ${n === 1 ? 'run' : 'runs'} (since ${formatDate(scrap.unhealthySince)})`);
+    }
+    if (scrap.lastCronOutcome === 'skipped_unhealthy') {
+        return chalk.yellow('⚠ cron paused');
+    }
+    return chalk.dim('—');
+}
+/**
+ * #179 — old-server fallback for `list --unhealthy`, same shape as
+ * `warnIfUnconfirmedTier` above (#86 finding 4b): an older trawl_node that
+ * predates #1952 ignores the unknown `minConsecutiveFailures` query param
+ * entirely and returns the FULL unfiltered list — silently presenting every
+ * scrap as "unhealthy" is exactly the kind of unconfirmed-outcome lie that
+ * precedent exists to prevent. A confirming server attaches
+ * `consecutiveFailedRuns` to EVERY item once the param is set (the
+ * controller writes `s.consecutiveFailedRuns || 0` unconditionally in that
+ * branch — see trawl_node scraps.controller.js), so its total absence across
+ * a non-empty response is the honest tell. Stderr only (safe under --json —
+ * stdout purity is untouched), never blocks the (unreliable) results below.
+ */
+function healthFilterUnconfirmed(data, unhealthyWasRequested) {
+    if (!unhealthyWasRequested || data.length === 0)
+        return false;
+    return !data.some((s) => s.consecutiveFailedRuns !== undefined);
+}
+function warnIfUnconfirmedHealthFilter(data, unhealthyWasRequested) {
+    if (!healthFilterUnconfirmed(data, unhealthyWasRequested))
+        return;
+    console.error(chalk.yellow('⚠ Server did not confirm the --unhealthy filter (older server, predates trawl_node#1952) — results below are UNFILTERED, not just the unhealthy scraps.'));
+}
+/**
+ * #179 — the machine-readable counterpart to the stderr warning above, in the
+ * exact shape `withTierUnconfirmed` already uses for the tier case. A --json
+ * caller has no reliable reason to read stderr, so without this a script sees
+ * a full unfiltered list and cannot tell it is not the unhealthy set.
+ * Emitted ONLY when the filter went unconfirmed — never fabricated otherwise.
+ */
+function withHealthFilterUnconfirmed(rows, data, unhealthyWasRequested) {
+    if (!healthFilterUnconfirmed(data, unhealthyWasRequested))
+        return rows;
+    return { scraps: rows, _healthFilterUnconfirmed: true };
+}
 export const scraps = new Command('scraps').description('Manage scraps');
 // shared SSE streaming helper. `asJson` (#107) emits one raw JSON object per
 // line (NDJSON) on stdout instead of the human-formatted timestamped text —
@@ -426,6 +487,15 @@ export async function pollRunProgress(id, before, opts = {}) {
 // keeps `--status`'s allowed values and its error message in one place,
 // mirrored from lastStatus()'s own return type so the two can never drift.
 const VALID_LAST_STATUSES = ['success', 'failure', 'never', 'running', 'regression'];
+// #179 — mirrors `UNHEALTHY_STREAK_LENGTH` in trawl_node
+// modules/scraps/helpers/scrapCronHealth.js. `--unhealthy` is a boolean
+// convenience flag for that ONE product-defined threshold (also what the
+// server's own cron-skip gate and email use), not a general numeric filter —
+// the API's `minConsecutiveFailures` param accepts any threshold, but this
+// CLI only ever asks for the one value the rest of the product means by
+// "unhealthy". If trawl_node's threshold ever changes, update this constant
+// to match.
+const UNHEALTHY_STREAK_LENGTH = 3;
 // list — promoted to a top-level verb (#108, see the AttachOptions comment above)
 export function attachListCommand(parent, attachOpts = {}) {
     return parent
@@ -443,6 +513,14 @@ export function attachListCommand(parent, attachOpts = {}) {
         // (~line 660) and report the actual bad input in the error message.
         .option('--limit <n>', 'Show only the first N results')
         .option('--page <n>', 'Fetch a specific page only (50 per page, no auto-pagination)')
+        // #179 — backed by the server's `minConsecutiveFailures` filter
+        // (trawl_node#1952): restricts the result set server-side to scraps
+        // whose CURRENT consecutive-failure streak is >= UNHEALTHY_STREAK_LENGTH,
+        // and annotates each returned scrap with the exact `consecutiveFailedRuns`/
+        // `unhealthySince` the aggregation computed. Deliberately never derived
+        // client-side from `history[]` — that would drift from what the web
+        // dashboard shows (comes-io/trawl_vue#1299 reads the identical filter).
+        .option('--unhealthy', `Only show scraps failing their last ${UNHEALTHY_STREAK_LENGTH}+ runs (server-computed)`)
         .action(async (opts, cmd) => {
         // #149 item 3 — validate --status against its enum the same way
         // --tier already validates (usageError + return, checked first, before
@@ -473,19 +551,23 @@ export function attachListCommand(parent, attachOpts = {}) {
                 return;
             }
         }
+        // #179 — same param on every request this action can issue (both
+        // branches below), so a filtered result stays filtered across the
+        // fetch-all pagination loop, not just its first page.
+        const healthFilter = opts.unhealthy ? `&minConsecutiveFailures=${UNHEALTHY_STREAK_LENGTH}` : '';
         let data;
         try {
             data = await spin(async () => {
                 if (page !== undefined) {
                     // Single-page mode: explicit page requested, no loop
-                    return api.get(`/api/scraps?perPage=50&page=${page}`);
+                    return api.get(`/api/scraps?perPage=50&page=${page}${healthFilter}`);
                 }
                 // Fetch-all mode: paginate until a page returns < 200 items
                 const perPage = 200;
                 let result = [];
                 let pageNum = 1;
                 while (true) {
-                    const batch = await api.get(`/api/scraps?perPage=${perPage}&page=${pageNum}`);
+                    const batch = await api.get(`/api/scraps?perPage=${perPage}&page=${pageNum}${healthFilter}`);
                     result = result.concat(batch);
                     if (batch.length < perPage)
                         break;
@@ -519,21 +601,28 @@ export function attachListCommand(parent, attachOpts = {}) {
             process.exitCode = exitCode;
             return;
         }
+        // #179 — check the RAW response, before --status narrows it further,
+        // so a genuinely empty (all-healthy) result is never mistaken for an
+        // unconfirmed filter.
+        warnIfUnconfirmedHealthFilter(data, opts.unhealthy);
         if (opts.status)
             data = data.filter((s) => lastStatus(s) === opts.status);
         const totalMatched = data.length;
         const rows = limit !== undefined ? data.slice(0, limit) : data;
         if (opts.json)
-            return json(rows);
+            return json(withHealthFilterUnconfirmed(rows, data, opts.unhealthy));
         const tableRows = rows.map((s) => ({
             id: s._id,
             title: s.title || '(untitled)',
             cron: s.cron || '—',
             status: statusIcon(lastStatus(s)),
+            // #179 — next to the existing status badge, per the issue's own
+            // wording (see healthBadge() above for the two source signals).
+            health: healthBadge(s),
             'last run': lastRun(s),
             updated: formatDate(s.updatedAt),
         }));
-        table(tableRows, ['id', 'title', 'cron', 'status', 'last run', 'updated']);
+        table(tableRows, ['id', 'title', 'cron', 'status', 'health', 'last run', 'updated']);
         // Print footer when --limit truncates
         if (limit !== undefined && rows.length < totalMatched) {
             console.log(chalk.dim(`Showing ${rows.length} of ${totalMatched} — omit --limit to see all`));
@@ -556,6 +645,10 @@ export function attachGetCommand(parent, attachOpts = {}) {
         console.log(chalk.dim(`  ID:       `) + data._id);
         console.log(chalk.dim(`  Cron:     `) + (data.cron || '—'));
         console.log(chalk.dim(`  Status:   `) + statusIcon(lastStatus(data)));
+        // #179 — same badge `list` renders (see healthBadge() above); `get`
+        // never sends `minConsecutiveFailures`, so this always reads the
+        // free `lastCronOutcome` breadcrumb, never the live per-run count.
+        console.log(chalk.dim(`  Health:   `) + healthBadge(data));
         console.log(chalk.dim(`  Last run: `) + lastRun(data));
         console.log(chalk.dim(`  Updated:  `) + formatDate(data.updatedAt));
     });

package/docs/agent-quickstart.md

@@ -81,7 +81,7 @@ first-class on every one, and none of them ever blocks on a prompt (see
 ```
 trawl create <url> --prompt <goal> [--no-autofix] [--json]   # url also accepted as --url <url> (#116)
                                                               Create a persistent, self-healing scrap from a URL + a goal (AI-generated)
-trawl list|ls [--json] [--status <s>] [--limit <n>] [--page <n>]   List all scraps
+trawl list|ls [--json] [--status <s>] [--unhealthy] [--limit <n>] [--page <n>]   List all scraps
 trawl get <id> [--json]                        Get scrap details
 trawl run <id> [--watch] [--json]              Run a scrap (blocks until finished)
 trawl trigger <id> [--watch] [--wait] [--json] Launch a scrap as a background worker

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@trawlme/cli",
-  "version": "3.8.2",
+  "version": "3.9.0",
   "description": "Trawl CLI — manage scraps from the terminal",
   "type": "module",
   "bin": {