@trawlme/cli 3.5.0 → 3.6.0

package/README.md

@@ -6,7 +6,7 @@
 
 Command-line client for [Trawl](https://trawl.me) — manage your scraps from the terminal.
 
-> ⚠️ **Early stage.** APIs may change before `1.0`. Pin a version if you depend on this in CI.
+> 📌 **Versioned with semantic-release** — breaking CLI-surface changes always ship as a new major. Pin a major (`@trawlme/cli@^3`) if you depend on the CLI in CI.
 
 > 🤖 **Driving this CLI from an AI agent?** See [docs/agent-quickstart.md](docs/agent-quickstart.md) for the minimal surface (`create`, `--json`, `TRAWL_TOKEN`, exit codes) — this README is the full human/CI guide.
 
@@ -48,7 +48,7 @@ All commands accept a global `--debug` flag to show full error stack traces on f
 ```
 trawl create <url> --prompt <goal> [--no-autofix] [--json]   # url also accepted as --url <url> (#116)
                                                               Create a persistent, self-healing scrap from a URL + a goal (AI-generated)
-trawl run <id> [--watch] [--json]              Run a scrap
+trawl run <id> [--watch] [--json]              Run a scrap (blocks until finished)
 trawl list|ls [--json] [--status <success|failure|never|running|regression>] [--limit <n>] [--page <n>]
 trawl get <id> [--json]                        Get scrap details
 trawl data <id> [--json] [--fresh] [--errors]  Get scrap data (last persisted run — read-only, no quota; --fresh to launch one)
@@ -59,6 +59,8 @@ trawl whoami [--json]                          Show the authenticated user's ide
 trawl ping [--json]                            Health/version handshake against the Trawl API
 ```
 
+(`trawl create` AI-generates and persists a new scrap from a URL + a goal — distinct from `trawl scraps create`, which is raw manual scrap entry.)
+
 > **No breaking change:** every verb above is also still reachable under its pre-reorg path, `trawl scraps <verb>` (e.g. `trawl scraps list`, `trawl scraps run <id>`) — kept as a hidden alias so scripts written before the surface reorg keep working. `trawl --help` only shows the top-level form above; `trawl scraps --help` only shows the remaining scrap-management commands below.
 
 `create`/`whoami`/`ping` are fully non-interactive — all three read auth only from `TRAWL_TOKEN`/the stored login token, never prompt. `trawl create` runs the AI wizard server-side (`POST /api/ai/wizard`): generate scrap code from `--prompt` via LLM, persist the scrap, trigger its FIRST run, and auto-fix on failure (default on — `--no-autofix` disables it, sending `autoFix:false`). On a successful first run (human mode) it prints a small **data sample** (item count + first-item fields + one truncated value) as proof of value — best-effort, silent if the sample can't be fetched — and points `Next step` at `trawl data <id>` (the data), with `trawl get <id>` as the secondary detail view. `--json` skips the sample fetch and prints the raw wizard payload verbatim. `success` is an honest outcome of that first run, not "did the HTTP call succeed" — a failed first run is still a 200 response (the scrap was still created; auto-fix, when enabled, retries in the background), and the CLI exits 1 in that case (both human and `--json` modes) even though `--json` always prints the raw payload verbatim. The call legitimately takes 30–250s+ server-side (AI generation + a real run), same long-run timeout as `run`/`data --fresh`/`trigger --wait` below. `trawl whoami`/`trawl ping` mirror the MCP `trawl_whoami`/`trawl_health_ping` tools as closely as the REST surface allows (`GET /api/users/me` / `GET /api/health`) — `ping`'s `--json` payload is admin-enriched (version/uptime/db) and just `{"status":"ok"}` for anyone else.
@@ -243,6 +245,8 @@ This project uses `legacy-peer-deps` (see `.npmrc`) due to a transitive peer-dep
 
 Issues and PRs are welcome — please open an issue first for non-trivial changes so we can align on direction.
 
+Releases: a merge to `main` publishes to the npm **`next`** dist-tag (candidate channel — try it with `npm i -g @trawlme/cli@next`). The **`latest`** tag is promoted manually after QA, so a merge never ships to default installs directly.
+
 ## License
 
 MIT — see [LICENSE](./LICENSE).

package/dist/commands/create.js

@@ -88,7 +88,7 @@ async function printDataSample(historyId) {
     }
 }
 export const create = new Command('create')
-    .description('Create a persistent, self-healing scrap from a URL + a goal (AI-generated)')
+    .description('Create a persistent, self-healing scrap from a URL + a goal (AI-generated) — distinct from `trawl scraps create`, which is raw manual entry')
     .argument('[url]', 'Target public URL (http/https)')
     .requiredOption('--prompt <goal>', 'What to extract/scrape, in plain language (required)')
     .option('--url <url>', 'Target public URL — alias of the positional argument')

package/dist/commands/ping.js

@@ -6,7 +6,14 @@ export const ping = new Command('ping')
     .description('Health/version handshake against the Trawl API')
     .option('--json', 'Output as JSON')
     .action(async (opts) => {
-    const data = await api.get('/api/health');
+    // #148 — the server route is optionalAuth (public, enriched for an admin
+    // JWT — see the PingResponse doc above); `api.publicGet` mirrors that:
+    // it attaches a stored/env token when one happens to be available but
+    // never REQUIRES one, so `ping` works as a zero-config sanity check even
+    // with no `trawl login` ever run. `api.get` (the authenticated path)
+    // would throw notLoggedInError() locally before this ever reached the
+    // network, defeating the whole point of a pre-login handshake.
+    const data = await api.publicGet('/api/health');
     // #106 review (kimi) — honest exit code: a soft-degraded 200 (status !== 'ok')
     // must exit non-zero so `trawl ping || handle_degraded` doesn't treat a
     // degraded API as healthy. Applies in BOTH --json and human modes, alongside

package/dist/commands/scraps.js

@@ -9,6 +9,7 @@ import { classifyError, reportError, UsageError, RefusalError } from '../lib/err
 import { confirmDestructive, isInteractive } from '../lib/confirm.js';
 import { formatDoctor, formatAutofix, fetchRunAndFix, pickRun, pickFix } from './doctor.js';
 import { renderPinch, pinchEnabled } from '../lib/pinch.js';
+import { maybeShowReferralTip } from '../lib/tips.js';
 /**
  * Print a usage/validation error consistently: human text to stderr, or a
  * machine envelope on stdout under --json (never both — reportError is the
@@ -710,7 +711,7 @@ scraps
 export function attachRunCommand(parent, attachOpts = {}) {
     return parent
         .command('run <id>', attachOpts)
-        .description('Run a scrap')
+        .description('Run a scrap (blocks until finished)')
         .option('-w, --watch', 'Show progress after launching (polls — see `trawl trigger --watch`, #91)')
         .option('--json', 'Output the raw launch payload as JSON')
         .action(async (id, opts) => {
@@ -738,6 +739,16 @@ export function attachRunCommand(parent, attachOpts = {}) {
             // of what the watched run actually did.
             await pollRunProgress(id, beforeRun, { json: opts.json });
         }
+        // #151 — throttled post-run referral tip. Reached only when the
+        // launch call above didn't throw (a hard failure rejects it, unwinding
+        // straight to index.ts's central catch before this line ever runs)
+        // — and, when --watch polled to a terminal state, only when
+        // pollRunProgress didn't flag a genuine run failure/timeout/poll-error
+        // via a non-zero process.exitCode (#107 review F1). See lib/tips.ts
+        // for the throttle/TTY/json/opt-out gating itself.
+        const runFailed = typeof process.exitCode === 'number' && process.exitCode !== 0;
+        if (!runFailed)
+            maybeShowReferralTip({ json: opts.json });
     });
 }
 attachRunCommand(scraps, { hidden: true });

package/dist/index.js

@@ -293,7 +293,29 @@ export async function runCli(argv = process.argv) {
         }
     });
     try {
-        await program.parseAsync(argv);
+        // #148 — a bare `trawl` (no args at all) reads as "let me look around",
+        // not a failure: it should behave exactly like `trawl --help` (same
+        // text, stdout, exit 0). Left to commander's own default, a program with
+        // subcommands and no root action handler treats zero args as "probably
+        // missing subcommand" and calls `this.help({ error: true })` internally
+        // (see node_modules/commander/lib/command.js `_parseCommand`) — which
+        // writes the SAME help text to stderr and throws a CommanderError with
+        // exitCode 1 (caught below, bucketed into HELP_OR_VERSION_CODES since its
+        // code is 'commander.help', so only the exit code carries the mistake
+        // through — text already went to the wrong stream by then). Intercepting
+        // here, before parseAsync ever runs, avoids fighting that internal
+        // decision entirely and can't affect a genuine unknown top-level command
+        // (`trawl frobnicate`) — that path only runs when `isBareInvocation` is
+        // false, so it still reaches `this.unknownCommand()` unchanged (stderr,
+        // exit 2, code 'commander.unknownCommand' — never in
+        // HELP_OR_VERSION_CODES).
+        if (isBareInvocation(argv)) {
+            program.outputHelp();
+            process.exitCode = 0;
+        }
+        else {
+            await program.parseAsync(argv);
+        }
     }
     catch (err) {
         const { debug } = program.opts();

package/dist/lib/api.d.ts

@@ -60,6 +60,7 @@ export interface RequestOptions {
 }
 export declare const api: {
     get: <T>(path: string, opts?: RequestOptions) => Promise<T>;
+    publicGet: <T>(path: string, opts?: RequestOptions) => Promise<T>;
     getText: (path: string, opts?: RequestOptions) => Promise<string>;
     post: <T>(path: string, body?: unknown, opts?: RequestOptions) => Promise<T>;
     put: <T>(path: string, body?: unknown, opts?: RequestOptions) => Promise<T>;

package/dist/lib/api.js

@@ -314,6 +314,47 @@ async function publicPost(path, body, baseUrlOverride, reqOpts = {}) {
         throw new Error('Invalid JSON in server response');
     }
 }
+/**
+ * Unauthenticated GET — for endpoints the server itself treats as public/
+ * optional-auth (e.g. `GET /api/health`, home.route.js's `optionalAuth`
+ * middleware: enriches the response for an admin JWT, passes through
+ * otherwise). Unlike `request()`, this NEVER throws `notLoggedInError()` for
+ * a missing token — that guard exists for endpoints that genuinely require
+ * auth, and applying it here defeated the whole point of a pre-login
+ * sanity-check (#148). The token is still attached as a best-effort Cookie
+ * when one happens to be stored/env-set, so an already-authenticated caller
+ * still gets the admin-enriched payload — it's just never REQUIRED.
+ * `throwIfError(res, true)` mirrors `publicPost`'s "public endpoint" 401
+ * wording, even though this route's optionalAuth middleware never actually
+ * rejects a request for lacking/invalid credentials.
+ */
+async function publicGet(path, reqOpts = {}) {
+    const token = getToken();
+    const url = `${getApiUrl()}${path}`;
+    const timeoutMs = getTimeoutMs(reqOpts.timeoutMs);
+    const res = await safeFetch(url, {
+        headers: {
+            'User-Agent': USER_AGENT,
+            ...(token ? { Cookie: `TOKEN=${token}` } : {}),
+        },
+        signal: AbortSignal.timeout(timeoutMs),
+    }, timeoutMs);
+    await throwIfError(res, true);
+    const text = await res.text();
+    try {
+        if (!text)
+            return {};
+        const parsed = JSON.parse(text);
+        // Unwrap API envelope { type, message, data: T }
+        if (parsed !== null && typeof parsed === 'object' && 'data' in parsed) {
+            return parsed.data;
+        }
+        return parsed;
+    }
+    catch {
+        throw new Error('Invalid JSON in server response');
+    }
+}
 async function getText(path, reqOpts = {}) {
     const token = getToken();
     if (!token)
@@ -332,6 +373,7 @@ async function getText(path, reqOpts = {}) {
 }
 export const api = {
     get: (path, opts) => request(path, {}, opts),
+    publicGet: (path, opts) => publicGet(path, opts),
     getText: (path, opts) => getText(path, opts),
     post: (path, body, opts) => request(path, {
         method: 'POST',

package/dist/lib/config.d.ts

@@ -9,6 +9,10 @@ interface TrawlConfig {
      * only an explicit `false` disables it. No default entry needed since it's
      * optional. (#129) */
     updateNotifier?: boolean;
+    /** Opt-out switch for the throttled post-run referral tip (lib/tips.ts).
+     * Same optional/absent-means-enabled shape as `updateNotifier` above. */
+    tips?: boolean;
+    referralTipShownAt: number;
 }
 declare const config: Conf<TrawlConfig>;
 /**

package/dist/lib/config.js

@@ -16,6 +16,7 @@ const config = new Conf({
         token: '',
         telemetry: true,
         telemetryUserId: '',
+        referralTipShownAt: 0,
     },
 });
 /**

package/dist/lib/tips.d.ts

@@ -0,0 +1,30 @@
+/**
+ * Pure gate: true when the tip should print. Every external signal (json
+ * flag, TTY-ness, opt-out, persisted timestamp, current time) is a
+ * parameter rather than read internally, so this is trivially testable
+ * without mocking process.env/stdout/Date/config.
+ */
+export declare function isReferralTipDue(opts: {
+    json?: boolean;
+    isTTY: boolean;
+    optedOut: boolean;
+    lastShownAt: number;
+    now: number;
+}): boolean;
+/**
+ * Best-effort: print one subtle line inviting the user to refer a builder,
+ * throttled to once per 7 days via a timestamp persisted in the CLI's
+ * existing Conf-backed config store (lib/config.ts — same store as
+ * apiUrl/token/telemetry/updateNotifier).
+ *
+ * This only gates on throttle/TTY/`--json`/opt-out — it does NOT know
+ * whether the underlying command actually succeeded. Callers must only
+ * invoke it on a genuine success (see commands/scraps.ts `run <id>`'s
+ * action, which also checks pollRunProgress's exit code under `--watch`
+ * before calling this). Any state read/write failure (corrupt config file,
+ * disk error, …) is swallowed silently — a broken tip must never break a
+ * run.
+ */
+export declare function maybeShowReferralTip(opts?: {
+    json?: boolean;
+}): void;

package/dist/lib/tips.js

@@ -0,0 +1,69 @@
+/**
+ * Throttled post-run referral tip (#151) — after a successful `trawl run`,
+ * occasionally invite the user to refer a builder. Mirrors
+ * lib/updateNotifier.ts's shape: synchronous, instant, self-guarded so a
+ * broken tip can never turn a successful command into a failed one.
+ */
+import chalk from 'chalk';
+import config from './config.js';
+/** Max once per 7 days. */
+const TIP_THROTTLE_MS = 7 * 24 * 60 * 60 * 1000;
+const REFERRAL_TIP_TEXT = 'Invite a builder — they get 500, you get 1,000 compute → trawl.me → account → Referrals';
+/**
+ * Opted out via env (presence-based, mirrors NO_COLOR / the update
+ * notifier's TRAWL_NO_UPDATE_NOTIFIER — see lib/updateNotifier.ts) or an
+ * explicit `false` config value. Absent/undefined config key means enabled.
+ */
+function isOptedOut() {
+    if (process.env['TRAWL_NO_TIPS'] !== undefined)
+        return true;
+    return config.get('tips') === false;
+}
+/**
+ * Pure gate: true when the tip should print. Every external signal (json
+ * flag, TTY-ness, opt-out, persisted timestamp, current time) is a
+ * parameter rather than read internally, so this is trivially testable
+ * without mocking process.env/stdout/Date/config.
+ */
+export function isReferralTipDue(opts) {
+    if (opts.json)
+        return false;
+    if (!opts.isTTY)
+        return false;
+    if (opts.optedOut)
+        return false;
+    return opts.now - opts.lastShownAt >= TIP_THROTTLE_MS;
+}
+/**
+ * Best-effort: print one subtle line inviting the user to refer a builder,
+ * throttled to once per 7 days via a timestamp persisted in the CLI's
+ * existing Conf-backed config store (lib/config.ts — same store as
+ * apiUrl/token/telemetry/updateNotifier).
+ *
+ * This only gates on throttle/TTY/`--json`/opt-out — it does NOT know
+ * whether the underlying command actually succeeded. Callers must only
+ * invoke it on a genuine success (see commands/scraps.ts `run <id>`'s
+ * action, which also checks pollRunProgress's exit code under `--watch`
+ * before calling this). Any state read/write failure (corrupt config file,
+ * disk error, …) is swallowed silently — a broken tip must never break a
+ * run.
+ */
+export function maybeShowReferralTip(opts = {}) {
+    try {
+        const lastShownAt = config.get('referralTipShownAt') || 0;
+        const due = isReferralTipDue({
+            json: opts.json,
+            isTTY: Boolean(process.stdout.isTTY),
+            optedOut: isOptedOut(),
+            lastShownAt,
+            now: Date.now(),
+        });
+        if (!due)
+            return;
+        console.log(chalk.dim(REFERRAL_TIP_TEXT));
+        config.set('referralTipShownAt', Date.now());
+    }
+    catch {
+        // silent — #151: a broken tip must never break a run
+    }
+}

package/docs/agent-quickstart.md

@@ -27,7 +27,7 @@ first-class on every one, and none of them ever blocks on a prompt (see
 ```
 trawl create <url> --prompt <goal> [--no-autofix] [--json]   # url also accepted as --url <url> (#116)
                                                               Create a persistent, self-healing scrap from a URL + a goal (AI-generated)
-trawl run <id> [--watch] [--json]              Run a scrap
+trawl run <id> [--watch] [--json]              Run a scrap (blocks until finished)
 trawl list|ls [--json] [--status <s>] [--limit <n>] [--page <n>]   List all scraps
 trawl get <id> [--json]                        Get scrap details
 trawl data <id> [--json] [--fresh] [--errors]  Get scrap data (last persisted run, or --fresh to launch one)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@trawlme/cli",
-  "version": "3.5.0",
+  "version": "3.6.0",
   "description": "Trawl CLI — manage scraps from the terminal",
   "type": "module",
   "bin": {