@trawlme/cli 3.10.0 → 3.11.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -79,7 +79,7 @@ trawl spec [--json] Print a versioned, machine-readab
79
79
  - `list`/`get` also show a **health** badge next to the status badge. By default it reads the scrap's own `lastCronOutcome` field (always present, no extra cost): `⚠ cron paused` when the last scheduled tick was skipped for being unhealthy, `—` otherwise — a breadcrumb of the last tick, not a live read (never set on a no-cron scrap, can lag by one tick). `list --unhealthy` asks the API to filter to scraps whose *current* consecutive-failure streak is 3+ (trawl_node's own threshold) and annotates each with the exact `consecutiveFailedRuns`/`unhealthySince` — shown instead of the badge whenever present, and never re-derived client-side from history. `--json` carries whichever fields the request produced: `lastCronOutcome` always, `consecutiveFailedRuns`/`unhealthySince` only when `--unhealthy` was passed. **Old-server safety:** a server predating trawl_node#1952 silently ignores the unknown `--unhealthy` query param and returns everything, unfiltered — the CLI detects that (no returned item carries `consecutiveFailedRuns`) and prints a stderr warning instead of presenting the full list as "your unhealthy scraps".
80
80
  - **Long-running calls (`create`, `run`, `data --fresh`, `trigger --wait`):** these hit server-side paths that can legitimately take 30–250s+ (AI generation + scrap creation + a first run for `create`; proxy tier escalation + AI-fix retries for the other three) — the CLI arms a 300s timeout for exactly these four call sites instead of the generic 30s default. `TRAWL_TIMEOUT` (see below) still overrides ALL requests, including these — set it if you need a tighter or looser ceiling than 300s, but note a global override that tight also clamps `create`.
81
81
  - **`--watch` is poll-based, not a live stream:** the activities SSE endpoint has no backlog and, for the default async `trigger` (no `--wait`), runs in a separate cron-consumer pod whose events never reach the API pod holding the SSE connection — a naive "await the run, then open SSE" shows nothing. `run --watch` and `trigger --watch` instead poll `GET /api/scraps/:id` (terminal status) and the activities REST list until the run finishes, printing each new activity line as it appears. The watched run's outcome drives the exit code too, in BOTH human and `--json` mode: a genuinely failed terminal run, a poll timeout, or a persistently unreachable API all exit non-zero — a clean successful run is the only exit `0`. A run that never reaches a terminal status within 300s prints an honest timeout notice pointing at `scraps doctor <id>` (human mode) — see the `--json` shape below. `scraps watch <id>` (the standalone command, no trigger) is unchanged — it still opens the live SSE stream directly.
82
- - `spec --json` prints the CLI's own command tree — `{specVersion, cliVersion, commands[], exitCodes, errorKinds, kindExitCodes}` — DERIVED at runtime by walking the live commander tree (never a hand-maintained file, which would silently drift from reality). Each entry in `commands[]` carries its full path (e.g. `"scraps account session set"`), description, `hidden` (the legacy `scraps <verb>` aliases above), `leaf` (false for a pure namespace/group node like `scraps`/`skills`/`telemetry` — invoking one directly is a guaranteed-failing tool, not a real command), `aliases`, `arguments`, and `options`; `exitCodes`/`errorKinds`/`kindExitCodes` are read from the exact same source `classifyError` uses (see [Exit codes](#exit-codes)) — never a second copy. `kindExitCodes` is the inverse of `exitCodes`: `kind -> exitCode`, since exit code `1` alone is a shared bucket (`api`/`refused`/`unknown`/`in_progress`/`run_failed`/`upgrade_failed`) that `exitCodes`' flat label can't disambiguate. Without `--json`, `spec` prints one short human line (version + visible command count) pointing at `--json`.
82
+ - `spec --json` prints the CLI's own command tree — `{specVersion, cliVersion, commands[], exitCodes, errorKinds, kindExitCodes, docsUrl?, llmsUrl?}` — DERIVED at runtime by walking the live commander tree (never a hand-maintained file, which would silently drift from reality). Each entry in `commands[]` carries its full path (e.g. `"scraps account session set"`), description, `hidden` (the legacy `scraps <verb>` aliases above), `leaf` (false for a pure namespace/group node like `scraps`/`skills`/`telemetry` — invoking one directly is a guaranteed-failing tool, not a real command), `aliases`, `arguments`, `options`, and an optional per-command `docs` deep link (e.g. every `scraps account *` command points at the account-sessions guide); `exitCodes`/`errorKinds`/`kindExitCodes` are read from the exact same source `classifyError` uses (see [Exit codes](#exit-codes)) — never a second copy. `kindExitCodes` is the inverse of `exitCodes`: `kind -> exitCode`, since exit code `1` alone is a shared bucket (`api`/`refused`/`unknown`/`in_progress`/`run_failed`/`upgrade_failed`) that `exitCodes`' flat label can't disambiguate. `docsUrl` is resolved server-first (`externalDocs.url` on the configured API base's own OpenAPI document, when declared), else derived from a KNOWN first-party `trawl.me` API host, else **omitted entirely** — never a guessed URL pointed at the wrong docs host for a self-hosted install. `llmsUrl` and every per-command `docs` link are narrower: they only ever come from that same known-host derivation, NEVER from a server-declared `docsUrl` — this CLI's own guide slugs have no reason to exist on a third party's own docs root, so a self-hosted server that declares `externalDocs.url` gets a correct top-level `docsUrl` but no `llmsUrl` and no per-command `docs` links. A `failureKind:"auth"` run's JSON payload (`doctor`/`data --errors`/`run-info`) carries the same `docs` field, by the same known-host-only rule. Without `--json`, `spec` prints one short human line (version + visible command count) pointing at `--json`.
83
83
  - `run --json`/`trigger --json` bypass the spinner and print the raw launch/trigger payload on stdout; combined with `--watch`, every intermediate progress line stays suppressed (stdout stays pure JSON) and, once the watch reaches its outcome, exactly ONE final NDJSON line is emitted: `{"runId","status"}` (the honest terminal status — `success`/`error`/`empty`/`regression`/…), `{"runId","status":"timeout"}` on a poll timeout, or `{"runId","status":"poll_error","error"}` if the API stays unreachable for several consecutive polls — `process.exitCode` is non-zero for all three except a genuine success. `scraps watch --json` emits one raw JSON object per activity line (NDJSON) instead of the formatted `[time] message` text — there's no single final payload to wait for on a live stream.
84
84
 
85
85
  ### Scrap management
@@ -12,6 +12,34 @@ import { formatDoctor, formatAutofix, fetchRunAndFix, pickRun, pickFix, isAuthWa
12
12
  import { renderPinch, pinchEnabled } from '../lib/pinch.js';
13
13
  import { maybeShowReferralTip } from '../lib/tips.js';
14
14
  import { maybeSuggestSkillsForAuthWall } from '../lib/skillsNudge.js';
15
+ import { getApiUrl } from '../lib/config.js';
16
+ import { resolveDocsUrls, resolveFailureKindDocsUrl } from '../lib/docs.js';
17
+ /**
18
+ * #185 — `{ docs }` (spreadable, empty object when nothing to add) for a
19
+ * run's JSON payload (`doctor`/`data --errors`/`run-info`), keyed on
20
+ * trawl_node's `failureKind` (NOT errors.ts's unrelated `ErrorEnvelope.kind`
21
+ * — see docs.ts's `FAILURE_KIND_DOC_PATHS` doc comment for why those two
22
+ * `'auth'`s must never be conflated). Reuses `isAuthWall` — the SAME
23
+ * staleness guard `formatDoctor`'s human hint and the #184 skills nudge
24
+ * already share (a `failureKind:'auth'` stamp survives a later
25
+ * success/regression patch, see `isAuthWall`'s own doc comment) — so a run
26
+ * that ultimately succeeded or degraded never carries a stale `docs` link
27
+ * either, exactly the invariant `detectWallVendor`/`isAuthWall` already
28
+ * enforce for the human-facing surfaces. One call site, one rule, reused by
29
+ * all three JSON surfaces below instead of three copies that could drift.
30
+ */
31
+ function runDocsField(run) {
32
+ if (!isAuthWall(run))
33
+ return {};
34
+ // Whole resolved object (never just `.docsUrl`) so resolveFailureKindDocsUrl
35
+ // can gate on rung provenance — see docs.ts's DocsUrls.docsUrlIsDerived.
36
+ // This call never supplies `externalDocsUrl`, so `docsUrl` here can only
37
+ // ever be rung-2-derived or absent — never a rung-1 server-declared root —
38
+ // but the gate stays explicit rather than relying on that as an invariant.
39
+ const resolved = resolveDocsUrls({ apiBaseUrl: getApiUrl() });
40
+ const docs = resolveFailureKindDocsUrl(run.failureKind, resolved);
41
+ return docs ? { docs } : {};
42
+ }
15
43
  /**
16
44
  * Print a usage/validation error consistently: human text to stderr, or a
17
45
  * machine envelope on stdout under --json (never both — reportError is the
@@ -1065,8 +1093,12 @@ export function attachDataCommand(parent, attachOpts = {}) {
1065
1093
  // --json is honored for BOTH outcomes (success or failure) — an agent
1066
1094
  // parsing `data --errors --json` must always get the flat run object,
1067
1095
  // never prose gated behind a status check. (#71 finding 13)
1096
+ // #185 — `docs` rides alongside as a sibling field on the same flat
1097
+ // object (never nested under `run`, since there's no wrapper shape
1098
+ // here), present only for a live mapped failureKind (see
1099
+ // runDocsField's doc comment).
1068
1100
  if (opts.json)
1069
- return json(pickRun(result.run));
1101
+ return json({ ...pickRun(result.run), ...runDocsField(result.run) });
1070
1102
  if (result.run.status === true) {
1071
1103
  console.log(chalk.green('✓ Last run succeeded. No errors to show.'));
1072
1104
  return;
@@ -1270,6 +1302,11 @@ export function attachRunInfoCommand(parent, attachOpts = {}) {
1270
1302
  selector: h.errorSnapshot?.selector ?? null,
1271
1303
  emptyContext: h.emptyContext ?? null,
1272
1304
  createdAt: h.createdAt ?? null,
1305
+ // #185 — present only for a live mapped failureKind (see
1306
+ // runDocsField's doc comment); omitted, never `docs: undefined`, for
1307
+ // every other run so this object's own JSON.stringify never grows an
1308
+ // unexpected key on existing consumers.
1309
+ ...runDocsField(h),
1273
1310
  };
1274
1311
  if (opts.json) {
1275
1312
  json(info);
@@ -1662,8 +1699,11 @@ scraps
1662
1699
  console.log(chalk.dim('No runs yet.'));
1663
1700
  return;
1664
1701
  }
1702
+ // #185 — `docs` sits alongside `run`/`fix` (present only for a live
1703
+ // mapped failureKind — see runDocsField's doc comment), never nested
1704
+ // inside the allowlisted `run` object.
1665
1705
  if (opts.json)
1666
- return json({ run: pickRun(result.run), fix: pickFix(result.fix) });
1706
+ return json({ run: pickRun(result.run), fix: pickFix(result.fix), ...runDocsField(result.run) });
1667
1707
  console.log(formatDoctor(result.scrap.title, result.run, result.fix, id));
1668
1708
  if (opts.autofix && result.fix) {
1669
1709
  console.log('\n' + formatAutofix(result.fix));
@@ -1,4 +1,5 @@
1
1
  import { Command } from 'commander';
2
+ import { type DocsUrls } from '../lib/docs.js';
2
3
  /**
3
4
  * `trawl spec --json` (#170) — a versioned, machine-readable description of
4
5
  * the command tree, so an agent can learn the CLI's surface without parsing
@@ -54,6 +55,18 @@ export interface CliSpecCommand {
54
55
  aliases: string[];
55
56
  arguments: CliSpecArgument[];
56
57
  options: CliSpecOption[];
58
+ /**
59
+ * #185 — a deep link into a specific guide when one is mapped for this
60
+ * command (docs.ts's `COMMAND_DOC_PATHS`, e.g. every `scraps account *`
61
+ * command -> the account-sessions guide). Omitted (never a guessed/generic
62
+ * link) whenever no guide is mapped, the top-level `docsUrl` itself
63
+ * couldn't be resolved, OR `docsUrl` resolved from rung 1 (a server-
64
+ * declared `externalDocs.url`) rather than rung 2 (this CLI's own known-host
65
+ * derivation) — appending our guide slug onto a third party's own docs root
66
+ * is exactly the confidently-wrong-404 defect this omission closes. See
67
+ * `resolveCommandDocsUrl` / `DocsUrls.docsUrlIsDerived`.
68
+ */
69
+ docs?: string;
57
70
  }
58
71
  export interface CliSpec {
59
72
  specVersion: 1;
@@ -81,12 +94,30 @@ export interface CliSpec {
81
94
  * (errors.ts's KIND_EXIT_CODES) — never a second copy.
82
95
  */
83
96
  kindExitCodes: Record<string, number>;
97
+ /**
98
+ * #185 — where the human-readable guides live, resolved via docs.ts's
99
+ * ladder (server `externalDocs.url` > derive from the API base's known
100
+ * `trawl.me` host > omit). Absent (never a guessed URL) when neither rung
101
+ * resolves — e.g. a self-hosted install with no `externalDocs` declared.
102
+ */
103
+ docsUrl?: string;
104
+ /** #185 — the llms.txt entry point alongside `docsUrl`, when resolvable.
105
+ * See docs.ts's module doc comment for why this never rides along with a
106
+ * server-declared `docsUrl` it wasn't itself derived from. */
107
+ llmsUrl?: string;
84
108
  }
85
109
  /**
86
110
  * Build the full spec from a live, already-constructed program (e.g.
87
111
  * `createProgram()`'s return value). Includes every node in the tree —
88
112
  * hidden legacy aliases (`scraps list`, …) included, flagged via `hidden`,
89
113
  * so a consumer that wants only the advertised surface can filter on it.
114
+ *
115
+ * #185 — `docs` is an already-RESOLVED `DocsUrls` (docs.ts's ladder run to
116
+ * completion), never computed in here: `buildSpec` stays pure/synchronous
117
+ * (no network, no config read) exactly as before, so every existing caller
118
+ * (the README-vs-spec contract test, the synthetic-tree unit tests) keeps
119
+ * working unchanged by simply omitting the parameter. Only `spec.ts`'s own
120
+ * action resolves rung 1 (a bounded fetch) before calling this.
90
121
  */
91
- export declare function buildSpec(program: Command): CliSpec;
122
+ export declare function buildSpec(program: Command, docs?: DocsUrls): CliSpec;
92
123
  export declare const spec: Command;
@@ -1,6 +1,9 @@
1
1
  import { Command, Help } from 'commander';
2
2
  import { json } from '../lib/format.js';
3
3
  import { EXIT_CODE_LABELS, ENVELOPE_KINDS, KIND_EXIT_CODES } from '../lib/errors.js';
4
+ import { getApiUrl } from '../lib/config.js';
5
+ import { api } from '../lib/api.js';
6
+ import { resolveDocsUrls, resolveCommandDocsUrl } from '../lib/docs.js';
4
7
  function buildOption(option) {
5
8
  const entry = {
6
9
  long: option.long ?? '',
@@ -15,8 +18,8 @@ function buildOption(option) {
15
18
  entry.default = option.defaultValue;
16
19
  return entry;
17
20
  }
18
- function buildCommandEntry(cmd, name, hidden) {
19
- return {
21
+ function buildCommandEntry(cmd, name, hidden, docs = {}) {
22
+ const entry = {
20
23
  name,
21
24
  description: cmd.description(),
22
25
  hidden,
@@ -29,20 +32,29 @@ function buildCommandEntry(cmd, name, hidden) {
29
32
  })),
30
33
  options: cmd.options.map(buildOption),
31
34
  };
35
+ const docsLink = resolveCommandDocsUrl(name, docs);
36
+ if (docsLink)
37
+ entry.docs = docsLink;
38
+ return entry;
32
39
  }
33
40
  /**
34
41
  * Walk `cmd.commands` recursively, collecting one entry per node under its
35
42
  * FULL path name. `hidden` is read via commander's own
36
43
  * `Help#visibleCommands()` — the same idiom this codebase already uses
37
44
  * (scraps.test.ts's #108 describe block) — rather than reaching for the
38
- * private, untyped `_hidden` field directly.
45
+ * private, untyped `_hidden` field directly. The resolved `docs` object,
46
+ * when present, threads through so every entry can carry its own per-command
47
+ * `docs` deep link (#185) off the SAME resolved root — never a second lookup
48
+ * per node, and never flattened to a bare string (see docs.ts's
49
+ * `DocsUrls.docsUrlIsDerived` — the rung provenance must survive the walk
50
+ * unchanged so `resolveCommandDocsUrl` can still gate on it at each leaf).
39
51
  */
40
- function walk(cmd, prefix, out) {
52
+ function walk(cmd, prefix, out, docs = {}) {
41
53
  const visible = new Set(new Help().visibleCommands(cmd));
42
54
  for (const sub of cmd.commands) {
43
55
  const name = prefix ? `${prefix} ${sub.name()}` : sub.name();
44
- out.push(buildCommandEntry(sub, name, !visible.has(sub)));
45
- walk(sub, name, out);
56
+ out.push(buildCommandEntry(sub, name, !visible.has(sub), docs));
57
+ walk(sub, name, out, docs);
46
58
  }
47
59
  }
48
60
  /**
@@ -50,11 +62,18 @@ function walk(cmd, prefix, out) {
50
62
  * `createProgram()`'s return value). Includes every node in the tree —
51
63
  * hidden legacy aliases (`scraps list`, …) included, flagged via `hidden`,
52
64
  * so a consumer that wants only the advertised surface can filter on it.
65
+ *
66
+ * #185 — `docs` is an already-RESOLVED `DocsUrls` (docs.ts's ladder run to
67
+ * completion), never computed in here: `buildSpec` stays pure/synchronous
68
+ * (no network, no config read) exactly as before, so every existing caller
69
+ * (the README-vs-spec contract test, the synthetic-tree unit tests) keeps
70
+ * working unchanged by simply omitting the parameter. Only `spec.ts`'s own
71
+ * action resolves rung 1 (a bounded fetch) before calling this.
53
72
  */
54
- export function buildSpec(program) {
73
+ export function buildSpec(program, docs = {}) {
55
74
  const commands = [];
56
- walk(program, '', commands);
57
- return {
75
+ walk(program, '', commands, docs);
76
+ const spec = {
58
77
  specVersion: 1,
59
78
  cliVersion: program.version() ?? 'unknown',
60
79
  commands,
@@ -62,18 +81,71 @@ export function buildSpec(program) {
62
81
  errorKinds: [...ENVELOPE_KINDS],
63
82
  kindExitCodes: { ...KIND_EXIT_CODES },
64
83
  };
84
+ if (docs.docsUrl)
85
+ spec.docsUrl = docs.docsUrl;
86
+ if (docs.llmsUrl)
87
+ spec.llmsUrl = docs.llmsUrl;
88
+ return spec;
89
+ }
90
+ /**
91
+ * #185 — bounds the SOCKET, not just the promise, so a slow/dead server can
92
+ * never perceptibly slow `spec --json` down. Mirrors lib/tips.ts's
93
+ * FLAG_CHECK_TIMEOUT_MS for the same reason: this is the one surface allowed
94
+ * a network call at all, and only because it's a deliberate, once-per-
95
+ * invocation agent probe.
96
+ *
97
+ * This is a REAL wall-clock ceiling, not merely a promise-level one: an
98
+ * earlier version of this fetch used `api.publicGet` (Node's global
99
+ * `fetch()` + `AbortSignal.timeout()`), which bounds the promise but not the
100
+ * OS-level connection — against a routable-but-silently-dropping host (a
101
+ * dropped SYN, no RST; a realistic firewalled/air-gapped shape, the very
102
+ * audience this feature cites) the promise rejected at ~2005ms while the
103
+ * PROCESS didn't exit until ~10.5s later, waiting out Node's own internal
104
+ * connect-timeout. `api.probeJson` (see its own doc comment in lib/api.ts)
105
+ * fixes this by managing the raw socket directly — an independent timer
106
+ * that `req.destroy()`s the connection, never relying on the idle-based
107
+ * `http.request` `timeout` option. Re-measured after the fix, same
108
+ * black-holed host, via the exact repro command (`time env
109
+ * TRAWL_API_URL=https://192.0.2.1 node dist/index.js spec --json`): total
110
+ * process time ~2.43s/~2.47s across two runs (the 2000ms bound plus Node
111
+ * startup and spec-tree construction), not ~10.68s.
112
+ */
113
+ const EXTERNAL_DOCS_FETCH_TIMEOUT_MS = 2_000;
114
+ /**
115
+ * Rung 1 — read `externalDocs.url` off the OpenAPI document the configured
116
+ * API base itself serves at `/api/spec.json`. Unset (null) server-side
117
+ * today, so this resolves to `null` in practice; forward-looking for when
118
+ * it isn't. ANY failure — network error, timeout, a malformed/unexpected
119
+ * response shape — is indistinguishable from "not declared" here: resolves
120
+ * to `null`, never throws (api.probeJson's own contract — see its doc
121
+ * comment). `spec --json` must stay a pure, reliable, side-effect-free probe
122
+ * even against an unreachable or ancient server.
123
+ */
124
+ async function fetchExternalDocsUrl() {
125
+ const data = await api.probeJson('/api/spec.json', {
126
+ timeoutMs: EXTERNAL_DOCS_FETCH_TIMEOUT_MS,
127
+ });
128
+ const url = data?.externalDocs?.url;
129
+ return typeof url === 'string' && url ? url : null;
65
130
  }
66
131
  export const spec = new Command('spec')
67
132
  .description('Print a versioned, machine-readable description of the command tree')
68
133
  .option('--json', 'Output as JSON')
69
- .action((opts) => {
134
+ .action(async (opts) => {
70
135
  // `spec` is registered as a direct child of the program root
71
136
  // (index.ts's createProgram), so `.parent` IS that root by the time this
72
137
  // action ever runs — commander sets it in `addCommand()`. Falling back
73
138
  // to `spec` itself only matters for an isolated unit invocation (e.g. a
74
139
  // test driving `spec.parseAsync()` directly, unattached to a program).
75
140
  const root = spec.parent ?? spec;
76
- const cliSpec = buildSpec(root);
141
+ const apiBaseUrl = getApiUrl();
142
+ // #185 — rung 1 (the server fetch) only runs under --json: the
143
+ // plain-text branch below never reads docsUrl at all, so paying a
144
+ // network round-trip for it would be pure waste. Rungs 2/3 (sync,
145
+ // no network) still apply either way via the fallback below.
146
+ const externalDocsUrl = opts.json ? await fetchExternalDocsUrl() : null;
147
+ const docs = resolveDocsUrls({ apiBaseUrl, externalDocsUrl });
148
+ const cliSpec = buildSpec(root, docs);
77
149
  if (opts.json) {
78
150
  json(cliSpec);
79
151
  return;
package/dist/index.d.ts CHANGED
@@ -32,6 +32,15 @@ export declare function resolveCommandName(actionCommand: Command | undefined):
32
32
  */
33
33
  export declare function collectCommandNames(root: Command): string[];
34
34
  export declare function createProgram(): Command;
35
+ /**
36
+ * #185 — see the `addHelpText('afterAll', …)` call above for why this is
37
+ * unconditional (no TTY/--json gate) and why a thrown error must never
38
+ * escape: commander calls this synchronously while already writing help
39
+ * output, so a throw here would crash a `--help` invocation, the one code
40
+ * path this whole feature is not allowed to touch (see docs.ts's own
41
+ * `docsFooterLine` for the "never a guessed URL" contract this composes).
42
+ */
43
+ export declare function docsFooterText(): string;
35
44
  /**
36
45
  * True when this module is the process entrypoint (not merely imported by a test).
37
46
  *
@@ -96,6 +105,14 @@ export declare function isBareInvocation(argv: string[]): boolean;
96
105
  * overstates its own invariant is worse than no comment, because the next
97
106
  * reader trusts it.
98
107
  *
108
+ * #185 — `spec --json` (not plain `spec`) now DOES make its own bounded
109
+ * (2s), swallowed-on-failure GET for `externalDocs.url` (see spec.ts's
110
+ * `fetchExternalDocsUrl`) — this is not a regression of the "no mutation"
111
+ * invariant above (a GET mutates nothing, and any failure — including no
112
+ * egress at all — degrades to the sync host-derivation fallback, never a
113
+ * thrown error), just a second, narrower kind of side effect this guard was
114
+ * never meant to suppress in the first place.
115
+ *
99
116
  * Before this, `trawl spec --json` silently re-synced
100
117
  * `~/.claude/skills` (and `./.claude/skills`) and printed `trawl: re-synced
101
118
  * skill …` lines to stderr BEFORE the JSON payload — an agent's very first
package/dist/index.js CHANGED
@@ -20,6 +20,8 @@ import { initPostHog, captureCommand, shutdown, registerAllowedCommands } from '
20
20
  import { classifyError, reportError, stripCommanderErrorPrefix, retryFieldsFor } from './lib/errors.js';
21
21
  import { renderPinch, pinchEnabled } from './lib/pinch.js';
22
22
  import { maybeNotifyUpdate, scheduleUpdateCheck } from './lib/updateNotifier.js';
23
+ import { getApiUrl } from './lib/config.js';
24
+ import { resolveDocsUrls, docsFooterLine } from './lib/docs.js';
23
25
  const __dirname = dirname(fileURLToPath(import.meta.url));
24
26
  const pkg = JSON.parse(readFileSync(join(__dirname, '..', 'package.json'), 'utf8'));
25
27
  /**
@@ -142,8 +144,37 @@ export function createProgram() {
142
144
  program.addCommand(logout);
143
145
  program.addCommand(telemetry);
144
146
  program.addCommand(upgrade);
147
+ // #185 — one dim line pointing at the docs, appended after EVERY `--help`
148
+ // in the tree (registering 'afterAll' on the root fires for a subcommand's
149
+ // own `--help` too — commander emits the afterAllHelp event on every
150
+ // ancestor of whichever command's outputHelp() ran, root included).
151
+ // Gated the way lib/tips.ts gates its own footer-shaped output: a pure,
152
+ // synchronous helper (docsFooterText, exported for unit testing) wrapped
153
+ // in try/catch so a broken footer can never turn a clean `--help` into a
154
+ // crash — but deliberately WITHOUT tips.ts's TTY/--json gate: `--help`
155
+ // itself never emits a --json payload (nothing here can pollute that
156
+ // channel), and a docs line inside `--help | less` is exactly the kind of
157
+ // thing worth keeping, unlike a promotional nudge.
158
+ program.addHelpText('afterAll', docsFooterText);
145
159
  return program;
146
160
  }
161
+ /**
162
+ * #185 — see the `addHelpText('afterAll', …)` call above for why this is
163
+ * unconditional (no TTY/--json gate) and why a thrown error must never
164
+ * escape: commander calls this synchronously while already writing help
165
+ * output, so a throw here would crash a `--help` invocation, the one code
166
+ * path this whole feature is not allowed to touch (see docs.ts's own
167
+ * `docsFooterLine` for the "never a guessed URL" contract this composes).
168
+ */
169
+ export function docsFooterText() {
170
+ try {
171
+ const line = docsFooterLine(resolveDocsUrls({ apiBaseUrl: getApiUrl() }).docsUrl);
172
+ return line ? chalk.dim(line) : '';
173
+ }
174
+ catch {
175
+ return '';
176
+ }
177
+ }
147
178
  /**
148
179
  * True when this module is the process entrypoint (not merely imported by a test).
149
180
  *
@@ -228,6 +259,14 @@ export function isBareInvocation(argv) {
228
259
  * overstates its own invariant is worse than no comment, because the next
229
260
  * reader trusts it.
230
261
  *
262
+ * #185 — `spec --json` (not plain `spec`) now DOES make its own bounded
263
+ * (2s), swallowed-on-failure GET for `externalDocs.url` (see spec.ts's
264
+ * `fetchExternalDocsUrl`) — this is not a regression of the "no mutation"
265
+ * invariant above (a GET mutates nothing, and any failure — including no
266
+ * egress at all — degrades to the sync host-derivation fallback, never a
267
+ * thrown error), just a second, narrower kind of side effect this guard was
268
+ * never meant to suppress in the first place.
269
+ *
231
270
  * Before this, `trawl spec --json` silently re-synced
232
271
  * `~/.claude/skills` (and `./.claude/skills`) and printed `trawl: re-synced
233
272
  * skill …` lines to stderr BEFORE the JSON payload — an agent's very first
package/dist/lib/api.d.ts CHANGED
@@ -95,6 +95,11 @@ export interface RequestOptions {
95
95
  export declare const api: {
96
96
  get: <T>(path: string, opts?: RequestOptions) => Promise<T>;
97
97
  publicGet: <T>(path: string, opts?: RequestOptions) => Promise<T>;
98
+ /** See `probeJson`'s own doc comment — hard-bounded, probe-only, never for
99
+ * a command's real work. Today's one caller: spec.ts's fetchExternalDocsUrl. */
100
+ probeJson: <T>(path: string, opts: {
101
+ timeoutMs: number;
102
+ }) => Promise<T | null>;
98
103
  getText: (path: string, opts?: RequestOptions) => Promise<string>;
99
104
  post: <T>(path: string, body?: unknown, opts?: RequestOptions) => Promise<T>;
100
105
  put: <T>(path: string, body?: unknown, opts?: RequestOptions) => Promise<T>;
package/dist/lib/api.js CHANGED
@@ -1,6 +1,8 @@
1
1
  import { readFileSync } from 'node:fs';
2
2
  import { fileURLToPath } from 'node:url';
3
3
  import { dirname, resolve } from 'node:path';
4
+ import { request as httpRequest } from 'node:http';
5
+ import { request as httpsRequest } from 'node:https';
4
6
  import { getApiUrl, getToken, getAuthMode, getLiveAuthEnvVar } from './config.js';
5
7
  import { parseServerJson } from './json.js';
6
8
  const __dirname = dirname(fileURLToPath(import.meta.url));
@@ -518,6 +520,119 @@ async function publicGet(path, reqOpts = {}) {
518
520
  throw new Error('Invalid JSON in server response');
519
521
  }
520
522
  }
523
+ /**
524
+ * A hard-bounded, PROBE-ONLY GET — never use this for a command's real work
525
+ * (no auth headers, no retry, no NetworkError-with-cause classification).
526
+ *
527
+ * trawl_cli#185 defect: every OTHER fetch in this file rides Node's global
528
+ * `fetch()` + `AbortSignal.timeout()`, which bounds the PROMISE, not the
529
+ * process. Aborting a fetch mid-connect does not necessarily tear down an
530
+ * in-flight TCP/TLS handshake — a documented Node/undici characteristic —
531
+ * so against a routable-but-silently-dropping host (a dropped SYN, never a
532
+ * RST — a realistic firewalled/air-gapped shape) the dangling socket keeps
533
+ * the event loop alive until Node's OWN internal connect-timeout eventually
534
+ * fires. Measured on this host: a bare `fetch()` + `AbortSignal.timeout(2000)`
535
+ * against such a host rejects its promise at ~2005ms, but the PROCESS does
536
+ * not exit until ~10.5s later — not configurable from here, and not paid by
537
+ * `ECONNREFUSED` (~0.13s) or a failed DNS lookup (~0.29s), only by the
538
+ * silent-drop shape. Every other command in this file accepts that risk
539
+ * because the user explicitly asked for that network call (`trawl ping`
540
+ * hanging on an unreachable host is the command doing its job — nothing to
541
+ * fix there). This primitive exists for the one case where the network call
542
+ * ITSELF is optional: `spec --json`'s own doc-discovery probe (the one
543
+ * command an agent runs to orient itself) has no business costing 10.5s to
544
+ * learn a host is unreachable, so this manages the raw socket directly.
545
+ *
546
+ * The timeout here is an INDEPENDENT `setTimeout` that calls `req.destroy()`
547
+ * — deliberately NOT `http.request`'s own `timeout` option (idle-based via
548
+ * `socket.setTimeout`, so it only fires when zero bytes arrive for that
549
+ * long — a coincidentally-same condition against a silent black hole, but a
550
+ * dishonest bound in general: it would never fire against a host that
551
+ * connects instantly then trickles bytes just often enough to reset the
552
+ * idle clock). Destroying the request/socket on this timer is what actually
553
+ * frees the OS-level handle so the process can exit promptly instead of
554
+ * waiting out Node's own multi-second connect-timeout — verified empirically
555
+ * (same black-holed host): total process time dropped from ~10.5s to ~2.0s
556
+ * with this change.
557
+ *
558
+ * Resolves to `null` on ANY failure (timeout, refusal, non-2xx, malformed
559
+ * JSON, an unsupported protocol) — never throws — so the one caller
560
+ * (spec.ts's fetchExternalDocsUrl) needs no try/catch of its own. This is
561
+ * NOT automatic: `node:http`/`node:https`' own `request()` throws
562
+ * SYNCHRONOUSLY (`ERR_INVALID_PROTOCOL`) for a URL whose protocol doesn't
563
+ * match the module (e.g. a stray `ftp://` in a misconfigured `TRAWL_API_URL`)
564
+ * — inside a Promise executor a synchronous throw becomes a REJECTED
565
+ * promise, which would have propagated straight through `fetchExternalDocsUrl`
566
+ * and turned `spec --json` into an error envelope instead of the spec.
567
+ * Reproduced: `TRAWL_API_URL=ftp://x node dist/index.js spec --json` failed
568
+ * entirely before this guard was added. The explicit protocol check below
569
+ * closes that.
570
+ *
571
+ * Two more deliberate departures from every other call in this file:
572
+ * - `TRAWL_TIMEOUT` (getTimeoutMs's env override, #91) is NOT consulted
573
+ * here — `opts.timeoutMs` is absolute. An operator raising that ceiling
574
+ * for a genuinely long-running command (e.g. LONG_RUN_TIMEOUT_MS's 300s)
575
+ * must never make an OPTIONAL probe wait 300s too; the two knobs measure
576
+ * different things and must not share a dial.
577
+ * - Redirects are NOT followed (global `fetch()`, used everywhere else in
578
+ * this file, follows them by default; raw `http`/`https` `request()`
579
+ * does not). For this probe a 3xx is just another non-2xx -> `null` ->
580
+ * fall through to rung 2 — an acceptable degradation, not a bug.
581
+ */
582
+ function probeJson(path, opts) {
583
+ return new Promise((settle) => {
584
+ let url;
585
+ try {
586
+ url = new URL(`${getApiUrl()}${path}`);
587
+ }
588
+ catch {
589
+ settle(null);
590
+ return;
591
+ }
592
+ if (url.protocol !== 'http:' && url.protocol !== 'https:') {
593
+ settle(null);
594
+ return;
595
+ }
596
+ let settled = false;
597
+ const finish = (value) => {
598
+ if (settled)
599
+ return;
600
+ settled = true;
601
+ clearTimeout(timer);
602
+ settle(value);
603
+ };
604
+ const transport = url.protocol === 'http:' ? httpRequest : httpsRequest;
605
+ const req = transport(url, { headers: { 'User-Agent': USER_AGENT } }, (res) => {
606
+ if (!res.statusCode || res.statusCode < 200 || res.statusCode >= 300) {
607
+ res.resume(); // drain so the socket can be released
608
+ finish(null);
609
+ return;
610
+ }
611
+ let body = '';
612
+ res.setEncoding('utf8');
613
+ res.on('data', (chunk) => {
614
+ body += chunk;
615
+ });
616
+ res.on('end', () => {
617
+ try {
618
+ finish(JSON.parse(body));
619
+ }
620
+ catch {
621
+ finish(null);
622
+ }
623
+ });
624
+ res.on('error', () => finish(null));
625
+ });
626
+ // The independent, non-idle-based ceiling this function exists for —
627
+ // fires regardless of connect/response state and forcibly destroys the
628
+ // socket, unlike AbortSignal.timeout() everywhere else in this file.
629
+ const timer = setTimeout(() => {
630
+ req.destroy(new Error(`probe to ${url.href} timed out after ${opts.timeoutMs}ms`));
631
+ }, opts.timeoutMs);
632
+ req.on('error', () => finish(null));
633
+ req.end();
634
+ });
635
+ }
521
636
  async function getText(path, reqOpts = {}) {
522
637
  const token = getToken();
523
638
  if (!token)
@@ -537,6 +652,9 @@ async function getText(path, reqOpts = {}) {
537
652
  export const api = {
538
653
  get: (path, opts) => request(path, {}, opts),
539
654
  publicGet: (path, opts) => publicGet(path, opts),
655
+ /** See `probeJson`'s own doc comment — hard-bounded, probe-only, never for
656
+ * a command's real work. Today's one caller: spec.ts's fetchExternalDocsUrl. */
657
+ probeJson: (path, opts) => probeJson(path, opts),
540
658
  getText: (path, opts) => getText(path, opts),
541
659
  post: (path, body, opts) => request(path, {
542
660
  method: 'POST',
@@ -0,0 +1,140 @@
1
+ /**
2
+ * trawl_cli#185 — ONE source of truth for the docs URL(s) an agent (or
3
+ * human) can be pointed at from three surfaces: `spec --json` (top-level
4
+ * `docsUrl`/`llmsUrl` + a per-command `docs` deep link), a run's JSON
5
+ * payload (`doctor`/`data --errors`/`run-info`, keyed on `failureKind` —
6
+ * NOT errors.ts's unrelated `ErrorEnvelope.kind`, see the doc comment on
7
+ * `FAILURE_KIND_DOC_PATHS` below), and the `--help` footer. Three hardcoded
8
+ * lists here would diverge within months — that exact "same fact computed
9
+ * in two places" defect has recurred repeatedly across this epic — so every
10
+ * surface reads these same tables/functions, never its own copy.
11
+ *
12
+ * Resolution ladder (issue #185 — all three rungs REQUIRED, in this order,
13
+ * evaluated PER FIELD, never as one bundled decision — see `resolveDocsUrls`):
14
+ *
15
+ * 1. Prefer the server. `externalDocs.url` on the OpenAPI document at
16
+ * `<apiBase>/api/spec.json` is the standard OpenAPI field for exactly
17
+ * this, and reading it makes a self-hosted install work with ZERO CLI
18
+ * change. It is unset (null) server-side today, so this is
19
+ * forward-looking — build it anyway, and it must win outright over rung
20
+ * 2 when present. The one caller allowed to fetch it is `spec.ts`'s own
21
+ * action, bounded and swallowed-on-failure (mirrors lib/tips.ts's
22
+ * `isReferralProgramUserFacing`) — a deliberate, once-per-invocation
23
+ * agent probe, not "every command". Every other surface (an error
24
+ * payload on an arbitrary failing command, the `--help` footer) must
25
+ * never add a network call or a failure mode to a command nobody asked
26
+ * to hit the docs host for — see `resolveDocsUrls`'s `externalDocsUrl`
27
+ * parameter, which only ever arrives pre-fetched.
28
+ * 2. Else derive, by stripping a leading `api.` host label from the
29
+ * configured API base — but ONLY for a KNOWN first-party `trawl.me`
30
+ * host. `api.trawl.me` -> `trawl.me` (prod: API and docs are genuinely
31
+ * on different hosts); `dev.trawl.me` (no `api.` prefix) -> unchanged
32
+ * (dev serves docs on the SAME host as its API). A generic, unscoped
33
+ * `api.`-strip applied to ANY host is itself a guess — it assumes a
34
+ * self-hosted `api.acme.internal` serves docs at `acme.internal`, which
35
+ * nothing here can know. Scoping the strip to `trawl.me` is what makes
36
+ * rung 3 (below) ever actually fire for a self-hosted base.
37
+ * 3. Else OMIT the field entirely. Never emit a guessed URL (issue's Rule
38
+ * 3): a wrong URL sends an agent to a 404 WITH CONFIDENCE, worse than no
39
+ * URL at all — a self-hosted install with no server-declared
40
+ * `externalDocs` and a base outside `trawl.me` gets no `docsUrl`/
41
+ * `llmsUrl`, not a hopeful default pointed at OUR docs host.
42
+ *
43
+ * `llmsUrl` has no OpenAPI-standard field to read (rung 1 contributes
44
+ * nothing to it) — deriving it from the ORIGIN of a server-declared
45
+ * `externalDocs.url` would itself be a guess for a self-hosted install
46
+ * (rule 3 again), so `llmsUrl` resolves ONLY via rung 2 (the known-host
47
+ * derivation) or omission. It never rides along with a rung-1 `docsUrl`.
48
+ *
49
+ * PROVENANCE (defect fix, reviewer repro against a live mock server): the
50
+ * top-level `docsUrl` above is deliberately the FLATTENED "whichever rung
51
+ * won" value — right for a human reading `spec --json`'s top-level field,
52
+ * WRONG as an input to `resolveCommandDocsUrl`/`resolveFailureKindDocsUrl`
53
+ * below. Those two append one of THIS CLI's own hardcoded guide slugs
54
+ * (`COMMAND_DOC_PATHS`/`FAILURE_KIND_DOC_PATHS`) onto whatever `docsUrl`
55
+ * resolved — safe onto a rung-2 root we derived ourselves (we know
56
+ * trawl.me's guide tree), never safe onto a rung-1 root (an arbitrary
57
+ * third party's own docs site, self-hosted, with no reason to carry our
58
+ * slugs). A mock server declaring `externalDocs.url: 'http://h/guide'`
59
+ * used to get `http://h/guide/build-your-scrap/account-sessions` appended
60
+ * — a confidently-wrong 404. So `DocsUrls.docsUrlIsDerived` carries rung
61
+ * provenance ALONGSIDE the string (never flattened to a bare string again)
62
+ * and both deep-link functions now take the whole `DocsUrls`-shaped object
63
+ * and gate construction on that flag — see its own doc comment below.
64
+ */
65
+ export interface DocsUrls {
66
+ docsUrl?: string;
67
+ llmsUrl?: string;
68
+ /**
69
+ * True iff `docsUrl` came from rung 2 (THIS CLI's own derivation from a
70
+ * KNOWN `trawl.me` host) — the only case where it is safe to append one of
71
+ * this CLI's own guide slugs onto it (see `resolveCommandDocsUrl`/
72
+ * `resolveFailureKindDocsUrl`). False when `docsUrl` is a rung-1
73
+ * SERVER-declared root instead — an arbitrary third party's own docs site,
74
+ * which has no reason to host our guide slugs, even when that server
75
+ * happens to run on a `trawl.me` host. Always a defined boolean whenever
76
+ * `docsUrl` itself is present; undefined only when `docsUrl` is undefined
77
+ * too (rung 3 — nothing resolved at all).
78
+ */
79
+ docsUrlIsDerived?: boolean;
80
+ }
81
+ /**
82
+ * Rung 2 — see the module doc comment. Returns `{ protocol, host }` for a
83
+ * KNOWN first-party API base (preserving the base's own protocol rather
84
+ * than assuming `https:`), or `null` when the base isn't recognized
85
+ * (self-hosted, a custom domain, `localhost`, an unparseable string, …) —
86
+ * `null` must propagate to omission (rung 3), never to a guessed host.
87
+ */
88
+ export declare function deriveDocsOrigin(apiBaseUrl: string): {
89
+ protocol: string;
90
+ host: string;
91
+ } | null;
92
+ /** Same rung as `deriveDocsOrigin`, collapsed to just the host string —
93
+ * convenience for a caller that only needs the derived host, not the
94
+ * protocol (kept as its own export since `docs.test.ts` exercises the host
95
+ * derivation independently of protocol handling). */
96
+ export declare function deriveDocsHost(apiBaseUrl: string): string | null;
97
+ /**
98
+ * The full ladder, PER FIELD (see module doc comment for why `llmsUrl`
99
+ * cannot ride along with a rung-1 `docsUrl`). Pure and synchronous — no
100
+ * network, safe to call from any surface (an error payload, the `--help`
101
+ * footer) without adding latency or a new failure mode. `externalDocsUrl`
102
+ * is rung 1's input: only ever supplied by `spec.ts`'s action, after its
103
+ * own bounded, swallowed-on-failure fetch — every other caller omits it and
104
+ * gets rungs 2/3 only.
105
+ */
106
+ export declare function resolveDocsUrls(opts: {
107
+ apiBaseUrl: string;
108
+ externalDocsUrl?: string | null;
109
+ }): DocsUrls;
110
+ /**
111
+ * `resolveCommandDocsUrl` returns `undefined` (never a bare path or a
112
+ * relative link) whenever any of three things is missing — `docsUrl`
113
+ * unresolved (rung 3 already fired), `docsUrl` resolved but NOT derived
114
+ * (rung 1 — a server-declared root; see `DocsUrls.docsUrlIsDerived`'s doc
115
+ * comment for why appending our own guide slug onto a third party's root is
116
+ * exactly the confidently-wrong-404 defect this gate closes), or no guide is
117
+ * mapped for this command — so a spec consumer never has to special-case a
118
+ * partial value. Takes the whole resolved `DocsUrls` object (never a bare
119
+ * string) specifically so this provenance can never again be flattened away
120
+ * before it reaches here.
121
+ */
122
+ export declare function resolveCommandDocsUrl(commandName: string, docs: Pick<DocsUrls, 'docsUrl' | 'docsUrlIsDerived'>): string | undefined;
123
+ /** Same "undefined unless docsUrl is resolved AND derived (rung 2)" gate as
124
+ * `resolveCommandDocsUrl` above, same reason — never append this CLI's own
125
+ * guide slug onto a rung-1 server-declared root. Callers MUST gate the call
126
+ * on their own staleness rule first (e.g. `doctor.ts`'s `isAuthWall`) — this
127
+ * function only knows the string-to-path mapping, not whether a stamped
128
+ * `failureKind` is still live on a run patched afterward. */
129
+ export declare function resolveFailureKindDocsUrl(kind: string | null | undefined, docs: Pick<DocsUrls, 'docsUrl' | 'docsUrlIsDerived'>): string | undefined;
130
+ /**
131
+ * `--help` footer text (issue #185 scope item 3) — a single dim line, for
132
+ * humans, e.g. "Docs: https://trawl.me/docs". Returns `undefined` (never an
133
+ * empty or guessed line) when no `docsUrl` resolved — mirrors
134
+ * lib/tips.ts's shape (a pure function separated from IO, so it's testable
135
+ * without mocking chalk/console) but NOT its TTY gate: tips.ts suppresses a
136
+ * promotional nudge from piped output, but a docs line inside `--help |
137
+ * less` is exactly the kind of thing worth keeping. Un-colored here — the
138
+ * caller (index.ts) applies `chalk.dim` so this stays trivially testable.
139
+ */
140
+ export declare function docsFooterLine(docsUrl?: string): string | undefined;
@@ -0,0 +1,238 @@
1
+ /**
2
+ * trawl_cli#185 — ONE source of truth for the docs URL(s) an agent (or
3
+ * human) can be pointed at from three surfaces: `spec --json` (top-level
4
+ * `docsUrl`/`llmsUrl` + a per-command `docs` deep link), a run's JSON
5
+ * payload (`doctor`/`data --errors`/`run-info`, keyed on `failureKind` —
6
+ * NOT errors.ts's unrelated `ErrorEnvelope.kind`, see the doc comment on
7
+ * `FAILURE_KIND_DOC_PATHS` below), and the `--help` footer. Three hardcoded
8
+ * lists here would diverge within months — that exact "same fact computed
9
+ * in two places" defect has recurred repeatedly across this epic — so every
10
+ * surface reads these same tables/functions, never its own copy.
11
+ *
12
+ * Resolution ladder (issue #185 — all three rungs REQUIRED, in this order,
13
+ * evaluated PER FIELD, never as one bundled decision — see `resolveDocsUrls`):
14
+ *
15
+ * 1. Prefer the server. `externalDocs.url` on the OpenAPI document at
16
+ * `<apiBase>/api/spec.json` is the standard OpenAPI field for exactly
17
+ * this, and reading it makes a self-hosted install work with ZERO CLI
18
+ * change. It is unset (null) server-side today, so this is
19
+ * forward-looking — build it anyway, and it must win outright over rung
20
+ * 2 when present. The one caller allowed to fetch it is `spec.ts`'s own
21
+ * action, bounded and swallowed-on-failure (mirrors lib/tips.ts's
22
+ * `isReferralProgramUserFacing`) — a deliberate, once-per-invocation
23
+ * agent probe, not "every command". Every other surface (an error
24
+ * payload on an arbitrary failing command, the `--help` footer) must
25
+ * never add a network call or a failure mode to a command nobody asked
26
+ * to hit the docs host for — see `resolveDocsUrls`'s `externalDocsUrl`
27
+ * parameter, which only ever arrives pre-fetched.
28
+ * 2. Else derive, by stripping a leading `api.` host label from the
29
+ * configured API base — but ONLY for a KNOWN first-party `trawl.me`
30
+ * host. `api.trawl.me` -> `trawl.me` (prod: API and docs are genuinely
31
+ * on different hosts); `dev.trawl.me` (no `api.` prefix) -> unchanged
32
+ * (dev serves docs on the SAME host as its API). A generic, unscoped
33
+ * `api.`-strip applied to ANY host is itself a guess — it assumes a
34
+ * self-hosted `api.acme.internal` serves docs at `acme.internal`, which
35
+ * nothing here can know. Scoping the strip to `trawl.me` is what makes
36
+ * rung 3 (below) ever actually fire for a self-hosted base.
37
+ * 3. Else OMIT the field entirely. Never emit a guessed URL (issue's Rule
38
+ * 3): a wrong URL sends an agent to a 404 WITH CONFIDENCE, worse than no
39
+ * URL at all — a self-hosted install with no server-declared
40
+ * `externalDocs` and a base outside `trawl.me` gets no `docsUrl`/
41
+ * `llmsUrl`, not a hopeful default pointed at OUR docs host.
42
+ *
43
+ * `llmsUrl` has no OpenAPI-standard field to read (rung 1 contributes
44
+ * nothing to it) — deriving it from the ORIGIN of a server-declared
45
+ * `externalDocs.url` would itself be a guess for a self-hosted install
46
+ * (rule 3 again), so `llmsUrl` resolves ONLY via rung 2 (the known-host
47
+ * derivation) or omission. It never rides along with a rung-1 `docsUrl`.
48
+ *
49
+ * PROVENANCE (defect fix, reviewer repro against a live mock server): the
50
+ * top-level `docsUrl` above is deliberately the FLATTENED "whichever rung
51
+ * won" value — right for a human reading `spec --json`'s top-level field,
52
+ * WRONG as an input to `resolveCommandDocsUrl`/`resolveFailureKindDocsUrl`
53
+ * below. Those two append one of THIS CLI's own hardcoded guide slugs
54
+ * (`COMMAND_DOC_PATHS`/`FAILURE_KIND_DOC_PATHS`) onto whatever `docsUrl`
55
+ * resolved — safe onto a rung-2 root we derived ourselves (we know
56
+ * trawl.me's guide tree), never safe onto a rung-1 root (an arbitrary
57
+ * third party's own docs site, self-hosted, with no reason to carry our
58
+ * slugs). A mock server declaring `externalDocs.url: 'http://h/guide'`
59
+ * used to get `http://h/guide/build-your-scrap/account-sessions` appended
60
+ * — a confidently-wrong 404. So `DocsUrls.docsUrlIsDerived` carries rung
61
+ * provenance ALONGSIDE the string (never flattened to a bare string again)
62
+ * and both deep-link functions now take the whole `DocsUrls`-shaped object
63
+ * and gate construction on that flag — see its own doc comment below.
64
+ */
65
+ /** The one first-party domain this CLI knows to serve docs — see rung 2 in
66
+ * the module doc comment above. Deliberately not "any host", so an
67
+ * unrecognized (self-hosted/custom) base falls through to omission. */
68
+ const KNOWN_DOCS_DOMAIN = 'trawl.me';
69
+ /**
70
+ * Rung 2 — see the module doc comment. Returns `{ protocol, host }` for a
71
+ * KNOWN first-party API base (preserving the base's own protocol rather
72
+ * than assuming `https:`), or `null` when the base isn't recognized
73
+ * (self-hosted, a custom domain, `localhost`, an unparseable string, …) —
74
+ * `null` must propagate to omission (rung 3), never to a guessed host.
75
+ */
76
+ export function deriveDocsOrigin(apiBaseUrl) {
77
+ let url;
78
+ try {
79
+ url = new URL(apiBaseUrl);
80
+ }
81
+ catch {
82
+ return null;
83
+ }
84
+ const hostname = url.hostname.toLowerCase();
85
+ const stripped = hostname.startsWith('api.') ? hostname.slice('api.'.length) : hostname;
86
+ if (stripped === KNOWN_DOCS_DOMAIN || stripped.endsWith(`.${KNOWN_DOCS_DOMAIN}`)) {
87
+ return { protocol: url.protocol, host: stripped };
88
+ }
89
+ return null;
90
+ }
91
+ /** Same rung as `deriveDocsOrigin`, collapsed to just the host string —
92
+ * convenience for a caller that only needs the derived host, not the
93
+ * protocol (kept as its own export since `docs.test.ts` exercises the host
94
+ * derivation independently of protocol handling). */
95
+ export function deriveDocsHost(apiBaseUrl) {
96
+ return deriveDocsOrigin(apiBaseUrl)?.host ?? null;
97
+ }
98
+ /**
99
+ * The full ladder, PER FIELD (see module doc comment for why `llmsUrl`
100
+ * cannot ride along with a rung-1 `docsUrl`). Pure and synchronous — no
101
+ * network, safe to call from any surface (an error payload, the `--help`
102
+ * footer) without adding latency or a new failure mode. `externalDocsUrl`
103
+ * is rung 1's input: only ever supplied by `spec.ts`'s action, after its
104
+ * own bounded, swallowed-on-failure fetch — every other caller omits it and
105
+ * gets rungs 2/3 only.
106
+ */
107
+ export function resolveDocsUrls(opts) {
108
+ const origin = deriveDocsOrigin(opts.apiBaseUrl);
109
+ const derived = origin
110
+ ? {
111
+ docsUrl: `${origin.protocol}//${origin.host}/docs`,
112
+ llmsUrl: `${origin.protocol}//${origin.host}/llms.txt`,
113
+ docsUrlIsDerived: true,
114
+ }
115
+ : {};
116
+ if (opts.externalDocsUrl) {
117
+ // Rung 1 wins for docsUrl outright; llmsUrl stays rung-2-only (see doc
118
+ // comment) — a self-hosted `externalDocsUrl` outside `trawl.me` yields
119
+ // `{ docsUrl: <server's own> }` with `llmsUrl` omitted, never guessed.
120
+ // `docsUrlIsDerived: false` regardless of whether rung 2 ALSO resolved
121
+ // (e.g. a `trawl.me`-hosted server declaring its own externalDocs) — a
122
+ // server-declared root is never "derived" by THIS CLI, and never a safe
123
+ // base for our own guide slugs (see DocsUrls.docsUrlIsDerived).
124
+ return derived.llmsUrl
125
+ ? { docsUrl: opts.externalDocsUrl, llmsUrl: derived.llmsUrl, docsUrlIsDerived: false }
126
+ : { docsUrl: opts.externalDocsUrl, docsUrlIsDerived: false };
127
+ }
128
+ return derived;
129
+ }
130
+ /** Join a docs root (e.g. `https://trawl.me/docs`) with a guide path
131
+ * suffix (e.g. `build-your-scrap/account-sessions`) — plain string
132
+ * concatenation, deliberately NOT `new URL(suffix, docsUrl)`: a
133
+ * leading-slash suffix resolved against a URL with its own path component
134
+ * (`/docs`) would resolve relative to the ORIGIN, silently dropping
135
+ * `/docs` from the result (`new URL('/x', 'https://h/docs')` ==
136
+ * `https://h/x`, not `https://h/docs/x`). */
137
+ function joinDocsPath(docsUrl, suffix) {
138
+ return `${docsUrl.replace(/\/+$/, '')}/${suffix.replace(/^\/+/, '')}`;
139
+ }
140
+ /**
141
+ * The single source for the account-sessions guide slug — referenced by
142
+ * BOTH `COMMAND_DOC_PATHS` and `FAILURE_KIND_DOC_PATHS` below. Those two
143
+ * tables key on different fields (a CLI command's full path name vs a run's
144
+ * `failureKind`) and legitimately stay separate, but they name the SAME
145
+ * guide, so the path literal itself must exist exactly once: this is the
146
+ * "same fact computed in two places" defect class this module's own doc
147
+ * comment says it exists to prevent, and it has recurred repeatedly across
148
+ * this epic. Renaming the guide used to require editing two literals in
149
+ * lockstep — miss one and one surface (a run's `docs` field, or `spec
150
+ * --json`'s per-command deep link) silently 404s while the other still
151
+ * works (trawl_cli#185 review).
152
+ */
153
+ const ACCOUNT_SESSIONS_GUIDE_PATH = 'build-your-scrap/account-sessions';
154
+ /**
155
+ * Per-command deep links (issue #185 scope item 1) — a command's FULL path
156
+ * name (matches `CliSpecCommand.name` in spec.ts, e.g.
157
+ * `"scraps account session set"`) is matched against `prefix` either
158
+ * exactly or as a whole path SEGMENT prefix (`startsWith(prefix + ' ')`) —
159
+ * never a bare substring, so a hypothetical future `scraps accounting`
160
+ * command could never false-match the `scraps account` entry below.
161
+ */
162
+ const COMMAND_DOC_PATHS = Object.freeze([
163
+ // Every account/session-management command — set, delete, clear-session,
164
+ // status, and the nested `session set` — is covered by one prefix entry
165
+ // rather than one row per leaf, so a new leaf added under `scraps account`
166
+ // later (e.g. a future `session capture`, trawl_cli#183) inherits the
167
+ // link for free instead of needing its own row.
168
+ { prefix: 'scraps account', path: ACCOUNT_SESSIONS_GUIDE_PATH },
169
+ ]);
170
+ function docsPathForCommand(commandName) {
171
+ const match = COMMAND_DOC_PATHS.find((e) => commandName === e.prefix || commandName.startsWith(`${e.prefix} `));
172
+ return match ? match.path : null;
173
+ }
174
+ /**
175
+ * `resolveCommandDocsUrl` returns `undefined` (never a bare path or a
176
+ * relative link) whenever any of three things is missing — `docsUrl`
177
+ * unresolved (rung 3 already fired), `docsUrl` resolved but NOT derived
178
+ * (rung 1 — a server-declared root; see `DocsUrls.docsUrlIsDerived`'s doc
179
+ * comment for why appending our own guide slug onto a third party's root is
180
+ * exactly the confidently-wrong-404 defect this gate closes), or no guide is
181
+ * mapped for this command — so a spec consumer never has to special-case a
182
+ * partial value. Takes the whole resolved `DocsUrls` object (never a bare
183
+ * string) specifically so this provenance can never again be flattened away
184
+ * before it reaches here.
185
+ */
186
+ export function resolveCommandDocsUrl(commandName, docs) {
187
+ if (!docs.docsUrl || !docs.docsUrlIsDerived)
188
+ return undefined;
189
+ const path = docsPathForCommand(commandName);
190
+ return path ? joinDocsPath(docs.docsUrl, path) : undefined;
191
+ }
192
+ /**
193
+ * failureKind -> guide path (issue #185 scope item 2). Keyed on trawl_node's
194
+ * run-level `failureKind` field (trawl_node#1975, surfaced by this CLI's
195
+ * own `doctor`/`data --errors`/`run-info` JSON payloads since cli#182) —
196
+ * this is a DIFFERENT axis from errors.ts's `ErrorEnvelope.kind` (a CLI
197
+ * transport/execution outcome like `"auth"` meaning "you're not logged
198
+ * into trawl", `"network"`, `"usage"`, …). Both happen to use the string
199
+ * `"auth"` for unrelated things: this table's `'auth'` means "the SCRAPED
200
+ * TARGET site showed a login wall", errors.ts's `'auth'` means "the TRAWL
201
+ * API rejected your OWN credentials". Never merge these two tables — they
202
+ * are keyed off different fields on different objects, and conflating them
203
+ * would either miss the run-level guide or wrongly attach it to an
204
+ * unrelated CLI auth failure.
205
+ */
206
+ const FAILURE_KIND_DOC_PATHS = Object.freeze({
207
+ auth: ACCOUNT_SESSIONS_GUIDE_PATH,
208
+ });
209
+ function docsPathForFailureKind(kind) {
210
+ if (!kind)
211
+ return null;
212
+ return FAILURE_KIND_DOC_PATHS[kind] ?? null;
213
+ }
214
+ /** Same "undefined unless docsUrl is resolved AND derived (rung 2)" gate as
215
+ * `resolveCommandDocsUrl` above, same reason — never append this CLI's own
216
+ * guide slug onto a rung-1 server-declared root. Callers MUST gate the call
217
+ * on their own staleness rule first (e.g. `doctor.ts`'s `isAuthWall`) — this
218
+ * function only knows the string-to-path mapping, not whether a stamped
219
+ * `failureKind` is still live on a run patched afterward. */
220
+ export function resolveFailureKindDocsUrl(kind, docs) {
221
+ if (!docs.docsUrl || !docs.docsUrlIsDerived)
222
+ return undefined;
223
+ const path = docsPathForFailureKind(kind);
224
+ return path ? joinDocsPath(docs.docsUrl, path) : undefined;
225
+ }
226
+ /**
227
+ * `--help` footer text (issue #185 scope item 3) — a single dim line, for
228
+ * humans, e.g. "Docs: https://trawl.me/docs". Returns `undefined` (never an
229
+ * empty or guessed line) when no `docsUrl` resolved — mirrors
230
+ * lib/tips.ts's shape (a pure function separated from IO, so it's testable
231
+ * without mocking chalk/console) but NOT its TTY gate: tips.ts suppresses a
232
+ * promotional nudge from piped output, but a docs line inside `--help |
233
+ * less` is exactly the kind of thing worth keeping. Un-colored here — the
234
+ * caller (index.ts) applies `chalk.dim` so this stays trivially testable.
235
+ */
236
+ export function docsFooterLine(docsUrl) {
237
+ return docsUrl ? `Docs: ${docsUrl}` : undefined;
238
+ }
@@ -166,7 +166,15 @@ just to look helpful, so don't treat its absence as an error.
166
166
 
167
167
  For the full, versioned machine description of every command (arguments,
168
168
  options, aliases, exit codes, error kinds) — instead of parsing this doc —
169
- run `trawl spec --json`.
169
+ run `trawl spec --json`. It also carries `docsUrl` (resolved from the server
170
+ when declared, else derived from a known first-party API host, else omitted
171
+ — never a guessed URL) and, where a guide exists, a per-command `docs` deep
172
+ link — but the deep link and `llmsUrl` are narrower than `docsUrl`: they only
173
+ ever come from the known-first-party-host derivation, never from a
174
+ server-declared `docsUrl`, since this CLI's own guide slugs have no reason
175
+ to exist on a third party's own docs root. A `failureKind:"auth"` run's
176
+ payload (`doctor`/`data --errors`/`run-info`) carries the same `docs` field,
177
+ under the same rule.
170
178
 
171
179
  ## Non-interactive contract
172
180
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@trawlme/cli",
3
- "version": "3.10.0",
3
+ "version": "3.11.0",
4
4
  "description": "Trawl CLI — manage scraps from the terminal",
5
5
  "type": "module",
6
6
  "bin": {