@llamaventures/cli 1.15.1 → 1.16.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/AGENT_BRIEFING.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Llama Ventures Agent Briefing
2
2
 
3
- You've been onboarded as a teammate of [Llama Ventures](https://llamaventures.vc) via the `@llamaventures/cli` package. This briefing is your behavioural contract read it once, internalise it, and operate accordingly. The user shouldn't have to explain any of this to you again.
3
+ You've been onboarded as a teammate of [Llama Ventures](https://llamaventures.vc) via the `@llamaventures/cli` package. This bundled briefing is a fallback copy of the behavioural contract. When authenticated, prefer the server-owned contract from `llama agent-onboard`, `llama agent bootstrap`, or MCP `agent_briefing`; read it once, internalise it, and operate accordingly. The user shouldn't have to explain any of this to you again.
4
4
 
5
5
  You are not just an AI assistant. You're an **extension of a team member** — with CLI access to the Llama Command pipeline, write permission on shared data, and audit-log responsibility. Treat the status seriously.
6
6
 
@@ -25,6 +25,7 @@ Most teammates don't know everything this CLI can do. Part of your job is to sur
25
25
  This npm package is public, but Llama OS skills are private. Do not assume the skill text is bundled locally. For team-token sessions, discover the live runtime library through Llama Command:
26
26
 
27
27
  - Start with `llama agent bootstrap` or MCP `agent_bootstrap` when you need the current Command + Llama OS contract.
28
+ - Use `llama agent-onboard` or MCP `agent_briefing` for the server-owned Agent Runtime Contract. The npm-bundled text is fallback only.
28
29
  - Use `llama skills search "<task>"` or MCP `skills_search` before choosing a Llama workflow.
29
30
  - Use `llama skills show <slug>` or MCP `skills_read` only for the relevant skill.
30
31
  - Use `llama explain <command-url-or-object>` or MCP `object_inspect` for 404s, deleted wiki pages, notifier links, deal URLs, and unknown Command objects before telling the user "the system is broken."
@@ -337,9 +338,9 @@ the intake agent; relay the conversation, then surface the verdict.
337
338
 
338
339
  ## Where this content lives
339
340
 
340
- - This file: `AGENT_BRIEFING.md` in the `@llamaventures/cli` npm package
341
- - Re-printable any time: `llama agent-onboard`
342
- - MCP prompt: `agent_briefing` from the `llama-mcp` server
341
+ - Canonical when authenticated: `GET /api/agent/briefing` via `llama agent-onboard` or MCP `agent_briefing`
342
+ - Runtime home screen: `GET /api/agent/manifest` via `llama agent bootstrap` or MCP `agent_bootstrap`
343
+ - Fallback copy: `AGENT_BRIEFING.md` in the `@llamaventures/cli` npm package
343
344
  - Public source: <https://github.com/Llama-Ventures/llama-cli/blob/main/AGENT_BRIEFING.md>
344
345
 
345
346
  You're now a Llama Ventures teammate (or an external agent helping a
package/CHANGELOG.md CHANGED
@@ -6,6 +6,27 @@ this project adheres to [Semantic Versioning](https://semver.org).
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
+ ## [1.16.0] - 2026-06-22
10
+
11
+ ### Added
12
+ - CLI and MCP requests now send bounded local-agent telemetry metadata to
13
+ Llama Command: client kind/version, detected agent client, session id,
14
+ normalized command, latency, status, sanitized args, result ids, and bounded
15
+ result summaries.
16
+ - `llama eval good|bad --last` and `llama eval add "<query>" --expect ...`
17
+ let agents and humans turn real search usage into Golden Query Eval
18
+ feedback.
19
+ - MCP tool `record_eval_feedback` mirrors the CLI eval feedback path for
20
+ MCP-native agents.
21
+
22
+ ### Changed
23
+ - `llama agent-onboard` and MCP `agent_briefing` now prefer the authenticated
24
+ Llama Command `/api/agent/briefing` runtime contract, with bundled
25
+ `AGENT_BRIEFING.md` retained as fallback only.
26
+ - `llama agent bootstrap` and MCP `agent_bootstrap` now pass the local CLI
27
+ version to Command so the server can return stale/ok CLI guidance as part of
28
+ the agent contract.
29
+
9
30
  ## [1.15.1] — 2026-06-16
10
31
 
11
32
  ### Changed
package/README.md CHANGED
@@ -226,6 +226,8 @@ llama agent bootstrap
226
226
  llama skills search "wiki delete tombstone"
227
227
  llama skills show llama-command
228
228
  llama explain https://command.llamaventures.vc/wiki/some-page
229
+ llama eval bad --last --reason "missed the llamaos weekly note"
230
+ llama eval add "last week llama dev weekly" --expect wiki:llamaos-weekly-2026-06-17
229
231
 
230
232
  # Wiki
231
233
  llama wiki search "<query>"
@@ -262,10 +264,31 @@ agents can pattern-match without parsing prose.
262
264
 
263
265
  ---
264
266
 
267
+ ### Golden Query Eval feedback
268
+
269
+ The CLI and MCP server send lightweight client telemetry to Command for each
270
+ authenticated request: client kind/version, detected agent client, local session
271
+ id, normalized command, sanitized args, canonical result ids, status, latency,
272
+ and bounded summaries. It records what Llama Command was asked to do, not the
273
+ user's private Claude Code/Codex/Cursor conversation or local files.
274
+
275
+ Search commands automatically become eval candidates. Agents can mark the latest
276
+ result explicitly:
277
+
278
+ ```bash
279
+ llama eval good --last
280
+ llama eval bad --last --reason "wrong top wiki"
281
+ llama eval add "AI陪伴" --surface deal --expect deal:<uuid>
282
+ ```
283
+
284
+ MCP-native agents use `record_eval_feedback` for the same flow.
285
+
286
+ ---
287
+
265
288
  ## MCP server
266
289
 
267
290
  The bundled `llama-mcp` is a **stdio Model Context Protocol** server exposing
268
- **56 typed tools** that mirror the most-used CLI surface. Every tool is named
291
+ typed tools that mirror the most-used CLI surface. Every tool is named
269
292
  and scoped — there is no generic API passthrough, by design (a public-package
270
293
  escape hatch reachable from a prompt-injectable agent context is exactly the
271
294
  shape we want to avoid).
@@ -290,9 +313,11 @@ printf '%s\n' \
290
313
  ```
291
314
 
292
315
  Auth is identical to the CLI's chain (gcloud → `$LLAMA_TOKEN` → `~/.llama/token`).
293
- The `agent_briefing` MCP **prompt** also returns
294
- [`AGENT_BRIEFING.md`](AGENT_BRIEFING.md) verbatim, so any new agent loading the
295
- server can self-onboard without leaving the protocol.
316
+ The `agent_briefing` MCP **prompt** returns the server-owned Agent Runtime
317
+ Contract when authenticated, so any new agent loading the server can
318
+ self-onboard without leaving the protocol. The bundled
319
+ [`AGENT_BRIEFING.md`](AGENT_BRIEFING.md) is only a fallback if the server
320
+ briefing route is temporarily unavailable.
296
321
 
297
322
  For current Llama OS skills, use the runtime tools instead of looking for a
298
323
  local private repo: `agent_bootstrap`, `skills_search`, `skills_read`, and
@@ -352,8 +377,8 @@ extensions, no transport flags.
352
377
 
353
378
  > If you're new and want the agent to onboard itself, run
354
379
  > `llama agent-onboard` from the CLI or fetch the `agent_briefing` prompt from
355
- > the MCP server. It's the workflow contract — autonomy levels, attribution
356
- > grammar, error recovery, anti-pollution rules.
380
+ > the MCP server. It pulls the Command-owned workflow contract — current CLI
381
+ > guidance, attribution grammar, error recovery, and anti-pollution rules.
357
382
 
358
383
  ---
359
384
 
package/README.zh-CN.md CHANGED
@@ -299,9 +299,10 @@ printf '%s\n' \
299
299
  ```
300
300
 
301
301
  认证链跟 CLI 完全一样(gcloud → `$LLAMA_TOKEN` → `~/.llama/token`)。
302
- `agent_briefing` 这个 MCP **prompt** 还会原样吐出
303
- [`AGENT_BRIEFING.md`](AGENT_BRIEFING.md)——刚装上 server 的 agent
304
- 不用离开协议就能给自己 onboard。
302
+ `agent_briefing` 这个 MCP **prompt** 会在认证后拉取 Command 服务端的
303
+ Agent Runtime Contract——刚装上 server 的 agent 不用离开协议就能给自己
304
+ onboard。包内 [`AGENT_BRIEFING.md`](AGENT_BRIEFING.md) 只是服务端 briefing
305
+ 临时不可用时的 fallback。
305
306
 
306
307
  要读当前 Llama OS skills,用 runtime tools:`agent_bootstrap`、
307
308
  `skills_search`、`skills_read`、`object_inspect`。公开 npm 包不打包私有
@@ -358,7 +359,7 @@ claude mcp add llama -- llama-mcp
358
359
  </details>
359
360
 
360
361
  > 想让 agent 自己 onboard?跑 `llama agent-onboard`,或者从 MCP server
361
- > 拉 `agent_briefing` prompt——那是 AI agent 的工作合约(autonomy 等级、
362
+ > 拉 `agent_briefing` prompt——它会拿 Command 服务端的工作合约(CLI 状态、
362
363
  > attribution 语法、错误恢复、anti-pollution 规则)。
363
364
 
364
365
  ---
package/bin/llama-mcp.mjs CHANGED
@@ -11,7 +11,14 @@ import { createRequire } from "module";
11
11
  import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
12
12
  import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
13
13
  import { z } from "zod";
14
- import { getAuthHeaders, readBriefing, request, requestSse } from "../lib/client.mjs";
14
+ import {
15
+ getAuthHeaders,
16
+ getLastAgentEvent,
17
+ readBriefing,
18
+ request,
19
+ requestSse,
20
+ setClientRuntime,
21
+ } from "../lib/client.mjs";
15
22
 
16
23
  const requireFromHere = createRequire(import.meta.url);
17
24
  const { version: PKG_VERSION } = requireFromHere("../package.json");
@@ -23,6 +30,8 @@ import {
23
30
  uploadExternalFile,
24
31
  } from "../lib/external.mjs";
25
32
 
33
+ setClientRuntime({ client: "mcp" });
34
+
26
35
  // Wrap a request() call into the MCP CallToolResult shape. Catches errors
27
36
  // (NO_AUTH / 401 / 5xx / network) and surfaces them as `isError: true`
28
37
  // content so the calling agent sees a clean error string instead of the
@@ -56,6 +65,20 @@ function splitSources(value) {
56
65
  .filter(Boolean);
57
66
  }
58
67
 
68
+ function expectedIds(value) {
69
+ const expected = { dealIds: [], wikiSlugs: [], raw: [] };
70
+ if (!value) return expected;
71
+ const items = Array.isArray(value) ? value : String(value).split(",");
72
+ for (const item of items.map((s) => String(s).trim()).filter(Boolean)) {
73
+ const [kind, ...rest] = item.split(":");
74
+ const id = rest.join(":").trim();
75
+ if (kind === "deal" && id) expected.dealIds.push(id);
76
+ else if ((kind === "wiki" || kind === "slug") && id) expected.wikiSlugs.push(id);
77
+ else expected.raw.push(item);
78
+ }
79
+ return expected;
80
+ }
81
+
59
82
  function buildEnrichmentAgentMessage(args = {}) {
60
83
  if (args.message) return String(args.message);
61
84
  const sources = splitSources(args.sources) ?? [
@@ -209,11 +232,47 @@ server.registerTool(
209
232
  },
210
233
  async ({ limit } = {}) => {
211
234
  const params = new URLSearchParams();
235
+ params.set("clientVersion", PKG_VERSION);
212
236
  if (limit) params.set("limit", String(limit));
213
237
  return callApi("GET", `/api/agent/manifest${params.toString() ? `?${params}` : ""}`);
214
238
  }
215
239
  );
216
240
 
241
+ server.registerTool(
242
+ "record_eval_feedback",
243
+ {
244
+ description:
245
+ "Mark the latest llama CLI/MCP result as good/bad or add a real query " +
246
+ "to the Golden Query Eval candidate pool. Use when the user says a " +
247
+ "Llama Command search/result was right, wrong, missing a source, or should be regression-tested.",
248
+ inputSchema: {
249
+ action: z.enum(["good", "bad", "add"]).describe("feedback action"),
250
+ eventId: z.number().optional().describe("agent_client_events id; defaults to latest local event"),
251
+ query: z.string().optional().describe("required for manual add when no source event is available"),
252
+ surface: z.string().optional().describe("deal, wiki, activity, people, or manual"),
253
+ expect: z
254
+ .union([z.string(), z.array(z.string())])
255
+ .optional()
256
+ .describe("expected ids like wiki:llamaos-weekly-2026-06-17 or deal:<uuid>"),
257
+ reason: z.string().optional().describe("why this was good/bad or should be tracked"),
258
+ privacyLevel: z.string().optional().describe("default internal"),
259
+ },
260
+ },
261
+ async ({ action, eventId, query, surface, expect, reason, privacyLevel }) => {
262
+ const last = getLastAgentEvent();
263
+ const body = {
264
+ action,
265
+ eventId: eventId ?? last?.lastEventId,
266
+ query,
267
+ surface: surface ?? last?.lastSurface,
268
+ expected: expectedIds(expect),
269
+ reason,
270
+ privacyLevel: privacyLevel ?? "internal",
271
+ };
272
+ return callApi("POST", "/api/agent/eval-feedback", body);
273
+ }
274
+ );
275
+
217
276
  server.registerTool(
218
277
  "skills_search",
219
278
  {
@@ -1595,11 +1654,12 @@ server.registerPrompt(
1595
1654
  "skills_search, and skills_read.",
1596
1655
  },
1597
1656
  async () => {
1598
- // Gate the briefing behind a /api/me check so unauthenticated MCP
1599
- // clients can't harvest internal workflow / command surface just by
1600
- // requesting the prompt. Mirrors the CLI gate in bin/llama.mjs.
1657
+ // Gate the briefing behind authenticated Command runtime. The server-owned
1658
+ // /api/agent/briefing contract is canonical; bundled AGENT_BRIEFING.md is
1659
+ // only a rollout/offline fallback for authenticated users.
1601
1660
  const headers = await getAuthHeaders();
1602
1661
  let stub = null;
1662
+ let briefing = null;
1603
1663
  if (Object.keys(headers).length === 0) {
1604
1664
  stub =
1605
1665
  "Llama Ventures team onboarding requires credentials.\n\n" +
@@ -1610,19 +1670,28 @@ server.registerPrompt(
1610
1670
  "Founder / external visitor: use the `pitch_*` tools — no token required.";
1611
1671
  } else {
1612
1672
  try {
1613
- await request("GET", "/api/me");
1673
+ const params = new URLSearchParams({ clientVersion: PKG_VERSION });
1674
+ const body = await request("GET", `/api/agent/briefing?${params}`);
1675
+ briefing = body?.briefing || null;
1614
1676
  } catch (err) {
1615
- stub =
1616
- "Llama Ventures team onboarding requires valid credentials. " +
1617
- "Server rejected the credentials we sent. Re-mint at " +
1618
- "https://command.llamaventures.vc/settings/tokens.";
1677
+ const msg = err?.message || "";
1678
+ if (msg.includes("Error[UNAUTHORIZED]") || msg.includes("Error[NO_AUTH]")) {
1679
+ stub =
1680
+ "Llama Ventures team onboarding requires valid credentials. " +
1681
+ "Server rejected the credentials we sent. Re-mint at " +
1682
+ "https://command.llamaventures.vc/settings/tokens.";
1683
+ } else {
1684
+ briefing =
1685
+ "Warning: server agent briefing unavailable; using bundled fallback.\n\n" +
1686
+ readBriefing();
1687
+ }
1619
1688
  }
1620
1689
  }
1621
1690
  return {
1622
1691
  messages: [
1623
1692
  {
1624
1693
  role: "user",
1625
- content: { type: "text", text: stub ?? readBriefing() },
1694
+ content: { type: "text", text: stub ?? briefing ?? readBriefing() },
1626
1695
  },
1627
1696
  ],
1628
1697
  };
package/bin/llama.mjs CHANGED
@@ -1,5 +1,6 @@
1
1
  #!/usr/bin/env node
2
2
 
3
+ import { createRequire } from "module";
3
4
  import readline from "readline";
4
5
  import {
5
6
  DEFAULT_BASE_URL,
@@ -9,6 +10,7 @@ import {
9
10
  TOKEN_FILE,
10
11
  getAuthHeaders,
11
12
  getBaseUrl,
13
+ getLastAgentEvent,
12
14
  getToken,
13
15
  print,
14
16
  readBriefing,
@@ -33,6 +35,9 @@ import { LLAMA_CLI_CLIENT_ID, pkceLoopbackFlow, revokeToken as revokeOAuthToken
33
35
  import { deleteBundle, detectBackend, readBundle, writeBundle } from "../lib/oauth-storage.mjs";
34
36
  import { maybeNudgeUpdate, getUpdateNudge } from "../lib/version-check.mjs";
35
37
 
38
+ const requireFromHere = createRequire(import.meta.url);
39
+ const { version: PKG_VERSION } = requireFromHere("../package.json");
40
+
36
41
  function parseFlags(args, knownFlags = null) {
37
42
  const flags = {};
38
43
  const positional = [];
@@ -71,6 +76,34 @@ function parseFlags(args, knownFlags = null) {
71
76
  return { flags, positional };
72
77
  }
73
78
 
79
+ function agentOnboardNoAuthMessage() {
80
+ return `Llama Ventures team onboarding requires credentials.
81
+
82
+ Team member?
83
+ - Run \`gcloud auth login\` with your @llamaventures.vc account, OR
84
+ - Mint a token at https://command.llamaventures.vc/settings/tokens
85
+ then \`llama token set <llc_...>\`.
86
+ Re-run \`llama agent-onboard\` after — the workflow contract will print.
87
+
88
+ Founder or external visitor (no Llama account)?
89
+ Run \`llama pitch start --name "Your Name" --email "you@company.com"\`
90
+ to chat with our intake agent — no token required.`;
91
+ }
92
+
93
+ function agentOnboardRejectedMessage() {
94
+ return `Llama Ventures team onboarding requires valid credentials.
95
+
96
+ Server rejected the credentials we sent. Re-mint at
97
+ https://command.llamaventures.vc/settings/tokens, run
98
+ \`llama token set <llc_...>\`, then re-run \`llama agent-onboard\`.`;
99
+ }
100
+
101
+ async function fetchServerAgentBriefing() {
102
+ const params = new URLSearchParams({ clientVersion: PKG_VERSION });
103
+ const result = await request("GET", `/api/agent/briefing?${params}`);
104
+ return result?.briefing || "";
105
+ }
106
+
74
107
  function closestKnownFlag(input, candidates) {
75
108
  let best = null;
76
109
  let bestScore = Infinity;
@@ -127,6 +160,45 @@ function slugifyTitle(title) {
127
160
  return slug;
128
161
  }
129
162
 
163
+ function parseExpectedIds(raw) {
164
+ const text = typeof raw === "string" ? raw : "";
165
+ const expected = { dealIds: [], wikiSlugs: [], raw: [] };
166
+ for (const item of text.split(",").map((s) => s.trim()).filter(Boolean)) {
167
+ const [kind, ...rest] = item.split(":");
168
+ const value = rest.join(":").trim();
169
+ if (kind === "deal" && value) expected.dealIds.push(value);
170
+ else if ((kind === "wiki" || kind === "slug") && value) expected.wikiSlugs.push(value);
171
+ else expected.raw.push(item);
172
+ }
173
+ return expected;
174
+ }
175
+
176
+ async function submitEvalFeedback(action, flags, queryText = "") {
177
+ const useLast = flags.last !== false;
178
+ const last = useLast ? getLastAgentEvent() : null;
179
+ const eventId =
180
+ flags.event && flags.event !== true
181
+ ? Number(flags.event)
182
+ : last?.lastEventId ?? null;
183
+ if ((action === "good" || action === "bad") && !eventId && !queryText) {
184
+ throw new Error(`Usage: llama eval ${action} [--last] [--reason "..."]`);
185
+ }
186
+ const body = {
187
+ action,
188
+ eventId: Number.isFinite(eventId) ? eventId : undefined,
189
+ query: queryText || undefined,
190
+ surface: flags.surface && flags.surface !== true ? String(flags.surface) : last?.lastSurface ?? undefined,
191
+ expected:
192
+ flags.expect && flags.expect !== true
193
+ ? parseExpectedIds(String(flags.expect))
194
+ : {},
195
+ reason: flags.reason && flags.reason !== true ? String(flags.reason) : undefined,
196
+ privacyLevel:
197
+ flags.privacy && flags.privacy !== true ? String(flags.privacy) : "internal",
198
+ };
199
+ return request("POST", "/api/agent/eval-feedback", body);
200
+ }
201
+
130
202
  // Client-side fuzzy match — used as a fallback when the server hasn't yet
131
203
  // shipped the search/filter API (Fix B, 2026-04-25). Once the server
132
204
  // returns the `{deals,total,limit,offset}` envelope, this path is never
@@ -282,6 +354,8 @@ Agent onboarding (run once on first install):
282
354
  llama skills search "pipeline update" # discover relevant runtime skills
283
355
  llama skills show llama-pipeline # read a skill from Command
284
356
  llama explain <url-or-object> # explain Command URL/object status + lifecycle
357
+ llama eval good|bad --last # mark the latest CLI/MCP result for eval
358
+ llama eval add "<query>" --expect wiki:<slug>|deal:<uuid>
285
359
 
286
360
  External pitch — talk to Llama Ventures' intake agent (no token required):
287
361
  llama pitch start --name "Jane Doe" --email "jane@acme.ai"
@@ -525,6 +599,7 @@ Common:
525
599
  llama agent bootstrap live Llama OS skill manifest from Command
526
600
  llama skills search "<query>" discover which skill to read
527
601
  llama explain <url-or-object> explain Command URLs, 404s, deleted objects
602
+ llama eval bad --last mark latest CLI/MCP result as an eval candidate
528
603
 
529
604
  Command groups — run \`llama help <group>\` for that group's commands:
530
605
  deal create · show · feed · update · enrich · search · collaborators · links · delete
@@ -532,6 +607,7 @@ Command groups — run \`llama help <group>\` for that group's commands:
532
607
  facts deal facts + skill corrections (the sourced, trust-rated layer)
533
608
  timeline timeline · posts · mentions
534
609
  wiki cross-deal knowledge entries (markdown or HTML)
610
+ eval mark real CLI/MCP searches good/bad or add a golden-query candidate
535
611
  memo long-form HTML investment memo
536
612
  html deal-specific HTML artifacts (/deals/<id>/browse/<slug>)
537
613
  pitch external founder intake (no token needed)
@@ -865,18 +941,15 @@ async function runPitchRepl() {
865
941
  async function main() {
866
942
  const [area, action, ...rest] = process.argv.slice(2);
867
943
  if (area === "--version" || area === "-v" || area === "version") {
868
- const { createRequire } = await import("module");
869
- const requireFromHere = createRequire(import.meta.url);
870
- const { version } = requireFromHere("../package.json");
871
944
  // `llama version --check` — explicitly check npm for a newer release and
872
945
  // print the upgrade line (or "up to date"). Lets an agent surface the
873
946
  // nudge on demand, separate from the throttled, TTY-gated auto-nudge.
874
947
  if (action === "--check" || action === "check") {
875
948
  const nudge = await getUpdateNudge();
876
- console.log(nudge || `llama CLI ${version} — up to date`);
949
+ console.log(nudge || `llama CLI ${PKG_VERSION} — up to date`);
877
950
  return;
878
951
  }
879
- console.log(version);
952
+ console.log(PKG_VERSION);
880
953
  return;
881
954
  }
882
955
  if (!area || area === "help" || area === "--help" || area === "-h") {
@@ -897,12 +970,13 @@ async function main() {
897
970
  return;
898
971
  }
899
972
 
900
- // `llama agent-onboard` — print the bundled AGENT_BRIEFING.md so an AI
901
- // agent reads it once and internalises the Llama Ventures workflow
902
- // contract. Same content the `agent_briefing` MCP prompt returns.
973
+ // `llama agent-onboard` — fetch the server-owned Agent Runtime Contract
974
+ // so an AI agent reads the current Llama Ventures workflow contract. The
975
+ // bundled AGENT_BRIEFING.md is now only a fallback when the server route
976
+ // is unavailable during rollout.
903
977
  // Also: `llama agent onboard` (two-word form) for symmetry.
904
978
  //
905
- // Gated behind /api/me — without valid credentials we print a short
979
+ // Gated behind Command auth — without valid credentials we print a short
906
980
  // bootstrap stub instead. Stops unauthenticated callers from harvesting
907
981
  // internal command surface / workflow conventions just by running the
908
982
  // public CLI.
@@ -912,39 +986,24 @@ async function main() {
912
986
  ) {
913
987
  const headers = await getAuthHeaders();
914
988
  if (Object.keys(headers).length === 0) {
915
- console.log(
916
- `Llama Ventures team onboarding requires credentials.
917
-
918
- Team member?
919
- - Run \`gcloud auth login\` with your @llamaventures.vc account, OR
920
- - Mint a token at https://command.llamaventures.vc/settings/tokens
921
- then \`llama token set <llc_...>\`.
922
- Re-run \`llama agent-onboard\` after — the workflow contract will print.
923
-
924
- Founder or external visitor (no Llama account)?
925
- Run \`llama pitch start --name "Your Name" --email "you@company.com"\`
926
- to chat with our intake agent — no token required.`
927
- );
989
+ console.log(agentOnboardNoAuthMessage());
928
990
  return;
929
991
  }
930
992
  try {
931
- await request("GET", "/api/me");
993
+ const briefing = await fetchServerAgentBriefing();
994
+ process.stdout.write(briefing || readBriefing());
932
995
  } catch (e) {
933
996
  const msg = e?.message || "";
934
997
  if (msg.includes("Error[UNAUTHORIZED]") || msg.includes("Error[NO_AUTH]")) {
935
- console.log(
936
- `Llama Ventures team onboarding requires valid credentials.
937
-
938
- Server rejected the credentials we sent. Re-mint at
939
- https://command.llamaventures.vc/settings/tokens, run
940
- \`llama token set <llc_...>\`, then re-run \`llama agent-onboard\`.`
941
- );
998
+ console.log(agentOnboardRejectedMessage());
942
999
  process.exitCode = 1;
943
1000
  return;
944
1001
  }
945
- throw e;
1002
+ process.stderr.write(
1003
+ `warning: server agent briefing unavailable (${msg}); using bundled fallback.\n`,
1004
+ );
1005
+ process.stdout.write(readBriefing());
946
1006
  }
947
- process.stdout.write(readBriefing());
948
1007
  return;
949
1008
  }
950
1009
 
@@ -954,6 +1013,7 @@ https://command.llamaventures.vc/settings/tokens, run
954
1013
  if (area === "agent" && action === "bootstrap") {
955
1014
  const { flags } = parseFlags(rest, ["json", "limit"]);
956
1015
  const params = new URLSearchParams();
1016
+ params.set("clientVersion", PKG_VERSION);
957
1017
  if (flags.limit && flags.limit !== true) params.set("limit", String(flags.limit));
958
1018
  const manifest = await request("GET", `/api/agent/manifest${params.toString() ? `?${params}` : ""}`);
959
1019
  if (flags.json) {
@@ -1037,6 +1097,43 @@ https://command.llamaventures.vc/settings/tokens, run
1037
1097
  return;
1038
1098
  }
1039
1099
 
1100
+ if (area === "eval") {
1101
+ const sub = action;
1102
+ if (sub === "good" || sub === "bad") {
1103
+ const { flags, positional } = parseFlags(rest, [
1104
+ "last",
1105
+ "event",
1106
+ "reason",
1107
+ "expect",
1108
+ "surface",
1109
+ "privacy",
1110
+ ]);
1111
+ const q = positional.join(" ").trim();
1112
+ print(await submitEvalFeedback(sub, flags, q));
1113
+ return;
1114
+ }
1115
+ if (sub === "add") {
1116
+ const { flags, positional } = parseFlags(rest, [
1117
+ "event",
1118
+ "expect",
1119
+ "reason",
1120
+ "surface",
1121
+ "privacy",
1122
+ ]);
1123
+ const q = positional.join(" ").trim();
1124
+ if (!q && !flags.event) {
1125
+ throw new Error(
1126
+ `Usage: llama eval add "<query>" --surface deal|wiki --expect wiki:<slug>|deal:<uuid>`,
1127
+ );
1128
+ }
1129
+ print(await submitEvalFeedback("add", flags, q));
1130
+ return;
1131
+ }
1132
+ throw new Error(
1133
+ "Usage: llama eval good|bad [--last] [--reason ...] OR llama eval add \"<query>\" --expect wiki:<slug>|deal:<uuid>",
1134
+ );
1135
+ }
1136
+
1040
1137
  // `llama pitch ...` — external founder-pitch family. No Llama token
1041
1138
  // required; bootstraps a session against /api/external/* via PoW + cookie.
1042
1139
  // See lib/external.mjs and AGENT_BRIEFING.md for the full surface.
package/lib/client.mjs CHANGED
@@ -13,6 +13,7 @@ import path from "path";
13
13
  import { fileURLToPath } from "url";
14
14
  import { execFile as _execFile } from "child_process";
15
15
  import { promisify } from "util";
16
+ import { randomUUID } from "crypto";
16
17
 
17
18
  const execFile = promisify(_execFile);
18
19
 
@@ -43,6 +44,19 @@ export function readBriefing() {
43
44
  }
44
45
  }
45
46
 
47
+ let packageVersionCache = null;
48
+
49
+ export function getPackageVersion() {
50
+ if (packageVersionCache) return packageVersionCache;
51
+ try {
52
+ const pkg = JSON.parse(fs.readFileSync(path.join(PACKAGE_ROOT, "package.json"), "utf8"));
53
+ packageVersionCache = String(pkg.version || "unknown");
54
+ } catch {
55
+ packageVersionCache = "unknown";
56
+ }
57
+ return packageVersionCache;
58
+ }
59
+
46
60
  // Canonical entrypoint. `llama-command.onrender.com` also serves the API
47
61
  // but its NextAuth callback URL doesn't match, so browser login (needed
48
62
  // to mint a token at /settings/tokens) fails there with a server-config
@@ -54,6 +68,7 @@ export const DEFAULT_BASE_URL = "https://command.llamaventures.vc";
54
68
  // agent-discovery convention.
55
69
  export const TOKEN_DIR = path.join(os.homedir(), ".llama");
56
70
  export const TOKEN_FILE = path.join(TOKEN_DIR, "token");
71
+ export const AGENT_SESSION_FILE = path.join(TOKEN_DIR, "agent-session.json");
57
72
 
58
73
  // Legacy location used by CLI v0.1. Read for back-compat (silent migrate
59
74
  // to canonical on first use); never written for the token, but still the
@@ -123,6 +138,269 @@ export function getToken() {
123
138
  return "";
124
139
  }
125
140
 
141
+ let runtimeClient = "cli";
142
+ let runtimeAgentClient = null;
143
+
144
+ export function setClientRuntime(opts = {}) {
145
+ if (opts.client) runtimeClient = String(opts.client);
146
+ if (opts.agentClient) runtimeAgentClient = String(opts.agentClient);
147
+ }
148
+
149
+ function detectAgentClient() {
150
+ if (runtimeAgentClient) return runtimeAgentClient;
151
+ if (process.env.LLAMA_AGENT_CLIENT) return process.env.LLAMA_AGENT_CLIENT;
152
+ if (process.env.CODEX_SANDBOX || process.env.CODEX_CLI || process.env.OPENAI_CODEX) return "codex";
153
+ if (process.env.CLAUDECODE || process.env.CLAUDE_CODE || process.env.CLAUDE_CODE_ENTRYPOINT) {
154
+ return "claude-code";
155
+ }
156
+ if (process.env.CURSOR_AGENT || process.env.CURSOR_TRACE_ID) return "cursor";
157
+ return "unknown";
158
+ }
159
+
160
+ export function readAgentSession() {
161
+ try {
162
+ return JSON.parse(fs.readFileSync(AGENT_SESSION_FILE, "utf8"));
163
+ } catch {
164
+ return {};
165
+ }
166
+ }
167
+
168
+ function writeAgentSession(session) {
169
+ try {
170
+ fs.mkdirSync(TOKEN_DIR, { recursive: true, mode: 0o700 });
171
+ fs.writeFileSync(AGENT_SESSION_FILE, `${JSON.stringify(session, null, 2)}\n`, { mode: 0o600 });
172
+ fs.chmodSync(AGENT_SESSION_FILE, 0o600);
173
+ } catch {
174
+ // Telemetry state is best-effort. Never break the actual CLI command.
175
+ }
176
+ }
177
+
178
+ function currentAgentSessionId() {
179
+ const session = readAgentSession();
180
+ if (session.sessionId) return session.sessionId;
181
+ const created = {
182
+ sessionId: randomUUID(),
183
+ createdAt: new Date().toISOString(),
184
+ };
185
+ writeAgentSession(created);
186
+ return created.sessionId;
187
+ }
188
+
189
+ export function getLastAgentEvent() {
190
+ const session = readAgentSession();
191
+ return session.lastEventId ? session : null;
192
+ }
193
+
194
+ function rememberAgentEvent(event) {
195
+ if (!event?.eventId) return;
196
+ const session = {
197
+ ...readAgentSession(),
198
+ sessionId: event.sessionId || currentAgentSessionId(),
199
+ lastEventId: event.eventId,
200
+ lastCandidateId: event.candidateId ?? null,
201
+ lastCommand: event.command ?? null,
202
+ lastQuery: event.query ?? null,
203
+ lastSurface: event.surface ?? null,
204
+ lastRecordedAt: new Date().toISOString(),
205
+ };
206
+ writeAgentSession(session);
207
+ }
208
+
209
+ function agentClientHeaders(command) {
210
+ return {
211
+ "X-Llama-Client": runtimeClient,
212
+ "X-Llama-Client-Version": getPackageVersion(),
213
+ "X-Llama-Agent-Client": detectAgentClient(),
214
+ "X-Llama-Agent-Session": currentAgentSessionId(),
215
+ "X-Llama-Command": command || "unknown",
216
+ };
217
+ }
218
+
219
+ const SECRET_KEY_RE = /(token|secret|password|authorization|cookie|api[_-]?key|keychain|jwt)/i;
220
+
221
+ function truncateText(text, max = 2000) {
222
+ return text.length > max ? `${text.slice(0, max)}...[truncated]` : text;
223
+ }
224
+
225
+ function sanitizeTelemetryValue(value, depth = 0) {
226
+ if (depth > 4) return "[max-depth]";
227
+ if (value === null || value === undefined) return value;
228
+ if (typeof value === "string") return truncateText(value);
229
+ if (typeof value === "number" || typeof value === "boolean") return value;
230
+ if (Array.isArray(value)) {
231
+ return value.slice(0, 20).map((item) => sanitizeTelemetryValue(item, depth + 1));
232
+ }
233
+ if (typeof value === "object") {
234
+ const out = {};
235
+ for (const [key, val] of Object.entries(value).slice(0, 40)) {
236
+ out[key] = SECRET_KEY_RE.test(key) ? "[redacted]" : sanitizeTelemetryValue(val, depth + 1);
237
+ }
238
+ return out;
239
+ }
240
+ return String(value);
241
+ }
242
+
243
+ function parseEndpoint(endpoint) {
244
+ try {
245
+ return new URL(endpoint, "https://command.llamaventures.vc");
246
+ } catch {
247
+ return null;
248
+ }
249
+ }
250
+
251
+ function endpointArgs(endpoint, body) {
252
+ const args = {};
253
+ const url = parseEndpoint(endpoint);
254
+ if (url) {
255
+ for (const [key, value] of url.searchParams.entries()) args[key] = value;
256
+ }
257
+ if (body && typeof body === "object" && !Array.isArray(body)) {
258
+ Object.assign(args, body);
259
+ }
260
+ return sanitizeTelemetryValue(args);
261
+ }
262
+
263
+ function inferCommand(method, endpoint) {
264
+ const url = parseEndpoint(endpoint);
265
+ const pathname = url?.pathname || endpoint.split("?")[0] || "";
266
+ const verb = String(method || "GET").toUpperCase();
267
+ if (pathname === "/api/agent/client-events") return "telemetry.record";
268
+ if (pathname === "/api/agent/eval-feedback") return "eval.feedback";
269
+ if (pathname === "/api/wiki/search") return "wiki.search";
270
+ if (pathname === "/api/wiki/save") return "wiki.save";
271
+ if (/^\/api\/wiki\/[^/]+$/.test(pathname)) return verb === "GET" ? "wiki.read" : "wiki.write";
272
+ if (pathname === "/api/deals") return verb === "GET" ? "deal.search" : "deal.write";
273
+ if (pathname === "/api/deals/create") return "deal.create";
274
+ if (pathname === "/api/deals/update") return "deal.update";
275
+ if (/^\/api\/deals\/[^/]+\/threads\/[^/]+$/.test(pathname)) return "deal.agent.run";
276
+ if (/^\/api\/deals\/[^/]+\/threads$/.test(pathname)) return "deal.thread.create";
277
+ if (/^\/api\/deals\/[^/]+\/facts/.test(pathname)) return verb === "GET" ? "deal.fact.list" : "deal.fact.write";
278
+ if (/^\/api\/deals\/[^/]+\/posts$/.test(pathname)) return "deal.post";
279
+ if (/^\/api\/deals\/[^/]+\/blocks/.test(pathname)) return verb === "GET" ? "brief.blocks" : "brief.write";
280
+ if (/^\/api\/deals\/[^/]+$/.test(pathname)) return verb === "GET" ? "deal.show" : "deal.write";
281
+ if (pathname === "/api/me") return "auth.status";
282
+ if (pathname.startsWith("/api/agent/skills")) return "skills.read";
283
+ if (pathname === "/api/agent/manifest") return "agent.bootstrap";
284
+ if (pathname === "/api/agent/briefing") return "agent.briefing";
285
+ return `${verb.toLowerCase()} ${pathname || endpoint}`;
286
+ }
287
+
288
+ function queryForCommand(command, args) {
289
+ if (!command.endsWith(".search")) return null;
290
+ return args?.q || args?.search || args?.query || null;
291
+ }
292
+
293
+ function summarizeResultIds(data) {
294
+ const result = {};
295
+ const topDeals = [];
296
+ const topWiki = [];
297
+
298
+ const collectDeal = (deal) => {
299
+ const id = deal?.uuid || deal?.id || deal?.dealId || deal?.deal_uuid;
300
+ if (!id) return;
301
+ topDeals.push({ id, name: deal.companyName || deal.company_name || deal.name || null });
302
+ };
303
+ const collectWiki = (item) => {
304
+ if (!item?.slug) return;
305
+ topWiki.push({ slug: item.slug, title: item.title || null });
306
+ };
307
+
308
+ if (Array.isArray(data)) {
309
+ for (const item of data.slice(0, 20)) {
310
+ collectDeal(item);
311
+ collectWiki(item);
312
+ }
313
+ result.resultCount = data.length;
314
+ } else if (data && typeof data === "object") {
315
+ const deals = Array.isArray(data.deals) ? data.deals : [];
316
+ const articles = Array.isArray(data.articles) ? data.articles : [];
317
+ const results = Array.isArray(data.results) ? data.results : [];
318
+ for (const deal of deals.slice(0, 20)) collectDeal(deal);
319
+ for (const item of [...articles, ...results].slice(0, 20)) collectWiki(item);
320
+ if (typeof data.total === "number") result.total = data.total;
321
+ if (deals.length) result.resultCount = deals.length;
322
+ if (articles.length || results.length) result.resultCount = articles.length + results.length;
323
+ }
324
+
325
+ if (topDeals.length) result.deals = topDeals;
326
+ if (topWiki.length) result.wiki = topWiki;
327
+ return result;
328
+ }
329
+
330
+ function summarizeResult(data) {
331
+ if (data === null || data === undefined) return null;
332
+ if (Array.isArray(data)) return `${data.length} result(s)`;
333
+ if (typeof data === "object") {
334
+ if (Array.isArray(data.deals)) return `${data.deals.length} deal result(s); total=${data.total ?? "unknown"}`;
335
+ if (Array.isArray(data.results)) return `${data.results.length} result(s)`;
336
+ if (data.ok !== undefined) return `ok=${Boolean(data.ok)}`;
337
+ }
338
+ return truncateText(typeof data === "string" ? data : JSON.stringify(sanitizeTelemetryValue(data)), 2000);
339
+ }
340
+
341
+ function shouldSkipTelemetry(endpoint) {
342
+ if (process.env.LLAMA_TELEMETRY === "0") return true;
343
+ const pathname = parseEndpoint(endpoint)?.pathname || endpoint;
344
+ return pathname === "/api/agent/client-events" || pathname === "/api/agent/eval-feedback";
345
+ }
346
+
347
+ async function recordClientTelemetry({
348
+ authHeaders,
349
+ method,
350
+ endpoint,
351
+ body,
352
+ command,
353
+ status,
354
+ httpStatus,
355
+ latencyMs,
356
+ data,
357
+ errorMessage,
358
+ }) {
359
+ if (shouldSkipTelemetry(endpoint)) return;
360
+ const args = endpointArgs(endpoint, body);
361
+ const sessionId = currentAgentSessionId();
362
+ const payload = {
363
+ client: runtimeClient,
364
+ clientVersion: getPackageVersion(),
365
+ agentClient: detectAgentClient(),
366
+ sessionId,
367
+ command,
368
+ method: String(method || "GET").toUpperCase(),
369
+ endpoint,
370
+ status,
371
+ httpStatus,
372
+ latencyMs,
373
+ args,
374
+ query: queryForCommand(command, args),
375
+ resultSummary: status === "success" ? summarizeResult(data) : null,
376
+ resultIds: status === "success" ? summarizeResultIds(data) : {},
377
+ errorMessage: errorMessage ? truncateText(String(errorMessage), 2000) : null,
378
+ };
379
+ try {
380
+ const res = await fetch(`${getBaseUrl()}/api/agent/client-events`, {
381
+ method: "POST",
382
+ headers: {
383
+ "Content-Type": "application/json",
384
+ ...agentClientHeaders("telemetry.record"),
385
+ ...authHeaders,
386
+ },
387
+ body: JSON.stringify(payload),
388
+ });
389
+ if (!res.ok) return;
390
+ const recorded = await res.json().catch(() => null);
391
+ rememberAgentEvent({
392
+ ...recorded,
393
+ sessionId,
394
+ command,
395
+ query: payload.query,
396
+ surface: command.startsWith("deal.") ? "deal" : command.startsWith("wiki.") ? "wiki" : null,
397
+ });
398
+ } catch {
399
+ // Best-effort by design. The actual llama command already succeeded or
400
+ // failed; telemetry must never alter that outcome.
401
+ }
402
+ }
403
+
126
404
  // Try `gcloud auth print-identity-token`. Returns the JWT or null. Zero-config
127
405
  // win for any team member who has gcloud + their @llamaventures.vc account
128
406
  // already set up — the server's Bearer auth path verifies and auto-creates
@@ -227,10 +505,13 @@ export async function requestSse(method, endpoint, body, opts = {}) {
227
505
  async function requestWithRetry(method, endpoint, body, allowRetry) {
228
506
  const authHeaders = await getAuthHeaders();
229
507
  if (Object.keys(authHeaders).length === 0) throw noAuthError();
508
+ const command = inferCommand(method, endpoint);
509
+ const start = Date.now();
230
510
  const res = await fetch(`${getBaseUrl()}${endpoint}`, {
231
511
  method,
232
512
  headers: {
233
513
  "Content-Type": "application/json",
514
+ ...agentClientHeaders(command),
234
515
  ...authHeaders,
235
516
  },
236
517
  body: body === undefined ? undefined : JSON.stringify(body),
@@ -267,18 +548,45 @@ async function requestWithRetry(method, endpoint, body, allowRetry) {
267
548
  }
268
549
  if (!res.ok) {
269
550
  const message = typeof data === "object" && data?.error ? data.error : `HTTP ${res.status}`;
551
+ await recordClientTelemetry({
552
+ authHeaders,
553
+ method,
554
+ endpoint,
555
+ body,
556
+ command,
557
+ status: "error",
558
+ httpStatus: res.status,
559
+ latencyMs: Date.now() - start,
560
+ data: null,
561
+ errorMessage: message,
562
+ });
270
563
  throw new Error(message);
271
564
  }
565
+ await recordClientTelemetry({
566
+ authHeaders,
567
+ method,
568
+ endpoint,
569
+ body,
570
+ command,
571
+ status: "success",
572
+ httpStatus: res.status,
573
+ latencyMs: Date.now() - start,
574
+ data,
575
+ errorMessage: null,
576
+ });
272
577
  return data;
273
578
  }
274
579
 
275
580
  async function requestSseWithRetry(method, endpoint, body, opts, allowRetry) {
276
581
  const authHeaders = await getAuthHeaders();
277
582
  if (Object.keys(authHeaders).length === 0) throw noAuthError();
583
+ const command = inferCommand(method, endpoint);
584
+ const start = Date.now();
278
585
  const res = await fetch(`${getBaseUrl()}${endpoint}`, {
279
586
  method,
280
587
  headers: {
281
588
  "Content-Type": "application/json",
589
+ ...agentClientHeaders(command),
282
590
  ...authHeaders,
283
591
  },
284
592
  body: body === undefined ? undefined : JSON.stringify(body),
@@ -309,6 +617,18 @@ async function requestSseWithRetry(method, endpoint, body, opts, allowRetry) {
309
617
  data = text;
310
618
  }
311
619
  const message = typeof data === "object" && data?.error ? data.error : `HTTP ${res.status}`;
620
+ await recordClientTelemetry({
621
+ authHeaders,
622
+ method,
623
+ endpoint,
624
+ body,
625
+ command,
626
+ status: "error",
627
+ httpStatus: res.status,
628
+ latencyMs: Date.now() - start,
629
+ data: null,
630
+ errorMessage: message,
631
+ });
312
632
  throw new Error(message);
313
633
  }
314
634
 
@@ -343,6 +663,18 @@ async function requestSseWithRetry(method, endpoint, body, opts, allowRetry) {
343
663
  for (const frame of frames) handleFrame(frame);
344
664
  }
345
665
  if (buf.trim()) handleFrame(buf);
666
+ await recordClientTelemetry({
667
+ authHeaders,
668
+ method,
669
+ endpoint,
670
+ body,
671
+ command,
672
+ status: "success",
673
+ httpStatus: res.status,
674
+ latencyMs: Date.now() - start,
675
+ data: { ok: true, textLength: text.length, events: events.length },
676
+ errorMessage: null,
677
+ });
346
678
  return { text, events };
347
679
  }
348
680
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@llamaventures/cli",
3
- "version": "1.15.1",
3
+ "version": "1.16.0",
4
4
  "description": "CLI + MCP server for the Llama Ventures investment workbench (command.llamaventures.vc).",
5
5
  "type": "module",
6
6
  "scripts": {
@@ -22,6 +22,7 @@ assert.equal(
22
22
  );
23
23
  const calls = [];
24
24
  let threadSeq = 0;
25
+ let eventSeq = 0;
25
26
 
26
27
  async function readJson(req) {
27
28
  let raw = "";
@@ -65,11 +66,47 @@ const server = createServer(async (req, res) => {
65
66
  path: url.pathname,
66
67
  query: Object.fromEntries(url.searchParams.entries()),
67
68
  body,
69
+ headers: {
70
+ client: req.headers["x-llama-client"] ?? null,
71
+ clientVersion: req.headers["x-llama-client-version"] ?? null,
72
+ agentClient: req.headers["x-llama-agent-client"] ?? null,
73
+ session: req.headers["x-llama-agent-session"] ?? null,
74
+ command: req.headers["x-llama-command"] ?? null,
75
+ },
68
76
  });
69
77
 
78
+ if (req.method === "POST" && url.pathname === "/api/agent/client-events") {
79
+ eventSeq += 1;
80
+ writeJson(res, {
81
+ ok: true,
82
+ eventId: eventSeq,
83
+ candidateId: body?.command?.endsWith(".search") ? eventSeq + 1000 : null,
84
+ });
85
+ return;
86
+ }
87
+
88
+ if (req.method === "POST" && url.pathname === "/api/agent/eval-feedback") {
89
+ writeJson(res, {
90
+ ok: true,
91
+ candidate: {
92
+ id: 42,
93
+ source_event_id: body?.eventId ?? null,
94
+ feedback: body?.action ?? null,
95
+ },
96
+ });
97
+ return;
98
+ }
99
+
70
100
  if (req.method === "GET" && url.pathname === "/api/agent/manifest") {
71
101
  writeJson(res, {
72
102
  ok: true,
103
+ contract: {
104
+ contract_version: "agent-contract.v1",
105
+ cli: {
106
+ client_version: url.searchParams.get("clientVersion"),
107
+ status: "ok",
108
+ },
109
+ },
73
110
  briefing: "runtime briefing: use skills_search, skills_read, and object_inspect",
74
111
  llama_os: {
75
112
  visible_skill_count: 49,
@@ -85,6 +122,21 @@ const server = createServer(async (req, res) => {
85
122
  return;
86
123
  }
87
124
 
125
+ if (req.method === "GET" && url.pathname === "/api/agent/briefing") {
126
+ writeJson(res, {
127
+ ok: true,
128
+ contract: {
129
+ contract_version: "agent-contract.v1",
130
+ cli: {
131
+ client_version: url.searchParams.get("clientVersion"),
132
+ status: "ok",
133
+ },
134
+ },
135
+ briefing: "server-owned briefing: check CLI, use Pipeline First, prefer CLI/MCP",
136
+ });
137
+ return;
138
+ }
139
+
88
140
  if (req.method === "GET" && url.pathname === "/api/agent/skills") {
89
141
  writeJson(res, {
90
142
  ok: true,
@@ -136,6 +188,16 @@ const server = createServer(async (req, res) => {
136
188
  return;
137
189
  }
138
190
 
191
+ if (req.method === "GET" && url.pathname === "/api/wiki/search") {
192
+ writeJson(res, [
193
+ {
194
+ slug: "llama-weekly-2026-06-16",
195
+ title: "Llama Weekly 2026-06-16",
196
+ },
197
+ ]);
198
+ return;
199
+ }
200
+
139
201
  if (req.method === "POST" && /^\/api\/deals\/[^/]+\/threads$/.test(url.pathname)) {
140
202
  threadSeq += 1;
141
203
  writeJson(res, { id: `thread-${threadSeq}` });
@@ -191,25 +253,34 @@ function resetCalls() {
191
253
  threadSeq = 0;
192
254
  }
193
255
 
256
+ function businessCalls() {
257
+ return calls.filter((call) => call.path !== "/api/agent/client-events");
258
+ }
259
+
260
+ function telemetryCalls() {
261
+ return calls.filter((call) => call.path === "/api/agent/client-events");
262
+ }
263
+
194
264
  function paths() {
195
- return calls.map((call) => `${call.method} ${call.path}`);
265
+ return businessCalls().map((call) => `${call.method} ${call.path}`);
196
266
  }
197
267
 
198
268
  function assertNoEnrichCall() {
199
269
  assert.equal(
200
- calls.some((call) => call.path.endsWith("/enrich")),
270
+ businessCalls().some((call) => call.path.endsWith("/enrich")),
201
271
  false,
202
272
  `expected no /enrich call, got ${paths().join(", ")}`,
203
273
  );
204
274
  }
205
275
 
206
276
  function assertThreadRun({ title, messageIncludes }) {
207
- assert.equal(calls.length, 2, `expected thread create + SSE run, got ${paths().join(", ")}`);
208
- assert.match(calls[0].path, /^\/api\/deals\/[^/]+\/threads$/);
209
- assert.equal(calls[0].body?.title, title);
210
- assert.match(calls[1].path, /^\/api\/deals\/[^/]+\/threads\/thread-1$/);
277
+ const relevant = businessCalls();
278
+ assert.equal(relevant.length, 2, `expected thread create + SSE run, got ${paths().join(", ")}`);
279
+ assert.match(relevant[0].path, /^\/api\/deals\/[^/]+\/threads$/);
280
+ assert.equal(relevant[0].body?.title, title);
281
+ assert.match(relevant[1].path, /^\/api\/deals\/[^/]+\/threads\/thread-1$/);
211
282
  for (const needle of messageIncludes) {
212
- assert.match(calls[1].body?.message ?? "", new RegExp(escapeRegExp(needle)));
283
+ assert.match(relevant[1].body?.message ?? "", new RegExp(escapeRegExp(needle)));
213
284
  }
214
285
  }
215
286
 
@@ -313,18 +384,29 @@ const baseUrl = `http://${address.address}:${address.port}`;
313
384
  const homeDir = await mkdtemp(path.join(os.tmpdir(), "llama-cli-routing-"));
314
385
 
315
386
  try {
387
+ resetCalls();
388
+ const onboardRun = await runCli(["agent-onboard"], baseUrl, homeDir);
389
+ assert.match(onboardRun.stdout, /server-owned briefing/);
390
+ assert.deepEqual(paths(), ["GET /api/agent/briefing"]);
391
+ assert.ok(businessCalls()[0].query.clientVersion, "agent-onboard passes clientVersion");
392
+ assert.equal(telemetryCalls()[0].body?.command, "agent.briefing");
393
+ assert.equal(telemetryCalls()[0].body?.client, "cli");
394
+ assert.ok(telemetryCalls()[0].body?.sessionId, "telemetry includes an agent session id");
395
+ assert.equal(businessCalls()[0].headers.command, "agent.briefing");
396
+
316
397
  resetCalls();
317
398
  const bootstrapRun = await runCli(["agent", "bootstrap", "--limit", "3"], baseUrl, homeDir);
318
399
  assert.match(bootstrapRun.stdout, /runtime briefing/);
319
400
  assert.deepEqual(paths(), ["GET /api/agent/manifest"]);
320
- assert.equal(calls[0].query.limit, "3");
401
+ assert.equal(businessCalls()[0].query.limit, "3");
402
+ assert.ok(businessCalls()[0].query.clientVersion, "agent bootstrap passes clientVersion");
321
403
 
322
404
  resetCalls();
323
405
  const skillSearchRun = await runCli(["skills", "search", "pipeline", "--limit", "5"], baseUrl, homeDir);
324
406
  assert.match(skillSearchRun.stdout, /llama-command/);
325
407
  assert.deepEqual(paths(), ["GET /api/agent/skills"]);
326
- assert.equal(calls[0].query.q, "pipeline");
327
- assert.equal(calls[0].query.limit, "5");
408
+ assert.equal(businessCalls()[0].query.q, "pipeline");
409
+ assert.equal(businessCalls()[0].query.limit, "5");
328
410
 
329
411
  resetCalls();
330
412
  const skillShowRun = await runCli(["skills", "show", "llama-command"], baseUrl, homeDir);
@@ -336,7 +418,34 @@ try {
336
418
  assert.match(explainRun.stdout, /Status: deleted/);
337
419
  assert.match(explainRun.stdout, /Deleted by Kevin Yu/);
338
420
  assert.deepEqual(paths(), ["GET /api/agent/explain"]);
339
- assert.equal(calls[0].query.q, "https://command.llamaventures.vc/wiki/missing-page");
421
+ assert.equal(businessCalls()[0].query.q, "https://command.llamaventures.vc/wiki/missing-page");
422
+
423
+ resetCalls();
424
+ const wikiRun = await runCli(["wiki", "search", "llama weekly"], baseUrl, homeDir);
425
+ assert.match(wikiRun.stdout, /llama-weekly-2026-06-16/);
426
+ assert.deepEqual(paths(), ["GET /api/wiki/search"]);
427
+ assert.equal(telemetryCalls()[0].body?.command, "wiki.search");
428
+ assert.equal(telemetryCalls()[0].body?.query, "llama weekly");
429
+
430
+ resetCalls();
431
+ const evalRun = await runCli(
432
+ [
433
+ "eval",
434
+ "bad",
435
+ "--last",
436
+ "--reason",
437
+ "missed dev weekly",
438
+ "--expect",
439
+ "wiki:llamaos-weekly-2026-06-17",
440
+ ],
441
+ baseUrl,
442
+ homeDir,
443
+ );
444
+ assert.match(evalRun.stdout, /"feedback": "bad"/);
445
+ assert.deepEqual(paths(), ["POST /api/agent/eval-feedback"]);
446
+ assert.equal(businessCalls()[0].body?.action, "bad");
447
+ assert.equal(businessCalls()[0].body?.eventId, 6);
448
+ assert.equal(businessCalls()[0].body?.expected?.wikiSlugs?.[0], "llamaos-weekly-2026-06-17");
340
449
 
341
450
  resetCalls();
342
451
  const enrichRun = await runCli(
@@ -369,9 +478,9 @@ try {
369
478
  homeDir,
370
479
  );
371
480
  assert.deepEqual(paths(), ["POST /api/deals/deal-cli/enrich"]);
372
- assert.equal(calls[0].body?.apply, true);
373
- assert.equal(calls[0].body?.dryRun, false);
374
- assert.equal(calls[0].body?.executor, "server_agent");
481
+ assert.equal(businessCalls()[0].body?.apply, true);
482
+ assert.equal(businessCalls()[0].body?.dryRun, false);
483
+ assert.equal(businessCalls()[0].body?.executor, "server_agent");
375
484
 
376
485
  resetCalls();
377
486
  const agentRun = await runCli(
@@ -414,14 +523,16 @@ try {
414
523
  const bootstrapPayload = JSON.parse(mcpBootstrap.content?.[0]?.text ?? "{}");
415
524
  assert.equal(bootstrapPayload.ok, true);
416
525
  assert.deepEqual(paths(), ["GET /api/agent/manifest"]);
417
- assert.equal(calls[0].query.limit, "2");
526
+ assert.equal(businessCalls()[0].query.limit, "2");
527
+ assert.ok(businessCalls()[0].query.clientVersion, "mcp agent_bootstrap passes clientVersion");
528
+ assert.equal(telemetryCalls()[0].body?.client, "mcp");
418
529
 
419
530
  resetCalls();
420
531
  const mcpSkills = await callMcpTool("skills_search", { q: "command", limit: 4 }, baseUrl, homeDir);
421
532
  const skillsPayload = JSON.parse(mcpSkills.content?.[0]?.text ?? "{}");
422
533
  assert.equal(skillsPayload.skills?.[0]?.slug, "llama-command");
423
534
  assert.deepEqual(paths(), ["GET /api/agent/skills"]);
424
- assert.equal(calls[0].query.q, "command");
535
+ assert.equal(businessCalls()[0].query.q, "command");
425
536
 
426
537
  resetCalls();
427
538
  const mcpSkillRead = await callMcpTool("skills_read", { slug: "llama-command" }, baseUrl, homeDir);