@llamaventures/cli 1.15.1 → 1.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENT_BRIEFING.md +5 -4
- package/CHANGELOG.md +21 -0
- package/README.md +31 -6
- package/README.zh-CN.md +5 -4
- package/bin/llama-mcp.mjs +79 -10
- package/bin/llama.mjs +129 -32
- package/lib/client.mjs +332 -0
- package/package.json +1 -1
- package/scripts/verify-agent-routing.mjs +127 -16
package/AGENT_BRIEFING.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Llama Ventures Agent Briefing
|
|
2
2
|
|
|
3
|
-
You've been onboarded as a teammate of [Llama Ventures](https://llamaventures.vc) via the `@llamaventures/cli` package. This briefing is
|
|
3
|
+
You've been onboarded as a teammate of [Llama Ventures](https://llamaventures.vc) via the `@llamaventures/cli` package. This bundled briefing is a fallback copy of the behavioural contract. When authenticated, prefer the server-owned contract from `llama agent-onboard`, `llama agent bootstrap`, or MCP `agent_briefing`; read it once, internalise it, and operate accordingly. The user shouldn't have to explain any of this to you again.
|
|
4
4
|
|
|
5
5
|
You are not just an AI assistant. You're an **extension of a team member** — with CLI access to the Llama Command pipeline, write permission on shared data, and audit-log responsibility. Treat the status seriously.
|
|
6
6
|
|
|
@@ -25,6 +25,7 @@ Most teammates don't know everything this CLI can do. Part of your job is to sur
|
|
|
25
25
|
This npm package is public, but Llama OS skills are private. Do not assume the skill text is bundled locally. For team-token sessions, discover the live runtime library through Llama Command:
|
|
26
26
|
|
|
27
27
|
- Start with `llama agent bootstrap` or MCP `agent_bootstrap` when you need the current Command + Llama OS contract.
|
|
28
|
+
- Use `llama agent-onboard` or MCP `agent_briefing` for the server-owned Agent Runtime Contract. The npm-bundled text is fallback only.
|
|
28
29
|
- Use `llama skills search "<task>"` or MCP `skills_search` before choosing a Llama workflow.
|
|
29
30
|
- Use `llama skills show <slug>` or MCP `skills_read` only for the relevant skill.
|
|
30
31
|
- Use `llama explain <command-url-or-object>` or MCP `object_inspect` for 404s, deleted wiki pages, notifier links, deal URLs, and unknown Command objects before telling the user "the system is broken."
|
|
@@ -337,9 +338,9 @@ the intake agent; relay the conversation, then surface the verdict.
|
|
|
337
338
|
|
|
338
339
|
## Where this content lives
|
|
339
340
|
|
|
340
|
-
-
|
|
341
|
-
-
|
|
342
|
-
-
|
|
341
|
+
- Canonical when authenticated: `GET /api/agent/briefing` via `llama agent-onboard` or MCP `agent_briefing`
|
|
342
|
+
- Runtime home screen: `GET /api/agent/manifest` via `llama agent bootstrap` or MCP `agent_bootstrap`
|
|
343
|
+
- Fallback copy: `AGENT_BRIEFING.md` in the `@llamaventures/cli` npm package
|
|
343
344
|
- Public source: <https://github.com/Llama-Ventures/llama-cli/blob/main/AGENT_BRIEFING.md>
|
|
344
345
|
|
|
345
346
|
You're now a Llama Ventures teammate (or an external agent helping a
|
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,27 @@ this project adheres to [Semantic Versioning](https://semver.org).
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [1.16.0] - 2026-06-22
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
- CLI and MCP requests now send bounded local-agent telemetry metadata to
|
|
13
|
+
Llama Command: client kind/version, detected agent client, session id,
|
|
14
|
+
normalized command, latency, status, sanitized args, result ids, and bounded
|
|
15
|
+
result summaries.
|
|
16
|
+
- `llama eval good|bad --last` and `llama eval add "<query>" --expect ...`
|
|
17
|
+
let agents and humans turn real search usage into Golden Query Eval
|
|
18
|
+
feedback.
|
|
19
|
+
- MCP tool `record_eval_feedback` mirrors the CLI eval feedback path for
|
|
20
|
+
MCP-native agents.
|
|
21
|
+
|
|
22
|
+
### Changed
|
|
23
|
+
- `llama agent-onboard` and MCP `agent_briefing` now prefer the authenticated
|
|
24
|
+
Llama Command `/api/agent/briefing` runtime contract, with bundled
|
|
25
|
+
`AGENT_BRIEFING.md` retained as fallback only.
|
|
26
|
+
- `llama agent bootstrap` and MCP `agent_bootstrap` now pass the local CLI
|
|
27
|
+
version to Command so the server can return stale/ok CLI guidance as part of
|
|
28
|
+
the agent contract.
|
|
29
|
+
|
|
9
30
|
## [1.15.1] — 2026-06-16
|
|
10
31
|
|
|
11
32
|
### Changed
|
package/README.md
CHANGED
|
@@ -226,6 +226,8 @@ llama agent bootstrap
|
|
|
226
226
|
llama skills search "wiki delete tombstone"
|
|
227
227
|
llama skills show llama-command
|
|
228
228
|
llama explain https://command.llamaventures.vc/wiki/some-page
|
|
229
|
+
llama eval bad --last --reason "missed the llamaos weekly note"
|
|
230
|
+
llama eval add "last week llama dev weekly" --expect wiki:llamaos-weekly-2026-06-17
|
|
229
231
|
|
|
230
232
|
# Wiki
|
|
231
233
|
llama wiki search "<query>"
|
|
@@ -262,10 +264,31 @@ agents can pattern-match without parsing prose.
|
|
|
262
264
|
|
|
263
265
|
---
|
|
264
266
|
|
|
267
|
+
### Golden Query Eval feedback
|
|
268
|
+
|
|
269
|
+
The CLI and MCP server send lightweight client telemetry to Command for each
|
|
270
|
+
authenticated request: client kind/version, detected agent client, local session
|
|
271
|
+
id, normalized command, sanitized args, canonical result ids, status, latency,
|
|
272
|
+
and bounded summaries. It records what Llama Command was asked to do, not the
|
|
273
|
+
user's private Claude Code/Codex/Cursor conversation or local files.
|
|
274
|
+
|
|
275
|
+
Search commands automatically become eval candidates. Agents can mark the latest
|
|
276
|
+
result explicitly:
|
|
277
|
+
|
|
278
|
+
```bash
|
|
279
|
+
llama eval good --last
|
|
280
|
+
llama eval bad --last --reason "wrong top wiki"
|
|
281
|
+
llama eval add "AI陪伴" --surface deal --expect deal:<uuid>
|
|
282
|
+
```
|
|
283
|
+
|
|
284
|
+
MCP-native agents use `record_eval_feedback` for the same flow.
|
|
285
|
+
|
|
286
|
+
---
|
|
287
|
+
|
|
265
288
|
## MCP server
|
|
266
289
|
|
|
267
290
|
The bundled `llama-mcp` is a **stdio Model Context Protocol** server exposing
|
|
268
|
-
|
|
291
|
+
typed tools that mirror the most-used CLI surface. Every tool is named
|
|
269
292
|
and scoped — there is no generic API passthrough, by design (a public-package
|
|
270
293
|
escape hatch reachable from a prompt-injectable agent context is exactly the
|
|
271
294
|
shape we want to avoid).
|
|
@@ -290,9 +313,11 @@ printf '%s\n' \
|
|
|
290
313
|
```
|
|
291
314
|
|
|
292
315
|
Auth is identical to the CLI's chain (gcloud → `$LLAMA_TOKEN` → `~/.llama/token`).
|
|
293
|
-
The `agent_briefing` MCP **prompt**
|
|
294
|
-
|
|
295
|
-
|
|
316
|
+
The `agent_briefing` MCP **prompt** returns the server-owned Agent Runtime
|
|
317
|
+
Contract when authenticated, so any new agent loading the server can
|
|
318
|
+
self-onboard without leaving the protocol. The bundled
|
|
319
|
+
[`AGENT_BRIEFING.md`](AGENT_BRIEFING.md) is only a fallback if the server
|
|
320
|
+
briefing route is temporarily unavailable.
|
|
296
321
|
|
|
297
322
|
For current Llama OS skills, use the runtime tools instead of looking for a
|
|
298
323
|
local private repo: `agent_bootstrap`, `skills_search`, `skills_read`, and
|
|
@@ -352,8 +377,8 @@ extensions, no transport flags.
|
|
|
352
377
|
|
|
353
378
|
> If you're new and want the agent to onboard itself, run
|
|
354
379
|
> `llama agent-onboard` from the CLI or fetch the `agent_briefing` prompt from
|
|
355
|
-
> the MCP server. It
|
|
356
|
-
> grammar, error recovery, anti-pollution rules.
|
|
380
|
+
> the MCP server. It pulls the Command-owned workflow contract — current CLI
|
|
381
|
+
> guidance, attribution grammar, error recovery, and anti-pollution rules.
|
|
357
382
|
|
|
358
383
|
---
|
|
359
384
|
|
package/README.zh-CN.md
CHANGED
|
@@ -299,9 +299,10 @@ printf '%s\n' \
|
|
|
299
299
|
```
|
|
300
300
|
|
|
301
301
|
认证链跟 CLI 完全一样(gcloud → `$LLAMA_TOKEN` → `~/.llama/token`)。
|
|
302
|
-
`agent_briefing` 这个 MCP **prompt**
|
|
303
|
-
|
|
304
|
-
|
|
302
|
+
`agent_briefing` 这个 MCP **prompt** 会在认证后拉取 Command 服务端的
|
|
303
|
+
Agent Runtime Contract——刚装上 server 的 agent 不用离开协议就能给自己
|
|
304
|
+
onboard。包内 [`AGENT_BRIEFING.md`](AGENT_BRIEFING.md) 只是服务端 briefing
|
|
305
|
+
临时不可用时的 fallback。
|
|
305
306
|
|
|
306
307
|
要读当前 Llama OS skills,用 runtime tools:`agent_bootstrap`、
|
|
307
308
|
`skills_search`、`skills_read`、`object_inspect`。公开 npm 包不打包私有
|
|
@@ -358,7 +359,7 @@ claude mcp add llama -- llama-mcp
|
|
|
358
359
|
</details>
|
|
359
360
|
|
|
360
361
|
> 想让 agent 自己 onboard?跑 `llama agent-onboard`,或者从 MCP server
|
|
361
|
-
> 拉 `agent_briefing` prompt
|
|
362
|
+
> 拉 `agent_briefing` prompt——它会拿 Command 服务端的工作合约(CLI 状态、
|
|
362
363
|
> attribution 语法、错误恢复、anti-pollution 规则)。
|
|
363
364
|
|
|
364
365
|
---
|
package/bin/llama-mcp.mjs
CHANGED
|
@@ -11,7 +11,14 @@ import { createRequire } from "module";
|
|
|
11
11
|
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
|
|
12
12
|
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
|
|
13
13
|
import { z } from "zod";
|
|
14
|
-
import {
|
|
14
|
+
import {
|
|
15
|
+
getAuthHeaders,
|
|
16
|
+
getLastAgentEvent,
|
|
17
|
+
readBriefing,
|
|
18
|
+
request,
|
|
19
|
+
requestSse,
|
|
20
|
+
setClientRuntime,
|
|
21
|
+
} from "../lib/client.mjs";
|
|
15
22
|
|
|
16
23
|
const requireFromHere = createRequire(import.meta.url);
|
|
17
24
|
const { version: PKG_VERSION } = requireFromHere("../package.json");
|
|
@@ -23,6 +30,8 @@ import {
|
|
|
23
30
|
uploadExternalFile,
|
|
24
31
|
} from "../lib/external.mjs";
|
|
25
32
|
|
|
33
|
+
setClientRuntime({ client: "mcp" });
|
|
34
|
+
|
|
26
35
|
// Wrap a request() call into the MCP CallToolResult shape. Catches errors
|
|
27
36
|
// (NO_AUTH / 401 / 5xx / network) and surfaces them as `isError: true`
|
|
28
37
|
// content so the calling agent sees a clean error string instead of the
|
|
@@ -56,6 +65,20 @@ function splitSources(value) {
|
|
|
56
65
|
.filter(Boolean);
|
|
57
66
|
}
|
|
58
67
|
|
|
68
|
+
function expectedIds(value) {
|
|
69
|
+
const expected = { dealIds: [], wikiSlugs: [], raw: [] };
|
|
70
|
+
if (!value) return expected;
|
|
71
|
+
const items = Array.isArray(value) ? value : String(value).split(",");
|
|
72
|
+
for (const item of items.map((s) => String(s).trim()).filter(Boolean)) {
|
|
73
|
+
const [kind, ...rest] = item.split(":");
|
|
74
|
+
const id = rest.join(":").trim();
|
|
75
|
+
if (kind === "deal" && id) expected.dealIds.push(id);
|
|
76
|
+
else if ((kind === "wiki" || kind === "slug") && id) expected.wikiSlugs.push(id);
|
|
77
|
+
else expected.raw.push(item);
|
|
78
|
+
}
|
|
79
|
+
return expected;
|
|
80
|
+
}
|
|
81
|
+
|
|
59
82
|
function buildEnrichmentAgentMessage(args = {}) {
|
|
60
83
|
if (args.message) return String(args.message);
|
|
61
84
|
const sources = splitSources(args.sources) ?? [
|
|
@@ -209,11 +232,47 @@ server.registerTool(
|
|
|
209
232
|
},
|
|
210
233
|
async ({ limit } = {}) => {
|
|
211
234
|
const params = new URLSearchParams();
|
|
235
|
+
params.set("clientVersion", PKG_VERSION);
|
|
212
236
|
if (limit) params.set("limit", String(limit));
|
|
213
237
|
return callApi("GET", `/api/agent/manifest${params.toString() ? `?${params}` : ""}`);
|
|
214
238
|
}
|
|
215
239
|
);
|
|
216
240
|
|
|
241
|
+
server.registerTool(
|
|
242
|
+
"record_eval_feedback",
|
|
243
|
+
{
|
|
244
|
+
description:
|
|
245
|
+
"Mark the latest llama CLI/MCP result as good/bad or add a real query " +
|
|
246
|
+
"to the Golden Query Eval candidate pool. Use when the user says a " +
|
|
247
|
+
"Llama Command search/result was right, wrong, missing a source, or should be regression-tested.",
|
|
248
|
+
inputSchema: {
|
|
249
|
+
action: z.enum(["good", "bad", "add"]).describe("feedback action"),
|
|
250
|
+
eventId: z.number().optional().describe("agent_client_events id; defaults to latest local event"),
|
|
251
|
+
query: z.string().optional().describe("required for manual add when no source event is available"),
|
|
252
|
+
surface: z.string().optional().describe("deal, wiki, activity, people, or manual"),
|
|
253
|
+
expect: z
|
|
254
|
+
.union([z.string(), z.array(z.string())])
|
|
255
|
+
.optional()
|
|
256
|
+
.describe("expected ids like wiki:llamaos-weekly-2026-06-17 or deal:<uuid>"),
|
|
257
|
+
reason: z.string().optional().describe("why this was good/bad or should be tracked"),
|
|
258
|
+
privacyLevel: z.string().optional().describe("default internal"),
|
|
259
|
+
},
|
|
260
|
+
},
|
|
261
|
+
async ({ action, eventId, query, surface, expect, reason, privacyLevel }) => {
|
|
262
|
+
const last = getLastAgentEvent();
|
|
263
|
+
const body = {
|
|
264
|
+
action,
|
|
265
|
+
eventId: eventId ?? last?.lastEventId,
|
|
266
|
+
query,
|
|
267
|
+
surface: surface ?? last?.lastSurface,
|
|
268
|
+
expected: expectedIds(expect),
|
|
269
|
+
reason,
|
|
270
|
+
privacyLevel: privacyLevel ?? "internal",
|
|
271
|
+
};
|
|
272
|
+
return callApi("POST", "/api/agent/eval-feedback", body);
|
|
273
|
+
}
|
|
274
|
+
);
|
|
275
|
+
|
|
217
276
|
server.registerTool(
|
|
218
277
|
"skills_search",
|
|
219
278
|
{
|
|
@@ -1595,11 +1654,12 @@ server.registerPrompt(
|
|
|
1595
1654
|
"skills_search, and skills_read.",
|
|
1596
1655
|
},
|
|
1597
1656
|
async () => {
|
|
1598
|
-
// Gate the briefing behind
|
|
1599
|
-
//
|
|
1600
|
-
//
|
|
1657
|
+
// Gate the briefing behind authenticated Command runtime. The server-owned
|
|
1658
|
+
// /api/agent/briefing contract is canonical; bundled AGENT_BRIEFING.md is
|
|
1659
|
+
// only a rollout/offline fallback for authenticated users.
|
|
1601
1660
|
const headers = await getAuthHeaders();
|
|
1602
1661
|
let stub = null;
|
|
1662
|
+
let briefing = null;
|
|
1603
1663
|
if (Object.keys(headers).length === 0) {
|
|
1604
1664
|
stub =
|
|
1605
1665
|
"Llama Ventures team onboarding requires credentials.\n\n" +
|
|
@@ -1610,19 +1670,28 @@ server.registerPrompt(
|
|
|
1610
1670
|
"Founder / external visitor: use the `pitch_*` tools — no token required.";
|
|
1611
1671
|
} else {
|
|
1612
1672
|
try {
|
|
1613
|
-
|
|
1673
|
+
const params = new URLSearchParams({ clientVersion: PKG_VERSION });
|
|
1674
|
+
const body = await request("GET", `/api/agent/briefing?${params}`);
|
|
1675
|
+
briefing = body?.briefing || null;
|
|
1614
1676
|
} catch (err) {
|
|
1615
|
-
|
|
1616
|
-
|
|
1617
|
-
|
|
1618
|
-
|
|
1677
|
+
const msg = err?.message || "";
|
|
1678
|
+
if (msg.includes("Error[UNAUTHORIZED]") || msg.includes("Error[NO_AUTH]")) {
|
|
1679
|
+
stub =
|
|
1680
|
+
"Llama Ventures team onboarding requires valid credentials. " +
|
|
1681
|
+
"Server rejected the credentials we sent. Re-mint at " +
|
|
1682
|
+
"https://command.llamaventures.vc/settings/tokens.";
|
|
1683
|
+
} else {
|
|
1684
|
+
briefing =
|
|
1685
|
+
"Warning: server agent briefing unavailable; using bundled fallback.\n\n" +
|
|
1686
|
+
readBriefing();
|
|
1687
|
+
}
|
|
1619
1688
|
}
|
|
1620
1689
|
}
|
|
1621
1690
|
return {
|
|
1622
1691
|
messages: [
|
|
1623
1692
|
{
|
|
1624
1693
|
role: "user",
|
|
1625
|
-
content: { type: "text", text: stub ?? readBriefing() },
|
|
1694
|
+
content: { type: "text", text: stub ?? briefing ?? readBriefing() },
|
|
1626
1695
|
},
|
|
1627
1696
|
],
|
|
1628
1697
|
};
|
package/bin/llama.mjs
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
|
|
3
|
+
import { createRequire } from "module";
|
|
3
4
|
import readline from "readline";
|
|
4
5
|
import {
|
|
5
6
|
DEFAULT_BASE_URL,
|
|
@@ -9,6 +10,7 @@ import {
|
|
|
9
10
|
TOKEN_FILE,
|
|
10
11
|
getAuthHeaders,
|
|
11
12
|
getBaseUrl,
|
|
13
|
+
getLastAgentEvent,
|
|
12
14
|
getToken,
|
|
13
15
|
print,
|
|
14
16
|
readBriefing,
|
|
@@ -33,6 +35,9 @@ import { LLAMA_CLI_CLIENT_ID, pkceLoopbackFlow, revokeToken as revokeOAuthToken
|
|
|
33
35
|
import { deleteBundle, detectBackend, readBundle, writeBundle } from "../lib/oauth-storage.mjs";
|
|
34
36
|
import { maybeNudgeUpdate, getUpdateNudge } from "../lib/version-check.mjs";
|
|
35
37
|
|
|
38
|
+
const requireFromHere = createRequire(import.meta.url);
|
|
39
|
+
const { version: PKG_VERSION } = requireFromHere("../package.json");
|
|
40
|
+
|
|
36
41
|
function parseFlags(args, knownFlags = null) {
|
|
37
42
|
const flags = {};
|
|
38
43
|
const positional = [];
|
|
@@ -71,6 +76,34 @@ function parseFlags(args, knownFlags = null) {
|
|
|
71
76
|
return { flags, positional };
|
|
72
77
|
}
|
|
73
78
|
|
|
79
|
+
function agentOnboardNoAuthMessage() {
|
|
80
|
+
return `Llama Ventures team onboarding requires credentials.
|
|
81
|
+
|
|
82
|
+
Team member?
|
|
83
|
+
- Run \`gcloud auth login\` with your @llamaventures.vc account, OR
|
|
84
|
+
- Mint a token at https://command.llamaventures.vc/settings/tokens
|
|
85
|
+
then \`llama token set <llc_...>\`.
|
|
86
|
+
Re-run \`llama agent-onboard\` after — the workflow contract will print.
|
|
87
|
+
|
|
88
|
+
Founder or external visitor (no Llama account)?
|
|
89
|
+
Run \`llama pitch start --name "Your Name" --email "you@company.com"\`
|
|
90
|
+
to chat with our intake agent — no token required.`;
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
function agentOnboardRejectedMessage() {
|
|
94
|
+
return `Llama Ventures team onboarding requires valid credentials.
|
|
95
|
+
|
|
96
|
+
Server rejected the credentials we sent. Re-mint at
|
|
97
|
+
https://command.llamaventures.vc/settings/tokens, run
|
|
98
|
+
\`llama token set <llc_...>\`, then re-run \`llama agent-onboard\`.`;
|
|
99
|
+
}
|
|
100
|
+
|
|
101
|
+
async function fetchServerAgentBriefing() {
|
|
102
|
+
const params = new URLSearchParams({ clientVersion: PKG_VERSION });
|
|
103
|
+
const result = await request("GET", `/api/agent/briefing?${params}`);
|
|
104
|
+
return result?.briefing || "";
|
|
105
|
+
}
|
|
106
|
+
|
|
74
107
|
function closestKnownFlag(input, candidates) {
|
|
75
108
|
let best = null;
|
|
76
109
|
let bestScore = Infinity;
|
|
@@ -127,6 +160,45 @@ function slugifyTitle(title) {
|
|
|
127
160
|
return slug;
|
|
128
161
|
}
|
|
129
162
|
|
|
163
|
+
function parseExpectedIds(raw) {
|
|
164
|
+
const text = typeof raw === "string" ? raw : "";
|
|
165
|
+
const expected = { dealIds: [], wikiSlugs: [], raw: [] };
|
|
166
|
+
for (const item of text.split(",").map((s) => s.trim()).filter(Boolean)) {
|
|
167
|
+
const [kind, ...rest] = item.split(":");
|
|
168
|
+
const value = rest.join(":").trim();
|
|
169
|
+
if (kind === "deal" && value) expected.dealIds.push(value);
|
|
170
|
+
else if ((kind === "wiki" || kind === "slug") && value) expected.wikiSlugs.push(value);
|
|
171
|
+
else expected.raw.push(item);
|
|
172
|
+
}
|
|
173
|
+
return expected;
|
|
174
|
+
}
|
|
175
|
+
|
|
176
|
+
async function submitEvalFeedback(action, flags, queryText = "") {
|
|
177
|
+
const useLast = flags.last !== false;
|
|
178
|
+
const last = useLast ? getLastAgentEvent() : null;
|
|
179
|
+
const eventId =
|
|
180
|
+
flags.event && flags.event !== true
|
|
181
|
+
? Number(flags.event)
|
|
182
|
+
: last?.lastEventId ?? null;
|
|
183
|
+
if ((action === "good" || action === "bad") && !eventId && !queryText) {
|
|
184
|
+
throw new Error(`Usage: llama eval ${action} [--last] [--reason "..."]`);
|
|
185
|
+
}
|
|
186
|
+
const body = {
|
|
187
|
+
action,
|
|
188
|
+
eventId: Number.isFinite(eventId) ? eventId : undefined,
|
|
189
|
+
query: queryText || undefined,
|
|
190
|
+
surface: flags.surface && flags.surface !== true ? String(flags.surface) : last?.lastSurface ?? undefined,
|
|
191
|
+
expected:
|
|
192
|
+
flags.expect && flags.expect !== true
|
|
193
|
+
? parseExpectedIds(String(flags.expect))
|
|
194
|
+
: {},
|
|
195
|
+
reason: flags.reason && flags.reason !== true ? String(flags.reason) : undefined,
|
|
196
|
+
privacyLevel:
|
|
197
|
+
flags.privacy && flags.privacy !== true ? String(flags.privacy) : "internal",
|
|
198
|
+
};
|
|
199
|
+
return request("POST", "/api/agent/eval-feedback", body);
|
|
200
|
+
}
|
|
201
|
+
|
|
130
202
|
// Client-side fuzzy match — used as a fallback when the server hasn't yet
|
|
131
203
|
// shipped the search/filter API (Fix B, 2026-04-25). Once the server
|
|
132
204
|
// returns the `{deals,total,limit,offset}` envelope, this path is never
|
|
@@ -282,6 +354,8 @@ Agent onboarding (run once on first install):
|
|
|
282
354
|
llama skills search "pipeline update" # discover relevant runtime skills
|
|
283
355
|
llama skills show llama-pipeline # read a skill from Command
|
|
284
356
|
llama explain <url-or-object> # explain Command URL/object status + lifecycle
|
|
357
|
+
llama eval good|bad --last # mark the latest CLI/MCP result for eval
|
|
358
|
+
llama eval add "<query>" --expect wiki:<slug>|deal:<uuid>
|
|
285
359
|
|
|
286
360
|
External pitch — talk to Llama Ventures' intake agent (no token required):
|
|
287
361
|
llama pitch start --name "Jane Doe" --email "jane@acme.ai"
|
|
@@ -525,6 +599,7 @@ Common:
|
|
|
525
599
|
llama agent bootstrap live Llama OS skill manifest from Command
|
|
526
600
|
llama skills search "<query>" discover which skill to read
|
|
527
601
|
llama explain <url-or-object> explain Command URLs, 404s, deleted objects
|
|
602
|
+
llama eval bad --last mark latest CLI/MCP result as an eval candidate
|
|
528
603
|
|
|
529
604
|
Command groups — run \`llama help <group>\` for that group's commands:
|
|
530
605
|
deal create · show · feed · update · enrich · search · collaborators · links · delete
|
|
@@ -532,6 +607,7 @@ Command groups — run \`llama help <group>\` for that group's commands:
|
|
|
532
607
|
facts deal facts + skill corrections (the sourced, trust-rated layer)
|
|
533
608
|
timeline timeline · posts · mentions
|
|
534
609
|
wiki cross-deal knowledge entries (markdown or HTML)
|
|
610
|
+
eval mark real CLI/MCP searches good/bad or add a golden-query candidate
|
|
535
611
|
memo long-form HTML investment memo
|
|
536
612
|
html deal-specific HTML artifacts (/deals/<id>/browse/<slug>)
|
|
537
613
|
pitch external founder intake (no token needed)
|
|
@@ -865,18 +941,15 @@ async function runPitchRepl() {
|
|
|
865
941
|
async function main() {
|
|
866
942
|
const [area, action, ...rest] = process.argv.slice(2);
|
|
867
943
|
if (area === "--version" || area === "-v" || area === "version") {
|
|
868
|
-
const { createRequire } = await import("module");
|
|
869
|
-
const requireFromHere = createRequire(import.meta.url);
|
|
870
|
-
const { version } = requireFromHere("../package.json");
|
|
871
944
|
// `llama version --check` — explicitly check npm for a newer release and
|
|
872
945
|
// print the upgrade line (or "up to date"). Lets an agent surface the
|
|
873
946
|
// nudge on demand, separate from the throttled, TTY-gated auto-nudge.
|
|
874
947
|
if (action === "--check" || action === "check") {
|
|
875
948
|
const nudge = await getUpdateNudge();
|
|
876
|
-
console.log(nudge || `llama CLI ${
|
|
949
|
+
console.log(nudge || `llama CLI ${PKG_VERSION} — up to date`);
|
|
877
950
|
return;
|
|
878
951
|
}
|
|
879
|
-
console.log(
|
|
952
|
+
console.log(PKG_VERSION);
|
|
880
953
|
return;
|
|
881
954
|
}
|
|
882
955
|
if (!area || area === "help" || area === "--help" || area === "-h") {
|
|
@@ -897,12 +970,13 @@ async function main() {
|
|
|
897
970
|
return;
|
|
898
971
|
}
|
|
899
972
|
|
|
900
|
-
// `llama agent-onboard` —
|
|
901
|
-
//
|
|
902
|
-
//
|
|
973
|
+
// `llama agent-onboard` — fetch the server-owned Agent Runtime Contract
|
|
974
|
+
// so an AI agent reads the current Llama Ventures workflow contract. The
|
|
975
|
+
// bundled AGENT_BRIEFING.md is now only a fallback when the server route
|
|
976
|
+
// is unavailable during rollout.
|
|
903
977
|
// Also: `llama agent onboard` (two-word form) for symmetry.
|
|
904
978
|
//
|
|
905
|
-
// Gated behind
|
|
979
|
+
// Gated behind Command auth — without valid credentials we print a short
|
|
906
980
|
// bootstrap stub instead. Stops unauthenticated callers from harvesting
|
|
907
981
|
// internal command surface / workflow conventions just by running the
|
|
908
982
|
// public CLI.
|
|
@@ -912,39 +986,24 @@ async function main() {
|
|
|
912
986
|
) {
|
|
913
987
|
const headers = await getAuthHeaders();
|
|
914
988
|
if (Object.keys(headers).length === 0) {
|
|
915
|
-
console.log(
|
|
916
|
-
`Llama Ventures team onboarding requires credentials.
|
|
917
|
-
|
|
918
|
-
Team member?
|
|
919
|
-
- Run \`gcloud auth login\` with your @llamaventures.vc account, OR
|
|
920
|
-
- Mint a token at https://command.llamaventures.vc/settings/tokens
|
|
921
|
-
then \`llama token set <llc_...>\`.
|
|
922
|
-
Re-run \`llama agent-onboard\` after — the workflow contract will print.
|
|
923
|
-
|
|
924
|
-
Founder or external visitor (no Llama account)?
|
|
925
|
-
Run \`llama pitch start --name "Your Name" --email "you@company.com"\`
|
|
926
|
-
to chat with our intake agent — no token required.`
|
|
927
|
-
);
|
|
989
|
+
console.log(agentOnboardNoAuthMessage());
|
|
928
990
|
return;
|
|
929
991
|
}
|
|
930
992
|
try {
|
|
931
|
-
await
|
|
993
|
+
const briefing = await fetchServerAgentBriefing();
|
|
994
|
+
process.stdout.write(briefing || readBriefing());
|
|
932
995
|
} catch (e) {
|
|
933
996
|
const msg = e?.message || "";
|
|
934
997
|
if (msg.includes("Error[UNAUTHORIZED]") || msg.includes("Error[NO_AUTH]")) {
|
|
935
|
-
console.log(
|
|
936
|
-
`Llama Ventures team onboarding requires valid credentials.
|
|
937
|
-
|
|
938
|
-
Server rejected the credentials we sent. Re-mint at
|
|
939
|
-
https://command.llamaventures.vc/settings/tokens, run
|
|
940
|
-
\`llama token set <llc_...>\`, then re-run \`llama agent-onboard\`.`
|
|
941
|
-
);
|
|
998
|
+
console.log(agentOnboardRejectedMessage());
|
|
942
999
|
process.exitCode = 1;
|
|
943
1000
|
return;
|
|
944
1001
|
}
|
|
945
|
-
|
|
1002
|
+
process.stderr.write(
|
|
1003
|
+
`warning: server agent briefing unavailable (${msg}); using bundled fallback.\n`,
|
|
1004
|
+
);
|
|
1005
|
+
process.stdout.write(readBriefing());
|
|
946
1006
|
}
|
|
947
|
-
process.stdout.write(readBriefing());
|
|
948
1007
|
return;
|
|
949
1008
|
}
|
|
950
1009
|
|
|
@@ -954,6 +1013,7 @@ https://command.llamaventures.vc/settings/tokens, run
|
|
|
954
1013
|
if (area === "agent" && action === "bootstrap") {
|
|
955
1014
|
const { flags } = parseFlags(rest, ["json", "limit"]);
|
|
956
1015
|
const params = new URLSearchParams();
|
|
1016
|
+
params.set("clientVersion", PKG_VERSION);
|
|
957
1017
|
if (flags.limit && flags.limit !== true) params.set("limit", String(flags.limit));
|
|
958
1018
|
const manifest = await request("GET", `/api/agent/manifest${params.toString() ? `?${params}` : ""}`);
|
|
959
1019
|
if (flags.json) {
|
|
@@ -1037,6 +1097,43 @@ https://command.llamaventures.vc/settings/tokens, run
|
|
|
1037
1097
|
return;
|
|
1038
1098
|
}
|
|
1039
1099
|
|
|
1100
|
+
if (area === "eval") {
|
|
1101
|
+
const sub = action;
|
|
1102
|
+
if (sub === "good" || sub === "bad") {
|
|
1103
|
+
const { flags, positional } = parseFlags(rest, [
|
|
1104
|
+
"last",
|
|
1105
|
+
"event",
|
|
1106
|
+
"reason",
|
|
1107
|
+
"expect",
|
|
1108
|
+
"surface",
|
|
1109
|
+
"privacy",
|
|
1110
|
+
]);
|
|
1111
|
+
const q = positional.join(" ").trim();
|
|
1112
|
+
print(await submitEvalFeedback(sub, flags, q));
|
|
1113
|
+
return;
|
|
1114
|
+
}
|
|
1115
|
+
if (sub === "add") {
|
|
1116
|
+
const { flags, positional } = parseFlags(rest, [
|
|
1117
|
+
"event",
|
|
1118
|
+
"expect",
|
|
1119
|
+
"reason",
|
|
1120
|
+
"surface",
|
|
1121
|
+
"privacy",
|
|
1122
|
+
]);
|
|
1123
|
+
const q = positional.join(" ").trim();
|
|
1124
|
+
if (!q && !flags.event) {
|
|
1125
|
+
throw new Error(
|
|
1126
|
+
`Usage: llama eval add "<query>" --surface deal|wiki --expect wiki:<slug>|deal:<uuid>`,
|
|
1127
|
+
);
|
|
1128
|
+
}
|
|
1129
|
+
print(await submitEvalFeedback("add", flags, q));
|
|
1130
|
+
return;
|
|
1131
|
+
}
|
|
1132
|
+
throw new Error(
|
|
1133
|
+
"Usage: llama eval good|bad [--last] [--reason ...] OR llama eval add \"<query>\" --expect wiki:<slug>|deal:<uuid>",
|
|
1134
|
+
);
|
|
1135
|
+
}
|
|
1136
|
+
|
|
1040
1137
|
// `llama pitch ...` — external founder-pitch family. No Llama token
|
|
1041
1138
|
// required; bootstraps a session against /api/external/* via PoW + cookie.
|
|
1042
1139
|
// See lib/external.mjs and AGENT_BRIEFING.md for the full surface.
|
package/lib/client.mjs
CHANGED
|
@@ -13,6 +13,7 @@ import path from "path";
|
|
|
13
13
|
import { fileURLToPath } from "url";
|
|
14
14
|
import { execFile as _execFile } from "child_process";
|
|
15
15
|
import { promisify } from "util";
|
|
16
|
+
import { randomUUID } from "crypto";
|
|
16
17
|
|
|
17
18
|
const execFile = promisify(_execFile);
|
|
18
19
|
|
|
@@ -43,6 +44,19 @@ export function readBriefing() {
|
|
|
43
44
|
}
|
|
44
45
|
}
|
|
45
46
|
|
|
47
|
+
let packageVersionCache = null;
|
|
48
|
+
|
|
49
|
+
export function getPackageVersion() {
|
|
50
|
+
if (packageVersionCache) return packageVersionCache;
|
|
51
|
+
try {
|
|
52
|
+
const pkg = JSON.parse(fs.readFileSync(path.join(PACKAGE_ROOT, "package.json"), "utf8"));
|
|
53
|
+
packageVersionCache = String(pkg.version || "unknown");
|
|
54
|
+
} catch {
|
|
55
|
+
packageVersionCache = "unknown";
|
|
56
|
+
}
|
|
57
|
+
return packageVersionCache;
|
|
58
|
+
}
|
|
59
|
+
|
|
46
60
|
// Canonical entrypoint. `llama-command.onrender.com` also serves the API
|
|
47
61
|
// but its NextAuth callback URL doesn't match, so browser login (needed
|
|
48
62
|
// to mint a token at /settings/tokens) fails there with a server-config
|
|
@@ -54,6 +68,7 @@ export const DEFAULT_BASE_URL = "https://command.llamaventures.vc";
|
|
|
54
68
|
// agent-discovery convention.
|
|
55
69
|
export const TOKEN_DIR = path.join(os.homedir(), ".llama");
|
|
56
70
|
export const TOKEN_FILE = path.join(TOKEN_DIR, "token");
|
|
71
|
+
export const AGENT_SESSION_FILE = path.join(TOKEN_DIR, "agent-session.json");
|
|
57
72
|
|
|
58
73
|
// Legacy location used by CLI v0.1. Read for back-compat (silent migrate
|
|
59
74
|
// to canonical on first use); never written for the token, but still the
|
|
@@ -123,6 +138,269 @@ export function getToken() {
|
|
|
123
138
|
return "";
|
|
124
139
|
}
|
|
125
140
|
|
|
141
|
+
let runtimeClient = "cli";
|
|
142
|
+
let runtimeAgentClient = null;
|
|
143
|
+
|
|
144
|
+
export function setClientRuntime(opts = {}) {
|
|
145
|
+
if (opts.client) runtimeClient = String(opts.client);
|
|
146
|
+
if (opts.agentClient) runtimeAgentClient = String(opts.agentClient);
|
|
147
|
+
}
|
|
148
|
+
|
|
149
|
+
function detectAgentClient() {
|
|
150
|
+
if (runtimeAgentClient) return runtimeAgentClient;
|
|
151
|
+
if (process.env.LLAMA_AGENT_CLIENT) return process.env.LLAMA_AGENT_CLIENT;
|
|
152
|
+
if (process.env.CODEX_SANDBOX || process.env.CODEX_CLI || process.env.OPENAI_CODEX) return "codex";
|
|
153
|
+
if (process.env.CLAUDECODE || process.env.CLAUDE_CODE || process.env.CLAUDE_CODE_ENTRYPOINT) {
|
|
154
|
+
return "claude-code";
|
|
155
|
+
}
|
|
156
|
+
if (process.env.CURSOR_AGENT || process.env.CURSOR_TRACE_ID) return "cursor";
|
|
157
|
+
return "unknown";
|
|
158
|
+
}
|
|
159
|
+
|
|
160
|
+
export function readAgentSession() {
|
|
161
|
+
try {
|
|
162
|
+
return JSON.parse(fs.readFileSync(AGENT_SESSION_FILE, "utf8"));
|
|
163
|
+
} catch {
|
|
164
|
+
return {};
|
|
165
|
+
}
|
|
166
|
+
}
|
|
167
|
+
|
|
168
|
+
function writeAgentSession(session) {
|
|
169
|
+
try {
|
|
170
|
+
fs.mkdirSync(TOKEN_DIR, { recursive: true, mode: 0o700 });
|
|
171
|
+
fs.writeFileSync(AGENT_SESSION_FILE, `${JSON.stringify(session, null, 2)}\n`, { mode: 0o600 });
|
|
172
|
+
fs.chmodSync(AGENT_SESSION_FILE, 0o600);
|
|
173
|
+
} catch {
|
|
174
|
+
// Telemetry state is best-effort. Never break the actual CLI command.
|
|
175
|
+
}
|
|
176
|
+
}
|
|
177
|
+
|
|
178
|
+
function currentAgentSessionId() {
|
|
179
|
+
const session = readAgentSession();
|
|
180
|
+
if (session.sessionId) return session.sessionId;
|
|
181
|
+
const created = {
|
|
182
|
+
sessionId: randomUUID(),
|
|
183
|
+
createdAt: new Date().toISOString(),
|
|
184
|
+
};
|
|
185
|
+
writeAgentSession(created);
|
|
186
|
+
return created.sessionId;
|
|
187
|
+
}
|
|
188
|
+
|
|
189
|
+
export function getLastAgentEvent() {
|
|
190
|
+
const session = readAgentSession();
|
|
191
|
+
return session.lastEventId ? session : null;
|
|
192
|
+
}
|
|
193
|
+
|
|
194
|
+
function rememberAgentEvent(event) {
|
|
195
|
+
if (!event?.eventId) return;
|
|
196
|
+
const session = {
|
|
197
|
+
...readAgentSession(),
|
|
198
|
+
sessionId: event.sessionId || currentAgentSessionId(),
|
|
199
|
+
lastEventId: event.eventId,
|
|
200
|
+
lastCandidateId: event.candidateId ?? null,
|
|
201
|
+
lastCommand: event.command ?? null,
|
|
202
|
+
lastQuery: event.query ?? null,
|
|
203
|
+
lastSurface: event.surface ?? null,
|
|
204
|
+
lastRecordedAt: new Date().toISOString(),
|
|
205
|
+
};
|
|
206
|
+
writeAgentSession(session);
|
|
207
|
+
}
|
|
208
|
+
|
|
209
|
+
function agentClientHeaders(command) {
|
|
210
|
+
return {
|
|
211
|
+
"X-Llama-Client": runtimeClient,
|
|
212
|
+
"X-Llama-Client-Version": getPackageVersion(),
|
|
213
|
+
"X-Llama-Agent-Client": detectAgentClient(),
|
|
214
|
+
"X-Llama-Agent-Session": currentAgentSessionId(),
|
|
215
|
+
"X-Llama-Command": command || "unknown",
|
|
216
|
+
};
|
|
217
|
+
}
|
|
218
|
+
|
|
219
|
+
const SECRET_KEY_RE = /(token|secret|password|authorization|cookie|api[_-]?key|keychain|jwt)/i;
|
|
220
|
+
|
|
221
|
+
function truncateText(text, max = 2000) {
|
|
222
|
+
return text.length > max ? `${text.slice(0, max)}...[truncated]` : text;
|
|
223
|
+
}
|
|
224
|
+
|
|
225
|
+
function sanitizeTelemetryValue(value, depth = 0) {
|
|
226
|
+
if (depth > 4) return "[max-depth]";
|
|
227
|
+
if (value === null || value === undefined) return value;
|
|
228
|
+
if (typeof value === "string") return truncateText(value);
|
|
229
|
+
if (typeof value === "number" || typeof value === "boolean") return value;
|
|
230
|
+
if (Array.isArray(value)) {
|
|
231
|
+
return value.slice(0, 20).map((item) => sanitizeTelemetryValue(item, depth + 1));
|
|
232
|
+
}
|
|
233
|
+
if (typeof value === "object") {
|
|
234
|
+
const out = {};
|
|
235
|
+
for (const [key, val] of Object.entries(value).slice(0, 40)) {
|
|
236
|
+
out[key] = SECRET_KEY_RE.test(key) ? "[redacted]" : sanitizeTelemetryValue(val, depth + 1);
|
|
237
|
+
}
|
|
238
|
+
return out;
|
|
239
|
+
}
|
|
240
|
+
return String(value);
|
|
241
|
+
}
|
|
242
|
+
|
|
243
|
+
function parseEndpoint(endpoint) {
|
|
244
|
+
try {
|
|
245
|
+
return new URL(endpoint, "https://command.llamaventures.vc");
|
|
246
|
+
} catch {
|
|
247
|
+
return null;
|
|
248
|
+
}
|
|
249
|
+
}
|
|
250
|
+
|
|
251
|
+
function endpointArgs(endpoint, body) {
|
|
252
|
+
const args = {};
|
|
253
|
+
const url = parseEndpoint(endpoint);
|
|
254
|
+
if (url) {
|
|
255
|
+
for (const [key, value] of url.searchParams.entries()) args[key] = value;
|
|
256
|
+
}
|
|
257
|
+
if (body && typeof body === "object" && !Array.isArray(body)) {
|
|
258
|
+
Object.assign(args, body);
|
|
259
|
+
}
|
|
260
|
+
return sanitizeTelemetryValue(args);
|
|
261
|
+
}
|
|
262
|
+
|
|
263
|
+
function inferCommand(method, endpoint) {
|
|
264
|
+
const url = parseEndpoint(endpoint);
|
|
265
|
+
const pathname = url?.pathname || endpoint.split("?")[0] || "";
|
|
266
|
+
const verb = String(method || "GET").toUpperCase();
|
|
267
|
+
if (pathname === "/api/agent/client-events") return "telemetry.record";
|
|
268
|
+
if (pathname === "/api/agent/eval-feedback") return "eval.feedback";
|
|
269
|
+
if (pathname === "/api/wiki/search") return "wiki.search";
|
|
270
|
+
if (pathname === "/api/wiki/save") return "wiki.save";
|
|
271
|
+
if (/^\/api\/wiki\/[^/]+$/.test(pathname)) return verb === "GET" ? "wiki.read" : "wiki.write";
|
|
272
|
+
if (pathname === "/api/deals") return verb === "GET" ? "deal.search" : "deal.write";
|
|
273
|
+
if (pathname === "/api/deals/create") return "deal.create";
|
|
274
|
+
if (pathname === "/api/deals/update") return "deal.update";
|
|
275
|
+
if (/^\/api\/deals\/[^/]+\/threads\/[^/]+$/.test(pathname)) return "deal.agent.run";
|
|
276
|
+
if (/^\/api\/deals\/[^/]+\/threads$/.test(pathname)) return "deal.thread.create";
|
|
277
|
+
if (/^\/api\/deals\/[^/]+\/facts/.test(pathname)) return verb === "GET" ? "deal.fact.list" : "deal.fact.write";
|
|
278
|
+
if (/^\/api\/deals\/[^/]+\/posts$/.test(pathname)) return "deal.post";
|
|
279
|
+
if (/^\/api\/deals\/[^/]+\/blocks/.test(pathname)) return verb === "GET" ? "brief.blocks" : "brief.write";
|
|
280
|
+
if (/^\/api\/deals\/[^/]+$/.test(pathname)) return verb === "GET" ? "deal.show" : "deal.write";
|
|
281
|
+
if (pathname === "/api/me") return "auth.status";
|
|
282
|
+
if (pathname.startsWith("/api/agent/skills")) return "skills.read";
|
|
283
|
+
if (pathname === "/api/agent/manifest") return "agent.bootstrap";
|
|
284
|
+
if (pathname === "/api/agent/briefing") return "agent.briefing";
|
|
285
|
+
return `${verb.toLowerCase()} ${pathname || endpoint}`;
|
|
286
|
+
}
|
|
287
|
+
|
|
288
|
+
function queryForCommand(command, args) {
|
|
289
|
+
if (!command.endsWith(".search")) return null;
|
|
290
|
+
return args?.q || args?.search || args?.query || null;
|
|
291
|
+
}
|
|
292
|
+
|
|
293
|
+
function summarizeResultIds(data) {
|
|
294
|
+
const result = {};
|
|
295
|
+
const topDeals = [];
|
|
296
|
+
const topWiki = [];
|
|
297
|
+
|
|
298
|
+
const collectDeal = (deal) => {
|
|
299
|
+
const id = deal?.uuid || deal?.id || deal?.dealId || deal?.deal_uuid;
|
|
300
|
+
if (!id) return;
|
|
301
|
+
topDeals.push({ id, name: deal.companyName || deal.company_name || deal.name || null });
|
|
302
|
+
};
|
|
303
|
+
const collectWiki = (item) => {
|
|
304
|
+
if (!item?.slug) return;
|
|
305
|
+
topWiki.push({ slug: item.slug, title: item.title || null });
|
|
306
|
+
};
|
|
307
|
+
|
|
308
|
+
if (Array.isArray(data)) {
|
|
309
|
+
for (const item of data.slice(0, 20)) {
|
|
310
|
+
collectDeal(item);
|
|
311
|
+
collectWiki(item);
|
|
312
|
+
}
|
|
313
|
+
result.resultCount = data.length;
|
|
314
|
+
} else if (data && typeof data === "object") {
|
|
315
|
+
const deals = Array.isArray(data.deals) ? data.deals : [];
|
|
316
|
+
const articles = Array.isArray(data.articles) ? data.articles : [];
|
|
317
|
+
const results = Array.isArray(data.results) ? data.results : [];
|
|
318
|
+
for (const deal of deals.slice(0, 20)) collectDeal(deal);
|
|
319
|
+
for (const item of [...articles, ...results].slice(0, 20)) collectWiki(item);
|
|
320
|
+
if (typeof data.total === "number") result.total = data.total;
|
|
321
|
+
if (deals.length) result.resultCount = deals.length;
|
|
322
|
+
if (articles.length || results.length) result.resultCount = articles.length + results.length;
|
|
323
|
+
}
|
|
324
|
+
|
|
325
|
+
if (topDeals.length) result.deals = topDeals;
|
|
326
|
+
if (topWiki.length) result.wiki = topWiki;
|
|
327
|
+
return result;
|
|
328
|
+
}
|
|
329
|
+
|
|
330
|
+
function summarizeResult(data) {
|
|
331
|
+
if (data === null || data === undefined) return null;
|
|
332
|
+
if (Array.isArray(data)) return `${data.length} result(s)`;
|
|
333
|
+
if (typeof data === "object") {
|
|
334
|
+
if (Array.isArray(data.deals)) return `${data.deals.length} deal result(s); total=${data.total ?? "unknown"}`;
|
|
335
|
+
if (Array.isArray(data.results)) return `${data.results.length} result(s)`;
|
|
336
|
+
if (data.ok !== undefined) return `ok=${Boolean(data.ok)}`;
|
|
337
|
+
}
|
|
338
|
+
return truncateText(typeof data === "string" ? data : JSON.stringify(sanitizeTelemetryValue(data)), 2000);
|
|
339
|
+
}
|
|
340
|
+
|
|
341
|
+
function shouldSkipTelemetry(endpoint) {
|
|
342
|
+
if (process.env.LLAMA_TELEMETRY === "0") return true;
|
|
343
|
+
const pathname = parseEndpoint(endpoint)?.pathname || endpoint;
|
|
344
|
+
return pathname === "/api/agent/client-events" || pathname === "/api/agent/eval-feedback";
|
|
345
|
+
}
|
|
346
|
+
|
|
347
|
+
async function recordClientTelemetry({
|
|
348
|
+
authHeaders,
|
|
349
|
+
method,
|
|
350
|
+
endpoint,
|
|
351
|
+
body,
|
|
352
|
+
command,
|
|
353
|
+
status,
|
|
354
|
+
httpStatus,
|
|
355
|
+
latencyMs,
|
|
356
|
+
data,
|
|
357
|
+
errorMessage,
|
|
358
|
+
}) {
|
|
359
|
+
if (shouldSkipTelemetry(endpoint)) return;
|
|
360
|
+
const args = endpointArgs(endpoint, body);
|
|
361
|
+
const sessionId = currentAgentSessionId();
|
|
362
|
+
const payload = {
|
|
363
|
+
client: runtimeClient,
|
|
364
|
+
clientVersion: getPackageVersion(),
|
|
365
|
+
agentClient: detectAgentClient(),
|
|
366
|
+
sessionId,
|
|
367
|
+
command,
|
|
368
|
+
method: String(method || "GET").toUpperCase(),
|
|
369
|
+
endpoint,
|
|
370
|
+
status,
|
|
371
|
+
httpStatus,
|
|
372
|
+
latencyMs,
|
|
373
|
+
args,
|
|
374
|
+
query: queryForCommand(command, args),
|
|
375
|
+
resultSummary: status === "success" ? summarizeResult(data) : null,
|
|
376
|
+
resultIds: status === "success" ? summarizeResultIds(data) : {},
|
|
377
|
+
errorMessage: errorMessage ? truncateText(String(errorMessage), 2000) : null,
|
|
378
|
+
};
|
|
379
|
+
try {
|
|
380
|
+
const res = await fetch(`${getBaseUrl()}/api/agent/client-events`, {
|
|
381
|
+
method: "POST",
|
|
382
|
+
headers: {
|
|
383
|
+
"Content-Type": "application/json",
|
|
384
|
+
...agentClientHeaders("telemetry.record"),
|
|
385
|
+
...authHeaders,
|
|
386
|
+
},
|
|
387
|
+
body: JSON.stringify(payload),
|
|
388
|
+
});
|
|
389
|
+
if (!res.ok) return;
|
|
390
|
+
const recorded = await res.json().catch(() => null);
|
|
391
|
+
rememberAgentEvent({
|
|
392
|
+
...recorded,
|
|
393
|
+
sessionId,
|
|
394
|
+
command,
|
|
395
|
+
query: payload.query,
|
|
396
|
+
surface: command.startsWith("deal.") ? "deal" : command.startsWith("wiki.") ? "wiki" : null,
|
|
397
|
+
});
|
|
398
|
+
} catch {
|
|
399
|
+
// Best-effort by design. The actual llama command already succeeded or
|
|
400
|
+
// failed; telemetry must never alter that outcome.
|
|
401
|
+
}
|
|
402
|
+
}
|
|
403
|
+
|
|
126
404
|
// Try `gcloud auth print-identity-token`. Returns the JWT or null. Zero-config
|
|
127
405
|
// win for any team member who has gcloud + their @llamaventures.vc account
|
|
128
406
|
// already set up — the server's Bearer auth path verifies and auto-creates
|
|
@@ -227,10 +505,13 @@ export async function requestSse(method, endpoint, body, opts = {}) {
|
|
|
227
505
|
async function requestWithRetry(method, endpoint, body, allowRetry) {
|
|
228
506
|
const authHeaders = await getAuthHeaders();
|
|
229
507
|
if (Object.keys(authHeaders).length === 0) throw noAuthError();
|
|
508
|
+
const command = inferCommand(method, endpoint);
|
|
509
|
+
const start = Date.now();
|
|
230
510
|
const res = await fetch(`${getBaseUrl()}${endpoint}`, {
|
|
231
511
|
method,
|
|
232
512
|
headers: {
|
|
233
513
|
"Content-Type": "application/json",
|
|
514
|
+
...agentClientHeaders(command),
|
|
234
515
|
...authHeaders,
|
|
235
516
|
},
|
|
236
517
|
body: body === undefined ? undefined : JSON.stringify(body),
|
|
@@ -267,18 +548,45 @@ async function requestWithRetry(method, endpoint, body, allowRetry) {
|
|
|
267
548
|
}
|
|
268
549
|
if (!res.ok) {
|
|
269
550
|
const message = typeof data === "object" && data?.error ? data.error : `HTTP ${res.status}`;
|
|
551
|
+
await recordClientTelemetry({
|
|
552
|
+
authHeaders,
|
|
553
|
+
method,
|
|
554
|
+
endpoint,
|
|
555
|
+
body,
|
|
556
|
+
command,
|
|
557
|
+
status: "error",
|
|
558
|
+
httpStatus: res.status,
|
|
559
|
+
latencyMs: Date.now() - start,
|
|
560
|
+
data: null,
|
|
561
|
+
errorMessage: message,
|
|
562
|
+
});
|
|
270
563
|
throw new Error(message);
|
|
271
564
|
}
|
|
565
|
+
await recordClientTelemetry({
|
|
566
|
+
authHeaders,
|
|
567
|
+
method,
|
|
568
|
+
endpoint,
|
|
569
|
+
body,
|
|
570
|
+
command,
|
|
571
|
+
status: "success",
|
|
572
|
+
httpStatus: res.status,
|
|
573
|
+
latencyMs: Date.now() - start,
|
|
574
|
+
data,
|
|
575
|
+
errorMessage: null,
|
|
576
|
+
});
|
|
272
577
|
return data;
|
|
273
578
|
}
|
|
274
579
|
|
|
275
580
|
async function requestSseWithRetry(method, endpoint, body, opts, allowRetry) {
|
|
276
581
|
const authHeaders = await getAuthHeaders();
|
|
277
582
|
if (Object.keys(authHeaders).length === 0) throw noAuthError();
|
|
583
|
+
const command = inferCommand(method, endpoint);
|
|
584
|
+
const start = Date.now();
|
|
278
585
|
const res = await fetch(`${getBaseUrl()}${endpoint}`, {
|
|
279
586
|
method,
|
|
280
587
|
headers: {
|
|
281
588
|
"Content-Type": "application/json",
|
|
589
|
+
...agentClientHeaders(command),
|
|
282
590
|
...authHeaders,
|
|
283
591
|
},
|
|
284
592
|
body: body === undefined ? undefined : JSON.stringify(body),
|
|
@@ -309,6 +617,18 @@ async function requestSseWithRetry(method, endpoint, body, opts, allowRetry) {
|
|
|
309
617
|
data = text;
|
|
310
618
|
}
|
|
311
619
|
const message = typeof data === "object" && data?.error ? data.error : `HTTP ${res.status}`;
|
|
620
|
+
await recordClientTelemetry({
|
|
621
|
+
authHeaders,
|
|
622
|
+
method,
|
|
623
|
+
endpoint,
|
|
624
|
+
body,
|
|
625
|
+
command,
|
|
626
|
+
status: "error",
|
|
627
|
+
httpStatus: res.status,
|
|
628
|
+
latencyMs: Date.now() - start,
|
|
629
|
+
data: null,
|
|
630
|
+
errorMessage: message,
|
|
631
|
+
});
|
|
312
632
|
throw new Error(message);
|
|
313
633
|
}
|
|
314
634
|
|
|
@@ -343,6 +663,18 @@ async function requestSseWithRetry(method, endpoint, body, opts, allowRetry) {
|
|
|
343
663
|
for (const frame of frames) handleFrame(frame);
|
|
344
664
|
}
|
|
345
665
|
if (buf.trim()) handleFrame(buf);
|
|
666
|
+
await recordClientTelemetry({
|
|
667
|
+
authHeaders,
|
|
668
|
+
method,
|
|
669
|
+
endpoint,
|
|
670
|
+
body,
|
|
671
|
+
command,
|
|
672
|
+
status: "success",
|
|
673
|
+
httpStatus: res.status,
|
|
674
|
+
latencyMs: Date.now() - start,
|
|
675
|
+
data: { ok: true, textLength: text.length, events: events.length },
|
|
676
|
+
errorMessage: null,
|
|
677
|
+
});
|
|
346
678
|
return { text, events };
|
|
347
679
|
}
|
|
348
680
|
|
package/package.json
CHANGED
|
@@ -22,6 +22,7 @@ assert.equal(
|
|
|
22
22
|
);
|
|
23
23
|
const calls = [];
|
|
24
24
|
let threadSeq = 0;
|
|
25
|
+
let eventSeq = 0;
|
|
25
26
|
|
|
26
27
|
async function readJson(req) {
|
|
27
28
|
let raw = "";
|
|
@@ -65,11 +66,47 @@ const server = createServer(async (req, res) => {
|
|
|
65
66
|
path: url.pathname,
|
|
66
67
|
query: Object.fromEntries(url.searchParams.entries()),
|
|
67
68
|
body,
|
|
69
|
+
headers: {
|
|
70
|
+
client: req.headers["x-llama-client"] ?? null,
|
|
71
|
+
clientVersion: req.headers["x-llama-client-version"] ?? null,
|
|
72
|
+
agentClient: req.headers["x-llama-agent-client"] ?? null,
|
|
73
|
+
session: req.headers["x-llama-agent-session"] ?? null,
|
|
74
|
+
command: req.headers["x-llama-command"] ?? null,
|
|
75
|
+
},
|
|
68
76
|
});
|
|
69
77
|
|
|
78
|
+
if (req.method === "POST" && url.pathname === "/api/agent/client-events") {
|
|
79
|
+
eventSeq += 1;
|
|
80
|
+
writeJson(res, {
|
|
81
|
+
ok: true,
|
|
82
|
+
eventId: eventSeq,
|
|
83
|
+
candidateId: body?.command?.endsWith(".search") ? eventSeq + 1000 : null,
|
|
84
|
+
});
|
|
85
|
+
return;
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
if (req.method === "POST" && url.pathname === "/api/agent/eval-feedback") {
|
|
89
|
+
writeJson(res, {
|
|
90
|
+
ok: true,
|
|
91
|
+
candidate: {
|
|
92
|
+
id: 42,
|
|
93
|
+
source_event_id: body?.eventId ?? null,
|
|
94
|
+
feedback: body?.action ?? null,
|
|
95
|
+
},
|
|
96
|
+
});
|
|
97
|
+
return;
|
|
98
|
+
}
|
|
99
|
+
|
|
70
100
|
if (req.method === "GET" && url.pathname === "/api/agent/manifest") {
|
|
71
101
|
writeJson(res, {
|
|
72
102
|
ok: true,
|
|
103
|
+
contract: {
|
|
104
|
+
contract_version: "agent-contract.v1",
|
|
105
|
+
cli: {
|
|
106
|
+
client_version: url.searchParams.get("clientVersion"),
|
|
107
|
+
status: "ok",
|
|
108
|
+
},
|
|
109
|
+
},
|
|
73
110
|
briefing: "runtime briefing: use skills_search, skills_read, and object_inspect",
|
|
74
111
|
llama_os: {
|
|
75
112
|
visible_skill_count: 49,
|
|
@@ -85,6 +122,21 @@ const server = createServer(async (req, res) => {
|
|
|
85
122
|
return;
|
|
86
123
|
}
|
|
87
124
|
|
|
125
|
+
if (req.method === "GET" && url.pathname === "/api/agent/briefing") {
|
|
126
|
+
writeJson(res, {
|
|
127
|
+
ok: true,
|
|
128
|
+
contract: {
|
|
129
|
+
contract_version: "agent-contract.v1",
|
|
130
|
+
cli: {
|
|
131
|
+
client_version: url.searchParams.get("clientVersion"),
|
|
132
|
+
status: "ok",
|
|
133
|
+
},
|
|
134
|
+
},
|
|
135
|
+
briefing: "server-owned briefing: check CLI, use Pipeline First, prefer CLI/MCP",
|
|
136
|
+
});
|
|
137
|
+
return;
|
|
138
|
+
}
|
|
139
|
+
|
|
88
140
|
if (req.method === "GET" && url.pathname === "/api/agent/skills") {
|
|
89
141
|
writeJson(res, {
|
|
90
142
|
ok: true,
|
|
@@ -136,6 +188,16 @@ const server = createServer(async (req, res) => {
|
|
|
136
188
|
return;
|
|
137
189
|
}
|
|
138
190
|
|
|
191
|
+
if (req.method === "GET" && url.pathname === "/api/wiki/search") {
|
|
192
|
+
writeJson(res, [
|
|
193
|
+
{
|
|
194
|
+
slug: "llama-weekly-2026-06-16",
|
|
195
|
+
title: "Llama Weekly 2026-06-16",
|
|
196
|
+
},
|
|
197
|
+
]);
|
|
198
|
+
return;
|
|
199
|
+
}
|
|
200
|
+
|
|
139
201
|
if (req.method === "POST" && /^\/api\/deals\/[^/]+\/threads$/.test(url.pathname)) {
|
|
140
202
|
threadSeq += 1;
|
|
141
203
|
writeJson(res, { id: `thread-${threadSeq}` });
|
|
@@ -191,25 +253,34 @@ function resetCalls() {
|
|
|
191
253
|
threadSeq = 0;
|
|
192
254
|
}
|
|
193
255
|
|
|
256
|
+
function businessCalls() {
|
|
257
|
+
return calls.filter((call) => call.path !== "/api/agent/client-events");
|
|
258
|
+
}
|
|
259
|
+
|
|
260
|
+
function telemetryCalls() {
|
|
261
|
+
return calls.filter((call) => call.path === "/api/agent/client-events");
|
|
262
|
+
}
|
|
263
|
+
|
|
194
264
|
function paths() {
|
|
195
|
-
return
|
|
265
|
+
return businessCalls().map((call) => `${call.method} ${call.path}`);
|
|
196
266
|
}
|
|
197
267
|
|
|
198
268
|
function assertNoEnrichCall() {
|
|
199
269
|
assert.equal(
|
|
200
|
-
|
|
270
|
+
businessCalls().some((call) => call.path.endsWith("/enrich")),
|
|
201
271
|
false,
|
|
202
272
|
`expected no /enrich call, got ${paths().join(", ")}`,
|
|
203
273
|
);
|
|
204
274
|
}
|
|
205
275
|
|
|
206
276
|
function assertThreadRun({ title, messageIncludes }) {
|
|
207
|
-
|
|
208
|
-
assert.
|
|
209
|
-
assert.
|
|
210
|
-
assert.
|
|
277
|
+
const relevant = businessCalls();
|
|
278
|
+
assert.equal(relevant.length, 2, `expected thread create + SSE run, got ${paths().join(", ")}`);
|
|
279
|
+
assert.match(relevant[0].path, /^\/api\/deals\/[^/]+\/threads$/);
|
|
280
|
+
assert.equal(relevant[0].body?.title, title);
|
|
281
|
+
assert.match(relevant[1].path, /^\/api\/deals\/[^/]+\/threads\/thread-1$/);
|
|
211
282
|
for (const needle of messageIncludes) {
|
|
212
|
-
assert.match(
|
|
283
|
+
assert.match(relevant[1].body?.message ?? "", new RegExp(escapeRegExp(needle)));
|
|
213
284
|
}
|
|
214
285
|
}
|
|
215
286
|
|
|
@@ -313,18 +384,29 @@ const baseUrl = `http://${address.address}:${address.port}`;
|
|
|
313
384
|
const homeDir = await mkdtemp(path.join(os.tmpdir(), "llama-cli-routing-"));
|
|
314
385
|
|
|
315
386
|
try {
|
|
387
|
+
resetCalls();
|
|
388
|
+
const onboardRun = await runCli(["agent-onboard"], baseUrl, homeDir);
|
|
389
|
+
assert.match(onboardRun.stdout, /server-owned briefing/);
|
|
390
|
+
assert.deepEqual(paths(), ["GET /api/agent/briefing"]);
|
|
391
|
+
assert.ok(businessCalls()[0].query.clientVersion, "agent-onboard passes clientVersion");
|
|
392
|
+
assert.equal(telemetryCalls()[0].body?.command, "agent.briefing");
|
|
393
|
+
assert.equal(telemetryCalls()[0].body?.client, "cli");
|
|
394
|
+
assert.ok(telemetryCalls()[0].body?.sessionId, "telemetry includes an agent session id");
|
|
395
|
+
assert.equal(businessCalls()[0].headers.command, "agent.briefing");
|
|
396
|
+
|
|
316
397
|
resetCalls();
|
|
317
398
|
const bootstrapRun = await runCli(["agent", "bootstrap", "--limit", "3"], baseUrl, homeDir);
|
|
318
399
|
assert.match(bootstrapRun.stdout, /runtime briefing/);
|
|
319
400
|
assert.deepEqual(paths(), ["GET /api/agent/manifest"]);
|
|
320
|
-
assert.equal(
|
|
401
|
+
assert.equal(businessCalls()[0].query.limit, "3");
|
|
402
|
+
assert.ok(businessCalls()[0].query.clientVersion, "agent bootstrap passes clientVersion");
|
|
321
403
|
|
|
322
404
|
resetCalls();
|
|
323
405
|
const skillSearchRun = await runCli(["skills", "search", "pipeline", "--limit", "5"], baseUrl, homeDir);
|
|
324
406
|
assert.match(skillSearchRun.stdout, /llama-command/);
|
|
325
407
|
assert.deepEqual(paths(), ["GET /api/agent/skills"]);
|
|
326
|
-
assert.equal(
|
|
327
|
-
assert.equal(
|
|
408
|
+
assert.equal(businessCalls()[0].query.q, "pipeline");
|
|
409
|
+
assert.equal(businessCalls()[0].query.limit, "5");
|
|
328
410
|
|
|
329
411
|
resetCalls();
|
|
330
412
|
const skillShowRun = await runCli(["skills", "show", "llama-command"], baseUrl, homeDir);
|
|
@@ -336,7 +418,34 @@ try {
|
|
|
336
418
|
assert.match(explainRun.stdout, /Status: deleted/);
|
|
337
419
|
assert.match(explainRun.stdout, /Deleted by Kevin Yu/);
|
|
338
420
|
assert.deepEqual(paths(), ["GET /api/agent/explain"]);
|
|
339
|
-
assert.equal(
|
|
421
|
+
assert.equal(businessCalls()[0].query.q, "https://command.llamaventures.vc/wiki/missing-page");
|
|
422
|
+
|
|
423
|
+
resetCalls();
|
|
424
|
+
const wikiRun = await runCli(["wiki", "search", "llama weekly"], baseUrl, homeDir);
|
|
425
|
+
assert.match(wikiRun.stdout, /llama-weekly-2026-06-16/);
|
|
426
|
+
assert.deepEqual(paths(), ["GET /api/wiki/search"]);
|
|
427
|
+
assert.equal(telemetryCalls()[0].body?.command, "wiki.search");
|
|
428
|
+
assert.equal(telemetryCalls()[0].body?.query, "llama weekly");
|
|
429
|
+
|
|
430
|
+
resetCalls();
|
|
431
|
+
const evalRun = await runCli(
|
|
432
|
+
[
|
|
433
|
+
"eval",
|
|
434
|
+
"bad",
|
|
435
|
+
"--last",
|
|
436
|
+
"--reason",
|
|
437
|
+
"missed dev weekly",
|
|
438
|
+
"--expect",
|
|
439
|
+
"wiki:llamaos-weekly-2026-06-17",
|
|
440
|
+
],
|
|
441
|
+
baseUrl,
|
|
442
|
+
homeDir,
|
|
443
|
+
);
|
|
444
|
+
assert.match(evalRun.stdout, /"feedback": "bad"/);
|
|
445
|
+
assert.deepEqual(paths(), ["POST /api/agent/eval-feedback"]);
|
|
446
|
+
assert.equal(businessCalls()[0].body?.action, "bad");
|
|
447
|
+
assert.equal(businessCalls()[0].body?.eventId, 6);
|
|
448
|
+
assert.equal(businessCalls()[0].body?.expected?.wikiSlugs?.[0], "llamaos-weekly-2026-06-17");
|
|
340
449
|
|
|
341
450
|
resetCalls();
|
|
342
451
|
const enrichRun = await runCli(
|
|
@@ -369,9 +478,9 @@ try {
|
|
|
369
478
|
homeDir,
|
|
370
479
|
);
|
|
371
480
|
assert.deepEqual(paths(), ["POST /api/deals/deal-cli/enrich"]);
|
|
372
|
-
assert.equal(
|
|
373
|
-
assert.equal(
|
|
374
|
-
assert.equal(
|
|
481
|
+
assert.equal(businessCalls()[0].body?.apply, true);
|
|
482
|
+
assert.equal(businessCalls()[0].body?.dryRun, false);
|
|
483
|
+
assert.equal(businessCalls()[0].body?.executor, "server_agent");
|
|
375
484
|
|
|
376
485
|
resetCalls();
|
|
377
486
|
const agentRun = await runCli(
|
|
@@ -414,14 +523,16 @@ try {
|
|
|
414
523
|
const bootstrapPayload = JSON.parse(mcpBootstrap.content?.[0]?.text ?? "{}");
|
|
415
524
|
assert.equal(bootstrapPayload.ok, true);
|
|
416
525
|
assert.deepEqual(paths(), ["GET /api/agent/manifest"]);
|
|
417
|
-
assert.equal(
|
|
526
|
+
assert.equal(businessCalls()[0].query.limit, "2");
|
|
527
|
+
assert.ok(businessCalls()[0].query.clientVersion, "mcp agent_bootstrap passes clientVersion");
|
|
528
|
+
assert.equal(telemetryCalls()[0].body?.client, "mcp");
|
|
418
529
|
|
|
419
530
|
resetCalls();
|
|
420
531
|
const mcpSkills = await callMcpTool("skills_search", { q: "command", limit: 4 }, baseUrl, homeDir);
|
|
421
532
|
const skillsPayload = JSON.parse(mcpSkills.content?.[0]?.text ?? "{}");
|
|
422
533
|
assert.equal(skillsPayload.skills?.[0]?.slug, "llama-command");
|
|
423
534
|
assert.deepEqual(paths(), ["GET /api/agent/skills"]);
|
|
424
|
-
assert.equal(
|
|
535
|
+
assert.equal(businessCalls()[0].query.q, "command");
|
|
425
536
|
|
|
426
537
|
resetCalls();
|
|
427
538
|
const mcpSkillRead = await callMcpTool("skills_read", { slug: "llama-command" }, baseUrl, homeDir);
|