agent-led-growth 0.6.4 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +9 -0
- package/README.md +23 -3
- package/bin/cli.js +109 -34
- package/package.json +2 -1
- package/skills/alg-test/SKILL.md +2 -2
- package/src/connect/index.js +209 -0
- package/src/verify/index.js +14 -4
- package/src/verify/wizard.js +41 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agent-led-growth",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.8.0",
|
|
4
4
|
"description": "Score: evals for agents as customers. Runs a real agent through signup, key and first call on your product, before you ship. Track, the analytics half, measures which agents convert after.",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "Kat Laszlo",
|
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,14 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.8.0 (2026-10-23)
|
|
4
|
+
|
|
5
|
+
- `alg connect`: sets up Track's key without copying a secret by hand. It prints a code and opens the browser; once approved it writes `AGENT_TRAFFIC_KEY` (and `AGENT_TRAFFIC_PROJECT_ID`, not a secret) to `.env` or `--env-file`, replacing existing lines for those names and keeping the rest, and creates a new file with mode 0600. It warns when the file is not gitignored in a git repo and offers to add it. In a folder linked to Vercel, or with `--vercel`, it offers to run `vercel env add AGENT_TRAFFIC_KEY production` with the key on stdin. The key is never printed, logged or put on a command line. `--endpoint` or `AGENT_TRAFFIC_ENDPOINT` points it at another Track server, which is then written as `AGENT_TRAFFIC_ENDPOINT`.
|
|
6
|
+
|
|
7
|
+
## 0.7.0 (2026-10-23)
|
|
8
|
+
|
|
9
|
+
- `test` no longer stops when `alg.yml` has no `verify_call` and the product has no OpenAPI document. At a terminal it asks for a GET that needs a key (and the header that carries it, default `Authorization: Bearer {key}`), checks the answer, saves it to `alg.yml` and continues the run; Enter with nothing skips the check for that run. Answering "n" to an inferred call asks the same way instead of cancelling. Before, it printed `verify_call_invalid` and left the person to edit the file and start over.
|
|
10
|
+
- `test --skip-verify`: the signup runs and the agent's key is not checked, for CI and products with no API call to prove a key. The result passes when the agent saved a key, and says the key was not checked; `--json` carries `checker.skipped: true`. Without a terminal, `verify_call_invalid` names the flag.
|
|
11
|
+
|
|
3
12
|
## 0.6.0 (2026-10-22)
|
|
4
13
|
|
|
5
14
|
- One name. The command is `alg` (`npx agent-led-growth` still runs it); the config file is `alg.yml` and the run folder `.alg/`. The skills are `alg` and `alg-test`, the plugin `agent-led-growth`. `agent-ready`, `agent-ready.yml` and `.agent-ready/` are gone.
|
package/README.md
CHANGED
|
@@ -104,6 +104,14 @@ Add agent traffic to this API: read https://alg.katrinalaszlo.com/SKILL.md and f
|
|
|
104
104
|
Add agent traffic to this API: read https://alg.katrinalaszlo.com/SKILL.md and follow it.
|
|
105
105
|
```
|
|
106
106
|
|
|
107
|
+
To set up the key yourself, run this in your API's folder:
|
|
108
|
+
|
|
109
|
+
```bash
|
|
110
|
+
npx agent-led-growth connect
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
It prints a code and opens your browser; approve there, and it writes `AGENT_TRAFFIC_KEY` to `.env` (plus `AGENT_TRAFFIC_PROJECT_ID`, which is not a secret and is optional). Nothing to copy: the key is never printed, logged or put on a command line. If `.env` is not in `.gitignore` it says so and offers to add it. In a folder linked to Vercel (`.vercel/project.json`) it asks whether to also save the key in Vercel production, and passes it to `vercel env add` on stdin. Then restart your server and open `https://alg.katrinalaszlo.com/install` to watch the first event.
|
|
114
|
+
|
|
107
115
|
## Full command reference
|
|
108
116
|
|
|
109
117
|
### The three steps
|
|
@@ -142,7 +150,7 @@ Commit `alg.yml` so CI reuses your answers. `test` needs Claude Code installed a
|
|
|
142
150
|
|
|
143
151
|
### Installed
|
|
144
152
|
|
|
145
|
-
`npm i -g agent-led-growth`, then `alg check …` and `alg
|
|
153
|
+
`npm i -g agent-led-growth`, then `alg check …`, `alg test …` and `alg connect`.
|
|
146
154
|
|
|
147
155
|
### Flags
|
|
148
156
|
|
|
@@ -163,11 +171,23 @@ Commit `alg.yml` so CI reuses your answers. `test` needs Claude Code installed a
|
|
|
163
171
|
| --- | --- |
|
|
164
172
|
| `--check` | Show the plan, run nothing. Free. |
|
|
165
173
|
| `--yes` | Skip the confirmation; save an inferred `verify_call` to `alg.yml`. |
|
|
174
|
+
| `--skip-verify` | Test the signup only. The agent's key is not checked, so no `verify_call` is needed. |
|
|
166
175
|
| `--max-budget-usd <n>` | Model-use cap, default 5. Checked after each turn, so a run can end slightly above it. |
|
|
167
176
|
| `--max-turns <n>` | Agent turn budget, default 40. |
|
|
168
177
|
| `--inbox <address>` | An inbox you relay mail from, for products that email a code. Or set `AGENTMAIL_API_KEY` and each run gets its own. |
|
|
169
178
|
| `--json` | `agent-ready/verify@1` (or `verify-plan@1` with `--check`) to stdout. |
|
|
170
179
|
|
|
180
|
+
`connect`
|
|
181
|
+
|
|
182
|
+
| Flag | Does |
|
|
183
|
+
| --- | --- |
|
|
184
|
+
| `--endpoint <url>` | Track server. Default `AGENT_TRAFFIC_ENDPOINT`, else `https://alg.katrinalaszlo.com`. |
|
|
185
|
+
| `--env-file <path>` | File to write the key to. Default `.env`. Existing lines for the same names are replaced; the rest of the file is kept. A new file is created readable only by you. |
|
|
186
|
+
| `--vercel` | Also save the key in Vercel production. Asked by default when `.vercel/project.json` exists; without a terminal, only this flag does it. |
|
|
187
|
+
| `--no-vercel` | Do not offer Vercel. |
|
|
188
|
+
|
|
189
|
+
`connect` exits `0` when the key is saved and `1` when the sign-in is denied or expires.
|
|
190
|
+
|
|
171
191
|
Exit codes, both commands: `0` done or passed, `1` fixes found or test failed, `2` usage or setup, `3` the site did not answer or the run was inconclusive, `130` cancelled.
|
|
172
192
|
|
|
173
193
|
### Cost, accounts and data
|
|
@@ -201,7 +221,7 @@ The headline and the seven steps are the same whatever the answers; the answers
|
|
|
201
221
|
|
|
202
222
|
### alg.yml
|
|
203
223
|
|
|
204
|
-
`check` writes it. `test` reads it, and infers a `verify_call` from your OpenAPI document when there is none: a GET that needs a key and no parameters, preferring `/me`, `/account`, `/user`, `/whoami`. A complete file:
|
|
224
|
+
`check` writes it. `test` reads it, and infers a `verify_call` from your OpenAPI document when there is none: a GET that needs a key and no parameters, preferring `/me`, `/account`, `/user`, `/whoami`. With no OpenAPI document, `test` asks you at the prompt for a GET that needs a key (and the header that carries it), saves it to `alg.yml` and goes on; press Enter with nothing to skip the check for that run. Without a terminal, pass `--skip-verify`. A complete file:
|
|
205
225
|
|
|
206
226
|
```yaml
|
|
207
227
|
url: yourproduct.com
|
|
@@ -253,4 +273,4 @@ npx agent-led-growth test --json --yes # creates a real account; ask the dev
|
|
|
253
273
|
|
|
254
274
|
Each fix prompt is written for a coding agent: read `files.prompts[0]`, make the change, run its acceptance tests, deploy, run `check` again.
|
|
255
275
|
|
|
256
|
-
Source and
|
|
276
|
+
Source: https://github.com/katrinalaszlo/score (development) and https://github.com/katrinalaszlo/agent-led-growth-cli (the published build, where issues go). MIT.
|
package/bin/cli.js
CHANGED
|
@@ -32,8 +32,10 @@ import { runStandards } from "../src/audit/standards.js";
|
|
|
32
32
|
import { parseVerifySpec, buildVerifyTask, classify, failedStep, networkFor, describeSpec } from "../src/verify/index.js";
|
|
33
33
|
import { renderVerify, renderPlan, duration } from "../src/verify/render.js";
|
|
34
34
|
import { inferVerifyCall, latestInterface, latestAnswers } from "../src/verify/infer.js";
|
|
35
|
+
import { askVerifyCall } from "../src/verify/wizard.js";
|
|
35
36
|
import { renderPrompt } from "../src/audit/prompts.js";
|
|
36
37
|
import { patternInfo } from "../src/audit/index.js";
|
|
38
|
+
import { connect, resolveEndpoint } from "../src/connect/index.js";
|
|
37
39
|
import { createInterface } from "node:readline/promises";
|
|
38
40
|
import { spawnSync } from "node:child_process";
|
|
39
41
|
|
|
@@ -276,6 +278,7 @@ program
|
|
|
276
278
|
.alias("verify")
|
|
277
279
|
.description("A real agent (your Claude Code) signs up for and uses your product; a separate checker proves its key. Creates a real account, spends up to $5 of model use, asks first.")
|
|
278
280
|
.option("-y, --yes", "skip the confirmation and save an inferred verify_call to alg.yml")
|
|
281
|
+
.option("--skip-verify", "test the signup only: the agent's key is not checked, so no verify_call is needed")
|
|
279
282
|
.option("--inbox <address>", "an inbox you will relay mail from into the run's work/inbox/, for products that email a code")
|
|
280
283
|
.option("--max-turns <n>", "agent turn budget", (v) => parseInt(v, 10), 40)
|
|
281
284
|
.option("--max-budget-usd <usd>", "stop the agent once model use passes this many dollars (checked after each turn)", parseBudget, 5)
|
|
@@ -289,7 +292,9 @@ program
|
|
|
289
292
|
Examples:
|
|
290
293
|
alg test --check check Claude Code and alg.yml, print the plan, run nothing
|
|
291
294
|
alg test use url, task and verify_call from alg.yml
|
|
292
|
-
(no verify_call: one is taken from the product's OpenAPI document
|
|
295
|
+
(no verify_call: one is taken from the product's OpenAPI document,
|
|
296
|
+
or you are asked for one at the prompt)
|
|
297
|
+
alg test --skip-verify signup only; the key is not checked
|
|
293
298
|
alg test --yes --json no confirmation, JSON result (CI)`)
|
|
294
299
|
.exitOverride(usageExit)
|
|
295
300
|
.action(async (input, opts) => {
|
|
@@ -311,47 +316,77 @@ Examples:
|
|
|
311
316
|
const shown = configPath.startsWith(home + sep) ? `~${configPath.slice(home.length)}` : CONFIG_FILE;
|
|
312
317
|
console.error(` ${style.dim(`Not using ${shown}: it is for ${savedFor || "no product"}.`)}`);
|
|
313
318
|
}
|
|
314
|
-
const addCallHint = `Add the call that proves an agent's key works to ${CONFIG_FILE}, for example: verify_call: GET https://api.example.com/v1/me`;
|
|
319
|
+
const addCallHint = `Add the call that proves an agent's key works to ${CONFIG_FILE}, for example: verify_call: GET https://api.example.com/v1/me, or pass --skip-verify to test the signup without checking the key`;
|
|
320
|
+
const interactive = process.stdin.isTTY && !opts.json;
|
|
321
|
+
const askOnce = async (question) => {
|
|
322
|
+
const rl = createInterface({ input: process.stdin, output: process.stderr });
|
|
323
|
+
const reply = await rl.question(question).catch((err) => {
|
|
324
|
+
if (err.code === "ABORT_ERR") return "";
|
|
325
|
+
throw err;
|
|
326
|
+
});
|
|
327
|
+
rl.close();
|
|
328
|
+
return reply;
|
|
329
|
+
};
|
|
315
330
|
// No verify_call: take one from the product's own OpenAPI document. It is saved only once accepted, by a
|
|
316
|
-
// person at the prompt or by --yes; otherwise it is used for this run alone.
|
|
317
|
-
|
|
318
|
-
|
|
331
|
+
// person at the prompt or by --yes; otherwise it is used for this run alone. When the docs give none, a person
|
|
332
|
+
// at the terminal is asked for the call, or can skip the check; without a terminal, --skip-verify does that.
|
|
333
|
+
let chosen = null;
|
|
334
|
+
let skipped = Boolean(opts.skipVerify);
|
|
335
|
+
if (!skipped && !(config.verify_call || "").trim()) {
|
|
319
336
|
const live = !opts.json && Boolean(process.stderr.isTTY);
|
|
320
337
|
if (live) process.stderr.write(` ${style.dim("Finding a call that proves the agent's key works…")}`);
|
|
321
|
-
inferred = await inferVerifyCall({ url, version: pkg.version, cwd: process.cwd() });
|
|
338
|
+
const inferred = await inferVerifyCall({ url, version: pkg.version, cwd: process.cwd() });
|
|
322
339
|
if (live) process.stderr.write("\r\x1b[2K");
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
const offSiteNote = inferred.offSite ? `\n This API host is on a different site than ${host}; the key the agent gets is sent there.` : "";
|
|
340
|
+
if (inferred.error && !interactive) {
|
|
341
|
+
// For another product's file, the call goes in a file of this product's own, never in that one.
|
|
342
|
+
if (!useSaved) fail(opts, 2, "verify_call_invalid", `No verify_call for ${host}, and none could be taken from the product's docs: ${inferred.error}.`, `Run "alg check ${host}" in a folder of its own, then add the call that proves an agent's key works to the ${CONFIG_FILE} it writes, for example: verify_call: GET https://api.example.com/v1/me, or pass --skip-verify to test the signup without checking the key`);
|
|
343
|
+
fail(opts, 2, "verify_call_invalid", `${CONFIG_FILE} has no verify_call, and none could be taken from the product's docs: ${inferred.error}.`, addCallHint);
|
|
344
|
+
}
|
|
329
345
|
let accepted = Boolean(opts.yes);
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
rl.close();
|
|
339
|
-
if (!/^(y(es)?)?$/i.test(reply.trim())) {
|
|
340
|
-
console.error(` Cancelled. Nothing saved. To use a different call, add it to ${CONFIG_FILE}, for example: verify_call: GET https://api.example.com/v1/me`);
|
|
341
|
-
process.exit(130);
|
|
346
|
+
let typed = false;
|
|
347
|
+
if (inferred.error) {
|
|
348
|
+
const answer = await askVerifyCall({ ask: askOnce, say: (l) => console.error(l), url, reason: inferred.error, style });
|
|
349
|
+
if (answer.skip) skipped = true;
|
|
350
|
+
else {
|
|
351
|
+
chosen = { ...answer, source: "typed" };
|
|
352
|
+
accepted = true;
|
|
353
|
+
typed = true;
|
|
342
354
|
}
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
355
|
+
} else {
|
|
356
|
+
chosen = { ...inferred, source: "inferred" };
|
|
357
|
+
const offSiteNote = inferred.offSite ? `\n This API host is on a different site than ${host}; the key the agent gets is sent there.` : "";
|
|
358
|
+
if (!accepted && interactive) {
|
|
359
|
+
// A person reads the call and where it came from; the "Will check with" line stays for logs and agents.
|
|
360
|
+
const ask = `\n To prove the agent's key works, we'll call your API with it after signup:\n ${style.bold(inferred.call)} ${style.dim(`(from ${inferred.specUrl.replace(/^https?:\/\//, "")})`)}${offSiteNote}\n Use this call? [Y/n] `;
|
|
361
|
+
const reply = await askOnce(ask);
|
|
362
|
+
if (/^(y(es)?)?$/i.test(reply.trim())) accepted = true;
|
|
363
|
+
else {
|
|
364
|
+
// Not that one: the person names the call instead, or skips the check.
|
|
365
|
+
const answer = await askVerifyCall({ ask: askOnce, say: (l) => console.error(l), url, reason: null, style });
|
|
366
|
+
if (answer.skip) {
|
|
367
|
+
chosen = null;
|
|
368
|
+
skipped = true;
|
|
369
|
+
} else {
|
|
370
|
+
chosen = { ...answer, source: "typed" };
|
|
371
|
+
accepted = true;
|
|
372
|
+
typed = true;
|
|
373
|
+
}
|
|
374
|
+
}
|
|
375
|
+
} else if (!opts.json) console.error(`\n Will check with ${inferred.call} on ${inferred.apiHost} (${inferred.headerName}) (from ${inferred.specUrl}${inferred.specVia ? `, linked from ${inferred.specVia}` : ""}).${offSiteNote}`);
|
|
376
|
+
}
|
|
377
|
+
if (skipped) {
|
|
378
|
+
if (!opts.json) console.error(` ${style.dim("Verification skipped: the agent signs up, and its key is not checked.")}`);
|
|
379
|
+
} else if (!useSaved) {
|
|
346
380
|
if (!opts.json) console.error(` ${style.dim(`Used for this run only: ${CONFIG_FILE} here is not for ${host}.`)}`);
|
|
347
381
|
} else if (accepted) {
|
|
348
|
-
writeVerifyCall(configPath,
|
|
349
|
-
if (!opts.json) console.error(` ${style.dim(`Saved to ${CONFIG_FILE}.`)}`);
|
|
382
|
+
writeVerifyCall(configPath, chosen);
|
|
383
|
+
if (!opts.json) console.error(` ${style.dim(`Saved to ${CONFIG_FILE}${typed ? ": the next test uses it without asking" : ""}.`)}`);
|
|
350
384
|
} else if (!opts.json) console.error(` ${style.dim(`Not saved to ${CONFIG_FILE}; pass --yes or run in a terminal to save it.`)}`);
|
|
351
385
|
}
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
386
|
+
if (skipped && !opts.json && opts.skipVerify) console.error(`\n ${style.dim("Verification skipped (--skip-verify): the agent signs up, and its key is not checked.")}`);
|
|
387
|
+
const spec = skipped ? null : parseVerifySpec(chosen ? { ...config, verify_call: chosen.call, verify_header: chosen.header } : config);
|
|
388
|
+
if (spec?.error) fail(opts, 2, "verify_call_invalid", spec.error, addCallHint);
|
|
389
|
+
const checker = { inferred: chosen?.source === "inferred", inferredFrom: chosen?.source === "inferred" ? chosen.specUrl : null, ...(chosen?.specVia ? { inferredVia: chosen.specVia } : {}), ...(chosen ? { apiHost: chosen.apiHost, apiHostOffSite: chosen.offSite } : {}), ...(skipped ? { skipped: true } : {}) };
|
|
355
390
|
if (opts.check) {
|
|
356
391
|
const claude = opts.executor === "claude-print" ? claudeStatus() : { ok: true, detail: `executor ${opts.executor}` };
|
|
357
392
|
const inbox = process.env.AGENTMAIL_API_KEY ? "AgentMail: a fresh inbox for this run, deleted after" : opts.inbox ? `you relay mail from ${opts.inbox} into work/inbox/` : "none (set AGENTMAIL_API_KEY if the product emails a code or link)";
|
|
@@ -361,8 +396,8 @@ Examples:
|
|
|
361
396
|
target: { url, host },
|
|
362
397
|
task,
|
|
363
398
|
claudeCode: claude,
|
|
364
|
-
agent: { hosts: networkFor(url, spec), inbox, maxTurns: opts.maxTurns, maxBudgetUsd: opts.maxBudgetUsd, saves: ["AGENT_READY_KEY", ...spec.fields] },
|
|
365
|
-
checker: { call: describeSpec(spec), calls: ["with the agent's key: must answer " + spec.expect, "with no key: must be refused (400, 401 or 403)", "with a wrong key: must be refused"], ...checker },
|
|
399
|
+
agent: { hosts: networkFor(url, spec), inbox, maxTurns: opts.maxTurns, maxBudgetUsd: opts.maxBudgetUsd, saves: ["AGENT_READY_KEY", ...(spec ? spec.fields : [])] },
|
|
400
|
+
checker: { call: describeSpec(spec), calls: spec ? ["with the agent's key: must answer " + spec.expect, "with no key: must be refused (400, 401 or 403)", "with a wrong key: must be refused"] : ["none: a key the agent saves counts, unchecked"], ...checker },
|
|
366
401
|
};
|
|
367
402
|
// The onboarding pattern chosen in alg.yml, as the last check's interface.json describes it.
|
|
368
403
|
const onboarding = latestInterface(process.cwd(), url)?.doc.onboarding;
|
|
@@ -435,6 +470,46 @@ Examples:
|
|
|
435
470
|
process.exitCode = verdict.exitCode;
|
|
436
471
|
});
|
|
437
472
|
|
|
473
|
+
program
|
|
474
|
+
.command("connect")
|
|
475
|
+
.description("Connect this app to Track: approve in the browser, and the key is written to .env for you (and to Vercel if you say yes). The key is never printed.")
|
|
476
|
+
.option("--endpoint <url>", "Track server (default: AGENT_TRAFFIC_ENDPOINT, else https://alg.katrinalaszlo.com)")
|
|
477
|
+
.option("--env-file <path>", "env file to write the key to", ".env")
|
|
478
|
+
.option("--vercel", "also save the key in Vercel production (asked by default when .vercel/project.json exists)")
|
|
479
|
+
.option("--no-vercel", "do not offer to save the key in Vercel")
|
|
480
|
+
.option("--no-color", "plain output")
|
|
481
|
+
.addHelpText("after", `
|
|
482
|
+
Examples:
|
|
483
|
+
alg connect sign in, write AGENT_TRAFFIC_KEY to .env
|
|
484
|
+
alg connect --env-file .env.local write it to another file
|
|
485
|
+
alg connect --no-vercel skip the Vercel question`)
|
|
486
|
+
.exitOverride(usageExit)
|
|
487
|
+
.action(async (opts) => {
|
|
488
|
+
const endpoint = resolveEndpoint(opts.endpoint, process.env);
|
|
489
|
+
if (!/^https?:\/\/[^/]+/.test(endpoint)) fail(opts, 2, "invalid_address", `"${endpoint}" is not a web address.`, "Try: alg connect --endpoint https://alg.katrinalaszlo.com");
|
|
490
|
+
const style = makeStyle(Boolean(process.stderr.isTTY) && opts.color && !process.env.NO_COLOR);
|
|
491
|
+
const ask = async (question) => {
|
|
492
|
+
const rl = createInterface({ input: process.stdin, output: process.stderr });
|
|
493
|
+
const reply = await rl.question(question).catch((err) => {
|
|
494
|
+
if (err.code === "ABORT_ERR") return "n";
|
|
495
|
+
throw err;
|
|
496
|
+
});
|
|
497
|
+
rl.close();
|
|
498
|
+
return reply;
|
|
499
|
+
};
|
|
500
|
+
process.exitCode = await connect({
|
|
501
|
+
endpoint,
|
|
502
|
+
envFile: resolve(opts.envFile),
|
|
503
|
+
vercel: opts.vercel,
|
|
504
|
+
cwd: process.cwd(),
|
|
505
|
+
interactive: Boolean(process.stdin.isTTY),
|
|
506
|
+
sleep: (ms) => new Promise((done) => setTimeout(done, ms)),
|
|
507
|
+
ask,
|
|
508
|
+
say: (line) => console.error(line),
|
|
509
|
+
style,
|
|
510
|
+
});
|
|
511
|
+
});
|
|
512
|
+
|
|
438
513
|
// 0.3 kept a Tanso workspace for verify runs. check and test need no Tanso account now; these
|
|
439
514
|
// commands stay so scripts written for 0.3 get a clear answer instead of "unknown command".
|
|
440
515
|
for (const name of ["account", "login", "logout"]) {
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agent-led-growth",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.8.0",
|
|
4
4
|
"description": "Score: evals for agents as customers. Runs a real agent through signup, key and first call on your product, before you ship. Track, the analytics half, measures which agents convert after.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -15,6 +15,7 @@
|
|
|
15
15
|
"bin/cli.js",
|
|
16
16
|
"src/index.js",
|
|
17
17
|
"src/audit/",
|
|
18
|
+
"src/connect/",
|
|
18
19
|
"src/dashboard/strip.js",
|
|
19
20
|
"src/crash/",
|
|
20
21
|
"src/harness/",
|
package/skills/alg-test/SKILL.md
CHANGED
|
@@ -14,7 +14,7 @@ A run creates a real account on the product and uses the developer's Claude Code
|
|
|
14
14
|
|
|
15
15
|
`test` reads `alg.yml` in the current directory. If it is missing, run `npx agent-led-growth check <url> --json --yes` first, which writes it.
|
|
16
16
|
|
|
17
|
-
With no `verify_call`, `test` infers one from the product's OpenAPI document: a GET that needs a key (bearer, or an API key header) and has no required parameters, preferring `/me`, `/account`, `/user` or `/whoami`, on the document's first server. An API key in the `Authorization` header gets the prefix the document gives (`Authorization: Token {key}` when it says "Prefix your key with 'Token '"); with no documented prefix that call is skipped. The plan shows the call's API host (`checker.apiHost`) and, when it is on a different site than the product (`checker.apiHostOffSite`), says the key the agent gets is sent there. Tell the developer which host the key goes to. When the product's own OpenAPI document has none, it tries the OpenAPI documents its `llms.txt`, `auth.md` and other agent files link to on the same registrable domain (such as `app.` or `api.` hosts). It reuses the latest `.alg/<host>/*/interface.json` from `check`, or reads the public pages again. `--yes` accepts the call and saves it to `alg.yml`; without `--yes` and without a terminal it is used for that run and not saved. With `--json`, the plan and the result carry `checker.inferred: true` and `checker.inferredFrom`, and `checker.inferredVia` (the page that linked the document) when it came from a link. Show the developer the inferred call before a real run. When nothing fits, `test` exits `2` with `verify_call_invalid`;
|
|
17
|
+
With no `verify_call`, `test` infers one from the product's OpenAPI document: a GET that needs a key (bearer, or an API key header) and has no required parameters, preferring `/me`, `/account`, `/user` or `/whoami`, on the document's first server. An API key in the `Authorization` header gets the prefix the document gives (`Authorization: Token {key}` when it says "Prefix your key with 'Token '"); with no documented prefix that call is skipped. The plan shows the call's API host (`checker.apiHost`) and, when it is on a different site than the product (`checker.apiHostOffSite`), says the key the agent gets is sent there. Tell the developer which host the key goes to. When the product's own OpenAPI document has none, it tries the OpenAPI documents its `llms.txt`, `auth.md` and other agent files link to on the same registrable domain (such as `app.` or `api.` hosts). It reuses the latest `.alg/<host>/*/interface.json` from `check`, or reads the public pages again. `--yes` accepts the call and saves it to `alg.yml`; without `--yes` and without a terminal it is used for that run and not saved. With `--json`, the plan and the result carry `checker.inferred: true` and `checker.inferredFrom`, and `checker.inferredVia` (the page that linked the document) when it came from a link. Show the developer the inferred call before a real run. When nothing fits and a person is at the terminal, `test` asks for the call (a GET that needs a key, then the header that carries it), saves it to `alg.yml` and goes on; Enter with nothing skips the check for that run, and the result says the key was not checked (`checker.skipped: true`). Without a terminal, or with `--json`, it exits `2` with `verify_call_invalid`; pass `--skip-verify` to run the signup without checking the key, or add the call to `alg.yml`.
|
|
18
18
|
|
|
19
19
|
To choose the call, add it. Pick an authenticated read from the product's own API docs that refuses a request without a key:
|
|
20
20
|
|
|
@@ -63,7 +63,7 @@ The JSON is `agent-ready/verify@1`. Report `outcome`, `reason`, each entry in `c
|
|
|
63
63
|
| --- | --- | --- |
|
|
64
64
|
| `0` | `passed` or `handoff` | The key works and no key and a wrong key were refused; or the agent correctly stopped where the onboarding model says a person sets access up first. |
|
|
65
65
|
| `1` | `failed` | The agent got no key, or the checker's calls did not pass. `prompt` names a fix prompt in the run folder. |
|
|
66
|
-
| `2` | | Usage or setup: no `verify_call` and none could be inferred, or Claude Code missing or signed out. |
|
|
66
|
+
| `2` | | Usage or setup: no `verify_call` and none could be inferred (without a terminal; `--skip-verify` runs the signup anyway), or Claude Code missing or signed out. |
|
|
67
67
|
| `3` | `inconclusive` | Not a result about the product: server errors, a call that answers without a key, the agent could not connect, the spending cap, or a run that did not finish. Say why and suggest what to change. |
|
|
68
68
|
| `130` | | Cancelled. |
|
|
69
69
|
|
|
@@ -0,0 +1,209 @@
|
|
|
1
|
+
import { existsSync, readFileSync, writeFileSync, appendFileSync } from "node:fs";
|
|
2
|
+
import { basename, dirname, join, relative } from "node:path";
|
|
3
|
+
import { spawnSync } from "node:child_process";
|
|
4
|
+
|
|
5
|
+
// `alg connect`: a device login against Track. The person approves in the browser, and the key Track returns is
|
|
6
|
+
// written to the env file (and, if they say yes, to Vercel) without ever being printed, logged or put on a command
|
|
7
|
+
// line. Everything that touches the outside (fetch, child processes, the clock, the prompt) is passed in, so tests
|
|
8
|
+
// run it against a fake server.
|
|
9
|
+
|
|
10
|
+
export const DEFAULT_ENDPOINT = "https://alg.katrinalaszlo.com";
|
|
11
|
+
|
|
12
|
+
export function resolveEndpoint(flag, env) {
|
|
13
|
+
return (flag || env.AGENT_TRAFFIC_ENDPOINT || DEFAULT_ENDPOINT).replace(/\/+$/, "");
|
|
14
|
+
}
|
|
15
|
+
|
|
16
|
+
const defaultRun = (cmd, args, options = {}) => spawnSync(cmd, args, { encoding: "utf8", ...options });
|
|
17
|
+
|
|
18
|
+
async function postJson(fetchImpl, url, body) {
|
|
19
|
+
let res;
|
|
20
|
+
try {
|
|
21
|
+
res = await fetchImpl(url, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify(body) });
|
|
22
|
+
} catch (err) {
|
|
23
|
+
throw new Error(`Could not reach ${url}: ${err.cause?.message || err.message}`);
|
|
24
|
+
}
|
|
25
|
+
// A proxy or a 5xx page can answer in HTML: only a JSON answer is read, and the status alone says what happened.
|
|
26
|
+
const isJson = (res.headers.get("content-type") || "").includes("json");
|
|
27
|
+
return { status: res.status, body: isJson ? await res.json() : null };
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
export async function startDeviceLogin({ endpoint, fetchImpl }) {
|
|
31
|
+
const res = await postJson(fetchImpl, `${endpoint}/v1/connect/start`, { client: "agent-led-growth" });
|
|
32
|
+
if (res.status !== 201 || !res.body?.device_code || !res.body?.user_code || !res.body?.verification_url) {
|
|
33
|
+
throw new Error(`${endpoint} did not start a sign-in (${res.status}${res.body?.error ? ` ${res.body.error}` : ""}).`);
|
|
34
|
+
}
|
|
35
|
+
return res.body;
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
// Polls until the person approves, denies, or the code expires. Returns { ok: true, result } or
|
|
39
|
+
// { ok: false, reason, message }.
|
|
40
|
+
export async function pollForKey({ endpoint, start, fetchImpl, sleep, now }) {
|
|
41
|
+
let intervalSeconds = Number(start.interval) || 5;
|
|
42
|
+
const deadline = now() + (Number(start.expires_in) || 600) * 1000;
|
|
43
|
+
const expired = { ok: false, reason: "expired", message: "The sign-in code expired before it was approved. Run alg connect again." };
|
|
44
|
+
for (;;) {
|
|
45
|
+
await sleep(intervalSeconds * 1000);
|
|
46
|
+
if (now() >= deadline) return expired;
|
|
47
|
+
const res = await postJson(fetchImpl, `${endpoint}/v1/connect/poll`, { device_code: start.device_code });
|
|
48
|
+
if (res.status === 202) continue;
|
|
49
|
+
if (res.status === 429) {
|
|
50
|
+
intervalSeconds += 2;
|
|
51
|
+
continue;
|
|
52
|
+
}
|
|
53
|
+
if (res.status === 200 && res.body?.key) return { ok: true, result: res.body };
|
|
54
|
+
if (res.status === 403) return { ok: false, reason: "denied", message: "The sign-in was denied in the browser. Nothing was saved." };
|
|
55
|
+
if (res.status === 410) return expired;
|
|
56
|
+
return { ok: false, reason: "unexpected", message: `${endpoint} answered ${res.status}${res.body?.error ? ` ${res.body.error}` : ""} while waiting for approval. Nothing was saved.` };
|
|
57
|
+
}
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
// Sets each name in values in the env file: an existing line for that name is replaced in place (a second one is
|
|
61
|
+
// dropped), every other line is kept as it is, and a missing name is added at the end. A new file is created
|
|
62
|
+
// readable only by its owner. Returns { created }.
|
|
63
|
+
export function writeEnvFile(path, values) {
|
|
64
|
+
const created = !existsSync(path);
|
|
65
|
+
const lines = created ? [] : readFileSync(path, "utf8").split("\n");
|
|
66
|
+
if (lines.length > 0 && lines[lines.length - 1] === "") lines.pop();
|
|
67
|
+
const written = new Set();
|
|
68
|
+
const out = [];
|
|
69
|
+
for (const line of lines) {
|
|
70
|
+
const name = /^\s*(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=/.exec(line)?.[1];
|
|
71
|
+
if (name && name in values) {
|
|
72
|
+
if (written.has(name)) continue;
|
|
73
|
+
out.push(`${name}=${values[name]}`);
|
|
74
|
+
written.add(name);
|
|
75
|
+
continue;
|
|
76
|
+
}
|
|
77
|
+
out.push(line);
|
|
78
|
+
}
|
|
79
|
+
for (const [name, value] of Object.entries(values)) if (!written.has(name)) out.push(`${name}=${value}`);
|
|
80
|
+
const text = out.join("\n") + "\n";
|
|
81
|
+
if (created) writeFileSync(path, text, { mode: 0o600 });
|
|
82
|
+
else writeFileSync(path, text);
|
|
83
|
+
return { created };
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
// "ignored", "not_ignored", "no_repo" (not inside a git work tree) or "no_git" (git is not installed).
|
|
87
|
+
export function gitIgnoreStatus(path, run = defaultRun) {
|
|
88
|
+
const cwd = dirname(path);
|
|
89
|
+
const inside = run("git", ["rev-parse", "--is-inside-work-tree"], { cwd });
|
|
90
|
+
if (inside.error) return "no_git";
|
|
91
|
+
if (inside.status !== 0 || inside.stdout.trim() !== "true") return "no_repo";
|
|
92
|
+
const check = run("git", ["check-ignore", "-q", basename(path)], { cwd });
|
|
93
|
+
return check.status === 0 ? "ignored" : "not_ignored";
|
|
94
|
+
}
|
|
95
|
+
|
|
96
|
+
// Adds the file's name to the .gitignore next to it, creating that .gitignore if there is none.
|
|
97
|
+
export function addToGitignore(path) {
|
|
98
|
+
const gitignore = join(dirname(path), ".gitignore");
|
|
99
|
+
const before = existsSync(gitignore) ? readFileSync(gitignore, "utf8") : "";
|
|
100
|
+
appendFileSync(gitignore, `${before === "" || before.endsWith("\n") ? "" : "\n"}${basename(path)}\n`);
|
|
101
|
+
return gitignore;
|
|
102
|
+
}
|
|
103
|
+
|
|
104
|
+
export function openBrowser(url, run = defaultRun, platform = process.platform) {
|
|
105
|
+
if (platform === "darwin") return run("open", [url]);
|
|
106
|
+
if (platform === "win32") return run("cmd", ["/c", "start", '""', url.replace(/&/g, "^&")], { windowsVerbatimArguments: true });
|
|
107
|
+
return run("xdg-open", [url]);
|
|
108
|
+
}
|
|
109
|
+
|
|
110
|
+
const vercelByHand = "To add it yourself: run vercel env add AGENT_TRAFFIC_KEY production and paste the AGENT_TRAFFIC_KEY value from your env file when asked, or add it in the Vercel dashboard under Settings, Environment Variables.";
|
|
111
|
+
|
|
112
|
+
// The key goes to the child on stdin only. Child output is shown only on failure, with the key cut out.
|
|
113
|
+
export async function saveKeyInVercel({ key, ask, say, run = defaultRun, cwd }) {
|
|
114
|
+
const hide = (text) => String(text || "").split(key).join("[key]").trim();
|
|
115
|
+
const version = run("vercel", ["--version"], { cwd });
|
|
116
|
+
if (version.error) {
|
|
117
|
+
say(" The vercel command was not found, so the key was not saved in Vercel.");
|
|
118
|
+
say(` ${vercelByHand}`);
|
|
119
|
+
return "vercel_missing";
|
|
120
|
+
}
|
|
121
|
+
const list = run("vercel", ["env", "ls", "production"], { cwd });
|
|
122
|
+
if (list.status !== 0) {
|
|
123
|
+
say(` vercel env ls failed, so the key was not saved in Vercel: ${hide(list.stderr) || `exit ${list.status}`}`);
|
|
124
|
+
say(` ${vercelByHand}`);
|
|
125
|
+
return "failed";
|
|
126
|
+
}
|
|
127
|
+
if (/(^|\s)AGENT_TRAFFIC_KEY(\s|$)/m.test(list.stdout)) {
|
|
128
|
+
const reply = await ask(" AGENT_TRAFFIC_KEY is already set in Vercel production. Replace it? (Y/n) ");
|
|
129
|
+
if (!/^(y(es)?)?$/i.test(reply.trim())) {
|
|
130
|
+
say(" Kept the existing AGENT_TRAFFIC_KEY in Vercel production.");
|
|
131
|
+
return "kept";
|
|
132
|
+
}
|
|
133
|
+
const removed = run("vercel", ["env", "rm", "AGENT_TRAFFIC_KEY", "production", "--yes"], { cwd });
|
|
134
|
+
if (removed.status !== 0) {
|
|
135
|
+
say(` vercel env rm failed, so the key was not replaced in Vercel: ${hide(removed.stderr) || `exit ${removed.status}`}`);
|
|
136
|
+
say(` ${vercelByHand}`);
|
|
137
|
+
return "failed";
|
|
138
|
+
}
|
|
139
|
+
}
|
|
140
|
+
const added = run("vercel", ["env", "add", "AGENT_TRAFFIC_KEY", "production"], { cwd, input: key });
|
|
141
|
+
if (added.status !== 0) {
|
|
142
|
+
say(` vercel env add failed, so the key was not saved in Vercel: ${hide(added.stderr) || `exit ${added.status}`}`);
|
|
143
|
+
say(` ${vercelByHand}`);
|
|
144
|
+
return "failed";
|
|
145
|
+
}
|
|
146
|
+
say(" Saved AGENT_TRAFFIC_KEY in Vercel production. It takes effect on the next deploy.");
|
|
147
|
+
return "saved";
|
|
148
|
+
}
|
|
149
|
+
|
|
150
|
+
// Runs the whole flow and returns the exit code: 0 connected, 1 denied, expired or failed.
|
|
151
|
+
export async function connect({ endpoint, envFile, vercel, cwd, interactive, fetchImpl = fetch, sleep, now = Date.now, run = defaultRun, ask, say, style }) {
|
|
152
|
+
const bold = style?.bold || ((s) => s);
|
|
153
|
+
const dim = style?.dim || ((s) => s);
|
|
154
|
+
say("");
|
|
155
|
+
say(` ${bold("alg connect")} · ${new URL(endpoint).host}`);
|
|
156
|
+
const start = await startDeviceLogin({ endpoint, fetchImpl });
|
|
157
|
+
say("");
|
|
158
|
+
say(` Your code: ${bold(start.user_code)}`);
|
|
159
|
+
const opened = openBrowser(start.verification_url, run);
|
|
160
|
+
if (opened.error || opened.status) say(` Open ${start.verification_url} in your browser.`);
|
|
161
|
+
else say(` Opened ${start.verification_url} in your browser.`);
|
|
162
|
+
say(` Check the code there matches, then approve. ${dim("Waiting…")}`);
|
|
163
|
+
|
|
164
|
+
const polled = await pollForKey({ endpoint, start, fetchImpl, sleep, now });
|
|
165
|
+
if (!polled.ok) {
|
|
166
|
+
say("");
|
|
167
|
+
say(` ${polled.message}`);
|
|
168
|
+
return 1;
|
|
169
|
+
}
|
|
170
|
+
const { key, project_id: projectId, project_name: projectName } = polled.result;
|
|
171
|
+
const trackEndpoint = (polled.result.endpoint || endpoint).replace(/\/+$/, "");
|
|
172
|
+
|
|
173
|
+
const values = { AGENT_TRAFFIC_KEY: key };
|
|
174
|
+
if (projectId) values.AGENT_TRAFFIC_PROJECT_ID = projectId;
|
|
175
|
+
// The middleware sends to the default host unless told otherwise.
|
|
176
|
+
if (trackEndpoint !== DEFAULT_ENDPOINT) values.AGENT_TRAFFIC_ENDPOINT = trackEndpoint;
|
|
177
|
+
const { created } = writeEnvFile(envFile, values);
|
|
178
|
+
const shown = relative(cwd, envFile).startsWith("..") ? envFile : relative(cwd, envFile);
|
|
179
|
+
say("");
|
|
180
|
+
say(` Connected to ${bold(projectName || projectId || "your Track project")}.`);
|
|
181
|
+
say(` Saved ${Object.keys(values).join(", ")} to ${shown}${created ? " (new file, readable only by you)" : ""}.`);
|
|
182
|
+
|
|
183
|
+
const ignored = gitIgnoreStatus(envFile, run);
|
|
184
|
+
if (ignored === "not_ignored") {
|
|
185
|
+
say(` ${shown} is not in .gitignore, so git would commit the key.`);
|
|
186
|
+
const reply = interactive ? await ask(` Add ${basename(envFile)} to .gitignore? (Y/n) `) : "n";
|
|
187
|
+
if (/^(y(es)?)?$/i.test(reply.trim())) {
|
|
188
|
+
const gitignore = addToGitignore(envFile);
|
|
189
|
+
say(` Added ${basename(envFile)} to ${relative(cwd, gitignore).startsWith("..") ? gitignore : relative(cwd, gitignore)}.`);
|
|
190
|
+
} else say(` Add ${basename(envFile)} to .gitignore before you commit.`);
|
|
191
|
+
} else if (ignored === "no_git") say(` ${dim("git was not found, so .gitignore was not checked.")}`);
|
|
192
|
+
|
|
193
|
+
const linked = existsSync(join(cwd, ".vercel", "project.json"));
|
|
194
|
+
if (vercel === true || (vercel !== false && linked)) {
|
|
195
|
+
say("");
|
|
196
|
+
if (interactive) {
|
|
197
|
+
const reply = await ask(" Also save the key in Vercel production? (Y/n) ");
|
|
198
|
+
if (/^(y(es)?)?$/i.test(reply.trim())) await saveKeyInVercel({ key, ask, say, run, cwd });
|
|
199
|
+
else say(" Not saved in Vercel.");
|
|
200
|
+
} else if (vercel === true) {
|
|
201
|
+
await saveKeyInVercel({ key, ask: async () => "y", say, run, cwd });
|
|
202
|
+
} else say(` ${dim("This folder is linked to Vercel. Run alg connect --vercel to save the key there too.")}`);
|
|
203
|
+
}
|
|
204
|
+
|
|
205
|
+
say("");
|
|
206
|
+
say(` Next: restart your server, then open ${trackEndpoint}/install to watch the first event.`);
|
|
207
|
+
say("");
|
|
208
|
+
return 0;
|
|
209
|
+
}
|
package/src/verify/index.js
CHANGED
|
@@ -54,7 +54,9 @@ export function parseVerifySpec(config) {
|
|
|
54
54
|
return { method: (match[1] || "GET").toUpperCase(), url: match[2], body: config.verify_body || null, header: { name: header.slice(0, colon).trim(), template: header.slice(colon + 1).trim() }, expect, assert, fields, exchange, hosts, cli: registries };
|
|
55
55
|
}
|
|
56
56
|
|
|
57
|
+
// spec is null when verification was skipped: the agent signs up, and its key is not checked.
|
|
57
58
|
export function describeSpec(spec) {
|
|
59
|
+
if (!spec) return "none: verification skipped, the agent's key is not checked";
|
|
58
60
|
const assertion = spec.assert ? (spec.assert.equals === null ? `, ${spec.assert.path} present` : `, ${spec.assert.path} = ${spec.assert.equals}`) : "";
|
|
59
61
|
const first = spec.exchange ? `${spec.exchange.method} ${spec.exchange.url} for a token (${spec.exchange.tokenPath}), then ` : "";
|
|
60
62
|
return `${first}${spec.method} ${spec.url} with ${spec.header.name}: ${spec.header.template}; expect ${spec.expect}${assertion}`;
|
|
@@ -155,23 +157,30 @@ export async function checkKey(spec, key, { fetchImpl = fetch, fields = {} } = {
|
|
|
155
157
|
export function networkFor(targetUrl, spec) {
|
|
156
158
|
const host = new URL(targetUrl).hostname;
|
|
157
159
|
const apex = registrableDomain(host);
|
|
158
|
-
const hosts = [host, apex, ...PRODUCT_SUBDOMAINS.map((s) => `${s}.${apex}`)
|
|
160
|
+
const hosts = [host, apex, ...PRODUCT_SUBDOMAINS.map((s) => `${s}.${apex}`)];
|
|
161
|
+
if (spec) hosts.push(new URL(spec.url.replace(/\{[^}]+\}/g, "x")).hostname, ...(spec.exchange ? [new URL(spec.exchange.url.replace(/\{[^}]+\}/g, "x")).hostname] : []), ...(spec.hosts || []));
|
|
159
162
|
return [...new Set(hosts)];
|
|
160
163
|
}
|
|
161
164
|
|
|
165
|
+
// With no spec the check is skipped: a key the agent saved counts, and the result says it was not checked.
|
|
166
|
+
async function skippedCheck() {
|
|
167
|
+
const checks = [{ id: "key_works", label: "Key works", pass: null, detail: "not checked: verification skipped (no verify_call)" }];
|
|
168
|
+
return { ok: true, status: 0, detail: checks[0].detail, objects: { checks, checkerInvalid: false, skipped: true } };
|
|
169
|
+
}
|
|
170
|
+
|
|
162
171
|
export function buildVerifyTask({ url, task, spec, fetchImpl = fetch }) {
|
|
163
172
|
const host = new URL(url).hostname;
|
|
164
173
|
return defineSignupTask({
|
|
165
174
|
id: "verify",
|
|
166
175
|
name: host,
|
|
167
176
|
url,
|
|
168
|
-
apiHost: new URL(spec.url.replace(/\{[^}]+\}/g, "x")).hostname,
|
|
177
|
+
apiHost: spec ? new URL(spec.url.replace(/\{[^}]+\}/g, "x")).hostname : host,
|
|
169
178
|
network: networkFor(url, spec),
|
|
170
179
|
credentialEnvName: KEY_ENV,
|
|
171
|
-
extraFields: spec.fields,
|
|
180
|
+
extraFields: spec ? spec.fields : [],
|
|
172
181
|
keyPattern: /^\S{8,}$/,
|
|
173
182
|
taskText: `${task}. The product is ${host}.`,
|
|
174
|
-
verify: { describe: describeSpec(spec), call: (key, runId, fields) => checkKey(spec, key, { fetchImpl, fields }) },
|
|
183
|
+
verify: { describe: describeSpec(spec), call: spec ? (key, runId, fields) => checkKey(spec, key, { fetchImpl, fields }) : skippedCheck },
|
|
175
184
|
});
|
|
176
185
|
}
|
|
177
186
|
|
|
@@ -184,6 +193,7 @@ const REPORTED_5XX = /\b(?:HTTP\s*(?:status)?|status(?:\s*code)?)\W{0,4}5\d\d\b|
|
|
|
184
193
|
export function classify(result, onboarding, resultMd = "") {
|
|
185
194
|
const evaluation = result.evaluation || {};
|
|
186
195
|
const stages = result.reconciled?.stages || [];
|
|
196
|
+
if (evaluation.success && evaluation.objects?.skipped) return { outcome: "passed", exitCode: 0, reason: "A real agent got its own key. The key was not checked: verification was skipped. Add verify_call to alg.yml to prove it works." };
|
|
187
197
|
if (evaluation.success) return { outcome: "passed", exitCode: 0, reason: "A real agent got its own key and the checker confirmed it works." };
|
|
188
198
|
if (evaluation.objects?.checkerInvalid) return { outcome: "inconclusive", exitCode: 3, reason: "The verify call answers without a key, so it cannot prove the agent's key works. Pick a call that needs authentication." };
|
|
189
199
|
if (stages.some((s) => s.agent?.outcome === "product_unavailable")) return { outcome: "inconclusive", exitCode: 3, reason: "The product returned server errors during the run. Try again later." };
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
import { parseVerifySpec } from "./index.js";
|
|
2
|
+
import { sameSite } from "../interface/sources.js";
|
|
3
|
+
|
|
4
|
+
// `test` found no verify_call in alg.yml and none in the product's docs. Instead of stopping, it asks the person
|
|
5
|
+
// at the terminal for the call, or lets them skip the check. ask(question) returns the typed line; say(line)
|
|
6
|
+
// prints one. Returns { call, header, apiHost, offSite } for a call, or { skip: true }.
|
|
7
|
+
|
|
8
|
+
export const EXAMPLE_CALL = "GET https://api.example.com/v1/me";
|
|
9
|
+
export const DEFAULT_HEADER = "Authorization: Bearer {key}";
|
|
10
|
+
|
|
11
|
+
export async function askVerifyCall({ ask, say, url, reason, style }) {
|
|
12
|
+
const dim = style?.dim || ((s) => s);
|
|
13
|
+
const bold = style?.bold || ((s) => s);
|
|
14
|
+
say("");
|
|
15
|
+
say(` We couldn't find an OpenAPI spec to figure out how to test your API key${reason ? ` ${dim(`(${reason})`)}` : ""}.`);
|
|
16
|
+
say(" What's a simple GET endpoint we can use to verify the key works?");
|
|
17
|
+
say(` ${dim(`e.g. ${EXAMPLE_CALL}. Press Enter with nothing to skip verification: the agent signs up, and its key is not checked.`)}`);
|
|
18
|
+
let spec;
|
|
19
|
+
for (;;) {
|
|
20
|
+
const reply = (await ask(" Call: ")).trim();
|
|
21
|
+
if (!reply || /^skip$/i.test(reply)) return { skip: true };
|
|
22
|
+
spec = parseVerifySpec({ verify_call: reply });
|
|
23
|
+
if (!spec.error) break;
|
|
24
|
+
say(` ${spec.error.replace(/^alg\.yml has no verify_call\. /, "")}`);
|
|
25
|
+
}
|
|
26
|
+
const call = `${spec.method} ${spec.url}`;
|
|
27
|
+
let header;
|
|
28
|
+
for (;;) {
|
|
29
|
+
const reply = (await ask(` Header that carries the key ${dim(`[${DEFAULT_HEADER}]`)}: `)).trim() || DEFAULT_HEADER;
|
|
30
|
+
const withHeader = parseVerifySpec({ verify_call: call, verify_header: reply });
|
|
31
|
+
if (!withHeader.error) {
|
|
32
|
+
header = reply;
|
|
33
|
+
break;
|
|
34
|
+
}
|
|
35
|
+
say(` ${withHeader.error}`);
|
|
36
|
+
}
|
|
37
|
+
const apiHost = new URL(spec.url.replace(/\{[^}]+\}/g, "x")).hostname;
|
|
38
|
+
const offSite = !sameSite(spec.url, url);
|
|
39
|
+
say(` ${dim("Will check with")} ${bold(call)} ${dim(`(${header})`)}`);
|
|
40
|
+
return { call, header, apiHost, offSite };
|
|
41
|
+
}
|