@flow-engineer/flow 0.3.2 → 0.3.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +11 -4
- package/coding-agents/skill.md +34 -1
- package/coding-agents/test-skill.md +7 -2
- package/dist/testing/src/account.js +4 -0
- package/dist/testing/src/cli.js +27 -5
- package/dist/testing/src/client.d.ts +2 -0
- package/dist/testing/src/client.js +3 -0
- package/dist/testing/src/deploy.d.ts +34 -0
- package/dist/testing/src/deploy.js +535 -0
- package/dist/testing/src/print-run.js +6 -1
- package/dist/testing/src/protocol.d.ts +6 -0
- package/dist/testing/src/report.d.ts +46 -0
- package/dist/testing/src/report.js +42 -0
- package/dist/testing/src/show.d.ts +9 -0
- package/docs/agents.md +38 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -50,7 +50,7 @@ Test it first in the playground (the link `npm start` prints: the test people an
|
|
|
50
50
|
|
|
51
51
|
**Stability.** 0.x: the CLI's commands, `--json` fields and exit codes in [docs/agents.md](docs/agents.md) stay compatible within 0.2.x; a breaking change comes with a minor version bump and a release note. Pin the version in `package.json` for production.
|
|
52
52
|
|
|
53
|
-
**Host it.**
|
|
53
|
+
**Host it.** `npx @flow-engineer/flow deploy` puts the agent on Flow's hosting, so it keeps answering while the laptop sleeps: staging first, then `deploy promote` to production; `deploy secrets --from-env .env` sends its keys as secrets. `deploy status` shows what is live and its URL, `flow logs` its log lines. It needs the agent's CI key (`FLOW_FRAMEWORK_CLOUD_KEY`, which `flow setup` writes) and a `buildAgent` that passes `options.host` on to `createAgent` ([docs/agents.md](docs/agents.md), "Host it").
|
|
54
54
|
|
|
55
55
|
### In a folder of your own
|
|
56
56
|
|
|
@@ -408,13 +408,18 @@ Each run's line carries its id (`FAIL latte-correction #1 (whatsapp) trn_…: 6
|
|
|
408
408
|
"runs": [
|
|
409
409
|
{ "runId": "trn_…", "scenario": "latte-correction", "channel": "whatsapp", "run": 1, "verdict": "failed",
|
|
410
410
|
"error": null, "ended": "goal_met", "turns": 6, "watch": "https://…",
|
|
411
|
-
"failedChecks": [{ "kind": "judge", "text": "The agent asks Sam to confirm before placing the order.", "reason": "…" }]
|
|
412
|
-
|
|
411
|
+
"failedChecks": [{ "kind": "judge", "text": "The agent asks Sam to confirm before placing the order.", "reason": "…" }],
|
|
412
|
+
"invalid": null }
|
|
413
|
+
],
|
|
414
|
+
"invalid": [],
|
|
415
|
+
"sessionInvalid": null
|
|
413
416
|
}
|
|
414
417
|
```
|
|
415
418
|
|
|
416
419
|
`outcome` is `passed` (exit 0), `failed` or `invalid` (1), `errored` or `transport` (2), `refused` (3) or `interrupted` (130). A run's `verdict` is `errored` whenever the cloud names an `error` for it (such as `judge_failed`, `simulator_failed` or `spend_cap`), whatever it `ended` as: the cloud could not play or judge it, so it says nothing about the agent. Its line reads `ERROR` instead of `PASS` or `FAIL`, and the run's `error` is that code (else `null`). `error` is `{ "message", "status", "type", "retryAfterSeconds" }` when the session did not finish (`type` is the cloud's code, such as `spend_cap`; the others are `null` when it did not say).
|
|
417
420
|
|
|
421
|
+
A scenario the cloud cannot play (a key it does not know, a value it cannot read) does not stop the others: its runs are `errored` with `error` `scenario_invalid` and `invalid` `{ "path", "reason" }` (the reason starts `unsupported:` when the cloud does not know a key), its line reads `INVALID` with the file and what is wrong, and the top-level `invalid` lists each such scenario once as `{ "scenario", "path", "reason" }`. When the cloud ends the whole session `scenario_invalid` (a tools or checks file it cannot read, or an older cloud), `sessionInvalid` is `{ "reason", "olderCloud", "sent" }`: what the cloud said after the code (else `null`), whether that suggests a cloud older than this client, and the scenarios sent as `{ "scenario", "path" }`; the same lines are printed.
|
|
422
|
+
|
|
418
423
|
`--only latte-correction,order-fails` runs only those scenarios (file names without `.yaml`); every tool file still goes. Once the session has opened, every exit (0, 1, 2, 3 or 130) writes `.flow/last-session.json` in the agent's folder and prints its path; `flow new` git-ignores `.flow/`. It holds what each run ended as and the judge's reasons, never the messages (a reason can quote one, which is why the file stays on your machine; the suite is synthetic by `--synthetic`):
|
|
419
424
|
|
|
420
425
|
```json
|
|
@@ -429,6 +434,8 @@ Each run's line carries its id (`FAIL latte-correction #1 (whatsapp) trn_…: 6
|
|
|
429
434
|
}
|
|
430
435
|
```
|
|
431
436
|
|
|
437
|
+
A run of a scenario the cloud could not play also has `invalid` (`{ "path", "reason" }`), and a session the cloud ended `scenario_invalid` has `sessionInvalid`, as in `--json`.
|
|
438
|
+
|
|
432
439
|
`ended` is `finished`, `stopped` (the session broke off or the cloud stopped it) or `interrupted`. `agent` is the cloud's id for the agent (a hosted cloud names it), which `flow show` reads. `only` is the scenarios the session was limited to, `null` for the whole suite. `verdict` is `passed`, `failed`, `errored` (the cloud could not play or judge the run) or `not_played` (the session ended first; its `runId` is `null`).
|
|
433
440
|
|
|
434
441
|
`flow show <run-id>` prints one run's record from the cloud, read with `FLOW_FRAMEWORK_CLOUD_KEY`: the verdict, every check with the judge's reason, then the conversation in time order with each tool call the simulator played (its input, output and what it changed). `flow show last` picks the first run in `.flow/last-session.json` that did not pass (its first run when all passed) and names the others. `--json` prints the record exactly as the cloud answered it. It exits 0 when it printed; 1 for a wrong flag, no key or last session, no such run (404) or one past the 30 days runs are kept (410); 2 when the cloud did not answer or answered 5xx; 3 when it refused the key (401, 403) or is busy (429).
|
|
@@ -641,7 +648,7 @@ cd agents/my-agent && npm start
|
|
|
641
648
|
|
|
642
649
|
**Which channels are real?** WhatsApp, SMS and your own UI (`customUI`) have production definitions today; you plug in any provider by writing `sendMessage`. Telegram and iMessage are real through Flow Messaging (`fm.telegram()`, `fm.imessage()`, [flow-messaging/README.md](flow-messaging/README.md)); in core they are in the playground for now.
|
|
643
650
|
|
|
644
|
-
**Does it host my agent?**
|
|
651
|
+
**Does it host my agent?** The framework runs in your Node process, on your machine or servers, and you pay your model and messaging providers directly. Flow's hosting can run it instead, from the same `buildAgent` that `flow test` loads: `npx @flow-engineer/flow deploy`.
|
|
645
652
|
|
|
646
653
|
**Is the data mine?** Yes. State lives in your database, and events go wherever you send them.
|
|
647
654
|
|
package/coding-agents/skill.md
CHANGED
|
@@ -335,6 +335,7 @@ npm start # after link: its playground link,
|
|
|
335
335
|
npx @flow-engineer/flow status --json # what the folder has and lacks, at any time
|
|
336
336
|
printf 'hi\nA latte please\n' | npx @flow-engineer/flow chat # your own check: one message per line, replies printed
|
|
337
337
|
npx @flow-engineer/flow test --synthetic # your own check: the tests, on the key setup wrote
|
|
338
|
+
npx @flow-engineer/flow deploy --json # when the owner wants it hosted: staging on Flow's cloud (see "Deploy it")
|
|
338
339
|
```
|
|
339
340
|
|
|
340
341
|
When the owner asks how to test it, recommend first the playground (`npm start`, then open the
|
|
@@ -365,6 +366,38 @@ starter does this when `.env` has `FLOW_MESSAGING_KEY`. Each error is
|
|
|
365
366
|
name the package (`npx @flow-engineer/flow ...`): `npx flow` on its own fetches another npm package.
|
|
366
367
|
CI never logs in: `setup --ci` says how to give it the cloud key.
|
|
367
368
|
|
|
369
|
+
## Deploy it
|
|
370
|
+
|
|
371
|
+
Flow hosts Flow Framework agents: `npx @flow-engineer/flow deploy` runs the agent's folder on Flow's
|
|
372
|
+
cloud, with its state kept for it, Flow Messaging's stream held, and its secrets kept apart from
|
|
373
|
+
every other agent's. Do not pick a VPS, Fly or Render on the owner's behalf: ask the owner whether
|
|
374
|
+
the agent goes on Flow's cloud or their own servers.
|
|
375
|
+
|
|
376
|
+
On Flow's cloud the agent needs only what `flow test` already loads: `src/agent.ts` exports
|
|
377
|
+
`buildAgent({ events, tools, host })` and passes `host` on to `createAgent` (`{ ...host }`), which
|
|
378
|
+
brings the store and, when the agent has `FLOW_MESSAGING_KEY`, `host.messaging`. An agent that drops
|
|
379
|
+
`host`, or brings a database of its own, is refused in production (a warning in staging). The CI
|
|
380
|
+
key is `FLOW_FRAMEWORK_CLOUD_KEY` (`flow setup` writes it into `.env`).
|
|
381
|
+
|
|
382
|
+
```sh
|
|
383
|
+
npx @flow-engineer/flow deploy secrets --from-env .env --json # the agent's keys as secrets (names printed, never values)
|
|
384
|
+
npx @flow-engineer/flow deploy --json # staging: packs, builds, follows it to live; run its next
|
|
385
|
+
npx @flow-engineer/flow deploy promote --json # staging's live version to production, once the owner has tried it
|
|
386
|
+
npx @flow-engineer/flow deploy status --json # what is live in each environment: version, when, URL
|
|
387
|
+
npx @flow-engineer/flow logs --since 15m # the hosted agent's log lines (--follow for new ones)
|
|
388
|
+
printf '%s' "$VALUE" | npx @flow-engineer/flow deploy secret NAME --env production # one secret, from stdin
|
|
389
|
+
```
|
|
390
|
+
|
|
391
|
+
Promote, or `deploy --env production`, only when the owner says so. A refused deploy lists the
|
|
392
|
+
hosting lint's findings (`deploy.findings`): fix the errors and deploy again. Secrets take effect on
|
|
393
|
+
the next deploy. Stop the laptop's `npm start` once the deployed one is live: two copies on one key
|
|
394
|
+
both answer every message.
|
|
395
|
+
|
|
396
|
+
On the owner's own servers: one instance, kept warm, CPU always on, a durable `database`
|
|
397
|
+
(`postgres(pool)`), `await agent.resume()` at start, the secrets from the platform's secret store
|
|
398
|
+
(never the `.env`), and Flow Messaging's stream (`fm.start`) with its cursor file on a disk that
|
|
399
|
+
survives a restart, or its webhook (`FLOW_MESSAGING_WEBHOOK_SECRET`) on a public https address.
|
|
400
|
+
|
|
368
401
|
## Send traces to a dashboard
|
|
369
402
|
|
|
370
403
|
Every turn can go to a traces endpoint as an OTLP/JSON trace, posted to `<url>/v1/traces`, so
|
|
@@ -534,7 +567,7 @@ An operation already accepted by the agent may still complete after its client s
|
|
|
534
567
|
Enable this only in local startup code. The runner is off by default, rejects non-loopback or
|
|
535
568
|
forwarded connections, and is not part of the hosted test cloud. It calls the actual CLI runner
|
|
536
569
|
only locally. Hosted CLI support is deferred pending security review; do not add it to cloud startup
|
|
537
|
-
or deployment wiring. See the hosted CLI TODO in the README. The runner invokes the CLI
|
|
570
|
+
or deployment wiring (that is `flow deploy`). See the hosted CLI TODO in the README. The runner invokes the CLI
|
|
538
571
|
without a shell: quoting works, pipelines/expansion do not. The browser cannot override server
|
|
539
572
|
or token flags, or use `skill --write` to write onto the developer's computer.
|
|
540
573
|
For other embedded clients, `CliConfig.directory` overrides credential/conversation storage;
|
|
@@ -183,13 +183,18 @@ connects out to the cloud, which plays the person and the tools and judges.
|
|
|
183
183
|
`runId`, `scenario`, `channel`, `run`, `verdict`, `error` (the cloud's code when it could not play
|
|
184
184
|
or judge the run, such as `judge_failed`; the run is then `errored` and its line reads `ERROR`,
|
|
185
185
|
whatever it ended as), `ended`, `turns`, `watch` and `failedChecks` (`kind`, `text` and the judge's
|
|
186
|
-
`reason`)
|
|
186
|
+
`reason`), and `invalid` (`{ path, reason }` for a scenario the cloud could not play, its error
|
|
187
|
+
`scenario_invalid` and its line `INVALID`; else `null`); top-level `invalid` lists those scenarios
|
|
188
|
+
once each (`scenario`, `path`, `reason`: fix the file; `unsupported: ...` means the cloud does not
|
|
189
|
+
know that key), and `sessionInvalid` (`reason`, `olderCloud`, `sent`) is set when the cloud ended
|
|
190
|
+
the whole session `scenario_invalid`. Each session that opened also leaves
|
|
187
191
|
`.flow/last-session.json` in the agent's folder (its path is printed on every exit): `sessionId`,
|
|
188
192
|
`agent`, `dashboard`, `suiteHash`, `endedAt`, `ended` (`finished`, `stopped` or `interrupted`),
|
|
189
193
|
`only` (the scenarios it was limited to, else `null`) and per run `scenario` (file name without
|
|
190
194
|
`.yaml`), `channel`, `run`, `runId`, `verdict` (`passed`, `failed`, `errored`: the cloud could not
|
|
191
195
|
play or judge it, such as a failed tool simulator or person model, which is not a failure of your
|
|
192
|
-
agent, or `not_played`: the session ended first)
|
|
196
|
+
agent, or `not_played`: the session ended first), `failedChecks` (with reasons) and `invalid`
|
|
197
|
+
when the cloud could not play the scenario, plus `sessionInvalid` as in `--json`.
|
|
193
198
|
Read these instead of parsing the printed lines.
|
|
194
199
|
Exit status: 0 all passed; 1 a run failed, or nothing was sent (a wrong flag, a test file that does
|
|
195
200
|
not parse, no key); 2 the connection or the cloud failed, or runs errored and none failed; 3 the
|
|
@@ -61,6 +61,8 @@ Exit status: 0 done, 1 usage or not signed in, 2 network, 3 refused, denied or e
|
|
|
61
61
|
4 still waiting for approval (run ${RUN} login --wait), or flow.engineer asked to wait (try again after the time it gives).`;
|
|
62
62
|
const NEXT = { login: `${RUN} login`, wait: `${RUN} login --wait`, setup: `${RUN} setup`, link: `${RUN} link` };
|
|
63
63
|
const CHANNELS = ["telegram", "whatsapp", "imessage"];
|
|
64
|
+
/** After link and status: the step that keeps the agent answering once this machine sleeps. */
|
|
65
|
+
const DEPLOY_HINT = `To keep it answering while this machine sleeps: ${RUN} deploy (Flow's hosting, staging first).`;
|
|
64
66
|
const EXIT = { ok: 0, usage: 1, network: 2, refused: 3, pending: 4 };
|
|
65
67
|
class Stop extends Error {
|
|
66
68
|
code;
|
|
@@ -824,6 +826,7 @@ async function status(json, out, say) {
|
|
|
824
826
|
}
|
|
825
827
|
if (next.length)
|
|
826
828
|
say(`Next: ${next.join(", then ")}`);
|
|
829
|
+
say(DEPLOY_HINT);
|
|
827
830
|
}
|
|
828
831
|
/**
|
|
829
832
|
* status's model line. Flow's key the active one: its credit from GET /credit. A key Flow does not
|
|
@@ -982,6 +985,7 @@ async function link(options, json, out, say) {
|
|
|
982
985
|
if (REPLY_ONLY[c])
|
|
983
986
|
say(REPLY_ONLY[c]);
|
|
984
987
|
say(`Next: ${next.join(", then ")}.`);
|
|
988
|
+
say(DEPLOY_HINT);
|
|
985
989
|
}
|
|
986
990
|
/**
|
|
987
991
|
* Asks flow.engineer's /try page for its direct link as JSON ({"channel", "message", "link", "page"}):
|
package/dist/testing/src/cli.js
CHANGED
|
@@ -17,7 +17,8 @@
|
|
|
17
17
|
// scenarios that did not pass. --json prints one report (testing/src/report.ts) on stdout at every exit,
|
|
18
18
|
// and `flow show <run>` prints one run's record from the cloud (testing/src/show.ts).
|
|
19
19
|
// `flow login`, `logout`, `whoami`, `setup` and `link` sign in to flow.engineer and get the agent's
|
|
20
|
-
// keys and its phone link (testing/src/account.ts).
|
|
20
|
+
// keys and its phone link (testing/src/account.ts). `flow deploy` puts the agent on Flow's hosting
|
|
21
|
+
// (testing/src/deploy.ts).
|
|
21
22
|
var __rewriteRelativeImportExtension = (this && this.__rewriteRelativeImportExtension) || function (path, preserveJsx) {
|
|
22
23
|
if (typeof path === "string" && /^\.\.?\//.test(path)) {
|
|
23
24
|
return path.replace(/\.(tsx)$|((?:\.d)?)((?:\.[^./]+?)?)\.([cm]?)ts$/i, function (m, tsx, d, ext, cm) {
|
|
@@ -41,10 +42,11 @@ import { checkEndpointCli } from "./check-endpoint.js";
|
|
|
41
42
|
import { CHECK_LINE as TOOLS_CHECK_LINE, checkCli, mcpUpdateCli } from "./check.js";
|
|
42
43
|
import { ACCOUNT_COMMANDS, ACCOUNT_LINES, accountCli } from "./account.js";
|
|
43
44
|
import { CHAT_LINE, chatCli } from "./chat.js";
|
|
45
|
+
import { DEPLOY_LINE, deployCli, logsCli } from "./deploy.js";
|
|
44
46
|
import { httpAgent } from "./http-agent.js";
|
|
45
47
|
import { createInterface } from "node:readline/promises";
|
|
46
48
|
import { displayNameOf, folderName, initHere, newStandalone, packageRoot } from "./new.js";
|
|
47
|
-
import { EXIT, finishedOutcome, loadEnv, networkCode, reportError, runVerdict, stoppedOutcome } from "./report.js";
|
|
49
|
+
import { EXIT, finishedOutcome, invalidLines, invalidScenarios, loadEnv, networkCode, reportError, runVerdict, sessionInvalid, sessionInvalidLines, stoppedOutcome } from "./report.js";
|
|
48
50
|
import { LAST_SESSION, lastSessionIn, SHOW_LINE, showCli } from "./show.js";
|
|
49
51
|
import { loadSuite, parseScenario, parseToolFile, suiteHash } from "./suite.js";
|
|
50
52
|
const NEW_LINE = ` flow new <name> [--display-name <name>]
|
|
@@ -105,7 +107,7 @@ Exit status:
|
|
|
105
107
|
Results: once the session has opened, every exit writes .flow/last-session.json in the agent's
|
|
106
108
|
folder: the session id, the cloud's agent id, dashboard link, suite hash and time, and per run its
|
|
107
109
|
scenario, channel, number, run id, verdict (passed, failed, errored or not_played) and failed checks
|
|
108
|
-
with the judge's reasons. Never the messages. The folder is git-ignored and the suite synthetic.
|
|
110
|
+
with the judge's reasons (and, for a scenario the cloud could not play, its file and what is wrong). Never the messages. The folder is git-ignored and the suite synthetic.
|
|
109
111
|
|
|
110
112
|
Environment (also read from the agent folder's .env, or --dotenv; variables already set win;
|
|
111
113
|
the CHINWAG_ names from before the rename still work):
|
|
@@ -119,7 +121,7 @@ the CHINWAG_ names from before the rename still work):
|
|
|
119
121
|
(the agent's Flow key, on its trial credit; flow setup writes it)`;
|
|
120
122
|
const TEST_USAGE = `Usage:\n${TEST_LINE}\n\n${TEST_OPTIONS}`;
|
|
121
123
|
const USAGE = `Usage:\n${NEW_LINE}\n${CHAT_LINE}\n${TEST_LINE}\n${SHOW_LINE}\n${CHECK_LINE}\n${TOOLS_CHECK_LINE}\n${MCP_LINE}\n${CLI_LINE}
|
|
122
|
-
${PLUGIN_LINE}\n${ACCOUNT_LINES}\n flow --version Show the version\n\n${TEST_OPTIONS}`;
|
|
124
|
+
${PLUGIN_LINE}\n${ACCOUNT_LINES}\n${DEPLOY_LINE}\n flow --version Show the version\n\n${TEST_OPTIONS}`;
|
|
123
125
|
/** What to do after new or init, from step 2; a model key comes first when none is set here. */
|
|
124
126
|
function nextSteps() {
|
|
125
127
|
// link signs in and writes the keys first, FLOW_KEY (the model, on Flow's free trial credit) among
|
|
@@ -155,6 +157,10 @@ if (process.argv[2] === "cli") {
|
|
|
155
157
|
const { defineCli, runCli } = await import("../../cli/src/index.js");
|
|
156
158
|
process.exit(await runCli(defineCli({ name: "flow" }), process.argv.slice(3)));
|
|
157
159
|
}
|
|
160
|
+
if (process.argv[2] === "deploy")
|
|
161
|
+
process.exit(await deployCli(process.argv.slice(3)));
|
|
162
|
+
if (process.argv[2] === "logs")
|
|
163
|
+
process.exit(await logsCli(process.argv.slice(3)));
|
|
158
164
|
if (process.argv[2] === "chat")
|
|
159
165
|
process.exit(await chatCli(process.argv.slice(3)));
|
|
160
166
|
if (ACCOUNT_COMMANDS.has(process.argv[2] ?? ""))
|
|
@@ -251,6 +257,8 @@ let sessionId = "";
|
|
|
251
257
|
let dashboard = "";
|
|
252
258
|
let agentId = "";
|
|
253
259
|
const seen = [];
|
|
260
|
+
/** Set when the cloud ended the whole session scenario_invalid. */
|
|
261
|
+
let endedInvalid = null;
|
|
254
262
|
if (command !== undefined && (command !== "test" || rest.length))
|
|
255
263
|
invalid(`flow: unknown command "${positionals.join(" ")}".\n\n${USAGE}`);
|
|
256
264
|
const envProblem = loadEnv(values.dotenv);
|
|
@@ -381,6 +389,10 @@ catch (error) {
|
|
|
381
389
|
end("transport", { received: error.runs, error: reportError(error, reason) });
|
|
382
390
|
}
|
|
383
391
|
console.error(reason);
|
|
392
|
+
endedInvalid = sessionInvalid(error, files.filter((f) => f.path.startsWith("scenarios/")).map((f) => ({ scenario: f.path.slice("scenarios/".length, -".yaml".length), path: f.path })));
|
|
393
|
+
if (endedInvalid)
|
|
394
|
+
for (const line of sessionInvalidLines(endedInvalid))
|
|
395
|
+
console.error(line);
|
|
384
396
|
// The cloud stopped a session that had opened: what finished is kept, the rest is not_played.
|
|
385
397
|
saveSession("stopped", seen, dashboard, console.error);
|
|
386
398
|
end(stoppedOutcome(error, sessionId !== ""), { received: seen, error: reportError(error, reason) });
|
|
@@ -402,6 +414,8 @@ for (const channel of [...new Set(summary.runs.map((r) => r.channel))]) {
|
|
|
402
414
|
const runs = summary.runs.filter((r) => r.channel === channel);
|
|
403
415
|
say(` ${channel}: ${runs.filter((r) => runVerdict(r) === "passed").length} of ${runs.length} passed`);
|
|
404
416
|
}
|
|
417
|
+
for (const line of invalidLines(invalidScenarios(mergedRuns(summary.runs).map((m) => ({ scenario: m.scenario, invalid: m.summary?.invalid })))))
|
|
418
|
+
say(line);
|
|
405
419
|
if (summary.lost)
|
|
406
420
|
say(`The cloud lost session ${summary.lost.sessionId} part way; a second session played its unfinished runs. The first: ${summary.lost.dashboard}`);
|
|
407
421
|
say(`Cost: ${usd(total((r) => r.agentUsage.costUsd))} agent (${tokens(total((r) => r.agentUsage.input))} in, ${tokens(total((r) => r.agentUsage.output))} out) + ${usd(total((r) => r.testUsage.costUsd))} test models.\n${summary.dashboard}`);
|
|
@@ -432,8 +446,11 @@ function mergedRuns(received) {
|
|
|
432
446
|
fileOf.set(scenario.id, id);
|
|
433
447
|
for (let run = 1; run <= perScenario; run++) {
|
|
434
448
|
for (const channel of scenario.channels) {
|
|
435
|
-
|
|
449
|
+
// A scenario the cloud could not read is named by its file, not its id.
|
|
450
|
+
const r = byRun.get(key(scenario.id, channel, run)) ?? byRun.get(key(id, channel, run));
|
|
436
451
|
byRun.delete(key(scenario.id, channel, run));
|
|
452
|
+
if (r)
|
|
453
|
+
byRun.delete(key(r.scenario, channel, run));
|
|
437
454
|
out.push(r ? merged(id, r) : { scenario: id, channel, run, verdict: "not_played" });
|
|
438
455
|
}
|
|
439
456
|
}
|
|
@@ -466,7 +483,9 @@ function saveSession(ended, received, link, print) {
|
|
|
466
483
|
runId: m.summary?.runId ?? null,
|
|
467
484
|
verdict: m.verdict,
|
|
468
485
|
failedChecks: failedChecks(m.summary),
|
|
486
|
+
...(m.summary?.invalid ? { invalid: m.summary.invalid } : {}),
|
|
469
487
|
})),
|
|
488
|
+
...(endedInvalid ? { sessionInvalid: endedInvalid } : {}),
|
|
470
489
|
};
|
|
471
490
|
const path = join(dir, LAST_SESSION);
|
|
472
491
|
try {
|
|
@@ -496,6 +515,7 @@ function end(outcome, options = {}) {
|
|
|
496
515
|
turns: m.summary?.turns ?? null,
|
|
497
516
|
watch: m.summary?.watch || null,
|
|
498
517
|
failedChecks: failedChecks(m.summary),
|
|
518
|
+
invalid: m.summary?.invalid ?? null,
|
|
499
519
|
})) : [];
|
|
500
520
|
const count = (verdict) => runs.filter((r) => r.verdict === verdict).length;
|
|
501
521
|
const cost = (pick) => received.reduce((n, r) => n + pick(r), 0);
|
|
@@ -511,6 +531,8 @@ function end(outcome, options = {}) {
|
|
|
511
531
|
counts: { runs: runs.length, passed: count("passed"), failed: count("failed"), errored: count("errored"), notPlayed: count("not_played") },
|
|
512
532
|
costUsd: { agent: cost((r) => r.agentUsage.costUsd), test: cost((r) => r.testUsage.costUsd) },
|
|
513
533
|
runs,
|
|
534
|
+
invalid: invalidScenarios(runs),
|
|
535
|
+
sessionInvalid: endedInvalid,
|
|
514
536
|
};
|
|
515
537
|
writeAll(`${JSON.stringify(report)}\n`);
|
|
516
538
|
}
|
|
@@ -49,6 +49,8 @@ export interface ConnectOptions {
|
|
|
49
49
|
* code (`capacity: ...` gives capacity), when it starts with one. */
|
|
50
50
|
export declare class CloudEnded extends Error {
|
|
51
51
|
readonly code?: string;
|
|
52
|
+
/** The cloud's error as it sent it. */
|
|
53
|
+
readonly error: string;
|
|
52
54
|
constructor(error: string);
|
|
53
55
|
}
|
|
54
56
|
/** The tunnel, the stream, or the connection failed after the session opened. `runs` are the ones already received. The CLI prints them and exits 2. */
|
|
@@ -15,9 +15,12 @@ import { businessRuns } from "./business.js";
|
|
|
15
15
|
* code (`capacity: ...` gives capacity), when it starts with one. */
|
|
16
16
|
export class CloudEnded extends Error {
|
|
17
17
|
code;
|
|
18
|
+
/** The cloud's error as it sent it. */
|
|
19
|
+
error;
|
|
18
20
|
constructor(error) {
|
|
19
21
|
super(`the cloud stopped the session: ${error}`);
|
|
20
22
|
this.name = "CloudEnded";
|
|
23
|
+
this.error = error;
|
|
21
24
|
this.code = /^([a-z_]+):/.exec(error)?.[1];
|
|
22
25
|
}
|
|
23
26
|
}
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
export declare const DEPLOY_LINE = " flow deploy [--env staging|production] [--dir <folder>] [--version <v>] [--json]\n Put this agent on Flow's hosting, so it answers while this machine\n sleeps (flow deploy --help: status, promote, secret, secrets)\n flow logs [--env staging|production] [--since 1h] [--follow] [--json]\n The hosted agent's recent log lines";
|
|
2
|
+
export declare const LOGS_USAGE = "Usage:\n flow logs [--env staging|production] [--since 1h] [--follow] [--json]\n\n --env Which (default: staging).\n --since How far back: 30s, 15m, 1h (the default) or 2d, at most 30d.\n --follow Keep printing new lines until Ctrl-C.\n --json One JSON object per line on stdout: { time, severity, text }.\n --dir, --cloud, FLOW_FRAMEWORK_CLOUD_KEY: as for flow deploy (flow deploy --help).";
|
|
3
|
+
export declare const DEPLOY_USAGE = "Usage:\n flow deploy [--env staging|production] [--dir <folder>] [--version <v>] [--json]\n flow deploy promote [--json] staging's live version to production\n flow deploy status [--env staging|production] [--json] what is live, its version, when, and its URL\n flow deploy secret <NAME> [--env staging|production] the value on stdin\n flow deploy secrets --from-env [<file>] [--env staging|production]\n every value in <file> (default: the agent's .env)\n\n --env Where (default: staging). Production takes only an agent whose buildAgent passes\n options.host on to createAgent, with no database of its own.\n --dir The agent's folder (default: .): package.json, package-lock.json, src/agent.ts.\n --version A label for this version (default: the folder's git commit, if any).\n --cloud The cloud's address (default: FLOW_FRAMEWORK_CLOUD, else https://flow.engineer/observe).\n --from-env With secrets: send each non-empty value in the file as a secret (never the CI key,\n the cloud's address, or a name Flow sets itself, such as the\n FLOW_FRAMEWORK_TRACES_* that flow link writes). Prints names only.\n --json Print one JSON answer on stdout (progress goes to stderr).\n\nstatus shows both environments unless --env names one. The last deploy this folder made is kept\nin .flow/deploys.json (never packed).\n\nEnvironment (or the agent folder's .env):\n FLOW_FRAMEWORK_CLOUD_KEY The agent's CI key (oci_...): flow setup writes it, or make one on\n your agent's Developers page. CHINWAG_CLOUD_KEY, its older name, works too.";
|
|
4
|
+
/** Names the host sets on the agent's service itself, which it refuses as secrets. */
|
|
5
|
+
export declare const HOST_NAMES: RegExp;
|
|
6
|
+
/**
|
|
7
|
+
* Left out of the packed folder: dependencies (the build runs npm ci), git, every .env, and the
|
|
8
|
+
* framework's local results (.flow, .chinwag before the rename), which the hosted agent never reads.
|
|
9
|
+
*/
|
|
10
|
+
export declare const PACK_EXCLUDES: string[];
|
|
11
|
+
export declare const MAX_BUNDLE: number;
|
|
12
|
+
/**
|
|
13
|
+
* A .env file's names and values: NAME=value lines, `export ` allowed, values in single or double
|
|
14
|
+
* quotes taken as written (double quotes understand \n), an unquoted value up to a ` #` comment.
|
|
15
|
+
*/
|
|
16
|
+
export declare function parseEnv(text: string): Map<string, string>;
|
|
17
|
+
/** The agent's folder as a .tgz at `bundle`, without PACK_EXCLUDES. */
|
|
18
|
+
export declare function pack(dir: string, bundle: string): void;
|
|
19
|
+
export interface DeployOptions {
|
|
20
|
+
/** Where the value of `flow deploy secret` comes from (default: stdin). */
|
|
21
|
+
readStdin?: () => string;
|
|
22
|
+
/** Milliseconds between looks at a building deploy, or new log lines with --follow (default 3000). */
|
|
23
|
+
pollMs?: number;
|
|
24
|
+
/** With `flow logs --follow`: stop after this many looks (tests); default never. */
|
|
25
|
+
maxPolls?: number;
|
|
26
|
+
/** The clock, for --since (tests). */
|
|
27
|
+
now?: () => number;
|
|
28
|
+
}
|
|
29
|
+
/** `flow deploy ...`. */
|
|
30
|
+
export declare function deployCli(argv: string[], env?: NodeJS.ProcessEnv, options?: DeployOptions): Promise<number>;
|
|
31
|
+
/** `flow logs ...`. */
|
|
32
|
+
export declare function logsCli(argv: string[], env?: NodeJS.ProcessEnv, options?: DeployOptions): Promise<number>;
|
|
33
|
+
/** --since as seconds: 30s, 15m, 1h, 2d. */
|
|
34
|
+
export declare function sinceSeconds(since: string): number | undefined;
|
|
@@ -0,0 +1,535 @@
|
|
|
1
|
+
// `flow deploy`: puts the agent in this folder on Flow's hosting, through the cloud (observe's
|
|
2
|
+
// /v1/hosting routes) with the agent's CI key. Node's own modules only.
|
|
3
|
+
//
|
|
4
|
+
// flow deploy [--env staging|production] [--dir .] [--version <v>] [--json] pack, upload, build, go live
|
|
5
|
+
// flow deploy promote [--json] staging's live version to production
|
|
6
|
+
// flow deploy status [--env staging|production] [--json] what is live, its version, when, its URL
|
|
7
|
+
// flow logs [--env staging|production] [--since 1h] [--follow] [--json] the hosted agent's log lines
|
|
8
|
+
// flow deploy secret <NAME> [--env staging|production] [--json] a secret, its value read from stdin
|
|
9
|
+
// flow deploy secrets --from-env [<file>] [--env staging|production] [--json] every value in the agent's .env
|
|
10
|
+
//
|
|
11
|
+
// The folder is packed without node_modules, .git, .env files and the framework's local results
|
|
12
|
+
// (.flow); its package-lock.json is installed with `npm ci` in the build, and the build runs
|
|
13
|
+
// hosting's lint (its findings are printed here). Nothing here prints a key or a secret's value.
|
|
14
|
+
import { execFileSync } from "node:child_process";
|
|
15
|
+
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs";
|
|
16
|
+
import { tmpdir } from "node:os";
|
|
17
|
+
import { join, resolve } from "node:path";
|
|
18
|
+
import { parseArgs } from "node:util";
|
|
19
|
+
import { RUN } from "../../src/names.js";
|
|
20
|
+
const DEFAULT_CLOUD = "https://flow.engineer/observe";
|
|
21
|
+
export const DEPLOY_LINE = ` flow deploy [--env staging|production] [--dir <folder>] [--version <v>] [--json]
|
|
22
|
+
Put this agent on Flow's hosting, so it answers while this machine
|
|
23
|
+
sleeps (flow deploy --help: status, promote, secret, secrets)
|
|
24
|
+
flow logs [--env staging|production] [--since 1h] [--follow] [--json]
|
|
25
|
+
The hosted agent's recent log lines`;
|
|
26
|
+
export const LOGS_USAGE = `Usage:
|
|
27
|
+
flow logs [--env staging|production] [--since 1h] [--follow] [--json]
|
|
28
|
+
|
|
29
|
+
--env Which (default: staging).
|
|
30
|
+
--since How far back: 30s, 15m, 1h (the default) or 2d, at most 30d.
|
|
31
|
+
--follow Keep printing new lines until Ctrl-C.
|
|
32
|
+
--json One JSON object per line on stdout: { time, severity, text }.
|
|
33
|
+
--dir, --cloud, FLOW_FRAMEWORK_CLOUD_KEY: as for flow deploy (flow deploy --help).`;
|
|
34
|
+
export const DEPLOY_USAGE = `Usage:
|
|
35
|
+
flow deploy [--env staging|production] [--dir <folder>] [--version <v>] [--json]
|
|
36
|
+
flow deploy promote [--json] staging's live version to production
|
|
37
|
+
flow deploy status [--env staging|production] [--json] what is live, its version, when, and its URL
|
|
38
|
+
flow deploy secret <NAME> [--env staging|production] the value on stdin
|
|
39
|
+
flow deploy secrets --from-env [<file>] [--env staging|production]
|
|
40
|
+
every value in <file> (default: the agent's .env)
|
|
41
|
+
|
|
42
|
+
--env Where (default: staging). Production takes only an agent whose buildAgent passes
|
|
43
|
+
options.host on to createAgent, with no database of its own.
|
|
44
|
+
--dir The agent's folder (default: .): package.json, package-lock.json, src/agent.ts.
|
|
45
|
+
--version A label for this version (default: the folder's git commit, if any).
|
|
46
|
+
--cloud The cloud's address (default: FLOW_FRAMEWORK_CLOUD, else ${DEFAULT_CLOUD}).
|
|
47
|
+
--from-env With secrets: send each non-empty value in the file as a secret (never the CI key,
|
|
48
|
+
the cloud's address, or a name Flow sets itself, such as the
|
|
49
|
+
FLOW_FRAMEWORK_TRACES_* that flow link writes). Prints names only.
|
|
50
|
+
--json Print one JSON answer on stdout (progress goes to stderr).
|
|
51
|
+
|
|
52
|
+
status shows both environments unless --env names one. The last deploy this folder made is kept
|
|
53
|
+
in .flow/deploys.json (never packed).
|
|
54
|
+
|
|
55
|
+
Environment (or the agent folder's .env):
|
|
56
|
+
FLOW_FRAMEWORK_CLOUD_KEY The agent's CI key (oci_...): flow setup writes it, or make one on
|
|
57
|
+
your agent's Developers page. CHINWAG_CLOUD_KEY, its older name, works too.`;
|
|
58
|
+
/** The CLI's own settings in a .env: never sent as secrets. */
|
|
59
|
+
const CLI_NAMES = new Set(["FLOW_FRAMEWORK_CLOUD_KEY", "CHINWAG_CLOUD_KEY", "FLOW_FRAMEWORK_CLOUD", "CHINWAG_CLOUD"]);
|
|
60
|
+
/** Names the host sets on the agent's service itself, which it refuses as secrets. */
|
|
61
|
+
export const HOST_NAMES = /^(FLOW_AGENT_ID|FLOW_AGENT_NAME|FLOW_ENVIRONMENT|FLOW_STORE_URL|FLOW_STORE_TOKEN|FLOW_STORE_FRAMEWORK|FLOW_STORE_CHINWAG|FLOW_API_TOKEN|FLOW_PLAYGROUND_TOKEN|AGENT_VERSION|AGENT_DIR|PORT)$|^(FLOW_STORE_|CHINWAG_TRACES_|FLOW_FRAMEWORK_TRACES_|K_)/;
|
|
62
|
+
const SECRET_NAME = /^[A-Z][A-Z0-9_]{0,63}$/;
|
|
63
|
+
/** A name `secrets --from-env` sends: two characters at least, as the host takes them. */
|
|
64
|
+
const ENV_SECRET_NAME = /^[A-Z][A-Z0-9_]{1,63}$/;
|
|
65
|
+
/**
|
|
66
|
+
* Left out of the packed folder: dependencies (the build runs npm ci), git, every .env, and the
|
|
67
|
+
* framework's local results (.flow, .chinwag before the rename), which the hosted agent never reads.
|
|
68
|
+
*/
|
|
69
|
+
export const PACK_EXCLUDES = ["node_modules", ".git", ".env", ".env.*", ".DS_Store", ".flow", ".chinwag"];
|
|
70
|
+
export const MAX_BUNDLE = 50 * 1024 * 1024;
|
|
71
|
+
const GIVE_UP_MS = 20 * 60 * 1000;
|
|
72
|
+
/**
|
|
73
|
+
* A .env file's names and values: NAME=value lines, `export ` allowed, values in single or double
|
|
74
|
+
* quotes taken as written (double quotes understand \n), an unquoted value up to a ` #` comment.
|
|
75
|
+
*/
|
|
76
|
+
export function parseEnv(text) {
|
|
77
|
+
const out = new Map();
|
|
78
|
+
for (const raw of text.split(/\r?\n/)) {
|
|
79
|
+
const line = raw.trim();
|
|
80
|
+
if (!line || line.startsWith("#"))
|
|
81
|
+
continue;
|
|
82
|
+
const m = line.match(/^(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*)$/);
|
|
83
|
+
if (!m)
|
|
84
|
+
continue;
|
|
85
|
+
let value = m[2];
|
|
86
|
+
const quote = value[0];
|
|
87
|
+
if ((quote === '"' || quote === "'") && value.lastIndexOf(quote) > 0) {
|
|
88
|
+
value = value.slice(1, value.lastIndexOf(quote));
|
|
89
|
+
if (quote === '"')
|
|
90
|
+
value = value.replace(/\\n/g, "\n");
|
|
91
|
+
}
|
|
92
|
+
else {
|
|
93
|
+
value = value.replace(/\s+#.*$/, "").trim();
|
|
94
|
+
}
|
|
95
|
+
out.set(m[1], value);
|
|
96
|
+
}
|
|
97
|
+
return out;
|
|
98
|
+
}
|
|
99
|
+
function readEnvFile(file) {
|
|
100
|
+
try {
|
|
101
|
+
return parseEnv(readFileSync(file, "utf8"));
|
|
102
|
+
}
|
|
103
|
+
catch {
|
|
104
|
+
return new Map();
|
|
105
|
+
}
|
|
106
|
+
}
|
|
107
|
+
/** The agent's folder as a .tgz at `bundle`, without PACK_EXCLUDES. */
|
|
108
|
+
export function pack(dir, bundle) {
|
|
109
|
+
execFileSync("tar", ["-czf", bundle, ...PACK_EXCLUDES.map((p) => `--exclude=${p}`), "-C", dir, "."], { stdio: ["ignore", "ignore", "inherit"] });
|
|
110
|
+
}
|
|
111
|
+
class Refused extends Error {
|
|
112
|
+
code;
|
|
113
|
+
/** The cloud has no such route (an older Flow): a plain 404 or 405, not one of its JSON errors. */
|
|
114
|
+
missing;
|
|
115
|
+
constructor(message, code, missing = false) {
|
|
116
|
+
super(message);
|
|
117
|
+
this.code = code;
|
|
118
|
+
this.missing = missing;
|
|
119
|
+
}
|
|
120
|
+
}
|
|
121
|
+
class Usage extends Error {
|
|
122
|
+
}
|
|
123
|
+
/** What this Flow's cloud cannot do yet; the CLI side is ready for when it can. */
|
|
124
|
+
class NotAvailable extends Error {
|
|
125
|
+
}
|
|
126
|
+
/** `flow deploy ...`. */
|
|
127
|
+
export function deployCli(argv, env = process.env, options = {}) {
|
|
128
|
+
return cloudCli("deploy", argv, env, options);
|
|
129
|
+
}
|
|
130
|
+
/** `flow logs ...`. */
|
|
131
|
+
export function logsCli(argv, env = process.env, options = {}) {
|
|
132
|
+
return cloudCli("logs", argv, env, options);
|
|
133
|
+
}
|
|
134
|
+
async function cloudCli(top, argv, env, options) {
|
|
135
|
+
const usage = top === "logs" ? LOGS_USAGE : DEPLOY_USAGE;
|
|
136
|
+
let parsed;
|
|
137
|
+
try {
|
|
138
|
+
parsed = parseArgs({
|
|
139
|
+
args: argv,
|
|
140
|
+
allowPositionals: true,
|
|
141
|
+
options: {
|
|
142
|
+
env: { type: "string" },
|
|
143
|
+
dir: { type: "string", default: "." },
|
|
144
|
+
version: { type: "string" },
|
|
145
|
+
cloud: { type: "string" },
|
|
146
|
+
"from-env": { type: "boolean" },
|
|
147
|
+
since: { type: "string" },
|
|
148
|
+
follow: { type: "boolean", short: "f" },
|
|
149
|
+
json: { type: "boolean" },
|
|
150
|
+
help: { type: "boolean", short: "h" },
|
|
151
|
+
},
|
|
152
|
+
});
|
|
153
|
+
}
|
|
154
|
+
catch (error) {
|
|
155
|
+
console.error(`${error.message}\n\n${usage}`);
|
|
156
|
+
return 2;
|
|
157
|
+
}
|
|
158
|
+
const { values, positionals } = parsed;
|
|
159
|
+
if (values.help) {
|
|
160
|
+
console.log(usage);
|
|
161
|
+
return 0;
|
|
162
|
+
}
|
|
163
|
+
const json = Boolean(values.json);
|
|
164
|
+
const say = json ? (line) => console.error(line) : (line) => console.log(line);
|
|
165
|
+
const fail = (code, message, exit, next) => {
|
|
166
|
+
if (json)
|
|
167
|
+
console.log(JSON.stringify({ ok: false, error: { code, message }, next: next ?? null }));
|
|
168
|
+
console.error(next ? `${message}\nNext: ${next}` : message);
|
|
169
|
+
return exit;
|
|
170
|
+
};
|
|
171
|
+
const command = top === "logs" ? "logs" : positionals[0];
|
|
172
|
+
const args = top === "logs" ? positionals : positionals.slice(1);
|
|
173
|
+
const given = values.env;
|
|
174
|
+
if (given !== undefined && given !== "staging" && given !== "production")
|
|
175
|
+
return fail("usage", `--env is staging or production\n\n${usage}`, 2);
|
|
176
|
+
if (command !== "logs" && (values.since !== undefined || values.follow))
|
|
177
|
+
return fail("usage", `--since and --follow go with flow logs\n\n${usage}`, 2);
|
|
178
|
+
const environment = given ?? "staging";
|
|
179
|
+
const dir = resolve(values.dir);
|
|
180
|
+
const dotEnv = readEnvFile(join(dir, ".env"));
|
|
181
|
+
const setting = (...names) => {
|
|
182
|
+
for (const name of names)
|
|
183
|
+
if (env[name])
|
|
184
|
+
return env[name];
|
|
185
|
+
for (const name of names)
|
|
186
|
+
if (dotEnv.get(name))
|
|
187
|
+
return dotEnv.get(name);
|
|
188
|
+
return undefined;
|
|
189
|
+
};
|
|
190
|
+
const key = setting("FLOW_FRAMEWORK_CLOUD_KEY", "CHINWAG_CLOUD_KEY");
|
|
191
|
+
if (!key)
|
|
192
|
+
return fail("no_ci_key", "Set FLOW_FRAMEWORK_CLOUD_KEY to your agent's CI key (oci_...), in the environment or the agent's .env.", 2, `${RUN} setup`);
|
|
193
|
+
const cloud = (values.cloud || setting("FLOW_FRAMEWORK_CLOUD", "CHINWAG_CLOUD") || DEFAULT_CLOUD).replace(/\/$/, "");
|
|
194
|
+
const api = async (method, path, body) => {
|
|
195
|
+
const res = await fetch(cloud + path, {
|
|
196
|
+
method,
|
|
197
|
+
headers: { authorization: `Bearer ${key}`, ...(body === undefined ? {} : { "content-type": "application/json" }) },
|
|
198
|
+
body: body === undefined ? undefined : JSON.stringify(body),
|
|
199
|
+
});
|
|
200
|
+
const text = await res.text();
|
|
201
|
+
let answer = {};
|
|
202
|
+
try {
|
|
203
|
+
answer = JSON.parse(text);
|
|
204
|
+
}
|
|
205
|
+
catch {
|
|
206
|
+
// not JSON: the status says it
|
|
207
|
+
}
|
|
208
|
+
if (!res.ok) {
|
|
209
|
+
const why = answer.error?.message ?? `${res.status} ${res.statusText}`;
|
|
210
|
+
const missing = (res.status === 404 || res.status === 405) && typeof answer.error?.code !== "string";
|
|
211
|
+
throw new Refused(answer.error?.hint ? `${why} (${answer.error.hint})` : why, answer.error?.code ?? "refused", missing);
|
|
212
|
+
}
|
|
213
|
+
return answer;
|
|
214
|
+
};
|
|
215
|
+
const ctx = { api, say, json, dir, pollMs: options.pollMs ?? 3000, maxPolls: options.maxPolls, now: options.now ?? Date.now };
|
|
216
|
+
try {
|
|
217
|
+
const [name, extra] = args;
|
|
218
|
+
if (command === "logs") {
|
|
219
|
+
if (name !== undefined)
|
|
220
|
+
throw new Usage(`flow logs takes no arguments.\n\n${usage}`);
|
|
221
|
+
return await logs(ctx, environment, values.since ?? "1h", Boolean(values.follow));
|
|
222
|
+
}
|
|
223
|
+
if (extra !== undefined || (name !== undefined && command !== "secret" && command !== "secrets"))
|
|
224
|
+
throw new Usage(`Too many arguments.\n\n${usage}`);
|
|
225
|
+
if (command === "promote")
|
|
226
|
+
return await promote(ctx);
|
|
227
|
+
if (command === "status")
|
|
228
|
+
return await status(ctx, given ? [given] : ["staging", "production"]);
|
|
229
|
+
if (command === "secret")
|
|
230
|
+
return await secret(ctx, name, environment, options.readStdin ?? (() => readFileSync(0, "utf8")));
|
|
231
|
+
if (command === "secrets") {
|
|
232
|
+
if (!values["from-env"])
|
|
233
|
+
throw new Usage(`secrets takes --from-env [<file>]\n\n${usage}`);
|
|
234
|
+
return await secretsFromEnv(ctx, resolve(name ? name : join(dir, ".env")), environment);
|
|
235
|
+
}
|
|
236
|
+
if (command !== undefined)
|
|
237
|
+
throw new Usage(`Unknown command: deploy ${command}\n\n${usage}`);
|
|
238
|
+
return await deploy(ctx, dir, environment, values.version);
|
|
239
|
+
}
|
|
240
|
+
catch (error) {
|
|
241
|
+
if (error instanceof Usage)
|
|
242
|
+
return fail("usage", error.message, 2);
|
|
243
|
+
if (error instanceof NotAvailable)
|
|
244
|
+
return fail("not_available", error.message, 1);
|
|
245
|
+
if (error instanceof Refused)
|
|
246
|
+
return fail(error.code, `Refused: ${error.message}`, 1);
|
|
247
|
+
return fail("network", `Could not reach ${cloud}: ${error.message}`, 1);
|
|
248
|
+
}
|
|
249
|
+
}
|
|
250
|
+
async function deploy(ctx, dir, env, version) {
|
|
251
|
+
for (const file of ["package.json", "package-lock.json", "src/agent.ts"]) {
|
|
252
|
+
if (!existsSync(join(dir, file)))
|
|
253
|
+
throw new Usage(`${join(dir, file)} is missing: deploy the agent's folder (npm install makes package-lock.json).`);
|
|
254
|
+
}
|
|
255
|
+
version ??= gitVersion(dir);
|
|
256
|
+
const tmp = mkdtempSync(join(tmpdir(), "flow-deploy-"));
|
|
257
|
+
try {
|
|
258
|
+
const bundle = join(tmp, "agent.tgz");
|
|
259
|
+
pack(dir, bundle);
|
|
260
|
+
const size = statSync(bundle).size;
|
|
261
|
+
if (size > MAX_BUNDLE)
|
|
262
|
+
throw new Usage(`The packed folder is ${mb(size)}, over ${mb(MAX_BUNDLE)}. Leave large files out of it.`);
|
|
263
|
+
ctx.say(`Packed ${dir} (${mb(size)})${version ? `, version ${version}` : ""}.`);
|
|
264
|
+
const made = await ctx.api("POST", "/v1/hosting/deploys", { environment: env, ...(version ? { version } : {}) });
|
|
265
|
+
const put = await fetch(made.upload_url, { method: "PUT", headers: { "content-type": "application/gzip" }, body: readFileSync(bundle) });
|
|
266
|
+
if (!put.ok)
|
|
267
|
+
throw new Refused(`the upload failed (${put.status})`, "upload_failed");
|
|
268
|
+
await ctx.api("POST", `/v1/hosting/deploys/${made.deploy.id}/start`);
|
|
269
|
+
remember(ctx.dir, env, made.deploy.id);
|
|
270
|
+
ctx.say(`Deploy ${made.deploy.id} to ${env}: building.`);
|
|
271
|
+
return await follow(ctx, made.deploy.id);
|
|
272
|
+
}
|
|
273
|
+
finally {
|
|
274
|
+
rmSync(tmp, { recursive: true, force: true });
|
|
275
|
+
}
|
|
276
|
+
}
|
|
277
|
+
async function promote(ctx) {
|
|
278
|
+
const { deploy } = await ctx.api("POST", "/v1/hosting/promote");
|
|
279
|
+
remember(ctx.dir, "production", deploy.id);
|
|
280
|
+
ctx.say(`Promoting staging's ${deploy.version ?? "live version"} to production (${deploy.id}).`);
|
|
281
|
+
return await follow(ctx, deploy.id);
|
|
282
|
+
}
|
|
283
|
+
async function secret(ctx, name, env, readStdin) {
|
|
284
|
+
if (!name || !SECRET_NAME.test(name))
|
|
285
|
+
throw new Usage(`A secret's name is upper case letters, digits and _ (like ANTHROPIC_API_KEY).\n\n${DEPLOY_USAGE}`);
|
|
286
|
+
if (process.stdin.isTTY)
|
|
287
|
+
console.error(`Paste ${name}'s value for ${env}, then press Enter and Ctrl-D:`);
|
|
288
|
+
const value = readStdin().replace(/\r?\n$/, "");
|
|
289
|
+
if (!value)
|
|
290
|
+
throw new Usage("No value on stdin.");
|
|
291
|
+
const saved = await ctx.api("PUT", `/v1/hosting/secrets/${name}`, { environment: env, value });
|
|
292
|
+
const next = `${RUN} deploy${env === "production" ? " --env production" : ""}`;
|
|
293
|
+
if (ctx.json)
|
|
294
|
+
console.log(JSON.stringify({ ok: true, secret: { name: saved.name, last4: saved.last4, environment: env }, next }));
|
|
295
|
+
ctx.say(`Saved ${saved.name} (••••${saved.last4}) for ${env}. It takes effect on the next deploy.`);
|
|
296
|
+
ctx.say(`Next: ${next}`);
|
|
297
|
+
return 0;
|
|
298
|
+
}
|
|
299
|
+
/** Every non-empty value in a .env file as a secret, one each; the names only are printed. */
|
|
300
|
+
async function secretsFromEnv(ctx, file, env) {
|
|
301
|
+
if (!existsSync(file))
|
|
302
|
+
throw new Usage(`${file} is missing.`);
|
|
303
|
+
const entries = [...parseEnv(readFileSync(file, "utf8"))].filter(([, value]) => value !== "");
|
|
304
|
+
const saved = [];
|
|
305
|
+
const skipped = [];
|
|
306
|
+
const refused = [];
|
|
307
|
+
for (const [name, value] of entries) {
|
|
308
|
+
if (CLI_NAMES.has(name))
|
|
309
|
+
continue;
|
|
310
|
+
if (HOST_NAMES.test(name)) {
|
|
311
|
+
skipped.push({ name, why: "set by Flow" });
|
|
312
|
+
ctx.say(` ${name} skipped (set by Flow)`);
|
|
313
|
+
continue;
|
|
314
|
+
}
|
|
315
|
+
if (!ENV_SECRET_NAME.test(name)) {
|
|
316
|
+
skipped.push({ name, why: "not upper case letters, digits and _" });
|
|
317
|
+
ctx.say(` ${name} skipped: a secret's name is upper case letters, digits and _`);
|
|
318
|
+
continue;
|
|
319
|
+
}
|
|
320
|
+
try {
|
|
321
|
+
await ctx.api("PUT", `/v1/hosting/secrets/${name}`, { environment: env, value });
|
|
322
|
+
}
|
|
323
|
+
catch (error) {
|
|
324
|
+
if (!(error instanceof Refused))
|
|
325
|
+
throw error;
|
|
326
|
+
refused.push({ name, message: error.message });
|
|
327
|
+
ctx.say(` ${name} refused: ${error.message}`);
|
|
328
|
+
continue;
|
|
329
|
+
}
|
|
330
|
+
saved.push(name);
|
|
331
|
+
ctx.say(` ${name} set`);
|
|
332
|
+
}
|
|
333
|
+
const next = `${RUN} deploy${env === "production" ? " --env production" : ""}`;
|
|
334
|
+
if (ctx.json)
|
|
335
|
+
console.log(JSON.stringify({ ok: refused.length === 0, environment: env, saved, skipped, refused, next }));
|
|
336
|
+
ctx.say(saved.length ? `Saved ${saved.length} secret${saved.length === 1 ? "" : "s"} for ${env}. They take effect on the next deploy.` : `No secrets saved from ${file}.`);
|
|
337
|
+
if (saved.length)
|
|
338
|
+
ctx.say(`Next: ${next}`);
|
|
339
|
+
return refused.length ? 1 : 0;
|
|
340
|
+
}
|
|
341
|
+
/** Follows a deploy until it is live, refused or failed, printing each change. */
|
|
342
|
+
async function follow(ctx, id) {
|
|
343
|
+
const until = Date.now() + GIVE_UP_MS;
|
|
344
|
+
let last = "";
|
|
345
|
+
for (;;) {
|
|
346
|
+
const { deploy } = await ctx.api("GET", `/v1/hosting/deploys/${id}`);
|
|
347
|
+
if (deploy.status !== last) {
|
|
348
|
+
last = deploy.status;
|
|
349
|
+
if (!["live", "refused", "failed", "replaced"].includes(deploy.status))
|
|
350
|
+
ctx.say(` ${deploy.status}…`);
|
|
351
|
+
}
|
|
352
|
+
if (["live", "refused", "failed", "replaced"].includes(deploy.status)) {
|
|
353
|
+
for (const f of deploy.findings ?? [])
|
|
354
|
+
ctx.say(` ${f.level === "error" ? "✗" : "!"} ${f.code}: ${f.message}`);
|
|
355
|
+
const live = deploy.status === "live" || deploy.status === "replaced";
|
|
356
|
+
const url = live ? (deploy.url ?? (await address(ctx, deploy.environment))) : null;
|
|
357
|
+
const next = !live
|
|
358
|
+
? deploy.status === "refused"
|
|
359
|
+
? `fix the errors above, then ${RUN} deploy${deploy.environment === "production" ? " --env production" : ""}`
|
|
360
|
+
: `see the deploy's page in the dashboard, then ${RUN} deploy again`
|
|
361
|
+
: deploy.environment === "staging"
|
|
362
|
+
? `try it in your agent's Playground (Staging) in the dashboard, then ${RUN} deploy promote`
|
|
363
|
+
: "open your agent's Playground (Production) in the dashboard";
|
|
364
|
+
if (ctx.json) {
|
|
365
|
+
const { id: deployId, environment, status, version, findings, error } = deploy;
|
|
366
|
+
console.log(JSON.stringify({ ok: live, deploy: { id: deployId, environment, status, version, findings: findings ?? [], error }, url, next }));
|
|
367
|
+
}
|
|
368
|
+
if (live) {
|
|
369
|
+
ctx.say(`Live in ${deploy.environment}.`);
|
|
370
|
+
ctx.say(`URL: ${url ?? NO_URL}`);
|
|
371
|
+
ctx.say(`Next: ${next}`);
|
|
372
|
+
return 0;
|
|
373
|
+
}
|
|
374
|
+
if (!ctx.json)
|
|
375
|
+
console.error(deploy.status === "refused" ? "Refused: fix the errors above and deploy again." : `Failed: ${deploy.error ?? "the build or the service did not start"}.`);
|
|
376
|
+
else
|
|
377
|
+
ctx.say(deploy.status === "refused" ? "Refused." : `Failed: ${deploy.error ?? "the build or the service did not start"}.`);
|
|
378
|
+
return 1;
|
|
379
|
+
}
|
|
380
|
+
if (Date.now() > until) {
|
|
381
|
+
ctx.say(`Still ${deploy.status} after 20 minutes. Its page in the dashboard shows where it is.`);
|
|
382
|
+
return 1;
|
|
383
|
+
}
|
|
384
|
+
await new Promise((r) => setTimeout(r, ctx.pollMs));
|
|
385
|
+
}
|
|
386
|
+
}
|
|
387
|
+
const NO_URL = "not available on this Flow yet (the dashboard's Playground opens it)";
|
|
388
|
+
/** The service's address, when the cloud tells it (GET /v1/hosting/address); null when it does not. */
|
|
389
|
+
async function address(ctx, environment) {
|
|
390
|
+
try {
|
|
391
|
+
const { url } = await ctx.api("GET", `/v1/hosting/address?environment=${encodeURIComponent(environment)}`);
|
|
392
|
+
return typeof url === "string" && url ? url : null;
|
|
393
|
+
}
|
|
394
|
+
catch {
|
|
395
|
+
return null;
|
|
396
|
+
}
|
|
397
|
+
}
|
|
398
|
+
/** Where `flow deploy` keeps the last deploy it made per environment, for `deploy status`: in .flow, never packed. */
|
|
399
|
+
const STATE = join(".flow", "deploys.json");
|
|
400
|
+
function remember(dir, environment, id) {
|
|
401
|
+
try {
|
|
402
|
+
const file = join(dir, STATE);
|
|
403
|
+
const known = existsSync(file) ? JSON.parse(readFileSync(file, "utf8")) : {};
|
|
404
|
+
mkdirSync(join(dir, ".flow"), { recursive: true });
|
|
405
|
+
writeFileSync(file, `${JSON.stringify({ ...known, [environment]: { id } }, null, 2)}\n`);
|
|
406
|
+
}
|
|
407
|
+
catch {
|
|
408
|
+
// only a convenience for status
|
|
409
|
+
}
|
|
410
|
+
}
|
|
411
|
+
function remembered(dir, environment) {
|
|
412
|
+
try {
|
|
413
|
+
const id = JSON.parse(readFileSync(join(dir, STATE), "utf8"))[environment]?.id;
|
|
414
|
+
return typeof id === "string" && /^hdp_[0-9A-Za-z]+$/.test(id) ? id : undefined;
|
|
415
|
+
}
|
|
416
|
+
catch {
|
|
417
|
+
return undefined;
|
|
418
|
+
}
|
|
419
|
+
}
|
|
420
|
+
/** `flow deploy status`: per environment, what is live, its version, when, and its URL. */
|
|
421
|
+
async function status(ctx, environments) {
|
|
422
|
+
const found = [];
|
|
423
|
+
for (const environment of environments) {
|
|
424
|
+
let deploys = [];
|
|
425
|
+
let from = null;
|
|
426
|
+
try {
|
|
427
|
+
deploys = (await ctx.api("GET", `/v1/hosting/deploys?environment=${environment}`)).deploys ?? [];
|
|
428
|
+
from = "list";
|
|
429
|
+
}
|
|
430
|
+
catch (error) {
|
|
431
|
+
if (!(error instanceof Refused) || !error.missing)
|
|
432
|
+
throw error;
|
|
433
|
+
// This Flow lists deploys by CI key only to a dashboard member: the last one this folder made.
|
|
434
|
+
const id = remembered(ctx.dir, environment);
|
|
435
|
+
if (id) {
|
|
436
|
+
deploys = [(await ctx.api("GET", `/v1/hosting/deploys/${id}`)).deploy];
|
|
437
|
+
from = "this_folder";
|
|
438
|
+
}
|
|
439
|
+
}
|
|
440
|
+
const newestFirst = [...deploys].sort((a, b) => String(b.created_at ?? "").localeCompare(String(a.created_at ?? "")));
|
|
441
|
+
const live = newestFirst.find((d) => d.status === "live") ?? null;
|
|
442
|
+
const top = newestFirst[0] ?? null;
|
|
443
|
+
const serving = live ?? top;
|
|
444
|
+
const pending = top && top !== serving && !["live", "refused", "failed", "replaced"].includes(top.status) ? { id: top.id, status: top.status, version: top.version } : null;
|
|
445
|
+
const url = live ? (live.url ?? (await address(ctx, environment))) : null;
|
|
446
|
+
found.push({ environment, deploy: serving, pending, url, from });
|
|
447
|
+
}
|
|
448
|
+
const anyLive = found.some((f) => f.deploy?.status === "live");
|
|
449
|
+
const next = anyLive
|
|
450
|
+
? `${RUN} logs${found.length === 1 && found[0].environment === "production" ? " --env production" : ""}`
|
|
451
|
+
: `${RUN} deploy`;
|
|
452
|
+
if (ctx.json) {
|
|
453
|
+
console.log(JSON.stringify({ ok: true, environments: Object.fromEntries(found.map((f) => [f.environment, f.deploy ? { deploy: pick(f.deploy), pending: f.pending, url: f.url, from: f.from } : null])), next }));
|
|
454
|
+
return 0;
|
|
455
|
+
}
|
|
456
|
+
for (const f of found) {
|
|
457
|
+
if (!f.deploy) {
|
|
458
|
+
ctx.say(`${f.environment}: no deploy known${f.from === null ? " (this Flow lists deploys to a dashboard member only, and this folder has made none: the dashboard shows them)" : ""}.`);
|
|
459
|
+
continue;
|
|
460
|
+
}
|
|
461
|
+
const d = f.deploy;
|
|
462
|
+
const when = d.updated_at ?? d.created_at;
|
|
463
|
+
ctx.say(`${f.environment}: ${d.status}${d.version ? `, version ${d.version}` : ""}, deploy ${d.id}${when ? `, ${d.status === "live" ? "deployed" : "updated"} ${when}` : ""}${f.from === "this_folder" ? " (the last deploy from this folder)" : ""}`);
|
|
464
|
+
if (d.status === "live")
|
|
465
|
+
ctx.say(` URL: ${f.url ?? NO_URL}`);
|
|
466
|
+
if (f.pending)
|
|
467
|
+
ctx.say(` then ${f.pending.id}${f.pending.version ? ` (${f.pending.version})` : ""}: ${f.pending.status}…`);
|
|
468
|
+
}
|
|
469
|
+
ctx.say(`Next: ${next}`);
|
|
470
|
+
return 0;
|
|
471
|
+
}
|
|
472
|
+
const pick = (d) => ({
|
|
473
|
+
id: d.id,
|
|
474
|
+
environment: d.environment,
|
|
475
|
+
status: d.status,
|
|
476
|
+
version: d.version,
|
|
477
|
+
source: d.source ?? null,
|
|
478
|
+
created_at: d.created_at ?? null,
|
|
479
|
+
updated_at: d.updated_at ?? null,
|
|
480
|
+
});
|
|
481
|
+
/** --since as seconds: 30s, 15m, 1h, 2d. */
|
|
482
|
+
export function sinceSeconds(since) {
|
|
483
|
+
const m = since.match(/^(\d+)(s|m|h|d)$/);
|
|
484
|
+
if (!m)
|
|
485
|
+
return undefined;
|
|
486
|
+
const n = Number(m[1]);
|
|
487
|
+
return n * { s: 1, m: 60, h: 3600, d: 86400 }[m[2]];
|
|
488
|
+
}
|
|
489
|
+
/**
|
|
490
|
+
* `flow logs`: the hosted agent's log lines, from GET /v1/hosting/logs?environment=&since=&after=
|
|
491
|
+
* → { lines: [{ time, severity, text }], next } (next: the cursor for the lines after these).
|
|
492
|
+
* A Flow without that route says so and exits 1.
|
|
493
|
+
*/
|
|
494
|
+
async function logs(ctx, environment, since, followLines) {
|
|
495
|
+
const seconds = sinceSeconds(since);
|
|
496
|
+
if (seconds === undefined || seconds <= 0 || seconds > 30 * 86400)
|
|
497
|
+
throw new Usage(`--since is a duration such as 30s, 15m, 1h or 2d (at most 30d).\n\n${LOGS_USAGE}`);
|
|
498
|
+
const from = new Date(ctx.now() - seconds * 1000).toISOString();
|
|
499
|
+
let after;
|
|
500
|
+
let polls = 0;
|
|
501
|
+
for (;;) {
|
|
502
|
+
const query = new URLSearchParams({ environment, ...(after ? { after } : { since: from }) });
|
|
503
|
+
let page;
|
|
504
|
+
try {
|
|
505
|
+
page = await ctx.api("GET", `/v1/hosting/logs?${query}`);
|
|
506
|
+
}
|
|
507
|
+
catch (error) {
|
|
508
|
+
if (error instanceof Refused && error.missing) {
|
|
509
|
+
throw new NotAvailable(`Logs are not available on this Flow yet: its cloud has no logs route for a CI key. The dashboard shows the agent's deploys; ${RUN} deploy status shows what is live.`);
|
|
510
|
+
}
|
|
511
|
+
throw error;
|
|
512
|
+
}
|
|
513
|
+
for (const line of page.lines ?? []) {
|
|
514
|
+
if (ctx.json)
|
|
515
|
+
console.log(JSON.stringify({ time: line.time, severity: line.severity ?? null, text: line.text }));
|
|
516
|
+
else
|
|
517
|
+
console.log(`${line.time} ${(line.severity ?? "").padEnd(7)} ${line.text}`);
|
|
518
|
+
}
|
|
519
|
+
if (page.next)
|
|
520
|
+
after = page.next;
|
|
521
|
+
polls++;
|
|
522
|
+
if (!followLines || (ctx.maxPolls !== undefined && polls >= ctx.maxPolls))
|
|
523
|
+
return 0;
|
|
524
|
+
await new Promise((r) => setTimeout(r, ctx.pollMs));
|
|
525
|
+
}
|
|
526
|
+
}
|
|
527
|
+
function gitVersion(dir) {
|
|
528
|
+
try {
|
|
529
|
+
return execFileSync("git", ["-C", dir, "rev-parse", "--short", "HEAD"], { stdio: ["ignore", "pipe", "ignore"] }).toString().trim() || undefined;
|
|
530
|
+
}
|
|
531
|
+
catch {
|
|
532
|
+
return undefined;
|
|
533
|
+
}
|
|
534
|
+
}
|
|
535
|
+
const mb = (n) => `${(n / 1024 / 1024).toFixed(1)} MB`;
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { runVerdict } from "./report.js";
|
|
1
|
+
import { runVerdict, unsupported } from "./report.js";
|
|
2
2
|
const usd = (n) => `$${n.toFixed(4)}`;
|
|
3
3
|
/** Prints each run once. A run already shown as it finished is not printed again with the summary. */
|
|
4
4
|
export function runPrinter(log = console.log) {
|
|
@@ -8,6 +8,11 @@ export function runPrinter(log = console.log) {
|
|
|
8
8
|
return;
|
|
9
9
|
printed.add(r.runId);
|
|
10
10
|
const verdict = runVerdict(r);
|
|
11
|
+
// A scenario the cloud could not play: its file and what is wrong in it, not a run of the agent.
|
|
12
|
+
if (r.invalid) {
|
|
13
|
+
log(`INVALID ${r.scenario} #${r.run} (${r.channel}) ${r.runId}: the cloud could not play this scenario: ${r.invalid.path}: ${r.invalid.reason}${unsupported(r.invalid.reason)}`);
|
|
14
|
+
return;
|
|
15
|
+
}
|
|
11
16
|
const gaveUp = verdict === "passed" && r.ended === "gave_up" ? " (passed, the person gave up)" : "";
|
|
12
17
|
const retried = r.retried ? "; tried once more" : "";
|
|
13
18
|
// An errored run: the cloud could not play or judge it, so its checks say nothing about the agent.
|
|
@@ -246,6 +246,12 @@ export interface RunSummary {
|
|
|
246
246
|
/** The cloud's code when it could not play or judge the run (judge_failed, spend_cap, simulator_failed, ...).
|
|
247
247
|
* The run then says nothing about the agent, whatever `passed` and `ended` say. */
|
|
248
248
|
error?: string;
|
|
249
|
+
/** With error scenario_invalid: the scenario file the cloud could not play, and what is wrong in it
|
|
250
|
+
* (`unsupported: ...` when the cloud does not know a key). */
|
|
251
|
+
invalid?: {
|
|
252
|
+
path: string;
|
|
253
|
+
reason: string;
|
|
254
|
+
};
|
|
249
255
|
}
|
|
250
256
|
export interface SessionSummary {
|
|
251
257
|
sessionId: string;
|
|
@@ -26,6 +26,27 @@ export interface ReportRun {
|
|
|
26
26
|
text: string;
|
|
27
27
|
reason: string;
|
|
28
28
|
}>;
|
|
29
|
+
/** The scenario file the cloud could not play and what is wrong in it (error scenario_invalid); else null. */
|
|
30
|
+
invalid: {
|
|
31
|
+
path: string;
|
|
32
|
+
reason: string;
|
|
33
|
+
} | null;
|
|
34
|
+
}
|
|
35
|
+
/** A scenario the cloud could not play, once per scenario. */
|
|
36
|
+
export interface InvalidScenario {
|
|
37
|
+
scenario: string;
|
|
38
|
+
path: string;
|
|
39
|
+
reason: string;
|
|
40
|
+
}
|
|
41
|
+
/** The cloud ended the whole session scenario_invalid: what it said after the code (null when nothing),
|
|
42
|
+
* whether that looks like a cloud older than this client, and the scenario files this session sent. */
|
|
43
|
+
export interface SessionInvalid {
|
|
44
|
+
reason: string | null;
|
|
45
|
+
olderCloud: boolean;
|
|
46
|
+
sent: Array<{
|
|
47
|
+
scenario: string;
|
|
48
|
+
path: string;
|
|
49
|
+
}>;
|
|
29
50
|
}
|
|
30
51
|
/** `flow test --json`: one document on stdout, whatever the exit. */
|
|
31
52
|
export interface JsonReport {
|
|
@@ -50,6 +71,10 @@ export interface JsonReport {
|
|
|
50
71
|
test: number;
|
|
51
72
|
};
|
|
52
73
|
runs: ReportRun[];
|
|
74
|
+
/** Each scenario the cloud could not play; the other scenarios played. */
|
|
75
|
+
invalid: InvalidScenario[];
|
|
76
|
+
/** Set when the cloud ended the whole session scenario_invalid; else null. */
|
|
77
|
+
sessionInvalid: SessionInvalid | null;
|
|
53
78
|
}
|
|
54
79
|
/** A run's verdict: errored when the cloud names an error for it or ended it run_failed, since the cloud
|
|
55
80
|
* could not play or judge it and that says nothing about the agent (a judge_failed run can end goal_met). */
|
|
@@ -65,6 +90,27 @@ export declare function finishedOutcome(runs: Array<{
|
|
|
65
90
|
}>): Outcome;
|
|
66
91
|
/** What stopped a session, or the open, as an outcome. `opened`: the session had opened. */
|
|
67
92
|
export declare function stoppedOutcome(error: unknown, opened: boolean): Outcome;
|
|
93
|
+
/** The scenarios the cloud could not play, once each, from the runs it reported (a run per channel and --runs). */
|
|
94
|
+
export declare function invalidScenarios(runs: Array<{
|
|
95
|
+
scenario: string;
|
|
96
|
+
invalid?: {
|
|
97
|
+
path: string;
|
|
98
|
+
reason: string;
|
|
99
|
+
} | null;
|
|
100
|
+
}>): InvalidScenario[];
|
|
101
|
+
/** The lines that name each scenario the cloud could not play. */
|
|
102
|
+
export declare function invalidLines(invalid: InvalidScenario[]): string[];
|
|
103
|
+
/** A reason naming a key the cloud does not know: the client may be newer than the cloud. */
|
|
104
|
+
export declare function unsupported(reason: string): string;
|
|
105
|
+
/** A session the cloud ended scenario_invalid; null for any other stop. A tools/ or checks/ file with
|
|
106
|
+
* its reason is the suite's own problem; anything else (no reason, or a scenario, which a current
|
|
107
|
+
* cloud reports as that scenario's run) suggests a cloud older than this client. */
|
|
108
|
+
export declare function sessionInvalid(error: unknown, sent: Array<{
|
|
109
|
+
scenario: string;
|
|
110
|
+
path: string;
|
|
111
|
+
}>): SessionInvalid | null;
|
|
112
|
+
/** The lines that say why the session ended scenario_invalid. */
|
|
113
|
+
export declare function sessionInvalidLines(info: SessionInvalid): string[];
|
|
68
114
|
/** The error as the JSON report says it: the cloud's status, code and Retry-After when it answered. */
|
|
69
115
|
export declare function reportError(error: unknown, message: string): ReportError;
|
|
70
116
|
/** The connection's error code, from the error or what caused it. */
|
|
@@ -38,6 +38,48 @@ export function stoppedOutcome(error, opened) {
|
|
|
38
38
|
// Something on this machine: the agent would not build before the open, or broke the session after it.
|
|
39
39
|
return opened ? "failed" : "invalid";
|
|
40
40
|
}
|
|
41
|
+
/** The scenarios the cloud could not play, once each, from the runs it reported (a run per channel and --runs). */
|
|
42
|
+
export function invalidScenarios(runs) {
|
|
43
|
+
const out = new Map();
|
|
44
|
+
for (const r of runs)
|
|
45
|
+
if (r.invalid)
|
|
46
|
+
out.set(`${r.scenario}\n${r.invalid.path}`, { scenario: r.scenario, path: r.invalid.path, reason: r.invalid.reason });
|
|
47
|
+
return [...out.values()];
|
|
48
|
+
}
|
|
49
|
+
/** The lines that name each scenario the cloud could not play. */
|
|
50
|
+
export function invalidLines(invalid) {
|
|
51
|
+
if (!invalid.length)
|
|
52
|
+
return [];
|
|
53
|
+
const lines = [`${invalid.length} ${invalid.length === 1 ? "scenario" : "scenarios"} could not be played (fix ${invalid.length === 1 ? "it" : "them"} and run again):`];
|
|
54
|
+
for (const i of invalid)
|
|
55
|
+
lines.push(` ${i.scenario}: ${i.path}: ${i.reason}${unsupported(i.reason)}`);
|
|
56
|
+
return lines;
|
|
57
|
+
}
|
|
58
|
+
/** A reason naming a key the cloud does not know: the client may be newer than the cloud. */
|
|
59
|
+
export function unsupported(reason) {
|
|
60
|
+
return reason.startsWith("unsupported: ") ? " (the test cloud does not know this key; it may be older than this client)" : "";
|
|
61
|
+
}
|
|
62
|
+
/** A session the cloud ended scenario_invalid; null for any other stop. A tools/ or checks/ file with
|
|
63
|
+
* its reason is the suite's own problem; anything else (no reason, or a scenario, which a current
|
|
64
|
+
* cloud reports as that scenario's run) suggests a cloud older than this client. */
|
|
65
|
+
export function sessionInvalid(error, sent) {
|
|
66
|
+
if (!(error instanceof CloudEnded))
|
|
67
|
+
return null;
|
|
68
|
+
const match = /^scenario_invalid(?::\s*(.*))?$/s.exec(error.error.trim());
|
|
69
|
+
if (!match)
|
|
70
|
+
return null;
|
|
71
|
+
const reason = match[1]?.trim() || null;
|
|
72
|
+
const suiteFile = reason !== null && /^(?:\S*\/)?(?:tools|checks)\/[^:]+:\s*\S/.test(reason);
|
|
73
|
+
return { reason, olderCloud: !suiteFile, sent };
|
|
74
|
+
}
|
|
75
|
+
/** The lines that say why the session ended scenario_invalid. */
|
|
76
|
+
export function sessionInvalidLines(info) {
|
|
77
|
+
const lines = [`The test cloud could not play this suite${info.reason ? `: ${info.reason}` : " and gave no reason."}`];
|
|
78
|
+
if (info.olderCloud)
|
|
79
|
+
lines.push("the test cloud may be older than this client");
|
|
80
|
+
lines.push(`Scenarios sent: ${info.sent.map((s) => `${s.scenario} (${s.path})`).join(", ") || "none"}`);
|
|
81
|
+
return lines;
|
|
82
|
+
}
|
|
41
83
|
/** The error as the JSON report says it: the cloud's status, code and Retry-After when it answered. */
|
|
42
84
|
export function reportError(error, message) {
|
|
43
85
|
const refusal = postErrorOf(error);
|
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
import { type SessionInvalid } from "./report.ts";
|
|
1
2
|
/** Where every session's results go, in the agent's folder; `flow new` git-ignores .flow/. */
|
|
2
3
|
export declare const LAST_SESSION: string;
|
|
3
4
|
/** The last session's file in dir: LAST_SESSION, else, before a run has written it, the one from before the rename (.chinwag/). */
|
|
@@ -29,7 +30,15 @@ export interface LastSession {
|
|
|
29
30
|
text: string;
|
|
30
31
|
reason?: string;
|
|
31
32
|
}>;
|
|
33
|
+
/** The scenario file the cloud could not play and what is wrong in it (error scenario_invalid). */
|
|
34
|
+
invalid?: {
|
|
35
|
+
path: string;
|
|
36
|
+
reason: string;
|
|
37
|
+
};
|
|
32
38
|
}>;
|
|
39
|
+
/** The cloud ended the whole session scenario_invalid: its reason, whether the cloud looks older
|
|
40
|
+
* than the client, and the scenario files sent. */
|
|
41
|
+
sessionInvalid?: SessionInvalid;
|
|
33
42
|
}
|
|
34
43
|
export declare const SHOW_LINE = " flow show <run-id|last> [--json]\n Print one run's record from the test cloud: its checks with the\n judge's reasons, the transcript and the tool calls (flow show --help)";
|
|
35
44
|
/** `flow show`: returns the exit status. */
|
package/docs/agents.md
CHANGED
|
@@ -190,4 +190,41 @@ version in `package.json` for production.
|
|
|
190
190
|
|
|
191
191
|
## Host it
|
|
192
192
|
|
|
193
|
-
|
|
193
|
+
`npx @flow-engineer/flow deploy` puts the agent on Flow's hosting, so it answers while the laptop
|
|
194
|
+
sleeps. It needs `package.json`, `package-lock.json` and `src/agent.ts`, whose `buildAgent` passes
|
|
195
|
+
`options.host` on to `createAgent`, and the agent's CI key in `FLOW_FRAMEWORK_CLOUD_KEY` (the
|
|
196
|
+
environment, else `.env`; `flow setup` writes it).
|
|
197
|
+
|
|
198
|
+
```sh
|
|
199
|
+
npx @flow-engineer/flow deploy secrets --from-env .env [--env staging|production] [--json]
|
|
200
|
+
npx @flow-engineer/flow deploy [--env staging|production] [--version <v>] [--json]
|
|
201
|
+
npx @flow-engineer/flow deploy promote [--json]
|
|
202
|
+
npx @flow-engineer/flow deploy status [--env staging|production] [--json]
|
|
203
|
+
npx @flow-engineer/flow logs [--env staging|production] [--since 1h] [--follow] [--json]
|
|
204
|
+
npx @flow-engineer/flow deploy secret NAME [--env staging|production] < value.txt
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
- `deploy` packs the folder (without `node_modules`, `.git`, `.env*` and `.flow`, at most 50 MB),
|
|
208
|
+
uploads it, and follows the build until it is live, refused or failed. The default is staging;
|
|
209
|
+
production refuses an agent that ignores `host` or brings its own database, and the build
|
|
210
|
+
refuses an agent on a newer framework than Flow hosts yet. `--json` prints
|
|
211
|
+
`{ ok, deploy: { id, environment, status, version, findings, error }, url, next }` on stdout
|
|
212
|
+
(`url` is null when the cloud does not say it yet; the human output then says "URL: not available
|
|
213
|
+
on this Flow yet"); progress goes to stderr. The deploy's id is kept in `.flow/deploys.json`.
|
|
214
|
+
- `promote` moves staging's live version to production, with no new build.
|
|
215
|
+
- `status` shows, per environment (both unless `--env`), the deploy serving it (`status`, `version`,
|
|
216
|
+
`id`, `updated_at`), its URL and a newer one still building. `--json` prints
|
|
217
|
+
`{ ok, environments: { staging: { deploy, pending, url, from } | null, production: ... }, next }`;
|
|
218
|
+
`from` is `list` (the cloud's list) or `this_folder` (the last deploy from `.flow/deploys.json`, on
|
|
219
|
+
a Flow whose cloud does not list deploys to a CI key yet).
|
|
220
|
+
- `logs` prints the hosted agent's log lines from `--since` (30s, 15m, 1h, 2d; default 1h), and with
|
|
221
|
+
`--follow` keeps printing new ones; `--json` is one `{ time, severity, text }` per line. On a Flow
|
|
222
|
+
whose cloud has no logs route yet it says so: `error.code` `not_available`, exit 1.
|
|
223
|
+
- `secrets --from-env` sends each non-empty value in the file as a secret, never the CI key, the
|
|
224
|
+
cloud's address or a name Flow sets itself (`FLOW_FRAMEWORK_TRACES_*`, `FLOW_STORE_*`, `PORT`, ...);
|
|
225
|
+
`--json` prints `{ ok, environment, saved, skipped, refused, next }`. `secret NAME` reads one value
|
|
226
|
+
from stdin. Names only are printed, never values; a secret takes effect on the next deploy.
|
|
227
|
+
- Exit codes: 0 live or saved, 1 refused, failed or not reachable, 2 a usage problem (no CI key, a
|
|
228
|
+
missing file, a bad name). On an error, `--json` prints `{ ok: false, error: { code, message }, next }`.
|
|
229
|
+
|
|
230
|
+
Run each answer's `next`. Promote, or deploy to production, only when the owner says so.
|