@flow-engineer/flow 0.3.2 → 0.3.3

package/README.md

@@ -408,13 +408,18 @@ Each run's line carries its id (`FAIL  latte-correction #1 (whatsapp) trn_…: 6
   "runs": [
     { "runId": "trn_…", "scenario": "latte-correction", "channel": "whatsapp", "run": 1, "verdict": "failed",
       "error": null, "ended": "goal_met", "turns": 6, "watch": "https://…",
-      "failedChecks": [{ "kind": "judge", "text": "The agent asks Sam to confirm before placing the order.", "reason": "…" }] }
-  ]
+      "failedChecks": [{ "kind": "judge", "text": "The agent asks Sam to confirm before placing the order.", "reason": "…" }],
+      "invalid": null }
+  ],
+  "invalid": [],
+  "sessionInvalid": null
 }
 ```
 
 `outcome` is `passed` (exit 0), `failed` or `invalid` (1), `errored` or `transport` (2), `refused` (3) or `interrupted` (130). A run's `verdict` is `errored` whenever the cloud names an `error` for it (such as `judge_failed`, `simulator_failed` or `spend_cap`), whatever it `ended` as: the cloud could not play or judge it, so it says nothing about the agent. Its line reads `ERROR` instead of `PASS` or `FAIL`, and the run's `error` is that code (else `null`). `error` is `{ "message", "status", "type", "retryAfterSeconds" }` when the session did not finish (`type` is the cloud's code, such as `spend_cap`; the others are `null` when it did not say).
 
+A scenario the cloud cannot play (a key it does not know, a value it cannot read) does not stop the others: its runs are `errored` with `error` `scenario_invalid` and `invalid` `{ "path", "reason" }` (the reason starts `unsupported:` when the cloud does not know a key), its line reads `INVALID` with the file and what is wrong, and the top-level `invalid` lists each such scenario once as `{ "scenario", "path", "reason" }`. When the cloud ends the whole session `scenario_invalid` (a tools or checks file it cannot read, or an older cloud), `sessionInvalid` is `{ "reason", "olderCloud", "sent" }`: what the cloud said after the code (else `null`), whether that suggests a cloud older than this client, and the scenarios sent as `{ "scenario", "path" }`; the same lines are printed.
+
 `--only latte-correction,order-fails` runs only those scenarios (file names without `.yaml`); every tool file still goes. Once the session has opened, every exit (0, 1, 2, 3 or 130) writes `.flow/last-session.json` in the agent's folder and prints its path; `flow new` git-ignores `.flow/`. It holds what each run ended as and the judge's reasons, never the messages (a reason can quote one, which is why the file stays on your machine; the suite is synthetic by `--synthetic`):
 
 ```json
@@ -429,6 +434,8 @@ Each run's line carries its id (`FAIL  latte-correction #1 (whatsapp) trn_…: 6
 }
 ```
 
+A run of a scenario the cloud could not play also has `invalid` (`{ "path", "reason" }`), and a session the cloud ended `scenario_invalid` has `sessionInvalid`, as in `--json`.
+
 `ended` is `finished`, `stopped` (the session broke off or the cloud stopped it) or `interrupted`. `agent` is the cloud's id for the agent (a hosted cloud names it), which `flow show` reads. `only` is the scenarios the session was limited to, `null` for the whole suite. `verdict` is `passed`, `failed`, `errored` (the cloud could not play or judge the run) or `not_played` (the session ended first; its `runId` is `null`).
 
 `flow show <run-id>` prints one run's record from the cloud, read with `FLOW_FRAMEWORK_CLOUD_KEY`: the verdict, every check with the judge's reason, then the conversation in time order with each tool call the simulator played (its input, output and what it changed). `flow show last` picks the first run in `.flow/last-session.json` that did not pass (its first run when all passed) and names the others. `--json` prints the record exactly as the cloud answered it. It exits 0 when it printed; 1 for a wrong flag, no key or last session, no such run (404) or one past the 30 days runs are kept (410); 2 when the cloud did not answer or answered 5xx; 3 when it refused the key (401, 403) or is busy (429).

package/coding-agents/test-skill.md

@@ -183,13 +183,18 @@ connects out to the cloud, which plays the person and the tools and judges.
    `runId`, `scenario`, `channel`, `run`, `verdict`, `error` (the cloud's code when it could not play
    or judge the run, such as `judge_failed`; the run is then `errored` and its line reads `ERROR`,
    whatever it ended as), `ended`, `turns`, `watch` and `failedChecks` (`kind`, `text` and the judge's
-   `reason`). Each session that opened also leaves
+   `reason`), and `invalid` (`{ path, reason }` for a scenario the cloud could not play, its error
+   `scenario_invalid` and its line `INVALID`; else `null`); top-level `invalid` lists those scenarios
+   once each (`scenario`, `path`, `reason`: fix the file; `unsupported: ...` means the cloud does not
+   know that key), and `sessionInvalid` (`reason`, `olderCloud`, `sent`) is set when the cloud ended
+   the whole session `scenario_invalid`. Each session that opened also leaves
    `.flow/last-session.json` in the agent's folder (its path is printed on every exit): `sessionId`,
    `agent`, `dashboard`, `suiteHash`, `endedAt`, `ended` (`finished`, `stopped` or `interrupted`),
    `only` (the scenarios it was limited to, else `null`) and per run `scenario` (file name without
    `.yaml`), `channel`, `run`, `runId`, `verdict` (`passed`, `failed`, `errored`: the cloud could not
    play or judge it, such as a failed tool simulator or person model, which is not a failure of your
-   agent, or `not_played`: the session ended first) and `failedChecks` (with reasons).
+   agent, or `not_played`: the session ended first), `failedChecks` (with reasons) and `invalid`
+   when the cloud could not play the scenario, plus `sessionInvalid` as in `--json`.
    Read these instead of parsing the printed lines.
    Exit status: 0 all passed; 1 a run failed, or nothing was sent (a wrong flag, a test file that does
    not parse, no key); 2 the connection or the cloud failed, or runs errored and none failed; 3 the

package/dist/testing/src/cli.js

@@ -44,7 +44,7 @@ import { CHAT_LINE, chatCli } from "./chat.js";
 import { httpAgent } from "./http-agent.js";
 import { createInterface } from "node:readline/promises";
 import { displayNameOf, folderName, initHere, newStandalone, packageRoot } from "./new.js";
-import { EXIT, finishedOutcome, loadEnv, networkCode, reportError, runVerdict, stoppedOutcome } from "./report.js";
+import { EXIT, finishedOutcome, invalidLines, invalidScenarios, loadEnv, networkCode, reportError, runVerdict, sessionInvalid, sessionInvalidLines, stoppedOutcome } from "./report.js";
 import { LAST_SESSION, lastSessionIn, SHOW_LINE, showCli } from "./show.js";
 import { loadSuite, parseScenario, parseToolFile, suiteHash } from "./suite.js";
 const NEW_LINE = `  flow new <name> [--display-name <name>]
@@ -105,7 +105,7 @@ Exit status:
 Results: once the session has opened, every exit writes .flow/last-session.json in the agent's
 folder: the session id, the cloud's agent id, dashboard link, suite hash and time, and per run its
 scenario, channel, number, run id, verdict (passed, failed, errored or not_played) and failed checks
-with the judge's reasons. Never the messages. The folder is git-ignored and the suite synthetic.
+with the judge's reasons (and, for a scenario the cloud could not play, its file and what is wrong). Never the messages. The folder is git-ignored and the suite synthetic.
 
 Environment (also read from the agent folder's .env, or --dotenv; variables already set win;
 the CHINWAG_ names from before the rename still work):
@@ -251,6 +251,8 @@ let sessionId = "";
 let dashboard = "";
 let agentId = "";
 const seen = [];
+/** Set when the cloud ended the whole session scenario_invalid. */
+let endedInvalid = null;
 if (command !== undefined && (command !== "test" || rest.length))
     invalid(`flow: unknown command "${positionals.join(" ")}".\n\n${USAGE}`);
 const envProblem = loadEnv(values.dotenv);
@@ -381,6 +383,10 @@ catch (error) {
         end("transport", { received: error.runs, error: reportError(error, reason) });
     }
     console.error(reason);
+    endedInvalid = sessionInvalid(error, files.filter((f) => f.path.startsWith("scenarios/")).map((f) => ({ scenario: f.path.slice("scenarios/".length, -".yaml".length), path: f.path })));
+    if (endedInvalid)
+        for (const line of sessionInvalidLines(endedInvalid))
+            console.error(line);
     // The cloud stopped a session that had opened: what finished is kept, the rest is not_played.
     saveSession("stopped", seen, dashboard, console.error);
     end(stoppedOutcome(error, sessionId !== ""), { received: seen, error: reportError(error, reason) });
@@ -402,6 +408,8 @@ for (const channel of [...new Set(summary.runs.map((r) => r.channel))]) {
     const runs = summary.runs.filter((r) => r.channel === channel);
     say(`  ${channel}: ${runs.filter((r) => runVerdict(r) === "passed").length} of ${runs.length} passed`);
 }
+for (const line of invalidLines(invalidScenarios(mergedRuns(summary.runs).map((m) => ({ scenario: m.scenario, invalid: m.summary?.invalid })))))
+    say(line);
 if (summary.lost)
     say(`The cloud lost session ${summary.lost.sessionId} part way; a second session played its unfinished runs. The first: ${summary.lost.dashboard}`);
 say(`Cost: ${usd(total((r) => r.agentUsage.costUsd))} agent (${tokens(total((r) => r.agentUsage.input))} in, ${tokens(total((r) => r.agentUsage.output))} out) + ${usd(total((r) => r.testUsage.costUsd))} test models.\n${summary.dashboard}`);
@@ -432,8 +440,11 @@ function mergedRuns(received) {
         fileOf.set(scenario.id, id);
         for (let run = 1; run <= perScenario; run++) {
             for (const channel of scenario.channels) {
-                const r = byRun.get(key(scenario.id, channel, run));
+                // A scenario the cloud could not read is named by its file, not its id.
+                const r = byRun.get(key(scenario.id, channel, run)) ?? byRun.get(key(id, channel, run));
                 byRun.delete(key(scenario.id, channel, run));
+                if (r)
+                    byRun.delete(key(r.scenario, channel, run));
                 out.push(r ? merged(id, r) : { scenario: id, channel, run, verdict: "not_played" });
             }
         }
@@ -466,7 +477,9 @@ function saveSession(ended, received, link, print) {
             runId: m.summary?.runId ?? null,
             verdict: m.verdict,
             failedChecks: failedChecks(m.summary),
+            ...(m.summary?.invalid ? { invalid: m.summary.invalid } : {}),
         })),
+        ...(endedInvalid ? { sessionInvalid: endedInvalid } : {}),
     };
     const path = join(dir, LAST_SESSION);
     try {
@@ -496,6 +509,7 @@ function end(outcome, options = {}) {
             turns: m.summary?.turns ?? null,
             watch: m.summary?.watch || null,
             failedChecks: failedChecks(m.summary),
+            invalid: m.summary?.invalid ?? null,
         })) : [];
         const count = (verdict) => runs.filter((r) => r.verdict === verdict).length;
         const cost = (pick) => received.reduce((n, r) => n + pick(r), 0);
@@ -511,6 +525,8 @@ function end(outcome, options = {}) {
             counts: { runs: runs.length, passed: count("passed"), failed: count("failed"), errored: count("errored"), notPlayed: count("not_played") },
             costUsd: { agent: cost((r) => r.agentUsage.costUsd), test: cost((r) => r.testUsage.costUsd) },
             runs,
+            invalid: invalidScenarios(runs),
+            sessionInvalid: endedInvalid,
         };
         writeAll(`${JSON.stringify(report)}\n`);
     }

package/dist/testing/src/client.d.ts

@@ -49,6 +49,8 @@ export interface ConnectOptions {
  * code (`capacity: ...` gives capacity), when it starts with one. */
 export declare class CloudEnded extends Error {
     readonly code?: string;
+    /** The cloud's error as it sent it. */
+    readonly error: string;
     constructor(error: string);
 }
 /** The tunnel, the stream, or the connection failed after the session opened. `runs` are the ones already received. The CLI prints them and exits 2. */

package/dist/testing/src/client.js

@@ -15,9 +15,12 @@ import { businessRuns } from "./business.js";
  * code (`capacity: ...` gives capacity), when it starts with one. */
 export class CloudEnded extends Error {
     code;
+    /** The cloud's error as it sent it. */
+    error;
     constructor(error) {
         super(`the cloud stopped the session: ${error}`);
         this.name = "CloudEnded";
+        this.error = error;
         this.code = /^([a-z_]+):/.exec(error)?.[1];
     }
 }

package/dist/testing/src/print-run.js

@@ -1,4 +1,4 @@
-import { runVerdict } from "./report.js";
+import { runVerdict, unsupported } from "./report.js";
 const usd = (n) => `$${n.toFixed(4)}`;
 /** Prints each run once. A run already shown as it finished is not printed again with the summary. */
 export function runPrinter(log = console.log) {
@@ -8,6 +8,11 @@ export function runPrinter(log = console.log) {
             return;
         printed.add(r.runId);
         const verdict = runVerdict(r);
+        // A scenario the cloud could not play: its file and what is wrong in it, not a run of the agent.
+        if (r.invalid) {
+            log(`INVALID  ${r.scenario} #${r.run} (${r.channel}) ${r.runId}: the cloud could not play this scenario: ${r.invalid.path}: ${r.invalid.reason}${unsupported(r.invalid.reason)}`);
+            return;
+        }
         const gaveUp = verdict === "passed" && r.ended === "gave_up" ? " (passed, the person gave up)" : "";
         const retried = r.retried ? "; tried once more" : "";
         // An errored run: the cloud could not play or judge it, so its checks say nothing about the agent.

package/dist/testing/src/protocol.d.ts

@@ -246,6 +246,12 @@ export interface RunSummary {
     /** The cloud's code when it could not play or judge the run (judge_failed, spend_cap, simulator_failed, ...).
      * The run then says nothing about the agent, whatever `passed` and `ended` say. */
     error?: string;
+    /** With error scenario_invalid: the scenario file the cloud could not play, and what is wrong in it
+     * (`unsupported: ...` when the cloud does not know a key). */
+    invalid?: {
+        path: string;
+        reason: string;
+    };
 }
 export interface SessionSummary {
     sessionId: string;

package/dist/testing/src/report.d.ts

@@ -26,6 +26,27 @@ export interface ReportRun {
         text: string;
         reason: string;
     }>;
+    /** The scenario file the cloud could not play and what is wrong in it (error scenario_invalid); else null. */
+    invalid: {
+        path: string;
+        reason: string;
+    } | null;
+}
+/** A scenario the cloud could not play, once per scenario. */
+export interface InvalidScenario {
+    scenario: string;
+    path: string;
+    reason: string;
+}
+/** The cloud ended the whole session scenario_invalid: what it said after the code (null when nothing),
+ * whether that looks like a cloud older than this client, and the scenario files this session sent. */
+export interface SessionInvalid {
+    reason: string | null;
+    olderCloud: boolean;
+    sent: Array<{
+        scenario: string;
+        path: string;
+    }>;
 }
 /** `flow test --json`: one document on stdout, whatever the exit. */
 export interface JsonReport {
@@ -50,6 +71,10 @@ export interface JsonReport {
         test: number;
     };
     runs: ReportRun[];
+    /** Each scenario the cloud could not play; the other scenarios played. */
+    invalid: InvalidScenario[];
+    /** Set when the cloud ended the whole session scenario_invalid; else null. */
+    sessionInvalid: SessionInvalid | null;
 }
 /** A run's verdict: errored when the cloud names an error for it or ended it run_failed, since the cloud
  * could not play or judge it and that says nothing about the agent (a judge_failed run can end goal_met). */
@@ -65,6 +90,27 @@ export declare function finishedOutcome(runs: Array<{
 }>): Outcome;
 /** What stopped a session, or the open, as an outcome. `opened`: the session had opened. */
 export declare function stoppedOutcome(error: unknown, opened: boolean): Outcome;
+/** The scenarios the cloud could not play, once each, from the runs it reported (a run per channel and --runs). */
+export declare function invalidScenarios(runs: Array<{
+    scenario: string;
+    invalid?: {
+        path: string;
+        reason: string;
+    } | null;
+}>): InvalidScenario[];
+/** The lines that name each scenario the cloud could not play. */
+export declare function invalidLines(invalid: InvalidScenario[]): string[];
+/** A reason naming a key the cloud does not know: the client may be newer than the cloud. */
+export declare function unsupported(reason: string): string;
+/** A session the cloud ended scenario_invalid; null for any other stop. A tools/ or checks/ file with
+ * its reason is the suite's own problem; anything else (no reason, or a scenario, which a current
+ * cloud reports as that scenario's run) suggests a cloud older than this client. */
+export declare function sessionInvalid(error: unknown, sent: Array<{
+    scenario: string;
+    path: string;
+}>): SessionInvalid | null;
+/** The lines that say why the session ended scenario_invalid. */
+export declare function sessionInvalidLines(info: SessionInvalid): string[];
 /** The error as the JSON report says it: the cloud's status, code and Retry-After when it answered. */
 export declare function reportError(error: unknown, message: string): ReportError;
 /** The connection's error code, from the error or what caused it. */

package/dist/testing/src/report.js

@@ -38,6 +38,48 @@ export function stoppedOutcome(error, opened) {
     // Something on this machine: the agent would not build before the open, or broke the session after it.
     return opened ? "failed" : "invalid";
 }
+/** The scenarios the cloud could not play, once each, from the runs it reported (a run per channel and --runs). */
+export function invalidScenarios(runs) {
+    const out = new Map();
+    for (const r of runs)
+        if (r.invalid)
+            out.set(`${r.scenario}\n${r.invalid.path}`, { scenario: r.scenario, path: r.invalid.path, reason: r.invalid.reason });
+    return [...out.values()];
+}
+/** The lines that name each scenario the cloud could not play. */
+export function invalidLines(invalid) {
+    if (!invalid.length)
+        return [];
+    const lines = [`${invalid.length} ${invalid.length === 1 ? "scenario" : "scenarios"} could not be played (fix ${invalid.length === 1 ? "it" : "them"} and run again):`];
+    for (const i of invalid)
+        lines.push(`  ${i.scenario}: ${i.path}: ${i.reason}${unsupported(i.reason)}`);
+    return lines;
+}
+/** A reason naming a key the cloud does not know: the client may be newer than the cloud. */
+export function unsupported(reason) {
+    return reason.startsWith("unsupported: ") ? " (the test cloud does not know this key; it may be older than this client)" : "";
+}
+/** A session the cloud ended scenario_invalid; null for any other stop. A tools/ or checks/ file with
+ * its reason is the suite's own problem; anything else (no reason, or a scenario, which a current
+ * cloud reports as that scenario's run) suggests a cloud older than this client. */
+export function sessionInvalid(error, sent) {
+    if (!(error instanceof CloudEnded))
+        return null;
+    const match = /^scenario_invalid(?::\s*(.*))?$/s.exec(error.error.trim());
+    if (!match)
+        return null;
+    const reason = match[1]?.trim() || null;
+    const suiteFile = reason !== null && /^(?:\S*\/)?(?:tools|checks)\/[^:]+:\s*\S/.test(reason);
+    return { reason, olderCloud: !suiteFile, sent };
+}
+/** The lines that say why the session ended scenario_invalid. */
+export function sessionInvalidLines(info) {
+    const lines = [`The test cloud could not play this suite${info.reason ? `: ${info.reason}` : " and gave no reason."}`];
+    if (info.olderCloud)
+        lines.push("the test cloud may be older than this client");
+    lines.push(`Scenarios sent: ${info.sent.map((s) => `${s.scenario} (${s.path})`).join(", ") || "none"}`);
+    return lines;
+}
 /** The error as the JSON report says it: the cloud's status, code and Retry-After when it answered. */
 export function reportError(error, message) {
     const refusal = postErrorOf(error);

package/dist/testing/src/show.d.ts

@@ -1,3 +1,4 @@
+import { type SessionInvalid } from "./report.ts";
 /** Where every session's results go, in the agent's folder; `flow new` git-ignores .flow/. */
 export declare const LAST_SESSION: string;
 /** The last session's file in dir: LAST_SESSION, else, before a run has written it, the one from before the rename (.chinwag/). */
@@ -29,7 +30,15 @@ export interface LastSession {
             text: string;
             reason?: string;
         }>;
+        /** The scenario file the cloud could not play and what is wrong in it (error scenario_invalid). */
+        invalid?: {
+            path: string;
+            reason: string;
+        };
     }>;
+    /** The cloud ended the whole session scenario_invalid: its reason, whether the cloud looks older
+     * than the client, and the scenario files sent. */
+    sessionInvalid?: SessionInvalid;
 }
 export declare const SHOW_LINE = "  flow show <run-id|last> [--json]\n                            Print one run's record from the test cloud: its checks with the\n                            judge's reasons, the transcript and the tool calls (flow show --help)";
 /** `flow show`: returns the exit status. */

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@flow-engineer/flow",
-  "version": "0.3.2",
+  "version": "0.3.3",
   "description": "An open-source runtime for chat agents that feel like a person texting back.",
   "license": "Apache-2.0",
   "repository": {