@mercury-fw/cli 0.30.0 → 0.31.1

package/CHANGELOG.md

@@ -1,5 +1,27 @@
 # @mercury-fw/cli
 
+## 0.31.1
+
+### Patch Changes
+
+- Updated dependencies [7926aa2]
+  - @mercury-fw/core@0.31.1
+
+## 0.31.0
+
+### Minor Changes
+
+- 6ae8133: - `mfw local-packages <folder>` makes an app install `@mercury-fw/*` packages from local tarballs (`bun pm pack`) instead of the registry, transitive dependencies included; `--off` goes back to the registry.
+  - `mfw e2e` runs end-to-end tests against an app's real model through its REPL: cases of turns with checks on the tool calls and the answer, repeated runs, results kept in `e2e/results/`.
+  - Test files declare their cases with `e2e()` from `@mercury-fw/cli/e2e`.
+  - A new app's Dockerfile copies `.packs/` when present, and its `.gitignore` leaves out `.packs/` and `e2e/results/`.
+  - In the REPL, `/dump` after a confirmation no longer writes the turn before it.
+
+### Patch Changes
+
+- Updated dependencies [6ae8133]
+  - @mercury-fw/core@0.31.0
+
 ## 0.30.0
 
 ### Minor Changes

package/README.md

@@ -20,6 +20,8 @@
   - [`mfw credentials set <plugin> [--from <dir>] [--print]`](#mfw-credentials-set-plugin---from-dir---print)
   - [`mfw credentials reset <plugin>`](#mfw-credentials-reset-plugin)
   - [`mfw google-chat set-key <key-file> [--subscription <name>]`](#mfw-google-chat-set-key-key-file---subscription-name)
+  - [`mfw local-packages <folder>` / `mfw local-packages --off`](#mfw-local-packages-folder--mfw-local-packages---off)
+  - [`mfw e2e [tests...] [--repeat N]`](#mfw-e2e-tests---repeat-n)
   - [`mfw upgrade`](#mfw-upgrade)
 - [Help](#help)
 
@@ -131,7 +133,7 @@ Maintains the wiki vault, which lives on a Docker volume and not in the app's fo
 |---|---|
 | `list` | Every note. |
 | `read <path>` | One note (a path that isn't one says so, exit 1). |
-| `grep <pattern>` | Every line matching `<pattern>`, a regular expression, as `path:line:text`. A pattern starting with `-` goes after `--` (`mfw vault grep -- -h`). |
+| `grep <pattern>` | Every line matching `<pattern>`, a regular expression (case ignored), as `path:line:text`. A pattern starting with `-` goes after `--` (`mfw vault grep -- -h`). |
 | `write-curated <path> [--author NAME]` | Writes a curated note, the body read from stdin. |
 | `write-raw <path>` | Writes raw material for the nightly review to triage, the body read from stdin. |
 
@@ -212,6 +214,88 @@ mfw google-chat set-key key.json --subscription projects/my-project/subscription
 mfw google-chat set-key new-key.json
 ```
 
+### `mfw local-packages <folder>` / `mfw local-packages --off`
+
+Makes the app install `@mercury-fw/*` packages from local tarballs instead of the registry: the way to try a framework or plugin change in a real app before it's published. `<folder>` holds the `.tgz` files `bun pm pack` writes, one per package; the command copies them into the app's `.packs/`, points each package at its tarball with `overrides` in `package.json`, and runs `bun install`.
+
+Overrides, and not the dependencies themselves, because a packed package names the packages it depends on by version, and the registry has those versions too: only an override sends them to the tarballs as well (a packed `@mercury-fw/core` depends on `@mercury-fw/plugin-types`, for example). The Dockerfile `mfw create` writes copies `.packs/` before installing, so the image gets the same packages (an app created before 0.31.0 needs `COPY --chown=mercury:mercury .pack[s] ./.packs/` added after the line copying `package.json`); `.gitignore` leaves it out of the repository.
+
+Run it again after packing anew: it replaces the tarballs and the overrides of the previous run, and leaves alone the overrides you wrote yourself. `--off` takes the app back to the registry: the overrides it wrote and `.packs/` go, then `bun install`. After either, `mfw start` rebuilds the image with the packages now installed.
+
+```bash
+mfw local-packages ../mercury-fw/apps/testbed/.packs
+mfw local-packages --off
+```
+
+### `mfw e2e [tests...] [--repeat N]`
+
+Runs end-to-end tests against the app's real model: each test case sends its turns to the app's REPL in the container, as `mfw repl` would, and checks what each turn did, the tool calls with their inputs and results, and the answer. It's how you find out whether the model actually uses a plugin the way its skill says, at the first try, with the app's own model, configuration, wiki and credentials. That's also why it doesn't belong in CI: the results depend on the model, on what's in the vault and in Qdrant, and on accounts a CI runner shouldn't have.
+
+Without arguments it runs every `e2e/*.e2e.ts` in the app; with files, those (they can live anywhere). It prints every check of every run, keeps everything (each turn's calls and answer, each check) in `e2e/results/<time>/`, which `.gitignore` leaves out, and exits 1 when a case didn't pass. The app must be built and its `.env` filled in, as for `mfw repl`. `--repeat N` runs each case N times, overriding its own `repeat`.
+
+#### Writing a test
+
+A test is a TypeScript file whose default export is `e2e({ … })`, from `@mercury-fw/cli/e2e` (every app has the CLI among its devDependencies, so the types are there):
+
+```ts
+import { e2e } from "@mercury-fw/cli/e2e";
+
+export default e2e({
+  // What the app must have, by catalog id: checked before anything runs.
+  plugins: ["jira"],
+  channels: [],
+  cases: [
+    {
+      name: "project key, at the first try",
+      turns: ["On Jira, what is the project key of Customer Support?"],
+      repeat: 3, // the model isn't deterministic
+      minPasses: 3, // how many runs must pass; default every one
+      check: (run, expect) => {
+        expect.everyCall("jiraCommand", (c) => String((c.input as { command: string }).command).includes("--select "));
+        expect.noFailedCalls();
+        expect.callCount({ max: 3 });
+        expect.answer(/\bCS\b/);
+      },
+    },
+  ],
+});
+```
+
+Each run of a case gets a fresh REPL session. `check` receives the run, `run.turns` in order and `run.last`, each turn with:
+
+- `calls`: the tool calls, each with `tool`, `input`, `output`, `ok` (false for a failed call, or one that never got a result) and `pending` (an irreversible command staged for confirmation: it worked, and its output holds the token);
+- `answer`: the turn's final text (what the model wrote; a list `present` adds afterwards isn't part of it);
+- `seconds`: how long it took, for the report.
+
+The `expect` helpers record a named check each and never stop the others, so a failed run shows every check that failed:
+
+| Helper | Passes when |
+|---|---|
+| `call(tool, match?)` | at least one call to `tool` (matching `match`, when given) |
+| `everyCall(tool, match)` | there are calls to `tool`, and every one matches |
+| `noFailedCalls()` | no call failed |
+| `callCount({ min?, max? }, tool?)` | the number of calls, to any tool or to `tool`, is within the bounds |
+| `answer(text \| regex)` | the last answer contains the text, or matches |
+| `answerNot(text \| regex)` | it doesn't |
+| `that(label, condition)` | `condition` is true |
+
+The call helpers look at every turn of the run, the answer helpers at the last one; each takes an optional label as its last argument. A case that records no check fails: it proves nothing.
+
+A turn can be a function of the one before, for a follow-up or a confirmation: `(previous) => …` returns the next message from `previous.calls` and `previous.answer` (an irreversible command comes back as a call whose output holds the pending confirmation's token, and sending the token as the next turn confirms it). Each turn is one line, as the REPL reads them.
+
+`before`, `after` and `check` also get `cli(command)`, which runs `sh -c command` in the app's container, outside the model, and resolves to its exit code and output: to prepare data (an issue to work on, a note in the wiki with the vault CLI), to check what a turn changed (`jira issue get KEY --select fields.assignee.displayName` after an assignment), to clean up. `after` runs even when the run or the check failed. A test that changes an external system has to clean up after itself, so start from read-only ones.
+
+#### What it can and can't test
+
+It can test how the model uses a plugin through its skill (the commands and flags it picks, rejected or failed calls, how many calls it takes), what the answer says or must not say, what a turn changed in an external system, the confirmation of an irreversible command, conversations over several turns, several plugins working together, and what the wiki and memory make of a conversation.
+
+It can't test how a channel shows a turn (Google Chat's cards, the HTTP channel's events: the REPL runs the same turn, without them), the background jobs (the nightly wiki review, idle-session capture), or the exact wording of an answer: checks look for properties, and `repeat` with `minPasses` says how steady the behaviour is. Durations are in the report and in `e2e/results/`, never a check.
+
+```bash
+mfw e2e
+mfw e2e e2e/jira.e2e.ts --repeat 3
+```
+
 ### `mfw upgrade`
 
 Installs the registry's latest `mfw` globally (`bun add -g @mercury-fw/cli@<latest>`) when it's newer than the one running, and says it's already the latest otherwise; a registry that doesn't answer exits 1. It's about the global `mfw` only: an app's framework, its own CLI included, moves with `bun update` in the app. `mfw create` run by a stale global `mfw` creates the app with the latest one anyway, and says to run this.

package/dist/src/app/commands.d.ts

@@ -74,6 +74,19 @@ export declare function appCommands(app: App, deps: AppDeps): {
     googleChatSetKey: (keyFile: string, { subscription }: {
         subscription?: string;
     }) => Promise<number>;
+    /** Makes the app install the packages in `from`'s tarballs (`bun pm
+     * pack`) instead of the registry's: copied into `.packs/` (which the
+     * image copies too), overridden in the manifest, then installed.
+     * Everything is checked before anything is written. */
+    localPackages: (from: string) => Promise<number>;
+    /** Runs e2e tests (`tests`, or the app's `e2e/*.e2e.ts`) against the app's
+     * REPL in its container, keeping the turns and checks in
+     * `e2e/results/<time>/`; returns 1 when a case didn't pass. */
+    e2e: (tests: string[], { repeat }: {
+        repeat?: number;
+    }) => Promise<number>;
+    /** Undoes `localPackages`: the app installs from the registry again. */
+    localPackagesOff: () => Promise<number>;
 };
 /** The real deps: docker on the user's terminal, questions on `input`
  * (stdin by default). */

package/dist/src/app/local-packages.d.ts

@@ -0,0 +1,26 @@
+/**
+ * What `mfw local-packages` writes: the app's `package.json` with `overrides`
+ * pointing packages at local tarballs (`bun pm pack`) copied into `.packs/`,
+ * which the image copies before `bun install`. Overrides, and not the
+ * dependencies themselves, because a packed package names its siblings by
+ * version, which the registry has too: only an override sends those
+ * transitive dependencies to the tarballs as well.
+ */
+/** The tarballs' folder inside the app, as the Dockerfile copies it. */
+export declare const LOCAL_PACKS_DIR = ".packs";
+type Manifest = Record<string, unknown> & {
+    overrides?: Record<string, string>;
+};
+/** `pkg` with each of `packs` (a package name and its tarball's file name in
+ * `.packs/`) as an override, in place of any left by an earlier run. */
+export declare function withLocalOverrides(pkg: Manifest, packs: Array<{
+    name: string;
+    file: string;
+}>): Manifest;
+/** `pkg` without the overrides this command writes; without `overrides` at
+ * all when none of the user's own is left. */
+export declare function withoutLocalOverrides(pkg: Manifest): Manifest;
+/** The package name in a `bun pm pack` tarball, read from its
+ * `package/package.json`. */
+export declare function packageNameOf(tarball: string): Promise<string>;
+export {};

package/dist/src/e2e/define.d.ts

@@ -0,0 +1,73 @@
+/**
+ * The shape of an e2e test, as a test file imports it
+ * (`import { e2e } from "@mercury-fw/cli/e2e"`): the plugins and channels the
+ * app must have, and cases of turns sent to the app's real model through its
+ * REPL, each with checks on the calls it made and the answer it gave.
+ * `mfw e2e` runs them (`runner.ts`).
+ */
+import type { Call, TurnData } from "./dump.ts";
+export type { Call } from "./dump.ts";
+/** One turn as a check sees it: its calls, its answer, how long it took. */
+export type Turn = TurnData & {
+    seconds: number;
+};
+/** A case's run: every turn in order, and the last one. */
+export type Run = {
+    turns: Turn[];
+    last: Turn;
+};
+/** Runs `command` in the app's container (`sh -c`), outside the model:
+ * preparing data, reading what a turn changed, cleaning up. */
+export type Cli = (command: string) => Promise<{
+    code: number;
+    output: string;
+}>;
+/** What `before`, `after` and `check` can reach besides the run. */
+export type Context = {
+    cli: Cli;
+};
+/** The checks a case makes. Call helpers look at every turn's calls, answer
+ * helpers at the last answer; each records a named check and never throws,
+ * so a case reports every failure at once. */
+export type Expect = {
+    /** At least one call to `tool`, matching `match` when given. */
+    call(tool: string, match?: (call: Call) => boolean, label?: string): void;
+    /** Every call to `tool` matches `match` (and there is at least one). */
+    everyCall(tool: string, match: (call: Call) => boolean, label?: string): void;
+    /** No call failed or went without a result. */
+    noFailedCalls(label?: string): void;
+    /** The number of calls, to every tool or to `tool`, within the bounds. */
+    callCount(bounds: {
+        min?: number;
+        max?: number;
+    }, tool?: string, label?: string): void;
+    /** The answer contains `pattern`, or matches it. */
+    answer(pattern: string | RegExp, label?: string): void;
+    /** The answer doesn't contain `pattern`, or doesn't match it. */
+    answerNot(pattern: string | RegExp, label?: string): void;
+    /** Anything else. */
+    that(label: string, condition: boolean): void;
+};
+/** One case: the turns sent, in one REPL session, and the checks on them. */
+export type E2eCase = {
+    name: string;
+    /** A message, or a function of the turn before (a follow-up, a confirmation token). */
+    turns: Array<string | ((previous: Turn) => string)>;
+    /** How many times to run it (the model isn't deterministic); default 1. */
+    repeat?: number;
+    /** How many runs must pass; default every one. */
+    minPasses?: number;
+    before?: (ctx: Context) => unknown;
+    after?: (ctx: Context) => unknown;
+    check: (run: Run, expect: Expect, ctx: Context) => unknown;
+};
+/** An e2e test: what the app must have, and its cases. */
+export type E2eTest = {
+    /** Catalog ids of the tool plugins the app must have (`jira`, …). */
+    plugins?: string[];
+    /** Catalog ids of the channels the app must have (`http`, …). */
+    channels?: string[];
+    cases: E2eCase[];
+};
+/** Declares an e2e test: the identity, there for the types. */
+export declare function e2e(test: E2eTest): E2eTest;

package/dist/src/e2e/dump.d.ts

@@ -0,0 +1,24 @@
+/**
+ * A turn read out of the file the REPL's `/dump` writes: the AI SDK's step
+ * results for the last turn, whose `content` parts are the tool calls, their
+ * results (or errors) and the model's text. Read as data, not through the
+ * SDK's types: only the parts listed here matter.
+ */
+/** One tool call as a check sees it. `ok` is false when the call failed or
+ * never got a result; a result without an `ok` of its own counts as worked.
+ * `pending` is an irreversible command staged for confirmation, which worked
+ * (its output has the token) though it reports `ok: false`. */
+export type Call = {
+    tool: string;
+    input: unknown;
+    output: unknown;
+    ok: boolean;
+    pending: boolean;
+};
+/** What a turn did: its tool calls in order, and its final text. */
+export type TurnData = {
+    calls: Call[];
+    answer: string;
+};
+/** The calls and the answer in `dump`, the parsed content of a `/dump` file. */
+export declare function turnFromDump(dump: unknown): TurnData;

package/dist/src/e2e/expect.d.ts

@@ -0,0 +1,12 @@
+import type { Expect, Run } from "./define.ts";
+/** One check's outcome; `detail` says what was found when it failed. */
+export type Check = {
+    label: string;
+    ok: boolean;
+    detail?: string;
+};
+/** An `expect` bound to `run`, and the checks it recorded so far. */
+export declare function createExpect(run: Run): {
+    expect: Expect;
+    checks: () => Check[];
+};

package/dist/src/e2e/load.d.ts

@@ -0,0 +1,9 @@
+import type { E2eTest } from "./define.ts";
+/** The files to run: `named` resolved from `cwd`, or, when none is named,
+ * the app's `e2e/*.e2e.ts` in name order. */
+export declare function findTests(named: string[], { appDir, cwd }: {
+    appDir: string;
+    cwd: string;
+}): string[];
+/** The test `file` exports as default; throws naming the file when it isn't one. */
+export declare function loadTest(file: string): Promise<E2eTest>;

package/dist/src/e2e/runner.d.ts

@@ -0,0 +1,42 @@
+import type { Context, E2eTest, Turn } from "./define.ts";
+import { type Check } from "./expect.ts";
+/** A REPL session in the app: `turn` sends one line and resolves with what
+ * `/dump` wrote for it and what the REPL printed meanwhile. */
+export type Session = {
+    turn: (line: string) => Promise<{
+        dump: unknown;
+        output: string;
+    }>;
+    close: () => Promise<void>;
+};
+export type RunnerDeps = {
+    openSession: () => Promise<Session>;
+    cli: Context["cli"];
+    /** The app's dependencies, to check the test's plugins and channels against. */
+    appPackages: Record<string, string>;
+    print: (line: string) => void;
+    /** Milliseconds, for each turn's duration. */
+    now: () => number;
+    writeReport: (report: CaseReport[]) => Promise<void>;
+};
+/** One run of a case, as the report keeps it. */
+export type RunReport = {
+    ok: boolean;
+    turns: Turn[];
+    checks: Check[];
+};
+export type CaseReport = {
+    file: string;
+    case: string;
+    passed: boolean;
+    runs: RunReport[];
+};
+/** Runs the tests in `tests` (each with the file it came from); returns the
+ * exit code: 0 when every case passed enough runs. `repeat` overrides each
+ * case's own. */
+export declare function runE2e(tests: Array<{
+    file: string;
+    test: E2eTest;
+}>, opts: {
+    repeat?: number;
+}, deps: RunnerDeps): Promise<number>;

package/dist/src/e2e/session.d.ts

@@ -0,0 +1,17 @@
+import type { Session } from "./runner.ts";
+export type ReplSessionOptions = {
+    /** The command that starts the REPL. */
+    argv: string[];
+    cwd: string;
+    /** Where the dumps are, on this side, and as the REPL sees the same folder. */
+    hostDir: string;
+    replDir: string;
+    /** Prefix of this session's dump files, unique among sessions sharing the folder. */
+    name: string;
+    /** How long one turn may take. */
+    timeoutMs: number;
+};
+/** Starts the REPL and returns the session over it once the REPL is ready
+ * (its first prompt is out), so a turn's time is only the turn's; throws
+ * when the REPL exits or doesn't get there in time. */
+export declare function openReplSession(opts: ReplSessionOptions): Promise<Session>;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mercury-fw/cli",
-  "version": "0.30.0",
+  "version": "0.31.1",
   "type": "module",
   "license": "MIT",
   "repository": {
@@ -27,6 +27,11 @@
       "mercury-fw-source": "./src/main.ts",
       "types": "./dist/src/main.d.ts",
       "default": "./src/main.ts"
+    },
+    "./e2e": {
+      "mercury-fw-source": "./src/e2e/define.ts",
+      "types": "./dist/src/e2e/define.d.ts",
+      "default": "./src/e2e/define.ts"
     }
   },
   "scripts": {
@@ -36,16 +41,16 @@
   "comment:shape": "The Mercury CLI (`mfw`). `mfw create <dir>` writes a new Mercury app from the template. The framework packages move in lockstep with it, so a new app gets them at the CLI's own version; plugins and channels at the registry's latest. The plugin and channel packages are devDependencies only for the catalog test. `main(argv)` is exported for create-mercury-agent.",
   "dependencies": {
     "@clack/prompts": "^1.8.1",
-    "@mercury-fw/core": "0.30.0",
+    "@mercury-fw/core": "0.31.1",
     "commander": "^15.0.0"
   },
   "devDependencies": {
     "@mercury-fw/channel-google-chat": "0.1.3",
-    "@mercury-fw/channel-http": "0.1.0",
-    "@mercury-fw/formatter": "0.30.0",
+    "@mercury-fw/channel-http": "0.1.1",
+    "@mercury-fw/formatter": "0.31.1",
     "@mercury-fw/plugin-atlassian-admin": "0.1.0",
     "@mercury-fw/plugin-bitbucket": "0.1.0",
-    "@mercury-fw/plugin-jira": "0.3.0",
+    "@mercury-fw/plugin-jira": "0.4.1",
     "@mercury-fw/typescript-config": "*",
     "@types/bun": "^1.4.2",
     "typescript": "^6.0.3"

package/src/app/commands.ts

@@ -5,12 +5,16 @@
  * and validated before any of this runs (`program.ts`); what runs a command is
  * injected (`AppDeps`), which is how the tests see the exact calls.
  */
-import { readFileSync } from "node:fs";
+import { chmodSync, cpSync, existsSync, mkdirSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
 import { homedir } from "node:os";
-import { join, resolve } from "node:path";
+import { join, relative, resolve } from "node:path";
 import { CATALOG, type CliCredentials } from "../catalog.ts";
 import { packCredentials, readServiceAccountKey, setEnvVar } from "./credentials.ts";
 import type { App } from "./find-app.ts";
+import { LOCAL_PACKS_DIR, packageNameOf, withLocalOverrides, withoutLocalOverrides } from "./local-packages.ts";
+import { findTests, loadTest } from "../e2e/load.ts";
+import { runE2e } from "../e2e/runner.ts";
+import { openReplSession } from "../e2e/session.ts";
 
 export type AppDeps = {
   /** Runs `argv` in `cwd` on the user's terminal (stdin, stdout, stderr) and returns its exit code. */
@@ -48,6 +52,16 @@ export const RESET_TARGETS = {
 } as const;
 export type ResetTarget = keyof typeof RESET_TARGETS;
 
+/** How long one e2e turn may take: a local model on a long, tool-heavy turn is slow. */
+const E2E_TURN_TIMEOUT_MS = 10 * 60_000;
+
+/** Runs `argv` in `cwd`, returning its exit code and its output (stdout and stderr together). */
+async function captureCode(argv: string[], cwd: string): Promise<{ code: number; output: string }> {
+  const proc = Bun.spawn(argv, { cwd, stdin: "ignore", stdout: "pipe", stderr: "pipe" });
+  const [stdout, stderr, code] = await Promise.all([new Response(proc.stdout).text(), new Response(proc.stderr).text(), proc.exited]);
+  return { code, output: stdout + stderr };
+}
+
 /** The compose calls that build and start the app; `recreate` restarts containers even when nothing changed. */
 function startCalls(noCache: boolean, recreate: boolean): string[][] {
   const up = [...COMPOSE, "up", "-d", ...(noCache ? [] : ["--build"]), ...(recreate ? ["--force-recreate"] : [])];
@@ -174,8 +188,77 @@ export function appCommands(app: App, deps: AppDeps) {
       deps.print(`Delete ${source} now, the env file holds the key. mfw start applies it to a running app.`);
       return 0;
     },
+    /** Makes the app install the packages in `from`'s tarballs (`bun pm
+     * pack`) instead of the registry's: copied into `.packs/` (which the
+     * image copies too), overridden in the manifest, then installed.
+     * Everything is checked before anything is written. */
+    localPackages: async (from: string) => {
+      const source = resolve(from);
+      const target = join(app.dir, LOCAL_PACKS_DIR);
+      if (source === target) throw new Error(`${source} is the app's own .packs/: give the folder the tarballs were packed into.`);
+      if (!existsSync(source)) throw new Error(`${source} doesn't exist.`);
+      const files = readdirSync(source).filter((f) => f.endsWith(".tgz")).sort();
+      if (files.length === 0) throw new Error(`No .tgz in ${source}: pack the packages there first (bun pm pack).`);
+      const packs = await Promise.all(files.map(async (file) => ({ name: await packageNameOf(join(source, file)), file })));
+      const manifest = withLocalOverrides(readManifest(), packs);
+      rmSync(target, { recursive: true, force: true });
+      mkdirSync(target);
+      for (const { file } of packs) cpSync(join(source, file), join(target, file));
+      writeManifest(manifest);
+      deps.print(`${packs.length} local packages in ${target}: ${packs.map((p) => p.name).join(", ")}.`);
+      return deps.run(["bun", "install"], { cwd: app.dir });
+    },
+    /** Runs e2e tests (`tests`, or the app's `e2e/*.e2e.ts`) against the app's
+     * REPL in its container, keeping the turns and checks in
+     * `e2e/results/<time>/`; returns 1 when a case didn't pass. */
+    e2e: async (tests: string[], { repeat }: { repeat?: number }) => {
+      const files = findTests(tests, { appDir: app.dir, cwd: process.cwd() });
+      const loaded = await Promise.all(files.map(async (file) => ({ file: relative(process.cwd(), file) || file, test: await loadTest(file) })));
+      const results = join(app.dir, "e2e", "results", new Date().toISOString().replace(/[:.]/g, "-"));
+      mkdirSync(results, { recursive: true });
+      // The container's user writes the dumps here; on a Linux host it isn't
+      // the folder's owner.
+      chmodSync(results, 0o777);
+      let sessions = 0;
+      return runE2e(loaded, repeat === undefined ? {} : { repeat }, {
+        openSession: async () =>
+          openReplSession({
+            argv: [...COMPOSE, "run", "--rm", "-T", "-v", `${results}:/e2e`, SERVICE, "bun", "run", "repl"],
+            cwd: app.dir,
+            hostDir: results,
+            replDir: "/e2e",
+            name: `run-${++sessions}`,
+            timeoutMs: E2E_TURN_TIMEOUT_MS,
+          }),
+        cli: (command) => captureCode([...COMPOSE, "run", "--rm", "-T", "--no-deps", SERVICE, "sh", "-c", command], app.dir),
+        appPackages: (readManifest() as { dependencies?: Record<string, string> }).dependencies ?? {},
+        print: deps.print,
+        now: Date.now,
+        writeReport: async (report) => {
+          writeFileSync(join(results, "report.json"), `${JSON.stringify(report, null, 2)}\n`);
+          deps.print(`Turns and checks in ${results}`);
+        },
+      });
+    },
+    /** Undoes `localPackages`: the app installs from the registry again. */
+    localPackagesOff: async () => {
+      writeManifest(withoutLocalOverrides(readManifest()));
+      rmSync(join(app.dir, LOCAL_PACKS_DIR), { recursive: true, force: true });
+      deps.print("Local packages removed: installing from the registry.");
+      return deps.run(["bun", "install"], { cwd: app.dir });
+    },
   };
 
+  /** The app's manifest, as an object. */
+  function readManifest(): Record<string, unknown> {
+    return JSON.parse(readFileSync(join(app.dir, "package.json"), "utf-8")) as Record<string, unknown>;
+  }
+
+  /** Writes the app's manifest, two-space indented like the one `mfw create` writes. */
+  function writeManifest(manifest: Record<string, unknown>): void {
+    writeFileSync(join(app.dir, "package.json"), `${JSON.stringify(manifest, null, 2)}\n`);
+  }
+
   /** Stops `service`, runs `steps`, starts it again; stops at the first
    * failure and, once the service is stopped, says it's down and how to bring
    * it back. Returns the exit code. */

package/src/app/local-packages.ts

@@ -0,0 +1,51 @@
+/**
+ * What `mfw local-packages` writes: the app's `package.json` with `overrides`
+ * pointing packages at local tarballs (`bun pm pack`) copied into `.packs/`,
+ * which the image copies before `bun install`. Overrides, and not the
+ * dependencies themselves, because a packed package names its siblings by
+ * version, which the registry has too: only an override sends those
+ * transitive dependencies to the tarballs as well.
+ */
+
+/** The tarballs' folder inside the app, as the Dockerfile copies it. */
+export const LOCAL_PACKS_DIR = ".packs";
+
+/** The prefix of every override this command writes, how it tells them from the user's own. */
+const LOCAL_OVERRIDE = `file:./${LOCAL_PACKS_DIR}/`;
+
+type Manifest = Record<string, unknown> & { overrides?: Record<string, string> };
+
+/** The user's own overrides in `pkg`: every one this command didn't write. */
+function ownOverrides(pkg: Manifest): Record<string, string> {
+  return Object.fromEntries(Object.entries(pkg.overrides ?? {}).filter(([, spec]) => !spec.startsWith(LOCAL_OVERRIDE)));
+}
+
+/** `pkg` with each of `packs` (a package name and its tarball's file name in
+ * `.packs/`) as an override, in place of any left by an earlier run. */
+export function withLocalOverrides(pkg: Manifest, packs: Array<{ name: string; file: string }>): Manifest {
+  const local = Object.fromEntries(packs.map((p) => [p.name, `${LOCAL_OVERRIDE}${p.file}`]));
+  return { ...pkg, overrides: { ...ownOverrides(pkg), ...local } };
+}
+
+/** `pkg` without the overrides this command writes; without `overrides` at
+ * all when none of the user's own is left. */
+export function withoutLocalOverrides(pkg: Manifest): Manifest {
+  const { overrides: _, ...rest } = pkg;
+  const own = ownOverrides(pkg);
+  return Object.keys(own).length > 0 ? { ...rest, overrides: own } : rest;
+}
+
+/** The package name in a `bun pm pack` tarball, read from its
+ * `package/package.json`. */
+export async function packageNameOf(tarball: string): Promise<string> {
+  const proc = Bun.spawn(["tar", "-xOzf", tarball, "package/package.json"], { stdout: "pipe", stderr: "pipe" });
+  const [manifest, code] = await Promise.all([new Response(proc.stdout).text(), proc.exited]);
+  try {
+    if (code !== 0) throw new Error(`tar exited with ${code}`);
+    const name = (JSON.parse(manifest) as { name?: unknown }).name;
+    if (typeof name !== "string") throw new Error("no name in its package.json");
+    return name;
+  } catch (err) {
+    throw new Error(`${tarball} isn't a package tarball: ${err instanceof Error ? err.message : String(err)}`);
+  }
+}

package/src/e2e/define.ts

@@ -0,0 +1,71 @@
+/**
+ * The shape of an e2e test, as a test file imports it
+ * (`import { e2e } from "@mercury-fw/cli/e2e"`): the plugins and channels the
+ * app must have, and cases of turns sent to the app's real model through its
+ * REPL, each with checks on the calls it made and the answer it gave.
+ * `mfw e2e` runs them (`runner.ts`).
+ */
+import type { Call, TurnData } from "./dump.ts";
+
+export type { Call } from "./dump.ts";
+
+/** One turn as a check sees it: its calls, its answer, how long it took. */
+export type Turn = TurnData & { seconds: number };
+
+/** A case's run: every turn in order, and the last one. */
+export type Run = { turns: Turn[]; last: Turn };
+
+/** Runs `command` in the app's container (`sh -c`), outside the model:
+ * preparing data, reading what a turn changed, cleaning up. */
+export type Cli = (command: string) => Promise<{ code: number; output: string }>;
+
+/** What `before`, `after` and `check` can reach besides the run. */
+export type Context = { cli: Cli };
+
+/** The checks a case makes. Call helpers look at every turn's calls, answer
+ * helpers at the last answer; each records a named check and never throws,
+ * so a case reports every failure at once. */
+export type Expect = {
+  /** At least one call to `tool`, matching `match` when given. */
+  call(tool: string, match?: (call: Call) => boolean, label?: string): void;
+  /** Every call to `tool` matches `match` (and there is at least one). */
+  everyCall(tool: string, match: (call: Call) => boolean, label?: string): void;
+  /** No call failed or went without a result. */
+  noFailedCalls(label?: string): void;
+  /** The number of calls, to every tool or to `tool`, within the bounds. */
+  callCount(bounds: { min?: number; max?: number }, tool?: string, label?: string): void;
+  /** The answer contains `pattern`, or matches it. */
+  answer(pattern: string | RegExp, label?: string): void;
+  /** The answer doesn't contain `pattern`, or doesn't match it. */
+  answerNot(pattern: string | RegExp, label?: string): void;
+  /** Anything else. */
+  that(label: string, condition: boolean): void;
+};
+
+/** One case: the turns sent, in one REPL session, and the checks on them. */
+export type E2eCase = {
+  name: string;
+  /** A message, or a function of the turn before (a follow-up, a confirmation token). */
+  turns: Array<string | ((previous: Turn) => string)>;
+  /** How many times to run it (the model isn't deterministic); default 1. */
+  repeat?: number;
+  /** How many runs must pass; default every one. */
+  minPasses?: number;
+  before?: (ctx: Context) => unknown;
+  after?: (ctx: Context) => unknown;
+  check: (run: Run, expect: Expect, ctx: Context) => unknown;
+};
+
+/** An e2e test: what the app must have, and its cases. */
+export type E2eTest = {
+  /** Catalog ids of the tool plugins the app must have (`jira`, …). */
+  plugins?: string[];
+  /** Catalog ids of the channels the app must have (`http`, …). */
+  channels?: string[];
+  cases: E2eCase[];
+};
+
+/** Declares an e2e test: the identity, there for the types. */
+export function e2e(test: E2eTest): E2eTest {
+  return test;
+}

package/src/e2e/dump.ts

@@ -0,0 +1,48 @@
+/**
+ * A turn read out of the file the REPL's `/dump` writes: the AI SDK's step
+ * results for the last turn, whose `content` parts are the tool calls, their
+ * results (or errors) and the model's text. Read as data, not through the
+ * SDK's types: only the parts listed here matter.
+ */
+
+/** One tool call as a check sees it. `ok` is false when the call failed or
+ * never got a result; a result without an `ok` of its own counts as worked.
+ * `pending` is an irreversible command staged for confirmation, which worked
+ * (its output has the token) though it reports `ok: false`. */
+export type Call = { tool: string; input: unknown; output: unknown; ok: boolean; pending: boolean };
+
+/** What a turn did: its tool calls in order, and its final text. */
+export type TurnData = { calls: Call[]; answer: string };
+
+type Part = { type?: unknown; toolCallId?: unknown; toolName?: unknown; input?: unknown; output?: unknown; error?: unknown; text?: unknown };
+
+/** The calls and the answer in `dump`, the parsed content of a `/dump` file. */
+export function turnFromDump(dump: unknown): TurnData {
+  if (!Array.isArray(dump)) throw new Error("the file isn't a /dump of steps");
+  const parts: Part[][] = dump.map((step) =>
+    Array.isArray((step as { content?: unknown } | null)?.content) ? ((step as { content: Part[] }).content) : [],
+  );
+
+  const calls: Array<Call & { id: unknown }> = [];
+  for (const part of parts.flat()) {
+    if (part.type === "tool-call") {
+      calls.push({ id: part.toolCallId, tool: String(part.toolName), input: part.input, output: undefined, ok: false, pending: false });
+      continue;
+    }
+    const call = calls.find((c) => c.id === part.toolCallId);
+    if (call === undefined) continue;
+    if (part.type === "tool-result") {
+      call.output = part.output;
+      const result = part.output as { ok?: unknown; pendingConfirmation?: unknown } | null;
+      call.pending = result?.pendingConfirmation === true;
+      call.ok = call.pending || (typeof result?.ok === "boolean" ? result.ok : true);
+    } else if (part.type === "tool-error") {
+      call.output = { error: part.error };
+      call.ok = false;
+    }
+  }
+
+  const last = parts.at(-1) ?? [];
+  const answer = last.filter((p) => p.type === "text" && typeof p.text === "string").map((p) => p.text as string).join("");
+  return { calls: calls.map(({ id: _, ...call }) => call), answer };
+}

package/src/e2e/expect.ts

@@ -0,0 +1,59 @@
+/**
+ * The `expect` a case's `check` receives: helpers that record named checks
+ * against a run, passed or failed with a detail, and never throw.
+ */
+import type { Call } from "./dump.ts";
+import type { Expect, Run } from "./define.ts";
+
+/** One check's outcome; `detail` says what was found when it failed. */
+export type Check = { label: string; ok: boolean; detail?: string };
+
+/** A call, one line, for a failure's detail. */
+const describe = (c: Call) => `${c.tool} ${JSON.stringify(c.input)}`;
+
+/** `pattern`, as a label says it. */
+const said = (pattern: string | RegExp) => (typeof pattern === "string" ? `contains "${pattern}"` : `matches ${pattern}`);
+const saidNot = (pattern: string | RegExp) => (typeof pattern === "string" ? `doesn't contain "${pattern}"` : `doesn't match ${pattern}`);
+const found = (text: string, pattern: string | RegExp) => (typeof pattern === "string" ? text.includes(pattern) : pattern.test(text));
+
+/** The checks' bounds, as a label says them. */
+function boundsLabel({ min, max }: { min?: number; max?: number }, what: string): string {
+  if (min !== undefined && max !== undefined) return `${min} to ${max} ${what}`;
+  if (max !== undefined) return `at most ${max} ${what}`;
+  return `at least ${min ?? 0} ${what}`;
+}
+
+/** An `expect` bound to `run`, and the checks it recorded so far. */
+export function createExpect(run: Run): { expect: Expect; checks: () => Check[] } {
+  const checks: Check[] = [];
+  const record = (label: string, ok: boolean, detail?: string) =>
+    void checks.push(ok || detail === undefined ? { label, ok } : { label, ok, detail });
+  const calls = run.turns.flatMap((t) => t.calls);
+  const of = (tool: string) => calls.filter((c) => c.tool === tool);
+
+  const expect: Expect = {
+    call: (tool, match, label) => {
+      const ok = of(tool).some((c) => match?.(c) ?? true);
+      record(label ?? `a ${tool} call`, ok, `calls: ${calls.map((c) => c.tool).join(", ") || "none"}`);
+    },
+    everyCall: (tool, match, label) => {
+      const mine = of(tool);
+      const bad = mine.find((c) => !match(c));
+      record(label ?? `every ${tool} call matches`, mine.length > 0 && bad === undefined, bad ? describe(bad) : `no ${tool} call`);
+    },
+    noFailedCalls: (label) => {
+      const failed = calls.filter((c) => !c.ok);
+      record(label ?? "no failed calls", failed.length === 0, failed.map(describe).join("; "));
+    },
+    callCount: (bounds, tool, label) => {
+      const n = (tool === undefined ? calls : of(tool)).length;
+      const ok = n >= (bounds.min ?? 0) && n <= (bounds.max ?? Infinity);
+      const what = tool === undefined ? "calls" : `${tool} call${bounds.max === 1 || bounds.min === 1 ? "" : "s"}`;
+      record(label ?? boundsLabel(bounds, what), ok, String(n));
+    },
+    answer: (pattern, label) => record(label ?? `answer ${said(pattern)}`, found(run.last.answer, pattern), run.last.answer),
+    answerNot: (pattern, label) => record(label ?? `answer ${saidNot(pattern)}`, !found(run.last.answer, pattern), run.last.answer),
+    that: (label, condition) => record(label, condition),
+  };
+  return { expect, checks: () => [...checks] };
+}

package/src/e2e/load.ts

@@ -0,0 +1,40 @@
+/**
+ * Which e2e test files `mfw e2e` runs, and loading one: Bun imports the
+ * TypeScript file as it is, and its default export is checked to look like
+ * a test before anything starts.
+ */
+import { existsSync, readdirSync } from "node:fs";
+import { join, resolve } from "node:path";
+import { pathToFileURL } from "node:url";
+import type { E2eTest } from "./define.ts";
+
+/** The files to run: `named` resolved from `cwd`, or, when none is named,
+ * the app's `e2e/*.e2e.ts` in name order. */
+export function findTests(named: string[], { appDir, cwd }: { appDir: string; cwd: string }): string[] {
+  if (named.length > 0) return named.map((f) => resolve(cwd, f));
+  const folder = join(appDir, "e2e");
+  const found = existsSync(folder) ? readdirSync(folder).filter((f) => f.endsWith(".e2e.ts")).sort() : [];
+  if (found.length === 0) throw new Error(`No tests: none named, and no *.e2e.ts in ${folder}`);
+  return found.map((f) => join(folder, f));
+}
+
+/** The test `file` exports as default; throws naming the file when it isn't one. */
+export async function loadTest(file: string): Promise<E2eTest> {
+  const loaded = ((await import(pathToFileURL(file).href)) as { default?: unknown }).default as Partial<E2eTest> | undefined;
+  if (loaded === undefined || loaded === null || !Array.isArray(loaded.cases)) {
+    throw new Error(`${file} doesn't export a test as default (export default e2e({ … }))`);
+  }
+  for (const [i, c] of loaded.cases.entries()) {
+    const which = `${file}: case ${i + 1} ("${c?.name ?? "?"}")`;
+    if (typeof c?.name !== "string") throw new Error(`${which} has no name`);
+    if (!Array.isArray(c.turns) || c.turns.length === 0) throw new Error(`${which} has no turns`);
+    if (typeof c.check !== "function") throw new Error(`${which} has no check`);
+    const whole = (n: unknown) => typeof n === "number" && Number.isInteger(n) && n >= 1;
+    if (c.repeat !== undefined && !whole(c.repeat)) throw new Error(`${which} repeat takes a positive whole number`);
+    if (c.minPasses !== undefined && !whole(c.minPasses)) throw new Error(`${which} minPasses takes a positive whole number`);
+    if (c.minPasses !== undefined && c.minPasses > (c.repeat ?? 1)) {
+      throw new Error(`${which} minPasses (${c.minPasses}) is more than repeat (${c.repeat ?? 1})`);
+    }
+  }
+  return loaded as E2eTest;
+}

package/src/e2e/runner.ts

@@ -0,0 +1,153 @@
+/**
+ * Runs e2e tests against an app: for each case (as many times as it
+ * repeats), a fresh REPL session gets the case's turns, the checks run on
+ * what each turn did, and every check is printed; the exit code says whether
+ * every case passed enough runs. The session, the container commands and the
+ * report's destination are injected (`session.ts` and `commands.ts` hold the
+ * real ones), so the tests drive the runner with scripted turns.
+ */
+import { CATALOG } from "../catalog.ts";
+import type { Context, E2eCase, E2eTest, Run, Turn } from "./define.ts";
+import { turnFromDump } from "./dump.ts";
+import { createExpect, type Check } from "./expect.ts";
+
+/** A REPL session in the app: `turn` sends one line and resolves with what
+ * `/dump` wrote for it and what the REPL printed meanwhile. */
+export type Session = {
+  turn: (line: string) => Promise<{ dump: unknown; output: string }>;
+  close: () => Promise<void>;
+};
+
+export type RunnerDeps = {
+  openSession: () => Promise<Session>;
+  cli: Context["cli"];
+  /** The app's dependencies, to check the test's plugins and channels against. */
+  appPackages: Record<string, string>;
+  print: (line: string) => void;
+  /** Milliseconds, for each turn's duration. */
+  now: () => number;
+  writeReport: (report: CaseReport[]) => Promise<void>;
+};
+
+/** One run of a case, as the report keeps it. */
+export type RunReport = { ok: boolean; turns: Turn[]; checks: Check[] };
+export type CaseReport = { file: string; case: string; passed: boolean; runs: RunReport[] };
+
+/** The prompt the REPL prints after a reply, with its context-usage suffix. */
+const PROMPT = /(\[[^\]\n]*\] )?> $/;
+
+/** A turn's printed output as an answer: no styling, no trailing prompt. */
+function answerFromOutput(output: string): string {
+  return output.replace(/\u001b\[[0-9;]*m/g, "").replace(PROMPT, "").trim();
+}
+
+/** The test's plugins and channels the app doesn't depend on, as a message, or undefined. */
+function missing(test: E2eTest, packages: Record<string, string>): string | undefined {
+  const wanted = [
+    ...(test.plugins ?? []).map((id) => ({ id, kind: "tool" as const, word: "plugin" })),
+    ...(test.channels ?? []).map((id) => ({ id, kind: "channel" as const, word: "channel" })),
+  ];
+  const lacking = wanted.flatMap(({ id, kind, word }) => {
+    const pkg = CATALOG.find((e) => e.kind === kind && e.id === id)?.package;
+    if (pkg === undefined) return [`${word} ${id} (not in the catalog)`];
+    return packages[pkg] === undefined ? [`${word} ${id} (${pkg})`] : [];
+  });
+  return lacking.length > 0 ? `the app lacks what the test needs: ${lacking.join(", ")}` : undefined;
+}
+
+/** Runs the tests in `tests` (each with the file it came from); returns the
+ * exit code: 0 when every case passed enough runs. `repeat` overrides each
+ * case's own. */
+export async function runE2e(
+  tests: Array<{ file: string; test: E2eTest }>,
+  opts: { repeat?: number },
+  deps: RunnerDeps,
+): Promise<number> {
+  const report: CaseReport[] = [];
+  let code = 0;
+  for (const { file, test } of tests) {
+    deps.print(file);
+    const lacking = missing(test, deps.appPackages);
+    if (lacking !== undefined) {
+      deps.print(`  ${lacking}`);
+      code = 1;
+      continue;
+    }
+    for (const c of test.cases) {
+      const result = await runCase(c, opts.repeat, deps);
+      report.push({ file, case: c.name, ...result });
+      if (!result.passed) code = 1;
+    }
+  }
+  await deps.writeReport(report);
+  return code;
+}
+
+/** Runs one case as many times as it repeats, printing each run's checks and the verdict. */
+async function runCase(c: E2eCase, repeatOverride: number | undefined, deps: RunnerDeps): Promise<{ passed: boolean; runs: RunReport[] }> {
+  const repeat = repeatOverride ?? c.repeat ?? 1;
+  const minPasses = Math.min(c.minPasses ?? repeat, repeat);
+  deps.print(`  ${c.name}`);
+  const runs: RunReport[] = [];
+  for (let i = 1; i <= repeat; i++) {
+    if (repeat > 1) deps.print(`    run ${i}/${repeat}`);
+    const run = await runOnce(c, deps);
+    const indent = repeat > 1 ? "      " : "    ";
+    for (const check of run.checks) {
+      deps.print(`${indent}${check.ok ? "✓" : "✗"} ${check.label}${check.ok || check.detail === undefined ? "" : `: ${check.detail}`}`);
+    }
+    runs.push(run);
+  }
+  const passes = runs.filter((r) => r.ok).length;
+  const passed = passes >= minPasses;
+  deps.print(
+    repeat > 1
+      ? `  ${c.name}: ${passes}/${repeat} runs passed (needs ${minPasses})${passed ? "" : " FAILED"}`
+      : `  ${c.name}: ${passed ? "passed" : "FAILED"}`,
+  );
+  return { passed, runs };
+}
+
+/** One run of a case in a fresh session: `before`, the turns, the checks, `after`. */
+async function runOnce(c: E2eCase, deps: RunnerDeps): Promise<RunReport> {
+  const ctx: Context = { cli: deps.cli };
+  const turns: Turn[] = [];
+  let checks: Check[] = [];
+  try {
+    await c.before?.(ctx);
+    const session = await deps.openSession();
+    try {
+      for (const next of c.turns) {
+        const line = typeof next === "string" ? next : next(turns.at(-1)!);
+        if (line.includes("\n")) throw new Error("a turn must be one line (the REPL reads one line per turn)");
+        const start = deps.now();
+        const { dump, output } = await session.turn(line);
+        const seconds = (deps.now() - start) / 1000;
+        const data = turnFromDump(dump);
+        const answer = data.calls.length === 0 && data.answer === "" ? answerFromOutput(output) : data.answer;
+        turns.push({ calls: data.calls, answer, seconds });
+      }
+    } finally {
+      await session.close();
+    }
+    const run: Run = { turns, last: turns.at(-1)! };
+    const recorder = createExpect(run);
+    try {
+      await c.check(run, recorder.expect, ctx);
+      checks = recorder.checks();
+      // A case that checks nothing proves nothing.
+      if (checks.length === 0) checks = [{ label: "the case made no checks", ok: false }];
+    } catch (err) {
+      checks = [...recorder.checks(), { label: `check threw: ${err instanceof Error ? err.message : String(err)}`, ok: false }];
+    }
+  } catch (err) {
+    checks = [{ label: `run failed: ${err instanceof Error ? err.message : String(err)}`, ok: false }];
+  } finally {
+    try {
+      await c.after?.(ctx);
+    } catch (err) {
+      checks = [...checks, { label: `after threw: ${err instanceof Error ? err.message : String(err)}`, ok: false }];
+    }
+  }
+  return { ok: checks.every((ch) => ch.ok), turns, checks };
+}

package/src/e2e/session.ts

@@ -0,0 +1,104 @@
+/**
+ * The REPL session `mfw e2e` drives: the app's REPL as a child process (in
+ * the app's container, through `docker compose run`), fed one turn at a time.
+ * A turn writes its line and then `/dump <file>` together: the REPL handles
+ * one line after the other, so the dump runs once the turn is over, and its
+ * "wrote … to <file>" line is the turn's end, unlike the prompt, which an
+ * answer's own text could look like. The dump is read from the host side of
+ * the folder the REPL writes it into.
+ */
+import { readFileSync } from "node:fs";
+import { join } from "node:path";
+import type { Session } from "./runner.ts";
+
+export type ReplSessionOptions = {
+  /** The command that starts the REPL. */
+  argv: string[];
+  cwd: string;
+  /** Where the dumps are, on this side, and as the REPL sees the same folder. */
+  hostDir: string;
+  replDir: string;
+  /** Prefix of this session's dump files, unique among sessions sharing the folder. */
+  name: string;
+  /** How long one turn may take. */
+  timeoutMs: number;
+};
+
+/** Starts the REPL and returns the session over it once the REPL is ready
+ * (its first prompt is out), so a turn's time is only the turn's; throws
+ * when the REPL exits or doesn't get there in time. */
+export async function openReplSession(opts: ReplSessionOptions): Promise<Session> {
+  const proc = Bun.spawn(opts.argv, { cwd: opts.cwd, stdin: "pipe", stdout: "pipe", stderr: "pipe" });
+  let stdout = "";
+  let stderr = "";
+  let exited: number | undefined;
+  /** Wakes whoever waits for more output or for the exit. */
+  let notify = () => {};
+  const pump = async (stream: ReadableStream<Uint8Array>, add: (text: string) => void) => {
+    const decoder = new TextDecoder();
+    for await (const chunk of stream) {
+      add(decoder.decode(chunk, { stream: true }));
+      notify();
+    }
+  };
+  void pump(proc.stdout, (t) => (stdout += t));
+  void pump(proc.stderr, (t) => (stderr += t));
+  void proc.exited.then((code) => {
+    exited = code;
+    notify();
+  });
+
+  /** Waits until `done` holds on the output so far; throws when the REPL exits or time runs out. */
+  const waitFor = async <T>(done: () => T | undefined): Promise<T> => {
+    const deadline = Date.now() + opts.timeoutMs;
+    for (;;) {
+      const result = done();
+      if (result !== undefined) return result;
+      if (exited !== undefined) throw new Error(`the REPL exited with code ${exited}: ${stderr.trim().split("\n").slice(-3).join(" ")}`);
+      const left = deadline - Date.now();
+      if (left <= 0) throw new Error(`no reply within ${opts.timeoutMs / 1000} s`);
+      await new Promise<void>((resolve) => {
+        const timer = setTimeout(resolve, left);
+        notify = () => {
+          clearTimeout(timer);
+          resolve();
+        };
+      });
+    }
+  };
+
+  // What the REPL prints while starting isn't part of any turn. One that
+  // doesn't get there is stopped: a docker compose run container otherwise
+  // keeps running.
+  try {
+    await waitFor(() => (/> $/.test(stdout) ? true : undefined));
+  } catch (err) {
+    proc.kill();
+    throw err;
+  }
+
+  let turns = 0;
+  return {
+    turn: async (line) => {
+      turns++;
+      const file = `${opts.name}-turn-${turns}.json`;
+      const marker = `to ${opts.replDir}/${file}`;
+      const from = stdout.length;
+      proc.stdin.write(`${line}\n/dump ${opts.replDir}/${file}\n`);
+      proc.stdin.flush();
+      return waitFor(() => {
+        const at = stdout.indexOf(marker, from);
+        if (at === -1) return undefined;
+        // What the turn printed: everything before the dump's own line.
+        const output = stdout.slice(from, stdout.lastIndexOf("wrote ", at));
+        return { dump: JSON.parse(readFileSync(join(opts.hostDir, file), "utf-8")) as unknown, output };
+      });
+    },
+    close: async () => {
+      if (exited !== undefined) return;
+      proc.stdin.end();
+      const done = await Promise.race([proc.exited, new Promise((r) => setTimeout(() => r("timeout"), 10_000))]);
+      if (done === "timeout") proc.kill();
+    },
+  };
+}

package/src/program.ts

@@ -24,12 +24,14 @@ export type ProgramHandlers = {
   app: () => AppCommands;
 };
 
-/** `--limit`'s value: a positive whole number. */
-function positiveInt(value: string): string {
-  if (!/^\d+$/.test(value) || Number(value) < 1) {
-    throw new InvalidArgumentError(`--limit takes a positive whole number (got "${value}").`);
-  }
-  return value;
+/** A parser for `flag`'s value: a positive whole number, kept as typed. */
+function positiveInt(flag: string): (value: string) => string {
+  return (value) => {
+    if (!/^\d+$/.test(value) || Number(value) < 1) {
+      throw new InvalidArgumentError(`${flag} takes a positive whole number (got "${value}").`);
+    }
+    return value;
+  };
 }
 
 /** The help's closing paragraph for the commands that run inside an app. */
@@ -199,7 +201,7 @@ Examples:
       "Prints a collection's points, each as its id and one line per payload field: newest first where the collection has a timestamp index, in Qdrant's own order otherwise.",
     )
     .argument("<collection>", "as mfw memory list prints it")
-    .option("--limit <n>", "how many points (default: 20)", positiveInt)
+    .option("--limit <n>", "how many points (default: 20)", positiveInt("--limit"))
     .action(async (collection: string, opts: { limit?: string }) =>
       inApp((app) => app.memory(["read", collection, ...(opts.limit === undefined ? [] : ["--limit", opts.limit])]))(),
     );
@@ -267,6 +269,33 @@ Examples:
       )(),
     );
 
+  program
+    .command("local-packages")
+    .summary("installs @mercury-fw packages from local tarballs")
+    .description(
+      "Makes the app install the packages in <folder>'s tarballs (bun pm pack) instead of the registry's: copies them into .packs/ (the image copies it too), points each package at its tarball with overrides in package.json, so packages that depend on it get it as well, and runs bun install. Run it again after packing anew. --off removes them and installs from the registry.",
+    )
+    .argument("[folder]", "the folder holding the .tgz files")
+    .option("--off", "go back to the registry's packages")
+    .addHelpText("after", `\nExamples:\n  mfw local-packages ../mercury-fw/apps/testbed/.packs\n  mfw local-packages --off${INSIDE_AN_APP}`)
+    .action(async (folder: string | undefined, opts: { off?: boolean }) => {
+      if ((folder === undefined) === (opts.off !== true)) throw new Error("local-packages takes a folder of tarballs, or --off");
+      await inApp((app) => (folder === undefined ? app.localPackagesOff() : app.localPackages(folder)))();
+    });
+
+  program
+    .command("e2e")
+    .summary("runs the app's end-to-end tests against its model")
+    .description(
+      "Runs e2e tests: each case's turns go to the app's real model through its REPL (in the container, like mfw repl), and its checks run on the tool calls each turn made and the answer it gave. Without files, every e2e/*.e2e.ts in the app. Prints every check, exits 1 when a case didn't pass enough runs, and keeps the turns and checks in e2e/results/<time>/. The app must be built and its env file filled in, as for mfw repl.",
+    )
+    .argument("[tests...]", "test files (default: e2e/*.e2e.ts)")
+    .option("--repeat <n>", "runs of each case, overriding its own repeat", positiveInt("--repeat"))
+    .addHelpText("after", `\nExamples:\n  mfw e2e\n  mfw e2e e2e/jira.e2e.ts --repeat 3${INSIDE_AN_APP}`)
+    .action(async (tests: string[], opts: { repeat?: string }) =>
+      inApp((app) => app.e2e(tests, opts.repeat === undefined ? {} : { repeat: Number(opts.repeat) }))(),
+    );
+
   return program;
 }
 

package/template/Dockerfile.tpl

@@ -12,8 +12,11 @@ RUN groupadd -r mercury && useradd -r -g mercury mercury
 WORKDIR /app
 
 # Each tool plugin downloads its pinned CLI in its postinstall (allowed by
-# package.json's trustedDependencies); the binaries go on PATH.
+# package.json's trustedDependencies); the binaries go on PATH. .packs/ holds
+# local tarballs when the app installs them (mfw local-packages), and is
+# skipped when it isn't there.
 COPY --chown=mercury:mercury package.json bun.lock* ./
+COPY --chown=mercury:mercury .pack[s] ./.packs/
 RUN bun install --production && chown -R mercury:mercury node_modules
 RUN find /app/node_modules -path '*/@mercury-fw/*/bin/*' -type f -exec ln -sf {} /usr/local/bin/ \;
 

package/template/gitignore.tpl

@@ -4,3 +4,5 @@ node_modules
 !.env.example
 *.log
 .DS_Store
+.packs/
+e2e/results/