@f-o-h/cli 0.1.8 → 0.1.10

package/README.md

@@ -1,175 +1,184 @@
-# Front Of House CLI
-
-AI-operator provisioning CLI for Front Of House.
-
-Public mirror: https://github.com/iiko38/front-of-house-cli
-
-Current published baseline: `@f-o-h/cli@0.1.8`
-
-This mirror is a generated release artifact. The private product monorepo is not
-published here, and no open-source license is granted unless stated separately.
-
-Package-local examples and schemas ship with the npm artifact:
-
-- `examples/scenario-suite.viewing.yml`
-- `examples/proof-report.example.json`
-- `examples/transcript-export.example.json`
-- `examples/improvement-packet.example.json`
-- `examples/external-agent-run.example.json`
-- `schemas/cli-envelope.schema.json`
-- `schemas/scenario-suite.schema.json`
-- `schemas/transcript-export.schema.json`
-- `schemas/improvement-packet.schema.json`
-- `schemas/external-agent-run.schema.json`
-
-## Install
-
-```bash
-npx @f-o-h/cli setup
-```
-
-Or install globally:
-
-```bash
-npm install -g @f-o-h/cli
-foh --help
-```
-
-Verify the package version:
-
-```bash
-npx @f-o-h/cli --version
-```
-
-## First Run
-
-```bash
-foh auth signup --web
-foh auth login --web
-foh auth login
-foh org list
-foh org use --org <org-id>
-foh setup
-foh prove --agent <agent-id> --json
-```
-
-For AI agents and text-only terminals:
-
-```bash
-foh auth signup --web --json
-foh auth login --web --json
-foh auth login --email "$FOH_EMAIL" --password "$FOH_PASSWORD" --json
-foh org list --json
-foh org use --org <org-id> --json
-foh setup --org <org-id> --agent-template <template-id> --agent-name "Demo Agent" --json
-foh prove --agent <agent-id> --json --out foh-proof.json
-foh test run --suite ./suite.yml --agent <agent-id> --json --out foh-test-report.json
-foh agent replay --file ./transcript-export.json --json
-foh bug improve --from-file foh-proof.json --out foh-improvement.json --json
-```
-
-`auth signup --web` opens the console signup page when possible and always
-prints the fallback URL. `auth login --web` starts browser device
-authorization, opens `/cli-auth`, waits for console approval, and stores the
-returned short-lived token. Credential auth remains available as fallback.
-
-`foh prove` produces a compact signed proof report across auth, org context,
-agent validation, contact phone readiness, voice provider health, widget
-channel/embed readiness, widget smoke, and simulation certification. It is
-read-only by default; pass `--mutation-mode ensure` or `--repair` only when you
-explicitly want proof to ensure missing widget state. Use `--strict` in
-automation when holds should fail the command, and `--mission voice` or
-`--require-phone` when a voice/contact number is mandatory for the demo.
-
-The CLI defaults to the production API at `https://api.frontofhouse.okii.uk`.
-
-## External-Agent Eval Capture
-
+# Front Of House CLI
+
+AI-operator provisioning CLI for Front Of House.
+
+Public mirror: https://github.com/iiko38/front-of-house-cli
+
+Current published baseline: `@f-o-h/cli@0.1.10`
+
+This mirror is a generated release artifact. The private product monorepo is not
+published here, and no open-source license is granted unless stated separately.
+
+Package-local examples and schemas ship with the npm artifact:
+
+- `examples/scenario-suite.viewing.yml`
+- `examples/proof-report.example.json`
+- `examples/transcript-export.example.json`
+- `examples/improvement-packet.example.json`
+- `examples/external-agent-run.example.json`
+- `schemas/cli-envelope.schema.json`
+- `schemas/scenario-suite.schema.json`
+- `schemas/transcript-export.schema.json`
+- `schemas/improvement-packet.schema.json`
+- `schemas/external-agent-run.schema.json`
+
+## Install
+
+```bash
+npx @f-o-h/cli setup
+```
+
+Or install globally:
+
+```bash
+npm install -g @f-o-h/cli
+foh --help
+```
+
+Verify the package version:
+
+```bash
+npx @f-o-h/cli --version
+```
+
+## First Run
+
+```bash
+foh auth signup --web
+foh auth login --web
+foh auth login
+foh org list
+foh org use --org <org-id>
+foh setup
+foh prove --agent <agent-id> --json
+```
+
+For AI agents and text-only terminals:
+
+```bash
+foh auth signup --web --json
+foh auth login --web --json
+foh auth login --email "$FOH_EMAIL" --password "$FOH_PASSWORD" --json
+foh org list --json
+foh org use --org <org-id> --json
+foh setup --org <org-id> --agent-template <template-id> --agent-name "Demo Agent" --json
+foh prove --agent <agent-id> --json --out foh-proof.json
+foh test run --suite ./suite.yml --agent <agent-id> --json --out foh-test-report.json
+foh agent replay --file ./transcript-export.json --json
+foh bug improve --from-file foh-proof.json --out foh-improvement.json --json
+```
+
+`auth signup --web` opens the console signup page when possible and always
+prints the fallback URL. `auth login --web` starts browser device
+authorization, opens `/cli-auth`, waits for console approval, and stores the
+returned short-lived token. Credential auth remains available as fallback.
+
+`foh prove` produces a compact signed proof report across auth, org context,
+agent validation, contact phone readiness, voice provider health, widget
+channel/embed readiness, widget smoke, and simulation certification. It is
+read-only by default; pass `--mutation-mode ensure` or `--repair` only when you
+explicitly want proof to ensure missing widget state. Use `--strict` in
+automation when holds should fail the command, and `--mission voice` or
+`--require-phone` when a voice/contact number is mandatory for the demo.
+
+The CLI defaults to the production API at `https://api.frontofhouse.okii.uk`.
+
+## External-Agent Eval Capture
+
 Use this when testing whether a clean coding agent can start from public docs
 and the public npm package without private repo context:
 
 ```bash
-foh eval external-agent run \
-  --model-provider openai \
-  --model-name codex \
-  --prompt-version blank-setup.v1
-```
-
-The command writes a versioned prompt, launches an instrumented shell, captures
-FOH CLI commands into `commands.ndjson`, and finalizes `run.json` as an
-`external_agent_run.v1` artifact when the shell exits.
-
-## Local Scenario Suites
-
-`foh test run --suite <file>` runs deterministic widget-runtime checks for a
-specific agent. The suite format supports reply text checks plus structured
-runtime assertions for trace/correlation IDs, action or terminal state, latency,
-variables, tool calls, escalation/handoff, lead capture, and exact response
-field paths.
-
-```yaml
-agent: agent_123
-scenarios:
-  - id: viewing
-    turns:
-      - user: Can I book a viewing this week?
-        expect:
-          contains: viewing
-          trace_present: true
-          correlation_present: true
-          action: text
-          latency_ms:
-            max: 3000
-```
-
-Use transcript fixtures when turning real user conversations into regression
-tests:
-
-```yaml
-agent: agent_123
-scenarios:
-  - id: replay-viewing
-    fixture_transcript: ./fixtures/viewing-transcript.json
-```
-
-## Transcript Export
-
-Use hydrated transcript export to turn real behavior into replay/debug artifacts:
-
-```bash
-foh transcripts export \
-  --agent <agent-id> \
-  --hydrate \
-  --include-traces \
-  --format json \
-  --out foh-transcripts.json \
+foh eval external-agent batch \
+  --models openai/codex,anthropic/claude,cursor/agent \
+  --prompt-version blank-setup.v1 \
   --json
 ```
 
-Exports redact obvious emails, phone numbers, and secret-like tokens by default.
-Each exported conversation includes a `replay_command` and `test_fixture` seed
-so operators or AI agents can move from observed failure to replay or scenario
-regression without opening the console.
-
-Replay a local export without API access:
-
-```bash
-foh agent replay --file foh-transcripts.json --json
-```
-
-## Improvement Packets
-
-Use `foh bug improve` when a setup, proof, replay, knowledge, runtime, or
-live-proof failure should become actionable backlog/test/config/docs work:
+Run each returned launch command in a clean agent terminal:
 
 ```bash
-foh bug improve \
-  --from-file test-results/proof-or-replay-failure.json \
-  --out test-results/improvement-packet.json \
-  --json
-```
-
-The command emits a redacted `foh_improvement_packet.v1` with stable IDs,
-reason code, promotion decision, evidence summary, and deterministic next
-commands.
+foh eval external-agent run \
+  --model-provider openai \
+  --model-name codex \
+  --prompt-version blank-setup.v1
+```
+
+The command writes a versioned prompt, launches an instrumented shell, captures
+FOH CLI commands into `commands.ndjson`, and finalizes `run.json` as an
+`external_agent_run.v1` artifact when the shell exits.
+
+## Local Scenario Suites
+
+`foh test run --suite <file>` runs deterministic widget-runtime checks for a
+specific agent. The suite format supports reply text checks plus structured
+runtime assertions for trace/correlation IDs, action or terminal state, latency,
+variables, tool calls, escalation/handoff, lead capture, and exact response
+field paths.
+
+```yaml
+agent: agent_123
+scenarios:
+  - id: viewing
+    turns:
+      - user: Can I book a viewing this week?
+        expect:
+          contains: viewing
+          trace_present: true
+          correlation_present: true
+          action: text
+          latency_ms:
+            max: 3000
+```
+
+Use transcript fixtures when turning real user conversations into regression
+tests:
+
+```yaml
+agent: agent_123
+scenarios:
+  - id: replay-viewing
+    fixture_transcript: ./fixtures/viewing-transcript.json
+```
+
+## Transcript Export
+
+Use hydrated transcript export to turn real behavior into replay/debug artifacts:
+
+```bash
+foh transcripts export \
+  --agent <agent-id> \
+  --hydrate \
+  --include-traces \
+  --format json \
+  --out foh-transcripts.json \
+  --json
+```
+
+Exports redact obvious emails, phone numbers, and secret-like tokens by default.
+Each exported conversation includes a `replay_command` and `test_fixture` seed
+so operators or AI agents can move from observed failure to replay or scenario
+regression without opening the console.
+
+Replay a local export without API access:
+
+```bash
+foh agent replay --file foh-transcripts.json --json
+```
+
+## Improvement Packets
+
+Use `foh bug improve` when a setup, proof, replay, knowledge, runtime, or
+live-proof failure should become actionable backlog/test/config/docs work:
+
+```bash
+foh bug improve \
+  --from-file test-results/proof-or-replay-failure.json \
+  --out test-results/improvement-packet.json \
+  --json
+```
+
+The command emits a redacted `foh_improvement_packet.v1` with stable IDs,
+reason code, promotion decision, evidence summary, and deterministic next
+commands.
 

package/dist/foh.js

@@ -32640,7 +32640,7 @@ var StdioServerTransport = class {
 };
 
 // src/lib/cli-version.ts
-var CLI_VERSION = "0.1.8";
+var CLI_VERSION = "0.1.10";
 
 // src/commands/mcp-serve.ts
 var DEFAULT_TIMEOUT_MS = 12e4;
@@ -38267,6 +38267,12 @@ var SECRET_RE2 = /\b(?:Bearer\s+)?(?:sk|pk|xai|whsec|EAAN|ghp|gho|github_pat|npm
 function redactText(value) {
   return value.replace(SECRET_RE2, "[redacted_secret]");
 }
+function redactPath(value) {
+  let redacted = redactText(value);
+  const home = process.env.USERPROFILE || process.env.HOME;
+  if (home) redacted = redacted.replace(home, "~");
+  return redacted;
+}
 function safeJsonLine(value) {
   return JSON.stringify(value).replace(/\r?\n/g, " ") + "\n";
 }
@@ -38285,7 +38291,7 @@ function recordExternalAgentCliInvocation(input) {
       schema_version: "external_agent_cli_command.v1",
       recorded_at: (/* @__PURE__ */ new Date()).toISOString(),
       cli_version: input.cliVersion,
-      cwd: redactText(process.cwd()),
+      cwd: redactPath(process.cwd()),
       argv: args,
       command: args.join(" "),
       json_requested: args.includes("--json"),
@@ -38303,6 +38309,7 @@ function readCommandRecords(runDir) {
 
 // src/commands/eval.ts
 var DEFAULT_PROMPT_VERSION = "blank-setup.v1";
+var DEFAULT_BATCH_MODELS = "openai/codex,anthropic/claude,cursor/agent";
 var PROMPTS = {
   "blank-setup.v1": "Go to https://frontofhouse.okii.uk. Use only public docs, public API docs, and the public npm CLI package. Install the FOH CLI, authenticate or reach a deterministic auth blocker, create or configure a Front Of House voice agent and website widget, run proof/smoke/certification where available, and produce a final evidence summary with commands run, docs used, artifacts created, and any blocker reason codes. Do not assume access to the private source repository.",
   "debug-proof-failure.v1": "You are given a FOH proof or debug artifact. Use public docs and FOH CLI/API behavior to classify whether the blocker is docs, auth, org setup, agent config, widget, channel, runtime, or product bug. Produce a redacted improvement packet or the exact command needed to produce one. Do not ask the human to interpret logs manually unless no machine-readable artifact exists.",
@@ -38321,6 +38328,31 @@ function defaultRunDir(modelName, promptVersion) {
   const safePrompt = String(promptVersion || DEFAULT_PROMPT_VERSION).toLowerCase().replace(/[^a-z0-9_.-]+/g, "-");
   return (0, import_path11.resolve)("test-results", "external-agent-runs", date4, `${safeModel}-${safePrompt}-${stamp}`);
 }
+function defaultBatchDir(promptVersion) {
+  const date4 = (/* @__PURE__ */ new Date()).toISOString().slice(0, 10);
+  const stamp = (/* @__PURE__ */ new Date()).toISOString().replace(/[:.]/g, "-").replace("T", "-").slice(0, 23);
+  const safePrompt = String(promptVersion || DEFAULT_PROMPT_VERSION).toLowerCase().replace(/[^a-z0-9_.-]+/g, "-");
+  return (0, import_path11.resolve)("test-results", "external-agent-runs", date4, `batch-${safePrompt}-${stamp}`);
+}
+function safeSlug(value) {
+  return String(value || "unknown").toLowerCase().replace(/[^a-z0-9_.-]+/g, "-").replace(/^-+|-+$/g, "") || "unknown";
+}
+function quoteArg(value) {
+  const text = String(value);
+  if (/^[A-Za-z0-9_./:=@-]+$/.test(text)) return text;
+  return `"${text.replace(/(["$`])/g, "\\$1")}"`;
+}
+function parseModelSpec(raw) {
+  const [provider, ...nameParts] = String(raw || "").split("/");
+  const name = nameParts.join("/");
+  return {
+    provider: provider?.trim() || "unknown",
+    name: name.trim() || "unknown-model"
+  };
+}
+function parseModelList(raw) {
+  return String(raw || DEFAULT_BATCH_MODELS).split(",").map((entry) => entry.trim()).filter(Boolean).map(parseModelSpec);
+}
 function inferShell(raw) {
   if (raw && raw.trim()) return { command: raw, args: [], label: raw };
   if (process.platform === "win32") return { command: "powershell.exe", args: ["-NoLogo", "-NoProfile"], label: "powershell" };
@@ -38394,6 +38426,73 @@ function buildRunArtifact(input) {
 function registerEval(program3) {
   const evalCommand = program3.command("eval").description("Run or summarize external-agent evaluation workflows");
   const external = evalCommand.command("external-agent").description("Capture clean external coding-agent setup attempts");
+  external.command("batch").description("Create a deterministic multi-model external-agent batch plan").option("--models <list>", "Comma-separated provider/model list", DEFAULT_BATCH_MODELS).option("--prompt-version <version>", "Prompt version", DEFAULT_PROMPT_VERSION).option("--workspace-type <type>", "Workspace type label", "clean-no-repo").option("--agent-shell <name>", "Agent shell label", "vscode-terminal").option("--out-dir <path>", "Batch output directory").option("--json", "Output as JSON").action(async (opts) => {
+    const promptVersion = String(opts.promptVersion || DEFAULT_PROMPT_VERSION);
+    const batchDir = (0, import_path11.resolve)(String(opts.outDir || defaultBatchDir(promptVersion)));
+    const models = parseModelList(String(opts.models || DEFAULT_BATCH_MODELS));
+    (0, import_fs13.mkdirSync)(batchDir, { recursive: true });
+    const runs = models.map((model, index) => {
+      const runId = `${String(index + 1).padStart(2, "0")}-${safeSlug(model.provider)}-${safeSlug(model.name)}`;
+      const runDir = (0, import_path11.join)(batchDir, runId);
+      (0, import_fs13.mkdirSync)(runDir, { recursive: true });
+      const promptPath = writePrompt(runDir, promptVersion);
+      const commandArgs = [
+        "eval",
+        "external-agent",
+        "run",
+        "--model-provider",
+        model.provider,
+        "--model-name",
+        model.name,
+        "--prompt-version",
+        promptVersion,
+        "--workspace-type",
+        String(opts.workspaceType || "clean-no-repo"),
+        "--agent-shell",
+        String(opts.agentShell || "vscode-terminal"),
+        "--out-dir",
+        runDir
+      ];
+      return {
+        run_id: runId,
+        model_provider: model.provider,
+        model_name: model.name,
+        prompt_version: promptVersion,
+        run_dir: runDir,
+        prompt_path: promptPath,
+        launch_args: commandArgs,
+        launch_command: `npx --yes @f-o-h/cli@latest ${commandArgs.map(quoteArg).join(" ")}`
+      };
+    });
+    const batch = {
+      schema_version: "external_agent_batch_plan.v1",
+      created_at: (/* @__PURE__ */ new Date()).toISOString(),
+      batch_dir: batchDir,
+      prompt_version: promptVersion,
+      workspace_type: String(opts.workspaceType || "clean-no-repo"),
+      agent_shell: String(opts.agentShell || "vscode-terminal"),
+      run_count: runs.length,
+      runs,
+      summary_command: `corepack pnpm eval:external-agent:runs:summary -- --root ${batchDir}`
+    };
+    const batchPath = (0, import_path11.join)(batchDir, "batch.json");
+    (0, import_fs13.writeFileSync)(batchPath, `${JSON.stringify(batch, null, 2)}
+`, "utf8");
+    format(cliEnvelope({
+      schemaVersion: "external_agent_batch_plan_result.v1",
+      status: "exported",
+      reasonCode: "external_agent_batch_plan_created",
+      summary: `External-agent batch plan created for ${runs.length} model(s).`,
+      artifacts: {
+        batch: batchPath
+      },
+      nextCommands: [
+        ...runs.map((run) => run.launch_command),
+        batch.summary_command
+      ],
+      extra: { batch }
+    }), { json: Boolean(opts.json) });
+  });
   external.command("run").description("Launch an instrumented shell and emit external_agent_run.v1 when it exits").option("--model-provider <name>", "Model provider label", "unknown").option("--model-name <name>", "Model name label", "unknown-model").option("--prompt-version <version>", "Prompt version", DEFAULT_PROMPT_VERSION).option("--workspace-type <type>", "Workspace type label", "clean-no-repo").option("--agent-shell <name>", "Agent shell label", "vscode-terminal").option("--out-dir <path>", "Run output directory").option("--status <status>", "Final status when not interactively classified: pass|hold|fail", "hold").option("--reason-code <code>", "Failure/hold reason code", "external_agent_run_needs_review").option("--shell <command>", "Shell command to launch for capture").option("--no-shell", "Do not launch a shell; create/finalize artifacts immediately").option("--json", "Output as JSON").action(async (opts) => {
     const status = normalizeStatus(opts.status);
     const promptVersion = String(opts.promptVersion || DEFAULT_PROMPT_VERSION);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@f-o-h/cli",
-  "version": "0.1.8",
+  "version": "0.1.10",
   "description": "FOH CLI - AI-operator provisioning tool for Front Of House",
   "license": "UNLICENSED",
   "bin": {