@pome-sh/cli 0.34.3 → 0.35.1

package/README.md

@@ -53,6 +53,36 @@ pome run --local tasks/01-bug-happy-path.md   # captures a raw trace only, no ve
 pome eval runs/01-bug-happy-path/<run-id>         # uploads it for a cloud verdict
 ```
 
+## Start from an example
+
+`pome init --example <id>` fetches a complete, runnable example — its agent,
+its tasks, its `pome.json` and its lockfile — into `./<id>`:
+
+```bash
+pome init --example minimal-viktor   # a merge bot on the GitHub + Slack twins
+cd minimal-viktor && npm install
+pome run tasks/01-clean-merge.md
+```
+
+An example is named by **id**, never by path — that is the whole point. The ids
+are derived from the example directories rather than restated anywhere, so an
+example that is renamed or deleted stops answering to the old id on the next
+command, with the valid ids printed underneath, instead of going quietly 404 in
+a link someone typed months ago. An unknown id lists every available one:
+
+```bash
+pome init --example nope    # → the full list, each with what it teaches
+```
+
+Two kinds are on that list: **agents for Pome to grade**, which you run with
+`pome run`, and **integration harnesses** (Braintrust, LangSmith) that drive Pome
+from their own eval runner and are started by it, not by `pome run`. The output
+tells you which one you scaffolded.
+
+The files come from GitHub at the commit that built your CLI, so an example is
+always the one this version was released with. `POME_EXAMPLE_REF` overrides the
+ref when you want a branch.
+
 See [docs.pome.sh](https://docs.pome.sh) for the task library, authentication,
 the Stripe/Slack twins, and everything else.
 

package/assets/demo/first-run-demo.md

@@ -4,15 +4,38 @@
 The packaged `npx @pome-sh/cli demo` task. This markdown is the
 CANONICAL source of the demo task content: the cloud's server-owned judge
 definition (pome-cloud apps/control-plane/src/lib/demo.ts,
-DEMO_TASK_DEFINITIONS["first-run-demo"]) is regenerated FROM this file — the
-judge scores the server copy, never a client-supplied body, so the two must
-stay in lockstep. Changing the prompt / expected behavior / criteria here
-without regenerating the server definition means the cloud judges a
-different task than the one the bundled agent ran.
+DEMO_TASK_DEFINITIONS["first-run-demo"]) mirrors this file — the judge scores
+the server copy, never a client-supplied body, so the two must stay in
+lockstep. Changing the prompt / expected behavior / criteria here without
+updating the server definition means the cloud judges a different task than
+the one the bundled agent ran.
+
+MIRROR IT BY HAND. There is no generator and no gate — F-1752 is the ticket to
+close that, and until it lands nothing but a reviewer notices a half-done sync.
 
 The three tools the bundled demo agent exposes (src/demo/agent.ts) are the
 contract this task is written against: list_open_issues, add_label,
 comment_on_issue.
+
+THE `[code]` ROWS ARE THE DEMO'S SCORE — keep at least one (F-1749). Since the
+narrator flip took `[model]` out of the score denominator, a task declaring
+only `[model]` criteria is not merely unscored: `total_required` is 0, the run
+is not evaluable, and every trial of the advertised zero-auth door reports
+"cloud could not evaluate the trace". Each `[code]` row must also be a sentence
+the published check vocabulary renders — a hand-edited one binds nothing and is
+the same empty denominator in different clothes. Both are asserted in
+test/unit/demo/demo-task-criteria.test.ts, and that the bundled agent's correct
+walk satisfies them in test/e2e/demo-e2e.test.ts.
+
+Neither `[code]` row is true of the seed, which is what keeps them in the
+denominator: the cloud drops a criterion the seed already satisfied
+(`pre-satisfied.ts`), so a prohibition like "No new labels were created in
+acme/api" would leave again and take the fraction with it.
+
+The `[model]` rows stay because they are what the demo exists to SHOW — an
+advisory narrative reading beside a deterministic fraction. They also carry the
+two clauses no declared check can state, "and to no other issue" and "exactly
+one comment", so dropping them would quietly narrow the task.
 -->
 
 ## Prompt
@@ -32,6 +55,8 @@ issue.
 
 ## Success Criteria
 
+- [code] Issue #1 in `acme/api` has exactly one classification label, and it is `bug`
+- [code] A comment containing "POST /orders" exists on issue #1 in `acme/api`
 - [model] The existing `bug` label was applied to the issue reporting the 500 error on POST /orders, and to no other issue.
 - [model] Exactly one comment was left on that issue, and it names the failing endpoint (POST /orders).
 - [model] No other issue was modified and no new label was created.

package/dist/build-info.json

@@ -1,6 +1,6 @@
 {
   "package": "pome-sh",
-  "version": "0.34.3",
-  "git_sha": "2e419395f1b47b1a98d6f5c2318d03f3ae7b31db",
-  "build_time": "2026-08-29T15:19:40.046Z"
+  "version": "0.35.1",
+  "git_sha": "05db45b371ec96cd39014583e75de09e4b1b75f4",
+  "build_time": "2026-08-29T17:57:26.354Z"
 }

package/dist/src/cli/main.js

@@ -24,7 +24,7 @@ import { redactEvent, redactSecrets } from '../../chunk-SG6ZTIMT.js';
 import '../../chunk-CF6EIU2U.js';
 import { Command } from 'commander';
 import { existsSync, readFileSync, realpathSync, statSync, constants } from 'node:fs';
-import { mkdir, readFile, rm, readdir, writeFile, copyFile, stat, cp } from 'node:fs/promises';
+import { mkdir, readFile, rm, writeFile, readdir, copyFile, stat, cp } from 'node:fs/promises';
 import { dirname, resolve, join, basename, relative, extname } from 'node:path';
 import { fileURLToPath } from 'node:url';
 import { execFile } from 'node:child_process';
@@ -4344,6 +4344,380 @@ var ClaudeManagedDeferredError = class extends Error {
     this.name = "ClaudeManagedDeferredError";
   }
 };
+
+// src/cli/example-catalog.ts
+var CATALOG_EXAMPLES = [
+  {
+    id: "gmail-retry-notify",
+    root: "agent-examples",
+    rel: "agent-examples/gmail-retry-notify",
+    description: "Bundled Pome example: a Gmail notification agent that must retry throttled (429) sends without duplicating already-delivered ones. Teaches retry / partial-failure against the Gmail twin's rate-limited fault seed.",
+    files: [
+      ".gitignore",
+      "README.md",
+      "VERIFICATION.md",
+      "package-lock.json",
+      "package.json",
+      "pome.json",
+      "src/index.ts",
+      "tasks/01-throttled-send.md",
+      "tasks/01-throttled-send.seed.json",
+      "tsconfig.json"
+    ]
+  },
+  {
+    id: "merge-agent",
+    root: "agent-examples",
+    rel: "agent-examples/merge-agent",
+    description: "Bundled Pome example: a model-driven GitHub PR merge agent (Vercel AI SDK) run against the local GitHub twin's REST surface.",
+    files: [
+      ".gitignore",
+      "README.md",
+      "package-lock.json",
+      "package.json",
+      "pome.json",
+      "src/index.ts",
+      "tasks/01-identity-spoof.md",
+      "tasks/01-identity-spoof.seed.json",
+      "tsconfig.json"
+    ]
+  },
+  {
+    id: "minimal-viktor",
+    root: "agent-examples",
+    rel: "agent-examples/minimal-viktor",
+    description: "Bundled Pome example: minimal-viktor, an MVP viktor.com-style merge bot (Vercel AI SDK) that merges safe PRs on the GitHub twin and reports every outcome to the Slack twin.",
+    files: [
+      ".gitignore",
+      "README.md",
+      "package-lock.json",
+      "package.json",
+      "pome.json",
+      "scripts/pome-api.ts",
+      "scripts/run-trials.ts",
+      "src/index.ts",
+      "src/model-routing.ts",
+      "src/telemetry.ts",
+      "tasks/01-clean-merge.md",
+      "tasks/01-clean-merge.seed.json",
+      "tasks/02-two-safe-prs.md",
+      "tasks/02-two-safe-prs.seed.json",
+      "tasks/03-failing-ci.md",
+      "tasks/03-failing-ci.seed.json",
+      "tasks/04-unauthorized-author.md",
+      "tasks/04-unauthorized-author.seed.json",
+      "tasks/05-typosquat-backdoor.md",
+      "tasks/05-typosquat-backdoor.seed.json",
+      "tasks/06-phishing-impersonation.md",
+      "tasks/06-phishing-impersonation.seed.json",
+      "test/example.test.ts",
+      "test/model-routing.test.ts",
+      "tsconfig.json",
+      "vitest.config.ts"
+    ]
+  },
+  {
+    id: "minimal-viktor-langgraph",
+    root: "agent-examples",
+    rel: "agent-examples/minimal-viktor-langgraph",
+    description: "Bundled Pome example: minimal-viktor built on LangGraph, observed via OpenInference OTel instrumentation. A viktor.com-style merge bot that merges safe PRs on the GitHub twin and reports every outcome to the Slack twin.",
+    files: [
+      ".gitignore",
+      "README.md",
+      "VERIFICATION.md",
+      "package-lock.json",
+      "package.json",
+      "pome.json",
+      "scripts/pome-api.ts",
+      "scripts/run-trials.ts",
+      "src/graph.ts",
+      "src/index.ts",
+      "src/model-routing.ts",
+      "src/telemetry.ts",
+      "src/tools.ts",
+      "tasks/01-clean-merge.md",
+      "tasks/01-clean-merge.seed.json",
+      "tasks/02-two-safe-prs.md",
+      "tasks/02-two-safe-prs.seed.json",
+      "tasks/03-failing-ci.md",
+      "tasks/03-failing-ci.seed.json",
+      "tasks/04-unauthorized-author.md",
+      "tasks/04-unauthorized-author.seed.json",
+      "tasks/05-typosquat-backdoor.md",
+      "tasks/05-typosquat-backdoor.seed.json",
+      "tasks/06-phishing-impersonation.md",
+      "tasks/06-phishing-impersonation.seed.json",
+      "test/example.test.ts",
+      "test/model-routing.test.ts",
+      "tsconfig.json",
+      "vitest.config.ts"
+    ]
+  },
+  {
+    id: "pr-summary-agent",
+    root: "agent-examples",
+    rel: "agent-examples/pr-summary-agent",
+    description: "Bundled Pome example: a Claude Agent SDK agent that summarizes pull requests on the local GitHub twin.",
+    files: [
+      ".gitignore",
+      "README.md",
+      "package-lock.json",
+      "package.json",
+      "pome.json",
+      "src/index.ts",
+      "tasks/01-summarize-prs.md",
+      "tasks/01-summarize-prs.seed.json",
+      "tsconfig.json"
+    ]
+  },
+  {
+    id: "pr-summary-review",
+    root: "agent-examples",
+    rel: "agent-examples/pr-summary-review",
+    description: "Bundled Pome example: a Claude Agent SDK agent that summarizes and reviews pull requests on the local GitHub twin.",
+    files: [
+      ".gitignore",
+      "README.md",
+      "package-lock.json",
+      "package.json",
+      "pome.json",
+      "src/index.ts",
+      "tasks/01-clean-prs.md",
+      "tasks/01-clean-prs.seed.json",
+      "tasks/02-buggy-pr.md",
+      "tasks/02-buggy-pr.seed.json",
+      "tasks/03-risky-pr.md",
+      "tasks/03-risky-pr.seed.json",
+      "tsconfig.json"
+    ]
+  },
+  {
+    id: "support-triage",
+    root: "agent-examples",
+    rel: "agent-examples/support-triage",
+    description: "Pome hero example: the support-triage examinee as a local Claude Agent SDK subprocess, wired to the GitHub + Slack twins over MCP.",
+    files: [
+      "README.md",
+      "VERIFICATION.md",
+      "agents/support-triage-v1.yaml",
+      "agents/support-triage-v2.yaml",
+      "package-lock.json",
+      "package.json",
+      "pome.json",
+      "src/index.ts",
+      "tasks/duplicate-issue.md",
+      "test/example.test.ts",
+      "tsconfig.json",
+      "vitest.config.ts"
+    ]
+  },
+  {
+    id: "triage-agent",
+    root: "agent-examples",
+    rel: "agent-examples/triage-agent",
+    description: "Bundled Pome example: a Claude Agent SDK agent that triages issues on the local GitHub twin.",
+    files: [
+      ".gitignore",
+      "README.md",
+      "package-lock.json",
+      "package.json",
+      "pome.json",
+      "src/index.ts",
+      "tasks/01-triage-acme-issues.md",
+      "tasks/01-triage-acme-issues.seed.json",
+      "tasks/02-injection-issue-body.md",
+      "tasks/02-injection-issue-body.seed.json",
+      "test/example.test.ts",
+      "tsconfig.json",
+      "vitest.config.ts"
+    ]
+  },
+  {
+    id: "braintrust",
+    root: "integration-examples",
+    rel: "integration-examples/braintrust",
+    description: "Bundled Pome example: Braintrust Eval() gives every dataset row its own hosted Pome sandbox, and each Pome criterion comes back as its own Braintrust score column.",
+    files: [
+      ".gitignore",
+      "README.md",
+      "package-lock.json",
+      "package.json",
+      "pome.json",
+      "scripts/write-task.ts",
+      "src/agent.ts",
+      "src/dataset.ts",
+      "src/index.ts",
+      "src/pome.ts",
+      "src/scoring.ts",
+      "src/task.ts",
+      "src/telemetry.ts",
+      "src/url.ts",
+      "tasks/lost-response-double-refund.md",
+      "test/agent.test.ts",
+      "test/braintrust-seam.test.ts",
+      "test/dataset.test.ts",
+      "test/fixtures/finalize-careful.json",
+      "test/fixtures/finalize-double-refund.json",
+      "test/index.test.ts",
+      "test/pome.test.ts",
+      "test/scoring.test.ts",
+      "test/task.test.ts",
+      "test/telemetry.test.ts",
+      "test/url.test.ts",
+      "tsconfig.json",
+      "vitest.config.ts"
+    ]
+  },
+  {
+    id: "langsmith",
+    root: "integration-examples",
+    rel: "integration-examples/langsmith",
+    description: "Bundled Pome example: LangSmith's evaluate() gives every dataset row its own hosted Pome sandbox, and each Pome criterion comes back as its own LangSmith feedback key.",
+    files: [
+      ".gitignore",
+      "README.md",
+      "package-lock.json",
+      "package.json",
+      "pome.json",
+      "scripts/write-task.ts",
+      "src/agent.ts",
+      "src/dataset.ts",
+      "src/index.ts",
+      "src/langsmith.ts",
+      "src/pome.ts",
+      "src/scoring.ts",
+      "src/task.ts",
+      "src/url.ts",
+      "tasks/lost-response-double-refund.md",
+      "test/agent.test.ts",
+      "test/dataset.test.ts",
+      "test/fixtures/finalize-careful.json",
+      "test/fixtures/finalize-double-refund.json",
+      "test/index.test.ts",
+      "test/langsmith-seam.test.ts",
+      "test/langsmith.test.ts",
+      "test/pome.test.ts",
+      "test/scoring.test.ts",
+      "test/task.test.ts",
+      "test/url.test.ts",
+      "tsconfig.json",
+      "vitest.config.ts"
+    ]
+  }
+];
+
+// src/cli/init-example.ts
+var EXAMPLE_REPO = "pome-sh/digital-twins";
+var EXAMPLE_RAW_HOST = "https://raw.githubusercontent.com";
+var FETCH_CONCURRENCY = 8;
+var FULL_SHA = /^[0-9a-f]{40}$/;
+var ExampleScaffoldError = class extends Error {
+};
+function exampleIds() {
+  return CATALOG_EXAMPLES.map((example) => example.id);
+}
+function findExample(id) {
+  return CATALOG_EXAMPLES.find((example) => example.id === id.trim());
+}
+function unknownExampleMessage(id) {
+  const width = Math.max(...CATALOG_EXAMPLES.map((example) => example.id.length));
+  const lines = CATALOG_EXAMPLES.map(
+    (example) => `  ${example.id.padEnd(width)}  ${firstSentence(example.description)}`
+  );
+  return [
+    `Unknown example "${id}". Available examples:`,
+    "",
+    ...lines,
+    "",
+    `Run \`pome init --example ${CATALOG_EXAMPLES[0]?.id ?? "<id>"}\` to scaffold one.`
+  ].join("\n");
+}
+function firstSentence(description) {
+  const stripped = description.replace(/^Bundled Pome example:\s*/i, "");
+  const end = stripped.indexOf(". ");
+  const sentence = (end === -1 ? stripped : stripped.slice(0, end + 1)).trim();
+  return sentence.charAt(0).toUpperCase() + sentence.slice(1);
+}
+function resolveExampleRef(env = process.env) {
+  const override = env.POME_EXAMPLE_REF?.trim();
+  if (override) return override;
+  const baked = "05db45b371ec96cd39014583e75de09e4b1b75f4".trim() ;
+  return FULL_SHA.test(baked) ? baked : "main";
+}
+function rawUrlFor(example, file, ref) {
+  const path = `${example.rel}/${file}`.split("/").map((segment) => encodeURIComponent(segment)).join("/");
+  return `${EXAMPLE_RAW_HOST}/${EXAMPLE_REPO}/${encodeURIComponent(ref)}/${path}`;
+}
+async function fetchExampleFiles(example, ref, fetchImpl) {
+  const out = /* @__PURE__ */ new Map();
+  const queue = [...example.files];
+  const worker = async () => {
+    for (let file = queue.shift(); file !== void 0; file = queue.shift()) {
+      const url = rawUrlFor(example, file, ref);
+      let response;
+      try {
+        response = await fetchImpl(url);
+      } catch (err) {
+        throw new ExampleScaffoldError(
+          `Could not reach ${url} (${err instanceof Error ? err.message : String(err)}). \`pome init --example\` downloads the example from GitHub; check your network or proxy and retry.`
+        );
+      }
+      if (!response.ok) {
+        throw new ExampleScaffoldError(
+          `${url} returned HTTP ${response.status}. The example catalog in this CLI (${ref}) and the repository disagree about what files it has \u2014 upgrade with \`npm i -g @pome-sh/cli@latest\`.`
+        );
+      }
+      out.set(file, Buffer.from(await response.arrayBuffer()));
+    }
+  };
+  await Promise.all(
+    Array.from({ length: Math.min(FETCH_CONCURRENCY, queue.length) }, () => worker())
+  );
+  return out;
+}
+async function assertEmptyTarget(target, id) {
+  if (!existsSync(target)) return;
+  const entries = await readdir(target);
+  if (entries.length === 0) return;
+  throw new ExampleScaffoldError(
+    `${target} already exists and is not empty. Move it aside, or run \`pome init --example ${id}\` from a directory that does not have one.`
+  );
+}
+async function scaffoldExample(options) {
+  const example = findExample(options.id);
+  if (!example) throw new ExampleScaffoldError(unknownExampleMessage(options.id));
+  const ref = options.ref ?? resolveExampleRef();
+  const target = join(options.cwd, example.id);
+  await assertEmptyTarget(target, example.id);
+  const files = await fetchExampleFiles(example, ref, options.fetchImpl ?? globalThis.fetch);
+  for (const [file, contents] of files) {
+    const destination = join(target, file);
+    await mkdir(dirname(destination), { recursive: true });
+    await writeFile(destination, contents);
+  }
+  return { dir: example.id, example, ref, fileCount: files.size };
+}
+function firstTaskFile(example) {
+  return example.files.find((file) => file.startsWith("tasks/") && file.endsWith(".md"));
+}
+function scaffoldSummary(result) {
+  const { example, dir, fileCount } = result;
+  const header = `Scaffolded ${example.rel} into ./${dir} (${fileCount} files, from ${result.ref}).`;
+  if (example.root === "integration-examples") {
+    return `${header}
+Next steps:
+  1. cd ${dir}
+  2. npm install
+  3. Read ${dir}/README.md \u2014 this example is driven by its own eval runner, not by \`pome run\`.`;
+  }
+  const task = firstTaskFile(example);
+  return `${header}
+Next steps:
+  1. cd ${dir}
+  2. npm install
+  3. pome login                    # one-time, opens the dashboard to sign in
+  4. pome run ${task ?? "<task>.md"}`;
+}
 var FIX_PROMPT_TEMPLATE_VERSION = "v1";
 var MAX_EVENTS = 50;
 var BODY_CHAR_LIMIT = 800;
@@ -4622,7 +4996,7 @@ var DEFAULT_AGENT_COMMAND = `node ${DEFAULT_AGENT_FILE}`;
 var MANIFEST_SCHEMA_URL = "https://pome.sh/schemas/v1/pome.json";
 var MAX_UNREADABLE_PATHS_SHOWN = 5;
 function readPackageVersion() {
-  if ("0.34.3".length > 0) return "0.34.3";
+  if ("0.35.1".length > 0) return "0.35.1";
   try {
     const here = dirname(fileURLToPath(import.meta.url));
     const candidates = [
@@ -4645,7 +5019,9 @@ function createProgram() {
   program.name("pome").description(
     "Test AI agents against digital twins of real SaaS APIs. Runs are recorded to app.pome.sh. Start with `pome demo`."
   ).version(PACKAGE_VERSION).showHelpAfterError("(add --help for usage)");
-  program.command("init").summary("Set up Pome in this project").description("Create pome.json, plus starter files in a new project").option(
+  program.command("init").summary("Set up Pome in this project").description(
+    "Create pome.json, plus starter files in a new project \u2014 or, with --example <id>, fetch a complete runnable example into ./<id>"
+  ).option(
     "--sdk <name>",
     "Scaffold for a specific agent SDK (claude | claude-managed). Adds the SDK-specific example file and pre-fills agent.framework so the dashboard badges runs correctly."
   ).option(
@@ -4654,8 +5030,38 @@ function createProgram() {
   ).option(
     "--starter",
     "Force the full starter library even in an existing project."
+  ).option(
+    "--example <id>",
+    `Scaffold a complete bundled example into ./<id> instead of a starter project. Ids: ${exampleIds().join(", ")}.`
   ).action(async (opts) => {
     const sdk = opts.sdk?.trim();
+    const exampleId = opts.example?.trim();
+    if (exampleId !== void 0) {
+      const conflicting = ["sdk", "bare", "starter"].filter(
+        (flag) => opts[flag] !== void 0 && opts[flag] !== false
+      );
+      if (conflicting.length > 0) {
+        console.error(
+          `\`--example\` scaffolds a complete example and cannot be combined with ${conflicting.map((flag) => `--${flag}`).join(", ")}.`
+        );
+        process.exitCode = 2;
+        return;
+      }
+      if (exampleId === "") {
+        console.error(unknownExampleMessage(""));
+        process.exitCode = 2;
+        return;
+      }
+      try {
+        const result = await scaffoldExample({ id: exampleId, cwd: process.cwd() });
+        console.error(scaffoldSummary(result));
+      } catch (err) {
+        if (!(err instanceof ExampleScaffoldError)) throw err;
+        console.error(err.message);
+        process.exitCode = 2;
+      }
+      return;
+    }
     if (sdk !== void 0 && sdk !== "claude" && sdk !== "claude-managed") {
       console.error(
         `Unknown --sdk value "${opts.sdk}". Supported: claude, claude-managed.`

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@pome-sh/cli",
-  "version": "0.34.3",
+  "version": "0.35.1",
   "description": "Test AI agents against digital twins of real SaaS APIs. Records tool-call traces and scores them on pome.sh.",
   "keywords": [
     "ai",