@eigenpal/cli 0.4.7 → 0.4.9

package/CHANGELOG.md

@@ -141,6 +141,16 @@
 
   Non-overlapping examples are listed in dedicated `only in A` / `only in B` blocks before the table so |Δ| sort stays meaningful.
 
+- 9dc84b9: CLI: agent-friendly experiment polling, watching, and comparison.
+
+  Four small additions so agents (and humans writing monitoring scripts) don't have to roll their own bash pollers, JSON parsers, or aggregators on top of `experiment status`.
+  - **`experiment status --json` now includes a top-level `summary` rollup** — `{ total, terminal, complete, completedCount, failedCount, cancelledCount, rejectedCount, runningCount, pendingCount }`. Polling scripts can read `.summary.complete` / `.summary.failedCount` directly instead of folding `executions[].status` themselves. `executions[]` stays as-is for backwards compat.
+  - **`experiment status --short`** — single-line, awk-friendly stdout summary (e.g. `6/6 done failed=0 cancelled=0 rejected=0`). Mutually exclusive with `--watch`. Drops the failure detail block + closing tip so monitoring loops stay clean.
+  - **New `experiment watch <workflow-id> <batchId>`** — polls until terminal, then auto-pulls eval results to `./results-<batchId>.<format>` in one command. Replaces the old "monitor + status + pull + score" recipe. Flags: `--interval`, `--max-wait`, `--format csv|json`, `--pull-on-complete <path>` to override destination, `--no-pull` to opt out. Exit codes match `experiment status --watch` (0 clean, 1 any failure, 2 deadline).
+  - **`experiment compare` adds a per-evaluator aggregate** — a "Per-evaluator deltas" table renders above the existing per-row table with rows, mean Δ, regressions, and improvements per evaluator (sorted by |meanΔ| desc). `--json` exposes it at `summary.byEvaluator[]`.
+
+  Updated SKILL.md so the canonical "run an experiment + collect results" recipe uses `experiment watch` and adds a "monitoring scripts" section that calls out `--short` and `summary.complete`.
+
 - 3da0612: CLI: pretty default rendering for `list`, `get`, and mutating commands; `--json` toggle for piping.
   - `workflow definition/dataset/experiment/version/step-type list` and `workflow execution list` render aligned ASCII tables. `--json` restores the legacy raw payload.
   - `workflow execution get` (no other flags) prints the vertical step list via `renderFrame` instead of dumping 1000+ lines of JSON. `--json` for the full payload; `--output-path` / `--step` / `--include` continue to print filtered JSON.

package/dist/cli.js

@@ -21310,7 +21310,6 @@ var init_extract = __esm(() => {
     prompt: exports_external.string().optional().describe("Custom prompt template for extraction"),
     provider: exports_external.string().optional().describe('Provider ID to use (e.g., "openai-gpt4o", "anthropic-claude")'),
     model: exports_external.string().optional().describe("Model to use (overrides provider default)"),
-    temperature: exports_external.number().min(0).max(2).default(0).describe("Model temperature (0 = deterministic)"),
     maxInputTokens: exports_external.number().int().optional().describe("Max input tokens. Truncates input text and logs a warning when exceeded. Omit for no limit.")
   });
 });
@@ -21844,6 +21843,9 @@ var init_execution_row = __esm(() => {
   ];
 });
 
+// ../types/src/llm-determinism.ts
+var DETERMINISTIC_TEMPERATURE = 0;
+
 // ../types/src/workflow/steps.ts
 function parseStepType(type) {
   const [category, operation] = type.split(".");
@@ -22110,7 +22112,6 @@ var init_step_configs = __esm(() => {
     prompt: exports_external.string().optional().describe("Custom prompt template for extraction"),
     provider: exports_external.string().optional().describe('Provider ID (e.g., "openai-gpt4o")'),
     model: exports_external.string().optional().describe("Model override"),
-    temperature: exports_external.number().min(0).max(2).default(0).optional().describe("Model temperature (0 = deterministic)"),
     maxInputTokens: exports_external.number().int().optional().describe("Max input tokens. Truncates input text and logs a warning when exceeded. Omit for no limit.")
   });
   AiExtractOutputSchema = exports_external.record(exports_external.string(), exports_external.unknown()).describe("Extracted structured data matching the provided schema");
@@ -22695,7 +22696,8 @@ var init_workflow = __esm(() => {
     inputs: exports_external.array(WorkflowInputDefSchema).optional(),
     steps: exports_external.array(StepSchema),
     output: exports_external.record(exports_external.string(), exports_external.string()).optional(),
-    settings: WorkflowSettingsSchema.optional()
+    settings: WorkflowSettingsSchema.optional(),
+    defaultModel: exports_external.string().optional()
   });
   StoredWorkflowDefinitionSchema = exports_external.object({
     id: exports_external.string(),
@@ -23463,8 +23465,7 @@ async function judgeScoreJson(params) {
   const response = await client.extract(userContent, {
     schema: SCORE_SCHEMA,
     prompt: systemPrompt,
-    model,
-    temperature: 0
+    model
   });
   const rawScore = response.data.score;
   const score = typeof rawScore === "number" ? rawScore : typeof rawScore === "string" ? parseFloat(rawScore) : NaN;
@@ -23484,8 +23485,7 @@ async function judgeLabelJson(params) {
   const response = await client.extract(userContent, {
     schema: buildLabelSchema(labels),
     prompt: systemPrompt,
-    model,
-    temperature: 0
+    model
   });
   const label = typeof response.data.label === "string" ? response.data.label.trim() : "";
   if (!labels.includes(label)) {
@@ -23870,6 +23870,7 @@ __export(exports_src, {
   DatasetImportRequestSchema: () => DatasetImportRequestSchema,
   DatasetImportModeSchema: () => DatasetImportModeSchema,
   DURABLE_EXECUTION_TYPES: () => DURABLE_EXECUTION_TYPES,
+  DETERMINISTIC_TEMPERATURE: () => DETERMINISTIC_TEMPERATURE,
   DEFAULT_PAGE_SIZE: () => DEFAULT_PAGE_SIZE,
   DEFAULT_CONCURRENCY: () => DEFAULT_CONCURRENCY,
   DATASET_NAME_PATTERN: () => DATASET_NAME_PATTERN,
@@ -143405,7 +143406,7 @@ import { fileURLToPath as fileURLToPath6 } from "url";
 // package.json
 var package_default = {
   name: "@eigenpal/cli",
-  version: "0.4.7",
+  version: "0.4.9",
   description: "Eigenpal CLI — author and run document workflows from your terminal. Agent-ready.",
   keywords: [
     "eigenpal",
@@ -147595,10 +147596,47 @@ function buildBatchDiff(args) {
       sharedExamples: sharedExampleSet.size,
       onlyInA,
       onlyInB,
-      regressionThreshold
+      regressionThreshold,
+      byEvaluator: aggregateByEvaluator(rows)
     }
   };
 }
+function aggregateByEvaluator(rows) {
+  const buckets = new Map;
+  for (const r5 of rows) {
+    const list = buckets.get(r5.evaluator);
+    if (list)
+      list.push(r5);
+    else
+      buckets.set(r5.evaluator, [r5]);
+  }
+  const aggregates = [];
+  for (const [evaluator, group2] of buckets) {
+    const comparable = group2.filter((r5) => r5.delta != null);
+    const meanDelta = comparable.length === 0 ? null : comparable.reduce((sum, r5) => sum + r5.delta, 0) / comparable.length;
+    aggregates.push({
+      evaluator,
+      comparable: comparable.length,
+      meanDelta,
+      regressions: group2.filter((r5) => r5.status === "regression").length,
+      improvements: group2.filter((r5) => r5.status === "improvement").length,
+      unchanged: group2.filter((r5) => r5.status === "unchanged").length
+    });
+  }
+  aggregates.sort((a5, b8) => {
+    if (a5.meanDelta == null && b8.meanDelta == null)
+      return a5.evaluator.localeCompare(b8.evaluator);
+    if (a5.meanDelta == null)
+      return 1;
+    if (b8.meanDelta == null)
+      return -1;
+    const diff = Math.abs(b8.meanDelta) - Math.abs(a5.meanDelta);
+    if (diff !== 0)
+      return diff;
+    return a5.evaluator.localeCompare(b8.evaluator);
+  });
+  return aggregates;
+}
 function sortDiffRows(rows, sort) {
   const tiebreak = (a5, b8) => {
     const exCmp = a5.example.localeCompare(b8.example);
@@ -147684,10 +147722,30 @@ function renderBatchDiffHuman(diff) {
   }
   process.stderr.write(`
 `);
+  if (diff.summary.byEvaluator.length > 0) {
+    process.stderr.write(`${ui.bold("Per-evaluator deltas")} ${ui.dim("(B − A):")}
+`);
+    console.log(table(diff.summary.byEvaluator, [
+      { key: "evaluator", header: "evaluator" },
+      { key: "comparable", header: "rows", align: "right" },
+      {
+        key: "meanDelta",
+        header: "mean Δ",
+        align: "right",
+        format: (v11) => formatDelta(v11)
+      },
+      { key: "regressions", header: "regressions", align: "right" },
+      { key: "improvements", header: "improvements", align: "right" }
+    ]));
+    process.stderr.write(`
+`);
+  }
   if (diff.rows.length === 0) {
     process.stderr.write(`${ui.dim("No shared (example, evaluator) pairs to compare.")}
 `);
   } else {
+    process.stderr.write(`${ui.bold("Per-row deltas:")}
+`);
     console.log(table(diff.rows, [
       { key: "example", header: "example" },
       { key: "evaluator", header: "evaluator" },
@@ -148444,7 +148502,7 @@ returns immediately with \`{ batchId, total }\`; poll
         return;
       }
       success2(`Started experiment ${ui.bold(start.batchId)} ${ui.dim(`(${start.total} execution${start.total === 1 ? "" : "s"} queued)`)}`);
-      process.stderr.write(`   ${ui.dim(`Watch: eigenpal workflow experiment status ${workflowId} ${start.batchId} --watch`)}
+      process.stderr.write(`   ${ui.dim(`Watch + auto-pull: eigenpal workflow experiment watch ${workflowId} ${start.batchId}`)}
 `);
       return;
     }
@@ -148461,20 +148519,32 @@ returns immediately with \`{ batchId, total }\`; poll
       await new Promise((r5) => setTimeout(r5, intervalMs));
     }
   }));
-  const statusCmd = experiment.command("status <workflow-id> <batchId>").description(descriptionFor("workflow_get_experiment_status")).option("--watch", "Poll until every execution reaches a terminal state (completed/failed/cancelled/rejected), then exit", false).option("--interval <seconds>", "Poll interval in seconds when --watch is set (default 5)", intArg, 5).option("--max-wait <seconds>", "Hard ceiling for --watch in seconds (default 1800 = 30 min)", intArg, 30 * 60).option("--include <kinds>", "Comma-separated extras to attach when --watch terminates: payload (full per-execution snapshot, can be hundreds of KB)", "");
+  const statusCmd = experiment.command("status <workflow-id> <batchId>").description(descriptionFor("workflow_get_experiment_status")).option("--watch", "Poll until every execution reaches a terminal state (completed/failed/cancelled/rejected), then exit", false).option("--short", "Single-line plain-text summary on stdout (e.g. `6/6 done failed=0 cancelled=0 rejected=0`). Pipe-friendly for monitoring loops; mutually exclusive with --watch.", false).option("--interval <seconds>", "Poll interval in seconds when --watch is set (default 5)", intArg, 5).option("--max-wait <seconds>", "Hard ceiling for --watch in seconds (default 1800 = 30 min)", intArg, 30 * 60).option("--include <kinds>", "Comma-separated extras to attach when --watch terminates: payload (full per-execution snapshot, can be hundreds of KB)", "");
   addJsonFlag(withBaseUrl(statusCmd)).action(action(async (workflow2, batchId, opts) => {
+    if (opts.short && opts.watch) {
+      error51("--short and --watch are mutually exclusive (short is for spot-check polling).");
+      process.exit(2);
+    }
     const includes = new Set((opts.include ?? "").split(",").map((s5) => s5.trim()).filter(Boolean));
     const includePayload = includes.has("payload");
     const { client, workflowId } = await buildClientForWorkflow(workflow2, opts);
     const url3 = `/api/v1/workflows/${workflowId}/experiments/${batchId}`;
     if (!opts.watch) {
       const result = await client.get(url3);
+      const execs = result.executions ?? [];
+      const summary = summarizeExperimentExecutions(execs);
       if (opts.json) {
-        printJson(result);
+        printJson({ ...result, summary: rollupForJson(summary) });
+        return;
+      }
+      if (opts.short) {
+        console.log(formatShortStatus(summary));
+        const failed2 = summary.counts.failed ?? 0;
+        const cancelled2 = summary.counts.cancelled ?? 0;
+        if (summary.done && failed2 + cancelled2 > 0)
+          process.exit(1);
         return;
       }
-      const execs = result.executions ?? [];
-      const summary = summarizeExperimentExecutions(execs);
       const counts = Object.entries(summary.counts).map(([k9, v11]) => `${k9}=${v11}`).sort().join(" ");
       console.log(`${ui.bold(batchId)}  ${summary.terminal}/${summary.total} terminal  ${ui.dim(`(${counts || "no executions"})`)}`);
       const failed = summary.counts.failed ?? 0;
@@ -148488,56 +148558,25 @@ returns immediately with \`{ batchId, total }\`; poll
         process.exit(1);
       return;
     }
-    const deadline = Date.now() + opts.maxWait * 1000;
-    let lastSummary = "";
-    while (true) {
-      const result = await client.get(url3);
-      const execs = result.executions ?? [];
-      const summary = summarizeExperimentExecutions(execs);
-      const counts = Object.entries(summary.counts).map(([k9, v11]) => `${k9}=${v11}`).sort().join(" ");
-      const line3 = `${summary.terminal}/${summary.total} terminal  ${ui.dim(`(${counts || "no executions"})`)}`;
-      if (line3 !== lastSummary) {
-        if (process.stderr.isTTY) {
-          process.stderr.write(`\r\x1B[K${line3}`);
-        } else {
-          process.stderr.write(`${line3}
-`);
-        }
-        lastSummary = line3;
-      }
-      const wantPayload = opts.json || includePayload;
-      if (summary.done) {
-        if (process.stderr.isTTY)
-          process.stderr.write(`
-`);
-        const failed = summary.counts.failed ?? 0;
-        const cancelled = summary.counts.cancelled ?? 0;
-        const rejected = summary.counts.rejected ?? 0;
-        const badCount = failed + cancelled + rejected;
-        if (badCount > 0) {
-          renderExperimentFailures(execs, summary.total);
-        }
+    const wantPayload = opts.json || includePayload;
+    await pollExperimentUntilTerminal({
+      client,
+      url: url3,
+      interval: opts.interval,
+      maxWait: opts.maxWait,
+      onTerminal: (result, summary) => {
         if (wantPayload) {
-          printJson(result);
+          printJson({ ...result, summary: rollupForJson(summary) });
         } else {
           process.stderr.write(`${ui.dim(`use --include payload (or --json) for the full per-execution snapshot`)}
 `);
         }
-        if (badCount > 0)
-          process.exit(1);
-        return;
-      }
-      if (Date.now() >= deadline) {
-        if (process.stderr.isTTY)
-          process.stderr.write(`
-`);
-        error51(`Timed out after ${opts.maxWait}s with ${summary.total - summary.terminal} non-terminal execution(s). Re-run to keep watching.`);
+      },
+      onDeadline: (result, summary) => {
         if (wantPayload)
-          printJson(result);
-        process.exit(2);
+          printJson({ ...result, summary: rollupForJson(summary) });
       }
-      await new Promise((r5) => setTimeout(r5, opts.interval * 1000));
-    }
+    });
   }));
   const cancelCmd = experiment.command("cancel <workflow-id> <batchId>").description("Cancel every execution in a batch. Idempotent.").option("--yes", "Required for non-TTY shells (CI, pipes). Acts immediately, no prompt.").addHelpText("after", `
 Examples:
@@ -148626,6 +148665,62 @@ batches must belong to the same workflow within the caller's tenant.
     }
     renderBatchDiffHuman(diff);
   }));
+  const watchCmd = experiment.command("watch <workflow-id> <batchId>").description("Poll until terminal, then auto-pull results — replaces `status --watch` + `results --out`.").option("--interval <seconds>", "Poll interval in seconds (default 5)", intArg, 5).option("--max-wait <seconds>", "Hard ceiling in seconds (default 1800 = 30 min)", intArg, 30 * 60).option("--pull-on-complete <path>", "Destination for the results file. Default: ./results-<batchId>.<format>").option("--format <csv|json>", "Results export format (default json)", parseExportFormat, "json").option("--no-pull", "Skip auto-pulling results on terminal (just watch)").addHelpText("after", `
+Examples:
+  $ eigenpal workflow experiment watch wf_abc123 evb_xyz789
+  $ eigenpal workflow experiment watch wf_abc123 evb_xyz789 --format csv
+  $ eigenpal workflow experiment watch wf_abc123 evb_xyz789 --pull-on-complete ./out/r.json
+  $ eigenpal workflow experiment watch wf_abc123 evb_xyz789 --no-pull
+
+Behavior:
+  - Polls every --interval seconds; redraws a one-line tick on a TTY,
+    appends a new line per tick when piped.
+  - On terminal: renders the failed-execution detail block (if any), then
+    downloads results to ./results-<batchId>.<format> unless --no-pull
+    or --pull-on-complete <path> says otherwise.
+  - 4xx errors (e.g. unknown batch id) abort immediately rather than spin.
+  - Pre-flight check: the destination directory must exist before the
+    poll loop starts, so a typo doesn't waste a 30-min wait.
+  - Exit codes (matching \`experiment status --watch\`):
+      0  every execution succeeded (or pull skipped via --no-pull on a clean batch)
+      1  at least one execution ended in failed/cancelled/rejected,
+         OR the export download failed after a clean batch
+      2  --max-wait deadline reached — re-run to keep watching;
+         OR misuse (e.g. --no-pull combined with --pull-on-complete)
+`);
+  withBaseUrl(watchCmd).action(action(async (workflow2, batchId, opts) => {
+    if (!opts.pull && opts.pullOnComplete) {
+      error51("--no-pull and --pull-on-complete are mutually exclusive — pass one or the other.");
+      process.exit(2);
+    }
+    const { client, workflowId } = await buildClientForWorkflow(workflow2, opts);
+    const url3 = `/api/v1/workflows/${workflowId}/experiments/${batchId}`;
+    const dest = opts.pull ? opts.pullOnComplete ?? `./results-${batchId}.${opts.format}` : null;
+    if (dest) {
+      const dir = path15.dirname(dest);
+      if (!existsSync9(dir)) {
+        error51(`Destination directory does not exist: ${dir}. Create it first or pass a path under an existing directory.`);
+        process.exit(2);
+      }
+    }
+    await pollExperimentUntilTerminal({
+      client,
+      url: url3,
+      interval: opts.interval,
+      maxWait: opts.maxWait,
+      onTerminal: async () => {
+        if (!dest)
+          return;
+        const params = new URLSearchParams({ format: opts.format, batchId });
+        try {
+          await downloadStreamToFile(client, `/api/v1/workflows/${workflowId}/eval-results/export?${params.toString()}`, dest);
+        } catch (err2) {
+          error51(`Watch finished but pulling results failed: ${err2 instanceof Error ? err2.message : String(err2)}. Re-run \`experiment results <wf> <batch>\` to retry.`);
+          process.exit(1);
+        }
+      }
+    });
+  }));
 }
 function renderExperimentFailures(execs, totalCount) {
   const failedExecs = execs.filter((e8) => e8.status === "failed" || e8.status === "cancelled" || e8.status === "rejected");
@@ -148664,6 +148759,83 @@ function summarizeExperimentExecutions(execs) {
     done: execs.length > 0 && terminal === execs.length
   };
 }
+function rollupForJson(summary) {
+  return {
+    total: summary.total,
+    terminal: summary.terminal,
+    complete: summary.done,
+    completedCount: summary.counts.completed ?? 0,
+    failedCount: summary.counts.failed ?? 0,
+    cancelledCount: summary.counts.cancelled ?? 0,
+    rejectedCount: summary.counts.rejected ?? 0,
+    runningCount: summary.counts.running ?? 0,
+    pendingCount: summary.counts.pending ?? 0
+  };
+}
+function formatShortStatus(summary) {
+  const state = summary.done ? "done" : "in-progress";
+  return [
+    `${summary.terminal}/${summary.total}`,
+    state,
+    `failed=${summary.counts.failed ?? 0}`,
+    `cancelled=${summary.counts.cancelled ?? 0}`,
+    `rejected=${summary.counts.rejected ?? 0}`
+  ].join(" ");
+}
+function renderTickLine(summary) {
+  const counts = Object.entries(summary.counts).map(([k9, v11]) => `${k9}=${v11}`).sort().join(" ");
+  return `${summary.terminal}/${summary.total} terminal  ${ui.dim(`(${counts || "no executions"})`)}`;
+}
+async function pollExperimentUntilTerminal({
+  client,
+  url: url3,
+  interval,
+  maxWait,
+  onTerminal,
+  onDeadline
+}) {
+  const deadline = Date.now() + maxWait * 1000;
+  let lastSummary = "";
+  while (true) {
+    const result = await client.get(url3);
+    const execs = result.executions ?? [];
+    const summary = summarizeExperimentExecutions(execs);
+    const line3 = renderTickLine(summary);
+    if (line3 !== lastSummary) {
+      if (process.stderr.isTTY) {
+        process.stderr.write(`\r\x1B[K${line3}`);
+      } else {
+        process.stderr.write(`${line3}
+`);
+      }
+      lastSummary = line3;
+    }
+    if (summary.done) {
+      if (process.stderr.isTTY)
+        process.stderr.write(`
+`);
+      const failed = summary.counts.failed ?? 0;
+      const cancelled = summary.counts.cancelled ?? 0;
+      const rejected = summary.counts.rejected ?? 0;
+      const badCount = failed + cancelled + rejected;
+      if (badCount > 0)
+        renderExperimentFailures(execs, summary.total);
+      await onTerminal(result, summary);
+      if (badCount > 0)
+        process.exit(1);
+      return;
+    }
+    if (Date.now() >= deadline) {
+      if (process.stderr.isTTY)
+        process.stderr.write(`
+`);
+      error51(`Timed out after ${maxWait}s with ${summary.total - summary.terminal} non-terminal execution(s). Re-run to keep watching.`);
+      onDeadline?.(result, summary);
+      process.exit(2);
+    }
+    await new Promise((r5) => setTimeout(r5, interval * 1000));
+  }
+}
 function registerVersionsCommands(parent) {
   const versions2 = parent.command("versions").description("Inspect and restore historical workflow versions.");
   const listCmd = versions2.command("list <workflow-id>").description(descriptionFor("workflow_list_versions"));

package/dist/skill/SKILL.md

@@ -145,6 +145,17 @@ to retry.
 eigenpal workflow experiment compare evb_old evb_new --regression-threshold 0.05
 ```
 
+Output is two stacked tables: a per-evaluator aggregate (rows per evaluator
+with mean Δ, regressions, improvements — sorted biggest mover first) followed
+by a per-(example, evaluator) detail table. No need to write a Python
+aggregator on top.
+
+```bash
+# JSON exposes both layers — `summary.byEvaluator` for the rollup,
+# `rows` for the full per-example detail.
+eigenpal workflow experiment compare evb_old evb_new --json | jq '.summary.byEvaluator'
+```
+
 Unlike its siblings, `compare` takes no `<workflow-id>` — the server resolves
 the owning workflow from each batch id. Both batches must live in the
 same workflow within your tenant. See [`reference/debugging.md`](reference/debugging.md#6-compare-two-experiment-batches)
@@ -218,12 +229,64 @@ eigenpal workflow experiment run <workflow-id>                       # → { bat
 
 # Or restrict to one example:
 eigenpal workflow experiment run <workflow-id> --example-id evx_…
+```
+
+**Recommended:** use `experiment watch` — it polls until terminal AND auto-pulls
+results to disk in one command. Replaces the old "run → status (poll) → results"
+chain so agents don't have to write a custom poller, parse `executions[].status`,
+or chain `monitor + status + pull + score` shell logic.
+
+```bash
+# One command: watch until terminal, then write ./results-<batchId>.json on completion.
+eigenpal workflow experiment watch <workflow-id> <batch-id>
+
+# CSV output:
+eigenpal workflow experiment watch <workflow-id> <batch-id> --format csv
+
+# Custom destination (overrides the default ./results-<batchId>.<format>):
+eigenpal workflow experiment watch <workflow-id> <batch-id> --pull-on-complete ./out/r.json
+
+# Just watch, skip the pull (e.g. you only need the live tick + exit code):
+eigenpal workflow experiment watch <workflow-id> <batch-id> --no-pull
+```
 
-# Poll until done. `experiment status` auto-detaches after 30 min; just re-run.
-eigenpal workflow experiment status <workflow-id> <batch-id>
+Exit codes match `experiment status --watch`: `0` clean, `1` any
+failed/cancelled/rejected execution, `2` `--max-wait` deadline reached
+(30 min default — re-run to keep watching).
 
-# Pull results — `--out` writes the file directly from the server's signed-URL
-# storage, so large CSVs don't go through the CLI process.
+### Monitoring scripts — use `--short` and the JSON rollup
+
+When a script needs to spot-check a batch (no waiting), prefer one of the two
+machine-readable paths instead of folding `executions[].status` yourself.
+
+```bash
+# Single line on stdout, awk-friendly:
+eigenpal workflow experiment status <wf> <batch> --short
+# → 6/6 done failed=0 cancelled=0 rejected=0   (or "in-progress" while pending)
+
+# Parse with awk:
+DONE_STATE=$(eigenpal workflow experiment status <wf> <batch> --short | awk '{print $2}')
+[ "$DONE_STATE" = "done" ] || exit 0   # not terminal yet, try again later
+```
+
+```bash
+# JSON includes a top-level `summary` rollup so you don't count statuses by hand:
+eigenpal workflow experiment status <wf> <batch> --json | jq '.summary'
+# → { total: 6, terminal: 6, complete: true,
+#     completedCount: 5, failedCount: 1,
+#     cancelledCount: 0, rejectedCount: 0,
+#     runningCount: 0, pendingCount: 0 }
+
+# `summary.complete` is the canonical poll-completion flag.
+```
+
+`--short` and `--json` are mutually exclusive with `--watch` — for live
+streaming use `experiment watch` instead.
+
+If you need the raw eval-results export after watching with `--no-pull`, or
+to refetch results outside a watch loop, fall back to:
+
+```bash
 eigenpal workflow experiment results <workflow-id> <batch-id> --format csv --out r.csv
 ```
 
@@ -433,10 +496,11 @@ eigenpal workflow clear-local                 # delete local execution artifacts
 
 # Experiments (batch eval runs)
 eigenpal workflow experiment run     <workflow-id> [--example-id <id> ...]
-eigenpal workflow experiment status  <workflow-id> <batch-id> [--watch]    # exit 1 on terminal w/ failures, 2 on --max-wait
+eigenpal workflow experiment status  <workflow-id> <batch-id> [--watch] [--short] [--json]    # exit 1 on terminal w/ failures, 2 on --max-wait. --json includes top-level `summary` rollup.
+eigenpal workflow experiment watch   <workflow-id> <batch-id> [--format csv|json] [--pull-on-complete <path>] [--no-pull]    # poll + auto-pull results in one shot
 eigenpal workflow experiment results <workflow-id> [batch-id] --format {csv|json} --out r.csv
 eigenpal workflow experiment list    <workflow-id>
-eigenpal workflow experiment compare <batch-a> <batch-b> [--regression-threshold] [--sort]   # no --workflow-id
+eigenpal workflow experiment compare <batch-a> <batch-b> [--regression-threshold] [--sort]   # no --workflow-id; prints per-evaluator aggregate + per-row table
 
 # Step-type / evaluator-type introspection
 eigenpal workflow step-type      list

package/dist/skill/reference/cli/workflow.md

@@ -38,6 +38,7 @@ Manage workflows: push, pull, run, evaluate.
   - [`eigenpal workflow experiment cancel [options] <workflow-id> <batchId>`](#eigenpal-workflow-experiment-cancel-options-workflow-id-batchid)
   - [`eigenpal workflow experiment results [options] <workflow-id> [batchId]`](#eigenpal-workflow-experiment-results-options-workflow-id-batchid)
   - [`eigenpal workflow experiment compare [options] <batchIdA> <batchIdB>`](#eigenpal-workflow-experiment-compare-options-batchida-batchidb)
+  - [`eigenpal workflow experiment watch [options] <workflow-id> <batchId>`](#eigenpal-workflow-experiment-watch-options-workflow-id-batchid)
   - [`eigenpal workflow execution run [options] <workflow-id> [examples...]`](#eigenpal-workflow-execution-run-options-workflow-id-examples)
   - [`eigenpal workflow execution get [options] <executionId>`](#eigenpal-workflow-execution-get-options-executionid)
   - [`eigenpal workflow execution list [options] <workflow-id>`](#eigenpal-workflow-execution-list-options-workflow-id)
@@ -79,7 +80,8 @@ workflow
 │   ├── status <workflow-id> <batchId>
 │   ├── cancel <workflow-id> <batchId>
 │   ├── results <workflow-id> [batchId]
-│   └── compare <batchIdA> <batchIdB>
+│   ├── compare <batchIdA> <batchIdB>
+│   └── watch <workflow-id> <batchId>
 ├── execution
 │   ├── run <workflow-id> [examples...]
 │   ├── get <executionId>
@@ -137,14 +139,15 @@ workflow
 
 ### Experiment
 
-| Command                                                                  | Description                                        |
-| ------------------------------------------------------------------------ | -------------------------------------------------- |
-| `eigenpal workflow experiment list [options] <workflow-id>`              | List executions for the workflow, newest first.    |
-| `eigenpal workflow experiment run [options] <workflow-id>`               | Start a batch eval against the workflow's dataset. |
-| `eigenpal workflow experiment status [options] <workflow-id> <batchId>`  | Aggregate progress for a batch by `batchId`.       |
-| `eigenpal workflow experiment cancel [options] <workflow-id> <batchId>`  | Cancel every execution in a batch. Idempotent.     |
-| `eigenpal workflow experiment results [options] <workflow-id> [batchId]` | Download eval results in CSV or JSON.              |
-| `eigenpal workflow experiment compare [options] <batchIdA> <batchIdB>`   | Diff eval scores between two experiment batches.   |
+| Command                                                                  | Description                                                                                |
+| ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------ |
+| `eigenpal workflow experiment list [options] <workflow-id>`              | List executions for the workflow, newest first.                                            |
+| `eigenpal workflow experiment run [options] <workflow-id>`               | Start a batch eval against the workflow's dataset.                                         |
+| `eigenpal workflow experiment status [options] <workflow-id> <batchId>`  | Aggregate progress for a batch by `batchId`.                                               |
+| `eigenpal workflow experiment cancel [options] <workflow-id> <batchId>`  | Cancel every execution in a batch. Idempotent.                                             |
+| `eigenpal workflow experiment results [options] <workflow-id> [batchId]` | Download eval results in CSV or JSON.                                                      |
+| `eigenpal workflow experiment compare [options] <batchIdA> <batchIdB>`   | Diff eval scores between two experiment batches.                                           |
+| `eigenpal workflow experiment watch [options] <workflow-id> <batchId>`   | Poll until terminal, then auto-pull results — replaces `status --watch` + `results --out`. |
 
 ### Execution
 
@@ -508,14 +511,15 @@ Aggregate progress for a batch by `batchId`.
 
 ### Options
 
-| Flag                   | Required | Default | Description                                                                                                            |
-| ---------------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------- |
-| `--watch`              | no       | `false` | Poll until every execution reaches a terminal state (completed/failed/cancelled/rejected), then exit                   |
-| `--interval <seconds>` | no       | `5`     | Poll interval in seconds when --watch is set (default 5)                                                               |
-| `--max-wait <seconds>` | no       | `1800`  | Hard ceiling for --watch in seconds (default 1800 = 30 min)                                                            |
-| `--include <kinds>`    | no       | `""`    | Comma-separated extras to attach when --watch terminates: payload (full per-execution snapshot, can be hundreds of KB) |
-| `--base-url <url>`     | no       |         | Server base URL                                                                                                        |
-| `--json`               | no       |         | Output the raw server response as JSON                                                                                 |
+| Flag                   | Required | Default | Description                                                                                                                                                      |
+| ---------------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `--watch`              | no       | `false` | Poll until every execution reaches a terminal state (completed/failed/cancelled/rejected), then exit                                                             |
+| `--short`              | no       | `false` | Single-line plain-text summary on stdout (e.g. `6/6 done failed=0 cancelled=0 rejected=0`). Pipe-friendly for monitoring loops; mutually exclusive with --watch. |
+| `--interval <seconds>` | no       | `5`     | Poll interval in seconds when --watch is set (default 5)                                                                                                         |
+| `--max-wait <seconds>` | no       | `1800`  | Hard ceiling for --watch in seconds (default 1800 = 30 min)                                                                                                      |
+| `--include <kinds>`    | no       | `""`    | Comma-separated extras to attach when --watch terminates: payload (full per-execution snapshot, can be hundreds of KB)                                           |
+| `--base-url <url>`     | no       |         | Server base URL                                                                                                                                                  |
+| `--json`               | no       |         | Output the raw server response as JSON                                                                                                                           |
 
 ### `eigenpal workflow experiment cancel [options] <workflow-id> <batchId>`
 
@@ -575,6 +579,28 @@ Diff eval scores between two experiment batches.
 | `--base-url <url>`                                     | no       |                    | Server base URL                                        |
 | `--json`                                               | no       |                    | Output the raw server response as JSON                 |
 
+### `eigenpal workflow experiment watch [options] <workflow-id> <batchId>`
+
+Poll until terminal, then auto-pull results — replaces `status --watch` + `results --out`.
+
+### Arguments
+
+| Name          | Required | Variadic | Description |
+| ------------- | -------- | -------- | ----------- |
+| `workflow-id` | yes      | no       |             |
+| `batchId`     | yes      | no       |             |
+
+### Options
+
+| Flag                        | Required | Default  | Description                                                             |
+| --------------------------- | -------- | -------- | ----------------------------------------------------------------------- |
+| `--interval <seconds>`      | no       | `5`      | Poll interval in seconds (default 5)                                    |
+| `--max-wait <seconds>`      | no       | `1800`   | Hard ceiling in seconds (default 1800 = 30 min)                         |
+| `--pull-on-complete <path>` | no       |          | Destination for the results file. Default: ./results-<batchId>.<format> |
+| `--format <csv\|json>`      | no       | `"json"` | Results export format (default json)                                    |
+| `--no-pull`                 | no       |          | Skip auto-pulling results on terminal (just watch)                      |
+| `--base-url <url>`          | no       |          | Server base URL                                                         |
+
 ### `eigenpal workflow execution run [options] <workflow-id> [examples...]`
 
 Run a saved workflow against local dataset examples.

package/dist/skill/reference/step-types.md

@@ -153,7 +153,6 @@ Extract structured data from text using AI with a JSON schema
 | `prompt` | string | no |  | Custom prompt template for extraction |
 | `provider` | string | no |  | Provider ID (e.g., "openai-gpt4o") |
 | `model` | string | no |  | Model override |
-| `temperature` | number | no | `0` | Model temperature (0 = deterministic) |
 | `maxInputTokens` | integer | no |  | Max input tokens. Truncates input text and logs a warning when exceeded. Omit for no limit. |
 
 **Output:** `record<string, unknown>`

package/dist/skill/reference/workflow-yaml.md

@@ -387,6 +387,7 @@ _Generated from `WorkflowDefinitionSchema` in `@eigenpal/types/src/workflow/work
 | `steps` | array<unknown> | yes |  |  |
 | `output` | record<string, string> | no |  |  |
 | `settings` | object | no |  |  |
+| `defaultModel` | string | no |  |  |
 
 
 ## Per-input fields

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@eigenpal/cli",
-  "version": "0.4.7",
+  "version": "0.4.9",
   "description": "Eigenpal CLI — author and run document workflows from your terminal. Agent-ready.",
   "keywords": [
     "eigenpal",