@eigenpal/cli 0.4.8 → 0.4.10

package/CHANGELOG.md

@@ -141,6 +141,16 @@
 
   Non-overlapping examples are listed in dedicated `only in A` / `only in B` blocks before the table so |Δ| sort stays meaningful.
 
+- 9dc84b9: CLI: agent-friendly experiment polling, watching, and comparison.
+
+  Four small additions so agents (and humans writing monitoring scripts) don't have to roll their own bash pollers, JSON parsers, or aggregators on top of `experiment status`.
+  - **`experiment status --json` now includes a top-level `summary` rollup** — `{ total, terminal, complete, completedCount, failedCount, cancelledCount, rejectedCount, runningCount, pendingCount }`. Polling scripts can read `.summary.complete` / `.summary.failedCount` directly instead of folding `executions[].status` themselves. `executions[]` stays as-is for backwards compat.
+  - **`experiment status --short`** — single-line, awk-friendly stdout summary (e.g. `6/6 done failed=0 cancelled=0 rejected=0`). Mutually exclusive with `--watch`. Drops the failure detail block + closing tip so monitoring loops stay clean.
+  - **New `experiment watch <workflow-id> <batchId>`** — polls until terminal, then auto-pulls eval results to `./results-<batchId>.<format>` in one command. Replaces the old "monitor + status + pull + score" recipe. Flags: `--interval`, `--max-wait`, `--format csv|json`, `--pull-on-complete <path>` to override destination, `--no-pull` to opt out. Exit codes match `experiment status --watch` (0 clean, 1 any failure, 2 deadline).
+  - **`experiment compare` adds a per-evaluator aggregate** — a "Per-evaluator deltas" table renders above the existing per-row table with rows, mean Δ, regressions, and improvements per evaluator (sorted by |meanΔ| desc). `--json` exposes it at `summary.byEvaluator[]`.
+
+  Updated SKILL.md so the canonical "run an experiment + collect results" recipe uses `experiment watch` and adds a "monitoring scripts" section that calls out `--short` and `summary.complete`.
+
 - 3da0612: CLI: pretty default rendering for `list`, `get`, and mutating commands; `--json` toggle for piping.
   - `workflow definition/dataset/experiment/version/step-type list` and `workflow execution list` render aligned ASCII tables. `--json` restores the legacy raw payload.
   - `workflow execution get` (no other flags) prints the vertical step list via `renderFrame` instead of dumping 1000+ lines of JSON. `--json` for the full payload; `--output-path` / `--step` / `--include` continue to print filtered JSON.
@@ -201,6 +211,25 @@
 
   `@` path syntax (`--code @file.js`, `--inputs items=@arr.json`) matches `curl -d @file` / `gh --body-file -`. Bare paths and `-` (stdin) work too.
 
+- 4137eb8: custom-script evaluator: the YAML now carries the entire
+  `function scoreScript(expected, actual) { ... }` declaration in a single
+  `function` field. Previous formats (`code`, then `body`) wrapped the
+  text implicitly; the new shape stores the function declaration verbatim
+  so the YAML matches the dashboard editor 1-for-1.
+
+  `workflow validate` and `workflow evaluators validate` flag YAML where
+  the function fails to parse, the signature does not match
+  `function scoreScript(expected, actual)` exactly, or the body contains
+  neither `return` nor `throw` — same checks the dashboard runs at form
+  save. The function must return a number in [0, 1]; throws are caught
+  and scored as 0.
+
+  Existing evaluator YAML must be ported by hand: rename `body` (or
+  `code`) to `function`, and add the `function scoreScript(expected,
+actual) { ... }` declaration around your statements. Parameter order is
+  load-bearing — swapping `expected` and `actual` would silently invert
+  every score.
+
 - 3da0612: All list commands now support `--limit` / `--offset` pagination consistently.
   - CLI: `workflow execution list`, `workflow version list`, and `workflow step-type list` previously had no `--offset` (or no pagination at all). All six list commands now accept `--limit`/`--offset` with defaults of 50/0.
   - Server: `GET /api/workflows/:id/versions` now reads `?limit=` and `?offset=` query params (defaults 50/0; max limit 100). The `listVersions` repo method gained an optional `{ limit, offset }` argument.
@@ -273,6 +302,11 @@
 
   Interactive profiles are unchanged: when you log in via `eigenpal auth login`, the profile stores both apiKey and baseUrl, and every subsequent command derives baseUrl from that profile. Set `EIGENPAL_PROFILE=<name>` to switch profiles per-shell without modifying state.
 
+- 542fe0e: CLI: surface evaluator entry-level fields (`name`, `description`, `weight`) so agents and humans can discover the optional `description` field on every evaluator.
+  - `workflow evaluator-type get <type>` now emits `entrySchema` alongside `configSchema`. Inspect with `eigenpal workflow evaluator-type get llm-judge | jq '.entrySchema'`.
+  - The skill reference now renders a "Common entry fields" table per type and a "Writing descriptions for stakeholders" section explaining the convention: each evaluator's `description` should be a one-sentence, plain-language summary aimed at non-technical reviewers in the dashboard.
+  - `init` templates `pdf-extraction` and `text-classification` now ship a `description:` line on their evaluator entries.
+
 - d41ee8b: CLI: fix `eigenpal status` error UX + harden `formatCliError` for real-world connection failures.
   - `eigenpal status` against an unreachable base URL no longer mis-diagnoses the failure as an auth problem. Previously, `EIGENPAL_BASE_URL=http://localhost:9999 eigenpal status` printed "Authentication check failed; try `eigenpal auth login`" — the wrong rabbit hole. The status command now rethrows non-auth errors (connection refused, DNS NXDOMAIN, timeout, HTML responses, 5xx) so `formatCliError` upstream can surface the right hint: `Could not connect to the server at <url> ... Set EIGENPAL_BASE_URL or pass --base-url <url>.` Only real 401/403 responses are treated as auth failures.
   - `formatCliError` now detects connection failures wrapped in any Error shape — bare `Error('ENOTFOUND ...')`, `Error` with `code: 'ECONNREFUSED'`, and `TypeError('fetch failed')` whose `.cause` carries the underlying error. Previously only `TypeError` whose message contained `fetch`/`URL` was recognized, so `getaddrinfo`-style errors fell through to the generic raw-message branch and lost the URL context.

package/dist/cli.js

@@ -21310,7 +21310,6 @@ var init_extract = __esm(() => {
     prompt: exports_external.string().optional().describe("Custom prompt template for extraction"),
     provider: exports_external.string().optional().describe('Provider ID to use (e.g., "openai-gpt4o", "anthropic-claude")'),
     model: exports_external.string().optional().describe("Model to use (overrides provider default)"),
-    temperature: exports_external.number().min(0).max(2).default(0).describe("Model temperature (0 = deterministic)"),
     maxInputTokens: exports_external.number().int().optional().describe("Max input tokens. Truncates input text and logs a warning when exceeded. Omit for no limit.")
   });
 });
@@ -21844,6 +21843,9 @@ var init_execution_row = __esm(() => {
   ];
 });
 
+// ../types/src/llm-determinism.ts
+var DETERMINISTIC_TEMPERATURE = 0;
+
 // ../types/src/workflow/steps.ts
 function parseStepType(type) {
   const [category, operation] = type.split(".");
@@ -22110,7 +22112,6 @@ var init_step_configs = __esm(() => {
     prompt: exports_external.string().optional().describe("Custom prompt template for extraction"),
     provider: exports_external.string().optional().describe('Provider ID (e.g., "openai-gpt4o")'),
     model: exports_external.string().optional().describe("Model override"),
-    temperature: exports_external.number().min(0).max(2).default(0).optional().describe("Model temperature (0 = deterministic)"),
     maxInputTokens: exports_external.number().int().optional().describe("Max input tokens. Truncates input text and logs a warning when exceeded. Omit for no limit.")
   });
   AiExtractOutputSchema = exports_external.record(exports_external.string(), exports_external.unknown()).describe("Extracted structured data matching the provided schema");
@@ -22695,7 +22696,8 @@ var init_workflow = __esm(() => {
     inputs: exports_external.array(WorkflowInputDefSchema).optional(),
     steps: exports_external.array(StepSchema),
     output: exports_external.record(exports_external.string(), exports_external.string()).optional(),
-    settings: WorkflowSettingsSchema.optional()
+    settings: WorkflowSettingsSchema.optional(),
+    defaultModel: exports_external.string().optional()
   });
   StoredWorkflowDefinitionSchema = exports_external.object({
     id: exports_external.string(),
@@ -23325,6 +23327,141 @@ var init_dataset_archive = __esm(() => {
   };
 });
 
+// ../types/src/eval/score-function.ts
+function stripLeadingTrivia(text) {
+  let i = 0;
+  while (i < text.length) {
+    const ch = text[i];
+    if (ch === " " || ch === "\t" || ch === `
+` || ch === "\r" || ch === "\v" || ch === "\f") {
+      i++;
+      continue;
+    }
+    if (ch === "/" && text[i + 1] === "/") {
+      const nl = text.indexOf(`
+`, i + 2);
+      if (nl === -1)
+        return "";
+      i = nl + 1;
+      continue;
+    }
+    if (ch === "/" && text[i + 1] === "*") {
+      const end = text.indexOf("*/", i + 2);
+      if (end === -1)
+        break;
+      i = end + 2;
+      continue;
+    }
+    break;
+  }
+  return text.slice(i);
+}
+function hasValidScoreSignature(fn) {
+  return STRICT_HEADER_REGEX.test(stripLeadingTrivia(fn));
+}
+function isParseableScoreFunction(fn) {
+  try {
+    new Function(fn);
+    return true;
+  } catch {
+    return false;
+  }
+}
+function getScoreFunctionRuntimeIssue(fn) {
+  if (/\basync\s+function\s+scoreScript\b/.test(fn)) {
+    return "scoreScript must be synchronous (drop the `async` keyword — the sandbox calls it sync, so a Promise return scores 0)";
+  }
+  if (/\bimport\s*\(/.test(fn)) {
+    return "dynamic `import(...)` is not supported in the score sandbox";
+  }
+  if (/\brequire\s*\(/.test(fn)) {
+    return "`require(...)` is not available in the score sandbox (no Node module system)";
+  }
+  return null;
+}
+function extractScoreFunctionBody(fn) {
+  const headerMatch = fn.match(ANY_HEADER_REGEX);
+  if (!headerMatch || headerMatch.index === undefined) {
+    return { body: fn, wrapperOk: false };
+  }
+  const start = headerMatch.index + headerMatch[0].length;
+  let depth = 1;
+  let i = start;
+  let inLine = false;
+  let inBlock = false;
+  let strQuote = null;
+  while (i < fn.length) {
+    const ch = fn[i];
+    const next = fn[i + 1];
+    if (inLine) {
+      if (ch === `
+`)
+        inLine = false;
+    } else if (inBlock) {
+      if (ch === "*" && next === "/") {
+        inBlock = false;
+        i++;
+      }
+    } else if (strQuote) {
+      if (ch === "\\")
+        i++;
+      else if (ch === strQuote)
+        strQuote = null;
+    } else if (ch === "/" && next === "/") {
+      inLine = true;
+      i++;
+    } else if (ch === "/" && next === "*") {
+      inBlock = true;
+      i++;
+    } else if (ch === '"' || ch === "'" || ch === "`") {
+      strQuote = ch;
+    } else if (ch === "{") {
+      depth++;
+    } else if (ch === "}") {
+      depth--;
+      if (depth === 0) {
+        const inner = fn.slice(start, i).replace(/^\n+|\n+$/g, "");
+        return { body: inner, wrapperOk: true };
+      }
+    }
+    i++;
+  }
+  return { body: fn.slice(start), wrapperOk: false };
+}
+function defaultScoreFunction() {
+  return `/**
+ * Score the workflow output against the example's expected output.
+ *
+ * @param {Expected} expected the example's expected output
+ * @param {Actual} actual the workflow's actual output
+ * @returns {number} a score in [0, 1]; throws are caught and scored as 0
+ */
+function scoreScript(expected, actual) {
+  // Return a number in [0, 1]; throws are caught and scored as 0.
+  return JSON.stringify(actual) === JSON.stringify(expected) ? 1 : 0;
+}
+`;
+}
+function wrapBodyAsScoreFunction(body) {
+  const trimmed = body.replace(/^\n+|\n+$/g, "");
+  return `/**
+ * Score the workflow output against the example's expected output.
+ *
+ * @param {Expected} expected the example's expected output
+ * @param {Actual} actual the workflow's actual output
+ * @returns {number} a score in [0, 1]; throws are caught and scored as 0
+ */
+function scoreScript(expected, actual) {
+${trimmed}
+}
+`;
+}
+var STRICT_HEADER_REGEX, ANY_HEADER_REGEX;
+var init_score_function = __esm(() => {
+  STRICT_HEADER_REGEX = /^(?:async\s+)?function\s+scoreScript\s*\(\s*expected\s*,\s*actual\s*\)\s*\{/;
+  ANY_HEADER_REGEX = /function\s+scoreScript\s*\([^)]*\)\s*\{/;
+});
+
 // ../types/src/eval/evaluator-config.ts
 function combineEvalScores(results) {
   let numerator = 0;
@@ -23339,9 +23476,10 @@ function combineEvalScores(results) {
     return null;
   return numerator / denominator;
 }
-var EvaluatorTypeSchema, ScoreInUnitInterval, LLM_JUDGE_PROMPT_MIN_LENGTH = 10, ExactDiffConfigSchema, JudgeModeSchema, LabelsRecord, LlmJudgeConfigSchema, CustomScriptConfigSchema, EvaluatorBaseEntrySchema, EvaluatorConfigSchema, EvalConfigYamlSchema, EvaluatorResultSchema;
+var EvaluatorTypeSchema, ScoreInUnitInterval, LLM_JUDGE_PROMPT_MIN_LENGTH = 10, ExactDiffConfigSchema, JudgeModeSchema, LabelsRecord, LlmJudgeConfigSchema, CUSTOM_SCRIPT_CONTRACT, CUSTOM_SCRIPT_MAX_BYTES = 1e4, CUSTOM_SCRIPT_MAX_TIMEOUT_MS = 30000, CUSTOM_SCRIPT_MAX_MEMORY_MB = 50, CUSTOM_SCRIPT_FUNCTION_DESCRIPTION, CustomScriptConfigSchema, EvaluatorBaseEntrySchema, EvaluatorConfigSchema, EvalConfigYamlSchema, EvaluatorResultSchema;
 var init_evaluator_config = __esm(() => {
   init_zod();
+  init_score_function();
   EvaluatorTypeSchema = exports_external.enum(["exact-diff", "llm-judge", "custom-script"]);
   ScoreInUnitInterval = exports_external.number().min(0, { message: "score must be >= 0" }).max(1, { message: "score must be <= 1" });
   ExactDiffConfigSchema = exports_external.object({
@@ -23370,14 +23508,41 @@ var init_evaluator_config = __esm(() => {
       }
     }
   });
+  CUSTOM_SCRIPT_CONTRACT = {
+    expected: "the example's expected output",
+    actual: "the workflow's actual output",
+    returns: "a number in [0, 1]",
+    throws: "caught and scored as 0"
+  };
+  CUSTOM_SCRIPT_FUNCTION_DESCRIPTION = `Full JavaScript declaration of \`function scoreScript(expected, actual) { ... }\`. ` + `Receives \`expected\` (${CUSTOM_SCRIPT_CONTRACT.expected}) ` + `and \`actual\` (${CUSTOM_SCRIPT_CONTRACT.actual}), returns ${CUSTOM_SCRIPT_CONTRACT.returns}. ` + `Throws are ${CUSTOM_SCRIPT_CONTRACT.throws}.`;
   CustomScriptConfigSchema = exports_external.object({
-    code: exports_external.string().min(1, { message: "code required" }).describe("Sandboxed JavaScript. Receives `actual`, `expected`, `input` as variables and must return { score, label? } where score is in [0, 1]."),
-    timeoutMs: exports_external.number().int().positive().default(5000).describe("Maximum wall-clock time the script can run before it's killed and the run is marked failed."),
-    memoryLimitMb: exports_external.number().int().positive().default(10).describe("Maximum memory the sandbox may allocate. Increase if the script processes large arrays or strings."),
+    function: exports_external.string().min(1, { message: "required" }).max(CUSTOM_SCRIPT_MAX_BYTES, {
+      message: `must be ≤ ${CUSTOM_SCRIPT_MAX_BYTES} bytes`
+    }).describe(CUSTOM_SCRIPT_FUNCTION_DESCRIPTION).refine(isParseableScoreFunction, {
+      message: "must parse as JavaScript"
+    }).refine(hasValidScoreSignature, {
+      message: "must declare `function scoreScript(expected, actual) { ... }` (parameter names + order matter)"
+    }).refine((fn) => {
+      const { body, wrapperOk } = extractScoreFunctionBody(fn);
+      return wrapperOk && /\b(return|throw)\b/.test(body);
+    }, {
+      message: "function body must `return` a number in [0, 1] (or `throw` to score 0)"
+    }).superRefine((fn, ctx) => {
+      const issue2 = getScoreFunctionRuntimeIssue(fn);
+      if (issue2)
+        ctx.addIssue({ code: exports_external.ZodIssueCode.custom, message: issue2 });
+    }),
+    timeoutMs: exports_external.number().int().positive().max(CUSTOM_SCRIPT_MAX_TIMEOUT_MS, {
+      message: `must be ≤ ${CUSTOM_SCRIPT_MAX_TIMEOUT_MS} ms`
+    }).default(5000).describe("Maximum wall-clock time the script can run before it's killed and the run is marked failed."),
+    memoryLimitMb: exports_external.number().int().positive().max(CUSTOM_SCRIPT_MAX_MEMORY_MB, {
+      message: `must be ≤ ${CUSTOM_SCRIPT_MAX_MEMORY_MB} MB`
+    }).default(10).describe("Maximum memory the sandbox may allocate. Increase if the script processes large arrays or strings."),
     passThreshold: ScoreInUnitInterval.default(1).describe("Minimum script score this evaluator must reach for a run to pass.")
   });
   EvaluatorBaseEntrySchema = exports_external.object({
     name: exports_external.string().min(1).max(64).regex(/^[a-z0-9-]+$/, "name must be lowercase kebab-case").describe("Identifier used in eval results and the weighted-mean formula. Use a short kebab-case slug like header-exact or covenants-recall."),
+    description: exports_external.string().trim().max(500).optional().describe("One-sentence explanation, in business language, of what this evaluator measures and why a stakeholder should care. Shown in the dashboard to non-technical reviewers (legal, ops, finance) who never see the YAML or judge prompt. Skip implementation detail (model names, thresholds, dot-paths); those live in `config`."),
     weight: exports_external.number().min(0).default(1).describe("Relative importance in the weighted mean. The overall score is Σ(weightᵢ × scoreᵢ) / Σweight. Set to 0 to score-only without affecting overall.")
   });
   EvaluatorConfigSchema = exports_external.discriminatedUnion("type", [
@@ -23463,8 +23628,7 @@ async function judgeScoreJson(params) {
   const response = await client.extract(userContent, {
     schema: SCORE_SCHEMA,
     prompt: systemPrompt,
-    model,
-    temperature: 0
+    model
   });
   const rawScore = response.data.score;
   const score = typeof rawScore === "number" ? rawScore : typeof rawScore === "string" ? parseFloat(rawScore) : NaN;
@@ -23484,8 +23648,7 @@ async function judgeLabelJson(params) {
   const response = await client.extract(userContent, {
     schema: buildLabelSchema(labels),
     prompt: systemPrompt,
-    model,
-    temperature: 0
+    model
   });
   const label = typeof response.data.label === "string" ? response.data.label.trim() : "";
   if (!labels.includes(label)) {
@@ -23569,6 +23732,7 @@ var init_eval = __esm(() => {
   init_dataset_archive();
   init_evaluator_config();
   init_judge();
+  init_score_function();
   init_workflow_tool_metadata();
 });
 
@@ -23637,6 +23801,7 @@ var init_paths = __esm(() => {
 // ../types/src/index.ts
 var exports_src = {};
 __export(exports_src, {
+  wrapBodyAsScoreFunction: () => wrapBodyAsScoreFunction,
   workflowTriggerConfigSchema: () => workflowTriggerConfigSchema,
   valuesMatch: () => valuesMatch,
   validateStepConfig: () => validateStepConfig,
@@ -23667,10 +23832,12 @@ __export(exports_src, {
   isValidJudgeModel: () => isValidJudgeModel,
   isTerminalExecutionStatus: () => isTerminalExecutionStatus,
   isStepCategory: () => isStepCategory,
+  isParseableScoreFunction: () => isParseableScoreFunction,
   isOutputSchemaUnknown: () => isOutputSchemaUnknown,
   isNonSuccessTerminalExecutionStatus: () => isNonSuccessTerminalExecutionStatus,
   isFilePathDescriptor: () => isFilePathDescriptor,
   isFileOutput: () => isFileOutput,
+  hasValidScoreSignature: () => hasValidScoreSignature,
   hasTriggerMethod: () => hasTriggerMethod,
   goldenFileSatisfiedInResult: () => goldenFileSatisfiedInResult,
   getTriggerMethods: () => getTriggerMethods,
@@ -23678,6 +23845,7 @@ __export(exports_src, {
   getStepSchema: () => getStepSchema,
   getStepOutputSchema: () => getStepOutputSchema,
   getStepConfigSchema: () => getStepConfigSchema,
+  getScoreFunctionRuntimeIssue: () => getScoreFunctionRuntimeIssue,
   getRequiredOutputKeys: () => getRequiredOutputKeys,
   getProcessorSchemas: () => getProcessorSchemas,
   getParserCategory: () => getParserCategory,
@@ -23693,11 +23861,13 @@ __export(exports_src, {
   formatDiffValue: () => formatDiffValue,
   fileFieldNamesFromJsonSchema: () => fileFieldNamesFromJsonSchema,
   fieldStemFromGoldenName: () => fieldStemFromGoldenName,
+  extractScoreFunctionBody: () => extractScoreFunctionBody,
   extractFieldMetadata: () => extractFieldMetadata,
   executionWallClockMs: () => executionWallClockMs,
   evalInputKey: () => evalInputKey,
   evalExpectedKey: () => evalExpectedKey,
   errorHintFor: () => errorHintFor,
+  defaultScoreFunction: () => defaultScoreFunction,
   compareOutput: () => compareOutput,
   compareExpectedValues: () => compareExpectedValues,
   combineEvalScores: () => combineEvalScores,
@@ -23870,6 +24040,7 @@ __export(exports_src, {
   DatasetImportRequestSchema: () => DatasetImportRequestSchema,
   DatasetImportModeSchema: () => DatasetImportModeSchema,
   DURABLE_EXECUTION_TYPES: () => DURABLE_EXECUTION_TYPES,
+  DETERMINISTIC_TEMPERATURE: () => DETERMINISTIC_TEMPERATURE,
   DEFAULT_PAGE_SIZE: () => DEFAULT_PAGE_SIZE,
   DEFAULT_CONCURRENCY: () => DEFAULT_CONCURRENCY,
   DATASET_NAME_PATTERN: () => DATASET_NAME_PATTERN,
@@ -23891,6 +24062,10 @@ __export(exports_src, {
   ControlApprovalOutputSchema: () => ControlApprovalOutputSchema,
   ControlApprovalConfigSchema: () => ControlApprovalConfigSchema,
   ConnectorSchema: () => ConnectorSchema,
+  CUSTOM_SCRIPT_MAX_TIMEOUT_MS: () => CUSTOM_SCRIPT_MAX_TIMEOUT_MS,
+  CUSTOM_SCRIPT_MAX_MEMORY_MB: () => CUSTOM_SCRIPT_MAX_MEMORY_MB,
+  CUSTOM_SCRIPT_MAX_BYTES: () => CUSTOM_SCRIPT_MAX_BYTES,
+  CUSTOM_SCRIPT_CONTRACT: () => CUSTOM_SCRIPT_CONTRACT,
   BoundingRegionSchema: () => BoundingRegionSchema,
   BlockStepSchema: () => BlockStepSchema,
   BaseEntitySchema: () => BaseEntitySchema,
@@ -143405,7 +143580,7 @@ import { fileURLToPath as fileURLToPath6 } from "url";
 // package.json
 var package_default = {
   name: "@eigenpal/cli",
-  version: "0.4.8",
+  version: "0.4.10",
   description: "Eigenpal CLI — author and run document workflows from your terminal. Agent-ready.",
   keywords: [
     "eigenpal",
@@ -146148,26 +146323,23 @@ function getExampleNames(dir, exampleNamesFilter) {
 function readInput(exampleDir) {
   const argsPath = join6(exampleDir, "input", "arguments.json");
   const inputDir = join6(exampleDir, "input");
-  const merged = JSON.parse(readFileSync5(argsPath, "utf-8"));
+  const scalars = JSON.parse(readFileSync5(argsPath, "utf-8"));
+  const files = [];
   for (const entry of readdirSync3(inputDir)) {
     const full = join6(inputDir, entry);
     if (!statSync2(full).isDirectory())
       continue;
-    const files = readdirSync3(full).filter((f) => statSync2(join6(full, f)).isFile()).sort();
-    if (files.length === 0)
-      continue;
-    const inlined = files.map((filename) => {
-      const buf = readFileSync5(join6(full, filename));
-      const mime = guessMimeType(filename);
-      return {
-        _inline: buf.toString("base64"),
+    const filenames = readdirSync3(full).filter((f) => statSync2(join6(full, f)).isFile()).sort();
+    for (const filename of filenames) {
+      files.push({
+        argument: entry,
         filename,
-        ...mime ? { mimeType: mime } : {}
-      };
-    });
-    merged[entry] = files.length === 1 ? inlined[0] : inlined;
+        content: readFileSync5(join6(full, filename)),
+        mimeType: guessMimeType(filename)
+      });
+    }
   }
-  return merged;
+  return { scalars, files };
 }
 function readOverrides(exampleDir) {
   const metaPath = join6(exampleDir, "meta.json");
@@ -146181,8 +146353,10 @@ function readOverrides(exampleDir) {
   return null;
 }
 function buildExamplePayload(exampleDir) {
+  const { scalars, files } = readInput(exampleDir);
   return {
-    input: readInput(exampleDir),
+    scalars,
+    files,
     overrides: readOverrides(exampleDir)
   };
 }
@@ -146664,11 +146838,19 @@ async function runExec(client, dir, workflowIdOrSlug, exampleNames, options8 = {
           progress.setRunning(index);
         else
           console.log(`${ui.dim("→")} ${name}`);
-        const res = await client.post(`/api/v1/workflows/${workflowId}/run`, {
-          input: example.input,
-          overrides: example.overrides ?? undefined,
-          trigger: "cli"
-        });
+        const form = new FormData;
+        const sidecar = { trigger: "cli" };
+        if (Object.keys(example.scalars).length > 0)
+          sidecar.input = example.scalars;
+        if (example.overrides)
+          sidecar.overrides = example.overrides;
+        form.append("_json", JSON.stringify(sidecar));
+        for (const file2 of example.files) {
+          form.append(file2.argument, new Blob([file2.content], {
+            type: file2.mimeType ?? "application/octet-stream"
+          }), file2.filename);
+        }
+        const res = await client.postFormData(`/api/v1/workflows/${workflowId}/run`, form);
         if (typeof res?.executionId !== "string") {
           throw new Error("Run API did not return executionId");
         }
@@ -147463,7 +147645,7 @@ var EVALUATOR_REGISTRY = {
   "custom-script": {
     type: "custom-script",
     name: "Custom Script",
-    description: "Run sandboxed JavaScript that returns { score, label? }. Receives `actual`, `expected`, `input`.",
+    description: "Score the workflow output with sandboxed JavaScript. Authors a `function scoreScript(expected, actual) { ... }` declaration that returns a number in [0, 1].",
     configSchema: CustomScriptConfigSchema
   }
 };
@@ -147505,6 +147687,7 @@ function registerEvaluatorTypeCommands(parent) {
       type: def.type,
       name: def.name,
       description: def.description,
+      entrySchema: z12.toJSONSchema(EvaluatorBaseEntrySchema, { target: "draft-7" }),
       configSchema: z12.toJSONSchema(def.configSchema, { target: "draft-7" })
     };
     const text = JSON.stringify(result, null, 2);
@@ -147595,10 +147778,47 @@ function buildBatchDiff(args) {
       sharedExamples: sharedExampleSet.size,
       onlyInA,
       onlyInB,
-      regressionThreshold
+      regressionThreshold,
+      byEvaluator: aggregateByEvaluator(rows)
     }
   };
 }
+function aggregateByEvaluator(rows) {
+  const buckets = new Map;
+  for (const r5 of rows) {
+    const list = buckets.get(r5.evaluator);
+    if (list)
+      list.push(r5);
+    else
+      buckets.set(r5.evaluator, [r5]);
+  }
+  const aggregates = [];
+  for (const [evaluator, group2] of buckets) {
+    const comparable = group2.filter((r5) => r5.delta != null);
+    const meanDelta = comparable.length === 0 ? null : comparable.reduce((sum, r5) => sum + r5.delta, 0) / comparable.length;
+    aggregates.push({
+      evaluator,
+      comparable: comparable.length,
+      meanDelta,
+      regressions: group2.filter((r5) => r5.status === "regression").length,
+      improvements: group2.filter((r5) => r5.status === "improvement").length,
+      unchanged: group2.filter((r5) => r5.status === "unchanged").length
+    });
+  }
+  aggregates.sort((a5, b8) => {
+    if (a5.meanDelta == null && b8.meanDelta == null)
+      return a5.evaluator.localeCompare(b8.evaluator);
+    if (a5.meanDelta == null)
+      return 1;
+    if (b8.meanDelta == null)
+      return -1;
+    const diff = Math.abs(b8.meanDelta) - Math.abs(a5.meanDelta);
+    if (diff !== 0)
+      return diff;
+    return a5.evaluator.localeCompare(b8.evaluator);
+  });
+  return aggregates;
+}
 function sortDiffRows(rows, sort) {
   const tiebreak = (a5, b8) => {
     const exCmp = a5.example.localeCompare(b8.example);
@@ -147684,10 +147904,30 @@ function renderBatchDiffHuman(diff) {
   }
   process.stderr.write(`
 `);
+  if (diff.summary.byEvaluator.length > 0) {
+    process.stderr.write(`${ui.bold("Per-evaluator deltas")} ${ui.dim("(B − A):")}
+`);
+    console.log(table(diff.summary.byEvaluator, [
+      { key: "evaluator", header: "evaluator" },
+      { key: "comparable", header: "rows", align: "right" },
+      {
+        key: "meanDelta",
+        header: "mean Δ",
+        align: "right",
+        format: (v11) => formatDelta(v11)
+      },
+      { key: "regressions", header: "regressions", align: "right" },
+      { key: "improvements", header: "improvements", align: "right" }
+    ]));
+    process.stderr.write(`
+`);
+  }
   if (diff.rows.length === 0) {
     process.stderr.write(`${ui.dim("No shared (example, evaluator) pairs to compare.")}
 `);
   } else {
+    process.stderr.write(`${ui.bold("Per-row deltas:")}
+`);
     console.log(table(diff.rows, [
       { key: "example", header: "example" },
       { key: "evaluator", header: "evaluator" },
@@ -147999,7 +148239,7 @@ flag would otherwise shadow it. Validation failures come back as
         error51(`Cannot update workflow ${ui.bold(workflowId)} — pass --bump <patch|minor|major> or --set-version <X.Y.Z>, or add a top-level \`version: X.Y.Z\` to ${filePath}.`);
         process.exit(1);
       }
-      const yamlToSend = cliWantsRewrite ? spliceWorkflowVersion(yamlOnDisk, nextVersion) : yamlOnDisk;
+      const yamlToSend = cliWantsRewrite ? spliceWorkflowVersion2(yamlOnDisk, nextVersion) : yamlOnDisk;
       const result = await client.post(`/api/v1/workflows/${workflowId}/versions`, {
         yaml: yamlToSend,
         version: nextVersion
@@ -148013,7 +148253,7 @@ flag would otherwise shadow it. Validation failures come back as
       if (opts.bump) {
         throw new Error("--bump requires --workflow-id (need a current version to bump from). For a new workflow, use --set-version <X.Y.Z> or omit both.");
       }
-      const yamlToSend = opts.setVersion && !extractWorkflowVersion(yamlOnDisk) ? spliceWorkflowVersion(yamlOnDisk, opts.setVersion) : yamlOnDisk;
+      const yamlToSend = opts.setVersion && !extractWorkflowVersion(yamlOnDisk) ? spliceWorkflowVersion2(yamlOnDisk, opts.setVersion) : yamlOnDisk;
       const result = await client.post("/api/v1/workflows", { yaml: yamlToSend });
       if (opts.json) {
         printJson(result);
@@ -148040,7 +148280,7 @@ function bumpSemver(current, level) {
     return `${major}.${minor + 1}.0`;
   return `${major}.${minor}.${patch + 1}`;
 }
-function spliceWorkflowVersion(yaml, next) {
+function spliceWorkflowVersion2(yaml, next) {
   if (/^version:\s*['"]?(\d+\.\d+\.\d+)['"]?\s*$/m.test(yaml)) {
     return yaml.replace(/^version:.*$/m, `version: ${next}`);
   }
@@ -148444,7 +148684,7 @@ returns immediately with \`{ batchId, total }\`; poll
         return;
       }
       success2(`Started experiment ${ui.bold(start.batchId)} ${ui.dim(`(${start.total} execution${start.total === 1 ? "" : "s"} queued)`)}`);
-      process.stderr.write(`   ${ui.dim(`Watch: eigenpal workflow experiment status ${workflowId} ${start.batchId} --watch`)}
+      process.stderr.write(`   ${ui.dim(`Watch + auto-pull: eigenpal workflow experiment watch ${workflowId} ${start.batchId}`)}
 `);
       return;
     }
@@ -148461,20 +148701,32 @@ returns immediately with \`{ batchId, total }\`; poll
       await new Promise((r5) => setTimeout(r5, intervalMs));
     }
   }));
-  const statusCmd = experiment.command("status <workflow-id> <batchId>").description(descriptionFor("workflow_get_experiment_status")).option("--watch", "Poll until every execution reaches a terminal state (completed/failed/cancelled/rejected), then exit", false).option("--interval <seconds>", "Poll interval in seconds when --watch is set (default 5)", intArg, 5).option("--max-wait <seconds>", "Hard ceiling for --watch in seconds (default 1800 = 30 min)", intArg, 30 * 60).option("--include <kinds>", "Comma-separated extras to attach when --watch terminates: payload (full per-execution snapshot, can be hundreds of KB)", "");
+  const statusCmd = experiment.command("status <workflow-id> <batchId>").description(descriptionFor("workflow_get_experiment_status")).option("--watch", "Poll until every execution reaches a terminal state (completed/failed/cancelled/rejected), then exit", false).option("--short", "Single-line plain-text summary on stdout (e.g. `6/6 done failed=0 cancelled=0 rejected=0`). Pipe-friendly for monitoring loops; mutually exclusive with --watch.", false).option("--interval <seconds>", "Poll interval in seconds when --watch is set (default 5)", intArg, 5).option("--max-wait <seconds>", "Hard ceiling for --watch in seconds (default 1800 = 30 min)", intArg, 30 * 60).option("--include <kinds>", "Comma-separated extras to attach when --watch terminates: payload (full per-execution snapshot, can be hundreds of KB)", "");
   addJsonFlag(withBaseUrl(statusCmd)).action(action(async (workflow2, batchId, opts) => {
+    if (opts.short && opts.watch) {
+      error51("--short and --watch are mutually exclusive (short is for spot-check polling).");
+      process.exit(2);
+    }
     const includes = new Set((opts.include ?? "").split(",").map((s5) => s5.trim()).filter(Boolean));
     const includePayload = includes.has("payload");
     const { client, workflowId } = await buildClientForWorkflow(workflow2, opts);
     const url3 = `/api/v1/workflows/${workflowId}/experiments/${batchId}`;
     if (!opts.watch) {
       const result = await client.get(url3);
+      const execs = result.executions ?? [];
+      const summary = summarizeExperimentExecutions(execs);
       if (opts.json) {
-        printJson(result);
+        printJson({ ...result, summary: rollupForJson(summary) });
+        return;
+      }
+      if (opts.short) {
+        console.log(formatShortStatus(summary));
+        const failed2 = summary.counts.failed ?? 0;
+        const cancelled2 = summary.counts.cancelled ?? 0;
+        if (summary.done && failed2 + cancelled2 > 0)
+          process.exit(1);
         return;
       }
-      const execs = result.executions ?? [];
-      const summary = summarizeExperimentExecutions(execs);
       const counts = Object.entries(summary.counts).map(([k9, v11]) => `${k9}=${v11}`).sort().join(" ");
       console.log(`${ui.bold(batchId)}  ${summary.terminal}/${summary.total} terminal  ${ui.dim(`(${counts || "no executions"})`)}`);
       const failed = summary.counts.failed ?? 0;
@@ -148488,56 +148740,25 @@ returns immediately with \`{ batchId, total }\`; poll
         process.exit(1);
       return;
     }
-    const deadline = Date.now() + opts.maxWait * 1000;
-    let lastSummary = "";
-    while (true) {
-      const result = await client.get(url3);
-      const execs = result.executions ?? [];
-      const summary = summarizeExperimentExecutions(execs);
-      const counts = Object.entries(summary.counts).map(([k9, v11]) => `${k9}=${v11}`).sort().join(" ");
-      const line3 = `${summary.terminal}/${summary.total} terminal  ${ui.dim(`(${counts || "no executions"})`)}`;
-      if (line3 !== lastSummary) {
-        if (process.stderr.isTTY) {
-          process.stderr.write(`\r\x1B[K${line3}`);
-        } else {
-          process.stderr.write(`${line3}
-`);
-        }
-        lastSummary = line3;
-      }
-      const wantPayload = opts.json || includePayload;
-      if (summary.done) {
-        if (process.stderr.isTTY)
-          process.stderr.write(`
-`);
-        const failed = summary.counts.failed ?? 0;
-        const cancelled = summary.counts.cancelled ?? 0;
-        const rejected = summary.counts.rejected ?? 0;
-        const badCount = failed + cancelled + rejected;
-        if (badCount > 0) {
-          renderExperimentFailures(execs, summary.total);
-        }
+    const wantPayload = opts.json || includePayload;
+    await pollExperimentUntilTerminal({
+      client,
+      url: url3,
+      interval: opts.interval,
+      maxWait: opts.maxWait,
+      onTerminal: (result, summary) => {
         if (wantPayload) {
-          printJson(result);
+          printJson({ ...result, summary: rollupForJson(summary) });
         } else {
           process.stderr.write(`${ui.dim(`use --include payload (or --json) for the full per-execution snapshot`)}
 `);
         }
-        if (badCount > 0)
-          process.exit(1);
-        return;
-      }
-      if (Date.now() >= deadline) {
-        if (process.stderr.isTTY)
-          process.stderr.write(`
-`);
-        error51(`Timed out after ${opts.maxWait}s with ${summary.total - summary.terminal} non-terminal execution(s). Re-run to keep watching.`);
+      },
+      onDeadline: (result, summary) => {
         if (wantPayload)
-          printJson(result);
-        process.exit(2);
+          printJson({ ...result, summary: rollupForJson(summary) });
       }
-      await new Promise((r5) => setTimeout(r5, opts.interval * 1000));
-    }
+    });
   }));
   const cancelCmd = experiment.command("cancel <workflow-id> <batchId>").description("Cancel every execution in a batch. Idempotent.").option("--yes", "Required for non-TTY shells (CI, pipes). Acts immediately, no prompt.").addHelpText("after", `
 Examples:
@@ -148626,6 +148847,62 @@ batches must belong to the same workflow within the caller's tenant.
     }
     renderBatchDiffHuman(diff);
   }));
+  const watchCmd = experiment.command("watch <workflow-id> <batchId>").description("Poll until terminal, then auto-pull results — replaces `status --watch` + `results --out`.").option("--interval <seconds>", "Poll interval in seconds (default 5)", intArg, 5).option("--max-wait <seconds>", "Hard ceiling in seconds (default 1800 = 30 min)", intArg, 30 * 60).option("--pull-on-complete <path>", "Destination for the results file. Default: ./results-<batchId>.<format>").option("--format <csv|json>", "Results export format (default json)", parseExportFormat, "json").option("--no-pull", "Skip auto-pulling results on terminal (just watch)").addHelpText("after", `
+Examples:
+  $ eigenpal workflow experiment watch wf_abc123 evb_xyz789
+  $ eigenpal workflow experiment watch wf_abc123 evb_xyz789 --format csv
+  $ eigenpal workflow experiment watch wf_abc123 evb_xyz789 --pull-on-complete ./out/r.json
+  $ eigenpal workflow experiment watch wf_abc123 evb_xyz789 --no-pull
+
+Behavior:
+  - Polls every --interval seconds; redraws a one-line tick on a TTY,
+    appends a new line per tick when piped.
+  - On terminal: renders the failed-execution detail block (if any), then
+    downloads results to ./results-<batchId>.<format> unless --no-pull
+    or --pull-on-complete <path> says otherwise.
+  - 4xx errors (e.g. unknown batch id) abort immediately rather than spin.
+  - Pre-flight check: the destination directory must exist before the
+    poll loop starts, so a typo doesn't waste a 30-min wait.
+  - Exit codes (matching \`experiment status --watch\`):
+      0  every execution succeeded (or pull skipped via --no-pull on a clean batch)
+      1  at least one execution ended in failed/cancelled/rejected,
+         OR the export download failed after a clean batch
+      2  --max-wait deadline reached — re-run to keep watching;
+         OR misuse (e.g. --no-pull combined with --pull-on-complete)
+`);
+  withBaseUrl(watchCmd).action(action(async (workflow2, batchId, opts) => {
+    if (!opts.pull && opts.pullOnComplete) {
+      error51("--no-pull and --pull-on-complete are mutually exclusive — pass one or the other.");
+      process.exit(2);
+    }
+    const { client, workflowId } = await buildClientForWorkflow(workflow2, opts);
+    const url3 = `/api/v1/workflows/${workflowId}/experiments/${batchId}`;
+    const dest = opts.pull ? opts.pullOnComplete ?? `./results-${batchId}.${opts.format}` : null;
+    if (dest) {
+      const dir = path15.dirname(dest);
+      if (!existsSync9(dir)) {
+        error51(`Destination directory does not exist: ${dir}. Create it first or pass a path under an existing directory.`);
+        process.exit(2);
+      }
+    }
+    await pollExperimentUntilTerminal({
+      client,
+      url: url3,
+      interval: opts.interval,
+      maxWait: opts.maxWait,
+      onTerminal: async () => {
+        if (!dest)
+          return;
+        const params = new URLSearchParams({ format: opts.format, batchId });
+        try {
+          await downloadStreamToFile(client, `/api/v1/workflows/${workflowId}/eval-results/export?${params.toString()}`, dest);
+        } catch (err2) {
+          error51(`Watch finished but pulling results failed: ${err2 instanceof Error ? err2.message : String(err2)}. Re-run \`experiment results <wf> <batch>\` to retry.`);
+          process.exit(1);
+        }
+      }
+    });
+  }));
 }
 function renderExperimentFailures(execs, totalCount) {
   const failedExecs = execs.filter((e8) => e8.status === "failed" || e8.status === "cancelled" || e8.status === "rejected");
@@ -148664,6 +148941,83 @@ function summarizeExperimentExecutions(execs) {
     done: execs.length > 0 && terminal === execs.length
   };
 }
+function rollupForJson(summary) {
+  return {
+    total: summary.total,
+    terminal: summary.terminal,
+    complete: summary.done,
+    completedCount: summary.counts.completed ?? 0,
+    failedCount: summary.counts.failed ?? 0,
+    cancelledCount: summary.counts.cancelled ?? 0,
+    rejectedCount: summary.counts.rejected ?? 0,
+    runningCount: summary.counts.running ?? 0,
+    pendingCount: summary.counts.pending ?? 0
+  };
+}
+function formatShortStatus(summary) {
+  const state = summary.done ? "done" : "in-progress";
+  return [
+    `${summary.terminal}/${summary.total}`,
+    state,
+    `failed=${summary.counts.failed ?? 0}`,
+    `cancelled=${summary.counts.cancelled ?? 0}`,
+    `rejected=${summary.counts.rejected ?? 0}`
+  ].join(" ");
+}
+function renderTickLine(summary) {
+  const counts = Object.entries(summary.counts).map(([k9, v11]) => `${k9}=${v11}`).sort().join(" ");
+  return `${summary.terminal}/${summary.total} terminal  ${ui.dim(`(${counts || "no executions"})`)}`;
+}
+async function pollExperimentUntilTerminal({
+  client,
+  url: url3,
+  interval,
+  maxWait,
+  onTerminal,
+  onDeadline
+}) {
+  const deadline = Date.now() + maxWait * 1000;
+  let lastSummary = "";
+  while (true) {
+    const result = await client.get(url3);
+    const execs = result.executions ?? [];
+    const summary = summarizeExperimentExecutions(execs);
+    const line3 = renderTickLine(summary);
+    if (line3 !== lastSummary) {
+      if (process.stderr.isTTY) {
+        process.stderr.write(`\r\x1B[K${line3}`);
+      } else {
+        process.stderr.write(`${line3}
+`);
+      }
+      lastSummary = line3;
+    }
+    if (summary.done) {
+      if (process.stderr.isTTY)
+        process.stderr.write(`
+`);
+      const failed = summary.counts.failed ?? 0;
+      const cancelled = summary.counts.cancelled ?? 0;
+      const rejected = summary.counts.rejected ?? 0;
+      const badCount = failed + cancelled + rejected;
+      if (badCount > 0)
+        renderExperimentFailures(execs, summary.total);
+      await onTerminal(result, summary);
+      if (badCount > 0)
+        process.exit(1);
+      return;
+    }
+    if (Date.now() >= deadline) {
+      if (process.stderr.isTTY)
+        process.stderr.write(`
+`);
+      error51(`Timed out after ${maxWait}s with ${summary.total - summary.terminal} non-terminal execution(s). Re-run to keep watching.`);
+      onDeadline?.(result, summary);
+      process.exit(2);
+    }
+    await new Promise((r5) => setTimeout(r5, interval * 1000));
+  }
+}
 function registerVersionsCommands(parent) {
   const versions2 = parent.command("versions").description("Inspect and restore historical workflow versions.");
   const listCmd = versions2.command("list <workflow-id>").description(descriptionFor("workflow_list_versions"));

package/dist/skill/SKILL.md

@@ -145,6 +145,17 @@ to retry.
 eigenpal workflow experiment compare evb_old evb_new --regression-threshold 0.05
 ```
 
+Output is two stacked tables: a per-evaluator aggregate (rows per evaluator
+with mean Δ, regressions, improvements — sorted biggest mover first) followed
+by a per-(example, evaluator) detail table. No need to write a Python
+aggregator on top.
+
+```bash
+# JSON exposes both layers — `summary.byEvaluator` for the rollup,
+# `rows` for the full per-example detail.
+eigenpal workflow experiment compare evb_old evb_new --json | jq '.summary.byEvaluator'
+```
+
 Unlike its siblings, `compare` takes no `<workflow-id>` — the server resolves
 the owning workflow from each batch id. Both batches must live in the
 same workflow within your tenant. See [`reference/debugging.md`](reference/debugging.md#6-compare-two-experiment-batches)
@@ -163,6 +174,12 @@ eigenpal workflow evaluators validate ./evaluators.yaml
 eigenpal workflow evaluators push <workflow-id> --file evaluators.yaml
 ```
 
+Always set `description:` on every evaluator entry. It's a short, plain-language
+sentence (≤500 chars; aim for one line) explaining what the evaluator measures
+and why it matters, written for a non-technical reviewer in the dashboard.
+Skip model names, thresholds, and dot-paths; those belong in `config`. See
+[`reference/evaluators.md`](reference/evaluators.md#writing-descriptions-for-stakeholders).
+
 ### Build + validate + upload a dataset
 
 ```bash
@@ -218,12 +235,64 @@ eigenpal workflow experiment run <workflow-id>                       # → { bat
 
 # Or restrict to one example:
 eigenpal workflow experiment run <workflow-id> --example-id evx_…
+```
+
+**Recommended:** use `experiment watch` — it polls until terminal AND auto-pulls
+results to disk in one command. Replaces the old "run → status (poll) → results"
+chain so agents don't have to write a custom poller, parse `executions[].status`,
+or chain `monitor + status + pull + score` shell logic.
+
+```bash
+# One command: watch until terminal, then write ./results-<batchId>.json on completion.
+eigenpal workflow experiment watch <workflow-id> <batch-id>
+
+# CSV output:
+eigenpal workflow experiment watch <workflow-id> <batch-id> --format csv
+
+# Custom destination (overrides the default ./results-<batchId>.<format>):
+eigenpal workflow experiment watch <workflow-id> <batch-id> --pull-on-complete ./out/r.json
+
+# Just watch, skip the pull (e.g. you only need the live tick + exit code):
+eigenpal workflow experiment watch <workflow-id> <batch-id> --no-pull
+```
 
-# Poll until done. `experiment status` auto-detaches after 30 min; just re-run.
-eigenpal workflow experiment status <workflow-id> <batch-id>
+Exit codes match `experiment status --watch`: `0` clean, `1` any
+failed/cancelled/rejected execution, `2` `--max-wait` deadline reached
+(30 min default — re-run to keep watching).
 
-# Pull results — `--out` writes the file directly from the server's signed-URL
-# storage, so large CSVs don't go through the CLI process.
+### Monitoring scripts — use `--short` and the JSON rollup
+
+When a script needs to spot-check a batch (no waiting), prefer one of the two
+machine-readable paths instead of folding `executions[].status` yourself.
+
+```bash
+# Single line on stdout, awk-friendly:
+eigenpal workflow experiment status <wf> <batch> --short
+# → 6/6 done failed=0 cancelled=0 rejected=0   (or "in-progress" while pending)
+
+# Parse with awk:
+DONE_STATE=$(eigenpal workflow experiment status <wf> <batch> --short | awk '{print $2}')
+[ "$DONE_STATE" = "done" ] || exit 0   # not terminal yet, try again later
+```
+
+```bash
+# JSON includes a top-level `summary` rollup so you don't count statuses by hand:
+eigenpal workflow experiment status <wf> <batch> --json | jq '.summary'
+# → { total: 6, terminal: 6, complete: true,
+#     completedCount: 5, failedCount: 1,
+#     cancelledCount: 0, rejectedCount: 0,
+#     runningCount: 0, pendingCount: 0 }
+
+# `summary.complete` is the canonical poll-completion flag.
+```
+
+`--short` and `--json` are mutually exclusive with `--watch` — for live
+streaming use `experiment watch` instead.
+
+If you need the raw eval-results export after watching with `--no-pull`, or
+to refetch results outside a watch loop, fall back to:
+
+```bash
 eigenpal workflow experiment results <workflow-id> <batch-id> --format csv --out r.csv
 ```
 
@@ -433,10 +502,11 @@ eigenpal workflow clear-local                 # delete local execution artifacts
 
 # Experiments (batch eval runs)
 eigenpal workflow experiment run     <workflow-id> [--example-id <id> ...]
-eigenpal workflow experiment status  <workflow-id> <batch-id> [--watch]    # exit 1 on terminal w/ failures, 2 on --max-wait
+eigenpal workflow experiment status  <workflow-id> <batch-id> [--watch] [--short] [--json]    # exit 1 on terminal w/ failures, 2 on --max-wait. --json includes top-level `summary` rollup.
+eigenpal workflow experiment watch   <workflow-id> <batch-id> [--format csv|json] [--pull-on-complete <path>] [--no-pull]    # poll + auto-pull results in one shot
 eigenpal workflow experiment results <workflow-id> [batch-id] --format {csv|json} --out r.csv
 eigenpal workflow experiment list    <workflow-id>
-eigenpal workflow experiment compare <batch-a> <batch-b> [--regression-threshold] [--sort]   # no --workflow-id
+eigenpal workflow experiment compare <batch-a> <batch-b> [--regression-threshold] [--sort]   # no --workflow-id; prints per-evaluator aggregate + per-row table
 
 # Step-type / evaluator-type introspection
 eigenpal workflow step-type      list

package/dist/skill/reference/cli/workflow.md

@@ -38,6 +38,7 @@ Manage workflows: push, pull, run, evaluate.
   - [`eigenpal workflow experiment cancel [options] <workflow-id> <batchId>`](#eigenpal-workflow-experiment-cancel-options-workflow-id-batchid)
   - [`eigenpal workflow experiment results [options] <workflow-id> [batchId]`](#eigenpal-workflow-experiment-results-options-workflow-id-batchid)
   - [`eigenpal workflow experiment compare [options] <batchIdA> <batchIdB>`](#eigenpal-workflow-experiment-compare-options-batchida-batchidb)
+  - [`eigenpal workflow experiment watch [options] <workflow-id> <batchId>`](#eigenpal-workflow-experiment-watch-options-workflow-id-batchid)
   - [`eigenpal workflow execution run [options] <workflow-id> [examples...]`](#eigenpal-workflow-execution-run-options-workflow-id-examples)
   - [`eigenpal workflow execution get [options] <executionId>`](#eigenpal-workflow-execution-get-options-executionid)
   - [`eigenpal workflow execution list [options] <workflow-id>`](#eigenpal-workflow-execution-list-options-workflow-id)
@@ -79,7 +80,8 @@ workflow
 │   ├── status <workflow-id> <batchId>
 │   ├── cancel <workflow-id> <batchId>
 │   ├── results <workflow-id> [batchId]
-│   └── compare <batchIdA> <batchIdB>
+│   ├── compare <batchIdA> <batchIdB>
+│   └── watch <workflow-id> <batchId>
 ├── execution
 │   ├── run <workflow-id> [examples...]
 │   ├── get <executionId>
@@ -137,14 +139,15 @@ workflow
 
 ### Experiment
 
-| Command                                                                  | Description                                        |
-| ------------------------------------------------------------------------ | -------------------------------------------------- |
-| `eigenpal workflow experiment list [options] <workflow-id>`              | List executions for the workflow, newest first.    |
-| `eigenpal workflow experiment run [options] <workflow-id>`               | Start a batch eval against the workflow's dataset. |
-| `eigenpal workflow experiment status [options] <workflow-id> <batchId>`  | Aggregate progress for a batch by `batchId`.       |
-| `eigenpal workflow experiment cancel [options] <workflow-id> <batchId>`  | Cancel every execution in a batch. Idempotent.     |
-| `eigenpal workflow experiment results [options] <workflow-id> [batchId]` | Download eval results in CSV or JSON.              |
-| `eigenpal workflow experiment compare [options] <batchIdA> <batchIdB>`   | Diff eval scores between two experiment batches.   |
+| Command                                                                  | Description                                                                                |
+| ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------ |
+| `eigenpal workflow experiment list [options] <workflow-id>`              | List executions for the workflow, newest first.                                            |
+| `eigenpal workflow experiment run [options] <workflow-id>`               | Start a batch eval against the workflow's dataset.                                         |
+| `eigenpal workflow experiment status [options] <workflow-id> <batchId>`  | Aggregate progress for a batch by `batchId`.                                               |
+| `eigenpal workflow experiment cancel [options] <workflow-id> <batchId>`  | Cancel every execution in a batch. Idempotent.                                             |
+| `eigenpal workflow experiment results [options] <workflow-id> [batchId]` | Download eval results in CSV or JSON.                                                      |
+| `eigenpal workflow experiment compare [options] <batchIdA> <batchIdB>`   | Diff eval scores between two experiment batches.                                           |
+| `eigenpal workflow experiment watch [options] <workflow-id> <batchId>`   | Poll until terminal, then auto-pull results — replaces `status --watch` + `results --out`. |
 
 ### Execution
 
@@ -508,14 +511,15 @@ Aggregate progress for a batch by `batchId`.
 
 ### Options
 
-| Flag                   | Required | Default | Description                                                                                                            |
-| ---------------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------- |
-| `--watch`              | no       | `false` | Poll until every execution reaches a terminal state (completed/failed/cancelled/rejected), then exit                   |
-| `--interval <seconds>` | no       | `5`     | Poll interval in seconds when --watch is set (default 5)                                                               |
-| `--max-wait <seconds>` | no       | `1800`  | Hard ceiling for --watch in seconds (default 1800 = 30 min)                                                            |
-| `--include <kinds>`    | no       | `""`    | Comma-separated extras to attach when --watch terminates: payload (full per-execution snapshot, can be hundreds of KB) |
-| `--base-url <url>`     | no       |         | Server base URL                                                                                                        |
-| `--json`               | no       |         | Output the raw server response as JSON                                                                                 |
+| Flag                   | Required | Default | Description                                                                                                                                                      |
+| ---------------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `--watch`              | no       | `false` | Poll until every execution reaches a terminal state (completed/failed/cancelled/rejected), then exit                                                             |
+| `--short`              | no       | `false` | Single-line plain-text summary on stdout (e.g. `6/6 done failed=0 cancelled=0 rejected=0`). Pipe-friendly for monitoring loops; mutually exclusive with --watch. |
+| `--interval <seconds>` | no       | `5`     | Poll interval in seconds when --watch is set (default 5)                                                                                                         |
+| `--max-wait <seconds>` | no       | `1800`  | Hard ceiling for --watch in seconds (default 1800 = 30 min)                                                                                                      |
+| `--include <kinds>`    | no       | `""`    | Comma-separated extras to attach when --watch terminates: payload (full per-execution snapshot, can be hundreds of KB)                                           |
+| `--base-url <url>`     | no       |         | Server base URL                                                                                                                                                  |
+| `--json`               | no       |         | Output the raw server response as JSON                                                                                                                           |
 
 ### `eigenpal workflow experiment cancel [options] <workflow-id> <batchId>`
 
@@ -575,6 +579,28 @@ Diff eval scores between two experiment batches.
 | `--base-url <url>`                                     | no       |                    | Server base URL                                        |
 | `--json`                                               | no       |                    | Output the raw server response as JSON                 |
 
+### `eigenpal workflow experiment watch [options] <workflow-id> <batchId>`
+
+Poll until terminal, then auto-pull results — replaces `status --watch` + `results --out`.
+
+### Arguments
+
+| Name          | Required | Variadic | Description |
+| ------------- | -------- | -------- | ----------- |
+| `workflow-id` | yes      | no       |             |
+| `batchId`     | yes      | no       |             |
+
+### Options
+
+| Flag                        | Required | Default  | Description                                                             |
+| --------------------------- | -------- | -------- | ----------------------------------------------------------------------- |
+| `--interval <seconds>`      | no       | `5`      | Poll interval in seconds (default 5)                                    |
+| `--max-wait <seconds>`      | no       | `1800`   | Hard ceiling in seconds (default 1800 = 30 min)                         |
+| `--pull-on-complete <path>` | no       |          | Destination for the results file. Default: ./results-<batchId>.<format> |
+| `--format <csv\|json>`      | no       | `"json"` | Results export format (default json)                                    |
+| `--no-pull`                 | no       |          | Skip auto-pulling results on terminal (just watch)                      |
+| `--base-url <url>`          | no       |          | Server base URL                                                         |
+
 ### `eigenpal workflow execution run [options] <workflow-id> [examples...]`
 
 Run a saved workflow against local dataset examples.

package/dist/skill/reference/evaluators.md

@@ -11,16 +11,41 @@ this is the agent-facing summary.
 ```yaml
 evaluators: # array of evaluator definitions, ≥1 entry
   - name: ... # required, unique within this file, [a-z0-9][a-z0-9-_]*
+    description: ... # optional, ≤500 chars; one-sentence business explanation (see below)
     type: ... # required, one of: exact-diff | llm-judge | custom-script
     config: # required, shape depends on `type`
       passThreshold: 0.95 # 0..1, optional (default 1.0)
       ...
 
-# OPTIONAL global config
-concurrency: 4 # max examples in parallel during `experiment run` (default 1)
-judge: # optional shared model defaults for llm-judge evaluators
-  model: gpt-4o
-  temperature: 0
+# OPTIONAL workflow-level pass threshold. A run passes when its weighted-mean
+# score across evaluators clears this number. Defaults to 0.7.
+passThreshold: 0.7
+```
+
+## Writing descriptions for stakeholders
+
+Every evaluator entry takes an optional `description`. Always set it. The
+dashboard surfaces this string to non-technical reviewers (legal, ops,
+finance) who never see the YAML or the judge prompt. Write it for them:
+
+- One short sentence, plain language.
+- Name what the evaluator measures and why a business reader should care.
+- Skip implementation detail (model names, thresholds, dot-paths). Those
+  belong in `config`.
+- Keep it under ~120 characters when you can; the schema caps it at 500.
+
+```yaml
+# Good. A non-technical reviewer immediately grasps the intent.
+- name: covenants-recall
+  description: Checks every covenant from the contract appears in the extraction.
+  type: llm-judge
+  config: ...
+
+# Bad. Restates the config; teaches a stakeholder nothing.
+- name: covenants-recall
+  description: llm-judge with passThreshold 0.85 over the covenants array.
+  type: llm-judge
+  config: ...
 ```
 
 ## Evaluator types
@@ -47,23 +72,34 @@ judge: # optional shared model defaults for llm-judge evaluators
 - name: covenants-recall
   type: llm-judge
   config:
-    model: gpt-4o # string — the OpenAI model id
-    temperature: 0 # number — for reproducibility
+    model: gpt-4o # OPTIONAL — falls back to the workspace default LLM provider
+    mode: continuous # continuous | discrete (default: continuous)
     passThreshold: 0.85
-    prompt: |
+    docLabel: extraction # what the judge calls the workflow output (default: "output")
+    promptExtension: |
       Score how well the actual output captures every covenant in the
-      expected output, on a scale of 0..1. Penalise missing covenants;
-      do not penalise over-extraction. Return JSON { score, rationale }.
-```
+      expected output. Penalise missing covenants; do not penalise
+      over-extraction. Be strict on covenants that change loan economics.
 
-The harness wraps your `prompt` and feeds:
-
-- `actual` — the workflow's output
-- `expected` — the example's `expected/output.json`
-- `meta` — example metadata
+# discrete mode — the judge picks a label and the score table maps to a number
+- name: tone-check
+  type: llm-judge
+  config:
+    mode: discrete
+    passThreshold: 0.7
+    labels:
+      excellent: 1.0
+      acceptable: 0.7
+      off-tone: 0.0
+    promptExtension: |
+      Pick the label that best matches the tone of the response.
+```
 
-It expects the model to return JSON with at least `score` (0..1) and
-`rationale` (string). The fixed harness handles parse failures + retries.
+`promptExtension` is the only required field — it's appended to the harness
+prompt and tells the judge what "good" looks like for this workflow. The
+harness handles JSON parsing, retries, and feeds the judge `actual`,
+`expected`, and example metadata. In `discrete` mode the judge MUST return
+one of the keys in `labels`; the harness then looks up the score.
 
 ### `custom-script` — JavaScript in the sandbox
 
@@ -72,16 +108,38 @@ It expects the model to return JSON with at least `score` (0..1) and
   type: custom-script
   config:
     passThreshold: 1.0
-    code: |
-      const a = actual.totalAmount ?? 0;
-      const e = expected.totalAmount ?? 0;
-      const score = Math.abs(a - e) <= 0.01 ? 1 : 0;
-      return { score, rationale: `actual=${a}, expected=${e}` };
+    function: |
+      /**
+       * @param {Expected} expected the example's expected output
+       * @param {Actual} actual the workflow's actual output
+       * @returns {number} a score in [0, 1]; throws are caught and scored as 0
+       */
+      function scoreScript(expected, actual) {
+        const a = actual.totalAmount ?? 0;
+        const e = expected.totalAmount ?? 0;
+        return Math.abs(a - e) <= 0.01 ? 1 : 0;
+      }
 ```
 
 Same WASM sandbox as `transform.script` (5s wall clock, 10 MB heap, no
-network). Receives `actual`, `expected`, `meta` in scope. Returns
-`{ score: number, label?: string, rationale?: string }`.
+network). The YAML stores the **whole** `function scoreScript(expected,
+actual) { ... }` declaration verbatim — same text the dashboard editor
+shows. The JSDoc preamble is optional but recommended: it pulls the
+workflow's output schema into IntelliSense for `expected.…` /
+`actual.…` in the dashboard.
+
+The signature shape is **locked**:
+
+- Function name MUST be `scoreScript`.
+- Parameters MUST be `(expected, actual)` in that order. Swapping them
+  would silently invert every score.
+- Body MUST `return` a number in `[0, 1]`, or `throw` (which is caught
+  and scored as 0).
+
+`workflow validate` and `workflow evaluators validate` flag YAML where
+the function fails to parse, the signature doesn't match, or the body
+contains neither `return` nor `throw` — so a hand-edited config errors
+before it reaches the worker.
 
 ## Pass / fail vs score
 
@@ -160,9 +218,11 @@ process — `--out` writes the file directly from the server's storage.
 LLM-judge scores have variance. To reduce flake:
 
 - Pin a specific model id (`gpt-4o-2024-08-06` rather than `gpt-4o`).
-- Set `temperature: 0`.
-- Keep the prompt narrow — score one dimension per evaluator. Use
+- Keep `promptExtension` narrow — score one dimension per evaluator. Use
   multiple judges instead of one omnibus prompt.
+- Prefer `mode: discrete` with a small label set when the judgement is
+  categorical (good / acceptable / bad) — it eliminates the model's
+  free-form numeric drift.
 - For high-stakes thresholds, run the experiment ≥3 times and average.
 
 <!-- GENERATED:EVALUATOR_CATALOG START -->
@@ -177,6 +237,14 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
 | `evaluators` | array<object> | no | `[]` |  |
 | `passThreshold` | number | no | `0.7` |  |
 
+### Common entry fields (every evaluator type)
+
+| Field | Type | Required | Default | Description |
+| --- | --- | --- | --- | --- |
+| `name` | string | yes |  | Identifier used in eval results and the weighted-mean formula. Use a short kebab-case slug like header-exact or covenants-recall. |
+| `description` | string | no |  | One-sentence explanation, in business language, of what this evaluator measures and why a stakeholder should care. Shown in the dashboard to non-technical reviewers (legal, ops, finance) who never see the YAML or judge prompt. Skip implementation detail (model names, thresholds, dot-paths); those live in `config`. |
+| `weight` | number | no | `1` | Relative importance in the weighted mean. The overall score is Σ(weightᵢ × scoreᵢ) / Σweight. Set to 0 to score-only without affecting overall. |
+
 ### `exact-diff` — JSON deep-diff against expected output
 
 | Field | Type | Required | Default | Description |
@@ -203,7 +271,7 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
 
 | Field | Type | Required | Default | Description |
 | --- | --- | --- | --- | --- |
-| `code` | string | yes |  | Sandboxed JavaScript. Receives `actual`, `expected`, `input` as variables and must return { score, label? } where score is in [0, 1]. |
+| `function` | string | yes |  | Full JavaScript declaration of `function scoreScript(expected, actual) { ... }`. Receives `expected` (the example's expected output) and `actual` (the workflow's actual output), returns a number in [0, 1]. Throws are caught and scored as 0. |
 | `timeoutMs` | integer | no | `5000` | Maximum wall-clock time the script can run before it's killed and the run is marked failed. |
 | `memoryLimitMb` | integer | no | `10` | Maximum memory the sandbox may allocate. Increase if the script processes large arrays or strings. |
 | `passThreshold` | number | no | `1` | Minimum script score this evaluator must reach for a run to pass. |

package/dist/skill/reference/step-types.md

@@ -153,7 +153,6 @@ Extract structured data from text using AI with a JSON schema
 | `prompt` | string | no |  | Custom prompt template for extraction |
 | `provider` | string | no |  | Provider ID (e.g., "openai-gpt4o") |
 | `model` | string | no |  | Model override |
-| `temperature` | number | no | `0` | Model temperature (0 = deterministic) |
 | `maxInputTokens` | integer | no |  | Max input tokens. Truncates input text and logs a warning when exceeded. Omit for no limit. |
 
 **Output:** `record<string, unknown>`

package/dist/skill/reference/workflow-yaml.md

@@ -387,6 +387,7 @@ _Generated from `WorkflowDefinitionSchema` in `@eigenpal/types/src/workflow/work
 | `steps` | array<unknown> | yes |  |  |
 | `output` | record<string, string> | no |  |  |
 | `settings` | object | no |  |  |
+| `defaultModel` | string | no |  |  |
 
 
 ## Per-input fields

package/dist/templates/pdf-extraction/evaluators.yaml

@@ -1,5 +1,6 @@
 evaluators:
   - name: exact-match
+    description: Checks the extracted fields match the expected output exactly.
     type: exact-diff
     config:
       passThreshold: 1.0

package/dist/templates/text-classification/evaluators.yaml

@@ -1,5 +1,6 @@
 evaluators:
   - name: label-match
+    description: Checks the predicted label matches the expected label.
     type: exact-diff
     config:
       passThreshold: 1.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@eigenpal/cli",
-  "version": "0.4.8",
+  "version": "0.4.10",
   "description": "Eigenpal CLI — author and run document workflows from your terminal. Agent-ready.",
   "keywords": [
     "eigenpal",