@eigenpal/cli 0.4.9 → 0.4.10

package/CHANGELOG.md

@@ -211,6 +211,25 @@
 
   `@` path syntax (`--code @file.js`, `--inputs items=@arr.json`) matches `curl -d @file` / `gh --body-file -`. Bare paths and `-` (stdin) work too.
 
+- 4137eb8: custom-script evaluator: the YAML now carries the entire
+  `function scoreScript(expected, actual) { ... }` declaration in a single
+  `function` field. Previous formats (`code`, then `body`) wrapped the
+  text implicitly; the new shape stores the function declaration verbatim
+  so the YAML matches the dashboard editor 1-for-1.
+
+  `workflow validate` and `workflow evaluators validate` flag YAML where
+  the function fails to parse, the signature does not match
+  `function scoreScript(expected, actual)` exactly, or the body contains
+  neither `return` nor `throw` — same checks the dashboard runs at form
+  save. The function must return a number in [0, 1]; throws are caught
+  and scored as 0.
+
+  Existing evaluator YAML must be ported by hand: rename `body` (or
+  `code`) to `function`, and add the `function scoreScript(expected,
+actual) { ... }` declaration around your statements. Parameter order is
+  load-bearing — swapping `expected` and `actual` would silently invert
+  every score.
+
 - 3da0612: All list commands now support `--limit` / `--offset` pagination consistently.
   - CLI: `workflow execution list`, `workflow version list`, and `workflow step-type list` previously had no `--offset` (or no pagination at all). All six list commands now accept `--limit`/`--offset` with defaults of 50/0.
   - Server: `GET /api/workflows/:id/versions` now reads `?limit=` and `?offset=` query params (defaults 50/0; max limit 100). The `listVersions` repo method gained an optional `{ limit, offset }` argument.
@@ -283,6 +302,11 @@
 
   Interactive profiles are unchanged: when you log in via `eigenpal auth login`, the profile stores both apiKey and baseUrl, and every subsequent command derives baseUrl from that profile. Set `EIGENPAL_PROFILE=<name>` to switch profiles per-shell without modifying state.
 
+- 542fe0e: CLI: surface evaluator entry-level fields (`name`, `description`, `weight`) so agents and humans can discover the optional `description` field on every evaluator.
+  - `workflow evaluator-type get <type>` now emits `entrySchema` alongside `configSchema`. Inspect with `eigenpal workflow evaluator-type get llm-judge | jq '.entrySchema'`.
+  - The skill reference now renders a "Common entry fields" table per type and a "Writing descriptions for stakeholders" section explaining the convention: each evaluator's `description` should be a one-sentence, plain-language summary aimed at non-technical reviewers in the dashboard.
+  - `init` templates `pdf-extraction` and `text-classification` now ship a `description:` line on their evaluator entries.
+
 - d41ee8b: CLI: fix `eigenpal status` error UX + harden `formatCliError` for real-world connection failures.
   - `eigenpal status` against an unreachable base URL no longer mis-diagnoses the failure as an auth problem. Previously, `EIGENPAL_BASE_URL=http://localhost:9999 eigenpal status` printed "Authentication check failed; try `eigenpal auth login`" — the wrong rabbit hole. The status command now rethrows non-auth errors (connection refused, DNS NXDOMAIN, timeout, HTML responses, 5xx) so `formatCliError` upstream can surface the right hint: `Could not connect to the server at <url> ... Set EIGENPAL_BASE_URL or pass --base-url <url>.` Only real 401/403 responses are treated as auth failures.
   - `formatCliError` now detects connection failures wrapped in any Error shape — bare `Error('ENOTFOUND ...')`, `Error` with `code: 'ECONNREFUSED'`, and `TypeError('fetch failed')` whose `.cause` carries the underlying error. Previously only `TypeError` whose message contained `fetch`/`URL` was recognized, so `getaddrinfo`-style errors fell through to the generic raw-message branch and lost the URL context.

package/dist/cli.js

@@ -23327,6 +23327,141 @@ var init_dataset_archive = __esm(() => {
   };
 });
 
+// ../types/src/eval/score-function.ts
+function stripLeadingTrivia(text) {
+  let i = 0;
+  while (i < text.length) {
+    const ch = text[i];
+    if (ch === " " || ch === "\t" || ch === `
+` || ch === "\r" || ch === "\v" || ch === "\f") {
+      i++;
+      continue;
+    }
+    if (ch === "/" && text[i + 1] === "/") {
+      const nl = text.indexOf(`
+`, i + 2);
+      if (nl === -1)
+        return "";
+      i = nl + 1;
+      continue;
+    }
+    if (ch === "/" && text[i + 1] === "*") {
+      const end = text.indexOf("*/", i + 2);
+      if (end === -1)
+        break;
+      i = end + 2;
+      continue;
+    }
+    break;
+  }
+  return text.slice(i);
+}
+function hasValidScoreSignature(fn) {
+  return STRICT_HEADER_REGEX.test(stripLeadingTrivia(fn));
+}
+function isParseableScoreFunction(fn) {
+  try {
+    new Function(fn);
+    return true;
+  } catch {
+    return false;
+  }
+}
+function getScoreFunctionRuntimeIssue(fn) {
+  if (/\basync\s+function\s+scoreScript\b/.test(fn)) {
+    return "scoreScript must be synchronous (drop the `async` keyword — the sandbox calls it sync, so a Promise return scores 0)";
+  }
+  if (/\bimport\s*\(/.test(fn)) {
+    return "dynamic `import(...)` is not supported in the score sandbox";
+  }
+  if (/\brequire\s*\(/.test(fn)) {
+    return "`require(...)` is not available in the score sandbox (no Node module system)";
+  }
+  return null;
+}
+function extractScoreFunctionBody(fn) {
+  const headerMatch = fn.match(ANY_HEADER_REGEX);
+  if (!headerMatch || headerMatch.index === undefined) {
+    return { body: fn, wrapperOk: false };
+  }
+  const start = headerMatch.index + headerMatch[0].length;
+  let depth = 1;
+  let i = start;
+  let inLine = false;
+  let inBlock = false;
+  let strQuote = null;
+  while (i < fn.length) {
+    const ch = fn[i];
+    const next = fn[i + 1];
+    if (inLine) {
+      if (ch === `
+`)
+        inLine = false;
+    } else if (inBlock) {
+      if (ch === "*" && next === "/") {
+        inBlock = false;
+        i++;
+      }
+    } else if (strQuote) {
+      if (ch === "\\")
+        i++;
+      else if (ch === strQuote)
+        strQuote = null;
+    } else if (ch === "/" && next === "/") {
+      inLine = true;
+      i++;
+    } else if (ch === "/" && next === "*") {
+      inBlock = true;
+      i++;
+    } else if (ch === '"' || ch === "'" || ch === "`") {
+      strQuote = ch;
+    } else if (ch === "{") {
+      depth++;
+    } else if (ch === "}") {
+      depth--;
+      if (depth === 0) {
+        const inner = fn.slice(start, i).replace(/^\n+|\n+$/g, "");
+        return { body: inner, wrapperOk: true };
+      }
+    }
+    i++;
+  }
+  return { body: fn.slice(start), wrapperOk: false };
+}
+function defaultScoreFunction() {
+  return `/**
+ * Score the workflow output against the example's expected output.
+ *
+ * @param {Expected} expected the example's expected output
+ * @param {Actual} actual the workflow's actual output
+ * @returns {number} a score in [0, 1]; throws are caught and scored as 0
+ */
+function scoreScript(expected, actual) {
+  // Return a number in [0, 1]; throws are caught and scored as 0.
+  return JSON.stringify(actual) === JSON.stringify(expected) ? 1 : 0;
+}
+`;
+}
+function wrapBodyAsScoreFunction(body) {
+  const trimmed = body.replace(/^\n+|\n+$/g, "");
+  return `/**
+ * Score the workflow output against the example's expected output.
+ *
+ * @param {Expected} expected the example's expected output
+ * @param {Actual} actual the workflow's actual output
+ * @returns {number} a score in [0, 1]; throws are caught and scored as 0
+ */
+function scoreScript(expected, actual) {
+${trimmed}
+}
+`;
+}
+var STRICT_HEADER_REGEX, ANY_HEADER_REGEX;
+var init_score_function = __esm(() => {
+  STRICT_HEADER_REGEX = /^(?:async\s+)?function\s+scoreScript\s*\(\s*expected\s*,\s*actual\s*\)\s*\{/;
+  ANY_HEADER_REGEX = /function\s+scoreScript\s*\([^)]*\)\s*\{/;
+});
+
 // ../types/src/eval/evaluator-config.ts
 function combineEvalScores(results) {
   let numerator = 0;
@@ -23341,9 +23476,10 @@ function combineEvalScores(results) {
     return null;
   return numerator / denominator;
 }
-var EvaluatorTypeSchema, ScoreInUnitInterval, LLM_JUDGE_PROMPT_MIN_LENGTH = 10, ExactDiffConfigSchema, JudgeModeSchema, LabelsRecord, LlmJudgeConfigSchema, CustomScriptConfigSchema, EvaluatorBaseEntrySchema, EvaluatorConfigSchema, EvalConfigYamlSchema, EvaluatorResultSchema;
+var EvaluatorTypeSchema, ScoreInUnitInterval, LLM_JUDGE_PROMPT_MIN_LENGTH = 10, ExactDiffConfigSchema, JudgeModeSchema, LabelsRecord, LlmJudgeConfigSchema, CUSTOM_SCRIPT_CONTRACT, CUSTOM_SCRIPT_MAX_BYTES = 1e4, CUSTOM_SCRIPT_MAX_TIMEOUT_MS = 30000, CUSTOM_SCRIPT_MAX_MEMORY_MB = 50, CUSTOM_SCRIPT_FUNCTION_DESCRIPTION, CustomScriptConfigSchema, EvaluatorBaseEntrySchema, EvaluatorConfigSchema, EvalConfigYamlSchema, EvaluatorResultSchema;
 var init_evaluator_config = __esm(() => {
   init_zod();
+  init_score_function();
   EvaluatorTypeSchema = exports_external.enum(["exact-diff", "llm-judge", "custom-script"]);
   ScoreInUnitInterval = exports_external.number().min(0, { message: "score must be >= 0" }).max(1, { message: "score must be <= 1" });
   ExactDiffConfigSchema = exports_external.object({
@@ -23372,14 +23508,41 @@ var init_evaluator_config = __esm(() => {
       }
     }
   });
+  CUSTOM_SCRIPT_CONTRACT = {
+    expected: "the example's expected output",
+    actual: "the workflow's actual output",
+    returns: "a number in [0, 1]",
+    throws: "caught and scored as 0"
+  };
+  CUSTOM_SCRIPT_FUNCTION_DESCRIPTION = `Full JavaScript declaration of \`function scoreScript(expected, actual) { ... }\`. ` + `Receives \`expected\` (${CUSTOM_SCRIPT_CONTRACT.expected}) ` + `and \`actual\` (${CUSTOM_SCRIPT_CONTRACT.actual}), returns ${CUSTOM_SCRIPT_CONTRACT.returns}. ` + `Throws are ${CUSTOM_SCRIPT_CONTRACT.throws}.`;
   CustomScriptConfigSchema = exports_external.object({
-    code: exports_external.string().min(1, { message: "code required" }).describe("Sandboxed JavaScript. Receives `actual`, `expected`, `input` as variables and must return { score, label? } where score is in [0, 1]."),
-    timeoutMs: exports_external.number().int().positive().default(5000).describe("Maximum wall-clock time the script can run before it's killed and the run is marked failed."),
-    memoryLimitMb: exports_external.number().int().positive().default(10).describe("Maximum memory the sandbox may allocate. Increase if the script processes large arrays or strings."),
+    function: exports_external.string().min(1, { message: "required" }).max(CUSTOM_SCRIPT_MAX_BYTES, {
+      message: `must be ≤ ${CUSTOM_SCRIPT_MAX_BYTES} bytes`
+    }).describe(CUSTOM_SCRIPT_FUNCTION_DESCRIPTION).refine(isParseableScoreFunction, {
+      message: "must parse as JavaScript"
+    }).refine(hasValidScoreSignature, {
+      message: "must declare `function scoreScript(expected, actual) { ... }` (parameter names + order matter)"
+    }).refine((fn) => {
+      const { body, wrapperOk } = extractScoreFunctionBody(fn);
+      return wrapperOk && /\b(return|throw)\b/.test(body);
+    }, {
+      message: "function body must `return` a number in [0, 1] (or `throw` to score 0)"
+    }).superRefine((fn, ctx) => {
+      const issue2 = getScoreFunctionRuntimeIssue(fn);
+      if (issue2)
+        ctx.addIssue({ code: exports_external.ZodIssueCode.custom, message: issue2 });
+    }),
+    timeoutMs: exports_external.number().int().positive().max(CUSTOM_SCRIPT_MAX_TIMEOUT_MS, {
+      message: `must be ≤ ${CUSTOM_SCRIPT_MAX_TIMEOUT_MS} ms`
+    }).default(5000).describe("Maximum wall-clock time the script can run before it's killed and the run is marked failed."),
+    memoryLimitMb: exports_external.number().int().positive().max(CUSTOM_SCRIPT_MAX_MEMORY_MB, {
+      message: `must be ≤ ${CUSTOM_SCRIPT_MAX_MEMORY_MB} MB`
+    }).default(10).describe("Maximum memory the sandbox may allocate. Increase if the script processes large arrays or strings."),
     passThreshold: ScoreInUnitInterval.default(1).describe("Minimum script score this evaluator must reach for a run to pass.")
   });
   EvaluatorBaseEntrySchema = exports_external.object({
     name: exports_external.string().min(1).max(64).regex(/^[a-z0-9-]+$/, "name must be lowercase kebab-case").describe("Identifier used in eval results and the weighted-mean formula. Use a short kebab-case slug like header-exact or covenants-recall."),
+    description: exports_external.string().trim().max(500).optional().describe("One-sentence explanation, in business language, of what this evaluator measures and why a stakeholder should care. Shown in the dashboard to non-technical reviewers (legal, ops, finance) who never see the YAML or judge prompt. Skip implementation detail (model names, thresholds, dot-paths); those live in `config`."),
     weight: exports_external.number().min(0).default(1).describe("Relative importance in the weighted mean. The overall score is Σ(weightᵢ × scoreᵢ) / Σweight. Set to 0 to score-only without affecting overall.")
   });
   EvaluatorConfigSchema = exports_external.discriminatedUnion("type", [
@@ -23569,6 +23732,7 @@ var init_eval = __esm(() => {
   init_dataset_archive();
   init_evaluator_config();
   init_judge();
+  init_score_function();
   init_workflow_tool_metadata();
 });
 
@@ -23637,6 +23801,7 @@ var init_paths = __esm(() => {
 // ../types/src/index.ts
 var exports_src = {};
 __export(exports_src, {
+  wrapBodyAsScoreFunction: () => wrapBodyAsScoreFunction,
   workflowTriggerConfigSchema: () => workflowTriggerConfigSchema,
   valuesMatch: () => valuesMatch,
   validateStepConfig: () => validateStepConfig,
@@ -23667,10 +23832,12 @@ __export(exports_src, {
   isValidJudgeModel: () => isValidJudgeModel,
   isTerminalExecutionStatus: () => isTerminalExecutionStatus,
   isStepCategory: () => isStepCategory,
+  isParseableScoreFunction: () => isParseableScoreFunction,
   isOutputSchemaUnknown: () => isOutputSchemaUnknown,
   isNonSuccessTerminalExecutionStatus: () => isNonSuccessTerminalExecutionStatus,
   isFilePathDescriptor: () => isFilePathDescriptor,
   isFileOutput: () => isFileOutput,
+  hasValidScoreSignature: () => hasValidScoreSignature,
   hasTriggerMethod: () => hasTriggerMethod,
   goldenFileSatisfiedInResult: () => goldenFileSatisfiedInResult,
   getTriggerMethods: () => getTriggerMethods,
@@ -23678,6 +23845,7 @@ __export(exports_src, {
   getStepSchema: () => getStepSchema,
   getStepOutputSchema: () => getStepOutputSchema,
   getStepConfigSchema: () => getStepConfigSchema,
+  getScoreFunctionRuntimeIssue: () => getScoreFunctionRuntimeIssue,
   getRequiredOutputKeys: () => getRequiredOutputKeys,
   getProcessorSchemas: () => getProcessorSchemas,
   getParserCategory: () => getParserCategory,
@@ -23693,11 +23861,13 @@ __export(exports_src, {
   formatDiffValue: () => formatDiffValue,
   fileFieldNamesFromJsonSchema: () => fileFieldNamesFromJsonSchema,
   fieldStemFromGoldenName: () => fieldStemFromGoldenName,
+  extractScoreFunctionBody: () => extractScoreFunctionBody,
   extractFieldMetadata: () => extractFieldMetadata,
   executionWallClockMs: () => executionWallClockMs,
   evalInputKey: () => evalInputKey,
   evalExpectedKey: () => evalExpectedKey,
   errorHintFor: () => errorHintFor,
+  defaultScoreFunction: () => defaultScoreFunction,
   compareOutput: () => compareOutput,
   compareExpectedValues: () => compareExpectedValues,
   combineEvalScores: () => combineEvalScores,
@@ -23892,6 +24062,10 @@ __export(exports_src, {
   ControlApprovalOutputSchema: () => ControlApprovalOutputSchema,
   ControlApprovalConfigSchema: () => ControlApprovalConfigSchema,
   ConnectorSchema: () => ConnectorSchema,
+  CUSTOM_SCRIPT_MAX_TIMEOUT_MS: () => CUSTOM_SCRIPT_MAX_TIMEOUT_MS,
+  CUSTOM_SCRIPT_MAX_MEMORY_MB: () => CUSTOM_SCRIPT_MAX_MEMORY_MB,
+  CUSTOM_SCRIPT_MAX_BYTES: () => CUSTOM_SCRIPT_MAX_BYTES,
+  CUSTOM_SCRIPT_CONTRACT: () => CUSTOM_SCRIPT_CONTRACT,
   BoundingRegionSchema: () => BoundingRegionSchema,
   BlockStepSchema: () => BlockStepSchema,
   BaseEntitySchema: () => BaseEntitySchema,
@@ -143406,7 +143580,7 @@ import { fileURLToPath as fileURLToPath6 } from "url";
 // package.json
 var package_default = {
   name: "@eigenpal/cli",
-  version: "0.4.9",
+  version: "0.4.10",
   description: "Eigenpal CLI — author and run document workflows from your terminal. Agent-ready.",
   keywords: [
     "eigenpal",
@@ -146149,26 +146323,23 @@ function getExampleNames(dir, exampleNamesFilter) {
 function readInput(exampleDir) {
   const argsPath = join6(exampleDir, "input", "arguments.json");
   const inputDir = join6(exampleDir, "input");
-  const merged = JSON.parse(readFileSync5(argsPath, "utf-8"));
+  const scalars = JSON.parse(readFileSync5(argsPath, "utf-8"));
+  const files = [];
   for (const entry of readdirSync3(inputDir)) {
     const full = join6(inputDir, entry);
     if (!statSync2(full).isDirectory())
       continue;
-    const files = readdirSync3(full).filter((f) => statSync2(join6(full, f)).isFile()).sort();
-    if (files.length === 0)
-      continue;
-    const inlined = files.map((filename) => {
-      const buf = readFileSync5(join6(full, filename));
-      const mime = guessMimeType(filename);
-      return {
-        _inline: buf.toString("base64"),
+    const filenames = readdirSync3(full).filter((f) => statSync2(join6(full, f)).isFile()).sort();
+    for (const filename of filenames) {
+      files.push({
+        argument: entry,
         filename,
-        ...mime ? { mimeType: mime } : {}
-      };
-    });
-    merged[entry] = files.length === 1 ? inlined[0] : inlined;
+        content: readFileSync5(join6(full, filename)),
+        mimeType: guessMimeType(filename)
+      });
+    }
   }
-  return merged;
+  return { scalars, files };
 }
 function readOverrides(exampleDir) {
   const metaPath = join6(exampleDir, "meta.json");
@@ -146182,8 +146353,10 @@ function readOverrides(exampleDir) {
   return null;
 }
 function buildExamplePayload(exampleDir) {
+  const { scalars, files } = readInput(exampleDir);
   return {
-    input: readInput(exampleDir),
+    scalars,
+    files,
     overrides: readOverrides(exampleDir)
   };
 }
@@ -146665,11 +146838,19 @@ async function runExec(client, dir, workflowIdOrSlug, exampleNames, options8 = {
           progress.setRunning(index);
         else
           console.log(`${ui.dim("→")} ${name}`);
-        const res = await client.post(`/api/v1/workflows/${workflowId}/run`, {
-          input: example.input,
-          overrides: example.overrides ?? undefined,
-          trigger: "cli"
-        });
+        const form = new FormData;
+        const sidecar = { trigger: "cli" };
+        if (Object.keys(example.scalars).length > 0)
+          sidecar.input = example.scalars;
+        if (example.overrides)
+          sidecar.overrides = example.overrides;
+        form.append("_json", JSON.stringify(sidecar));
+        for (const file2 of example.files) {
+          form.append(file2.argument, new Blob([file2.content], {
+            type: file2.mimeType ?? "application/octet-stream"
+          }), file2.filename);
+        }
+        const res = await client.postFormData(`/api/v1/workflows/${workflowId}/run`, form);
         if (typeof res?.executionId !== "string") {
           throw new Error("Run API did not return executionId");
         }
@@ -147464,7 +147645,7 @@ var EVALUATOR_REGISTRY = {
   "custom-script": {
     type: "custom-script",
     name: "Custom Script",
-    description: "Run sandboxed JavaScript that returns { score, label? }. Receives `actual`, `expected`, `input`.",
+    description: "Score the workflow output with sandboxed JavaScript. Authors a `function scoreScript(expected, actual) { ... }` declaration that returns a number in [0, 1].",
     configSchema: CustomScriptConfigSchema
   }
 };
@@ -147506,6 +147687,7 @@ function registerEvaluatorTypeCommands(parent) {
       type: def.type,
       name: def.name,
       description: def.description,
+      entrySchema: z12.toJSONSchema(EvaluatorBaseEntrySchema, { target: "draft-7" }),
       configSchema: z12.toJSONSchema(def.configSchema, { target: "draft-7" })
     };
     const text = JSON.stringify(result, null, 2);
@@ -148057,7 +148239,7 @@ flag would otherwise shadow it. Validation failures come back as
         error51(`Cannot update workflow ${ui.bold(workflowId)} — pass --bump <patch|minor|major> or --set-version <X.Y.Z>, or add a top-level \`version: X.Y.Z\` to ${filePath}.`);
         process.exit(1);
       }
-      const yamlToSend = cliWantsRewrite ? spliceWorkflowVersion(yamlOnDisk, nextVersion) : yamlOnDisk;
+      const yamlToSend = cliWantsRewrite ? spliceWorkflowVersion2(yamlOnDisk, nextVersion) : yamlOnDisk;
       const result = await client.post(`/api/v1/workflows/${workflowId}/versions`, {
         yaml: yamlToSend,
         version: nextVersion
@@ -148071,7 +148253,7 @@ flag would otherwise shadow it. Validation failures come back as
       if (opts.bump) {
         throw new Error("--bump requires --workflow-id (need a current version to bump from). For a new workflow, use --set-version <X.Y.Z> or omit both.");
       }
-      const yamlToSend = opts.setVersion && !extractWorkflowVersion(yamlOnDisk) ? spliceWorkflowVersion(yamlOnDisk, opts.setVersion) : yamlOnDisk;
+      const yamlToSend = opts.setVersion && !extractWorkflowVersion(yamlOnDisk) ? spliceWorkflowVersion2(yamlOnDisk, opts.setVersion) : yamlOnDisk;
       const result = await client.post("/api/v1/workflows", { yaml: yamlToSend });
       if (opts.json) {
         printJson(result);
@@ -148098,7 +148280,7 @@ function bumpSemver(current, level) {
     return `${major}.${minor + 1}.0`;
   return `${major}.${minor}.${patch + 1}`;
 }
-function spliceWorkflowVersion(yaml, next) {
+function spliceWorkflowVersion2(yaml, next) {
   if (/^version:\s*['"]?(\d+\.\d+\.\d+)['"]?\s*$/m.test(yaml)) {
     return yaml.replace(/^version:.*$/m, `version: ${next}`);
   }

package/dist/skill/SKILL.md

@@ -174,6 +174,12 @@ eigenpal workflow evaluators validate ./evaluators.yaml
 eigenpal workflow evaluators push <workflow-id> --file evaluators.yaml
 ```
 
+Always set `description:` on every evaluator entry. It's a short, plain-language
+sentence (≤500 chars; aim for one line) explaining what the evaluator measures
+and why it matters, written for a non-technical reviewer in the dashboard.
+Skip model names, thresholds, and dot-paths; those belong in `config`. See
+[`reference/evaluators.md`](reference/evaluators.md#writing-descriptions-for-stakeholders).
+
 ### Build + validate + upload a dataset
 
 ```bash

package/dist/skill/reference/evaluators.md

@@ -11,16 +11,41 @@ this is the agent-facing summary.
 ```yaml
 evaluators: # array of evaluator definitions, ≥1 entry
   - name: ... # required, unique within this file, [a-z0-9][a-z0-9-_]*
+    description: ... # optional, ≤500 chars; one-sentence business explanation (see below)
     type: ... # required, one of: exact-diff | llm-judge | custom-script
     config: # required, shape depends on `type`
       passThreshold: 0.95 # 0..1, optional (default 1.0)
       ...
 
-# OPTIONAL global config
-concurrency: 4 # max examples in parallel during `experiment run` (default 1)
-judge: # optional shared model defaults for llm-judge evaluators
-  model: gpt-4o
-  temperature: 0
+# OPTIONAL workflow-level pass threshold. A run passes when its weighted-mean
+# score across evaluators clears this number. Defaults to 0.7.
+passThreshold: 0.7
+```
+
+## Writing descriptions for stakeholders
+
+Every evaluator entry takes an optional `description`. Always set it. The
+dashboard surfaces this string to non-technical reviewers (legal, ops,
+finance) who never see the YAML or the judge prompt. Write it for them:
+
+- One short sentence, plain language.
+- Name what the evaluator measures and why a business reader should care.
+- Skip implementation detail (model names, thresholds, dot-paths). Those
+  belong in `config`.
+- Keep it under ~120 characters when you can; the schema caps it at 500.
+
+```yaml
+# Good. A non-technical reviewer immediately grasps the intent.
+- name: covenants-recall
+  description: Checks every covenant from the contract appears in the extraction.
+  type: llm-judge
+  config: ...
+
+# Bad. Restates the config; teaches a stakeholder nothing.
+- name: covenants-recall
+  description: llm-judge with passThreshold 0.85 over the covenants array.
+  type: llm-judge
+  config: ...
 ```
 
 ## Evaluator types
@@ -47,23 +72,34 @@ judge: # optional shared model defaults for llm-judge evaluators
 - name: covenants-recall
   type: llm-judge
   config:
-    model: gpt-4o # string — the OpenAI model id
-    temperature: 0 # number — for reproducibility
+    model: gpt-4o # OPTIONAL — falls back to the workspace default LLM provider
+    mode: continuous # continuous | discrete (default: continuous)
     passThreshold: 0.85
-    prompt: |
+    docLabel: extraction # what the judge calls the workflow output (default: "output")
+    promptExtension: |
       Score how well the actual output captures every covenant in the
-      expected output, on a scale of 0..1. Penalise missing covenants;
-      do not penalise over-extraction. Return JSON { score, rationale }.
-```
+      expected output. Penalise missing covenants; do not penalise
+      over-extraction. Be strict on covenants that change loan economics.
 
-The harness wraps your `prompt` and feeds:
-
-- `actual` — the workflow's output
-- `expected` — the example's `expected/output.json`
-- `meta` — example metadata
+# discrete mode — the judge picks a label and the score table maps to a number
+- name: tone-check
+  type: llm-judge
+  config:
+    mode: discrete
+    passThreshold: 0.7
+    labels:
+      excellent: 1.0
+      acceptable: 0.7
+      off-tone: 0.0
+    promptExtension: |
+      Pick the label that best matches the tone of the response.
+```
 
-It expects the model to return JSON with at least `score` (0..1) and
-`rationale` (string). The fixed harness handles parse failures + retries.
+`promptExtension` is the only required field — it's appended to the harness
+prompt and tells the judge what "good" looks like for this workflow. The
+harness handles JSON parsing, retries, and feeds the judge `actual`,
+`expected`, and example metadata. In `discrete` mode the judge MUST return
+one of the keys in `labels`; the harness then looks up the score.
 
 ### `custom-script` — JavaScript in the sandbox
 
@@ -72,16 +108,38 @@ It expects the model to return JSON with at least `score` (0..1) and
   type: custom-script
   config:
     passThreshold: 1.0
-    code: |
-      const a = actual.totalAmount ?? 0;
-      const e = expected.totalAmount ?? 0;
-      const score = Math.abs(a - e) <= 0.01 ? 1 : 0;
-      return { score, rationale: `actual=${a}, expected=${e}` };
+    function: |
+      /**
+       * @param {Expected} expected the example's expected output
+       * @param {Actual} actual the workflow's actual output
+       * @returns {number} a score in [0, 1]; throws are caught and scored as 0
+       */
+      function scoreScript(expected, actual) {
+        const a = actual.totalAmount ?? 0;
+        const e = expected.totalAmount ?? 0;
+        return Math.abs(a - e) <= 0.01 ? 1 : 0;
+      }
 ```
 
 Same WASM sandbox as `transform.script` (5s wall clock, 10 MB heap, no
-network). Receives `actual`, `expected`, `meta` in scope. Returns
-`{ score: number, label?: string, rationale?: string }`.
+network). The YAML stores the **whole** `function scoreScript(expected,
+actual) { ... }` declaration verbatim — same text the dashboard editor
+shows. The JSDoc preamble is optional but recommended: it pulls the
+workflow's output schema into IntelliSense for `expected.…` /
+`actual.…` in the dashboard.
+
+The signature shape is **locked**:
+
+- Function name MUST be `scoreScript`.
+- Parameters MUST be `(expected, actual)` in that order. Swapping them
+  would silently invert every score.
+- Body MUST `return` a number in `[0, 1]`, or `throw` (which is caught
+  and scored as 0).
+
+`workflow validate` and `workflow evaluators validate` flag YAML where
+the function fails to parse, the signature doesn't match, or the body
+contains neither `return` nor `throw` — so a hand-edited config errors
+before it reaches the worker.
 
 ## Pass / fail vs score
 
@@ -160,9 +218,11 @@ process — `--out` writes the file directly from the server's storage.
 LLM-judge scores have variance. To reduce flake:
 
 - Pin a specific model id (`gpt-4o-2024-08-06` rather than `gpt-4o`).
-- Set `temperature: 0`.
-- Keep the prompt narrow — score one dimension per evaluator. Use
+- Keep `promptExtension` narrow — score one dimension per evaluator. Use
   multiple judges instead of one omnibus prompt.
+- Prefer `mode: discrete` with a small label set when the judgement is
+  categorical (good / acceptable / bad) — it eliminates the model's
+  free-form numeric drift.
 - For high-stakes thresholds, run the experiment ≥3 times and average.
 
 <!-- GENERATED:EVALUATOR_CATALOG START -->
@@ -177,6 +237,14 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
 | `evaluators` | array<object> | no | `[]` |  |
 | `passThreshold` | number | no | `0.7` |  |
 
+### Common entry fields (every evaluator type)
+
+| Field | Type | Required | Default | Description |
+| --- | --- | --- | --- | --- |
+| `name` | string | yes |  | Identifier used in eval results and the weighted-mean formula. Use a short kebab-case slug like header-exact or covenants-recall. |
+| `description` | string | no |  | One-sentence explanation, in business language, of what this evaluator measures and why a stakeholder should care. Shown in the dashboard to non-technical reviewers (legal, ops, finance) who never see the YAML or judge prompt. Skip implementation detail (model names, thresholds, dot-paths); those live in `config`. |
+| `weight` | number | no | `1` | Relative importance in the weighted mean. The overall score is Σ(weightᵢ × scoreᵢ) / Σweight. Set to 0 to score-only without affecting overall. |
+
 ### `exact-diff` — JSON deep-diff against expected output
 
 | Field | Type | Required | Default | Description |
@@ -203,7 +271,7 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
 
 | Field | Type | Required | Default | Description |
 | --- | --- | --- | --- | --- |
-| `code` | string | yes |  | Sandboxed JavaScript. Receives `actual`, `expected`, `input` as variables and must return { score, label? } where score is in [0, 1]. |
+| `function` | string | yes |  | Full JavaScript declaration of `function scoreScript(expected, actual) { ... }`. Receives `expected` (the example's expected output) and `actual` (the workflow's actual output), returns a number in [0, 1]. Throws are caught and scored as 0. |
 | `timeoutMs` | integer | no | `5000` | Maximum wall-clock time the script can run before it's killed and the run is marked failed. |
 | `memoryLimitMb` | integer | no | `10` | Maximum memory the sandbox may allocate. Increase if the script processes large arrays or strings. |
 | `passThreshold` | number | no | `1` | Minimum script score this evaluator must reach for a run to pass. |

package/dist/templates/pdf-extraction/evaluators.yaml

@@ -1,5 +1,6 @@
 evaluators:
   - name: exact-match
+    description: Checks the extracted fields match the expected output exactly.
     type: exact-diff
     config:
       passThreshold: 1.0

package/dist/templates/text-classification/evaluators.yaml

@@ -1,5 +1,6 @@
 evaluators:
   - name: label-match
+    description: Checks the predicted label matches the expected label.
     type: exact-diff
     config:
       passThreshold: 1.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@eigenpal/cli",
-  "version": "0.4.9",
+  "version": "0.4.10",
   "description": "Eigenpal CLI — author and run document workflows from your terminal. Agent-ready.",
   "keywords": [
     "eigenpal",