@fede0089/skill-eval 3.2.0 → 3.3.0

package/README.md

@@ -92,7 +92,7 @@ skill-eval functional --workspace <path> --skill <path> [options] [agent]
 | `--eval-file <name>` | no | all | Run only the evals from this file in `evals/` (e.g. `edge-cases.json`) |
 | `--compare-ref [refs...]` | no | — | Git references to compare against (variadic — put `[agent]` before it, not after) |
 | `--compare-baseline` | no | `false` | Also run the no-skill baseline alongside the skill |
-| `-v, --debug` | no | `false` | Enable verbose debug logging |
+| `-v, --debug` | no | `false` | Print verbose logs to the console (trial transcripts are always saved) |
 | `[agent]` | no | `gemini-cli` | Agent backend to use |
 
 Supported runners:
@@ -222,24 +222,55 @@ Refer to your runner's documentation for the full set of settings and policy key
 
 ## Reports
 
-Each run writes to `.project-skill-evals/runs/<timestamp>/` and includes per-trial logs, the raw eval JSON, and a self-contained HTML report you can open in any browser. The report shows pass@k aggregates per eval, lets you expand each trial, and color-codes triggering vs. functional outcomes.
+Each run writes to `.project-skill-evals/runs/<timestamp>/`: one log per trial and a self-contained HTML report you can open in any browser.
+
+Expanding an eval gives you three sections:
+
+- **Summary** — success rate, average tokens and average time, per variant.
+- **Trials** — one row per trial, with its score, anomaly flags, cost and an exclude control. Expanding a row shows the agent's final output, its stats and a link to the full transcript.
+- **Expectations** — one row per expectation, with per-variant pass rates. Clicking a cell shows the judge's verdict for every trial.
+
+The report carries the full run data and computes every figure in the browser, so excluding a trial updates all of them at once.
 
 A published sample report is available at [fede0089.github.io/skill-eval/sample-report.html](https://fede0089.github.io/skill-eval/sample-report.html), generated from this project root with:
 
 ```sh
-skill-eval functional --workspace . --skill mock-skill --trials 2 --compare-baseline --debug claude-code
+skill-eval functional --workspace . --skill mock-skill --trials 2 --compare-baseline claude-code
 ```
 
 ![Sample HTML report](docs/sample-report.png)
 
-### Debug logs
+### Trial transcripts
 
-When a trial misbehaves, pass `-v` / `--debug` to capture the full transcripts to disk. Each trial writes a `task_<id>_<variant>_trial_<n>.log` file inside the run directory with two sections appended in order:
+Every trial writes a `task_<id>_<variant>_trial_<n>.log` file inside the run directory, with two sections appended in order:
 
 - `# SECTION: <MODE> AGENT RUN` — the initial prompt sent to the agent and its raw streamed response.
 - `# SECTION: <MODE> JUDGE RUN` — the prompt sent to the LLM judge and its verdict (only present for `functional` runs; `trigger` is graded programmatically and produces no judge section).
 
-Without `--debug` these files are not written, so reach for the flag when you need to see exactly what the agent — or the judge — saw.
+These are always written, and the report links to each one from its trial row, so you can go from a suspicious number to exactly what the agent — or the judge — saw. `-v` / `--debug` is unrelated: it only makes the console output verbose.
+
+## Excluding trials
+
+Agents are non-deterministic, and a trial sometimes fails for reasons that have nothing to do with the skill: the model degenerates into a single word, stops before answering, or trips over the environment. Those trials drag the score down and, worse, compress the very difference an A/B run exists to measure.
+
+The report flags the likely ones and lets you drop them. Each flag compares a trial against its own cohort — the sibling trials of the same eval and variant — rather than against a fixed threshold:
+
+| Flag | Meaning |
+|------|---------|
+| `degenerate-output` | The final output is a fraction of the cohort's median length |
+| `zero-assertions` | Nothing passed, while the cohort median is well above zero |
+| `premature-stop` | The run ended without a success status, or produced no output |
+| `resource-outlier` | Token spend far above the cohort median |
+
+Flags never exclude anything on their own — no threshold can reliably separate "the skill failed" from "the model went off the rails this time", so the call is yours. Open a trial row, read what the agent actually produced, and press **Exclude**; the reason is pre-filled from the strongest flag and can be changed, along with a free-text note.
+
+Excluding recomputes every figure in the report, under a few rules that keep the result honest:
+
+- **The raw number stays visible.** Every adjusted rate is shown next to the unadjusted one and the effective sample size (`raw 36% · n=3/5`). Below three usable trials the figure is marked low-confidence.
+- **Unbalanced exclusions raise a warning.** Dropping more trials from one variant than another moves the delta you are measuring, so the report says so.
+- **The exclusion rate is itself a result.** If four of ten trials degenerated, that is a finding about the model or the prompt, not noise to sweep away — so it is shown at the top.
+
+Exclusions are kept in the browser for that run. **Download reviewed copy** writes a `report-reviewed.html` with them baked in, for sharing or committing alongside the run.
 
 ## Try it out
 

package/dist/commands/functional.js

@@ -11,10 +11,10 @@ import { preflight } from '../core/preflight.js';
 import { ConfigError } from '../core/errors.js';
 import { withRetry } from '../core/trial-utils.js';
 import { renderFunctionalTable, renderRunHeader } from '../utils/table-renderer.js';
-import { JsonReporter } from '../reporters/index.js';
+import { HtmlReporter } from '../reporters/index.js';
 import chalk from 'chalk';
 import { git } from '../utils/git.js';
-export async function functionalCommand(agent, workspace, skillPath, maxAgents = 4, injectedSuite, numTrials = 3, reporter = new JsonReporter(), timeoutMs, evalId, compareRefs = [], compareBaseline = false, evalFile) {
+export async function functionalCommand(agent, workspace, skillPath, maxAgents = 4, injectedSuite, numTrials = 3, reporter = new HtmlReporter(), timeoutMs, evalId, compareRefs = [], compareBaseline = false, evalFile) {
     if (!injectedSuite)
         preflight(agent, workspace, skillPath);
     const suite = injectedSuite || evalLoader.loadEvalSuite(skillPath, evalFile);
@@ -45,8 +45,7 @@ export async function functionalCommand(agent, workspace, skillPath, maxAgents =
     const cleanup = () => { env.teardown().finally(() => process.exit(1)); };
     process.once('SIGINT', cleanup);
     process.once('SIGTERM', cleanup);
-    // Setup Artifacts Directory (Always create, even if not in debug mode, for 'show' command)
-    const debug = !!process.env.DEBUG;
+    // Every run gets its own directory: trial logs and the report always land here.
     const startTime = new Date();
     const timestamp = startTime.toISOString().replace(/[:.]/g, '-');
     const runDir = path.resolve(workspace, '.project-skill-evals', 'runs', timestamp);
@@ -55,7 +54,7 @@ export async function functionalCommand(agent, workspace, skillPath, maxAgents =
     const variantRunners = new Map();
     // 1. Local Runner
     variantRunners.set('local', new EvalRunner({
-        agent, workspace, skillPath, skillName: skill_name, runDir, isBaseline: false, debug, timeoutMs,
+        agent, workspace, skillPath, skillName: skill_name, runDir, isBaseline: false, timeoutMs,
         variant: 'local'
     }));
     // 2. Historical Runners
@@ -71,14 +70,13 @@ export async function functionalCommand(agent, workspace, skillPath, maxAgents =
             skillName: skill_name,
             runDir,
             isBaseline: false,
-            debug,
             timeoutMs,
             variant: `ref:${ref}`
         }));
     }
     // 3. Baseline Runner
     const withoutSkillRunner = compareBaseline ? new EvalRunner({
-        agent, workspace, skillPath, skillName: skill_name, runDir, isBaseline: true, debug, timeoutMs,
+        agent, workspace, skillPath, skillName: skill_name, runDir, isBaseline: true, timeoutMs,
         variant: 'baseline'
     }) : undefined;
     const taskResults = [];
@@ -273,7 +271,6 @@ export async function functionalCommand(agent, workspace, skillPath, maxAgents =
         Logger.write(`──────────────────────────────────────────────────\n`);
         renderFunctionalTable(report);
         Logger.write('\n');
-        new JsonReporter().generate(report, runDir);
         reporter.generate(report, runDir);
     }
     finally {

package/dist/commands/trigger.js

@@ -10,10 +10,10 @@ import { aggregatePassAtK, aggregateAssertionPassRate, aggregateTokenStats, aggr
 import { preflight } from '../core/preflight.js';
 import { withRetry } from '../core/trial-utils.js';
 import { renderTriggerTable, renderRunHeader } from '../utils/table-renderer.js';
-import { JsonReporter } from '../reporters/index.js';
+import { HtmlReporter } from '../reporters/index.js';
 import chalk from 'chalk';
 import { git } from '../utils/git.js';
-export async function triggerCommand(agent, workspace, skillPath, maxAgents = 4, injectedSuite, numTrials = 3, reporter = new JsonReporter(), timeoutMs, evalId, compareRefs = [], evalFile) {
+export async function triggerCommand(agent, workspace, skillPath, maxAgents = 4, injectedSuite, numTrials = 3, reporter = new HtmlReporter(), timeoutMs, evalId, compareRefs = [], evalFile) {
     if (!injectedSuite)
         preflight(agent, workspace, skillPath);
     const suite = injectedSuite || evalLoader.loadEvalSuite(skillPath, evalFile);
@@ -32,8 +32,7 @@ export async function triggerCommand(agent, workspace, skillPath, maxAgents = 4,
     const cleanup = () => { env.teardown().finally(() => process.exit(1)); };
     process.once('SIGINT', cleanup);
     process.once('SIGTERM', cleanup);
-    // Setup Artifacts Directory (Always create, even if not in debug mode, for 'show' command)
-    const debug = !!process.env.DEBUG;
+    // Every run gets its own directory: trial logs and the report always land here.
     const startTime = new Date();
     const timestamp = startTime.toISOString().replace(/[:.]/g, '-');
     const runDir = path.resolve(workspace, '.project-skill-evals', 'runs', timestamp);
@@ -42,7 +41,7 @@ export async function triggerCommand(agent, workspace, skillPath, maxAgents = 4,
     const variantRunners = new Map();
     // 1. Local Runner
     variantRunners.set('local', new EvalRunner({
-        agent, workspace, skillPath, skillName: skill_name, runDir, isBaseline: false, debug, timeoutMs,
+        agent, workspace, skillPath, skillName: skill_name, runDir, isBaseline: false, timeoutMs,
         variant: 'local'
     }));
     // 2. Historical Runners
@@ -58,7 +57,6 @@ export async function triggerCommand(agent, workspace, skillPath, maxAgents = 4,
             skillName: skill_name,
             runDir,
             isBaseline: false,
-            debug,
             timeoutMs,
             variant: `ref:${ref}`
         }));
@@ -203,7 +201,6 @@ export async function triggerCommand(agent, workspace, skillPath, maxAgents = 4,
         Logger.write(`──────────────────────────────────────────────────\n`);
         renderTriggerTable(report);
         Logger.write('\n\n');
-        new JsonReporter().generate(report, runDir);
         reporter.generate(report, runDir);
     }
     finally {

package/dist/core/anomalies.js

@@ -0,0 +1,63 @@
+/** Fraction of the cohort's median output length below which text reads as degenerate. */
+const DEGENERATE_OUTPUT_RATIO = 0.15;
+/** Multiple of the cohort's median token spend above which a trial is an outlier. */
+const RESOURCE_OUTLIER_RATIO = 2.5;
+function median(values) {
+    if (values.length === 0)
+        return 0;
+    const sorted = [...values].sort((a, b) => a - b);
+    const mid = Math.floor(sorted.length / 2);
+    return sorted.length % 2 === 1 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2;
+}
+function passedAssertions(trial) {
+    return trial.assertionResults.filter(r => r.passed).length;
+}
+/**
+ * Flags a trial against its cohort — the sibling trials of the same eval and
+ * variant. Comparisons are relative to the cohort median rather than absolute
+ * thresholds, so the signals hold whether an eval has 3 assertions or 30 and
+ * whether a task normally takes 20K tokens or 2M.
+ *
+ * Infrastructure errors return no anomalies: they are already excluded from
+ * every metric, and flagging them again would only pad the review queue.
+ *
+ * @param trial  The trial to inspect.
+ * @param cohort All trials of the same eval and variant, including `trial`.
+ */
+export function detectAnomalies(trial, cohort) {
+    if (trial.isError || !trial.summary)
+        return [];
+    // Errored siblings produced no output; letting them into the baseline would
+    // drag the median down and mask the degenerate trials this exists to catch.
+    const peers = cohort.filter(t => !t.isError && t.summary);
+    const anomalies = [];
+    const medianOutputLen = median(peers.map(t => t.summary.outputLen));
+    if (medianOutputLen > 0 && trial.summary.outputLen < medianOutputLen * DEGENERATE_OUTPUT_RATIO) {
+        anomalies.push({
+            tag: 'degenerate-output',
+            reason: `Final output is ${trial.summary.outputLen} characters; the cohort median is ${Math.round(medianOutputLen)}.`,
+        });
+    }
+    const medianPassed = median(peers.map(passedAssertions));
+    if (trial.assertionResults.length > 0 && passedAssertions(trial) === 0 && medianPassed > 0) {
+        anomalies.push({
+            tag: 'zero-assertions',
+            reason: `Passed 0 of ${trial.assertionResults.length} assertions; the cohort median is ${medianPassed}.`,
+        });
+    }
+    if (trial.summary.outputLen === 0 || trial.summary.stopStatus !== 'success') {
+        anomalies.push({
+            tag: 'premature-stop',
+            reason: `Run ended with status "${trial.summary.stopStatus ?? 'unknown'}" and ${trial.summary.outputLen} characters of output.`,
+        });
+    }
+    const medianTokens = median(peers.filter(t => t.tokenStats).map(t => t.tokenStats.totalTokens));
+    if (trial.tokenStats && medianTokens > 0 &&
+        trial.tokenStats.totalTokens > medianTokens * RESOURCE_OUTLIER_RATIO) {
+        anomalies.push({
+            tag: 'resource-outlier',
+            reason: `Spent ${trial.tokenStats.totalTokens.toLocaleString('en-US')} tokens against a cohort median of ${Math.round(medianTokens).toLocaleString('en-US')}.`,
+        });
+    }
+    return anomalies;
+}

package/dist/core/eval-runner.js

@@ -4,6 +4,7 @@ import { executor } from '../utils/exec.js';
 import { EvalEnvironment } from './environment.js';
 import { RunnerFactory } from '../runners/index.js';
 import { TriggerGrader, ModelBasedGrader } from './evaluator.js';
+import { buildTrialSummary } from './trial-utils.js';
 import { parseStreamResult, parseTokenStats } from '../utils/ndjson.js';
 // Replace filesystem-unsafe characters so variants like 'ref:main' or 'feature/x' can be used in filenames.
 function slugifyVariant(v) {
@@ -26,7 +27,7 @@ export class EvalRunner {
     async runTriggerTask(task, index, trialId, uiCtx, attempt = 0) {
         const variantSlug = slugifyVariant(this.options.variant ?? 'local');
         const logFileName = `task_${task.id}_${variantSlug}_trial_${trialId}.log`;
-        const logPath = this.options.debug ? path.join(this.options.runDir, logFileName) : undefined;
+        const logPath = path.join(this.options.runDir, logFileName);
         let worktreePath;
         let transcript = null;
         let durationMs;
@@ -65,6 +66,8 @@ export class EvalRunner {
         const tokenStats = transcript
             ? parseTokenStats(transcript.response || '') ?? undefined
             : undefined;
+        // Compact evidence for the report. The full stream stays in the trial log.
+        const summary = buildTrialSummary(transcript?.response || '', logFileName);
         // Negative evals (should_trigger: false) assert the opposite: the skill must NOT activate.
         const shouldTrigger = task.should_trigger !== false;
         const assertionLabel = shouldTrigger ? 'Skill was triggered' : 'Skill was not triggered';
@@ -99,7 +102,8 @@ export class EvalRunner {
                     trialPassed: false,
                     isError: true,
                     tokenStats,
-                    durationMs
+                    durationMs,
+                    summary
                 };
             }
             else {
@@ -129,7 +133,8 @@ export class EvalRunner {
                 trialPassed: false,
                 isError: true,
                 tokenStats,
-                durationMs
+                durationMs,
+                summary
             };
         }
         return {
@@ -138,7 +143,8 @@ export class EvalRunner {
             assertionResults: assertionResults,
             trialPassed,
             tokenStats,
-            durationMs
+            durationMs,
+            summary
         };
     }
     async runFunctionalTask(task, index, trialId, uiCtx, attempt = 0) {
@@ -149,7 +155,7 @@ export class EvalRunner {
             : `${task.prompt}\n\nIMPORTANT: You must use the '${this.options.skillName}' skill/tool to solve this task.`;
         const variantSlug = slugifyVariant(this.options.variant ?? (skillDisabled ? 'baseline' : 'local'));
         const logFileName = `task_${task.id}_${variantSlug}_trial_${trialId}.log`;
-        const logPath = this.options.debug ? path.join(this.options.runDir, logFileName) : undefined;
+        const logPath = path.join(this.options.runDir, logFileName);
         let worktreePath;
         let assertionResults = [];
         let trialPassed = false;
@@ -188,6 +194,8 @@ export class EvalRunner {
             const tokenStats = transcript
                 ? parseTokenStats(transcript.response || '') ?? undefined
                 : undefined;
+            // Compact evidence for the report. The full stream stays in the trial log.
+            const summary = buildTrialSummary(transcript?.response || '', logFileName);
             if (transcript && !transcript.error) {
                 if (skillDisabled && this.triggerGrader.detectSkillAttempt(transcript)) {
                     return {
@@ -201,7 +209,8 @@ export class EvalRunner {
                             }],
                         trialPassed: false,
                         tokenStats,
-                        durationMs
+                        durationMs,
+                        summary
                     };
                 }
                 if (!skillDisabled && !this.triggerGrader.gradeTrigger(transcript)) {
@@ -216,7 +225,8 @@ export class EvalRunner {
                             }],
                         trialPassed: false,
                         tokenStats,
-                        durationMs
+                        durationMs,
+                        summary
                     };
                 }
                 let context = 'No changes detected or git not available.';
@@ -273,7 +283,8 @@ export class EvalRunner {
                             })),
                             trialPassed: false,
                             tokenStats,
-                            durationMs
+                            durationMs,
+                            summary
                         };
                     }
                     trialPassed = assertionResults.every(r => r.passed);
@@ -287,7 +298,8 @@ export class EvalRunner {
                     assertionResults: assertionResults,
                     trialPassed,
                     tokenStats,
-                    durationMs
+                    durationMs,
+                    summary
                 };
             }
             else {
@@ -308,7 +320,8 @@ export class EvalRunner {
                     trialPassed: false,
                     isError: true,
                     tokenStats,
-                    durationMs
+                    durationMs,
+                    summary
                 };
             }
         }
@@ -322,13 +335,16 @@ export class EvalRunner {
                     graderType: 'model-based'
                 }));
             }
-            // isError return — finally still runs cleanup
+            // isError return — finally still runs cleanup.
+            // The agent crashed before a summary could be built, but the log may hold
+            // a partial stream, so keep the pointer to it.
             return {
                 id: trialId,
                 transcript: { error: errorMsg },
                 assertionResults,
                 trialPassed: false,
-                isError: true
+                isError: true,
+                summary: buildTrialSummary('', logFileName)
             };
         }
         finally {

package/dist/core/statistics.js

@@ -1,9 +1,10 @@
+import { isCounted } from './trial-utils.js';
 /**
- * Computes the fraction of individual assertions that passed across all non-error trials.
- * Returns 0 if there are no non-error trials or no assertions.
+ * Computes the fraction of individual assertions that passed across all counted trials.
+ * Returns 0 if there are no counted trials or no assertions.
  */
 export function computeAssertionPassRate(trials) {
-    const relevant = trials.filter(t => !t.isError);
+    const relevant = trials.filter(isCounted);
     if (relevant.length === 0)
         return 0;
     const total = relevant.reduce((s, t) => s + t.assertionResults.length, 0);
@@ -21,14 +22,16 @@ export function aggregateAssertionPassRate(results, trialSelector) {
     return results.reduce((sum, r) => sum + computeAssertionPassRate(trialSelector(r)), 0) / results.length;
 }
 /**
- * Computes the pass rate (pass@1) for a set of trials.
- * Returns the fraction of trials that passed: c / n.
+ * Computes the pass rate (pass@1) for a set of trials: c / n over counted trials.
+ * Infrastructure errors leave the denominator rather than counting as failures,
+ * matching computeAssertionPassRate — see isCounted().
+ * Returns 0 when no trial counted.
  */
 export function computePassAtK(trials, _k = 1) {
-    const n = trials.length;
-    if (n === 0)
+    const relevant = trials.filter(isCounted);
+    if (relevant.length === 0)
         return 0;
-    return trials.filter(t => t.trialPassed).length / n;
+    return relevant.filter(t => t.trialPassed).length / relevant.length;
 }
 /**
  * Aggregates pass@1 across all task results.

package/dist/core/trial-utils.js

@@ -1,3 +1,28 @@
+import { parseStreamResult, parseStreamStats } from '../utils/ndjson.js';
+/**
+ * Cap on the agent text carried into the report. The full text always stays in
+ * the trial log, which the report links to; this bound keeps a run with dozens
+ * of trials from producing a multi-megabyte HTML file.
+ */
+export const MAX_SUMMARY_OUTPUT = 4000;
+/**
+ * Builds the compact record the report shows for a trial.
+ * `rawStream` is the agent's raw NDJSON stdout; an empty or unparsable stream
+ * yields an empty output, which is itself the signal that the agent produced
+ * nothing usable.
+ */
+export function buildTrialSummary(rawStream, logFile) {
+    const parsed = parseStreamResult(rawStream);
+    const text = parsed && 'response' in parsed ? parsed.response : '';
+    const { toolCalls, status } = parseStreamStats(rawStream);
+    return {
+        output: text.slice(0, MAX_SUMMARY_OUTPUT),
+        outputLen: text.length,
+        toolCalls,
+        stopStatus: status,
+        logFile,
+    };
+}
 /**
  * Returns true when a trial represents an infrastructure failure (timeout, blocked
  * interactive prompt, runner crash, etc.) rather than a legitimate judge verdict.
@@ -6,6 +31,21 @@
 export function isTrialError(trial) {
     return trial.isError === true;
 }
+/**
+ * Whether a trial contributes to metrics.
+ *
+ * Infrastructure errors (timeout, blocked interactive prompt, runner crash)
+ * never reached a verdict, so they leave the denominator entirely instead of
+ * counting as failures: a harness timeout says nothing about the skill. The
+ * report shows the effective n alongside every rate so an incomplete run is
+ * still visible as one.
+ *
+ * The report applies the same rule to trials a reviewer excluded. Exclusions
+ * are made after the run, against the report, so they are not visible here.
+ */
+export function isCounted(trial) {
+    return !trial.isError;
+}
 /**
  * Runs fn(), retrying up to maxRetries additional times with exponential backoff
  * whenever the result is an infrastructure-error trial (isTrialError returns true).
@@ -33,7 +73,11 @@ export async function withRetry(fn, maxRetries = 2, baseDelayMs = 1000, onRetry)
 }
 /**
  * Pads the trials array up to targetCount when a trial loop aborts early.
- * Ensures that pass@k calculations always reflect the full requested trial count.
+ *
+ * The padding records that N trials were requested but never ran, so the report
+ * can show "n=2/5" instead of silently presenting a two-trial measurement as if
+ * it were the full run. It does not push the rates down: padded entries are
+ * infrastructure errors, and isCounted() keeps those out of every denominator.
  *
  * @param trials     Trials collected so far (may be shorter than targetCount).
  * @param targetCount The requested number of trials (numTrials).

package/dist/index.js

@@ -41,7 +41,7 @@ program
     .name('skill-eval')
     .description('CLI to evaluate agent skills triggering and functionality')
     .version(pkg.version)
-    .option('-v, --debug', 'Enable debug logging', false);
+    .option('-v, --debug', 'Print verbose logs to the console (trial transcripts are always saved)', false);
 program.on('option:debug', () => {
     process.env.DEBUG = 'true';
 });