@cogitator-ai/evals 0.13.2 → 0.14.0

package/README.md

@@ -27,6 +27,7 @@ Website docs: [Evals](https://cogitator.app/docs/evals), [Datasets](https://cogi
 - **Assertions** — threshold, noRegression, custom assertion with auto-detection of lower-is-better metrics
 - **A/B Testing** - EvalComparison with McNemar's test (exact for few discordant pairs), the paired t-test or Wilcoxon signed-rank, Holm's correction across metrics, bootstrap confidence intervals, effect sizes, latency and cost comparisons and pairwise judging
 - **Experiment Tracking** - named runs with metadata saved as versioned run files, a `RunStore` directory of runs, `compareRuns()` with the A/B statistics, per-case regressions and improvements, and warnings when datasets, prompt versions, metrics or judges differ. Agent cases keep the `traceId` and versioned `prompt` of their runs
+- **Plugin Evals** - `experimental_runPluginEvals()` runs a plugin's eval suite in the format of Claude Code's `claude plugin eval`, with and without the plugin, six grader types, MCP mocks and a JSON and HTML report
 - **8 Reporters** - console (colored table), JSON, CSV, JUnit XML, Markdown (pull request comments), self-contained HTML, GitHub Actions annotations and job summary, CI (exit code on failure), plus `reportComparison()` for comparisons
 - **Vitest** - `describeEval` from `@cogitator-ai/evals/vitest` runs a suite as a test file, one test per case plus one for the assertions, and `evalMatchers` from `@cogitator-ai/evals/matchers` (Vitest 5) adds `toHaveCalledTool`, `toFollowTrajectory`, `toHaveRetrieved` and `toScoreAtLeast`
 - **Builder API** — Fluent `EvalBuilder` for composable eval pipelines
@@ -676,6 +677,83 @@ if (base) {
 
 ---
 
+## Plugin Evals
+
+Experimental. `experimental_runPluginEvals()` runs a plugin's eval suite in the format of Claude Code's `claude plugin eval`: each case's prompt several times with the plugin loaded (the with-arm) and with no plugin (the without-arm), scores every run with the case's graders and writes `aggregate-result.json` and a self-contained `report.html`. From the command line it is `cogitator plugin eval`. Needs `@cogitator-ai/core`.
+
+```typescript
+import { experimental_runPluginEvals } from '@cogitator-ai/evals';
+
+const { result, reportFile } = await experimental_runPluginEvals({
+  target: './my-plugin',
+  llm: { defaultModel: 'anthropic/claude-sonnet-5-5' },
+  judgeModel: 'anthropic/claude-haiku-4-5',
+  threshold: 0.8,
+  maxCostUsd: 5,
+});
+
+console.log(result.aggregates.overallScore, result.aggregates.meanDelta, reportFile);
+```
+
+A suite lives in `evals/` of the plugin, one directory per case with a `prompt.md` (frontmatter with the run settings, body the prompt) or a `case.yaml`, and graders in `graders/<name>.md` or under `graders:` of `case.yaml`:
+
+```text
+my-plugin/
+├── .claude-plugin/plugin.json
+├── skills/commit-message/SKILL.md
+└── evals/
+    ├── writes-commit-message/
+    │   ├── prompt.md
+    │   └── graders/
+    │       ├── skill-fired.md
+    │       └── criteria.md
+    └── mocks/<server>/<tool>.md
+```
+
+```markdown
+---
+runs: 3
+max_turns: 10
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+Write me a commit message for this change: I renamed getUser to fetchUser and updated the three call sites.
+```
+
+Each run gets a fresh temporary workspace, a new runtime with only the plugin under test (no user or project settings, hooks, memory or other plugins) and only the tools the case may use: the read-only tools its `allowed_tools` lists (`Read`, `Glob`, `Grep`, `Skill`, `Agent`, `TodoWrite` and the rest of Claude Code's read-only set) plus what `allowTools` grants for the whole run, such as `Write` or `Bash(npm test *)`. A tool that is not granted is not given to the model, and runs never stop to ask. Claude Code's tool names map to Cogitator's (`Read` to `read_file`, `Bash` to `run_command`, `Skill` to `use_skill`, `Agent` to `agent`), and graders see the calls under Claude Code's names and input shapes. A `read_skill_file` call keeps its own name, it is not a `Skill` call.
+
+The six grader types are `regex` (over `last_message`, `trace`, `files`, `mock_calls` or `{ source: file, path }`), `tool_used`, `tool_order`, `file_exists`, `llm` and `baseline`. The last two ask the judge three times and pass on two PASS votes. A run's score is the weighted share of its graders that passed, a case's score the mean of its with-arm runs, and its delta the with-arm score minus the without-arm score. In a two-arm run, graders that can only pass with the plugin (`tool_used: Skill`, mock call graders of the plugin's own servers, `arm: with-only`) are reported as plugin-fired indicators with `scored: false`.
+
+| Option        | Default                                             | Description                                                                                                                |
+| ------------- | --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
+| `target`      | the working directory                               | A plugin directory, a `prompt.md` or `case.yaml`, or an installed plugin by name                                           |
+| `llm`         | required                                            | The LLM configuration of the runs and the judge: providers, keys, backends, model                                          |
+| `model`       | each case's `model`, else `llm.defaultModel`        | Model of the agent under test                                                                                              |
+| `judgeModel`  | each case's agent model                             | Model of `llm` and `baseline` graders                                                                                      |
+| `runs`        | each case's `runs`, else 3                          | Runs per case per arm, 1 to 50                                                                                             |
+| `concurrency` | 1                                                   | Agent runs at once, 1 to 8. Results keep case order                                                                        |
+| `ablation`    | decided per case                                    | `none` runs the with-arm only, `with-without` adds the baseline to every case                                              |
+| `threshold`   | 1                                                   | A case passes when its with-arm score is at least this                                                                     |
+| `maxCostUsd`  | none                                                | Once the cost estimate reaches it no further run starts, and the result is partial                                         |
+| `allowTools`  | none                                                | Tools granted beyond the read-only set, as permission rules                                                                |
+| `scaffold`    | false                                               | Run each case's `context.scaffold_script`                                                                                  |
+| `caseGlob`    | none                                                | Only the cases whose name matches                                                                                          |
+| `tags`        | none                                                | Only the cases with any of these tags                                                                                      |
+| `evalDir`     | `experimental.evals` of `plugin.json`, else `evals` | The eval directory, a relative path of plain directory names such as `./qa`. In `plugin.json` a list takes its first entry |
+| `outputDir`   | `<evalDir>/results/<timestamp>/`                    | Where `aggregate-result.json` and `report.html` go                                                                         |
+| `keepTemp`    | false                                               | Keep every run's temporary directory                                                                                       |
+| `exec`        | none                                                | How granted `Bash` commands are spawned, such as a `launch` that adds an OS sandbox                                        |
+| `signal`      | none                                                | Stops starting runs: the result is partial, `interrupted`                                                                  |
+| `onProgress`  | none                                                | Hears warnings, each case with its notices, each finished run, and skipped runs                                            |
+
+The result, `PluginEvalResult`, is the document written to `aggregate-result.json`: `schemaVersion: 1`, `cogitatorVersion`, `outcome` (`passed`, `failed`, `partial` or `interrupted`), `partial` and `partialReason` (`cost_ceiling`, `interrupted` or `auth_failed`), `aggregates` (`overallScore`, `casesPassed`, `casesTotal`, `meanDelta`, `baselineScore`, `perfectRuns`), `apiErrors`, `loadErrors`, `costUsd`, `durationSeconds`, and `cases[]` with `aggregates.score` and `delta` and every run of `arms.with` and `arms.without` with its `error`, `aborted`, `skippedPaidGraders` and grader results with explanations and judge votes. A suite that cannot start (no plugin, no case, an invalid option) throws `PluginEvalError`, a case that does not load is reported with its `error` and scores 0.
+
+Through `cogitator plugin eval` the outcome is the exit code: `0` when every case met the threshold, `1` for a case below it, a case that did not load, a run that ended on a provider error or an invalid or unknown option, `2` for a partial run, `130` when interrupted and `143` when terminated. Claude Code's `--trust-plugin`, `--no-publish`, `--publish-report` and `--mocks record` are accepted and change nothing, `--mocks off` and `--allow-real-servers` exit 1.
+
+Supported from Claude Code's format: `prompt.md` and `case.yaml` with every field (an unknown key is an error), `context.scaffold_script`, `context.history_file` (such a case runs one arm by default), `context.add_dirs` (read-only), `env` keys matching `EVAL_[A-Z0-9_]*`, every grader type and option, the exclusion rules of two-arm runs, and `fixed` MCP mocks with `{{input.<field>}}` and `{{file:...}}` substitutions, `expect:` guards, `error: true` and `_tools.json`. Not supported: `type: agent` and `_server.md` mocks (a case that uses the server, through its own `mocks/` or a plugin that declares it, fails to load and names the file, other cases run), `--mocks off` and the plugin's real MCP servers, skills-directory plugins (`name@skills-dir`), images shown to the `llm` judge (a binary file fails the grader), the trust prompt and report publishing. Granted `Bash` commands run in the workspace directory without an OS sandbox unless `exec.launch` adds one, and `env` reaches those commands only. The judge votes at temperature 1 on each case's agent model unless `judgeModel` names another.
+
+---
+
 ## Builder API
 
 `EvalBuilder` provides a fluent interface for constructing eval suites.
@@ -754,20 +832,22 @@ result.report('ci');
 
 ### Core
 
-| Export           | Description                                                         |
-| ---------------- | ------------------------------------------------------------------- |
-| `EvalSuite`      | Main evaluation runner                                              |
-| `EvalComparison` | A/B testing runner with statistical significance                    |
-| `calibrateJudge` | Agreement of a judge metric with labelled examples                  |
-| `EvalBuilder`    | Fluent builder for EvalSuite                                        |
-| `compareRuns`    | Compares two saved runs or suite results case by case               |
-| `RunStore`       | Directory of run files: save, list, load, latest                    |
-| `loadRun`        | Reads and validates a run file                                      |
-| `caseKey`        | The key a case is matched by across runs                            |
-| `Dataset`        | Immutable dataset with from/fromJsonl/fromCsv/filter/sample/shuffle |
-| `loadJsonl`      | Low-level JSONL file loader                                         |
-| `loadCsv`        | Low-level CSV file loader                                           |
-| `isLLMMetric`    | Type guard: checks if a `MetricFn` is an `LLMMetricFn`              |
+| Export                        | Description                                                         |
+| ----------------------------- | ------------------------------------------------------------------- |
+| `EvalSuite`                   | Main evaluation runner                                              |
+| `EvalComparison`              | A/B testing runner with statistical significance                    |
+| `calibrateJudge`              | Agreement of a judge metric with labelled examples                  |
+| `EvalBuilder`                 | Fluent builder for EvalSuite                                        |
+| `compareRuns`                 | Compares two saved runs or suite results case by case               |
+| `RunStore`                    | Directory of run files: save, list, load, latest                    |
+| `loadRun`                     | Reads and validates a run file                                      |
+| `caseKey`                     | The key a case is matched by across runs                            |
+| `Dataset`                     | Immutable dataset with from/fromJsonl/fromCsv/filter/sample/shuffle |
+| `loadJsonl`                   | Low-level JSONL file loader                                         |
+| `loadCsv`                     | Low-level CSV file loader                                           |
+| `isLLMMetric`                 | Type guard: checks if a `MetricFn` is an `LLMMetricFn`              |
+| `experimental_runPluginEvals` | Runs a plugin's eval suite, Claude Code's plugin eval format        |
+| `PluginEvalError`             | Thrown when a plugin eval run cannot start                          |
 
 ### Metrics
 

package/dist/index.d.ts

@@ -49,5 +49,7 @@ export { pearson, spearman } from './stats/correlation.js';
 export { mean, median, stdDev, percentile, aggregate } from './stats/percentiles.js';
 export type { EvalCase, ExpectedToolCall, EvalSuiteConfig, JudgeConfig, JudgeConfigInput, JudgeCache, JudgeCogitator, EvalComparisonConfig, } from './schema.js';
 export { createRunEvalTool, evalTools } from './tools.js';
+export { experimental_runPluginEvals, PluginEvalError } from './plugin-evals/index.js';
+export type { PluginEvalAblation, PluginEvalAbort, PluginEvalArm, PluginEvalBaselineGrader, PluginEvalCaseAggregates, PluginEvalCaseResult, PluginEvalExpectation, PluginEvalExpectType, PluginEvalFileExistsGrader, PluginEvalGrader, PluginEvalGraderResult, PluginEvalLlmGrader, PluginEvalMockCall, PluginEvalOptions, PluginEvalOutcome, PluginEvalPartialReason, PluginEvalProgressEvent, PluginEvalRegexGrader, PluginEvalReport, PluginEvalResult, PluginEvalRunResult, PluginEvalTarget, PluginEvalToolCall, PluginEvalToolMatcher, PluginEvalToolOrderGrader, PluginEvalToolUsedGrader, PluginEvalVote, } from './plugin-evals/index.js';
 export type { EvalTool } from './tools.js';
 //# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAIA,eAAO,MAAM,OAAO,EAAE,MAA4B,CAAC;AAEnD,OAAO,EAAE,SAAS,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AACtD,YAAY,EACV,UAAU,EACV,iBAAiB,EACjB,QAAQ,EACR,mBAAmB,EACnB,YAAY,EACZ,cAAc,EACd,gBAAgB,EAChB,eAAe,EACf,cAAc,GACf,MAAM,cAAc,CAAC;AACtB,OAAO,EAAE,cAAc,EAAE,MAAM,mBAAmB,CAAC;AACnD,YAAY,EACV,qBAAqB,EACrB,wBAAwB,EACxB,gBAAgB,EAChB,kBAAkB,EAClB,gBAAgB,EAChB,cAAc,EACd,gBAAgB,GACjB,MAAM,mBAAmB,CAAC;AAC3B,YAAY,EACV,eAAe,EACf,cAAc,EACd,YAAY,EACZ,eAAe,EACf,YAAY,EACZ,eAAe,GAChB,MAAM,YAAY,CAAC;AACpB,OAAO,EAAE,cAAc,EAAE,MAAM,mBAAmB,CAAC;AACnD,YAAY,EACV,kBAAkB,EAClB,qBAAqB,EACrB,eAAe,EACf,gBAAgB,GACjB,MAAM,mBAAmB,CAAC;AAC3B,OAAO,EAAE,WAAW,EAAE,MAAM,gBAAgB,CAAC;AAE7C,OAAO,EAAE,OAAO,EAAE,MAAM,YAAY,CAAC;AACrC,OAAO,EAAE,SAAS,EAAE,MAAM,YAAY,CAAC;AACvC,OAAO,EAAE,OAAO,EAAE,MAAM,YAAY,CAAC;AAErC,OAAO,EACL,UAAU,EACV,QAAQ,EACR,WAAW,EACX,WAAW,EACX,UAAU,EACV,OAAO,EACP,KAAK,EACL,UAAU,GACX,MAAM,yBAAyB,CAAC;AACjC,OAAO,EACL,YAAY,EACZ,SAAS,EACT,SAAS,EACT,WAAW,EACX,SAAS,EACT,gBAAgB,EAChB,gBAAgB,GACjB,MAAM,qBAAqB,CAAC;AAC7B,YAAY,EACV,WAAW,EACX,gBAAgB,EAChB,mBAAmB,EACnB,mBAAmB,EACnB,WAAW,EACX,YAAY,EACZ,YAAY,EACZ,eAAe,EACf,WAAW,EACX,YAAY,EACZ,UAAU,EACV,UAAU,GACX,MAAM,qBAAqB,CAAC;AAC7B,OAAO,EAAE,KAAK,EAAE,MAAM,kBAAkB,CAAC;AACzC,YAAY,EAAE,YAAY,EAAE,MAAM,kBAAkB,CAAC;AACrD,OAAO,EACL,OAAO,EACP,IAAI,EACJ,UAAU,EACV,OAAO,EACP,eAAe,EACf,eAAe,GAChB,MAAM,uBAAuB,CAAC;AAC/B,YAAY,EAAE,cAAc,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAC;AAC7E,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,eAAe,EAAE,eAAe,EAAE,MAAM,sBAAsB,CAAC;AAC/F,YAAY,EACV,gBAAgB,EAChB,iBAAiB,EACjB,kBAAkB,EAClB,cAAc,EACd,eAAe,GAChB,MAAM,sBAAsB,CAAC;AAC9B,OAAO,EACL,eAAe,EACf,kBAAkB,EAClB,YAAY,EACZ,iBAAiB,GAClB,MAAM,qBAAqB,CAAC;AAC7B,OAAO,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAC;AACpD,OAAO,EAAE,MAAM,EAAE,MAAM,kBAAkB,CAAC;AAC1C,YAAY,EACV,QAAQ,EACR,WAAW,EACX,WAAW,EACX,aAAa,EACb,cAAc,EACd,UAAU,EACV,gBAAgB,EAChB,YAAY,EACZ,cAAc,EACd,cAAc,EACd,OAAO,EACP,mBAAmB,EACnB,gBAAgB,GACjB,MAAM,iBAAiB,CAAC;AACzB,YAAY,EAAE,kBAAkB,EAAE,gBAAgB,EAAE,MAAM,kBAAkB,CAAC;AAE7E,OAAO,EAAE,SAAS,EAAE,MAAM,cAAc,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,SAAS,EAAE,MAAM,cAAc,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,YAAY,EAAE,WAAW,EAAE,eAAe,EAAE,gBAAgB,EAAE,SAAS,EAAE,MAAM,cAAc,CAAC;AAE9F,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AACrC,YAAY,EAAE,YAAY,EAAE,eAAe,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC/E,OAAO,EAAE,gBAAgB,EAAE,kBAAkB,EAAE,cAAc,EAAE,MAAM,wBAAwB,CAAC;AAC9F,YAAY,EAAE,oBAAoB,EAAE,sBAAsB,EAAE,MAAM,wBAAwB,CAAC;AAE3F,OAAO,EACL,OAAO,EACP,OAAO,EACP,QAAQ,EACR,WAAW,EACX,aAAa,EACb,eAAe,EACf,gBAAgB,GACjB,MAAM,QAAQ,CAAC;AAChB,YAAY,EACV,OAAO,EACP,WAAW,EACX,aAAa,EACb,aAAa,EACb,YAAY,EACZ,UAAU,EACV,SAAS,EACT,eAAe,EACf,cAAc,EACd,kBAAkB,EAClB,aAAa,EACb,mBAAmB,EACnB,aAAa,EACb,UAAU,EACV,MAAM,EACN,SAAS,GACV,MAAM,QAAQ,CAAC;AAEhB,OAAO,EAAE,WAAW,EAAE,MAAM,gBAAgB,CAAC;AAC7C,YAAY,EAAE,WAAW,EAAE,MAAM,gBAAgB,CAAC;AAClD,OAAO,EAAE,YAAY,EAAE,YAAY,EAAE,MAAM,iBAAiB,CAAC;AAC7D,YAAY,EAAE,aAAa,EAAE,kBAAkB,EAAE,MAAM,iBAAiB,CAAC;AACzE,OAAO,EAAE,kBAAkB,EAAE,MAAM,kBAAkB,CAAC;AACtD,YAAY,EAAE,cAAc,EAAE,MAAM,kBAAkB,CAAC;AACvD,OAAO,EAAE,WAAW,EAAE,MAAM,mBAAmB,CAAC;AAChD,YAAY,EAAE,gBAAgB,EAAE,iBAAiB,EAAE,MAAM,mBAAmB,CAAC;AAC7E,OAAO,EAAE,OAAO,EAAE,MAAM,qBAAqB,CAAC;AAC9C,OAAO,EAAE,IAAI,EAAE,MAAM,cAAc,CAAC;AACpC,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,MAAM,qBAAqB,CAAC;AACxD,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,UAAU,EAAE,SAAS,EAAE,MAAM,qBAAqB,CAAC;AAElF,YAAY,EACV,QAAQ,EACR,gBAAgB,EAChB,eAAe,EACf,WAAW,EACX,gBAAgB,EAChB,UAAU,EACV,cAAc,EACd,oBAAoB,GACrB,MAAM,UAAU,CAAC;AAElB,OAAO,EAAE,iBAAiB,EAAE,SAAS,EAAE,MAAM,SAAS,CAAC;AACvD,YAAY,EAAE,QAAQ,EAAE,MAAM,SAAS,CAAC"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAIA,eAAO,MAAM,OAAO,EAAE,MAA4B,CAAC;AAEnD,OAAO,EAAE,SAAS,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AACtD,YAAY,EACV,UAAU,EACV,iBAAiB,EACjB,QAAQ,EACR,mBAAmB,EACnB,YAAY,EACZ,cAAc,EACd,gBAAgB,EAChB,eAAe,EACf,cAAc,GACf,MAAM,cAAc,CAAC;AACtB,OAAO,EAAE,cAAc,EAAE,MAAM,mBAAmB,CAAC;AACnD,YAAY,EACV,qBAAqB,EACrB,wBAAwB,EACxB,gBAAgB,EAChB,kBAAkB,EAClB,gBAAgB,EAChB,cAAc,EACd,gBAAgB,GACjB,MAAM,mBAAmB,CAAC;AAC3B,YAAY,EACV,eAAe,EACf,cAAc,EACd,YAAY,EACZ,eAAe,EACf,YAAY,EACZ,eAAe,GAChB,MAAM,YAAY,CAAC;AACpB,OAAO,EAAE,cAAc,EAAE,MAAM,mBAAmB,CAAC;AACnD,YAAY,EACV,kBAAkB,EAClB,qBAAqB,EACrB,eAAe,EACf,gBAAgB,GACjB,MAAM,mBAAmB,CAAC;AAC3B,OAAO,EAAE,WAAW,EAAE,MAAM,gBAAgB,CAAC;AAE7C,OAAO,EAAE,OAAO,EAAE,MAAM,YAAY,CAAC;AACrC,OAAO,EAAE,SAAS,EAAE,MAAM,YAAY,CAAC;AACvC,OAAO,EAAE,OAAO,EAAE,MAAM,YAAY,CAAC;AAErC,OAAO,EACL,UAAU,EACV,QAAQ,EACR,WAAW,EACX,WAAW,EACX,UAAU,EACV,OAAO,EACP,KAAK,EACL,UAAU,GACX,MAAM,yBAAyB,CAAC;AACjC,OAAO,EACL,YAAY,EACZ,SAAS,EACT,SAAS,EACT,WAAW,EACX,SAAS,EACT,gBAAgB,EAChB,gBAAgB,GACjB,MAAM,qBAAqB,CAAC;AAC7B,YAAY,EACV,WAAW,EACX,gBAAgB,EAChB,mBAAmB,EACnB,mBAAmB,EACnB,WAAW,EACX,YAAY,EACZ,YAAY,EACZ,eAAe,EACf,WAAW,EACX,YAAY,EACZ,UAAU,EACV,UAAU,GACX,MAAM,qBAAqB,CAAC;AAC7B,OAAO,EAAE,KAAK,EAAE,MAAM,kBAAkB,CAAC;AACzC,YAAY,EAAE,YAAY,EAAE,MAAM,kBAAkB,CAAC;AACrD,OAAO,EACL,OAAO,EACP,IAAI,EACJ,UAAU,EACV,OAAO,EACP,eAAe,EACf,eAAe,GAChB,MAAM,uBAAuB,CAAC;AAC/B,YAAY,EAAE,cAAc,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAC;AAC7E,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,eAAe,EAAE,eAAe,EAAE,MAAM,sBAAsB,CAAC;AAC/F,YAAY,EACV,gBAAgB,EAChB,iBAAiB,EACjB,kBAAkB,EAClB,cAAc,EACd,eAAe,GAChB,MAAM,sBAAsB,CAAC;AAC9B,OAAO,EACL,eAAe,EACf,kBAAkB,EAClB,YAAY,EACZ,iBAAiB,GAClB,MAAM,qBAAqB,CAAC;AAC7B,OAAO,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAC;AACpD,OAAO,EAAE,MAAM,EAAE,MAAM,kBAAkB,CAAC;AAC1C,YAAY,EACV,QAAQ,EACR,WAAW,EACX,WAAW,EACX,aAAa,EACb,cAAc,EACd,UAAU,EACV,gBAAgB,EAChB,YAAY,EACZ,cAAc,EACd,cAAc,EACd,OAAO,EACP,mBAAmB,EACnB,gBAAgB,GACjB,MAAM,iBAAiB,CAAC;AACzB,YAAY,EAAE,kBAAkB,EAAE,gBAAgB,EAAE,MAAM,kBAAkB,CAAC;AAE7E,OAAO,EAAE,SAAS,EAAE,MAAM,cAAc,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,SAAS,EAAE,MAAM,cAAc,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,YAAY,EAAE,WAAW,EAAE,eAAe,EAAE,gBAAgB,EAAE,SAAS,EAAE,MAAM,cAAc,CAAC;AAE9F,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AACrC,YAAY,EAAE,YAAY,EAAE,eAAe,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC/E,OAAO,EAAE,gBAAgB,EAAE,kBAAkB,EAAE,cAAc,EAAE,MAAM,wBAAwB,CAAC;AAC9F,YAAY,EAAE,oBAAoB,EAAE,sBAAsB,EAAE,MAAM,wBAAwB,CAAC;AAE3F,OAAO,EACL,OAAO,EACP,OAAO,EACP,QAAQ,EACR,WAAW,EACX,aAAa,EACb,eAAe,EACf,gBAAgB,GACjB,MAAM,QAAQ,CAAC;AAChB,YAAY,EACV,OAAO,EACP,WAAW,EACX,aAAa,EACb,aAAa,EACb,YAAY,EACZ,UAAU,EACV,SAAS,EACT,eAAe,EACf,cAAc,EACd,kBAAkB,EAClB,aAAa,EACb,mBAAmB,EACnB,aAAa,EACb,UAAU,EACV,MAAM,EACN,SAAS,GACV,MAAM,QAAQ,CAAC;AAEhB,OAAO,EAAE,WAAW,EAAE,MAAM,gBAAgB,CAAC;AAC7C,YAAY,EAAE,WAAW,EAAE,MAAM,gBAAgB,CAAC;AAClD,OAAO,EAAE,YAAY,EAAE,YAAY,EAAE,MAAM,iBAAiB,CAAC;AAC7D,YAAY,EAAE,aAAa,EAAE,kBAAkB,EAAE,MAAM,iBAAiB,CAAC;AACzE,OAAO,EAAE,kBAAkB,EAAE,MAAM,kBAAkB,CAAC;AACtD,YAAY,EAAE,cAAc,EAAE,MAAM,kBAAkB,CAAC;AACvD,OAAO,EAAE,WAAW,EAAE,MAAM,mBAAmB,CAAC;AAChD,YAAY,EAAE,gBAAgB,EAAE,iBAAiB,EAAE,MAAM,mBAAmB,CAAC;AAC7E,OAAO,EAAE,OAAO,EAAE,MAAM,qBAAqB,CAAC;AAC9C,OAAO,EAAE,IAAI,EAAE,MAAM,cAAc,CAAC;AACpC,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,MAAM,qBAAqB,CAAC;AACxD,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,UAAU,EAAE,SAAS,EAAE,MAAM,qBAAqB,CAAC;AAElF,YAAY,EACV,QAAQ,EACR,gBAAgB,EAChB,eAAe,EACf,WAAW,EACX,gBAAgB,EAChB,UAAU,EACV,cAAc,EACd,oBAAoB,GACrB,MAAM,UAAU,CAAC;AAElB,OAAO,EAAE,iBAAiB,EAAE,SAAS,EAAE,MAAM,SAAS,CAAC;AAEvD,OAAO,EAAE,2BAA2B,EAAE,eAAe,EAAE,MAAM,gBAAgB,CAAC;AAC9E,YAAY,EACV,kBAAkB,EAClB,eAAe,EACf,aAAa,EACb,wBAAwB,EACxB,wBAAwB,EACxB,oBAAoB,EACpB,qBAAqB,EACrB,oBAAoB,EACpB,0BAA0B,EAC1B,gBAAgB,EAChB,sBAAsB,EACtB,mBAAmB,EACnB,kBAAkB,EAClB,iBAAiB,EACjB,iBAAiB,EACjB,uBAAuB,EACvB,uBAAuB,EACvB,qBAAqB,EACrB,gBAAgB,EAChB,gBAAgB,EAChB,mBAAmB,EACnB,gBAAgB,EAChB,kBAAkB,EAClB,qBAAqB,EACrB,yBAAyB,EACzB,wBAAwB,EACxB,cAAc,GACf,MAAM,gBAAgB,CAAC;AACxB,YAAY,EAAE,QAAQ,EAAE,MAAM,SAAS,CAAC"}

package/dist/index.js

@@ -32,4 +32,5 @@ export { holm } from './stats/holm.js';
 export { pearson, spearman } from './stats/correlation.js';
 export { mean, median, stdDev, percentile, aggregate } from './stats/percentiles.js';
 export { createRunEvalTool, evalTools } from './tools.js';
+export { experimental_runPluginEvals, PluginEvalError } from './plugin-evals/index.js';
 //# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -1 +1 @@
-{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAE5C,MAAM,WAAW,GAAG,aAAa,CAAC,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,iBAAiB,CAAwB,CAAC;AAE7F,MAAM,CAAC,MAAM,OAAO,GAAW,WAAW,CAAC,OAAO,CAAC;AAEnD,OAAO,EAAE,SAAS,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AAYtD,OAAO,EAAE,cAAc,EAAE,MAAM,mBAAmB,CAAC;AAkBnD,OAAO,EAAE,cAAc,EAAE,MAAM,mBAAmB,CAAC;AAOnD,OAAO,EAAE,WAAW,EAAE,MAAM,gBAAgB,CAAC;AAE7C,OAAO,EAAE,OAAO,EAAE,MAAM,YAAY,CAAC;AACrC,OAAO,EAAE,SAAS,EAAE,MAAM,YAAY,CAAC;AACvC,OAAO,EAAE,OAAO,EAAE,MAAM,YAAY,CAAC;AAErC,OAAO,EACL,UAAU,EACV,QAAQ,EACR,WAAW,EACX,WAAW,EACX,UAAU,EACV,OAAO,EACP,KAAK,EACL,UAAU,GACX,MAAM,yBAAyB,CAAC;AACjC,OAAO,EACL,YAAY,EACZ,SAAS,EACT,SAAS,EACT,WAAW,EACX,SAAS,EACT,gBAAgB,EAChB,gBAAgB,GACjB,MAAM,qBAAqB,CAAC;AAe7B,OAAO,EAAE,KAAK,EAAE,MAAM,kBAAkB,CAAC;AAEzC,OAAO,EACL,OAAO,EACP,IAAI,EACJ,UAAU,EACV,OAAO,EACP,eAAe,EACf,eAAe,GAChB,MAAM,uBAAuB,CAAC;AAE/B,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,eAAe,EAAE,eAAe,EAAE,MAAM,sBAAsB,CAAC;AAQ/F,OAAO,EACL,eAAe,EACf,kBAAkB,EAClB,YAAY,EACZ,iBAAiB,GAClB,MAAM,qBAAqB,CAAC;AAC7B,OAAO,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAC;AACpD,OAAO,EAAE,MAAM,EAAE,MAAM,kBAAkB,CAAC;AAkB1C,OAAO,EAAE,SAAS,EAAE,MAAM,cAAc,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,SAAS,EAAE,MAAM,cAAc,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAG5C,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAErC,OAAO,EAAE,gBAAgB,EAAE,kBAAkB,EAAE,cAAc,EAAE,MAAM,wBAAwB,CAAC;AAG9F,OAAO,EACL,OAAO,EACP,OAAO,EACP,QAAQ,EACR,WAAW,EACX,aAAa,EACb,eAAe,EACf,gBAAgB,GACjB,MAAM,QAAQ,CAAC;AAoBhB,OAAO,EAAE,WAAW,EAAE,MAAM,gBAAgB,CAAC;AAE7C,OAAO,EAAE,YAAY,EAAE,YAAY,EAAE,MAAM,iBAAiB,CAAC;AAE7D,OAAO,EAAE,kBAAkB,EAAE,MAAM,kBAAkB,CAAC;AAEtD,OAAO,EAAE,WAAW,EAAE,MAAM,mBAAmB,CAAC;AAEhD,OAAO,EAAE,OAAO,EAAE,MAAM,qBAAqB,CAAC;AAC9C,OAAO,EAAE,IAAI,EAAE,MAAM,cAAc,CAAC;AACpC,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,MAAM,qBAAqB,CAAC;AACxD,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,UAAU,EAAE,SAAS,EAAE,MAAM,qBAAqB,CAAC;AAalF,OAAO,EAAE,iBAAiB,EAAE,SAAS,EAAE,MAAM,SAAS,CAAC"}
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAE5C,MAAM,WAAW,GAAG,aAAa,CAAC,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,iBAAiB,CAAwB,CAAC;AAE7F,MAAM,CAAC,MAAM,OAAO,GAAW,WAAW,CAAC,OAAO,CAAC;AAEnD,OAAO,EAAE,SAAS,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AAYtD,OAAO,EAAE,cAAc,EAAE,MAAM,mBAAmB,CAAC;AAkBnD,OAAO,EAAE,cAAc,EAAE,MAAM,mBAAmB,CAAC;AAOnD,OAAO,EAAE,WAAW,EAAE,MAAM,gBAAgB,CAAC;AAE7C,OAAO,EAAE,OAAO,EAAE,MAAM,YAAY,CAAC;AACrC,OAAO,EAAE,SAAS,EAAE,MAAM,YAAY,CAAC;AACvC,OAAO,EAAE,OAAO,EAAE,MAAM,YAAY,CAAC;AAErC,OAAO,EACL,UAAU,EACV,QAAQ,EACR,WAAW,EACX,WAAW,EACX,UAAU,EACV,OAAO,EACP,KAAK,EACL,UAAU,GACX,MAAM,yBAAyB,CAAC;AACjC,OAAO,EACL,YAAY,EACZ,SAAS,EACT,SAAS,EACT,WAAW,EACX,SAAS,EACT,gBAAgB,EAChB,gBAAgB,GACjB,MAAM,qBAAqB,CAAC;AAe7B,OAAO,EAAE,KAAK,EAAE,MAAM,kBAAkB,CAAC;AAEzC,OAAO,EACL,OAAO,EACP,IAAI,EACJ,UAAU,EACV,OAAO,EACP,eAAe,EACf,eAAe,GAChB,MAAM,uBAAuB,CAAC;AAE/B,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,eAAe,EAAE,eAAe,EAAE,MAAM,sBAAsB,CAAC;AAQ/F,OAAO,EACL,eAAe,EACf,kBAAkB,EAClB,YAAY,EACZ,iBAAiB,GAClB,MAAM,qBAAqB,CAAC;AAC7B,OAAO,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAC;AACpD,OAAO,EAAE,MAAM,EAAE,MAAM,kBAAkB,CAAC;AAkB1C,OAAO,EAAE,SAAS,EAAE,MAAM,cAAc,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,SAAS,EAAE,MAAM,cAAc,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAG5C,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAErC,OAAO,EAAE,gBAAgB,EAAE,kBAAkB,EAAE,cAAc,EAAE,MAAM,wBAAwB,CAAC;AAG9F,OAAO,EACL,OAAO,EACP,OAAO,EACP,QAAQ,EACR,WAAW,EACX,aAAa,EACb,eAAe,EACf,gBAAgB,GACjB,MAAM,QAAQ,CAAC;AAoBhB,OAAO,EAAE,WAAW,EAAE,MAAM,gBAAgB,CAAC;AAE7C,OAAO,EAAE,YAAY,EAAE,YAAY,EAAE,MAAM,iBAAiB,CAAC;AAE7D,OAAO,EAAE,kBAAkB,EAAE,MAAM,kBAAkB,CAAC;AAEtD,OAAO,EAAE,WAAW,EAAE,MAAM,mBAAmB,CAAC;AAEhD,OAAO,EAAE,OAAO,EAAE,MAAM,qBAAqB,CAAC;AAC9C,OAAO,EAAE,IAAI,EAAE,MAAM,cAAc,CAAC;AACpC,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,MAAM,qBAAqB,CAAC;AACxD,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,UAAU,EAAE,SAAS,EAAE,MAAM,qBAAqB,CAAC;AAalF,OAAO,EAAE,iBAAiB,EAAE,SAAS,EAAE,MAAM,SAAS,CAAC;AAEvD,OAAO,EAAE,2BAA2B,EAAE,eAAe,EAAE,MAAM,gBAAgB,CAAC"}

package/dist/plugin-evals/execute.d.ts

@@ -0,0 +1,45 @@
+import type { CogitatorConfig } from '@cogitator-ai/core';
+import type { RunCapture } from './graders.js';
+import { type ToolGrant } from './tools.js';
+import type { PluginEvalAbort, PluginEvalArm, PluginEvalCase, PluginEvalOptions } from './types.js';
+type Core = typeof import('@cogitator-ai/core');
+/** How one run goes: the case, its arm and what the run may use */
+export interface RunSpec {
+    core: Core;
+    case: PluginEvalCase;
+    arm: PluginEvalArm;
+    llm: CogitatorConfig['llm'];
+    model: string | undefined;
+    grants: readonly ToolGrant[];
+    /** The MCP servers each plugin of the case declares, by plugin name */
+    pluginServers: ReadonlyMap<string, ReadonlySet<string>>;
+    scaffold: boolean;
+    exec: PluginEvalOptions['exec'];
+    keepTemp: boolean;
+    signal: AbortSignal;
+}
+/** How a run ended and what it left for its graders */
+export interface RunOutcome {
+    /** What the graders read, absent when the run is not graded */
+    capture?: RunCapture;
+    error: string | null;
+    apiError?: boolean;
+    authFailed?: boolean;
+    interrupted?: boolean;
+    aborted?: PluginEvalAbort;
+    costUsd: number;
+    durationSeconds: number;
+    tempDir?: string;
+    /** Removes the run's temporary directory unless it is kept */
+    cleanup(): void;
+}
+/** The tool name of a mocked MCP tool: `mcp__plugin_<plugin>_<server>__<tool>` for a plugin's server */
+export declare function mockToolName(plugin: string | undefined, server: string, tool: string): string;
+/**
+ * Runs the case once in a fresh temporary workspace: the scaffold, the read-only fixture
+ * directories, an agent with the granted tools and, in the with-arm, the case's plugins. The run
+ * never stops to ask: what is not granted is not given
+ */
+export declare function executeRun(spec: RunSpec): Promise<RunOutcome>;
+export {};
+//# sourceMappingURL=execute.d.ts.map

package/dist/plugin-evals/execute.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"execute.d.ts","sourceRoot":"","sources":["../../src/plugin-evals/execute.ts"],"names":[],"mappings":"AAaA,OAAO,KAAK,EACV,eAAe,EAMhB,MAAM,oBAAoB,CAAC;AAE5B,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,WAAW,CAAC;AAG5C,OAAO,EAAyD,KAAK,SAAS,EAAE,MAAM,SAAS,CAAC;AAEhG,OAAO,KAAK,EACV,eAAe,EACf,aAAa,EACb,cAAc,EAGd,iBAAiB,EAElB,MAAM,SAAS,CAAC;AAEjB,KAAK,IAAI,GAAG,cAAc,oBAAoB,CAAC,CAAC;AAkBhD,mEAAmE;AACnE,MAAM,WAAW,OAAO;IACtB,IAAI,EAAE,IAAI,CAAC;IACX,IAAI,EAAE,cAAc,CAAC;IACrB,GAAG,EAAE,aAAa,CAAC;IACnB,GAAG,EAAE,eAAe,CAAC,KAAK,CAAC,CAAC;IAC5B,KAAK,EAAE,MAAM,GAAG,SAAS,CAAC;IAC1B,MAAM,EAAE,SAAS,SAAS,EAAE,CAAC;IAC7B,uEAAuE;IACvE,aAAa,EAAE,WAAW,CAAC,MAAM,EAAE,WAAW,CAAC,MAAM,CAAC,CAAC,CAAC;IACxD,QAAQ,EAAE,OAAO,CAAC;IAClB,IAAI,EAAE,iBAAiB,CAAC,MAAM,CAAC,CAAC;IAChC,QAAQ,EAAE,OAAO,CAAC;IAClB,MAAM,EAAE,WAAW,CAAC;CACrB;AAED,uDAAuD;AACvD,MAAM,WAAW,UAAU;IACzB,+DAA+D;IAC/D,OAAO,CAAC,EAAE,UAAU,CAAC;IACrB,KAAK,EAAE,MAAM,GAAG,IAAI,CAAC;IACrB,QAAQ,CAAC,EAAE,OAAO,CAAC;IACnB,UAAU,CAAC,EAAE,OAAO,CAAC;IACrB,WAAW,CAAC,EAAE,OAAO,CAAC;IACtB,OAAO,CAAC,EAAE,eAAe,CAAC;IAC1B,OAAO,EAAE,MAAM,CAAC;IAChB,eAAe,EAAE,MAAM,CAAC;IACxB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,8DAA8D;IAC9D,OAAO,IAAI,IAAI,CAAC;CACjB;AAaD,wGAAwG;AACxG,wBAAgB,YAAY,CAAC,MAAM,EAAE,MAAM,GAAG,SAAS,EAAE,MAAM,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,GAAG,MAAM,CAG7F;AAsTD;;;;GAIG;AACH,wBAAsB,UAAU,CAAC,IAAI,EAAE,OAAO,GAAG,OAAO,CAAC,UAAU,CAAC,CA4LnE"}