snapeval 1.0.1

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Matan Tsach
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,194 @@
+# snapeval
+
+Semantic snapshot testing for AI skills. Zero assertions. AI-driven. Free inference.
+
+[![CI](https://github.com/matantsach/snapeval/actions/workflows/ci.yml/badge.svg)](https://github.com/matantsach/snapeval/actions/workflows/ci.yml)
+[![npm version](https://img.shields.io/npm/v/snapeval.svg)](https://www.npmjs.com/package/snapeval)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+snapeval evaluates [agentskills.io](https://agentskills.io) skills through semantic snapshot testing. It generates test cases from your skill's `SKILL.md`, captures baseline outputs, and detects regressions through a tiered comparison pipeline — all with zero manual test authoring.
+
+## Why snapeval?
+
+- **Zero assertions** — AI generates test cases from your SKILL.md. You never write test logic.
+- **Semantic comparison** — Three-tier pipeline: schema check (free) → embedding similarity (cheap) → LLM judge with order-swap debiasing (expensive). Most checks cost $0.
+- **Free inference** — Uses gpt-5-mini via Copilot CLI (0x multiplier on paid plans) and GitHub Models API (free with GITHUB_TOKEN).
+- **Non-determinism handling** — Variance envelope from N baseline runs prevents false regressions.
+- **Platform-agnostic** — Adapter-based architecture. Copilot CLI first, Claude Code and others coming.
+
+## Quick Start
+
+### As a Copilot CLI Plugin
+
+Install directly from the GitHub repo:
+
+```bash
+gh copilot -- plugin install matantsach/snapeval
+```
+
+Or register the marketplace first, then install by name:
+
+```bash
+gh copilot -- plugin marketplace add matantsach/snapeval
+gh copilot -- plugin install snapeval@snapeval-marketplace
+```
+
+Then in Copilot CLI interactive mode, just ask naturally:
+
+```
+> evaluate my code-reviewer skill
+> check skills/code-reviewer for regressions
+> approve scenario 3
+```
+
+The agent will use the snapeval skill automatically based on your prompt.
+
+### As a CLI
+
+```bash
+npx snapeval init <skill-path>       # AI generates test cases from SKILL.md
+npx snapeval capture <skill-path>    # Run tests, save baseline snapshots
+npx snapeval check <skill-path>      # Compare current output to baselines
+npx snapeval approve [--scenario N]  # Accept new behavior as baseline
+npx snapeval report <skill-path>     # Generate benchmark.json
+```
+
+### Local Development
+
+For development without `npx`, clone and use `tsx` directly:
+
+```bash
+git clone https://github.com/matantsach/snapeval.git
+cd snapeval && npm install
+npx tsx bin/snapeval.ts init <skill-path>
+```
+
+Or load as a local plugin during development:
+
+```bash
+gh copilot -- --plugin-dir /path/to/snapeval
+```
+
+### In CI
+
+Commit your `evals.json` and `snapshots/` directory, then add a workflow:
+
+```yaml
+# .github/workflows/skill-eval.yml
+name: Skill Evaluation
+on: [pull_request]
+
+jobs:
+  eval:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+      - run: npm ci
+      - run: npx tsx bin/snapeval.ts check skills/my-skill --ci --skip-embedding
+```
+
+> **Note:** The `--skip-embedding` flag runs Tier 1 (schema) and Tier 3 (LLM judge) only, skipping Tier 2 which requires the GitHub Models embedding API. For Tier 1-only checks (fastest, free, no API needed), committed baselines with stable output structures will pass without any inference calls.
+
+## How It Works
+
+```
+SKILL.md → AI generates test scenarios → Capture baseline snapshots
+                                                    ↓
+         Modify skill → Re-run scenarios → Compare via tiered pipeline
+                                                    ↓
+                              Schema match? → PASS (free, instant)
+                              Embedding > 0.85? → PASS (cheap)
+                              LLM Judge agrees? → PASS/REGRESSED (expensive)
+```
+
+### Comparison Pipeline
+
+| Tier | Method | Cost | When Used |
+|------|--------|------|-----------|
+| 1 | Schema check | Free | Structural skeleton matches |
+| 2 | Embedding similarity | Cheap | Schema differs but meaning similar |
+| 3 | LLM judge (order-swap) | Expensive | Ambiguous cases only |
+
+Most stable skills are checked entirely at Tier 1 — $0.00 per run.
+
+## Eval Format
+
+snapeval follows the [agentskills.io evaluation standard](https://agentskills.io/skill-creation/evaluating-skills):
+
+```
+my-skill/
+├── SKILL.md
+└── evals/
+    ├── evals.json          ← AI-generated test cases
+    ├── snapshots/          ← Captured baseline outputs
+    └── results/
+        └── iteration-N/
+            ├── grading.json
+            ├── timing.json
+            └── benchmark.json
+```
+
+## Configuration
+
+Create `snapeval.config.json` in your skill or project root:
+
+```json
+{
+  "adapter": "copilot-cli",
+  "inference": "auto",
+  "threshold": 0.85,
+  "runs": 3,
+  "budget": "unlimited"
+}
+```
+
+CLI flags override config file values.
+
+## CLI Reference
+
+```
+snapeval init [skill-dir]         Generate test cases from SKILL.md using AI
+snapeval capture [skill-dir]      Run skill against all scenarios, save baselines
+snapeval check [skill-dir]        Compare current output against baselines
+snapeval approve [skill-dir]      Approve regressed scenarios as new baselines
+snapeval report [skill-dir]       Write results to evals/results/iteration-N/
+```
+
+**Common flags:**
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--adapter <name>` | Skill adapter | `copilot-cli` |
+| `--inference <name>` | Inference adapter | `auto` |
+| `--threshold <n>` | Embedding similarity threshold | `0.85` |
+| `--budget <amount>` | Spend cap in USD | `unlimited` |
+| `--runs <n>` | Baseline runs per scenario | `1` |
+| `--ci` | CI mode: exit 1 on regressions | off |
+| `--skip-embedding` | Skip Tier 2 (embedding) | off |
+| `--scenario <ids>` | Comma-separated scenario IDs | all |
+| `--verbose` | Verbose output | off |
+
+## Architecture
+
+Three surfaces over a shared core engine:
+
+- **Plugin** (SKILL.md) — Interactive product. AI handles everything.
+- **CLI** (`npx snapeval`) — Headless backend for CI and power users.
+- **GitHub Action** — CI wrapper (coming in v2).
+
+Three adapter layers for platform independence:
+
+- **SkillAdapter** — How to invoke a skill (Copilot CLI, Claude Code, generic)
+- **InferenceAdapter** — Where to get LLM capabilities (Copilot gpt-5-mini, GitHub Models API)
+- **ReportAdapter** — How to present results (terminal, JSON, PR comment)
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+
+## License
+
+[MIT](LICENSE)

package/bin/snapeval.ts

@@ -0,0 +1,226 @@
+#!/usr/bin/env tsx
+import { Command } from 'commander';
+import { resolveConfig } from '../src/config.js';
+import { resolveInference } from '../src/adapters/inference/resolve.js';
+import { CopilotCLIAdapter } from '../src/adapters/skill/copilot-cli.js';
+import { TerminalReporter } from '../src/adapters/report/terminal.js';
+import { initCommand } from '../src/commands/init.js';
+import { captureCommand } from '../src/commands/capture.js';
+import { checkCommand } from '../src/commands/check.js';
+import { approveCommand, approveFromResults } from '../src/commands/approve.js';
+import { reportCommand } from '../src/commands/report.js';
+import { SnapevalError } from '../src/errors.js';
+import * as path from 'node:path';
+
+const program = new Command();
+
+program
+  .name('snapeval')
+  .description('Semantic snapshot testing for AI skills')
+  .version('1.0.0');
+
+// --- init ---
+program
+  .command('init')
+  .description('Generate test cases from SKILL.md using AI')
+  .option('--adapter <adapter>', 'Skill adapter to use', 'copilot-cli')
+  .option('--inference <inference>', 'Inference adapter to use', 'auto')
+  .option('--verbose', 'Verbose output')
+  .argument('[skill-dir]', 'Path to skill directory', process.cwd())
+  .action(async (skillDir: string, opts: Record<string, string | boolean>) => {
+    try {
+      const skillPath = path.resolve(skillDir);
+      const config = resolveConfig(
+        { adapter: opts.adapter as string, inference: opts.inference as string },
+        process.cwd(),
+        skillPath
+      );
+      const inference = resolveInference(config.inference);
+      await initCommand(skillPath, inference);
+      console.log(`Generated evals at ${path.join(skillPath, 'evals', 'evals.json')}`);
+      process.exit(0);
+    } catch (err) {
+      handleError(err);
+    }
+  });
+
+// --- capture ---
+program
+  .command('capture')
+  .description('Run skill against all scenarios and save baseline snapshots')
+  .option('--adapter <adapter>', 'Skill adapter to use', 'copilot-cli')
+  .option('--inference <inference>', 'Inference adapter to use', 'auto')
+  .option('--runs <n>', 'Number of runs per scenario', '1')
+  .option('--verbose', 'Verbose output')
+  .argument('[skill-dir]', 'Path to skill directory', process.cwd())
+  .action(async (skillDir: string, opts: Record<string, string | boolean>) => {
+    try {
+      const skillPath = path.resolve(skillDir);
+      const config = resolveConfig(
+        {
+          adapter: opts.adapter as string,
+          inference: opts.inference as string,
+          runs: opts.runs ? parseInt(opts.runs as string, 10) : undefined,
+        },
+        process.cwd(),
+        skillPath
+      );
+      const skillAdapter = resolveSkillAdapter(config.adapter);
+      await captureCommand(skillPath, skillAdapter, { runs: config.runs });
+      console.log(`Captured baselines at ${path.join(skillPath, 'evals', 'snapshots')}`);
+      process.exit(0);
+    } catch (err) {
+      handleError(err);
+    }
+  });
+
+// --- check ---
+program
+  .command('check')
+  .description('Compare current skill output against baselines')
+  .option('--adapter <adapter>', 'Skill adapter to use', 'copilot-cli')
+  .option('--inference <inference>', 'Inference adapter to use', 'auto')
+  .option('--threshold <n>', 'Similarity threshold (0–1)', '0.85')
+  .option('--budget <amount>', 'Spend cap in USD (or "unlimited")', 'unlimited')
+  .option('--ci', 'CI mode: exit 1 on regressions, no interactive prompts')
+  .option('--skip-embedding', 'Skip embedding tier (tier 2)')
+  .option('--verbose', 'Verbose output')
+  .option('--scenario <ids>', 'Comma-separated scenario IDs to check')
+  .argument('[skill-dir]', 'Path to skill directory', process.cwd())
+  .action(async (skillDir: string, opts: Record<string, string | boolean>) => {
+    try {
+      const skillPath = path.resolve(skillDir);
+      const config = resolveConfig(
+        {
+          adapter: opts.adapter as string,
+          inference: opts.inference as string,
+          threshold: opts.threshold ? parseFloat(opts.threshold as string) : undefined,
+          budget: opts.budget as string,
+        },
+        process.cwd(),
+        skillPath
+      );
+      const skillAdapter = resolveSkillAdapter(config.adapter);
+      const inference = resolveInference(config.inference);
+
+      const results = await checkCommand(skillPath, skillAdapter, inference, {
+        threshold: config.threshold,
+        budget: config.budget,
+        skipEmbedding: Boolean(opts.skipEmbedding),
+      });
+
+      // Always print terminal report
+      const reporter = new TerminalReporter();
+      await reporter.report(results);
+
+      const hasRegressions = results.summary.regressed > 0;
+      if (hasRegressions) {
+        process.exit(1);
+      }
+      process.exit(0);
+    } catch (err) {
+      handleError(err);
+    }
+  });
+
+// --- approve ---
+program
+  .command('approve')
+  .description('Approve regressed scenarios as new baselines')
+  .option('--adapter <adapter>', 'Skill adapter to use', 'copilot-cli')
+  .option('--inference <inference>', 'Inference adapter to use', 'auto')
+  .option('--scenario <ids>', 'Comma-separated scenario IDs to approve (default: all)')
+  .option('--verbose', 'Verbose output')
+  .argument('[skill-dir]', 'Path to skill directory', process.cwd())
+  .action(async (skillDir: string, opts: Record<string, string | boolean>) => {
+    try {
+      const skillPath = path.resolve(skillDir);
+      const config = resolveConfig(
+        { adapter: opts.adapter as string, inference: opts.inference as string },
+        process.cwd(),
+        skillPath
+      );
+      const skillAdapter = resolveSkillAdapter(config.adapter);
+
+      const scenarioIds = opts.scenario
+        ? (opts.scenario as string).split(',').map((s) => parseInt(s.trim(), 10))
+        : undefined;
+
+      await approveCommand(skillPath, skillAdapter, { scenarioIds });
+      console.log('Approved snapshots updated.');
+      process.exit(0);
+    } catch (err) {
+      handleError(err);
+    }
+  });
+
+// --- report ---
+program
+  .command('report')
+  .description('Write latest check results to evals/results/iteration-N/')
+  .option('--adapter <adapter>', 'Skill adapter to use', 'copilot-cli')
+  .option('--inference <inference>', 'Inference adapter to use', 'auto')
+  .option('--threshold <n>', 'Similarity threshold (0–1)', '0.85')
+  .option('--budget <amount>', 'Spend cap in USD (or "unlimited")', 'unlimited')
+  .option('--skip-embedding', 'Skip embedding tier (tier 2)')
+  .option('--verbose', 'Verbose output')
+  .argument('[skill-dir]', 'Path to skill directory', process.cwd())
+  .action(async (skillDir: string, opts: Record<string, string | boolean>) => {
+    try {
+      const skillPath = path.resolve(skillDir);
+      const config = resolveConfig(
+        {
+          adapter: opts.adapter as string,
+          inference: opts.inference as string,
+          threshold: opts.threshold ? parseFloat(opts.threshold as string) : undefined,
+          budget: opts.budget as string,
+        },
+        process.cwd(),
+        skillPath
+      );
+      const skillAdapter = resolveSkillAdapter(config.adapter);
+      const inference = resolveInference(config.inference);
+
+      const results = await checkCommand(skillPath, skillAdapter, inference, {
+        threshold: config.threshold,
+        budget: config.budget,
+        skipEmbedding: Boolean(opts.skipEmbedding),
+      });
+
+      await reportCommand(skillPath, results, { verbose: Boolean(opts.verbose) });
+
+      const hasRegressions = results.summary.regressed > 0;
+      if (hasRegressions) {
+        process.exit(1);
+      }
+      process.exit(0);
+    } catch (err) {
+      handleError(err);
+    }
+  });
+
+// --- helpers ---
+
+function resolveSkillAdapter(adapterName: string) {
+  if (adapterName === 'copilot-cli') {
+    return new CopilotCLIAdapter();
+  }
+  throw new SnapevalError(
+    `Unknown skill adapter "${adapterName}". Valid options: copilot-cli.`
+  );
+}
+
+function handleError(err: unknown): never {
+  if (err instanceof SnapevalError) {
+    console.error(`Error: ${err.message}`);
+    process.exit(err.exitCode ?? 2);
+  }
+  if (err instanceof Error) {
+    console.error(`Error: ${err.message}`);
+    process.exit(2);
+  }
+  console.error('An unknown error occurred.');
+  process.exit(2);
+}
+
+program.parse(process.argv);

package/dist/bin/snapeval.d.ts

@@ -0,0 +1,2 @@
+#!/usr/bin/env tsx
+export {};

package/dist/bin/snapeval.js

@@ -0,0 +1,191 @@
+#!/usr/bin/env tsx
+import { Command } from 'commander';
+import { resolveConfig } from '../src/config.js';
+import { resolveInference } from '../src/adapters/inference/resolve.js';
+import { CopilotCLIAdapter } from '../src/adapters/skill/copilot-cli.js';
+import { TerminalReporter } from '../src/adapters/report/terminal.js';
+import { initCommand } from '../src/commands/init.js';
+import { captureCommand } from '../src/commands/capture.js';
+import { checkCommand } from '../src/commands/check.js';
+import { approveCommand } from '../src/commands/approve.js';
+import { reportCommand } from '../src/commands/report.js';
+import { SnapevalError } from '../src/errors.js';
+import * as path from 'node:path';
+const program = new Command();
+program
+    .name('snapeval')
+    .description('Semantic snapshot testing for AI skills')
+    .version('1.0.0');
+// --- init ---
+program
+    .command('init')
+    .description('Generate test cases from SKILL.md using AI')
+    .option('--adapter <adapter>', 'Skill adapter to use', 'copilot-cli')
+    .option('--inference <inference>', 'Inference adapter to use', 'auto')
+    .option('--verbose', 'Verbose output')
+    .argument('[skill-dir]', 'Path to skill directory', process.cwd())
+    .action(async (skillDir, opts) => {
+    try {
+        const skillPath = path.resolve(skillDir);
+        const config = resolveConfig({ adapter: opts.adapter, inference: opts.inference }, process.cwd(), skillPath);
+        const inference = resolveInference(config.inference);
+        await initCommand(skillPath, inference);
+        console.log(`Generated evals at ${path.join(skillPath, 'evals', 'evals.json')}`);
+        process.exit(0);
+    }
+    catch (err) {
+        handleError(err);
+    }
+});
+// --- capture ---
+program
+    .command('capture')
+    .description('Run skill against all scenarios and save baseline snapshots')
+    .option('--adapter <adapter>', 'Skill adapter to use', 'copilot-cli')
+    .option('--inference <inference>', 'Inference adapter to use', 'auto')
+    .option('--runs <n>', 'Number of runs per scenario', '1')
+    .option('--verbose', 'Verbose output')
+    .argument('[skill-dir]', 'Path to skill directory', process.cwd())
+    .action(async (skillDir, opts) => {
+    try {
+        const skillPath = path.resolve(skillDir);
+        const config = resolveConfig({
+            adapter: opts.adapter,
+            inference: opts.inference,
+            runs: opts.runs ? parseInt(opts.runs, 10) : undefined,
+        }, process.cwd(), skillPath);
+        const skillAdapter = resolveSkillAdapter(config.adapter);
+        await captureCommand(skillPath, skillAdapter, { runs: config.runs });
+        console.log(`Captured baselines at ${path.join(skillPath, 'evals', 'snapshots')}`);
+        process.exit(0);
+    }
+    catch (err) {
+        handleError(err);
+    }
+});
+// --- check ---
+program
+    .command('check')
+    .description('Compare current skill output against baselines')
+    .option('--adapter <adapter>', 'Skill adapter to use', 'copilot-cli')
+    .option('--inference <inference>', 'Inference adapter to use', 'auto')
+    .option('--threshold <n>', 'Similarity threshold (0–1)', '0.85')
+    .option('--budget <amount>', 'Spend cap in USD (or "unlimited")', 'unlimited')
+    .option('--ci', 'CI mode: exit 1 on regressions, no interactive prompts')
+    .option('--skip-embedding', 'Skip embedding tier (tier 2)')
+    .option('--verbose', 'Verbose output')
+    .option('--scenario <ids>', 'Comma-separated scenario IDs to check')
+    .argument('[skill-dir]', 'Path to skill directory', process.cwd())
+    .action(async (skillDir, opts) => {
+    try {
+        const skillPath = path.resolve(skillDir);
+        const config = resolveConfig({
+            adapter: opts.adapter,
+            inference: opts.inference,
+            threshold: opts.threshold ? parseFloat(opts.threshold) : undefined,
+            budget: opts.budget,
+        }, process.cwd(), skillPath);
+        const skillAdapter = resolveSkillAdapter(config.adapter);
+        const inference = resolveInference(config.inference);
+        const results = await checkCommand(skillPath, skillAdapter, inference, {
+            threshold: config.threshold,
+            budget: config.budget,
+            skipEmbedding: Boolean(opts.skipEmbedding),
+        });
+        // Always print terminal report
+        const reporter = new TerminalReporter();
+        await reporter.report(results);
+        const hasRegressions = results.summary.regressed > 0;
+        if (hasRegressions) {
+            process.exit(1);
+        }
+        process.exit(0);
+    }
+    catch (err) {
+        handleError(err);
+    }
+});
+// --- approve ---
+program
+    .command('approve')
+    .description('Approve regressed scenarios as new baselines')
+    .option('--adapter <adapter>', 'Skill adapter to use', 'copilot-cli')
+    .option('--inference <inference>', 'Inference adapter to use', 'auto')
+    .option('--scenario <ids>', 'Comma-separated scenario IDs to approve (default: all)')
+    .option('--verbose', 'Verbose output')
+    .argument('[skill-dir]', 'Path to skill directory', process.cwd())
+    .action(async (skillDir, opts) => {
+    try {
+        const skillPath = path.resolve(skillDir);
+        const config = resolveConfig({ adapter: opts.adapter, inference: opts.inference }, process.cwd(), skillPath);
+        const skillAdapter = resolveSkillAdapter(config.adapter);
+        const scenarioIds = opts.scenario
+            ? opts.scenario.split(',').map((s) => parseInt(s.trim(), 10))
+            : undefined;
+        await approveCommand(skillPath, skillAdapter, { scenarioIds });
+        console.log('Approved snapshots updated.');
+        process.exit(0);
+    }
+    catch (err) {
+        handleError(err);
+    }
+});
+// --- report ---
+program
+    .command('report')
+    .description('Write latest check results to evals/results/iteration-N/')
+    .option('--adapter <adapter>', 'Skill adapter to use', 'copilot-cli')
+    .option('--inference <inference>', 'Inference adapter to use', 'auto')
+    .option('--threshold <n>', 'Similarity threshold (0–1)', '0.85')
+    .option('--budget <amount>', 'Spend cap in USD (or "unlimited")', 'unlimited')
+    .option('--skip-embedding', 'Skip embedding tier (tier 2)')
+    .option('--verbose', 'Verbose output')
+    .argument('[skill-dir]', 'Path to skill directory', process.cwd())
+    .action(async (skillDir, opts) => {
+    try {
+        const skillPath = path.resolve(skillDir);
+        const config = resolveConfig({
+            adapter: opts.adapter,
+            inference: opts.inference,
+            threshold: opts.threshold ? parseFloat(opts.threshold) : undefined,
+            budget: opts.budget,
+        }, process.cwd(), skillPath);
+        const skillAdapter = resolveSkillAdapter(config.adapter);
+        const inference = resolveInference(config.inference);
+        const results = await checkCommand(skillPath, skillAdapter, inference, {
+            threshold: config.threshold,
+            budget: config.budget,
+            skipEmbedding: Boolean(opts.skipEmbedding),
+        });
+        await reportCommand(skillPath, results, { verbose: Boolean(opts.verbose) });
+        const hasRegressions = results.summary.regressed > 0;
+        if (hasRegressions) {
+            process.exit(1);
+        }
+        process.exit(0);
+    }
+    catch (err) {
+        handleError(err);
+    }
+});
+// --- helpers ---
+function resolveSkillAdapter(adapterName) {
+    if (adapterName === 'copilot-cli') {
+        return new CopilotCLIAdapter();
+    }
+    throw new SnapevalError(`Unknown skill adapter "${adapterName}". Valid options: copilot-cli.`);
+}
+function handleError(err) {
+    if (err instanceof SnapevalError) {
+        console.error(`Error: ${err.message}`);
+        process.exit(err.exitCode ?? 2);
+    }
+    if (err instanceof Error) {
+        console.error(`Error: ${err.message}`);
+        process.exit(2);
+    }
+    console.error('An unknown error occurred.');
+    process.exit(2);
+}
+program.parse(process.argv);
+//# sourceMappingURL=snapeval.js.map

package/dist/bin/snapeval.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"snapeval.js","sourceRoot":"","sources":["../../bin/snapeval.ts"],"names":[],"mappings":";AACA,OAAO,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AACpC,OAAO,EAAE,aAAa,EAAE,MAAM,kBAAkB,CAAC;AACjD,OAAO,EAAE,gBAAgB,EAAE,MAAM,sCAAsC,CAAC;AACxE,OAAO,EAAE,iBAAiB,EAAE,MAAM,sCAAsC,CAAC;AACzE,OAAO,EAAE,gBAAgB,EAAE,MAAM,oCAAoC,CAAC;AACtE,OAAO,EAAE,WAAW,EAAE,MAAM,yBAAyB,CAAC;AACtD,OAAO,EAAE,cAAc,EAAE,MAAM,4BAA4B,CAAC;AAC5D,OAAO,EAAE,YAAY,EAAE,MAAM,0BAA0B,CAAC;AACxD,OAAO,EAAE,cAAc,EAAsB,MAAM,4BAA4B,CAAC;AAChF,OAAO,EAAE,aAAa,EAAE,MAAM,2BAA2B,CAAC;AAC1D,OAAO,EAAE,aAAa,EAAE,MAAM,kBAAkB,CAAC;AACjD,OAAO,KAAK,IAAI,MAAM,WAAW,CAAC;AAElC,MAAM,OAAO,GAAG,IAAI,OAAO,EAAE,CAAC;AAE9B,OAAO;KACJ,IAAI,CAAC,UAAU,CAAC;KAChB,WAAW,CAAC,yCAAyC,CAAC;KACtD,OAAO,CAAC,OAAO,CAAC,CAAC;AAEpB,eAAe;AACf,OAAO;KACJ,OAAO,CAAC,MAAM,CAAC;KACf,WAAW,CAAC,4CAA4C,CAAC;KACzD,MAAM,CAAC,qBAAqB,EAAE,sBAAsB,EAAE,aAAa,CAAC;KACpE,MAAM,CAAC,yBAAyB,EAAE,0BAA0B,EAAE,MAAM,CAAC;KACrE,MAAM,CAAC,WAAW,EAAE,gBAAgB,CAAC;KACrC,QAAQ,CAAC,aAAa,EAAE,yBAAyB,EAAE,OAAO,CAAC,GAAG,EAAE,CAAC;KACjE,MAAM,CAAC,KAAK,EAAE,QAAgB,EAAE,IAAsC,EAAE,EAAE;IACzE,IAAI,CAAC;QACH,MAAM,SAAS,GAAG,IAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;QACzC,MAAM,MAAM,GAAG,aAAa,CAC1B,EAAE,OAAO,EAAE,IAAI,CAAC,OAAiB,EAAE,SAAS,EAAE,IAAI,CAAC,SAAmB,EAAE,EACxE,OAAO,CAAC,GAAG,EAAE,EACb,SAAS,CACV,CAAC;QACF,MAAM,SAAS,GAAG,gBAAgB,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACrD,MAAM,WAAW,CAAC,SAAS,EAAE,SAAS,CAAC,CAAC;QACxC,OAAO,CAAC,GAAG,CAAC,sBAAsB,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,OAAO,EAAE,YAAY,CAAC,EAAE,CAAC,CAAC;QACjF,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,WAAW,CAAC,GAAG,CAAC,CAAC;IACnB,CAAC;AACH,CAAC,CAAC,CAAC;AAEL,kBAAkB;AAClB,OAAO;KACJ,OAAO,CAAC,SAAS,CAAC;KAClB,WAAW,CAAC,6DAA6D,CAAC;KAC1E,MAAM,CAAC,qBAAqB,EAAE,sBAAsB,EAAE,aAAa,CAAC;KACpE,MAAM,CAAC,yBAAyB,EAAE,0BAA0B,EAAE,MAAM,CAAC;KACrE,MAAM,CAAC,YAAY,EAAE,6BAA6B,EAAE,GAAG,CAAC;KACxD,MAAM,CAAC,WAAW,EAAE,gBAAgB,CAAC;KACrC,QAAQ,CAAC,aAAa,EAAE,yBAAyB,EAAE,OAAO,CAAC,GAAG,EAAE,CAAC;KACjE,MAAM,CAAC,KAAK,EAAE,QAAgB,EAAE,IAAsC,EAAE,EAAE;IACzE,IAAI,CAAC;QACH,MAAM,SAAS,GAAG,IAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;QACzC,MAAM,MAAM,GAAG,aAAa,CAC1B;YACE,OAAO,EAAE,IAAI,CAAC,OAAiB;YAC/B,SAAS,EAAE,IAAI,CAAC,SAAmB;YACnC,IAAI,EAAE,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,QAAQ,CAAC,IAAI,CAAC,IAAc,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,SAAS;SAChE,EACD,OAAO,CAAC,GAAG,EAAE,EACb,SAAS,CACV,CAAC;QACF,MAAM,YAAY,GAAG,mBAAmB,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;QACzD,MAAM,cAAc,CAAC,SAAS,EAAE,YAAY,EAAE,EAAE,IAAI,EAAE,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC;QACrE,OAAO,CAAC,GAAG,CAAC,yBAAyB,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,OAAO,EAAE,WAAW,CAAC,EAAE,CAAC,CAAC;QACnF,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,WAAW,CAAC,GAAG,CAAC,CAAC;IACnB,CAAC;AACH,CAAC,CAAC,CAAC;AAEL,gBAAgB;AAChB,OAAO;KACJ,OAAO,CAAC,OAAO,CAAC;KAChB,WAAW,CAAC,gDAAgD,CAAC;KAC7D,MAAM,CAAC,qBAAqB,EAAE,sBAAsB,EAAE,aAAa,CAAC;KACpE,MAAM,CAAC,yBAAyB,EAAE,0BAA0B,EAAE,MAAM,CAAC;KACrE,MAAM,CAAC,iBAAiB,EAAE,4BAA4B,EAAE,MAAM,CAAC;KAC/D,MAAM,CAAC,mBAAmB,EAAE,mCAAmC,EAAE,WAAW,CAAC;KAC7E,MAAM,CAAC,MAAM,EAAE,wDAAwD,CAAC;KACxE,MAAM,CAAC,kBAAkB,EAAE,8BAA8B,CAAC;KAC1D,MAAM,CAAC,WAAW,EAAE,gBAAgB,CAAC;KACrC,MAAM,CAAC,kBAAkB,EAAE,uCAAuC,CAAC;KACnE,QAAQ,CAAC,aAAa,EAAE,yBAAyB,EAAE,OAAO,CAAC,GAAG,EAAE,CAAC;KACjE,MAAM,CAAC,KAAK,EAAE,QAAgB,EAAE,IAAsC,EAAE,EAAE;IACzE,IAAI,CAAC;QACH,MAAM,SAAS,GAAG,IAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;QACzC,MAAM,MAAM,GAAG,aAAa,CAC1B;YACE,OAAO,EAAE,IAAI,CAAC,OAAiB;YAC/B,SAAS,EAAE,IAAI,CAAC,SAAmB;YACnC,SAAS,EAAE,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,UAAU,CAAC,IAAI,CAAC,SAAmB,CAAC,CAAC,CAAC,CAAC,SAAS;YAC5E,MAAM,EAAE,IAAI,CAAC,MAAgB;SAC9B,EACD,OAAO,CAAC,GAAG,EAAE,EACb,SAAS,CACV,CAAC;QACF,MAAM,YAAY,GAAG,mBAAmB,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;QACzD,MAAM,SAAS,GAAG,gBAAgB,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAErD,MAAM,OAAO,GAAG,MAAM,YAAY,CAAC,SAAS,EAAE,YAAY,EAAE,SAAS,EAAE;YACrE,SAAS,EAAE,MAAM,CAAC,SAAS;YAC3B,MAAM,EAAE,MAAM,CAAC,MAAM;YACrB,aAAa,EAAE,OAAO,CAAC,IAAI,CAAC,aAAa,CAAC;SAC3C,CAAC,CAAC;QAEH,+BAA+B;QAC/B,MAAM,QAAQ,GAAG,IAAI,gBAAgB,EAAE,CAAC;QACxC,MAAM,QAAQ,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;QAE/B,MAAM,cAAc,GAAG,OAAO,CAAC,OAAO,CAAC,SAAS,GAAG,CAAC,CAAC;QACrD,IAAI,cAAc,EAAE,CAAC;YACnB,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;QAClB,CAAC;QACD,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,WAAW,CAAC,GAAG,CAAC,CAAC;IACnB,CAAC;AACH,CAAC,CAAC,CAAC;AAEL,kBAAkB;AAClB,OAAO;KACJ,OAAO,CAAC,SAAS,CAAC;KAClB,WAAW,CAAC,8CAA8C,CAAC;KAC3D,MAAM,CAAC,qBAAqB,EAAE,sBAAsB,EAAE,aAAa,CAAC;KACpE,MAAM,CAAC,yBAAyB,EAAE,0BAA0B,EAAE,MAAM,CAAC;KACrE,MAAM,CAAC,kBAAkB,EAAE,wDAAwD,CAAC;KACpF,MAAM,CAAC,WAAW,EAAE,gBAAgB,CAAC;KACrC,QAAQ,CAAC,aAAa,EAAE,yBAAyB,EAAE,OAAO,CAAC,GAAG,EAAE,CAAC;KACjE,MAAM,CAAC,KAAK,EAAE,QAAgB,EAAE,IAAsC,EAAE,EAAE;IACzE,IAAI,CAAC;QACH,MAAM,SAAS,GAAG,IAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;QACzC,MAAM,MAAM,GAAG,aAAa,CAC1B,EAAE,OAAO,EAAE,IAAI,CAAC,OAAiB,EAAE,SAAS,EAAE,IAAI,CAAC,SAAmB,EAAE,EACxE,OAAO,CAAC,GAAG,EAAE,EACb,SAAS,CACV,CAAC;QACF,MAAM,YAAY,GAAG,mBAAmB,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;QAEzD,MAAM,WAAW,GAAG,IAAI,CAAC,QAAQ;YAC/B,CAAC,CAAE,IAAI,CAAC,QAAmB,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,EAAE,EAAE,EAAE,CAAC,CAAC;YACzE,CAAC,CAAC,SAAS,CAAC;QAEd,MAAM,cAAc,CAAC,SAAS,EAAE,YAAY,EAAE,EAAE,WAAW,EAAE,CAAC,CAAC;QAC/D,OAAO,CAAC,GAAG,CAAC,6BAA6B,CAAC,CAAC;QAC3C,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,WAAW,CAAC,GAAG,CAAC,CAAC;IACnB,CAAC;AACH,CAAC,CAAC,CAAC;AAEL,iBAAiB;AACjB,OAAO;KACJ,OAAO,CAAC,QAAQ,CAAC;KACjB,WAAW,CAAC,0DAA0D,CAAC;KACvE,MAAM,CAAC,qBAAqB,EAAE,sBAAsB,EAAE,aAAa,CAAC;KACpE,MAAM,CAAC,yBAAyB,EAAE,0BAA0B,EAAE,MAAM,CAAC;KACrE,MAAM,CAAC,iBAAiB,EAAE,4BAA4B,EAAE,MAAM,CAAC;KAC/D,MAAM,CAAC,mBAAmB,EAAE,mCAAmC,EAAE,WAAW,CAAC;KAC7E,MAAM,CAAC,kBAAkB,EAAE,8BAA8B,CAAC;KAC1D,MAAM,CAAC,WAAW,EAAE,gBAAgB,CAAC;KACrC,QAAQ,CAAC,aAAa,EAAE,yBAAyB,EAAE,OAAO,CAAC,GAAG,EAAE,CAAC;KACjE,MAAM,CAAC,KAAK,EAAE,QAAgB,EAAE,IAAsC,EAAE,EAAE;IACzE,IAAI,CAAC;QACH,MAAM,SAAS,GAAG,IAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;QACzC,MAAM,MAAM,GAAG,aAAa,CAC1B;YACE,OAAO,EAAE,IAAI,CAAC,OAAiB;YAC/B,SAAS,EAAE,IAAI,CAAC,SAAmB;YACnC,SAAS,EAAE,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,UAAU,CAAC,IAAI,CAAC,SAAmB,CAAC,CAAC,CAAC,CAAC,SAAS;YAC5E,MAAM,EAAE,IAAI,CAAC,MAAgB;SAC9B,EACD,OAAO,CAAC,GAAG,EAAE,EACb,SAAS,CACV,CAAC;QACF,MAAM,YAAY,GAAG,mBAAmB,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;QACzD,MAAM,SAAS,GAAG,gBAAgB,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAErD,MAAM,OAAO,GAAG,MAAM,YAAY,CAAC,SAAS,EAAE,YAAY,EAAE,SAAS,EAAE;YACrE,SAAS,EAAE,MAAM,CAAC,SAAS;YAC3B,MAAM,EAAE,MAAM,CAAC,MAAM;YACrB,aAAa,EAAE,OAAO,CAAC,IAAI,CAAC,aAAa,CAAC;SAC3C,CAAC,CAAC;QAEH,MAAM,aAAa,CAAC,SAAS,EAAE,OAAO,EAAE,EAAE,OAAO,EAAE,OAAO,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC,CAAC;QAE5E,MAAM,cAAc,GAAG,OAAO,CAAC,OAAO,CAAC,SAAS,GAAG,CAAC,CAAC;QACrD,IAAI,cAAc,EAAE,CAAC;YACnB,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;QAClB,CAAC;QACD,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,WAAW,CAAC,GAAG,CAAC,CAAC;IACnB,CAAC;AACH,CAAC,CAAC,CAAC;AAEL,kBAAkB;AAElB,SAAS,mBAAmB,CAAC,WAAmB;IAC9C,IAAI,WAAW,KAAK,aAAa,EAAE,CAAC;QAClC,OAAO,IAAI,iBAAiB,EAAE,CAAC;IACjC,CAAC;IACD,MAAM,IAAI,aAAa,CACrB,0BAA0B,WAAW,gCAAgC,CACtE,CAAC;AACJ,CAAC;AAED,SAAS,WAAW,CAAC,GAAY;IAC/B,IAAI,GAAG,YAAY,aAAa,EAAE,CAAC;QACjC,OAAO,CAAC,KAAK,CAAC,UAAU,GAAG,CAAC,OAAO,EAAE,CAAC,CAAC;QACvC,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC,QAAQ,IAAI,CAAC,CAAC,CAAC;IAClC,CAAC;IACD,IAAI,GAAG,YAAY,KAAK,EAAE,CAAC;QACzB,OAAO,CAAC,KAAK,CAAC,UAAU,GAAG,CAAC,OAAO,EAAE,CAAC,CAAC;QACvC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IACD,OAAO,CAAC,KAAK,CAAC,4BAA4B,CAAC,CAAC;IAC5C,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;AAClB,CAAC;AAED,OAAO,CAAC,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC"}

package/dist/src/adapters/inference/copilot.d.ts

@@ -0,0 +1,9 @@
+import type { InferenceAdapter, Message, ChatOptions } from '../../types.js';
+export declare class CopilotInference implements InferenceAdapter {
+    private readonly fallback?;
+    readonly name = "copilot";
+    constructor(fallback?: InferenceAdapter | undefined);
+    chat(messages: Message[], _options?: ChatOptions): Promise<string>;
+    embed(text: string): Promise<number[]>;
+    estimateCost(_tokens: number): number;
+}