@pauly4010/evalai-sdk 1.8.0 → 1.9.1

package/CHANGELOG.md

@@ -5,6 +5,60 @@ All notable changes to the @pauly4010/evalai-sdk package will be documented in t
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.9.0] - 2026-02-27
+
+### ✨ Added
+
+#### CLI — One-Command CI Loop (`evalai ci`)
+
+- **`evalai ci`** — Single command teams put in GitHub workflows and never think about again
+- **Complete CI pipeline**: discover → manifest → impact → run → diff → PR summary → safe failure → "next step"
+- **Automatic manifest building**: Builds manifest if missing, no manual steps required
+- **Impact analysis integration**: `--impacted-only` flag for targeted testing
+- **Smart exit codes**: 0=clean, 1=regressions, 2=config/infra issues
+- **Self-documenting failures**: Always prints copy/paste next step for debugging
+- **GitHub Step Summary integration**: Automatic PR summaries with regressions and artifacts
+
+#### CLI — Durable Run History & Diff System
+
+- **Run artifact retention**: Timestamped artifacts in `.evalai/runs/run-<runId>.json`
+- **Run index file**: `.evalai/runs/index.json` tracks all runs with metadata
+- **Schema versioning**: `RunResult` and `DiffResult` include `schemaVersion` for compatibility
+- **Base/head shortcuts**: `--base baseline`, `--base last`, `--head last` for common cases
+- **Floating point normalization**: Consistent score/delta calculations across runs
+- **Comprehensive diff comparison**: Classifies regressions, improvements, added, removed specs
+
+#### CLI — Centralized Architecture
+
+- **Environment detection**: `isCI()`, `isGitHubActions()`, `getGitHubStepSummaryPath()` unified
+- **Workspace resolution**: `resolveEvalWorkspace()` provides all `.evalai` paths
+- **Git reference detection**: Comprehensive patterns for branches, tags, and ranges
+- **No more duplication**: All commands use shared utilities for consistency
+
+#### CLI — CI Friendliness
+
+- **Fail-safe base resolution**: Clear error messages when base artifacts missing in CI
+- **GitHub Step Summary**: Rich markdown summaries with metrics, regressions, and artifact links
+- **CI-specific error handling**: Exit code 2 for config issues with helpful guidance
+- **Artifact download instructions**: Exact commands for manual base artifact setup
+
+### 🔧 Changed
+
+- **Exit codes standardized**: 0=clean, 1=regressions, 2=config/infra issues across all commands
+- **Schema compatibility**: Added `schemaVersion` validation for future-proofing
+- **Path resolution**: All commands use centralized workspace helpers
+- **Error messages**: More actionable and context-aware guidance
+
+### 📊 New Features Summary
+
+- **One-command CI**: `evalai ci` replaces multi-step workflows
+- **Durable history**: Run artifacts preserved with smart indexing
+- **Smart diffing**: Automated regression detection with GitHub integration
+- **Centralized utilities**: Environment detection and workspace resolution unified
+- **Self-documenting**: Clear next steps for any failure scenario
+
+---
+
 ## [1.8.0] - 2026-02-26
 
 ### ✨ Added

package/README.md

@@ -7,41 +7,150 @@
 [![Contract Version](https://img.shields.io/badge/report%20schema-v1-blue.svg)](#)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-**Stop LLM regressions in CI in minutes.**
+**One-command CI for AI evaluation. Complete pipeline: discover → manifest → impact → run → diff → PR summary.**
 
-Zero to gate in under 5 minutes. No infra. No lock-in. Remove anytime.
+Zero to production CI in 60 seconds. No infra. No lock-in. Remove anytime.
 
 ---
 
-## Quick Start (2 minutes)
+## Quick Start (60 seconds)
+
+Add this to your `.github/workflows/evalai.yml`:
+
+```yaml
+name: EvalAI CI
+on: [push, pull_request]
+jobs:
+  evalai:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+      - run: npm ci
+      - run: npx @pauly4010/evalai-sdk ci --format github --write-results --base main
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: evalai-results
+          path: .evalai/
+```
+
+Create `eval/your-spec.spec.ts`:
+
+```typescript
+import { defineEval } from "@pauly4010/evalai-sdk";
+
+defineEval({
+  name: "Basic Math Operations",
+  description: "Test fundamental arithmetic",
+  prompt: "Test: 1+1=2, string concatenation, array includes",
+  expected: "All tests should pass",
+  tags: ["basic", "math"],
+  category: "unit-test"
+});
+```
 
 ```bash
-npx @pauly4010/evalai-sdk init
-git add evals/ .github/workflows/evalai-gate.yml evalai.config.json
-git commit -m "chore: add EvalAI regression gate"
+git add .github/workflows/evalai.yml eval/
+git commit -m "feat: add EvalAI CI pipeline"
 git push
 ```
 
-That's it. Open a PR and CI blocks regressions automatically.
-
-`evalai init` detects your project, creates a baseline from your current tests, and installs a GitHub Actions workflow. No manual config needed.
+That's it! Your CI now:
+- ✅ Discovers evaluation specs automatically
+- ✅ Runs only impacted specs (smart caching)
+- ✅ Compares results against base branch
+- ✅ Posts rich summary in PR with regressions
+- ✅ Exits with proper codes (0=clean, 1=regressions, 2=config)
 
 ---
 
-## What `evalai init` does
+## 🚀 New in v1.9.0: One-Command CI
+
+### `evalai ci` - Complete CI Pipeline
+
+```bash
+npx @pauly4010/evalai-sdk ci --format github --write-results --base main
+```
 
-1. **Detects** your Node repo and package manager (npm/yarn/pnpm)
-2. **Runs your tests** to capture a real baseline (pass/fail + test count)
-3. **Creates `evals/baseline.json`** with provenance metadata
-4. **Installs `.github/workflows/evalai-gate.yml`** (package-manager aware)
-5. **Creates `evalai.config.json`**
-6. **Prints next steps** — just commit and push
+**What it does:**
+1. **Discover** - Finds all evaluation specs automatically
+2. **Manifest** - Builds stable manifest if missing
+3. **Impact Analysis** - Runs only specs impacted by changes (optional)
+4. **Run** - Executes evaluations with artifact retention
+5. **Diff** - Compares results against base branch
+6. **PR Summary** - Posts rich markdown summary to GitHub
+7. **Debug Flow** - Prints copy/paste next step on failure
+
+**Advanced Options:**
+```bash
+npx @pauly4010/evalai-sdk ci --base main --impacted-only    # Run only impacted specs
+npx @pauly4010/evalai-sdk ci --format json --write-results   # JSON output for automation
+npx @pauly4010/evalai-sdk ci --base develop                  # Custom base branch
+```
+
+### Smart Diffing & GitHub Integration
+
+```bash
+npx @pauly4010/evalai-sdk diff --base main --head last --format github
+```
+
+**Features:**
+- 📊 Pass rate delta and score changes
+- 🚨 Regression detection with classifications
+- 📈 Improvements and new specs
+- 📁 Artifact links and technical details
+- 🎯 Exit codes: 0=clean, 1=regressions, 2=config
+
+### Self-Documenting Failures
+
+Every failure prints a clear next step:
+
+```
+🔧 Next step for debugging:
+   Download base artifact and run: evalai diff --base .evalai/base-run.json --head .evalai/last-run.json
+   Artifacts: .evalai/runs/
+```
 
 ---
 
 ## CLI Commands
 
-### Regression Gate (local, no account needed)
+### 🚀 One-Command CI (v1.9.0)
+
+| Command | Description |
+|---------|-------------|
+| `npx evalai ci` | Complete CI pipeline: discover → manifest → impact → run → diff → PR summary |
+| `npx evalai ci --base main` | Run CI with diff against main branch |
+| `npx evalai ci --impacted-only` | Run only specs impacted by changes |
+| `npx evalai ci --format github` | GitHub Step Summary with rich markdown |
+| `npx evalai ci --format json` | JSON output for automation |
+
+### Discovery & Manifest
+
+| Command | Description |
+|---------|-------------|
+| `npx evalai discover` | Find and analyze evaluation specs |
+| `npx evalai discover --manifest` | Generate stable manifest for incremental analysis |
+
+### Impact Analysis
+
+| Command | Description |
+|---------|-------------|
+| `npx evalai impact-analysis --base main` | Analyze impact of changes |
+| `npx evalai impact-analysis --changed-files file1.ts,file2.ts` | Analyze specific changed files |
+
+### Run & Diff
+
+| Command | Description |
+|---------|-------------|
+| `npx evalai run` | Run evaluation specifications |
+| `npx evalai run --write-results` | Run with artifact retention |
+| `npx evalai diff --base main` | Compare results against base branch |
+| `npx evalai diff --base last --head last` | Compare last two runs |
+| `npx evalai diff --format github` | GitHub Step Summary with regressions |
+
+### Legacy Regression Gate (local, no account needed)
 
 | Command | Description |
 |---------|-------------|
@@ -68,25 +177,29 @@ That's it. Open a PR and CI blocks regressions automatically.
 | `npx evalai explain` | Offline report explainer — top failures, root cause classification, suggested fixes |
 | `npx evalai print-config` | Show resolved config with source-of-truth annotations (file/env/default/arg) |
 
+### Migration Tools
+
+| Command | Description |
+|---------|-------------|
+| `npx evalai migrate config --in evalai.config.json --out eval/migrated.spec.ts` | Convert legacy config to DSL |
+
 **Guided failure flow:**
 
 ```
-evalai check  →  fails  →  "Next: evalai explain"
+evalai ci  →  fails  →  "Next: evalai explain --report .evalai/last-run.json"
                               ↓
                    evalai explain  →  root causes + fixes
 ```
 
-**GitHub Actions step summary** — gate result at a glance:
+**GitHub Actions step summary** — CI result at a glance with regressions and artifacts:
 
-![GitHub Actions step summary showing gate pass/fail with delta table](../../docs/images/evalai-gate-step-summary.svg)
+![GitHub Actions step summary showing CI pass/fail with delta table](../../docs/images/evalai-gate-step-summary.svg)
 
 **`evalai explain` terminal output** — root causes + fix commands:
 
 ![Terminal output of evalai explain showing top failures and suggested fixes](../../docs/images/evalai-explain-terminal.svg)
 
-`check` automatically writes `.evalai/last-report.json` so `explain` works with zero flags.
-
-`doctor` uses exit codes: **0** = ready, **2** = not ready, **3** = infra error. Use `--report` for a JSON diagnostic bundle.
+All commands automatically write artifacts so `explain` works with zero flags.
 
 ### Gate Exit Codes
 

package/dist/assertions.js

@@ -86,7 +86,9 @@ class Expectation {
             expected: substring,
             actual: text,
             message: message ||
-                (passed ? `Text contains "${substring}"` : `Text does not contain "${substring}"`),
+                (passed
+                    ? `Text contains "${substring}"`
+                    : `Text does not contain "${substring}"`),
         };
     }
     /**
@@ -103,7 +105,9 @@ class Expectation {
             expected: keywords,
             actual: text,
             message: message ||
-                (passed ? `Contains all keywords` : `Missing keywords: ${missingKeywords.join(", ")}`),
+                (passed
+                    ? `Contains all keywords`
+                    : `Missing keywords: ${missingKeywords.join(", ")}`),
         };
     }
     /**
@@ -119,7 +123,9 @@ class Expectation {
             expected: `not containing "${substring}"`,
             actual: text,
             message: message ||
-                (passed ? `Text does not contain "${substring}"` : `Text contains "${substring}"`),
+                (passed
+                    ? `Text does not contain "${substring}"`
+                    : `Text contains "${substring}"`),
         };
     }
     /**
@@ -144,7 +150,8 @@ class Expectation {
             passed,
             expected: "no PII",
             actual: foundPII.length > 0 ? `Found: ${foundPII.join(", ")}` : "no PII",
-            message: message || (passed ? "No PII detected" : `PII detected: ${foundPII.join(", ")}`),
+            message: message ||
+                (passed ? "No PII detected" : `PII detected: ${foundPII.join(", ")}`),
         };
     }
     /**
@@ -159,7 +166,10 @@ class Expectation {
             passed,
             expected: pattern.toString(),
             actual: text,
-            message: message || (passed ? `Matches pattern ${pattern}` : `Does not match pattern ${pattern}`),
+            message: message ||
+                (passed
+                    ? `Matches pattern ${pattern}`
+                    : `Does not match pattern ${pattern}`),
         };
     }
     /**
@@ -205,7 +215,8 @@ class Expectation {
             passed,
             expected: schema,
             actual: parsedJson,
-            message: message || (passed ? "JSON matches schema" : "JSON does not match schema"),
+            message: message ||
+                (passed ? "JSON matches schema" : "JSON does not match schema"),
         };
     }
     /**
@@ -253,7 +264,10 @@ class Expectation {
             passed,
             expected,
             actual,
-            message: message || (passed ? `Sentiment is ${expected}` : `Expected ${expected}, got ${actual}`),
+            message: message ||
+                (passed
+                    ? `Sentiment is ${expected}`
+                    : `Expected ${expected}, got ${actual}`),
         };
     }
     /**
@@ -269,7 +283,10 @@ class Expectation {
             passed,
             expected: range,
             actual: length,
-            message: message || (passed ? `Length ${length} is within range` : `Length ${length} not in range`),
+            message: message ||
+                (passed
+                    ? `Length ${length} is within range`
+                    : `Length ${length} not in range`),
         };
     }
     /**
@@ -284,9 +301,13 @@ class Expectation {
             name: "toNotHallucinate",
             passed,
             expected: "all ground truth facts",
-            actual: missingFacts.length > 0 ? `Missing: ${missingFacts.join(", ")}` : "all facts present",
+            actual: missingFacts.length > 0
+                ? `Missing: ${missingFacts.join(", ")}`
+                : "all facts present",
             message: message ||
-                (passed ? "No hallucinations detected" : `Missing facts: ${missingFacts.join(", ")}`),
+                (passed
+                    ? "No hallucinations detected"
+                    : `Missing facts: ${missingFacts.join(", ")}`),
         };
     }
     /**
@@ -301,7 +322,10 @@ class Expectation {
             passed,
             expected: `<= ${maxMs}ms`,
             actual: `${duration}ms`,
-            message: message || (passed ? `${duration}ms within limit` : `${duration}ms exceeds ${maxMs}ms`),
+            message: message ||
+                (passed
+                    ? `${duration}ms within limit`
+                    : `${duration}ms exceeds ${maxMs}ms`),
         };
     }
     /**
@@ -344,7 +368,8 @@ class Expectation {
             passed,
             expected: `> ${expected}`,
             actual: value,
-            message: message || (passed ? `${value} > ${expected}` : `${value} <= ${expected}`),
+            message: message ||
+                (passed ? `${value} > ${expected}` : `${value} <= ${expected}`),
         };
     }
     /**
@@ -359,7 +384,8 @@ class Expectation {
             passed,
             expected: `< ${expected}`,
             actual: value,
-            message: message || (passed ? `${value} < ${expected}` : `${value} >= ${expected}`),
+            message: message ||
+                (passed ? `${value} < ${expected}` : `${value} >= ${expected}`),
         };
     }
     /**
@@ -374,7 +400,8 @@ class Expectation {
             passed,
             expected: `between ${min} and ${max}`,
             actual: value,
-            message: message || (passed ? `${value} is within range` : `${value} is outside range`),
+            message: message ||
+                (passed ? `${value} is within range` : `${value} is outside range`),
         };
     }
     /**
@@ -389,7 +416,8 @@ class Expectation {
             passed: hasCodeBlock,
             expected: "code block",
             actual: text,
-            message: message || (hasCodeBlock ? "Contains code block" : "No code block found"),
+            message: message ||
+                (hasCodeBlock ? "Contains code block" : "No code block found"),
         };
     }
     /**
@@ -405,9 +433,13 @@ class Expectation {
             name: "toBeProfessional",
             passed,
             expected: "professional tone",
-            actual: foundProfanity.length > 0 ? `Found: ${foundProfanity.join(", ")}` : "professional",
+            actual: foundProfanity.length > 0
+                ? `Found: ${foundProfanity.join(", ")}`
+                : "professional",
             message: message ||
-                (passed ? "Professional tone" : `Unprofessional language: ${foundProfanity.join(", ")}`),
+                (passed
+                    ? "Professional tone"
+                    : `Unprofessional language: ${foundProfanity.join(", ")}`),
         };
     }
     /**
@@ -432,7 +464,8 @@ class Expectation {
             passed,
             expected: "proper grammar",
             actual: issues.length > 0 ? `Issues: ${issues.join(", ")}` : "proper grammar",
-            message: message || (passed ? "Proper grammar" : `Grammar issues: ${issues.join(", ")}`),
+            message: message ||
+                (passed ? "Proper grammar" : `Grammar issues: ${issues.join(", ")}`),
         };
     }
 }

package/dist/batch.js

@@ -81,7 +81,8 @@ class RequestBatcher {
                         pendingRequest.resolve(response.data);
                     }
                     else {
-                        pendingRequest.reject(new Error(response.error || `Request failed with status ${response.status}`));
+                        pendingRequest.reject(new Error(response.error ||
+                            `Request failed with status ${response.status}`));
                     }
                 }
             }
@@ -149,7 +150,12 @@ function canBatch(method, endpoint) {
     if (method !== "GET") {
         return false;
     }
-    const batchableEndpoints = ["/traces", "/evaluations", "/annotations", "/results"];
+    const batchableEndpoints = [
+        "/traces",
+        "/evaluations",
+        "/annotations",
+        "/results",
+    ];
     return batchableEndpoints.some((pattern) => endpoint.includes(pattern));
 }
 /**

package/dist/cli/api.js

@@ -73,7 +73,9 @@ async function publishShare(baseUrl, apiKey, evaluationId, exportData, evaluatio
         exportData,
         shareScope: "run",
         evaluationRunId,
-        ...(options?.expiresInDays != null && { expiresInDays: options.expiresInDays }),
+        ...(options?.expiresInDays != null && {
+            expiresInDays: options.expiresInDays,
+        }),
     };
     const url = `${baseUrl.replace(/\/$/, "")}/api/evaluations/${evaluationId}/publish`;
     try {

package/dist/cli/check.js

@@ -183,7 +183,11 @@ function parseArgs(argv) {
         };
     }
     if (Number.isNaN(minScore) || minScore < 0 || minScore > 100) {
-        return { ok: false, exitCode: constants_1.EXIT.BAD_ARGS, message: "Error: --minScore must be 0-100" };
+        return {
+            ok: false,
+            exitCode: constants_1.EXIT.BAD_ARGS,
+            message: "Error: --minScore must be 0-100",
+        };
     }
     if (minN !== undefined && (Number.isNaN(minN) || minN < 1)) {
         return {
@@ -210,9 +214,15 @@ function parseArgs(argv) {
             onFail,
             share,
             prCommentOut,
-            maxCostUsd: maxCostUsd != null && !Number.isNaN(maxCostUsd) ? maxCostUsd : undefined,
-            maxLatencyMs: maxLatencyMs != null && !Number.isNaN(maxLatencyMs) ? maxLatencyMs : undefined,
-            maxCostDeltaUsd: maxCostDeltaUsd != null && !Number.isNaN(maxCostDeltaUsd) ? maxCostDeltaUsd : undefined,
+            maxCostUsd: maxCostUsd != null && !Number.isNaN(maxCostUsd)
+                ? maxCostUsd
+                : undefined,
+            maxLatencyMs: maxLatencyMs != null && !Number.isNaN(maxLatencyMs)
+                ? maxLatencyMs
+                : undefined,
+            maxCostDeltaUsd: maxCostDeltaUsd != null && !Number.isNaN(maxCostDeltaUsd)
+                ? maxCostDeltaUsd
+                : undefined,
         },
     };
 }
@@ -297,7 +307,8 @@ async function runCheck(args) {
         runDetails?.results &&
         quality?.evaluationRunId) {
         const importResults = runDetails.results
-            .filter((r) => r.testCaseId != null && (r.status === "passed" || r.status === "failed"))
+            .filter((r) => r.testCaseId != null &&
+            (r.status === "passed" || r.status === "failed"))
             .map((r) => ({
             testCaseId: r.testCaseId,
             status: r.status,
@@ -306,7 +317,9 @@ async function runCheck(args) {
             assertionsJson: r.assertionsJson,
         }));
         if (importResults.length > 0) {
-            const idempotencyKey = ci ? (0, ci_context_1.computeIdempotencyKey)(args.evaluationId, ci) : undefined;
+            const idempotencyKey = ci
+                ? (0, ci_context_1.computeIdempotencyKey)(args.evaluationId, ci)
+                : undefined;
             const importRes = await (0, api_1.importRunOnFail)(args.baseUrl, args.apiKey, args.evaluationId, importResults, {
                 idempotencyKey,
                 ci,

package/dist/cli/ci-context.js

@@ -89,7 +89,9 @@ function captureCiContext() {
         provider,
         repo,
         sha,
-        branch: ref?.startsWith("refs/heads/") ? ref.slice("refs/heads/".length) : ref,
+        branch: ref?.startsWith("refs/heads/")
+            ? ref.slice("refs/heads/".length)
+            : ref,
         runUrl,
         actor,
         pr,

package/dist/cli/ci.d.ts

@@ -0,0 +1,45 @@
+/**
+ * UX-401: One-command CI loop (evalai ci)
+ *
+ * Provides a single command teams put in .github/workflows/* and never think about again.
+ */
+import type { DiffResult } from "./diff";
+import type { RunResult } from "./run";
+/**
+ * CI command options
+ */
+export interface CIOptions {
+    /** Base reference for diff comparison */
+    base?: string;
+    /** Run only impacted specs */
+    impactedOnly?: boolean;
+    /** Output format */
+    format?: "human" | "json" | "github";
+    /** Write run results */
+    writeResults?: boolean;
+}
+/**
+ * CI execution result
+ */
+export interface CIResult {
+    /** Success status */
+    success: boolean;
+    /** Exit code */
+    exitCode: number;
+    /** Execution narrative */
+    narrative: string;
+    /** Run result (if executed) */
+    runResult?: RunResult;
+    /** Diff result (if executed) */
+    diffResult?: DiffResult;
+    /** Error message (if failed) */
+    error?: string;
+}
+/**
+ * Run CI command
+ */
+export declare function runCI(options: CIOptions, projectRoot?: string): Promise<CIResult>;
+/**
+ * CLI entry point
+ */
+export declare function runCICLI(options: CIOptions): Promise<void>;