@pauly4010/evalai-sdk 1.6.0 → 1.7.0

package/CHANGELOG.md

@@ -5,6 +5,50 @@ All notable changes to the @pauly4010/evalai-sdk package will be documented in t
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.7.0] - 2026-02-25
+
+### ✨ Added
+
+#### CLI — `evalai init` Full Project Scaffolder
+
+- **`evalai init`** — Zero-to-gate in under 5 minutes:
+  - Detects Node repo + package manager (npm/yarn/pnpm)
+  - Runs existing tests to capture real pass/fail + test count
+  - Creates `evals/baseline.json` with provenance metadata
+  - Installs `.github/workflows/evalai-gate.yml` (package-manager aware)
+  - Creates `evalai.config.json`
+  - Prints copy-paste next steps — just commit and push
+  - Idempotent: skips files that already exist
+
+#### CLI — `evalai upgrade --full` (Tier 1 → Tier 2)
+
+- **`evalai upgrade --full`** — Upgrade from built-in gate to full gate:
+  - Creates `scripts/regression-gate.ts` (full gate with `--update-baseline`)
+  - Adds `eval:regression-gate` + `eval:baseline-update` npm scripts
+  - Creates `.github/workflows/baseline-governance.yml` (label + diff enforcement)
+  - Upgrades `evalai-gate.yml` to project mode
+  - Adds `CODEOWNERS` entry for `evals/baseline.json`
+
+#### Gate Output — Machine-Readable Improvements
+
+- **`detectRunner()`** — Identifies test runner from `package.json` scripts: vitest, jest, mocha, node:test, ava, tap, or unknown
+- **BuiltinReport** now always emits: `durationMs`, `command`, `runner`, `baseline` metadata
+- Report schema updated with optional `durationMs`, `command`, `runner` properties
+
+#### Init Scaffolder Integration Tests
+
+- 4 fixtures: npm+jest, pnpm+vitest, yarn+jest, pnpm monorepo
+- 25 tests: files created, YAML valid, pm-aware workflow, idempotent runs
+- All fixtures use `node:test` (zero external deps)
+
+### 🔧 Changed
+
+- CLI help text updated to include `upgrade` command
+- Gate report includes runner detection and timing metadata
+- SDK test count: 147 → 172 tests (12 → 15 contract tests)
+
+---
+
 ## [1.6.0] - 2026-02-24
 
 ### ✨ Added

package/README.md

@@ -1,22 +1,108 @@
 # @pauly4010/evalai-sdk
 
-[![npm version](https://img.shields.io/npm/v/@pauly4010/evalai-sdk.svg)](https://www.npmjs.com/package/@pauly4010/evalai-sdk)  
+[![npm version](https://img.shields.io/npm/v/@pauly4010/evalai-sdk.svg)](https://www.npmjs.com/package/@pauly4010/evalai-sdk)
 [![npm downloads](https://img.shields.io/npm/dm/@pauly4010/evalai-sdk.svg)](https://www.npmjs.com/package/@pauly4010/evalai-sdk)
 
 **Stop LLM regressions in CI in minutes.**
 
-Evaluate locally in 60 seconds. Add an optional CI gate in 2 minutes.  
-No lock-in — remove by deleting `evalai.config.json`.
+Zero to gate in under 5 minutes. No infra. No lock-in. Remove anytime.
 
 ---
 
-# 🚀 1) 60 seconds: Run locally (no account)
+## Quick Start (2 minutes)
 
-Install, run, get a score.  
-No EvalAI account. No API key. No dashboard required.
+```bash
+npx @pauly4010/evalai-sdk init
+git add evals/ .github/workflows/evalai-gate.yml evalai.config.json
+git commit -m "chore: add EvalAI regression gate"
+git push
+```
+
+That's it. Open a PR and CI blocks regressions automatically.
+
+`evalai init` detects your project, creates a baseline from your current tests, and installs a GitHub Actions workflow. No manual config needed.
+
+---
+
+## What `evalai init` does
+
+1. **Detects** your Node repo and package manager (npm/yarn/pnpm)
+2. **Runs your tests** to capture a real baseline (pass/fail + test count)
+3. **Creates `evals/baseline.json`** with provenance metadata
+4. **Installs `.github/workflows/evalai-gate.yml`** (package-manager aware)
+5. **Creates `evalai.config.json`**
+6. **Prints next steps** — just commit and push
+
+---
+
+## CLI Commands
+
+### Regression Gate (local, no account needed)
+
+| Command | Description |
+|---------|-------------|
+| `npx evalai init` | Full project scaffolder — creates everything you need |
+| `npx evalai gate` | Run regression gate locally |
+| `npx evalai gate --format json` | Machine-readable JSON output |
+| `npx evalai gate --format github` | GitHub Step Summary with delta table |
+| `npx evalai baseline init` | Create starter `evals/baseline.json` |
+| `npx evalai baseline update` | Re-run tests and update baseline with real scores |
+| `npx evalai upgrade --full` | Upgrade from Tier 1 (built-in) to Tier 2 (full gate) |
+
+### API Gate (requires account)
+
+| Command | Description |
+|---------|-------------|
+| `npx evalai check` | Gate on quality score from dashboard |
+| `npx evalai doctor` | Verify CI/CD setup |
+| `npx evalai share` | Create share link for a run |
+
+### Gate Exit Codes
+
+| Code | Meaning |
+|------|---------|
+| 0 | Pass — no regression |
+| 1 | Regression detected |
+| 2 | Infra error (baseline missing, tests crashed) |
+
+### Check Exit Codes (API mode)
+
+| Code | Meaning |
+|------|---------|
+| 0 | Pass |
+| 1 | Score below threshold |
+| 2 | Regression failure |
+| 3 | Policy violation |
+| 4 | API error |
+| 5 | Bad arguments |
+| 6 | Low test count |
+| 7 | Weak evidence |
+| 8 | Warn (soft regression) |
+
+---
+
+## How the Gate Works
+
+**Built-in mode** (any Node project, no config needed):
+- Runs `<pm> test`, captures exit code + test count
+- Compares against `evals/baseline.json`
+- Writes `evals/regression-report.json`
+- Fails CI if tests regress
+
+**Project mode** (advanced, for full regression gate):
+- If `eval:regression-gate` script exists in `package.json`, delegates to it
+- Supports golden eval scores, confidence tests, p95 latency, cost tracking
+- Full delta table with tolerances
+
+---
+
+## Run a Regression Test Locally (no account)
 
 ```bash
 npm install @pauly4010/evalai-sdk openai
+```
+
+```typescript
 import { openAIChatEval } from "@pauly4010/evalai-sdk";
 
 await openAIChatEval({
@@ -26,14 +112,14 @@ await openAIChatEval({
     { input: "2 + 2 = ?", expectedOutput: "4" },
   ],
 });
-Set:
+```
 
-OPENAI_API_KEY=...
-✅ Vitest integration (recommended)
-import {
-  openAIChatEval,
-  extendExpectWithToPassGate,
-} from "@pauly4010/evalai-sdk";
+Output: `PASS 2/2 (score: 100)`. No account needed. Just a score.
+
+### Vitest Integration
+
+```typescript
+import { openAIChatEval, extendExpectWithToPassGate } from "@pauly4010/evalai-sdk";
 import { expect } from "vitest";
 
 extendExpectWithToPassGate(expect);
@@ -46,278 +132,113 @@ it("passes gate", async () => {
       { input: "2 + 2 = ?", expectedOutput: "4" },
     ],
   });
-
   expect(result).toPassGate();
 });
-Example output
-PASS 2/2 (score: 100)
-
-Tip: Want dashboards and history?
-Set EVALAI_API_KEY and connect this to the platform.
-Failures show:
-
-FAIL 9/10 (score: 90)
-with failed cases and CI guidance.
-
-⚡ 2) Optional: Add a CI gate (2 minutes)
-When you're ready to gate PRs on quality and regressions:
-
-npx -y @pauly4010/evalai-sdk@^1 init
-Create an evaluation in the dashboard and paste its ID into:
-
-{
-  "evaluationId": "42"
-}
-Add to your CI:
-
-- name: EvalAI gate
-  env:
-    EVALAI_API_KEY: ${{ secrets.EVALAI_API_KEY }}
-  run: npx -y @pauly4010/evalai-sdk@^1 check --format github --onFail import --warnDrop 1
-You’ll get:
-
-GitHub annotations
-
-Step summary
-
-Optional dashboard link
-
-PASS / WARN / FAIL (v1.5.7)
-EvalAI introduces a WARN band so teams can see meaningful regressions without always blocking merges.
-
-Behavior
-
-PASS → within thresholds
-
-WARN → regression > warnDrop but < maxDrop
-
-FAIL → regression > maxDrop
-
-Key flags
-
---warnDrop → soft regression warning
-
---maxDrop → hard regression fail
-
---fail-on-flake → fail if unknown test is unstable
-
-This lets teams tune signal vs noise in CI.
-
-🔒 3) No lock-in
-To stop using EvalAI:
-
-rm evalai.config.json
-Your local openAIChatEval runs continue to work exactly the same.
-
-No account cancellation. No data export required.
-
-📦 Installation
-npm install @pauly4010/evalai-sdk openai
-# or
-yarn add @pauly4010/evalai-sdk openai
-# or
-pnpm add @pauly4010/evalai-sdk openai
-🖥️ Environment Support
-This SDK works in both Node.js and browsers, with some Node-only features.
-
-✅ Works Everywhere (Node.js + Browser)
-Traces API
-
-Evaluations API
-
-LLM Judge API
-
-Annotations API
-
-Developer API (API Keys, Webhooks, Usage)
-
-Organizations API
-
-Assertions Library
-
-Test Suites
-
-Error Handling
-
-CJS/ESM Compatibility
+```
 
-🟡 Node.js Only
-These require Node.js:
+---
 
-Snapshot Testing
+## SDK Exports
 
-Local Storage Mode
+### Regression Gate Constants
 
-CLI Tool
+```typescript
+import {
+  GATE_EXIT,           // { PASS: 0, REGRESSION: 1, INFRA_ERROR: 2, ... }
+  GATE_CATEGORY,       // { PASS: "pass", REGRESSION: "regression", INFRA_ERROR: "infra_error" }
+  REPORT_SCHEMA_VERSION,
+  ARTIFACTS,           // { BASELINE, REGRESSION_REPORT, CONFIDENCE_SUMMARY, LATENCY_BENCHMARK }
+} from "@pauly4010/evalai-sdk";
 
-Export to File
+// Or tree-shakeable:
+import { GATE_EXIT } from "@pauly4010/evalai-sdk/regression";
+```
 
-🔄 Context Propagation
-Node.js: full async context via AsyncLocalStorage
+### Types
+
+```typescript
+import type {
+  RegressionReport,
+  RegressionDelta,
+  Baseline,
+  BaselineTolerance,
+  GateExitCode,
+  GateCategory,
+} from "@pauly4010/evalai-sdk/regression";
+```
 
-Browser: basic support (not safe across all async boundaries)
+### Platform Client
 
-🧠 AIEvalClient (Platform API)
+```typescript
 import { AIEvalClient } from "@pauly4010/evalai-sdk";
 
-// From env
-const client = AIEvalClient.init();
-
-// Explicit
-const client2 = new AIEvalClient({
-  apiKey: "your-api-key",
-  organizationId: 123,
-  debug: true,
-});
-🧪 evalai CLI (v1.5.7)
-The CLI gates deployments on quality, regression, and policy.
-
-Quick start
-npx -y @pauly4010/evalai-sdk@^1 check \
-  --evaluationId 42 \
-  --apiKey $EVALAI_API_KEY
-evalai check
-Option	Description
---evaluationId <id>	Required. Evaluation to gate on
---apiKey <key>	API key (or EVALAI_API_KEY)
---format <fmt>	human, json, or github
---onFail import	Import failing run to dashboard
---explain	Show score breakdown
---minScore <n>	Fail if score < n
---warnDrop <n>	Warn if regression exceeds n
---maxDrop <n>	Fail if regression exceeds n
---minN <n>	Fail if test count < n
---allowWeakEvidence	Permit weak evidence
---policy <name>	HIPAA, SOC2, GDPR, PCI_DSS, FINRA_4511
---baseline <mode>	published, previous, production
---fail-on-flake	Fail if unknown case is flaky
---baseUrl <url>	Override API base URL
-
-Exit codes
-Code	Meaning
-0	PASS
-8	WARN
-1	Score below threshold
-2	Regression failure
-3	Policy violation
-4	API error
-5	Bad arguments
-6	Low test count
-7	Weak evidence
-evalai doctor
-Verify CI setup before running the gate:
-
-npx -y @pauly4010/evalai-sdk@^1 doctor \
-  --evaluationId 42 \
-  --apiKey $EVALAI_API_KEY
-If doctor passes, check will work.
-
-🧯 Error Handling
-import { EvalAIError, RateLimitError } from "@pauly4010/evalai-sdk";
-
-try {
-  await client.traces.create({ name: "User Query" });
-} catch (err) {
-  if (err instanceof RateLimitError) {
-    console.log("Retry after:", err.retryAfter);
-  } else if (err instanceof EvalAIError) {
-    console.log(err.code, err.message, err.requestId);
-  }
-}
-
-
-🔍 Traces
-const trace = await client.traces.create({
-  name: "User Query",
-  traceId: "trace-123",
-  metadata: { userId: "456" },
-});
-
-
-📝 Evaluations
-import { EvaluationTemplates } from "@pauly4010/evalai-sdk";
-
-const evaluation = await client.evaluations.create({
-  name: "Chatbot Responses",
-  type: EvaluationTemplates.OUTPUT_QUALITY,
-  createdBy: userId,
-});
+const client = AIEvalClient.init(); // from EVALAI_API_KEY env
+// or
+const client = new AIEvalClient({ apiKey: "...", organizationId: 123 });
+```
 
+### Framework Integrations
 
-🔌 Framework Integrations
+```typescript
 import { traceOpenAI } from "@pauly4010/evalai-sdk/integrations/openai";
-import OpenAI from "openai";
-
-const openai = traceOpenAI(new OpenAI(), client);
-
-await openai.chat.completions.create({
-  model: "gpt-4",
-  messages: [{ role: "user", content: "Hello" }],
-});
-
-
-🧭 Changelog
-v1.5.8 (Latest)
-Fixed secureRoute TypeScript overload compatibility
-
-Fixed test infrastructure (expect.any, NextRequest constructor)
-
-Fixed 304 response handling in exports API
-
-Improved error catalog test coverage
-
-v1.5.7
-Documentation updates for CJS compatibility
-
-Version alignment across README and changelog
-
-Environment support section updated
-
-v1.5.6
-PASS/WARN/FAIL gate semantics
+import { traceAnthropic } from "@pauly4010/evalai-sdk/integrations/anthropic";
+```
 
---warnDrop soft regression band
+---
 
-Flake intelligence + per-case pass rates
+## Installation
 
---fail-on-flake enforcement
+```bash
+npm install @pauly4010/evalai-sdk
+# or
+yarn add @pauly4010/evalai-sdk
+# or
+pnpm add @pauly4010/evalai-sdk
+```
 
-Golden regression suite
+Add `openai` as a peer dependency if using `openAIChatEval`:
 
-Nightly determinism + performance audits
+```bash
+npm install openai
+```
 
-Audit trail, observability, retention, and migration safety docs
+## Environment Support
 
-CJS compatibility for all subpath exports
+| Feature | Node.js | Browser |
+|---------|---------|---------|
+| Platform APIs (Traces, Evaluations, LLM Judge) | ✅ | ✅ |
+| Assertions, Test Suites, Error Handling | ✅ | ✅ |
+| CJS/ESM | ✅ | ✅ |
+| CLI, Snapshots, File Export | ✅ | — |
+| Context Propagation | ✅ Full | ⚠️ Basic |
 
-v1.5.0
-GitHub annotations formatter
+## No Lock-in
 
-JSON formatter
+```bash
+rm evalai.config.json
+```
 
---onFail import
+Your local `openAIChatEval` runs continue to work. No account cancellation. No data export required.
 
---explain
+## Changelog
 
-evalai doctor
+See [CHANGELOG.md](CHANGELOG.md) for the full release history.
 
-CI pinned invocation guidance
+**v1.7.0** — `evalai init` scaffolder, `evalai upgrade --full`, `detectRunner()`, machine-readable gate output, init test matrix
 
+**v1.6.0** — `evalai gate`, `evalai baseline`, regression gate constants & types
 
-Environment Variable Safety
+**v1.5.8** — secureRoute fix, test infra fixes, 304 handling fix
 
-The SDK never assumes `process.env` exists. All environment reads are guarded, so the client can initialize safely in browser, edge, and server runtimes.
+**v1.5.5** — PASS/WARN/FAIL semantics, flake intelligence, golden regression suite
 
-If environment variables are unavailable, the SDK falls back to explicit config.
+**v1.5.0** — GitHub annotations, `--onFail import`, `evalai doctor`
 
+## License
 
-📄 License
 MIT
 
-🤝 Support
-Documentation:
-https://v0-ai-evaluation-platform-nu.vercel.app/documentation
+## Support
 
-Issues:
-https://github.com/pauly7610/ai-evaluation-platform/issues
-```
+- **Docs:** https://v0-ai-evaluation-platform-nu.vercel.app/documentation
+- **Issues:** https://github.com/pauly7610/ai-evaluation-platform/issues

package/dist/cli/baseline.js

@@ -142,7 +142,7 @@ function runBaselineUpdate(cwd) {
     }
     if (!pkg.scripts?.["eval:baseline-update"]) {
         console.error("❌ Missing 'eval:baseline-update' script in package.json.");
-        console.error("   Add it:  \"eval:baseline-update\": \"npx tsx scripts/regression-gate.ts --update-baseline\"");
+        console.error('   Add it:  "eval:baseline-update": "npx tsx scripts/regression-gate.ts --update-baseline"');
         return 1;
     }
     console.log("📊 Running baseline update...\n");

package/dist/cli/index.js

@@ -14,6 +14,7 @@ const doctor_1 = require("./doctor");
 const init_1 = require("./init");
 const regression_gate_1 = require("./regression-gate");
 const share_1 = require("./share");
+const upgrade_1 = require("./upgrade");
 const argv = process.argv.slice(2);
 const subcommand = argv[0];
 if (subcommand === "init") {
@@ -29,6 +30,10 @@ else if (subcommand === "gate") {
     const code = (0, regression_gate_1.runGate)(argv.slice(1));
     process.exit(code);
 }
+else if (subcommand === "upgrade") {
+    const code = (0, upgrade_1.runUpgrade)(argv.slice(1));
+    process.exit(code);
+}
 else if (subcommand === "doctor") {
     (0, doctor_1.runDoctor)(argv.slice(1))
         .then((code) => process.exit(code))
@@ -71,6 +76,7 @@ Usage:
   evalai baseline init       Create starter evals/baseline.json
   evalai baseline update     Run tests and update baseline with real scores
   evalai gate [options]      Run regression gate (local test-based)
+  evalai upgrade --full      Upgrade from Tier 1 to Tier 2 (full gate)
   evalai doctor [options]    Verify CI/CD setup (same endpoint as check)
   evalai check [options]     CI/CD evaluation gate (API-based)
   evalai share [options]     Create share link for a run

package/dist/cli/init.d.ts

@@ -1,7 +1,16 @@
 #!/usr/bin/env node
 /**
- * evalai init — Create evalai.config.json
+ * evalai init — Full project scaffolder
  *
- * Creates the smallest possible config file. Defaults belong in code.
+ * Zero-to-gate in under 5 minutes:
+ *   npx evalai init
+ *   git push
+ *   …CI starts blocking regressions.
+ *
+ * What it does:
+ *   1. Detects Node repo + package manager
+ *   2. Creates evals/ directory + baseline.json
+ *   3. Installs .github/workflows/evalai-gate.yml
+ *   4. Prints next steps (no docs required)
  */
 export declare function runInit(cwd?: string): boolean;