@pauly4010/evalai-sdk 1.7.0 → 1.8.0

package/CHANGELOG.md

@@ -5,6 +5,76 @@ All notable changes to the @pauly4010/evalai-sdk package will be documented in t
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.8.0] - 2026-02-26
+
+### ✨ Added
+
+#### CLI — `evalai doctor` Rewrite (Comprehensive Checklist)
+
+- **9 itemized checks** with pass/fail/warn/skip status and exact remediation commands:
+  1. Project detection (package.json + lockfile + package manager)
+  2. Config file validity (evalai.config.json)
+  3. Baseline file (evals/baseline.json — schema, staleness)
+  4. Authentication (API key presence, redacted display)
+  5. Evaluation target (evaluationId configured)
+  6. API connectivity (reachable, latency)
+  7. Evaluation access (permissions, baseline presence)
+  8. CI wiring (.github/workflows/evalai-gate.yml)
+  9. Provider env vars (OpenAI/Anthropic/Azure — optional)
+- **Exit codes**: `0` ready, `2` not ready, `3` infrastructure error
+- **`--report`** flag outputs full JSON diagnostic bundle (versions, hashes, latency, all checks)
+- **`--format json`** for machine-readable output
+
+#### CLI — `evalai explain` (New Command)
+
+- **Offline report explainer** — reads `.evalai/last-report.json` or `evals/regression-report.json` with zero flags
+- **Top 3 failing test cases** with input/expected/actual
+- **What changed** — baseline vs current with directional indicators
+- **Root cause classification**: prompt drift, retrieval drift, formatting drift, tool-use drift, safety/cost/latency regression, coverage drop, baseline stale
+- **Prioritized suggested fixes** with actionable commands
+- Works with both `evalai check` reports (CheckReport) and `evalai gate` reports (BuiltinReport)
+- **`--format json`** for CI pipeline consumption
+
+#### Guided Failure Flow
+
+- **`evalai check` now writes `.evalai/last-report.json`** automatically after every run
+- **Failure hint**: prints `Next: evalai explain` on gate failure
+- **GitHub step summary**: adds tip about `evalai explain` and report artifact location on failure
+
+#### CI Template Improvements
+
+- **Doctor preflight step** added to generated workflow (`continue-on-error: true`)
+- **Report artifact upload** now includes both `evals/regression-report.json` and `.evalai/last-report.json`
+
+#### `evalai init` Output Updated
+
+- First recommendation: `npx evalai doctor` (verify setup)
+- Full command reference: doctor, gate, check, explain, baseline update
+
+#### CLI — `evalai print-config` (New Command)
+
+- **Resolved config viewer** — prints every config field with its current value
+- **Source-of-truth annotations**: `[file]`, `[env]`, `[default]`, `[profile]`, `[arg]` for each field
+- **Secrets redacted** — API keys shown as `sk_t...abcd`
+- **Environment summary** — shows all relevant env vars (EVALAI_API_KEY, OPENAI_API_KEY, CI, etc.)
+- **`--format json`** for machine-readable output
+- Accepts `--evaluationId`, `--baseUrl`, etc. to show how CLI args would merge
+
+#### Minimal Green Example
+
+- **`examples/minimal-green/`** — passes on first run, no account needed
+- Zero dependencies, 3 `node:test` tests
+- Clone → init → doctor → gate → ✅
+
+### 🔧 Changed
+
+- `evalai doctor` exit codes changed: was `0`/`1`, now `0`/`2`/`3`
+- SDK README: added Debugging & Diagnostics section with guided flow diagram
+- SDK README: added Doctor Exit Codes table
+- Doctor test count: 4 → 29 tests; added 9 explain tests (38 total new tests)
+
+---
+
 ## [1.7.0] - 2026-02-25
 
 ### ✨ Added

package/README.md

@@ -2,6 +2,10 @@
 
 [![npm version](https://img.shields.io/npm/v/@pauly4010/evalai-sdk.svg)](https://www.npmjs.com/package/@pauly4010/evalai-sdk)
 [![npm downloads](https://img.shields.io/npm/dm/@pauly4010/evalai-sdk.svg)](https://www.npmjs.com/package/@pauly4010/evalai-sdk)
+[![TypeScript](https://img.shields.io/badge/TypeScript-strict-blue.svg)](https://www.typescriptlang.org/)
+[![SDK Tests](https://img.shields.io/badge/tests-172%20passed-brightgreen.svg)](#)
+[![Contract Version](https://img.shields.io/badge/report%20schema-v1-blue.svg)](#)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
 **Stop LLM regressions in CI in minutes.**
 
@@ -54,9 +58,36 @@ That's it. Open a PR and CI blocks regressions automatically.
 | Command | Description |
 |---------|-------------|
 | `npx evalai check` | Gate on quality score from dashboard |
-| `npx evalai doctor` | Verify CI/CD setup |
 | `npx evalai share` | Create share link for a run |
 
+### Debugging & Diagnostics
+
+| Command | Description |
+|---------|-------------|
+| `npx evalai doctor` | Comprehensive preflight checklist — verifies config, baseline, auth, API, CI wiring |
+| `npx evalai explain` | Offline report explainer — top failures, root cause classification, suggested fixes |
+| `npx evalai print-config` | Show resolved config with source-of-truth annotations (file/env/default/arg) |
+
+**Guided failure flow:**
+
+```
+evalai check  →  fails  →  "Next: evalai explain"
+                              ↓
+                   evalai explain  →  root causes + fixes
+```
+
+**GitHub Actions step summary** — gate result at a glance:
+
+![GitHub Actions step summary showing gate pass/fail with delta table](../../docs/images/evalai-gate-step-summary.svg)
+
+**`evalai explain` terminal output** — root causes + fix commands:
+
+![Terminal output of evalai explain showing top failures and suggested fixes](../../docs/images/evalai-explain-terminal.svg)
+
+`check` automatically writes `.evalai/last-report.json` so `explain` works with zero flags.
+
+`doctor` uses exit codes: **0** = ready, **2** = not ready, **3** = infra error. Use `--report` for a JSON diagnostic bundle.
+
 ### Gate Exit Codes
 
 | Code | Meaning |
@@ -79,6 +110,14 @@ That's it. Open a PR and CI blocks regressions automatically.
 | 7 | Weak evidence |
 | 8 | Warn (soft regression) |
 
+### Doctor Exit Codes
+
+| Code | Meaning |
+|------|---------|
+| 0 | Ready — all checks passed |
+| 2 | Not ready — one or more checks failed |
+| 3 | Infrastructure error |
+
 ---
 
 ## How the Gate Works
@@ -224,6 +263,8 @@ Your local `openAIChatEval` runs continue to work. No account cancellation. No d
 
 See [CHANGELOG.md](CHANGELOG.md) for the full release history.
 
+**v1.8.0** — `evalai doctor` rewrite (9-check checklist), `evalai explain` command, guided failure flow, CI template with doctor preflight
+
 **v1.7.0** — `evalai init` scaffolder, `evalai upgrade --full`, `detectRunner()`, machine-readable gate output, init test matrix
 
 **v1.6.0** — `evalai gate`, `evalai baseline`, regression gate constants & types

package/dist/cli/check.js

@@ -77,6 +77,7 @@ exports.EXIT = void 0;
 exports.parseArgs = parseArgs;
 exports.runCheck = runCheck;
 const fs = __importStar(require("node:fs"));
+const path = __importStar(require("node:path"));
 const api_1 = require("./api");
 const ci_context_1 = require("./ci-context");
 const config_1 = require("./config");
@@ -260,12 +261,26 @@ async function runCheck(args) {
         baselineRunId: quality?.baselineRunId ?? undefined,
         ciRunUrl: ci?.runUrl ?? undefined,
     });
+    // Persist report artifact so `evalai explain` works with zero flags
+    try {
+        const reportDir = path.join(process.cwd(), ".evalai");
+        if (!fs.existsSync(reportDir))
+            fs.mkdirSync(reportDir, { recursive: true });
+        fs.writeFileSync(path.join(reportDir, "last-report.json"), JSON.stringify(report, null, 2), "utf8");
+    }
+    catch {
+        // Non-fatal: best-effort artifact write
+    }
     const formatted = args.format === "json"
         ? (0, json_1.formatJson)(report)
         : args.format === "github"
             ? (0, github_1.formatGitHub)(report)
             : (0, human_1.formatHuman)(report);
     console.log(formatted);
+    // Guided flow hint on failure
+    if (!gateResult.passed) {
+        console.error("\nNext: evalai explain");
+    }
     // --pr-comment-out: write markdown to file for GitHub Action to post
     if (args.prCommentOut) {
         try {

package/dist/cli/doctor.d.ts

@@ -1,11 +1,88 @@
 /**
- * evalai doctor — Verify CI/CD setup.
- * Uses the same quality endpoint as check — if doctor passes, check works.
+ * evalai doctor — Comprehensive CI/CD readiness checklist.
+ *
+ * Runs itemized pass/fail checks with exact remediation commands.
+ *
+ * Exit codes:
+ *   0 — All checks passed (ready)
+ *   2 — One or more checks failed (not ready)
+ *   3 — Infrastructure error (couldn't complete checks)
+ *
+ * Flags:
+ *   --report          Output JSON diagnostic bundle (redacted)
+ *   --format <fmt>    Output format: human (default), json
+ *   --apiKey <key>    API key (or EVALAI_API_KEY env)
+ *   --baseUrl <url>   API base URL
+ *   --evaluationId <id>  Evaluation to verify
+ *   --baseline <mode> Baseline mode
  */
-export type DoctorArgs = {
+import { type EvalAIConfig } from "./config";
+export declare const DOCTOR_EXIT: {
+    readonly READY: 0;
+    readonly NOT_READY: 2;
+    readonly INFRA_ERROR: 3;
+};
+export type CheckStatus = "pass" | "fail" | "warn" | "skip";
+export interface CheckResult {
+    id: string;
+    label: string;
+    status: CheckStatus;
+    message: string;
+    remediation?: string;
+}
+export interface DiagnosticBundle {
+    timestamp: string;
+    cliVersion: string;
+    specVersion: string;
+    platform: string;
+    nodeVersion: string;
+    checks: CheckResult[];
+    config: Partial<EvalAIConfig> & {
+        path?: string | null;
+    };
+    baseline: {
+        path: string;
+        exists: boolean;
+        hash?: string;
+        schemaVersion?: number;
+        stale?: boolean;
+    } | null;
+    api: {
+        reachable: boolean;
+        latencyMs?: number;
+        scopes?: string[];
+    } | null;
+    ci: {
+        workflowPath: string;
+        exists: boolean;
+    } | null;
+    overall: "ready" | "not_ready" | "infra_error";
+}
+export interface DoctorFlags {
+    report: boolean;
+    format: "human" | "json";
+    strict: boolean;
     baseUrl: string;
     apiKey: string;
     evaluationId: string;
     baseline: "published" | "previous" | "production";
+}
+export declare function checkProject(cwd: string): CheckResult;
+export declare function checkConfig(cwd: string): CheckResult & {
+    config: EvalAIConfig | null;
+    configPath: string | null;
+};
+export declare function checkBaseline(cwd: string): CheckResult & {
+    baselineInfo: DiagnosticBundle["baseline"];
+};
+export declare function checkAuth(apiKey: string): CheckResult;
+export declare function checkConnectivity(baseUrl: string, apiKey: string): Promise<CheckResult & {
+    latencyMs?: number;
+}>;
+export declare function checkEvalTarget(evaluationId: string): CheckResult;
+export declare function checkEvalAccess(baseUrl: string, apiKey: string, evaluationId: string, baseline: string): Promise<CheckResult>;
+export declare function checkCiWiring(cwd: string): CheckResult & {
+    ciInfo: DiagnosticBundle["ci"];
 };
+export declare function checkProviderEnv(): CheckResult;
 export declare function runDoctor(argv: string[]): Promise<number>;