@pauly4010/evalai-sdk 1.4.1 → 1.5.5

package/CHANGELOG.md

@@ -5,6 +5,91 @@ All notable changes to the @pauly4010/evalai-sdk package will be documented in t
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.5.5] - 2026-02-19
+
+### ✨ Added
+
+#### Gate semantics (PASS / WARN / FAIL)
+
+- **`--warnDrop <n>`** — Introduce a WARN band when score drops > `warnDrop` but < `maxDrop`
+- **Gate verdicts:** PASS, WARN, FAIL
+- **Profiles:** `strict` (warnDrop: 0), `balanced` (warnDrop: 1), `fast` (warnDrop: 2)
+- **`--fail-on-flake`** — Fail the gate if any case is flagged as flaky (partial pass rate across determinism runs)
+
+#### Determinism & flake intelligence
+
+- **Adaptive variance thresholds** — Determinism audit passes if `absVariance ≤ 5` OR `relVariance ≤ 2%`
+- **Per-case variance reporting** — Reports per-case pass rate across N runs and flags `[FLAKY]` cases
+- **Golden dataset regression** — Added `evals/golden` with `pnpm eval:golden` to prevent semantic regressions
+- **Golden drift output** — Writes `evals/golden/golden-results.json` with `currentScore`, `baselineScore`, `delta`, `passed`, and timestamps
+
+#### CI audits & workflows
+
+- **Nightly audits** — Added `audit-nightly.yml` for determinism + performance budgets (skips without `OPENAI_API_KEY`)
+- **SDK compatibility matrix** — Added `sdk-compat.yml` to validate older SDK versions against current API
+- **New audits:** `audit:retention`, `audit:migrations`, `audit:performance`, `audit:determinism`
+
+#### Platform safety & governance (docs + proofs)
+
+- **Audit trail docs** — Added `docs/audit-trail.md`
+- **Observability docs** — Added `docs/observability.md` (log schema + requestId)
+- **Retention docs** — Added `docs/data-retention.md`
+- **Migration safety docs** — Added `docs/migration-safety.md`
+- **Adoption benchmark** — Added `docs/adoption-benchmark.md`
+- **Examples** — Added real-world example suites (RAG regression + agent tool-use)
+
+### 🔧 Changed
+
+- **Exit codes updated** — 0=pass, **8=warn**, failures remain as documented for score/regression/policy/API/config
+- **GitHub + human formatters** — Render WARN state, top contributors, and flake indicators where available
+- **Rate limiting** — Adds `Retry-After` header on 429 responses
+- **RequestId propagation** — `EvalAIError` surfaces `requestId` from response body or `x-request-id` header
+
+### 🧪 Testing
+
+- Added tests for:
+  - access boundaries (no tenant info leak)
+  - rate-limit abuse patterns + `Retry-After`
+  - executor failure modes (timeouts / upstream 429 / malformed responses)
+  - error catalog stability + graceful handling of unknown codes
+  - exports contract (retention visibility, 410 semantics)
+
+--
+
+## [1.5.0] - 2026-02-18
+
+### ✨ Added
+
+#### evalai CLI — CI DevX
+
+- **`--format github`** — GitHub Actions annotations + step summary (`$GITHUB_STEP_SUMMARY`)
+- **`--format json`** — Machine-readable output only
+- **`--onFail import`** — On gate failure, import run metadata + failures to dashboard (idempotent per CI run)
+- **`--explain`** — Show score breakdown (contribPts) and thresholds
+- **`evalai doctor`** — Verify CI setup (config, API key, quality endpoint, baseline)
+- **Pinned CLI invocation** — Use `npx -y @pauly4010/evalai-sdk@^1` for stable CI (avoids surprise v2 breaks)
+
+#### Documentation
+
+- **README** — 3-section adoption flow: 60s local → optional CI gate → no lock-in
+- **Init output** — Shows path written, pinned snippet with `--format github --onFail import`
+- **openAIChatEval** — "Gate this in CI" hint uses pinned invocation
+
+### 🔧 Changed
+
+- **evalai init** — Output: "Wrote evalai.config.json at {path}", one next step, uninstall line
+- **Baseline missing** — Treated as config failure (BAD_ARGS), not API error
+- **parseArgs** — Returns `{ ok, args }` or `{ ok: false }` (no `process.exit` inside) for testability
+
+### 📦 Internal
+
+- Refactored `check.ts` into modules: `api.ts`, `gate.ts`, `report/build-check-report.ts`, `formatters/`
+- Deterministic helpers: `truncateSnippet`, `sortFailedCases`
+- Formatter tests: `json.test.ts`, `github.test.ts`
+- Doctor tests: `doctor.test.ts`
+
+---
+
 ## [1.4.1] - 2026-02-18
 
 ### ✨ Added