evalgate-sdk 3.3.1__tar.gz

evalgate_sdk-3.3.1/.gitignore

@@ -0,0 +1,13 @@
+__pycache__/
+*.pyc
+*.pyo
+dist/
+build/
+*.egg-info/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+htmlcov/
+coverage/
+*.coverage
+junit-results.xml

evalgate_sdk-3.3.1/CHANGELOG.md

@@ -0,0 +1,146 @@
+# Changelog
+
+All notable changes to the EvalGate Python SDK will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+Version numbering is aligned with the TypeScript SDK (`@evalgate/sdk`) and the platform API.
+
+**Version history note:** The Python SDK jumped from 1.0.0 → 1.9.x → 2.0.0 to stay in sync with the TypeScript SDK. The TypeScript SDK had many releases (1.1–1.9) before the Python SDK existed. We now align both SDKs on the same major.minor version.
+
+## [3.3.1] - 2026-03-24
+
+### Changed
+- **PyPI distribution name** — package metadata publishes under `evalgate-sdk` on PyPI (import package remains `evalgate_sdk`).
+- **Install and extras guidance** — CLI/runtime error messages, examples, and docs now consistently use `evalgate-sdk`, `evalgate-sdk[cli]`, `evalgate-sdk[openai]`, and `evalgate-sdk[anthropic]`.
+- **Version alignment** — bumped Python SDK version and spec constants to `3.3.1` for synchronized release metadata.
+
+## [3.3.0] - 2026-03-23
+
+### Changed
+- **Version bump** — Python SDK version and spec constants bumped to 3.3.0, aligned with TypeScript SDK and platform.
+
+## [3.2.7] - 2026-03-22
+
+### Changed
+
+- **Release continuation** — Advanced the Python SDK release target to `3.2.7` to stay aligned with the coordinated npm and platform release after another day of shipped fixes.
+- **Package metadata alignment** — Updated `pyproject.toml` and release validation inputs so the next PyPI publish uses the new version number consistently.
+
+### Fixed
+
+- **Assertion contract normalization** — sync assertion helpers now consistently return `AssertionResult` with boolean truthiness, including schema, syntax, instruction-following, and required-field checks.
+- **Schema and negation parity** — `matches_schema()` / `expect(...).to_match_json()` accept JSON strings and embedded JSON snippets, and `expect(...).not_` now mirrors the TypeScript fluent negation flow.
+- **`run_assertions()` compatibility** — legacy boolean and mapping-returning assertion callables are coerced into `AssertionResult` so mixed assertion batches no longer fail at runtime.
+- **Starter templates** — Python scaffolded eval examples now use `.passed` when converting assertion results into runtime `create_result()` payloads.
+- **Async assertion configuration parity** — `configure_assertions()` now accepts keyword arguments like `provider=`, `api_key=`, `model=`, and `timeout_ms=` in addition to `AssertionLLMConfig(...)`.
+- **Optional dependency guidance** — async assertion errors, docstrings, and README examples now point to the correct extras such as `pauly4010-evalgate-sdk[openai]` and `pauly4010-evalgate-sdk[anthropic]`.
+- **PyPI package metadata** — author metadata now points to EvalGate organization branding instead of a personal handle.
+
+## [3.0.2] - 2026-03-09
+
+### Changed
+
+- **Version alignment** — bumped to v3.0.2 to stay in sync with the platform API and `@evalgate/sdk` TypeScript SDK
+- **Autonomous workflow guidance** — clarified that the newest autonomous loop, clustering, synthesis, and daemon workflows currently ship in the TypeScript CLI; the Python SDK remains compatible with the shared EvalGate dataset, run-artifact, and CI flows while those advanced controls mature cross-SDK.
+
+---
+
+## [2.2.1] - 2026-03-03
+
+### Changed
+
+- **Version alignment** — bumped to v2.2.1 to stay in sync with `@evalgate/sdk` TypeScript SDK
+- No Python-specific changes in this release; see [TypeScript SDK CHANGELOG](../../src/packages/sdk/CHANGELOG.md) for 2.2.1 details
+
+---
+
+## [2.2.0] - 2026-03-03
+
+### Changed
+
+- **Version alignment** — bumped to v2.2.0 to stay in sync with `@evalgate/sdk` TypeScript SDK
+- No Python-specific changes in this release; see [TypeScript SDK CHANGELOG](../../src/packages/sdk/CHANGELOG.md) for full 2.2.0 details
+
+## [2.1.2] - 2026-03-02
+
+### Fixed
+
+- **Type safety** — aligned with platform 2.1.2; all CI checks passing
+
+## [2.1.1] - 2026-03-02
+
+### Fixed
+
+- **Contract payload validation** - Fixed ruff errors in test_contract_payloads.py
+- **CI integration** - Resolved test suite compatibility issues
+- **Linting compliance** - Fixed SIM102, E501, SIM105, I001, SIM300 ruff violations
+
+### Changed
+
+- **Test coverage** - Improved test matrix for TypeScript/Python SDK compatibility
+- **Documentation** - Updated README with PyPI downloads badge and GitHub stars
+
+## [2.0.0] - 2026-03-01
+
+### Breaking
+
+- **Rebrand:** Package renamed `pauly4010-evalai-sdk` → `pauly4010-evalgate-sdk`, module `evalai_sdk` → `evalgate_sdk`
+- **CLI:** `evalai` → `evalgate`
+- **Config:** `.evalai/` → `.evalgate/` (legacy `.evalai/` still read, with deprecation warning)
+- **Env vars:** `EVALAI_*` → `EVALGATE_*` (legacy `EVALAI_*` still work, with deprecation warning)
+- **Error class:** `EvalAIError` → `EvalGateError`
+
+### Added
+
+- Deprecation warnings when using legacy env vars or config paths
+
+## [1.9.1] - 2026-03-01
+
+### Fixed
+
+- Align `SPEC_VERSION` with OpenAPI spec 1.9.1
+- Ruff lint and format fixes (SIM102, E501, SIM105, I001, SIM300)
+
+### Changed
+
+- Package metadata: Production/Stable status, improved description and keywords
+- README: Added PyPI downloads badge, GitHub stars, status section, changelog link
+
+## [1.9.0] - 2026-02-27
+
+### Added
+
+- Full parity with TypeScript SDK 1.9.0
+- `evalai ci` CLI command — one-command CI loop
+- Run artifact retention and diff system
+- Impact analysis integration
+- Schema versioning for run reports
+
+### Changed
+
+- Exit codes standardized: 0=clean, 1=regressions, 2=config/infra
+- CLI output improvements for CI environments
+
+## [1.0.1] - 2026-02-26
+
+### Fixed
+
+- CLI init template and credential resolution
+- Run output formatting
+
+## [1.0.0] - 2026-02-25
+
+### Added
+
+- Initial Python SDK release
+- `AIEvalClient` — async HTTP client for EvalGate API
+- 20+ assertions: `expect()`, `to_contain`, `to_not_contain_pii`, `to_be_professional`, etc.
+- Test suites: `create_test_suite`, `TestSuiteConfig`, `TestSuiteCase`
+- Integrations: OpenAI, Anthropic, LangChain, CrewAI, AutoGen tracing
+- Workflow tracing: `WorkflowTracer`, handoffs, cost tracking
+- Regression gates: `evaluate_regression`, `to_pass_gate`
+- CLI: `evalai init`, `evalai run`, `evalai gate`, `evalai ci`, `evalai doctor`
+- Batch processing, caching, pagination, structured errors
+- Full type hints (`py.typed`), mypy and Pyright compatible