evalgate-sdk 3.5.1__tar.gz → 3.13.0__tar.gz

evalgate_sdk-3.13.0/.evalgate/harness-events.jsonl

@@ -0,0 +1 @@
+{"schemaVersion":1,"sessionId":"evalgate-gate-32a53eb6f72d470c066c32d8","harnessKey":"evalgate","agent":"evalgate","outcome":"passed","inputTokens":0,"outputTokens":0,"recordedAt":"2026-10-13T02:13:37.125450Z","taskKey":"latest","captureSource":"evalgate_gate","evidenceHash":"32a53eb6f72d470c066c32d828dd4b6f64d3e4e5539a0f0ea7bdf247056fd3de","evidencePath":".evalgate/last-report.json","qualityScore":91.0}

evalgate_sdk-3.13.0/.evalgate/last-report.json

@@ -0,0 +1,23 @@
+{
+  "evaluationId": "42",
+  "verdict": "pass",
+  "gateApplied": true,
+  "gateMode": "enforced",
+  "reasonCode": "PASS",
+  "schemaVersion": 1,
+  "score": 91.0,
+  "thresholds": {
+    "minScore": 90,
+    "allowWeakEvidence": false,
+    "baseline": "published"
+  },
+  "n": 12,
+  "evidenceLevel": "strong",
+  "failedCases": [],
+  "requestId": "test-req",
+  "decisionPassed": true,
+  "reportingPassed": true,
+  "evidencePassed": true,
+  "releaseReady": true,
+  "evidenceMode": "offline"
+}

evalgate_sdk-3.13.0/CHANGELOG.md

@@ -0,0 +1,436 @@
+# Changelog
+
+All notable changes to the EvalGate Python SDK will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+Version numbering is aligned with the TypeScript SDK (`@evalgate/sdk`) and the platform API.
+
+**Version history note:** The Python SDK jumped from 1.0.0 → 1.9.x → 2.0.0 to stay in sync with the TypeScript SDK. The TypeScript SDK had many releases (1.1–1.9) before the Python SDK existed. We now align both SDKs on the same major.minor version.
+
+## [Unreleased]
+
+## [3.13.0] - 2026-10-11 (source candidate; PyPI publication pending)
+
+- Align the Python source and generated client contracts with October's
+  OpenAPI 13.1.1 and durable repository-scan enqueue/phase semantics.
+- Preserve Python 3.10+ runtime compatibility and strict type checks; refresh
+  reviewed Connections, scan, evaluation and CLI interface metadata.
+- A source version is not a registry release. PyPI remains separately
+  governed; run Python 3.10–3.13 tests, mypy, Ruff, build/wheel smoke and
+  registry publication validation before claiming a published SDK.
+
+## [3.12.5] - 2026-09-30
+
+- Align Python source package version with TypeScript `3.12.5` tip identity.
+  PyPI remains separately governed at its last published line.
+
+
+## [3.12.4] - 2026-09-29
+
+- Align Python source package version with TypeScript `3.12.4` tip identity.
+  PyPI remains separately governed at its last published line.
+
+
+## [3.12.3] - 2026-09-23
+
+Aligns Python source with the TypeScript 3.12.3 patch identity, including
+`login --scope` for consented collector reads. PyPI publication remains
+separately governed.
+
+- Source version tracks `@evalgate/sdk` 3.12.3 / OpenAPI 12.0.0.
+- Device login accepts validated `--scope`; defaults still omit `traces:read`.
+
+## [3.12.2] - 2026-09-23
+
+Aligns Python source with the TypeScript 3.12.2 patch identity. PyPI
+publication remains separately governed.
+
+- Source version tracks `@evalgate/sdk` 3.12.2 / OpenAPI 12.0.0.
+
+## [3.12.1] - 2026-09-23
+
+Aligns Python source with the TypeScript 3.12.1 patch identity. PyPI
+publication remains separately governed.
+
+- Source version tracks `@evalgate/sdk` 3.12.1 / OpenAPI 12.0.0.
+
+## [3.12.0] - 2026-09-23
+
+Aligns Python source with the TypeScript 3.12.0 release-delta identity and the
+refreshed OpenAPI 12.0.0 contract. PyPI publication remains separately
+governed.
+
+- Source version tracks `@evalgate/sdk` 3.12.0 / OpenAPI 12.0.0.
+- Regenerated operation registry includes adjudication prepare/activity and
+  evaluator-validation plan/resume.
+- No PyPI publish is claimed by this cut; registry publish remains blocked
+  with TypeScript under DISC-20260922-001 until Actions can start Release.
+
+## [3.11.0] - 2026-09-19
+
+Aligns Python source with the TypeScript 3.11.0 release-delta identity. PyPI
+publication remains separately governed.
+
+- Source version tracks `@evalgate/sdk` 3.11.0 / OpenAPI 7.0.0.
+- No PyPI publish is claimed by this cut; registry remains at 3.6.0 until a
+  separately governed publish.
+
+## [3.10.0] - 2026-09-09
+
+Aligns Python source with the TypeScript 3.10.0 credential-free local gate and
+release-evidence honesty release. PyPI publication remains separately governed.
+
+- Add source parity for credential-free local initialization, canonical local
+  gate readiness, baseline artifact hygiene, and intent-specific next actions
+  without creating hosted organization or repository identity.
+- Add targeted JSON-safe migration handling for removed lifecycle commands and
+  align Red Team evidence discovery/error categories with TypeScript.
+- Reframe the source README around distinct local and hosted workflows while
+  retaining the separately governed PyPI publication boundary.
+
+## [3.9.0] - 2026-09-04
+
+Aligns Python source with the TypeScript 3.9.0 Red Team release-evidence
+contract: exact campaign export, Ed25519 trust-anchor verification,
+lifecycle-aware strict profiles, fail-closed deferral semantics, cross-field
+evidence validation, and structured export failure categories. PyPI publication
+remains separately governed.
+
+Python source also converges on the clean CLI activation lifecycle: browser-
+approved `login`, identity-preserving `whoami`, durable non-secret repository
+`link`, split link/checkout/cloud/readiness `status`, control-plane `open`,
+remotely revoking `logout`, and TTY-aware `init`. Version 2 project context
+separates organization/repository/root identity from the last exact activation
+snapshot, so branch movement does not unlink the project and cloud work never
+silently substitutes a different commit. Explicit preview, JSON, and non-TTY
+init remain non-mutating without `--apply`. The prior auth/setup families and
+mutation aliases are removed.
+
+## [3.8.2] - 2026-08-31
+
+Aligns Python source with the TypeScript 3.8.2 baseline source-identity and
+init/exit contract fixes. PyPI publication remains separately governed.
+
+## [3.8.1] - 2026-08-30
+
+Aligns Python source with the TypeScript 3.8.1 release-evidence hotfix. Pending
+baselines fail closed, local and hosted gates share one envelope, and generated
+CI remains read-only. PyPI publication remains separately governed.
+
+## [3.8.0] - 2026-08-29
+
+Aligns Python source with the 3.8.0 reliability and strict gate envelope
+release. PyPI publication remains separately governed.
+
+### Changed
+
+- Regression gate and formatter output follow the same strict envelope and
+  trajectory-gate contracts as the TypeScript SDK.
+
+## [3.7.11] - 2026-08-28
+
+Aligns Python source with the 3.7.11 discovery, skill-contract, and public
+sharing hardening release. PyPI publication remains separately governed.
+
+### Discovery
+
+- Public metadata and generated operations stay aligned with the published
+  package identity and first-hop agent discovery surfaces.
+
+### Security and reliability
+
+- Public-sharing policy and execution recovery contracts match the TypeScript
+  release boundary.
+
+## [3.7.10] - 2026-08-27
+
+Hardens CLI credential storage for the Python package while keeping PyPI
+publication as a separately governed release boundary.
+
+### Security
+
+- **User-scoped credential storage** — `evalgate auth configure` stores bearer
+  credentials in the operating-system user configuration directory instead of
+  the repository. Repository configuration cannot supply API keys or redirect a
+  user/environment credential to another API origin. Trusted alternate origins
+  require HTTPS, except explicit localhost and loopback development.
+- **Safe upgrade boundary** — repository-local `.evalgate/config.json`
+  credentials are ignored without automatic migration. Re-run `auth configure`
+  or use `EVALGATE_API_KEY`, then delete and rotate any legacy secret that may
+  have entered Git history.
+
+## [3.7.9] - 2026-08-26
+
+The 3.7.9 source expands Python from evaluation execution into the same guided
+repository-to-release workflow as the TypeScript CLI. PyPI publication remains a
+separately governed release boundary; source availability is not a registry
+claim.
+
+### Connect and activate
+
+- Authentication and setup commands support scoped key validation, signed-in
+  repository installation, human approval, and durable activation status.
+- `connect --client` plans native Codex, Claude Code, and Cursor instructions or
+  a portable Skill without writing credentials or contacting a provider.
+
+### Measure and improve
+
+- On-demand application-health commands capture bounded, redacted HTTP and
+  command evidence without an always-on collector.
+- Harness optimization uses metadata-only capture, reviewed feedback, managed
+  Skill mutation, and rollback contracts aligned with TypeScript.
+- Product-understanding and quality-profile commands keep inferred context
+  provisional, ground cases in examples, and preview writes before apply.
+
+### Automate through the shared contract
+
+The generated OpenAPI client and capability map cover repository activation,
+execution state, agent authentication, and the expanded 3.7.9 source contract.
+
+## [3.7.8] - 2026-08-15
+
+### Fixed
+
+- **Spec identity** — the client advertises OpenAPI spec `3.7.8`, which documents five previously missing operations.
+
+## [3.7.7] - 2026-08-14
+
+### Added
+
+- **Understand-first CLI** — `evalgate understand` produces evidence-backed local product context, confidence, open questions, and apply-gated `evalgate.product.json` output aligned with the TypeScript SDK.
+- **Quality-profile synthesis** — `evalgate synthesize --profile` reads versioned preferred/undesired examples, previews by default, and writes only quarantined draft cases after explicit apply.
+
+### Fixed
+
+- **Package release identity** — advances the Python package/runtime version with the TypeScript SDK and advertises OpenAPI spec `3.7.7` (Copilot repository domain).
+- **Grounded dimension handling** — profile dimensions are validated as vocabulary unless examples provide explicit values, preventing synthetic contradictions.
+
+- **Python 3.10 compatibility** — `evaluators`, `pytest_plugin`, and the LangSmith migration CLI imported `datetime.UTC`, which doesn't exist before Python 3.11. Despite `requires-python = ">=3.10"`, using the SDK on 3.10 raised `ImportError` at import time. Switched to `datetime.timezone.utc`, which works on all supported versions.
+- **Annotation task/item routes 404'd** — `client.annotations.tasks` and `.tasks.items` called `/api/annotation-tasks`, a URL that doesn't exist on the backend. The real routes are `/api/annotations/tasks[...]`, matching the TypeScript SDK. `tasks.get()` also failed to unwrap the `{ task: ... }` response envelope. Neither had any test coverage, which is how this went unnoticed.
+- **`developer.get_usage()` / `get_usage_summary()` silently returned empty data** — `UsageStats` and `UsageSummary` modeled a flat shape, but the API nests fields under `analytics`/`period` and `summary`/`period` respectively. Every field would have come back at its default (usually `0`) with no error. Also, `get_usage_summary()` sent an `organizationId` query parameter the endpoint doesn't accept instead of `period`, and required an argument the endpoint doesn't need.
+
+### Added
+
+Closed a Python ⟷ TypeScript SDK parity audit — the following existed in `@evalgate/sdk` but not `evalgate-sdk`:
+
+- **`client.proposed_changes`** — list/get/approve/reject proposed prompt changes from the autonomous-improvement review queue.
+- **`client.governance`** — evaluate an agent decision against a configured compliance preset (SOC2/GDPR/HIPAA/PCI-DSS/etc).
+- **`client.human_review`** — create/fetch human-review sampling batches and submit reviewer verdicts.
+- **`client.evaluations.list_prompt_versions()` / `.set_active_prompt_version()` / `.rollback_prompt()`** — evaluation-scoped prompt version history and rollback.
+- **`client.organizations.list_members()` / `.list_invitations()` / `.invite_member()` / `.resend_invitation()` / `.revoke_invitation()` / `.update_member_role()` / `.remove_member()`** — organization member and invitation management.
+- **`client.datasets.update()` / `.archive()` / `.list_rows()` / `.from_traces()` / `.health()`** — the remaining Dataset Hub operations.
+
+### Security
+
+- **Raised the pydantic floor to 2.4** — `pydantic>=2.0` allowed installs on a version with a known regular-expression denial-of-service in email-string validation (fixed upstream in 2.4.0, [GHSA-mr82-8j83-vxmv](https://github.com/advisories/GHSA-mr82-8j83-vxmv)).
+
+## [3.7.4] - 2026-08-03
+
+### Added
+
+- **First-class Python and JavaScript package handlers** — the Python CLI now models npm, pnpm, Yarn, Bun, pip, Poetry, and uv with explicit install, execution, script, and lockfile behavior aligned to the TypeScript CLI.
+
+### Fixed
+
+- **Package-aware initialization and gates** — project discovery, generated commands, regression baselines, and gate execution retain the detected package manager instead of substituting a generic fallback.
+
+## [3.7.3] - 2026-08-02
+
+### Fixed
+
+- **Exact template transactions** — template-aware plans enumerate the files apply creates, rollback owns those files, and initialization no longer edits user `.gitignore` files.
+- **Accurate gate evidence** — repository-connected eval gates report `evalgate run`/`evalgate` and include evaluation execution time in their duration instead of labeling the work as pytest.
+
+## [3.7.2] - 2026-07-27
+
+### Added
+
+- **Evaluator measurement workflows** — adds typed validation-run, release, observation, metric, outcome, and measurement CLI contracts aligned with API spec `3.7.3`.
+
+### Changed
+
+- **Version alignment** — package and runtime SDK identity advance to `3.7.2` while `SPEC_VERSION` tracks the stacked evaluator-measurement API contract at `3.7.3`.
+
+## [3.7.1] - 2026-07-21
+
+### Added
+- **Production-loop scaffold parity** — `evalgate init --template production-loop`, `evalgate verify --local`, configured `discover`, JSON `validate`, and JSON/dry-run `gate` now work together from a clean Python SDK install.
+
+- **Complete generated operation registry** — Python parity metadata now covers all 603 source-verified public API operations.
+
+### Changed
+
+- **Version alignment** — package, runtime, and OpenAPI spec constants now identify the coordinated `3.7.1` release.
+
+### Fixed
+
+- **Trusted PyPI identity** — publishing runs as the top-level `publish-pypi.yml` workflow with the protected `pypi` environment, avoiding unsupported reusable-workflow OIDC claims.
+
+## [3.5.1] - 2026-06-26
+
+### Fixed
+- **CLI JSON contracts** — `check --format json` now emits one parseable JSON document.
+- **Zero-config scaffolds** — `start --template` writes config that `run` and `doctor` discover consistently.
+- **Runtime scoring defaults** — boolean/dict executor fallbacks now use the SDK's 0-100 score scale.
+- **Sync safety** — missing labeled datasets surface an explicit error, and prompt pulls write UTF-8 content on Windows.
+- **Default CLI install** — `pip install evalgate-sdk` now includes the runtime dependencies needed for the `evalgate` command.
+
+## [3.5.0] - 2026-04-09
+
+### Added
+- **Knowledge API parity** — new `client.knowledge.search()`, `upload_document()`, and `get_grounding_context()` methods matching the TypeScript SDK surface.
+- **Auto-session SDK methods** — `evaluations.create_auto_session()`, `list_auto_sessions()`, `get_auto_session()`, `get_quality_score()`, and `synthesize_test_cases()` for autonomous loop management.
+- **Batch trace ingestion** — `report_traces()` function for sending up to 100 traces per request.
+- **Offline trace persistence** — `WorkflowTracer` supports JSON file persistence when disconnected.
+
+### Changed
+- **Version alignment** — bumped Python SDK package metadata and spec constants to `3.5.0`.
+
+## [3.4.0] - 2026-03-29
+
+### Added
+- **Judge orchestration parity** — added Python client and CLI support for the normalized judge registry, presets, run/case judge inspection, and multi-judge evidence payloads.
+- **Workflow and controls parity** — expanded Python operator surfaces for workflow execution, system controls, and workflow-aware parity with the TypeScript SDK.
+- **Provider/PII safeguards** — outbound judge/assertion flows now enforce shared provider policy and pre-send PII protection semantics in the Python SDK path too.
+
+### Changed
+- **Canonical contract alignment** — updated Python types and CLI copy to match the workflow-native and judge-orchestration model used across the platform.
+- **Version alignment** — bumped Python SDK package metadata and spec constants to `3.4.0`.
+
+## [3.3.1] - 2026-03-24
+
+### Changed
+- **PyPI distribution name** — package metadata publishes under `evalgate-sdk` on PyPI (import package remains `evalgate_sdk`).
+- **Install and extras guidance** — CLI/runtime error messages, examples, and docs now consistently use `evalgate-sdk`, `evalgate-sdk[cli]`, `evalgate-sdk[openai]`, and `evalgate-sdk[anthropic]`.
+- **Version alignment** — bumped Python SDK version and spec constants to `3.3.1` for synchronized release metadata.
+
+## [3.3.0] - 2026-03-23
+
+### Changed
+- **Version bump** — Python SDK version and spec constants bumped to 3.3.0, aligned with TypeScript SDK and platform.
+
+## [3.2.7] - 2026-03-22
+
+### Changed
+
+- **Release continuation** — Advanced the Python SDK release target to `3.2.7` to stay aligned with the coordinated npm and platform release after another day of shipped fixes.
+- **Package metadata alignment** — Updated `pyproject.toml` and release validation inputs so the next PyPI publish uses the new version number consistently.
+
+### Fixed
+
+- **Assertion contract normalization** — sync assertion helpers now consistently return `AssertionResult` with boolean truthiness, including schema, syntax, instruction-following, and required-field checks.
+- **Schema and negation parity** — `matches_schema()` / `expect(...).to_match_json()` accept JSON strings and embedded JSON snippets, and `expect(...).not_` now mirrors the TypeScript fluent negation flow.
+- **`run_assertions()` compatibility** — legacy boolean and mapping-returning assertion callables are coerced into `AssertionResult` so mixed assertion batches no longer fail at runtime.
+- **Starter templates** — Python scaffolded eval examples now use `.passed` when converting assertion results into runtime `create_result()` payloads.
+- **Async assertion configuration parity** — `configure_assertions()` now accepts keyword arguments like `provider=`, `api_key=`, `model=`, and `timeout_ms=` in addition to `AssertionLLMConfig(...)`.
+- **Optional dependency guidance** — async assertion errors, docstrings, and README examples now point to the correct extras such as `pauly4010-evalgate-sdk[openai]` and `pauly4010-evalgate-sdk[anthropic]`.
+- **PyPI package metadata** — author metadata now points to EvalGate organization branding instead of a personal handle.
+
+## [3.0.2] - 2026-03-09
+
+### Changed
+
+- **Version alignment** — bumped to v3.0.2 to stay in sync with the platform API and `@evalgate/sdk` TypeScript SDK
+- **Autonomous workflow guidance** — clarified that the newest autonomous loop, clustering, synthesis, and daemon workflows currently ship in the TypeScript CLI; the Python SDK remains compatible with the shared EvalGate dataset, run-artifact, and CI flows while those advanced controls mature cross-SDK.
+
+---
+
+## [2.2.1] - 2026-03-03
+
+### Changed
+
+- **Version alignment** — bumped to v2.2.1 to stay in sync with `@evalgate/sdk` TypeScript SDK
+- No Python-specific changes in this release; see [TypeScript SDK CHANGELOG](../../src/packages/sdk/CHANGELOG.md) for 2.2.1 details
+
+---
+
+## [2.2.0] - 2026-03-03
+
+### Changed
+
+- **Version alignment** — bumped to v2.2.0 to stay in sync with `@evalgate/sdk` TypeScript SDK
+- No Python-specific changes in this release; see [TypeScript SDK CHANGELOG](../../src/packages/sdk/CHANGELOG.md) for full 2.2.0 details
+
+## [2.1.2] - 2026-03-02
+
+### Fixed
+
+- **Type safety** — aligned with platform 2.1.2; all CI checks passing
+
+## [2.1.1] - 2026-03-02
+
+### Fixed
+
+- **Contract payload validation** - Fixed ruff errors in test_contract_payloads.py
+- **CI integration** - Resolved test suite compatibility issues
+- **Linting compliance** - Fixed SIM102, E501, SIM105, I001, SIM300 ruff violations
+
+### Changed
+
+- **Test coverage** - Improved test matrix for TypeScript/Python SDK compatibility
+- **Documentation** - Updated README with PyPI downloads badge and GitHub stars
+
+## [2.0.0] - 2026-03-01
+
+### Breaking
+
+- **Rebrand:** Package renamed `pauly4010-evalai-sdk` → `pauly4010-evalgate-sdk`, module `evalai_sdk` → `evalgate_sdk`
+- **CLI:** `evalai` → `evalgate`
+- **Config:** `.evalai/` → `.evalgate/` (legacy `.evalai/` still read, with deprecation warning)
+- **Env vars:** `EVALAI_*` → `EVALGATE_*` (legacy `EVALAI_*` still work, with deprecation warning)
+- **Error class:** `EvalAIError` → `EvalGateError`
+
+### Added
+
+- Deprecation warnings when using legacy env vars or config paths
+
+## [1.9.1] - 2026-03-01
+
+### Fixed
+
+- Align `SPEC_VERSION` with OpenAPI spec 1.9.1
+- Ruff lint and format fixes (SIM102, E501, SIM105, I001, SIM300)
+
+### Changed
+
+- Package metadata: Production/Stable status, improved description and keywords
+- README: Added PyPI downloads badge, GitHub stars, status section, changelog link
+
+## [1.9.0] - 2026-02-27
+
+### Added
+
+- Full parity with TypeScript SDK 1.9.0
+- `evalai ci` CLI command — one-command CI loop
+- Run artifact retention and diff system
+- Impact analysis integration
+- Schema versioning for run reports
+
+### Changed
+
+- Exit codes standardized: 0=clean, 1=regressions, 2=config/infra
+- CLI output improvements for CI environments
+
+## [1.0.1] - 2026-02-26
+
+### Fixed
+
+- CLI init template and credential resolution
+- Run output formatting
+
+## [1.0.0] - 2026-02-25
+
+### Added
+
+- Initial Python SDK release
+- `AIEvalClient` — async HTTP client for EvalGate API
+- 20+ assertions: `expect()`, `to_contain`, `to_not_contain_pii`, `to_be_professional`, etc.
+- Test suites: `create_test_suite`, `TestSuiteConfig`, `TestSuiteCase`
+- Integrations: OpenAI, Anthropic, LangChain, CrewAI, AutoGen tracing
+- Workflow tracing: `WorkflowTracer`, handoffs, cost tracking
+- Regression gates: `evaluate_regression`, `to_pass_gate`
+- CLI: `evalai init`, `evalai run`, `evalai gate`, `evalai ci`, `evalai doctor`
+- Batch processing, caching, pagination, structured errors
+- Full type hints (`py.typed`), mypy and Pyright compatible