zentest-sdk 0.4.0__tar.gz

zentest_sdk-0.4.0/CHANGELOG.md

@@ -0,0 +1,109 @@
+# Changelog
+
+All notable changes to `zentest-sdk`, the Python SDK for the Zentest platform. The SDK implements the
+[SDK contract v1](https://github.com/ITHyd/Zentest/blob/main/docs/specs/sdk-contract.md); every release so far is
+additive within v1, so code written against an earlier release keeps working.
+
+Versions follow [Semantic Versioning](https://semver.org/). Until 1.0, a minor release may add to the API but does not
+remove or change what the contract marks as public (the adapter protocol, the `Verdict` shape and runner parity).
+Release steps are in [RELEASING.md](https://github.com/ITHyd/Zentest/blob/main/sdk/RELEASING.md).
+
+## 0.4.0 (2026-10-19)
+
+The first release published to PyPI (`pip install zentest-sdk`), built, tested and uploaded by an Azure Pipelines
+release pipeline. Local evaluation: the SDK becomes a testing library
+you can use without the platform, and that records on it when you want. Everything in 0.3.0 is unchanged.
+
+### Added
+- `zentest.metrics`: a metrics library. Deterministic checks (`PIILeak`, `Forbidden`, `Contains`, `Regex`, `ExactMatch`,
+  `Format`, `ReadingLevel`, `Disclaimer`, `Escalation`, `ToolCorrectness`, `JudgeGaming`, and any platform check through
+  `Check`), deterministic retrieval metrics (`Groundedness`, `ContextPrecision`, `ContextRecall`) and judge metrics
+  (`Faithfulness`, `AnswerRelevancy`, `ContextRelevance` (the RAG triad), `Hallucination`, `Toxicity`, `Bias`, `GEval`).
+  Every score records `model_snapshot`, `prompt_hash` and `judge_version`.
+- `zentest.judges`: the judge for judge metrics is pluggable. `PlatformJudge` scores through the platform's gateway
+  (`POST /api/sdk/evaluate`), `LocalJudge` calls Claude (`pip install "zentest-sdk[judge]"`) or an OpenRouter model with
+  your key, and `HeuristicJudge` makes no model call. Pick one with `judge=` or `ZENTEST_JUDGE`.
+- `assert_test`, `evaluate` and `EvalReport`, plus a pytest plugin (auto-registered): a summary table,
+  `--zentest-report` JSON, `--zentest-judge`, and `--zentest-sync` to record the session as a platform run.
+- `zentest test FILE`: declarative test files (cases, assertions, a Python or HTTP target, `repeats`, `pass_rate`),
+  with `--junit`, `--json-file`, `--baseline` (a case that passed before and fails now exits 1) and `--sync`.
+  `zentest init` scaffolds a test file and a pytest file.
+- `zt.redteam.scan` and `zentest redteam`: generate the product's red-team suite from the platform's attack catalogue,
+  run it against your agent, and get the per-class and OWASP report.
+- `zentest.guard`: runtime guardrails (prompt injection, PII in the input, topic denylist and any deterministic metric
+  on the output) with `block`, `redact`, `raise` or `log`, recorded as `guardrail` spans.
+- `zentest.simulate`: seeded multi-turn conversations with a persona and a goal, scored per turn (the platform's
+  simulator script).
+- `zentest.sync`: record local evaluations as a platform run (`POST /api/sdk/suites` finds or creates the suite).
+
+## 0.3.0 (2026-10-17)
+
+Prepared for PyPI but not published; 0.1.0 to 0.3.0 were installed from the repository.
+
+### Added
+- `gates.run` takes the run options `start` already had: `connection_id` (the HTTP connection an endpoint run calls when
+  the environment has several), `corpus` with `corpus_data_source` (UK-31: the retrieval corpus, scanned when the run
+  starts so contaminated cases are skipped), `osa_scope` (UK-10) and `trigger` (`"api"` or `"ci"`). `connection_id`
+  with an in-process `agent` raises `ValueError` before anything is created.
+- `start` gained `connection_id` (several HTTP connections per environment; the run pins the one it calls) and `corpus`
+  with `corpus_data_source`.
+- `Verdict.contamination` (UK-31): the cases found in the app's retrieval corpus, which were excluded, kept or skipped.
+- The SDK runner also submits results on a time interval, not only every `batch_size` trials: `flush_every_s` on
+  `gates.run` and `Zentest(...)`, or `ZENTEST_FLUSH_EVERY_S` (default 30 s; `0` sends full batches only), so a slow
+  app's run shows progress in the console as it goes (SDK-07).
+- CLI: `zentest run --junit PATH` writes the result as JUnit XML for CI test reports; a suite file can pin a
+  `connection`; a metric with no counted trials prints "no counted trials" instead of 0%.
+- Packaging: published metadata (readme, licence, authors, classifiers, project URLs), a PEP 561 `py.typed` marker, this
+  changelog and a release workflow with PyPI trusted publishing.
+
+### Changed
+- The version is written once, in `zentest/_version.py`; `pyproject.toml`, `zentest.__version__`, the `User-Agent` and
+  the SDK-19 minimum-version check all read it.
+- The offline check library (`zentest/_checks.py`, a copy of the platform's) follows the platform: it gains the
+  special-category data check, and the platform-only RAG, language and attack checks are refused offline as a
+  configuration error instead of failing to import.
+
+## 0.2.0 (2026-10-15)
+
+Completes contract v1 (SDK-01 to SDK-20). Installed from the repository (`pip install -e sdk`).
+
+### Added
+- Tracing (SDK-15): `@zt.traced` and `with zt.span(...)` record zentest.span.v1 spans, one trace per trial, uploaded and
+  linked to the trial's result. A run whose agent records no spans warns when it starts (`gates.last_run.traced`).
+- OpenTelemetry (SDK-16): `zentest.otel` (extra `otel`) exports existing GenAI or OpenInference spans and stamps them
+  with the trial's ids.
+- Offline runs (SDK-08): `zentest export` and `zentest run --offline` score deterministic metrics with the platform's
+  own check library and decision rule; skipped judge metrics make the result `incomplete` (exit 2) unless
+  `--allow-incomplete`.
+- The zentest CLI (SDK-08, SDK-09): `zentest run gate.zt` gates a CI pipeline on a platform verdict (exit 0 passed,
+  1 not passed, 2 no verdict), with `--json`, `--json-file`, `--summary`, `zentest cancel`, the REQ-077 override chain
+  and OSA scope conditions (UK-10); a replayed idempotency key for another gate exits 2.
+- Minimum-version check (SDK-19): one `GET /api/version` before the first call; `VersionError` with the upgrade command.
+- Typed errors for every refusal: `ResidencyError`, `IdempotencyConflictError`, `MissingInputsError` (`.missing`),
+  `VersionError`, alongside the earlier ones.
+- Per-trial `usage` and `metadata` (SDK-04), cut down rather than refused when too large; `packs` recorded on the run
+  and returned as `Verdict.packs` (SDK-10); `cancel` returns a `CancelResult` (SDK-18).
+- Version attribution (AI-49): `AgentResponse(model=..., prompt_version=..., config_version=...)` and
+  `Verdict.attribution`.
+- Client methods for the catalogue and case authoring (`products`, `versions`, `suites`, `suite_cases`, `add_cases`,
+  `update_case`), `results`, `power`, `override`, `span_schema`, `upload_spans`, `instrumentation`, `export_suite`,
+  `submit_corpus` and `submit_session`.
+- Extras `otel`, `yaml` and `schema`; the core dependency stays `httpx` alone (SDK-20).
+- Examples: `quickstart.py` and `mock_app.py` report attribution; `http_app_gate.py` wraps a login-protected chat API.
+
+### Changed
+- `zt.wait()` accepts a `RunHandle` as well as a run id; `zt.cancel(run)` and `RunCancelledError` (SDK-17, SDK-18).
+
+## 0.1.0 (2026-10-13)
+
+First version, installed from the repository.
+
+### Added
+- `Zentest` client: API token and pinned residency region on every call, retries on 5xx and connection errors.
+- Adapter protocol (SDK-02 to SDK-05): the agent's shape is detected from its type hint (`str`, `list[dict]` or
+  `AgentInput`), sync or async.
+- The in-process SDK runner with batched, idempotent result submission, and the manual loop (`start`, `cases`,
+  `submit`, `finish`).
+- `gates.run` with required `repeats` and the stated `deterministic` mode (SDK-10 to SDK-13), the typed `Verdict` and
+  the contract errors.
+- Examples: `mock_app.py` and `quickstart.py` (both runners).

zentest_sdk-0.4.0/MANIFEST.in

@@ -0,0 +1,6 @@
+# The sdist is public (PyPI): the package (from pyproject.toml), the README (the PyPI page), the changelog and the
+# release notes. The tests and examples stay in the repository: they name non-public apps and deployments, and the
+# README links the examples on GitHub.
+include CHANGELOG.md RELEASING.md
+prune tests
+prune examples

zentest_sdk-0.4.0/PKG-INFO

@@ -0,0 +1,243 @@
+Metadata-Version: 2.4
+Name: zentest-sdk
+Version: 0.4.0
+Summary: Zentest SDK: test your AI app in-process and gate releases on a judged verdict (SDK contract v1)
+Author: Nxzen
+License-Expression: LicenseRef-Proprietary
+Project-URL: Homepage, https://github.com/ITHyd/Zentest
+Project-URL: Repository, https://github.com/ITHyd/Zentest
+Project-URL: Documentation, https://github.com/ITHyd/Zentest/blob/main/docs/specs/sdk-contract.md
+Project-URL: Changelog, https://github.com/ITHyd/Zentest/blob/main/sdk/CHANGELOG.md
+Project-URL: Issues, https://github.com/ITHyd/Zentest/issues
+Keywords: llm,evaluation,testing,ai,release-gate,rag,opentelemetry
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: httpx>=0.27
+Provides-Extra: otel
+Requires-Dist: opentelemetry-sdk>=1.20; extra == "otel"
+Provides-Extra: yaml
+Requires-Dist: pyyaml>=6; extra == "yaml"
+Provides-Extra: schema
+Requires-Dist: jsonschema>=4; extra == "schema"
+Provides-Extra: judge
+Requires-Dist: anthropic>=0.100; extra == "judge"
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: pyyaml>=6; extra == "dev"
+Requires-Dist: opentelemetry-sdk>=1.20; extra == "dev"
+Requires-Dist: jsonschema>=4; extra == "dev"
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: twine>=6.1; extra == "dev"
+Requires-Dist: packaging>=23; extra == "dev"
+
+# zentest-sdk
+
+The Python SDK for [Zentest](https://github.com/ITHyd/Zentest), Nxzen's platform for testing AI apps (chatbots, RAG,
+voice agents). Run your app's test suite in your own process, or let the platform call its HTTP endpoint, and gate the
+release on a judged verdict with a confidence interval. The behaviour is fixed by the
+[SDK contract v1](https://github.com/ITHyd/Zentest/blob/main/docs/specs/sdk-contract.md) (SDK-01 to SDK-20).
+A TypeScript SDK for Node 18+ implements the same contract in [`sdk-ts/`](https://github.com/ITHyd/Zentest/tree/main/sdk-ts)
+(contract §2b); offline runs and the CLI are Python only.
+You need a Zentest workspace and an API token (console: Settings → API tokens).
+
+**Links:** [contract](https://github.com/ITHyd/Zentest/blob/main/docs/specs/sdk-contract.md) ·
+[changelog](https://github.com/ITHyd/Zentest/blob/main/sdk/CHANGELOG.md) ·
+[CI gate guide](https://github.com/ITHyd/Zentest/blob/main/docs/specs/ci-gate.md) ·
+[span format](https://github.com/ITHyd/Zentest/blob/main/docs/specs/zentest-span-v1.md) ·
+[examples](https://github.com/ITHyd/Zentest/tree/main/sdk/examples)
+
+## Install
+
+Python 3.11 or later.
+
+```bash
+pip install zentest-sdk                 # core: httpx only
+pip install "zentest-sdk[otel]"         # + OpenTelemetry exporter (zentest.otel)
+pip install "zentest-sdk[yaml]"         # + YAML suite files for the CLI
+pip install "zentest-sdk[schema]"       # + offline runs of a format metric with a JSON Schema
+pip install "zentest-sdk[judge]"        # + the local Claude judge (open-core mode, no platform)
+```
+
+Install it into a virtual environment, then check it: `zentest --version` prints the installed version. Add only
+the extras you need.
+
+Working on the SDK itself? Install it from a checkout of the repository instead: `pip install -e sdk` (extras as
+`pip install -e "sdk[otel,yaml]"`, and `sdk[dev]` for the tests).
+
+**Dependencies (SDK-20).** The client is installed into your application's environment, so the floor is thin: `httpx`
+and nothing else. Evaluation libraries stay on the platform; the deterministic checks used by offline runs are standard
+library only. Extras: `otel` (`opentelemetry-sdk`), `yaml` (`pyyaml`), `schema` (`jsonschema`). The package ships type
+hints (`py.typed`).
+
+## Evaluate in your own tests (0.4.0)
+
+```python
+from zentest import LLMTestCase, assert_test
+from zentest.metrics import PIILeak, Faithfulness, AnswerRelevancy, GEval
+
+def test_burst_pipe():
+    case = LLMTestCase(input="Is a burst pipe covered?", actual_output=app(q), retrieval_context=chunks)
+    assert_test(case, [PIILeak(), Faithfulness(threshold=0.8), AnswerRelevancy(),
+                       GEval("Plain English", "Short answer a customer understands")])
+```
+
+- `pytest` prints a Zentest summary table; `--zentest-sync --zentest-product P --zentest-version V` records the session
+  as a platform run. Deterministic metrics always run before judge metrics.
+- Judges: `ZENTEST_JUDGE=platform` (the platform's calibrated judge, needs `ZENTEST_API_KEY`), `local` (Claude with
+  `ANTHROPIC_API_KEY`, or OpenRouter with `OPENROUTER_API_KEY`) or `heuristic` (no model call). A local judge is not
+  calibrated: gate releases on the platform.
+- `zentest init` then `zentest test zentest.yaml`: the same checks as a YAML file in Git (`--baseline`, `--junit`,
+  `--sync`).
+- `zentest redteam --product P --version V --agent app:answer`: the product's red-team suite against your agent.
+- `zentest.guard.Guard(...).protect(answer)`: block prompt injection and personal data in production.
+- `zentest.simulate(answer, persona=..., goal=...)`: a seeded multi-turn conversation, scored per turn.
+
+A worked example on a real app lives in the repository under `sdk/examples` (it runs with no network in mock mode).
+
+## Quick start (release gate)
+
+```python
+from zentest import Zentest, AgentInput, AgentResponse
+
+def agent(x: AgentInput) -> AgentResponse:        # or (user_message: str) -> str, or (messages: list[dict]) -> str
+    reply, chunks = my_app.answer(x.messages[-1]["content"])
+    return AgentResponse(output=reply, retrieval_context=chunks,
+                         model="claude-sonnet-4.5", prompt_version="r12", config_version="v6")
+
+with Zentest(api_key="zt_live_…", base_url="http://localhost:8620", region="uk-south") as zt:
+    v = zt.gates.run("Claims Copilot", "v6", "Claims FAQ Golden Set", agent, repeats=3, deterministic=True)
+print(v.passed, v.reason, v.interval)
+```
+
+**Arguments**
+- `repeats`: required, with no default. A single run gives no interval.
+- `deterministic=True`: you are stating that the app runs with temperature 0 and a fixed seed. Without it the platform
+  refuses (`PrerequisiteError`).
+- `agent=None`: the platform calls the product's HTTP connection instead of your code (the endpoint runner, below).
+- `batch_size=50`, `flush_every_s=None`: results go to the platform every `batch_size` trials, or sooner once
+  `flush_every_s` seconds pass without a submission, so a slow app's run shows progress and time left in the console
+  as it goes. `None` takes the client's (`Zentest(flush_every_s=...)`, else `ZENTEST_FLUSH_EVERY_S`, else 30 s); `0`
+  sends full batches only.
+- `packs=[...]`: compliance pack ids from the workspace's catalogue (`eu-ai-act`, `owasp-llm`, …; any other id is a
+  422), recorded on the run and returned as `v.packs`.
+- `connection_id="…"` (with `agent=None`): the HTTP connection the platform calls when the environment has several;
+  without it, the environment's default connection, else its only one.
+- `corpus=[...]`, `corpus_data_source="synthetic" | "masked_production" | "production"`: your app's retrieval corpus
+  (chunks as strings or `{"text", "id"}`), scanned when the run starts. A test case found in it is skipped and reported on
+  `v.contamination`; too many and the run is refused (UK-31).
+- `osa_scope={...}`: the three Online Safety Act scope conditions for this release; a change comes back as
+  `zt.gates.last_run.scope_alert` (UK-10). `trigger="ci"` marks a run started from a pipeline.
+- `timeout=600`: seconds to wait for the verdict before `ZentestTimeout`.
+
+**Version attribution ([contract §2a](https://github.com/ITHyd/Zentest/blob/main/docs/specs/sdk-contract.md)):** return `AgentResponse(output=…, model=…, prompt_version=…, config_version=…)`
+from a shape C agent so every trial says what produced it. The values must equal what is registered on the version
+under test. Attribution is required by default: on a product created from P2a on, a trial that reports nothing does not
+count, and a run where nothing is reported does not pass (the verdict's reason says what to report). Products that
+predate it keep counting such trials, as *declared*, until the owner turns attribution required on;
+`v.attribution["coverage"]` shows the share the app reported. `usage` and `metadata` on the response are stored with
+each trial too (credential-named keys such as `api_key`, `accessToken` or `clientSecret` as `[REDACTED]`, personal data
+redacted like any output). They are optional and never fail a trial: nested usage (OpenAI's `prompt_tokens_details`)
+is sent flat as `prompt_tokens_details.cached_tokens`, at most 32 values, and metadata over 16 KB as JSON is trimmed
+key by key; each cut is a warning on `zt.gates.last_run.warnings`.
+
+**Tracing (SDK-15):** `@zt.traced` / `with zt.span(name, kind=...)` record steps (tool calls, retrievals, LLM calls,
+planning) as zentest.span.v1 spans. The SDK runner uploads one trace per trial and links it to the trial's result
+(`zt.results(run)` gives `trace_id` and `session_id`; a trial whose spans the platform did not store has no
+`trace_id`, and the run handle says which). Without spans the run still works, but the SDK warns when the run starts
+that trajectory and tool-choice metrics are unavailable, and `zt.gates.last_run.traced` is `False`.
+
+```python
+@zt.traced(kind="tool", name="lookup_claim")
+def lookup(claim_id: str) -> dict: ...
+
+with zt.span("answer", kind="llm", region="uk-south") as s:
+    s.set_usage(input_tokens=120, output_tokens=30, provider="anthropic")
+    s.set_output(text)
+```
+
+**Already on OpenTelemetry (SDK-16):** keep your instrumentation (GenAI `gen_ai.*` or OpenInference) and add
+`zentest.otel.install(zt, product="Claims Copilot", version="v6")`. Spans started inside a trial are stamped with the
+trial's ids. A collector can instead export to `POST /v1/traces` with the API token and `X-Zentest-Product` (or the
+resource attribute `zentest.product`), as **OTLP/JSON only**: set `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/json`
+(protobuf, the exporters' default, gets 415). `/v1/traces` is outside `/api`, so a proxy or ingress needs a route for
+it; see [zentest-span-v1 §4a](https://github.com/ITHyd/Zentest/blob/main/docs/specs/zentest-span-v1.md).
+
+**Endpoint runner (no code on your side):** the platform calls the product's HTTP connection (Connections in the
+console, per environment). For the mock app:
+
+| Field | Value | Why |
+|---|---|---|
+| endpoint | `http://127.0.0.1:8100/chat` | plain http only to localhost |
+| secret_ref | `env://MOCK_APP_KEY` (dev) or `kv://<vault>/<name>` | the platform stores a reference, never the key |
+| request_template | `{"message": "{{input}}", "session": "zt"}` | |
+| output_path | `$.reply.text` | the answer |
+| context_path | `$.sources[*].text` | **required** when the suite has a context-grounded judge metric (Faithfulness): without it the run is refused before it starts (NEW-43, `MissingInputsError`) |
+| model_path, prompt_version_path, config_version_path | `$.meta.model`, `$.meta.prompt_version`, `$.meta.config_version` | attribution (§2a); a new product does not count unattributed trials |
+
+**Manual loop:** `run = zt.start(...)`, then `zt.submit(run, case, output)` for each case in `zt.cases(run)`, then
+`zt.finish(run)`. `zt.cancel(run)` stops a run and returns what completed (`trials_received`, kept).
+
+**Catalogue and authoring:** `zt.products()`, `zt.versions(product)`, `zt.suites(product)`, `zt.suite_cases(suite_id)`,
+`zt.add_cases(suite_id, [...])`, `zt.update_case(case_id, input=…)`, `zt.results(run)`, `zt.span_schema()`,
+`zt.upload_spans(product, spans, otlp=…)`, `zt.instrumentation(product)`, `zt.export_suite(product, version, suite)`,
+`zt.submit_corpus(run, chunks, data_source=…)` (UK-31 corpus scan before gating) and
+`zt.submit_session(product, session_id, input, transcript=…)` (REQ-066 production sessions, a `sessions:write` token).
+
+**Errors:** `PrerequisiteError`, `CalibrationError`, `ResidencyError`, `IdempotencyConflictError`,
+`MissingInputsError` (`.missing` lists each gap), `VersionError`, `RunFailedError`, `RunCancelledError`, `AuthError`,
+`NotFoundError`, `ZentestTimeout`; all are `ZentestError`.
+
+**Minimum version (SDK-19):** the client reads `GET /api/version` once before its first call and raises `VersionError`
+with the upgrade command when it is older than the platform's `min_sdk_version`.
+
+**CLI** (`zentest`, [CI gate guide](https://github.com/ITHyd/Zentest/blob/main/docs/specs/ci-gate.md)): `zentest run gate.zt` (a platform gate in CI), `zentest cancel RUN_ID`, and
+offline (SDK-08):
+
+```bash
+zentest export --product "Claims Copilot" --version v6 --suite "Claims FAQ Golden Set" -o claims.ztx
+zentest run --offline claims.ztx --agent my_app.gate:agent --repeats 3      # exit 0 passed, 1 not passed, 2 error
+                                                                            #   or incomplete
+```
+
+The export holds cases, metric configs and rubrics, never a connection or secret. It does hold every case input, and
+once written it is outside the platform's retention and erasure: keep it where the suite's data may live, delete it
+after the job, and export again after an erasure. Offline, deterministic metrics are scored with the platform's own
+check library and the platform's decision rule. **Offline mode cannot** score judge metrics (they need the platform's
+judge gateway: calibration, family guard, residency), apply the rubric guards, span-level checks, the power check or
+the override chain, or produce an evidence pack; nothing is uploaded.
+
+**Incomplete runs exit 2 by default.** When judge metrics were skipped the result is `status: "incomplete"` (exit 2):
+a deterministic-only pass is not a release verdict. A definite not-pass still exits 1 (blocked by a gating
+deterministic metric, or not passed with every trial scored). `--allow-incomplete` accepts the deterministic-only
+result (exit 0/1). A trial with no metric scorable offline is not passed, and a suite with no deterministic metric has
+no offline verdict (`passed: null`, exit 2 even with `--allow-incomplete`). `--strict` is the default and still
+accepted.
+
+**Examples**
+- [`examples/mock_app.py`](https://github.com/ITHyd/Zentest/blob/main/sdk/examples/mock_app.py): a local stand-in app that reports attribution (`meta`).
+- [`examples/quickstart.py`](https://github.com/ITHyd/Zentest/blob/main/sdk/examples/quickstart.py): runs it through both runners, with tracing.
+- [`examples/http_app_gate.py`](https://github.com/ITHyd/Zentest/blob/main/sdk/examples/http_app_gate.py): wraps any login-protected chat API as an SDK agent from environment variables (logs in,
+  caches and refreshes the JWT, threads conversation ids, reads answer and context with JSONPaths, reports attribution).
+  `python sdk/examples/http_app_gate.py --smoke "question"` checks the configuration with one call.
+
+**Tests** (from a checkout of the repository): `cd sdk && python3 -m pytest -q tests` (no network:
+httpx.MockTransport; the span and OTel tests also run the platform's own validator from `backend/` when it is
+importable). `pip install -e "sdk[dev]"` adds `build` and `twine`, which the packaging tests use to build and check the
+release artifacts.
+
+## Versions and licence
+
+`zentest.__version__` is the installed version; [CHANGELOG.md](https://github.com/ITHyd/Zentest/blob/main/sdk/CHANGELOG.md)
+lists what each release changed. Releases are additive within contract v1, and the platform tells an SDK that is too
+old to upgrade (`VersionError`, above). Copyright Nxzen; proprietary licence (see the package metadata).

zentest_sdk-0.4.0/README.md

@@ -0,0 +1,200 @@
+# zentest-sdk
+
+The Python SDK for [Zentest](https://github.com/ITHyd/Zentest), Nxzen's platform for testing AI apps (chatbots, RAG,
+voice agents). Run your app's test suite in your own process, or let the platform call its HTTP endpoint, and gate the
+release on a judged verdict with a confidence interval. The behaviour is fixed by the
+[SDK contract v1](https://github.com/ITHyd/Zentest/blob/main/docs/specs/sdk-contract.md) (SDK-01 to SDK-20).
+A TypeScript SDK for Node 18+ implements the same contract in [`sdk-ts/`](https://github.com/ITHyd/Zentest/tree/main/sdk-ts)
+(contract §2b); offline runs and the CLI are Python only.
+You need a Zentest workspace and an API token (console: Settings → API tokens).
+
+**Links:** [contract](https://github.com/ITHyd/Zentest/blob/main/docs/specs/sdk-contract.md) ·
+[changelog](https://github.com/ITHyd/Zentest/blob/main/sdk/CHANGELOG.md) ·
+[CI gate guide](https://github.com/ITHyd/Zentest/blob/main/docs/specs/ci-gate.md) ·
+[span format](https://github.com/ITHyd/Zentest/blob/main/docs/specs/zentest-span-v1.md) ·
+[examples](https://github.com/ITHyd/Zentest/tree/main/sdk/examples)
+
+## Install
+
+Python 3.11 or later.
+
+```bash
+pip install zentest-sdk                 # core: httpx only
+pip install "zentest-sdk[otel]"         # + OpenTelemetry exporter (zentest.otel)
+pip install "zentest-sdk[yaml]"         # + YAML suite files for the CLI
+pip install "zentest-sdk[schema]"       # + offline runs of a format metric with a JSON Schema
+pip install "zentest-sdk[judge]"        # + the local Claude judge (open-core mode, no platform)
+```
+
+Install it into a virtual environment, then check it: `zentest --version` prints the installed version. Add only
+the extras you need.
+
+Working on the SDK itself? Install it from a checkout of the repository instead: `pip install -e sdk` (extras as
+`pip install -e "sdk[otel,yaml]"`, and `sdk[dev]` for the tests).
+
+**Dependencies (SDK-20).** The client is installed into your application's environment, so the floor is thin: `httpx`
+and nothing else. Evaluation libraries stay on the platform; the deterministic checks used by offline runs are standard
+library only. Extras: `otel` (`opentelemetry-sdk`), `yaml` (`pyyaml`), `schema` (`jsonschema`). The package ships type
+hints (`py.typed`).
+
+## Evaluate in your own tests (0.4.0)
+
+```python
+from zentest import LLMTestCase, assert_test
+from zentest.metrics import PIILeak, Faithfulness, AnswerRelevancy, GEval
+
+def test_burst_pipe():
+    case = LLMTestCase(input="Is a burst pipe covered?", actual_output=app(q), retrieval_context=chunks)
+    assert_test(case, [PIILeak(), Faithfulness(threshold=0.8), AnswerRelevancy(),
+                       GEval("Plain English", "Short answer a customer understands")])
+```
+
+- `pytest` prints a Zentest summary table; `--zentest-sync --zentest-product P --zentest-version V` records the session
+  as a platform run. Deterministic metrics always run before judge metrics.
+- Judges: `ZENTEST_JUDGE=platform` (the platform's calibrated judge, needs `ZENTEST_API_KEY`), `local` (Claude with
+  `ANTHROPIC_API_KEY`, or OpenRouter with `OPENROUTER_API_KEY`) or `heuristic` (no model call). A local judge is not
+  calibrated: gate releases on the platform.
+- `zentest init` then `zentest test zentest.yaml`: the same checks as a YAML file in Git (`--baseline`, `--junit`,
+  `--sync`).
+- `zentest redteam --product P --version V --agent app:answer`: the product's red-team suite against your agent.
+- `zentest.guard.Guard(...).protect(answer)`: block prompt injection and personal data in production.
+- `zentest.simulate(answer, persona=..., goal=...)`: a seeded multi-turn conversation, scored per turn.
+
+A worked example on a real app lives in the repository under `sdk/examples` (it runs with no network in mock mode).
+
+## Quick start (release gate)
+
+```python
+from zentest import Zentest, AgentInput, AgentResponse
+
+def agent(x: AgentInput) -> AgentResponse:        # or (user_message: str) -> str, or (messages: list[dict]) -> str
+    reply, chunks = my_app.answer(x.messages[-1]["content"])
+    return AgentResponse(output=reply, retrieval_context=chunks,
+                         model="claude-sonnet-4.5", prompt_version="r12", config_version="v6")
+
+with Zentest(api_key="zt_live_…", base_url="http://localhost:8620", region="uk-south") as zt:
+    v = zt.gates.run("Claims Copilot", "v6", "Claims FAQ Golden Set", agent, repeats=3, deterministic=True)
+print(v.passed, v.reason, v.interval)
+```
+
+**Arguments**
+- `repeats`: required, with no default. A single run gives no interval.
+- `deterministic=True`: you are stating that the app runs with temperature 0 and a fixed seed. Without it the platform
+  refuses (`PrerequisiteError`).
+- `agent=None`: the platform calls the product's HTTP connection instead of your code (the endpoint runner, below).
+- `batch_size=50`, `flush_every_s=None`: results go to the platform every `batch_size` trials, or sooner once
+  `flush_every_s` seconds pass without a submission, so a slow app's run shows progress and time left in the console
+  as it goes. `None` takes the client's (`Zentest(flush_every_s=...)`, else `ZENTEST_FLUSH_EVERY_S`, else 30 s); `0`
+  sends full batches only.
+- `packs=[...]`: compliance pack ids from the workspace's catalogue (`eu-ai-act`, `owasp-llm`, …; any other id is a
+  422), recorded on the run and returned as `v.packs`.
+- `connection_id="…"` (with `agent=None`): the HTTP connection the platform calls when the environment has several;
+  without it, the environment's default connection, else its only one.
+- `corpus=[...]`, `corpus_data_source="synthetic" | "masked_production" | "production"`: your app's retrieval corpus
+  (chunks as strings or `{"text", "id"}`), scanned when the run starts. A test case found in it is skipped and reported on
+  `v.contamination`; too many and the run is refused (UK-31).
+- `osa_scope={...}`: the three Online Safety Act scope conditions for this release; a change comes back as
+  `zt.gates.last_run.scope_alert` (UK-10). `trigger="ci"` marks a run started from a pipeline.
+- `timeout=600`: seconds to wait for the verdict before `ZentestTimeout`.
+
+**Version attribution ([contract §2a](https://github.com/ITHyd/Zentest/blob/main/docs/specs/sdk-contract.md)):** return `AgentResponse(output=…, model=…, prompt_version=…, config_version=…)`
+from a shape C agent so every trial says what produced it. The values must equal what is registered on the version
+under test. Attribution is required by default: on a product created from P2a on, a trial that reports nothing does not
+count, and a run where nothing is reported does not pass (the verdict's reason says what to report). Products that
+predate it keep counting such trials, as *declared*, until the owner turns attribution required on;
+`v.attribution["coverage"]` shows the share the app reported. `usage` and `metadata` on the response are stored with
+each trial too (credential-named keys such as `api_key`, `accessToken` or `clientSecret` as `[REDACTED]`, personal data
+redacted like any output). They are optional and never fail a trial: nested usage (OpenAI's `prompt_tokens_details`)
+is sent flat as `prompt_tokens_details.cached_tokens`, at most 32 values, and metadata over 16 KB as JSON is trimmed
+key by key; each cut is a warning on `zt.gates.last_run.warnings`.
+
+**Tracing (SDK-15):** `@zt.traced` / `with zt.span(name, kind=...)` record steps (tool calls, retrievals, LLM calls,
+planning) as zentest.span.v1 spans. The SDK runner uploads one trace per trial and links it to the trial's result
+(`zt.results(run)` gives `trace_id` and `session_id`; a trial whose spans the platform did not store has no
+`trace_id`, and the run handle says which). Without spans the run still works, but the SDK warns when the run starts
+that trajectory and tool-choice metrics are unavailable, and `zt.gates.last_run.traced` is `False`.
+
+```python
+@zt.traced(kind="tool", name="lookup_claim")
+def lookup(claim_id: str) -> dict: ...
+
+with zt.span("answer", kind="llm", region="uk-south") as s:
+    s.set_usage(input_tokens=120, output_tokens=30, provider="anthropic")
+    s.set_output(text)
+```
+
+**Already on OpenTelemetry (SDK-16):** keep your instrumentation (GenAI `gen_ai.*` or OpenInference) and add
+`zentest.otel.install(zt, product="Claims Copilot", version="v6")`. Spans started inside a trial are stamped with the
+trial's ids. A collector can instead export to `POST /v1/traces` with the API token and `X-Zentest-Product` (or the
+resource attribute `zentest.product`), as **OTLP/JSON only**: set `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/json`
+(protobuf, the exporters' default, gets 415). `/v1/traces` is outside `/api`, so a proxy or ingress needs a route for
+it; see [zentest-span-v1 §4a](https://github.com/ITHyd/Zentest/blob/main/docs/specs/zentest-span-v1.md).
+
+**Endpoint runner (no code on your side):** the platform calls the product's HTTP connection (Connections in the
+console, per environment). For the mock app:
+
+| Field | Value | Why |
+|---|---|---|
+| endpoint | `http://127.0.0.1:8100/chat` | plain http only to localhost |
+| secret_ref | `env://MOCK_APP_KEY` (dev) or `kv://<vault>/<name>` | the platform stores a reference, never the key |
+| request_template | `{"message": "{{input}}", "session": "zt"}` | |
+| output_path | `$.reply.text` | the answer |
+| context_path | `$.sources[*].text` | **required** when the suite has a context-grounded judge metric (Faithfulness): without it the run is refused before it starts (NEW-43, `MissingInputsError`) |
+| model_path, prompt_version_path, config_version_path | `$.meta.model`, `$.meta.prompt_version`, `$.meta.config_version` | attribution (§2a); a new product does not count unattributed trials |
+
+**Manual loop:** `run = zt.start(...)`, then `zt.submit(run, case, output)` for each case in `zt.cases(run)`, then
+`zt.finish(run)`. `zt.cancel(run)` stops a run and returns what completed (`trials_received`, kept).
+
+**Catalogue and authoring:** `zt.products()`, `zt.versions(product)`, `zt.suites(product)`, `zt.suite_cases(suite_id)`,
+`zt.add_cases(suite_id, [...])`, `zt.update_case(case_id, input=…)`, `zt.results(run)`, `zt.span_schema()`,
+`zt.upload_spans(product, spans, otlp=…)`, `zt.instrumentation(product)`, `zt.export_suite(product, version, suite)`,
+`zt.submit_corpus(run, chunks, data_source=…)` (UK-31 corpus scan before gating) and
+`zt.submit_session(product, session_id, input, transcript=…)` (REQ-066 production sessions, a `sessions:write` token).
+
+**Errors:** `PrerequisiteError`, `CalibrationError`, `ResidencyError`, `IdempotencyConflictError`,
+`MissingInputsError` (`.missing` lists each gap), `VersionError`, `RunFailedError`, `RunCancelledError`, `AuthError`,
+`NotFoundError`, `ZentestTimeout`; all are `ZentestError`.
+
+**Minimum version (SDK-19):** the client reads `GET /api/version` once before its first call and raises `VersionError`
+with the upgrade command when it is older than the platform's `min_sdk_version`.
+
+**CLI** (`zentest`, [CI gate guide](https://github.com/ITHyd/Zentest/blob/main/docs/specs/ci-gate.md)): `zentest run gate.zt` (a platform gate in CI), `zentest cancel RUN_ID`, and
+offline (SDK-08):
+
+```bash
+zentest export --product "Claims Copilot" --version v6 --suite "Claims FAQ Golden Set" -o claims.ztx
+zentest run --offline claims.ztx --agent my_app.gate:agent --repeats 3      # exit 0 passed, 1 not passed, 2 error
+                                                                            #   or incomplete
+```
+
+The export holds cases, metric configs and rubrics, never a connection or secret. It does hold every case input, and
+once written it is outside the platform's retention and erasure: keep it where the suite's data may live, delete it
+after the job, and export again after an erasure. Offline, deterministic metrics are scored with the platform's own
+check library and the platform's decision rule. **Offline mode cannot** score judge metrics (they need the platform's
+judge gateway: calibration, family guard, residency), apply the rubric guards, span-level checks, the power check or
+the override chain, or produce an evidence pack; nothing is uploaded.
+
+**Incomplete runs exit 2 by default.** When judge metrics were skipped the result is `status: "incomplete"` (exit 2):
+a deterministic-only pass is not a release verdict. A definite not-pass still exits 1 (blocked by a gating
+deterministic metric, or not passed with every trial scored). `--allow-incomplete` accepts the deterministic-only
+result (exit 0/1). A trial with no metric scorable offline is not passed, and a suite with no deterministic metric has
+no offline verdict (`passed: null`, exit 2 even with `--allow-incomplete`). `--strict` is the default and still
+accepted.
+
+**Examples**
+- [`examples/mock_app.py`](https://github.com/ITHyd/Zentest/blob/main/sdk/examples/mock_app.py): a local stand-in app that reports attribution (`meta`).
+- [`examples/quickstart.py`](https://github.com/ITHyd/Zentest/blob/main/sdk/examples/quickstart.py): runs it through both runners, with tracing.
+- [`examples/http_app_gate.py`](https://github.com/ITHyd/Zentest/blob/main/sdk/examples/http_app_gate.py): wraps any login-protected chat API as an SDK agent from environment variables (logs in,
+  caches and refreshes the JWT, threads conversation ids, reads answer and context with JSONPaths, reports attribution).
+  `python sdk/examples/http_app_gate.py --smoke "question"` checks the configuration with one call.
+
+**Tests** (from a checkout of the repository): `cd sdk && python3 -m pytest -q tests` (no network:
+httpx.MockTransport; the span and OTel tests also run the platform's own validator from `backend/` when it is
+importable). `pip install -e "sdk[dev]"` adds `build` and `twine`, which the packaging tests use to build and check the
+release artifacts.
+
+## Versions and licence
+
+`zentest.__version__` is the installed version; [CHANGELOG.md](https://github.com/ITHyd/Zentest/blob/main/sdk/CHANGELOG.md)
+lists what each release changed. Releases are additive within contract v1, and the platform tells an SDK that is too
+old to upgrade (`VersionError`, above). Copyright Nxzen; proprietary licence (see the package metadata).