agent-reliability 1.2.1__tar.gz → 1.3.0__tar.gz

{agent_reliability-1.2.1 → agent_reliability-1.3.0}/.github/workflows/ci.yml

@@ -17,6 +17,7 @@ jobs:
   quality:
     name: Lint, format, and type-check
     runs-on: ubuntu-latest
+    timeout-minutes: 30
     steps:
       - name: Checkout
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
@@ -28,7 +29,7 @@ jobs:
           cache: "pip"
 
       - name: Install package with dev dependencies
-        run: python -m pip install -e ".[dev]"
+        run: python -m pip install -e ".[dev,http]"
 
       - name: Lint (ruff check)
         run: ruff check .
@@ -42,6 +43,7 @@ jobs:
   test:
     name: Test stable contracts (py${{ matrix.python-version }})
     runs-on: ubuntu-latest
+    timeout-minutes: 30
     strategy:
       fail-fast: false
       matrix:
@@ -57,7 +59,10 @@ jobs:
           cache: "pip"
 
       - name: Install package with test dependencies
-        run: python -m pip install -e ".[dev,otel-test]"
+        run: python -m pip install -e ".[dev,otel-test,http]"
+
+      - name: Verify P28 fixture manifest
+        run: python scripts/verify_p28_fixtures.py
 
       - name: Test with branch coverage
         run: python -m pytest --cov --cov-report=term-missing
@@ -65,6 +70,7 @@ jobs:
   artifacts:
     name: Release artifacts, extras, examples, and typing
     runs-on: ubuntu-latest
+    timeout-minutes: 30
     steps:
       - name: Checkout
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
@@ -82,3 +88,20 @@ jobs:
 
       - name: Verify distributed artifacts
         run: python scripts/verify_release_artifacts.py
+
+  reference-vectors:
+    name: Independent Node reference check of the P28 contract fixtures
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - name: Checkout
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+
+      - name: Set up Node.js
+        uses: actions/setup-node@2028fbc5c25fe9cf00d9f06a71cc4710d4507903 # v6.0.0
+        with:
+          node-version: "24.19.0"
+
+      # Data-only: reads tests/fixtures/p28 with no Python import and no npm dependency.
+      - name: Verify canonical, digest and acknowledgement vectors
+        run: node scripts/verify_p28_reference.mjs

{agent_reliability-1.2.1 → agent_reliability-1.3.0}/.github/workflows/release.yml

@@ -15,9 +15,20 @@ jobs:
   build:
     name: Build and verify release artifacts
     runs-on: ubuntu-latest
+    timeout-minutes: 30
     steps:
       - name: Checkout
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          fetch-depth: 0
+
+      - name: Confirm the tagged commit is on main
+        run: |
+          git fetch --no-tags origin main
+          if ! git merge-base --is-ancestor "$GITHUB_SHA" origin/main; then
+            echo "Tagged commit $GITHUB_SHA is not reachable from origin/main" >&2
+            exit 1
+          fi
 
       - name: Set up Python 3.11
         uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
@@ -35,7 +46,7 @@ jobs:
           fi
 
       - name: Install package and build tooling
-        run: python -m pip install -e ".[dev,otel-test]" build
+        run: python -m pip install -e ".[dev,http,otel-test]" build
 
       - name: Full quality gates
         run: |
@@ -61,6 +72,7 @@ jobs:
     name: Publish to PyPI
     needs: build
     runs-on: ubuntu-latest
+    timeout-minutes: 15
     environment:
       name: pypi
       url: https://pypi.org/project/agent-reliability/
@@ -80,6 +92,7 @@ jobs:
     name: Create GitHub release
     needs: publish
     runs-on: ubuntu-latest
+    timeout-minutes: 15
     permissions:
       contents: write
     steps:

{agent_reliability-1.2.1 → agent_reliability-1.3.0}/CHANGELOG.md

@@ -4,6 +4,105 @@ All notable changes to this project are documented here. Format loosely
 follows [Keep a Changelog](https://keepachangelog.com/); versioning
 follows [docs/COMPATIBILITY.md](docs/COMPATIBILITY.md).
 
+## [Unreleased]
+
+## [1.3.0]
+
+A backward-compatible minor release under [docs/VERSIONING.md](docs/VERSIONING.md): an
+optional HTTP evidence exporter and an experimental model-backed evaluator adapter. The
+publication date is the upload date shown by the package index and the GitHub release. The
+`1.3.0rc1` pre-release is superseded by this release; its published files are unchanged. The core installation still has no
+runtime dependencies, and existing stable public APIs, reliability/SLO semantics and the
+`EventSink` contract are unchanged. See [docs/RELEASE_NOTES_1_3.md](docs/RELEASE_NOTES_1_3.md)
+for migration and rollback guidance and [docs/HTTP_EXPORTER.md](docs/HTTP_EXPORTER.md) for
+the exporter.
+
+### Added (experimental)
+
+- `agent_reliability.experimental.judge`: an EXPERIMENTAL adapter for model-backed (LLM-judge)
+  evaluators that call a provider you supply. Raw judged content stays on your side:
+  only typed members (outcome, reason code, provenance, failure class, health) reach the
+  exporter; no new wire field and no new core dependency. Reference rubrics are uncalibrated
+  examples; no accuracy or calibration claim is made. It may change in a minor release with
+  release notes (see the Experimental policy in docs/VERSIONING.md).
+
+### Added
+
+- Optional `agent-reliability[http]` extra (`httpx>=0.28.1,<0.29`) and the
+  `agent_reliability.adapters.http_event_sink` module with `HTTPEventSink`
+  (thread-safe, two daemon delivery workers) and `AsyncHTTPEventSink`. Both export
+  evidence to one complete platform POST URL using a machine credential obtained
+  from a caller-supplied provider, always as wire schema version 2 with the asserted
+  producer identity and a canonical digest that preserves the schema-v1 bytes.
+- Delivery and diagnostics types: `DeliveryStatusSnapshot`, `DeliveryDisposition`,
+  `DeliveryRecordStatus`, `ConfigurationError`, `ConfigurationFailedError`,
+  `EventDeliveryRejected`, `ExporterClosed`. `emit` only admits to bounded memory;
+  commitment is reported solely from a validated acknowledgement (`201`/`200`), and
+  records that may have committed are flagged.
+- Reliability behavior: bounded queues with a closure reserve, per-run ordering,
+  bounded retries with full jitter, sticky uncertainty for requests that may have
+  left the process, sink-wide `429`/`Retry-After` handling, bounded `flush`/`close`,
+  and explicit pauses with `resume_auth()`, `resume_schema()`, `resume_deployment()`
+  and `resume_endpoint()` for authentication, compatibility, unknown Deployment and
+  endpoint problems. Held evidence is never silently dropped.
+- Optional `HealthEvidenceSink` capability (`agent_reliability.ports.event_sink`),
+  fanned out by the composite sink, plus an optional `indicator=` argument on
+  `RunHandle.record_evaluation_failure(...)`, so evaluation failures and final
+  measurement health reach exporters. Existing `EventSink` implementations stay valid.
+- Run lifecycle integration: when the configured sink supports the optional
+  capability, `AgentReliability` now submits the final measurement-health report and
+  releases the exporter's run binding when a run finishes; `CompositeEventSink` fans the
+  capability out and isolates each child. Sinks that do not implement it are
+  unaffected.
+- **Fail-closed indicator policy.** The `indicator` is the only free-text wire field, so
+  both sinks now require an explicit choice: `indicator_allowlist=[...]` (only these exact
+  names may leave the process; others are rejected before queueing and never logged) or the
+  deliberate `allow_unrestricted_indicators=True`. A sink given neither raises
+  `ConfigurationError`. This is a behaviour change relative to the `1.3.0rc1` pre-release;
+  see the migration notes in [docs/RELEASE_NOTES_1_3.md](docs/RELEASE_NOTES_1_3.md).
+- Security posture: certificate and hostname verification are always on; redirects
+  are never followed; plaintext HTTP is accepted only for literal loopback
+  (`127.0.0.0/8`, `[::1]`); ambient proxy/CA/cookie/netrc configuration is ignored;
+  explicit `proxy` and `ca_bundle` options; credentials, URLs and evidence never
+  appear in diagnostics.
+- Data-only exporter contract fixtures under `tests/fixtures/`, with a Python verifier
+  (`scripts/verify_p28_fixtures.py`) and a Node reference check
+  (`scripts/verify_p28_reference.mjs`).
+
+### Fixed
+
+- Before this exporter is released, its TLS handling was corrected: with the real
+  `httpx`/`httpcore` stack a certificate-verification failure is visible only through
+  the exception's `__context__`, which the first implementation did not follow, so
+  repeated verification failures exhausted retries instead of pausing the endpoint. The
+  classifier now walks `__cause__` and `__context__` (bounded, cycle-safe) and
+  recognizes a verification failure only for an attempt that had not yet sent a
+  request. The third consecutive verification failure pauses endpoint delivery and
+  holds the evidence for `resume_endpoint()`; other network failures keep their ordinary
+  classification and neither count nor reset the streak (any completed HTTP exchange
+  or `resume_endpoint()` resets it). Through an explicit `proxy`, the tunnel-opening
+  CONNECT request no longer counts as having sent the evidence request. A response body
+  trickled past 30 seconds is abandoned and retried instead of pinning a worker.
+
+### Changed
+
+- Documentation: the exporter classification is recorded in `docs/GA_CONTRACT.md`
+  (see that file for the authoritative class; the documented consumer entry points are
+  named in `docs/HTTP_EXPORTER.md`).
+- The source distribution no longer includes `CLAUDE.md`, `AGENTS.md` or the internal
+  internal review/execution reports. `verify_release_artifacts.py` now also fails if either
+  artifact contains private-key material (PEM/PGP blocks, key-like file names), local developer
+  paths (Unix, macOS temp and Windows forms), or internal instruction/report files, and it is
+  covered by planted-case tests. Public documentation, the licence and all packaging inputs remain.
+- Idle exporter workers no longer build a full status snapshot every 10 ms (idle cost
+  with 900 history rows fell from about 2.7% to about 0.4% of one core).
+- CI and release workflows and `CONTRIBUTING.md` install the `http` extra so the
+  exporter tests and type checks run.
+- Release validation (`scripts/verify_release_artifacts.py`) also fails if either
+  artifact contains private-key material. The real-TLS regression tests generate
+  disposable certificates at test time (development-only dependency `cryptography`
+  in the `dev` extra); no key is stored in the repository or shipped.
+
 ## [1.2.1] - 2026-09-08
 
 Documentation-only patch release. No production code, public API, or

{agent_reliability-1.2.1 → agent_reliability-1.3.0}/CONTRIBUTING.md

@@ -12,7 +12,7 @@ Requires Python 3.11–3.13.
 python -m venv .venv
 # POSIX: source .venv/bin/activate
 # Windows PowerShell: .venv\Scripts\Activate.ps1
-python -m pip install -e ".[dev]"
+python -m pip install -e ".[dev,http,otel-test]"
 ```
 
 Branches use `<type>/<short-description>`, for example

{agent_reliability-1.2.1 → agent_reliability-1.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agent-reliability
-Version: 1.2.1
+Version: 1.3.0
 Summary: Vendor-neutral reliability engineering for autonomous AI agents.
 Project-URL: Homepage, https://github.com/kamleshd07/agent-reliability
 Project-URL: Documentation, https://github.com/kamleshd07/agent-reliability/tree/main/docs
@@ -20,12 +20,15 @@ Classifier: Typing :: Typed
 Requires-Python: >=3.11
 Provides-Extra: dev
 Requires-Dist: build>=1.2; extra == 'dev'
+Requires-Dist: cryptography>=42; extra == 'dev'
 Requires-Dist: hypothesis>=6.108; extra == 'dev'
 Requires-Dist: mypy>=1.11; extra == 'dev'
 Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
 Requires-Dist: pytest-cov>=5.0; extra == 'dev'
 Requires-Dist: pytest>=8.0; extra == 'dev'
 Requires-Dist: ruff>=0.6; extra == 'dev'
+Provides-Extra: http
+Requires-Dist: httpx<0.29,>=0.28.1; extra == 'http'
 Provides-Extra: otel
 Requires-Dist: opentelemetry-api<2,>=1.44; extra == 'otel'
 Provides-Extra: otel-test
@@ -42,7 +45,7 @@ It brings evaluations, SLOs, error budgets, burn rates, and measurement
 provenance to agent applications—and refuses to produce a misleading number
 when evaluation methodologies are incompatible.
 
-**Status: GA (`1.2.1`).** Public APIs documented as stable in
+**Status: GA (`1.3.0` is the latest stable release; the HTTP exporter and the LLM-judge adapter are Experimental, see [release notes](https://github.com/kamleshd07/agent-reliability/blob/main/docs/RELEASE_NOTES_1_3.md)).** Public APIs documented as stable in
 [GA_CONTRACT.md](https://github.com/kamleshd07/agent-reliability/blob/v1.0.0/docs/GA_CONTRACT.md) follow Semantic Versioning.
 See [compatibility](https://github.com/kamleshd07/agent-reliability/blob/v1.0.0/docs/COMPATIBILITY.md).
 
@@ -202,13 +205,24 @@ assertions for tests and CI.
 The SDK is designed to work standalone. It requires no hosted account,
 API key, or network service, and has zero mandatory runtime dependencies.
 
-**Current release: 1.2.1**
+**Current stable release: 1.3.0**
 
 The public API follows semantic versioning, with compatibility and
 installed-artifact verification included in the release process.
 
-No remote ingestion, dashboard, LLM judge, persistence, auto-instrumentation,
-or framework-specific adapter is included. See the [roadmap](https://github.com/kamleshd07/agent-reliability/blob/main/docs/ROADMAP.md).
+Releases before 1.3 include no remote ingestion, dashboard, LLM judge, persistence,
+auto-instrumentation, or framework-specific adapter. Release 1.3 adds an optional
+**Experimental** HTTP evidence exporter (`pip install 'agent-reliability[http]'`, see
+[HTTP exporter](https://github.com/kamleshd07/agent-reliability/blob/main/docs/HTTP_EXPORTER.md)) and an
+Experimental LLM-judge adapter (described below); it still has no dashboard, persistence or
+auto-instrumentation. See the [roadmap](https://github.com/kamleshd07/agent-reliability/blob/main/docs/ROADMAP.md).
+
+### LLM-judge evaluators (experimental)
+
+`agent_reliability.experimental.judge` is an Experimental adapter for model-backed evaluators that call a provider you
+supply. It adds no dependency and sends raw content only to your provider. The reference rubrics are uncalibrated examples. See
+[Evaluate](https://github.com/kamleshd07/agent-reliability/blob/main/docs/EVALUATE.md).
+A sync judge's timeout is best effort unless the provider is wrapped in `ExecutorSyncProvider`; the provider's own client timeout is the real bound (see the same page).
 
 ## Documentation
 
@@ -230,3 +244,9 @@ not a public issue.
 ## License
 
 Apache License 2.0. See [LICENSE](https://github.com/kamleshd07/agent-reliability/blob/v1.0.0/LICENSE).
+
+### HTTP evidence delivery
+
+The optional [HTTP exporter](https://github.com/kamleshd07/agent-reliability/blob/main/docs/HTTP_EXPORTER.md) sends schema-v2 evidence
+using `agent-reliability[http]`, bounded delivery queues and explicit shutdown.
+Platform integration and independent review remain required before release.

{agent_reliability-1.2.1 → agent_reliability-1.3.0}/README.md

@@ -8,7 +8,7 @@ It brings evaluations, SLOs, error budgets, burn rates, and measurement
 provenance to agent applications—and refuses to produce a misleading number
 when evaluation methodologies are incompatible.
 
-**Status: GA (`1.2.1`).** Public APIs documented as stable in
+**Status: GA (`1.3.0` is the latest stable release; the HTTP exporter and the LLM-judge adapter are Experimental, see [release notes](https://github.com/kamleshd07/agent-reliability/blob/main/docs/RELEASE_NOTES_1_3.md)).** Public APIs documented as stable in
 [GA_CONTRACT.md](https://github.com/kamleshd07/agent-reliability/blob/v1.0.0/docs/GA_CONTRACT.md) follow Semantic Versioning.
 See [compatibility](https://github.com/kamleshd07/agent-reliability/blob/v1.0.0/docs/COMPATIBILITY.md).
 
@@ -168,13 +168,24 @@ assertions for tests and CI.
 The SDK is designed to work standalone. It requires no hosted account,
 API key, or network service, and has zero mandatory runtime dependencies.
 
-**Current release: 1.2.1**
+**Current stable release: 1.3.0**
 
 The public API follows semantic versioning, with compatibility and
 installed-artifact verification included in the release process.
 
-No remote ingestion, dashboard, LLM judge, persistence, auto-instrumentation,
-or framework-specific adapter is included. See the [roadmap](https://github.com/kamleshd07/agent-reliability/blob/main/docs/ROADMAP.md).
+Releases before 1.3 include no remote ingestion, dashboard, LLM judge, persistence,
+auto-instrumentation, or framework-specific adapter. Release 1.3 adds an optional
+**Experimental** HTTP evidence exporter (`pip install 'agent-reliability[http]'`, see
+[HTTP exporter](https://github.com/kamleshd07/agent-reliability/blob/main/docs/HTTP_EXPORTER.md)) and an
+Experimental LLM-judge adapter (described below); it still has no dashboard, persistence or
+auto-instrumentation. See the [roadmap](https://github.com/kamleshd07/agent-reliability/blob/main/docs/ROADMAP.md).
+
+### LLM-judge evaluators (experimental)
+
+`agent_reliability.experimental.judge` is an Experimental adapter for model-backed evaluators that call a provider you
+supply. It adds no dependency and sends raw content only to your provider. The reference rubrics are uncalibrated examples. See
+[Evaluate](https://github.com/kamleshd07/agent-reliability/blob/main/docs/EVALUATE.md).
+A sync judge's timeout is best effort unless the provider is wrapped in `ExecutorSyncProvider`; the provider's own client timeout is the real bound (see the same page).
 
 ## Documentation
 
@@ -196,3 +207,9 @@ not a public issue.
 ## License
 
 Apache License 2.0. See [LICENSE](https://github.com/kamleshd07/agent-reliability/blob/v1.0.0/LICENSE).
+
+### HTTP evidence delivery
+
+The optional [HTTP exporter](https://github.com/kamleshd07/agent-reliability/blob/main/docs/HTTP_EXPORTER.md) sends schema-v2 evidence
+using `agent-reliability[http]`, bounded delivery queues and explicit shutdown.
+Platform integration and independent review remain required before release.

{agent_reliability-1.2.1 → agent_reliability-1.3.0}/docs/ARCHITECTURE.md

@@ -194,3 +194,18 @@ optional OTel ownership remain unchanged. See ADR-0009 and
 ADR-0001 ([docs/adr/0001-architecture-boundaries.md](adr/0001-architecture-boundaries.md))
 records only the layering and dependency-direction decision made at M0.
 The items above are explicitly not decided yet.
+
+## Language-neutral public HTTP contracts
+
+Gentlity public ingestion and control-plane APIs are defined by the platform's
+published OpenAPI, referenced JSON Schemas and normative protocol documentation,
+not by this Python package's dataclasses, exceptions, enum internals or
+serialization defaults. The SDK must validate and emit the documented wire
+types and versions, authenticate through standard HTTP, classify responses by
+HTTP status/header and stable error code, and obey documented idempotency,
+retry, rate-limit, `Retry-After`, timeout and compatibility rules.
+
+Shared conformance assets must be data-only and reusable by Python, Go, Java and
+TypeScript. The SDK and platform execute the same positive, negative, boundary,
+request/response and digest fixtures. Schema-v1 canonical digest bytes and
+goldens are frozen across upgrades; SDK behavior cannot redefine them.

{agent_reliability-1.2.1 → agent_reliability-1.3.0}/docs/COMPATIBILITY.md

@@ -31,16 +31,25 @@ and [OTEL_MAPPING.md](OTEL_MAPPING.md).
 
 ## Additive measurement-health extension
 
-M8 adds `agent_reliability.measurement` and additive `RunHandle` inspection
+The measurement-health API adds `agent_reliability.measurement` and additive `RunHandle` inspection
 and policy members. Existing 1.0 export sets, callable signatures, enum
 semantics, event payloads, and SLO/reliability calculations are unchanged.
 This public addition ships in 1.1.0 as a backward-compatible minor release.
-M9 adds tests, examples, documentation, and artifact checks only; the `1.1.1`
+The patch release adds tests, examples, documentation, and artifact checks only; the `1.1.1`
 release leaves the `1.1.0` public and semantic contract unchanged.
 
+## Additive HTTP exporter extension (1.3.0)
+
+1.3 adds the optional `agent-reliability[http]` extra and the HTTP exporter entry points,
+classed EXPERIMENTAL (see [GA_CONTRACT.md](GA_CONTRACT.md)), plus the experimental judge
+adapter. Existing 1.0-1.2 stable export sets, callable signatures, enum semantics, event
+payloads and SLO/reliability calculations are unchanged, and base runtime dependencies remain
+zero. This ships as a backward-compatible minor release (`1.3.0`); the earlier `1.3.0rc1`
+pre-release is superseded. Migration and rollback: [RELEASE_NOTES_1_3.md](RELEASE_NOTES_1_3.md).
+
 ## Additive developer-adoption extension
 
-DX-1 — Developer Adoption adds built-in evaluators and an `EvaluationOutcome`
+The developer-adoption extension adds built-in evaluators and an `EvaluationOutcome`
 re-export to `agent_reliability.evaluation`, `AggregationConflictError`/
 `evaluate_reliability_or_raise` to `agent_reliability.reliability`, and two
 new stable namespaces, `agent_reliability.reporting` and

{agent_reliability-1.2.1 → agent_reliability-1.3.0}/docs/CONCEPTS.md

@@ -51,7 +51,7 @@ itself change the measurement—it explains how much tolerance remains.
 ## Burn rate
 
 Observed bad-event rate divided by the SLO's allowed bad-event rate. It is not
-“current failures divided by total failures.” M5 calculates a burn rate only
+“current failures divided by total failures.” The local reliability engine calculates a burn rate only
 when the caller supplies an explicit lookback ratio; it does not retain data or
 select a time window.
 

{agent_reliability-1.2.1 → agent_reliability-1.3.0}/docs/ENGINEERING_PRINCIPLES.md

@@ -78,7 +78,7 @@ AutoGen, or any other framework to be present.
 
 ## 10. Dependency discipline
 
-Runtime dependencies are minimal by default and zero at M0. Every
+Runtime dependencies are minimal, with zero mandatory core dependencies. Every
 runtime dependency must justify itself in the PR that introduces it —
 "it's convenient" is not sufficient justification.
 

agent_reliability-1.3.0/docs/EVALUATE.md

@@ -0,0 +1,224 @@
+# Evaluate
+
+How to judge an indicator once you can execute your agent. See
+[Instrument](../examples/basic_reliability.py) first if you have not wrapped
+a run yet, and [Concepts](CONCEPTS.md) for what "evaluator," "indicator," and
+"provenance" mean.
+
+## Built-in evaluators
+
+All built-in evaluators live in `agent_reliability.evaluation` and share the
+same shape: construct one with an explicit `EvaluatorIdentity` and
+configuration, then call `.evaluate(value)` directly or through
+`EvaluatorRunner` (recommended — it attaches provenance and converts a raw
+exception into a distinct `EvaluationExecutionFailure`, never into `FAIL` or
+`UNKNOWN`).
+
+| Evaluator | Judges | UNKNOWN case |
+|---|---|---|
+| `EqualityEvaluator` | `value == expected` | never |
+| `PredicateEvaluator` | a trusted `Callable[[T], bool \| None]` | when the predicate returns `None` |
+| `ContainsEvaluator` | substring presence in a `str`, optional case-insensitivity | never |
+| `RegexMatchEvaluator` | a stdlib `re` pattern (`search` or `fullmatch`) against a `str` | never — a malformed pattern raises at construction, not evaluation |
+| `LatencyThresholdEvaluator` | an explicit `duration_seconds` against a threshold | when `duration_seconds` is `None` (no measurement captured) |
+| `ToolCalledEvaluator` / `ToolNotCalledEvaluator` | presence/absence of a tool name in a `frozenset[str]` you supply | never |
+
+Run [`examples/builtin_evaluators_quickstart.py`](../examples/builtin_evaluators_quickstart.py)
+to see `ContainsEvaluator` and `LatencyThresholdEvaluator` produce `PASS`,
+`FAIL`, and a legitimate `UNKNOWN` in one script, and
+[`examples/tool_call_evaluators.py`](../examples/tool_call_evaluators.py) for
+the tool-name evaluators.
+
+## What UNKNOWN means here
+
+`UNKNOWN` means the evaluator ran to completion but could not reach a
+definite judgment — for example, `LatencyThresholdEvaluator` receiving
+`duration_seconds=None` because your timing instrumentation did not capture
+a value for that run. It is never used for "the evaluator crashed" (that is
+an `EvaluationExecutionFailure`, isolated by `EvaluatorRunner`) and never
+silently becomes `FAIL`. See [Concepts](CONCEPTS.md#unknown) and
+[GA_CONTRACT.md](GA_CONTRACT.md#frozen-outcome-and-uncertainty-semantics).
+
+## Privacy boundary for tool evaluators
+
+`ToolCalledEvaluator`/`ToolNotCalledEvaluator` accept only a bounded
+`frozenset[str]` of tool-name identifiers you have already decided to
+disclose. They never accept, inspect, or retain tool call arguments, tool
+results, or framework-specific tool objects — see each class's docstring for
+the exact contract, and
+[`tests/security/test_m8_builtin_evaluator_privacy.py`](../tests/security/test_m8_builtin_evaluator_privacy.py)
+for the adversarial tests that enforce it.
+
+## Custom evaluators
+
+Implement the `SyncEvaluator`/`AsyncEvaluator` protocol directly — no
+inheritance or registration required. See
+[EVALUATOR_FRAMEWORK.md](EVALUATOR_FRAMEWORK.md) for the full contract and
+[`docs/EVALUATION_PROVENANCE.md`](EVALUATION_PROVENANCE.md) for why identity
+and determinism matter to aggregation later.
+
+## Reducing `AggregationConflict` boilerplate
+
+`evaluate_reliability(...)` returns `ReliabilityReport | AggregationConflict`
+so a genuine measurement incompatibility is never silently resolved. When a
+call site has no recovery path other than failing loudly, use
+`evaluate_reliability_or_raise(...)` instead — identical arguments, but it
+raises `AggregationConflictError` (carrying the original conflict on
+`.conflict`) instead of returning it. Keep using the explicit
+`evaluate_reliability` form wherever you want to inspect or handle a
+conflict programmatically.
+
+## LLM-judge evaluators (EXPERIMENTAL)
+
+`agent_reliability.experimental.judge` adds model-backed evaluators for semantic dimensions that equality and
+predicates cannot express. It is **Experimental** ([versioning policy](VERSIONING.md)): names, defaults, the prompt template (`jp1`)
+and vocabularies may change in a minor release with release notes. It is standard library only, ships no provider, reads no
+credentials or environment, and adds no wire member and no runtime dependency. A judge is never deterministic
+(`deterministic=False`).
+
+### Privacy statement
+
+Raw evaluated text (question, answer, context, reference, instructions, policy, task evidence) goes **only** to the provider object
+you supply, and only the roles the rubric lists. The SDK never retains, logs, hashes, serializes, copies or exports it. What can reach
+Gentlity through the existing evidence path is limited to the indicator name, the outcome, a closed reason code
+(`<dimension>.<code>`), the evaluator identity labels (rubric id, rubric version, `configuration_id`) and, for failures, one of ten
+fixed class names. Judge `reason` text, citations, confidence, token counts, latency, cost and the sampling rate stay local. What your
+provider does with the prompt (retention, region, training) is governed by your agreement with that provider; the SDK makes no
+statement about it. Labels you choose (`model_label`, indicator names) are stored by the platform and must not contain secrets,
+hostnames or customer names.
+
+### Quick start
+
+```python
+from agent_reliability.experimental.judge import (
+    ContentPart,
+    ContentRole,
+    EvaluationRequest,
+    IndicatorBinding,
+    JudgeCoordinator,
+    JudgeDimension,
+    LLMJudge,
+    load_reference_rubric,
+)
+
+judge = LLMJudge(
+    rubric=load_reference_rubric("judge-groundedness"),
+    provider=my_async_provider,  # any JudgeProvider; see examples/judge_providers.py
+    model_label="local-llama3-8b-q4",  # non-sensitive label, chars [A-Za-z0-9._-]
+)
+coordinator = JudgeCoordinator([IndicatorBinding("groundedness", judge)])
+async with sdk.run(agent_id="support-agent", name="Support", version="1.4.0") as run:
+    answer = await agent.answer(question)
+    async with coordinator.scope(run) as scope:
+        await scope.evaluate(
+            "groundedness",
+            EvaluationRequest(
+                JudgeDimension.GROUNDEDNESS,
+                (
+                    ContentPart(ContentRole.CONTEXT, snippet),
+                    ContentPart(ContentRole.ANSWER, answer),
+                ),
+            ),
+        )
+    print(coordinator.coverage().describe())
+```
+
+A runnable offline version with scripted fake providers is `examples/judge_fake_provider.py`. Use `SyncLLMJudge` and
+`scope.evaluate_sync` for blocking providers (no `asyncio.run` is used anywhere). Evaluations must finish inside the run block:
+the scope drains pending work up to `drain_timeout_seconds` and records `JudgeTimeoutError` evidence for what is unfinished; a late
+evaluation after the run is closed is not supported in V1.
+
+### Meaning of PASS per dimension
+
+The five reference rubrics (`judge-answer-correctness`, `judge-groundedness`, `judge-instruction-adherence`, `judge-safety-policy`,
+`judge-task-completion-evidence`) are **uncalibrated examples**: not calibrated by Gentlity, no accuracy claim, domain thresholds are
+yours. Load them with `load_reference_rubric(id)` or copy the JSON and edit it (change `rubric_version` whenever behavior changes).
+
+| Dimension (rubric) | PASS means | PASS does not mean |
+|---|---|---|
+| `answer-correctness` (judge-answer-correctness) | no contradiction with the supplied reference and the required facts are present | the answer is correct in general |
+| `groundedness` (judge-groundedness) | every material claim is supported by the supplied context blocks, with at least one citation | the context itself is true |
+| `instruction-adherence` (judge-instruction-adherence) | the checkable items stated in the instructions were followed | unstated expectations were met (deterministic checks are better where possible) |
+| `safety-policy` (judge-safety-policy) | this rubric found no violation of the supplied policy | the answer is safe |
+| `task-completion-evidence` (judge-task-completion-evidence) | the evidence TEXT supports the claim of completion | the business outcome occurred (never an authoritative outcome; a fabricated confirmation passes it by construction; use an outcome expectation for critical journeys) |
+
+### Outcomes, failures and what the platform sees
+
+A completed judgment is PASS, FAIL or UNKNOWN (low confidence and judge disagreement are UNKNOWN). Anything else (timeout, provider
+failure, refusal, invalid or oversize output, input over the limit, budget exhausted, queue shed) is an **execution failure**, recorded
+with the class name as `exception_type`, never as an outcome. Failures and UNKNOWN leave the reliability denominator, so reliability must
+be read **beside** the failure rate and the UNKNOWN rate: 99 failed attempts plus one PASS reads as reliability 1.0. The local
+`coordinator.coverage()` report shows attempted, determinate, failed, unknown and unsampled counts together and offers no single "healthy"
+value on purpose. Any failure marks the run's measurement health `UNAVAILABLE`, and the platform's health counts are not
+indicator-scoped, so every indicator in that scope and window shows an unavailable `measurement_state`. Only a window with no determinate
+evidence reaches a gate's `on_no_data`; a partial outage or a high UNKNOWN rate has no gate effect, so set `on_no_data` deliberately
+(BLOCK where an outage must stop deployment). Failure evidence reaches the platform only when the SDK build's `record_evaluation_failure`
+accepts `indicator=` and the sink supports health evidence; otherwise it is local run health only and the coordinator reports "failure
+evidence not delivered".
+
+### Timeouts, blocking providers and shutdown
+
+* **Async (`LLMJudge`).** The per-attempt timeout and the total deadline are enforced with `asyncio.timeout`; cancellation propagates and records nothing.
+* **Sync (`SyncLLMJudge`) is best effort.** A blocking call cannot be interrupted from outside. The judge passes the remaining deadline to your provider as `deadline_seconds` and
+  classifies an overrun as `JudgeTimeoutError` only *after the call returns*, discarding the late reply. A provider that never returns blocks `evaluate_sync` indefinitely, and the
+  scope's drain cannot help in the same thread. **The real bound is your provider's own socket/client timeout**: always set one (the example adapters pass `deadline_seconds` to the HTTP client).
+  When you need the deadline enforced regardless, wrap the provider in `ExecutorSyncProvider(inner, max_threads=N)`: the call runs in a bounded daemon helper thread and is abandoned at the deadline
+  (the thread itself cannot be killed and lives until the provider returns; beyond `max_threads` calls fail fast with `JudgeProviderUnavailableError`).
+* **Tokens of discarded late replies are not counted.** A reply that arrives after its deadline (or after a drain timeout) is discarded together with its reported usage, so token and cost budgets
+  can undercount what the provider billed. The documented overshoot bound applies; treat budgets as guard rails, not as billing.
+* **`jitter` and clocks.** A custom `jitter` callable must return a number in `[0, 1]`; anything else raises `ValueError("jitter must return a number in [0, 1]")`, a programmer error that is not one of the ten failure classes.
+
+### The recording gate: what runs inside it
+
+Concurrent judge *execution* is never concurrent *recording*. Every call that hands evidence to the run handle is made while holding the scope's one recording gate. `RunHandle.record_*`
+synchronously invokes your **sink and diagnostic handler inside that gate**, so they **must be non-blocking** (the HTTP exporter's `emit` is); a slow sink delays all recording of the scope.
+A sink or diagnostic handler must not call back into the same `EvaluationScope`: such a re-entrant call raises `RuntimeError` instead of deadlocking. Application code must not record on the same
+handle from another thread while a threaded scope has evaluations in flight.
+
+"Recorded" means *handed to the run handle*. The SDK isolates a failing sink, so an evaluation reported as `RECORDED_RESULT` or `RECORDED_FAILURE` may still have been dropped by a sink that
+raised; the coordinator cannot detect that. Watch your sink's own delivery status (for the HTTP exporter, its delivery snapshot) next to the coordinator's `coverage()`.
+
+### Measurement-health safeguards
+
+| Safeguard | What to do |
+|---|---|
+| Budget and overload limits | Size budgets and limits so budget exhaustion and overload do not occur (`JudgeBudget`, `JudgeLimits`, `max_pending`, `max_concurrency`); treat any `JudgeBudgetExceededError` or `JudgeOverloadedError` as an incident. Defaults are safe, not fast, and are not benchmarked |
+| Run health | One judge failure sets run health `UNAVAILABLE`, which changes `measurement_state` for every indicator in the scope and window |
+| Coverage monitoring | Read failure rate and UNKNOWN rate beside reliability (`coverage()`; platform reads of failure and UNKNOWN counts) |
+| Scope separation | Do not interpret `measurement_state` of deterministic indicators while judge indicators share their scope and window; consider a separate scope for judge evaluation |
+| Failure alerting | Wire a `JudgeObserver` (or poll `coverage()`) to your own alerting: failures by class, shed, skipped |
+| No-data policy | Set `on_no_data` explicitly; partial outage is not covered |
+
+### Identity and cohorts
+
+The wire identity is `name=<rubric_id>`, `version=<rubric_version>`, `configuration_id=<model_label>.jp1.mc-<min_confidence>.rp<repeats>.ct<0|1>`,
+all in the grammar `[A-Za-z0-9._-]` (1-128, alphanumeric ends) that both the SDK and the HTTP exporter accept. A `+`
+(for example `1.0.0+build5`) is accepted by `EvaluatorIdentity` but rejected by the exporter, so evidence for such an identity is never
+delivered; that hazard exists for any evaluator and is not changed here. The platform pools by indicator only: when a rubric or model
+change should start a new cohort, also change the indicator name (for example `groundedness-v2`).
+
+### Vocabulary notes
+
+The contract lists `judge_abstained`, `low_confidence` and `judge_disagreement` as always-present unknown codes. The SDK requires `low_confidence` in every rubric and `judge_disagreement` when
+`repeats > 1`; `judge_abstained` is *not* enforced (a rubric may omit it; the reference rubrics include it). `SamplingPolicy.salt` is validated more strictly than the contract's charset
+(alphanumeric first and last character, 1-64 characters of `[A-Za-z0-9._-]`, or empty) so that it can never be rejected by an identity grammar if it is later folded into a label; salts such as `_pilot` are rejected.
+A reply fenced with Windows line endings (`` ```json `` followed by CRLF) is rejected by the strict parser and costs one re-ask.
+
+### Calibration
+
+Calibration is a customer-local practice: `run_calibration` over your own labelled dataset with your own provider, producing a
+content-free report. Gentlity receives no dataset, enforces no threshold and publishes no accuracy figure; see the calibration section
+below. Results describe one rubric version, model label and dataset only.
+
+### Local-model walkthrough (manual, not a CI criterion)
+
+Prerequisites: a locally running OpenAI-compatible runtime (for example Ollama serving on `127.0.0.1:11434`), a model already pulled by
+you (the SDK never downloads models), Python 3.11+, and this package. Steps: (1) copy `examples/judge_providers.py` next to your code;
+(2) `provider = local_runtime_provider(model="<your model>")`; (3) build `SyncLLMJudge(rubric=load_reference_rubric(...), provider=provider,
+model_label="<non-sensitive label>")`; (4) run a few requests through `JudgeCoordinator` with `evaluate_sync`; (5) read
+`coordinator.coverage().describe()`. Model output is non-deterministic, so the output of this walkthrough is not an acceptance criterion
+and no result of it is a claim about quality.
+
+## Next
+
+[Measure reliability locally](MEASURE_RELIABILITY_LOCALLY.md).