agent-reliability 1.2.0__tar.gz → 1.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/.github/workflows/ci.yml +25 -2
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/.github/workflows/release.yml +14 -1
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/CHANGELOG.md +112 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/CONTRIBUTING.md +1 -1
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/PKG-INFO +41 -14
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/README.md +37 -13
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/ARCHITECTURE.md +15 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/COMPATIBILITY.md +12 -3
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/CONCEPTS.md +1 -1
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/ENGINEERING_PRINCIPLES.md +1 -1
- agent_reliability-1.3.0/docs/EVALUATE.md +224 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/EVALUATION_PROVENANCE.md +3 -3
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/GA_CONTRACT.md +30 -1
- agent_reliability-1.3.0/docs/HTTP_EXPORTER.md +309 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/MEASUREMENT_HEALTH.md +1 -1
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/OTEL_MAPPING.md +9 -9
- agent_reliability-1.3.0/docs/P29_PILOT_REPORT_TEMPLATE.json +66 -0
- agent_reliability-1.3.0/docs/P29_PILOT_RUNBOOK.md +64 -0
- agent_reliability-1.3.0/docs/P29_PILOT_SAFEGUARDS.md +95 -0
- agent_reliability-1.3.0/docs/PLATFORM_QUICKSTART.md +59 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/README.md +1 -14
- agent_reliability-1.3.0/docs/RELEASE_NOTES_1_3.md +112 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/RELEASING.md +2 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/ROADMAP.md +3 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/SECURITY_MODEL.md +29 -48
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/SLO_SEMANTICS.md +7 -7
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/TELEMETRY_SPEC.md +9 -25
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/VERSIONING.md +14 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/adr/0007-evaluator-architecture-and-provenance-semantics.md +5 -1
- agent_reliability-1.3.0/docs/adr/0010-experimental-judge-adapter-amends-adr-0007-decision-9.md +172 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/adr/README.md +2 -1
- agent_reliability-1.3.0/examples/judge_fake_provider.py +148 -0
- agent_reliability-1.3.0/examples/judge_pilot_demo.py +183 -0
- agent_reliability-1.3.0/examples/judge_pilot_replay.py +102 -0
- agent_reliability-1.3.0/examples/judge_providers.py +301 -0
- agent_reliability-1.3.0/examples/platform_quickstart.py +72 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/pyproject.toml +26 -1
- agent_reliability-1.3.0/scripts/p29_pilot_report_check.py +301 -0
- agent_reliability-1.3.0/scripts/p29_sg_evidence.py +111 -0
- agent_reliability-1.3.0/scripts/verify_p28_fixtures.py +40 -0
- agent_reliability-1.3.0/scripts/verify_p28_reference.mjs +68 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/scripts/verify_release_artifacts.py +183 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/__init__.py +1 -1
- agent_reliability-1.3.0/src/agent_reliability/adapters/_http_delivery.py +1109 -0
- agent_reliability-1.3.0/src/agent_reliability/adapters/_http_protocol.py +216 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/adapters/event_sinks.py +29 -1
- agent_reliability-1.3.0/src/agent_reliability/adapters/http_event_sink.py +683 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/__init__.py +151 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/_engine.py +317 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/_jsonschema.py +103 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/_schema.py +43 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/budget.py +84 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/calibration.py +596 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/coordinator.py +1002 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/core.py +417 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/errors.py +158 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/executor.py +100 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/identity.py +105 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/judge.py +194 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/provider.py +171 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/reference.py +34 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/rubric.py +434 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/rubrics/judge-answer-correctness.json +44 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/rubrics/judge-groundedness.json +45 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/rubrics/judge-instruction-adherence.json +40 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/rubrics/judge-safety-policy.json +40 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/rubrics/judge-task-completion-evidence.json +40 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/schemas/calibration-case.schema.json +106 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/schemas/calibration-report.schema.json +590 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/structured.py +92 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/sync.py +126 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/testing.py +134 -0
- agent_reliability-1.3.0/src/agent_reliability/experimental/judge/values.py +381 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/ports/event_sink.py +30 -1
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/sdk/client.py +73 -5
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/sdk/context.py +13 -1
- agent_reliability-1.3.0/tests/compat/test_experimental_http_exporter_api.py +215 -0
- agent_reliability-1.3.0/tests/compat/test_experimental_judge_api.py +319 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/compat/test_public_api_1_1.py +2 -1
- agent_reliability-1.3.0/tests/contract/test_p28_http_protocol.py +126 -0
- agent_reliability-1.3.0/tests/contract/test_p29_judge_vectors.py +292 -0
- agent_reliability-1.3.0/tests/contract/test_p29_wire_conformance.py +476 -0
- agent_reliability-1.3.0/tests/fakes/p29_baseline.py +37 -0
- agent_reliability-1.3.0/tests/fakes/p29_judge.py +328 -0
- agent_reliability-1.3.0/tests/fixtures/documentation/editorial-revisions.json +7 -0
- agent_reliability-1.3.0/tests/fixtures/documentation/runtime-revisions.json +12 -0
- agent_reliability-1.3.0/tests/fixtures/p28/legacy_platform_responses.json +161 -0
- agent_reliability-1.3.0/tests/fixtures/p28/manifest.json +22 -0
- agent_reliability-1.3.0/tests/fixtures/p28/operational_vectors.json +1537 -0
- agent_reliability-1.3.0/tests/fixtures/p28/v1_digest_vectors.json +1657 -0
- agent_reliability-1.3.0/tests/fixtures/p28/v2_digest_vectors_specification_only.json +226 -0
- agent_reliability-1.3.0/tests/fixtures/p29/calibration_known_confusion.json +760 -0
- agent_reliability-1.3.0/tests/fixtures/p29/failure_mapping_vectors.json +749 -0
- agent_reliability-1.3.0/tests/fixtures/p29/judge_output_vectors.json +1134 -0
- agent_reliability-1.3.0/tests/fixtures/p29/manifest.json +42 -0
- agent_reliability-1.3.0/tests/fixtures/p29/mapping_vectors.json +474 -0
- agent_reliability-1.3.0/tests/fixtures/p29/privacy_planted.json +41 -0
- agent_reliability-1.3.0/tests/fixtures/p29/prompt_golden_jp1.json +343 -0
- agent_reliability-1.3.0/tests/fixtures/p29/reference_rubric_cases.json +629 -0
- agent_reliability-1.3.0/tests/fixtures/p29/rubric_invalid.json +2432 -0
- agent_reliability-1.3.0/tests/fixtures/p29/rubric_valid.json +788 -0
- agent_reliability-1.3.0/tests/fixtures/p29/wire_expectation_vectors.json +578 -0
- agent_reliability-1.3.0/tests/property/test_experimental_judge_properties.py +234 -0
- agent_reliability-1.3.0/tests/security/test_experimental_judge_privacy.py +383 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/adapters/test_adapters.py +17 -4
- agent_reliability-1.3.0/tests/unit/experimental_judge/__init__.py +0 -0
- agent_reliability-1.3.0/tests/unit/experimental_judge/test_p29_pilot_report_check.py +253 -0
- agent_reliability-1.3.0/tests/unit/experimental_judge/test_p29_sg_safeguards.py +56 -0
- agent_reliability-1.3.0/tests/unit/experimental_judge/test_phase4_measurement_integrity.py +505 -0
- agent_reliability-1.3.0/tests/unit/experimental_judge/test_review_remediation.py +583 -0
- agent_reliability-1.3.0/tests/unit/experimental_judge/test_s00_s12_docs.py +242 -0
- agent_reliability-1.3.0/tests/unit/experimental_judge/test_s01_identity_grammar.py +158 -0
- agent_reliability-1.3.0/tests/unit/experimental_judge/test_s01_rubric.py +163 -0
- agent_reliability-1.3.0/tests/unit/experimental_judge/test_s01_values.py +365 -0
- agent_reliability-1.3.0/tests/unit/experimental_judge/test_s02_core.py +334 -0
- agent_reliability-1.3.0/tests/unit/experimental_judge/test_s03_llm_judge.py +769 -0
- agent_reliability-1.3.0/tests/unit/experimental_judge/test_s04_sync_judge.py +205 -0
- agent_reliability-1.3.0/tests/unit/experimental_judge/test_s05_coordinator.py +755 -0
- agent_reliability-1.3.0/tests/unit/experimental_judge/test_s05_recording_gate.py +335 -0
- agent_reliability-1.3.0/tests/unit/experimental_judge/test_s06_reference_and_examples.py +360 -0
- agent_reliability-1.3.0/tests/unit/experimental_judge/test_s07_s08_optional.py +221 -0
- agent_reliability-1.3.0/tests/unit/experimental_judge/test_s10_calibration.py +566 -0
- agent_reliability-1.3.0/tests/unit/experimental_judge/test_s11_pilot_artifacts.py +212 -0
- agent_reliability-1.3.0/tests/unit/test_http_exporter.py +869 -0
- agent_reliability-1.3.0/tests/unit/test_http_exporter_indicator_allowlist.py +544 -0
- agent_reliability-1.3.0/tests/unit/test_http_exporter_real_stack_uncertainty.py +136 -0
- agent_reliability-1.3.0/tests/unit/test_http_exporter_release_regressions.py +217 -0
- agent_reliability-1.3.0/tests/unit/test_http_exporter_review_regressions.py +424 -0
- agent_reliability-1.3.0/tests/unit/test_http_exporter_tls_real.py +680 -0
- agent_reliability-1.3.0/tests/unit/test_platform_quickstart.py +63 -0
- agent_reliability-1.3.0/tests/unit/test_release_artifact_guards.py +148 -0
- agent_reliability-1.3.0/tests/unit/test_workflow_policy.py +104 -0
- agent_reliability-1.2.0/docs/EVALUATE.md +0 -73
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/.github/ISSUE_TEMPLATE/bug_report.yml +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/.github/ISSUE_TEMPLATE/feature_request.yml +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/.github/pull_request_template.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/.gitignore +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/CODE_OF_CONDUCT.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/LICENSE +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/SECURITY.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/benchmarks/bench_evaluation.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/benchmarks/bench_measurement_health.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/benchmarks/bench_otel.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/benchmarks/bench_reliability.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/benchmarks/bench_sdk.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/DOMAIN_MODEL.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/DX_REVIEW.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/EVALUATOR_FRAMEWORK.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/GA_READINESS.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/INTEGRATIONS.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/LOCAL_RELIABILITY_ENGINE.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/M9_HARDENING_AUDIT.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/MEASUREMENT_HEALTH_POLICY_EXAMPLES.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/MEASURE_RELIABILITY_LOCALLY.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/PUBLIC_API_REVIEW.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/QUICKSTART.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/SDK_DESIGN.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/TESTING_STRATEGY.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/TEST_RELIABILITY.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/VISION.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/adr/0001-architecture-boundaries.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/adr/0002-reliability-mathematics-and-undefined-data-semantics.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/adr/0003-python-sdk-runtime-and-context-architecture.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/adr/0004-instrumentation-failure-isolation.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/adr/0005-instrumentation-initialization-degraded-mode.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/adr/0006-opentelemetry-interoperability-and-context-ownership.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/adr/0008-reliability-aggregation-and-provenance-compatibility.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/docs/adr/0009-measurement-health-and-policy-boundary.md +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/examples/.gitkeep +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/examples/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/examples/async_agent.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/examples/basic_reliability.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/examples/builtin_evaluators_quickstart.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/examples/opentelemetry_example.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/examples/policy_bounded_degradation.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/examples/policy_fail_closed.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/examples/policy_fail_open.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/examples/provenance_conflict.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/examples/slo_assertion.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/examples/tool_call_evaluators.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/adapters/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/adapters/otel/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/adapters/otel/run_context.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/adapters/system_clock.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/adapters/uuid_run_id_generator.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/application/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/application/measurement_policy.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/domain/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/domain/error_budget.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/domain/evaluation.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/domain/identity.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/domain/measurement_health.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/domain/runs.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/domain/sli.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/domain/slo.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/evaluation/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/evaluation/_validation.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/evaluation/builtins.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/evaluation/identity.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/evaluation/protocols.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/evaluation/result.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/experimental/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/measurement/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/ports/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/ports/clock.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/ports/events.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/ports/id_generator.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/ports/run_context.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/py.typed +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/reliability/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/reliability/engine.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/reliability/model.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/reporting/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/reporting/formatting.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/sdk/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/sdk/diagnostics.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/sdk/evaluator_runner.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/testing/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/src/agent_reliability/testing/assertions.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/compat/test_public_api_1_0.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/compat/test_public_api_1_2.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/compat/test_semantic_1_0.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/contract/.gitkeep +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/contract/test_examples.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/contract/test_readme_links.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/contract/test_reliability_m1_contract.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/contract/test_run_context_bridge.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/fakes/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/fakes/clock.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/fakes/diagnostics.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/fakes/id_generator.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/fakes/sinks.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/integration/.gitkeep +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/integration/test_measurement_health_otel.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/integration/test_otel_trace_context.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/property/test_error_budget_properties.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/property/test_evaluator_properties.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/property/test_measurement_health_properties.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/property/test_reliability_properties.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/property/test_sdk_properties.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/property/test_sli_properties.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/security/test_m8_builtin_evaluator_privacy.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/security/test_measurement_health_failure_privacy.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/security/test_measurement_health_privacy.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/security/test_privacy_contract.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/typing/installed_consumer.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/typing/mypy.ini +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/domain/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/domain/test_error_budget.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/domain/test_evaluation.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/domain/test_identity.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/domain/test_measurement_health.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/domain/test_runs.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/domain/test_sli.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/domain/test_slo.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/evaluation/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/evaluation/test_builtins_and_protocols.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/evaluation/test_identity_and_result.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/evaluation/test_m8_builtin_evaluators.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/evaluation/test_runner.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/examples/test_measurement_policy_examples.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/ports/test_events.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/reliability/test_engine.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/reliability/test_m8_convenience.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/reliability/test_measurement_health_integration.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/reliability/test_measurement_health_quadrants.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/reliability/test_model.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/reporting/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/reporting/test_formatting.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/sdk/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/sdk/test_evaluation_integration.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/sdk/test_failure_isolation.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/sdk/test_lifecycle_async.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/sdk/test_lifecycle_sync.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/sdk/test_measurement_health.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/sdk/test_measurement_health_hardening.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/sdk/test_memory.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/sdk/test_nesting.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/sdk/test_threading.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/test_package.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/testing_helpers/__init__.py +0 -0
- {agent_reliability-1.2.0 → agent_reliability-1.3.0}/tests/unit/testing_helpers/test_assertions.py +0 -0
|
@@ -17,6 +17,7 @@ jobs:
|
|
|
17
17
|
quality:
|
|
18
18
|
name: Lint, format, and type-check
|
|
19
19
|
runs-on: ubuntu-latest
|
|
20
|
+
timeout-minutes: 30
|
|
20
21
|
steps:
|
|
21
22
|
- name: Checkout
|
|
22
23
|
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
@@ -28,7 +29,7 @@ jobs:
|
|
|
28
29
|
cache: "pip"
|
|
29
30
|
|
|
30
31
|
- name: Install package with dev dependencies
|
|
31
|
-
run: python -m pip install -e ".[dev]"
|
|
32
|
+
run: python -m pip install -e ".[dev,http]"
|
|
32
33
|
|
|
33
34
|
- name: Lint (ruff check)
|
|
34
35
|
run: ruff check .
|
|
@@ -42,6 +43,7 @@ jobs:
|
|
|
42
43
|
test:
|
|
43
44
|
name: Test stable contracts (py${{ matrix.python-version }})
|
|
44
45
|
runs-on: ubuntu-latest
|
|
46
|
+
timeout-minutes: 30
|
|
45
47
|
strategy:
|
|
46
48
|
fail-fast: false
|
|
47
49
|
matrix:
|
|
@@ -57,7 +59,10 @@ jobs:
|
|
|
57
59
|
cache: "pip"
|
|
58
60
|
|
|
59
61
|
- name: Install package with test dependencies
|
|
60
|
-
run: python -m pip install -e ".[dev,otel-test]"
|
|
62
|
+
run: python -m pip install -e ".[dev,otel-test,http]"
|
|
63
|
+
|
|
64
|
+
- name: Verify P28 fixture manifest
|
|
65
|
+
run: python scripts/verify_p28_fixtures.py
|
|
61
66
|
|
|
62
67
|
- name: Test with branch coverage
|
|
63
68
|
run: python -m pytest --cov --cov-report=term-missing
|
|
@@ -65,6 +70,7 @@ jobs:
|
|
|
65
70
|
artifacts:
|
|
66
71
|
name: Release artifacts, extras, examples, and typing
|
|
67
72
|
runs-on: ubuntu-latest
|
|
73
|
+
timeout-minutes: 30
|
|
68
74
|
steps:
|
|
69
75
|
- name: Checkout
|
|
70
76
|
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
@@ -82,3 +88,20 @@ jobs:
|
|
|
82
88
|
|
|
83
89
|
- name: Verify distributed artifacts
|
|
84
90
|
run: python scripts/verify_release_artifacts.py
|
|
91
|
+
|
|
92
|
+
reference-vectors:
|
|
93
|
+
name: Independent Node reference check of the P28 contract fixtures
|
|
94
|
+
runs-on: ubuntu-latest
|
|
95
|
+
timeout-minutes: 10
|
|
96
|
+
steps:
|
|
97
|
+
- name: Checkout
|
|
98
|
+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
99
|
+
|
|
100
|
+
- name: Set up Node.js
|
|
101
|
+
uses: actions/setup-node@2028fbc5c25fe9cf00d9f06a71cc4710d4507903 # v6.0.0
|
|
102
|
+
with:
|
|
103
|
+
node-version: "24.19.0"
|
|
104
|
+
|
|
105
|
+
# Data-only: reads tests/fixtures/p28 with no Python import and no npm dependency.
|
|
106
|
+
- name: Verify canonical, digest and acknowledgement vectors
|
|
107
|
+
run: node scripts/verify_p28_reference.mjs
|
|
@@ -15,9 +15,20 @@ jobs:
|
|
|
15
15
|
build:
|
|
16
16
|
name: Build and verify release artifacts
|
|
17
17
|
runs-on: ubuntu-latest
|
|
18
|
+
timeout-minutes: 30
|
|
18
19
|
steps:
|
|
19
20
|
- name: Checkout
|
|
20
21
|
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
22
|
+
with:
|
|
23
|
+
fetch-depth: 0
|
|
24
|
+
|
|
25
|
+
- name: Confirm the tagged commit is on main
|
|
26
|
+
run: |
|
|
27
|
+
git fetch --no-tags origin main
|
|
28
|
+
if ! git merge-base --is-ancestor "$GITHUB_SHA" origin/main; then
|
|
29
|
+
echo "Tagged commit $GITHUB_SHA is not reachable from origin/main" >&2
|
|
30
|
+
exit 1
|
|
31
|
+
fi
|
|
21
32
|
|
|
22
33
|
- name: Set up Python 3.11
|
|
23
34
|
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
|
|
@@ -35,7 +46,7 @@ jobs:
|
|
|
35
46
|
fi
|
|
36
47
|
|
|
37
48
|
- name: Install package and build tooling
|
|
38
|
-
run: python -m pip install -e ".[dev,otel-test]" build
|
|
49
|
+
run: python -m pip install -e ".[dev,http,otel-test]" build
|
|
39
50
|
|
|
40
51
|
- name: Full quality gates
|
|
41
52
|
run: |
|
|
@@ -61,6 +72,7 @@ jobs:
|
|
|
61
72
|
name: Publish to PyPI
|
|
62
73
|
needs: build
|
|
63
74
|
runs-on: ubuntu-latest
|
|
75
|
+
timeout-minutes: 15
|
|
64
76
|
environment:
|
|
65
77
|
name: pypi
|
|
66
78
|
url: https://pypi.org/project/agent-reliability/
|
|
@@ -80,6 +92,7 @@ jobs:
|
|
|
80
92
|
name: Create GitHub release
|
|
81
93
|
needs: publish
|
|
82
94
|
runs-on: ubuntu-latest
|
|
95
|
+
timeout-minutes: 15
|
|
83
96
|
permissions:
|
|
84
97
|
contents: write
|
|
85
98
|
steps:
|
|
@@ -4,6 +4,118 @@ All notable changes to this project are documented here. Format loosely
|
|
|
4
4
|
follows [Keep a Changelog](https://keepachangelog.com/); versioning
|
|
5
5
|
follows [docs/COMPATIBILITY.md](docs/COMPATIBILITY.md).
|
|
6
6
|
|
|
7
|
+
## [Unreleased]
|
|
8
|
+
|
|
9
|
+
## [1.3.0]
|
|
10
|
+
|
|
11
|
+
A backward-compatible minor release under [docs/VERSIONING.md](docs/VERSIONING.md): an
|
|
12
|
+
optional HTTP evidence exporter and an experimental model-backed evaluator adapter. The
|
|
13
|
+
publication date is the upload date shown by the package index and the GitHub release. The
|
|
14
|
+
`1.3.0rc1` pre-release is superseded by this release; its published files are unchanged. The core installation still has no
|
|
15
|
+
runtime dependencies, and existing stable public APIs, reliability/SLO semantics and the
|
|
16
|
+
`EventSink` contract are unchanged. See [docs/RELEASE_NOTES_1_3.md](docs/RELEASE_NOTES_1_3.md)
|
|
17
|
+
for migration and rollback guidance and [docs/HTTP_EXPORTER.md](docs/HTTP_EXPORTER.md) for
|
|
18
|
+
the exporter.
|
|
19
|
+
|
|
20
|
+
### Added (experimental)
|
|
21
|
+
|
|
22
|
+
- `agent_reliability.experimental.judge`: an EXPERIMENTAL adapter for model-backed (LLM-judge)
|
|
23
|
+
evaluators that call a provider you supply. Raw judged content stays on your side:
|
|
24
|
+
only typed members (outcome, reason code, provenance, failure class, health) reach the
|
|
25
|
+
exporter; no new wire field and no new core dependency. Reference rubrics are uncalibrated
|
|
26
|
+
examples; no accuracy or calibration claim is made. It may change in a minor release with
|
|
27
|
+
release notes (see the Experimental policy in docs/VERSIONING.md).
|
|
28
|
+
|
|
29
|
+
### Added
|
|
30
|
+
|
|
31
|
+
- Optional `agent-reliability[http]` extra (`httpx>=0.28.1,<0.29`) and the
|
|
32
|
+
`agent_reliability.adapters.http_event_sink` module with `HTTPEventSink`
|
|
33
|
+
(thread-safe, two daemon delivery workers) and `AsyncHTTPEventSink`. Both export
|
|
34
|
+
evidence to one complete platform POST URL using a machine credential obtained
|
|
35
|
+
from a caller-supplied provider, always as wire schema version 2 with the asserted
|
|
36
|
+
producer identity and a canonical digest that preserves the schema-v1 bytes.
|
|
37
|
+
- Delivery and diagnostics types: `DeliveryStatusSnapshot`, `DeliveryDisposition`,
|
|
38
|
+
`DeliveryRecordStatus`, `ConfigurationError`, `ConfigurationFailedError`,
|
|
39
|
+
`EventDeliveryRejected`, `ExporterClosed`. `emit` only admits to bounded memory;
|
|
40
|
+
commitment is reported solely from a validated acknowledgement (`201`/`200`), and
|
|
41
|
+
records that may have committed are flagged.
|
|
42
|
+
- Reliability behavior: bounded queues with a closure reserve, per-run ordering,
|
|
43
|
+
bounded retries with full jitter, sticky uncertainty for requests that may have
|
|
44
|
+
left the process, sink-wide `429`/`Retry-After` handling, bounded `flush`/`close`,
|
|
45
|
+
and explicit pauses with `resume_auth()`, `resume_schema()`, `resume_deployment()`
|
|
46
|
+
and `resume_endpoint()` for authentication, compatibility, unknown Deployment and
|
|
47
|
+
endpoint problems. Held evidence is never silently dropped.
|
|
48
|
+
- Optional `HealthEvidenceSink` capability (`agent_reliability.ports.event_sink`),
|
|
49
|
+
fanned out by the composite sink, plus an optional `indicator=` argument on
|
|
50
|
+
`RunHandle.record_evaluation_failure(...)`, so evaluation failures and final
|
|
51
|
+
measurement health reach exporters. Existing `EventSink` implementations stay valid.
|
|
52
|
+
- Run lifecycle integration: when the configured sink supports the optional
|
|
53
|
+
capability, `AgentReliability` now submits the final measurement-health report and
|
|
54
|
+
releases the exporter's run binding when a run finishes; `CompositeEventSink` fans the
|
|
55
|
+
capability out and isolates each child. Sinks that do not implement it are
|
|
56
|
+
unaffected.
|
|
57
|
+
- **Fail-closed indicator policy.** The `indicator` is the only free-text wire field, so
|
|
58
|
+
both sinks now require an explicit choice: `indicator_allowlist=[...]` (only these exact
|
|
59
|
+
names may leave the process; others are rejected before queueing and never logged) or the
|
|
60
|
+
deliberate `allow_unrestricted_indicators=True`. A sink given neither raises
|
|
61
|
+
`ConfigurationError`. This is a behaviour change relative to the `1.3.0rc1` pre-release;
|
|
62
|
+
see the migration notes in [docs/RELEASE_NOTES_1_3.md](docs/RELEASE_NOTES_1_3.md).
|
|
63
|
+
- Security posture: certificate and hostname verification are always on; redirects
|
|
64
|
+
are never followed; plaintext HTTP is accepted only for literal loopback
|
|
65
|
+
(`127.0.0.0/8`, `[::1]`); ambient proxy/CA/cookie/netrc configuration is ignored;
|
|
66
|
+
explicit `proxy` and `ca_bundle` options; credentials, URLs and evidence never
|
|
67
|
+
appear in diagnostics.
|
|
68
|
+
- Data-only exporter contract fixtures under `tests/fixtures/`, with a Python verifier
|
|
69
|
+
(`scripts/verify_p28_fixtures.py`) and a Node reference check
|
|
70
|
+
(`scripts/verify_p28_reference.mjs`).
|
|
71
|
+
|
|
72
|
+
### Fixed
|
|
73
|
+
|
|
74
|
+
- Before this exporter is released, its TLS handling was corrected: with the real
|
|
75
|
+
`httpx`/`httpcore` stack a certificate-verification failure is visible only through
|
|
76
|
+
the exception's `__context__`, which the first implementation did not follow, so
|
|
77
|
+
repeated verification failures exhausted retries instead of pausing the endpoint. The
|
|
78
|
+
classifier now walks `__cause__` and `__context__` (bounded, cycle-safe) and
|
|
79
|
+
recognizes a verification failure only for an attempt that had not yet sent a
|
|
80
|
+
request. The third consecutive verification failure pauses endpoint delivery and
|
|
81
|
+
holds the evidence for `resume_endpoint()`; other network failures keep their ordinary
|
|
82
|
+
classification and neither count nor reset the streak (any completed HTTP exchange
|
|
83
|
+
or `resume_endpoint()` resets it). Through an explicit `proxy`, the tunnel-opening
|
|
84
|
+
CONNECT request no longer counts as having sent the evidence request. A response body
|
|
85
|
+
trickled past 30 seconds is abandoned and retried instead of pinning a worker.
|
|
86
|
+
|
|
87
|
+
### Changed
|
|
88
|
+
|
|
89
|
+
- Documentation: the exporter classification is recorded in `docs/GA_CONTRACT.md`
|
|
90
|
+
(see that file for the authoritative class; the documented consumer entry points are
|
|
91
|
+
named in `docs/HTTP_EXPORTER.md`).
|
|
92
|
+
- The source distribution no longer includes `CLAUDE.md`, `AGENTS.md` or the internal
|
|
93
|
+
internal review/execution reports. `verify_release_artifacts.py` now also fails if either
|
|
94
|
+
artifact contains private-key material (PEM/PGP blocks, key-like file names), local developer
|
|
95
|
+
paths (Unix, macOS temp and Windows forms), or internal instruction/report files, and it is
|
|
96
|
+
covered by planted-case tests. Public documentation, the licence and all packaging inputs remain.
|
|
97
|
+
- Idle exporter workers no longer build a full status snapshot every 10 ms (idle cost
|
|
98
|
+
with 900 history rows fell from about 2.7% to about 0.4% of one core).
|
|
99
|
+
- CI and release workflows and `CONTRIBUTING.md` install the `http` extra so the
|
|
100
|
+
exporter tests and type checks run.
|
|
101
|
+
- Release validation (`scripts/verify_release_artifacts.py`) also fails if either
|
|
102
|
+
artifact contains private-key material. The real-TLS regression tests generate
|
|
103
|
+
disposable certificates at test time (development-only dependency `cryptography`
|
|
104
|
+
in the `dev` extra); no key is stored in the repository or shipped.
|
|
105
|
+
|
|
106
|
+
## [1.2.1] - 2026-09-08
|
|
107
|
+
|
|
108
|
+
Documentation-only patch release. No production code, public API, or
|
|
109
|
+
runtime behavior changes.
|
|
110
|
+
|
|
111
|
+
### Documentation
|
|
112
|
+
|
|
113
|
+
- Reworked the public project description to focus on developer-facing
|
|
114
|
+
capabilities instead of internal milestone names.
|
|
115
|
+
- Clarified the standalone/local-first usage model and zero mandatory
|
|
116
|
+
runtime dependency guarantee.
|
|
117
|
+
- Improved first-run discoverability on PyPI/GitHub.
|
|
118
|
+
|
|
7
119
|
## [1.2.0] - 2026-09-08
|
|
8
120
|
|
|
9
121
|
Developer-adoption additive release ("DX-1 — Developer Adoption").
|
|
@@ -12,7 +12,7 @@ Requires Python 3.11–3.13.
|
|
|
12
12
|
python -m venv .venv
|
|
13
13
|
# POSIX: source .venv/bin/activate
|
|
14
14
|
# Windows PowerShell: .venv\Scripts\Activate.ps1
|
|
15
|
-
python -m pip install -e ".[dev]"
|
|
15
|
+
python -m pip install -e ".[dev,http,otel-test]"
|
|
16
16
|
```
|
|
17
17
|
|
|
18
18
|
Branches use `<type>/<short-description>`, for example
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: agent-reliability
|
|
3
|
-
Version: 1.
|
|
3
|
+
Version: 1.3.0
|
|
4
4
|
Summary: Vendor-neutral reliability engineering for autonomous AI agents.
|
|
5
5
|
Project-URL: Homepage, https://github.com/kamleshd07/agent-reliability
|
|
6
6
|
Project-URL: Documentation, https://github.com/kamleshd07/agent-reliability/tree/main/docs
|
|
@@ -20,12 +20,15 @@ Classifier: Typing :: Typed
|
|
|
20
20
|
Requires-Python: >=3.11
|
|
21
21
|
Provides-Extra: dev
|
|
22
22
|
Requires-Dist: build>=1.2; extra == 'dev'
|
|
23
|
+
Requires-Dist: cryptography>=42; extra == 'dev'
|
|
23
24
|
Requires-Dist: hypothesis>=6.108; extra == 'dev'
|
|
24
25
|
Requires-Dist: mypy>=1.11; extra == 'dev'
|
|
25
26
|
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
|
|
26
27
|
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
|
|
27
28
|
Requires-Dist: pytest>=8.0; extra == 'dev'
|
|
28
29
|
Requires-Dist: ruff>=0.6; extra == 'dev'
|
|
30
|
+
Provides-Extra: http
|
|
31
|
+
Requires-Dist: httpx<0.29,>=0.28.1; extra == 'http'
|
|
29
32
|
Provides-Extra: otel
|
|
30
33
|
Requires-Dist: opentelemetry-api<2,>=1.44; extra == 'otel'
|
|
31
34
|
Provides-Extra: otel-test
|
|
@@ -42,7 +45,7 @@ It brings evaluations, SLOs, error budgets, burn rates, and measurement
|
|
|
42
45
|
provenance to agent applications—and refuses to produce a misleading number
|
|
43
46
|
when evaluation methodologies are incompatible.
|
|
44
47
|
|
|
45
|
-
**Status: GA (`1.
|
|
48
|
+
**Status: GA (`1.3.0` is the latest stable release; the HTTP exporter and the LLM-judge adapter are Experimental, see [release notes](https://github.com/kamleshd07/agent-reliability/blob/main/docs/RELEASE_NOTES_1_3.md)).** Public APIs documented as stable in
|
|
46
49
|
[GA_CONTRACT.md](https://github.com/kamleshd07/agent-reliability/blob/v1.0.0/docs/GA_CONTRACT.md) follow Semantic Versioning.
|
|
47
50
|
See [compatibility](https://github.com/kamleshd07/agent-reliability/blob/v1.0.0/docs/COMPATIBILITY.md).
|
|
48
51
|
|
|
@@ -189,19 +192,37 @@ none of them and exports nothing by itself. See the
|
|
|
189
192
|
|
|
190
193
|
## Project status and scope
|
|
191
194
|
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
the adoption path and installed-artifact verification. M7 defines the GA
|
|
195
|
-
contract and release gates, released as `1.0.0` after `1.0.0rc1` was
|
|
196
|
-
published and independently reinstalled from PyPI. M8 adds run-scoped
|
|
197
|
-
measurement health and an application-owned policy extension boundary in
|
|
198
|
-
`1.1.0`; M9 hardens that contract with adversarial coverage and real-world
|
|
199
|
-
policy examples in `1.1.1`. DX-1 — Developer Adoption adds built-in
|
|
200
|
-
evaluators, ergonomic re-exports, local reporting, and a framework-neutral
|
|
201
|
-
SLO assertion helper in `1.2.0`, additive only.
|
|
195
|
+
Agent Reliability is a stable, local-first Python SDK for measuring and
|
|
196
|
+
testing the reliability of AI agents.
|
|
202
197
|
|
|
203
|
-
|
|
204
|
-
|
|
198
|
+
The SDK provides sync and async agent instrumentation, deterministic
|
|
199
|
+
evaluators, evaluator provenance, reliability aggregation, SLO and
|
|
200
|
+
error-budget semantics, measurement-health tracking, optional
|
|
201
|
+
OpenTelemetry interoperability, human-readable local reports,
|
|
202
|
+
machine-readable reliability results, and framework-neutral SLO
|
|
203
|
+
assertions for tests and CI.
|
|
204
|
+
|
|
205
|
+
The SDK is designed to work standalone. It requires no hosted account,
|
|
206
|
+
API key, or network service, and has zero mandatory runtime dependencies.
|
|
207
|
+
|
|
208
|
+
**Current stable release: 1.3.0**
|
|
209
|
+
|
|
210
|
+
The public API follows semantic versioning, with compatibility and
|
|
211
|
+
installed-artifact verification included in the release process.
|
|
212
|
+
|
|
213
|
+
Releases before 1.3 include no remote ingestion, dashboard, LLM judge, persistence,
|
|
214
|
+
auto-instrumentation, or framework-specific adapter. Release 1.3 adds an optional
|
|
215
|
+
**Experimental** HTTP evidence exporter (`pip install 'agent-reliability[http]'`, see
|
|
216
|
+
[HTTP exporter](https://github.com/kamleshd07/agent-reliability/blob/main/docs/HTTP_EXPORTER.md)) and an
|
|
217
|
+
Experimental LLM-judge adapter (described below); it still has no dashboard, persistence or
|
|
218
|
+
auto-instrumentation. See the [roadmap](https://github.com/kamleshd07/agent-reliability/blob/main/docs/ROADMAP.md).
|
|
219
|
+
|
|
220
|
+
### LLM-judge evaluators (experimental)
|
|
221
|
+
|
|
222
|
+
`agent_reliability.experimental.judge` is an Experimental adapter for model-backed evaluators that call a provider you
|
|
223
|
+
supply. It adds no dependency and sends raw content only to your provider. The reference rubrics are uncalibrated examples. See
|
|
224
|
+
[Evaluate](https://github.com/kamleshd07/agent-reliability/blob/main/docs/EVALUATE.md).
|
|
225
|
+
A sync judge's timeout is best effort unless the provider is wrapped in `ExecutorSyncProvider`; the provider's own client timeout is the real bound (see the same page).
|
|
205
226
|
|
|
206
227
|
## Documentation
|
|
207
228
|
|
|
@@ -223,3 +244,9 @@ not a public issue.
|
|
|
223
244
|
## License
|
|
224
245
|
|
|
225
246
|
Apache License 2.0. See [LICENSE](https://github.com/kamleshd07/agent-reliability/blob/v1.0.0/LICENSE).
|
|
247
|
+
|
|
248
|
+
### HTTP evidence delivery
|
|
249
|
+
|
|
250
|
+
The optional [HTTP exporter](https://github.com/kamleshd07/agent-reliability/blob/main/docs/HTTP_EXPORTER.md) sends schema-v2 evidence
|
|
251
|
+
using `agent-reliability[http]`, bounded delivery queues and explicit shutdown.
|
|
252
|
+
Platform integration and independent review remain required before release.
|
|
@@ -8,7 +8,7 @@ It brings evaluations, SLOs, error budgets, burn rates, and measurement
|
|
|
8
8
|
provenance to agent applications—and refuses to produce a misleading number
|
|
9
9
|
when evaluation methodologies are incompatible.
|
|
10
10
|
|
|
11
|
-
**Status: GA (`1.
|
|
11
|
+
**Status: GA (`1.3.0` is the latest stable release; the HTTP exporter and the LLM-judge adapter are Experimental, see [release notes](https://github.com/kamleshd07/agent-reliability/blob/main/docs/RELEASE_NOTES_1_3.md)).** Public APIs documented as stable in
|
|
12
12
|
[GA_CONTRACT.md](https://github.com/kamleshd07/agent-reliability/blob/v1.0.0/docs/GA_CONTRACT.md) follow Semantic Versioning.
|
|
13
13
|
See [compatibility](https://github.com/kamleshd07/agent-reliability/blob/v1.0.0/docs/COMPATIBILITY.md).
|
|
14
14
|
|
|
@@ -155,19 +155,37 @@ none of them and exports nothing by itself. See the
|
|
|
155
155
|
|
|
156
156
|
## Project status and scope
|
|
157
157
|
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
the adoption path and installed-artifact verification. M7 defines the GA
|
|
161
|
-
contract and release gates, released as `1.0.0` after `1.0.0rc1` was
|
|
162
|
-
published and independently reinstalled from PyPI. M8 adds run-scoped
|
|
163
|
-
measurement health and an application-owned policy extension boundary in
|
|
164
|
-
`1.1.0`; M9 hardens that contract with adversarial coverage and real-world
|
|
165
|
-
policy examples in `1.1.1`. DX-1 — Developer Adoption adds built-in
|
|
166
|
-
evaluators, ergonomic re-exports, local reporting, and a framework-neutral
|
|
167
|
-
SLO assertion helper in `1.2.0`, additive only.
|
|
158
|
+
Agent Reliability is a stable, local-first Python SDK for measuring and
|
|
159
|
+
testing the reliability of AI agents.
|
|
168
160
|
|
|
169
|
-
|
|
170
|
-
|
|
161
|
+
The SDK provides sync and async agent instrumentation, deterministic
|
|
162
|
+
evaluators, evaluator provenance, reliability aggregation, SLO and
|
|
163
|
+
error-budget semantics, measurement-health tracking, optional
|
|
164
|
+
OpenTelemetry interoperability, human-readable local reports,
|
|
165
|
+
machine-readable reliability results, and framework-neutral SLO
|
|
166
|
+
assertions for tests and CI.
|
|
167
|
+
|
|
168
|
+
The SDK is designed to work standalone. It requires no hosted account,
|
|
169
|
+
API key, or network service, and has zero mandatory runtime dependencies.
|
|
170
|
+
|
|
171
|
+
**Current stable release: 1.3.0**
|
|
172
|
+
|
|
173
|
+
The public API follows semantic versioning, with compatibility and
|
|
174
|
+
installed-artifact verification included in the release process.
|
|
175
|
+
|
|
176
|
+
Releases before 1.3 include no remote ingestion, dashboard, LLM judge, persistence,
|
|
177
|
+
auto-instrumentation, or framework-specific adapter. Release 1.3 adds an optional
|
|
178
|
+
**Experimental** HTTP evidence exporter (`pip install 'agent-reliability[http]'`, see
|
|
179
|
+
[HTTP exporter](https://github.com/kamleshd07/agent-reliability/blob/main/docs/HTTP_EXPORTER.md)) and an
|
|
180
|
+
Experimental LLM-judge adapter (described below); it still has no dashboard, persistence or
|
|
181
|
+
auto-instrumentation. See the [roadmap](https://github.com/kamleshd07/agent-reliability/blob/main/docs/ROADMAP.md).
|
|
182
|
+
|
|
183
|
+
### LLM-judge evaluators (experimental)
|
|
184
|
+
|
|
185
|
+
`agent_reliability.experimental.judge` is an Experimental adapter for model-backed evaluators that call a provider you
|
|
186
|
+
supply. It adds no dependency and sends raw content only to your provider. The reference rubrics are uncalibrated examples. See
|
|
187
|
+
[Evaluate](https://github.com/kamleshd07/agent-reliability/blob/main/docs/EVALUATE.md).
|
|
188
|
+
A sync judge's timeout is best effort unless the provider is wrapped in `ExecutorSyncProvider`; the provider's own client timeout is the real bound (see the same page).
|
|
171
189
|
|
|
172
190
|
## Documentation
|
|
173
191
|
|
|
@@ -189,3 +207,9 @@ not a public issue.
|
|
|
189
207
|
## License
|
|
190
208
|
|
|
191
209
|
Apache License 2.0. See [LICENSE](https://github.com/kamleshd07/agent-reliability/blob/v1.0.0/LICENSE).
|
|
210
|
+
|
|
211
|
+
### HTTP evidence delivery
|
|
212
|
+
|
|
213
|
+
The optional [HTTP exporter](https://github.com/kamleshd07/agent-reliability/blob/main/docs/HTTP_EXPORTER.md) sends schema-v2 evidence
|
|
214
|
+
using `agent-reliability[http]`, bounded delivery queues and explicit shutdown.
|
|
215
|
+
Platform integration and independent review remain required before release.
|
|
@@ -194,3 +194,18 @@ optional OTel ownership remain unchanged. See ADR-0009 and
|
|
|
194
194
|
ADR-0001 ([docs/adr/0001-architecture-boundaries.md](adr/0001-architecture-boundaries.md))
|
|
195
195
|
records only the layering and dependency-direction decision made at M0.
|
|
196
196
|
The items above are explicitly not decided yet.
|
|
197
|
+
|
|
198
|
+
## Language-neutral public HTTP contracts
|
|
199
|
+
|
|
200
|
+
Gentlity public ingestion and control-plane APIs are defined by the platform's
|
|
201
|
+
published OpenAPI, referenced JSON Schemas and normative protocol documentation,
|
|
202
|
+
not by this Python package's dataclasses, exceptions, enum internals or
|
|
203
|
+
serialization defaults. The SDK must validate and emit the documented wire
|
|
204
|
+
types and versions, authenticate through standard HTTP, classify responses by
|
|
205
|
+
HTTP status/header and stable error code, and obey documented idempotency,
|
|
206
|
+
retry, rate-limit, `Retry-After`, timeout and compatibility rules.
|
|
207
|
+
|
|
208
|
+
Shared conformance assets must be data-only and reusable by Python, Go, Java and
|
|
209
|
+
TypeScript. The SDK and platform execute the same positive, negative, boundary,
|
|
210
|
+
request/response and digest fixtures. Schema-v1 canonical digest bytes and
|
|
211
|
+
goldens are frozen across upgrades; SDK behavior cannot redefine them.
|
|
@@ -31,16 +31,25 @@ and [OTEL_MAPPING.md](OTEL_MAPPING.md).
|
|
|
31
31
|
|
|
32
32
|
## Additive measurement-health extension
|
|
33
33
|
|
|
34
|
-
|
|
34
|
+
The measurement-health API adds `agent_reliability.measurement` and additive `RunHandle` inspection
|
|
35
35
|
and policy members. Existing 1.0 export sets, callable signatures, enum
|
|
36
36
|
semantics, event payloads, and SLO/reliability calculations are unchanged.
|
|
37
37
|
This public addition ships in 1.1.0 as a backward-compatible minor release.
|
|
38
|
-
|
|
38
|
+
The patch release adds tests, examples, documentation, and artifact checks only; the `1.1.1`
|
|
39
39
|
release leaves the `1.1.0` public and semantic contract unchanged.
|
|
40
40
|
|
|
41
|
+
## Additive HTTP exporter extension (1.3.0)
|
|
42
|
+
|
|
43
|
+
1.3 adds the optional `agent-reliability[http]` extra and the HTTP exporter entry points,
|
|
44
|
+
classed EXPERIMENTAL (see [GA_CONTRACT.md](GA_CONTRACT.md)), plus the experimental judge
|
|
45
|
+
adapter. Existing 1.0-1.2 stable export sets, callable signatures, enum semantics, event
|
|
46
|
+
payloads and SLO/reliability calculations are unchanged, and base runtime dependencies remain
|
|
47
|
+
zero. This ships as a backward-compatible minor release (`1.3.0`); the earlier `1.3.0rc1`
|
|
48
|
+
pre-release is superseded. Migration and rollback: [RELEASE_NOTES_1_3.md](RELEASE_NOTES_1_3.md).
|
|
49
|
+
|
|
41
50
|
## Additive developer-adoption extension
|
|
42
51
|
|
|
43
|
-
|
|
52
|
+
The developer-adoption extension adds built-in evaluators and an `EvaluationOutcome`
|
|
44
53
|
re-export to `agent_reliability.evaluation`, `AggregationConflictError`/
|
|
45
54
|
`evaluate_reliability_or_raise` to `agent_reliability.reliability`, and two
|
|
46
55
|
new stable namespaces, `agent_reliability.reporting` and
|
|
@@ -51,7 +51,7 @@ itself change the measurement—it explains how much tolerance remains.
|
|
|
51
51
|
## Burn rate
|
|
52
52
|
|
|
53
53
|
Observed bad-event rate divided by the SLO's allowed bad-event rate. It is not
|
|
54
|
-
“current failures divided by total failures.”
|
|
54
|
+
“current failures divided by total failures.” The local reliability engine calculates a burn rate only
|
|
55
55
|
when the caller supplies an explicit lookback ratio; it does not retain data or
|
|
56
56
|
select a time window.
|
|
57
57
|
|
|
@@ -78,7 +78,7 @@ AutoGen, or any other framework to be present.
|
|
|
78
78
|
|
|
79
79
|
## 10. Dependency discipline
|
|
80
80
|
|
|
81
|
-
Runtime dependencies are minimal
|
|
81
|
+
Runtime dependencies are minimal, with zero mandatory core dependencies. Every
|
|
82
82
|
runtime dependency must justify itself in the PR that introduces it —
|
|
83
83
|
"it's convenient" is not sufficient justification.
|
|
84
84
|
|