agentic-evals 0.7.2__tar.gz → 0.9.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (138) hide show
  1. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/.gitignore +1 -0
  2. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/CHANGELOG.md +87 -0
  3. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/PKG-INFO +206 -1
  4. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/README.md +204 -0
  5. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/ROADMAP.md +155 -21
  6. agentic_evals-0.9.0/docs/chaos-0.7.md +18 -0
  7. agentic_evals-0.9.0/docs/chaos-0.8.md +27 -0
  8. agentic_evals-0.9.0/docs/cli.md +121 -0
  9. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/docs/docs/index.html +24 -0
  10. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/docs/index.html +3 -3
  11. agentic_evals-0.9.0/docs/judge-reliability.md +225 -0
  12. agentic_evals-0.9.0/docs/lens-0.10-routing.md +16 -0
  13. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/docs/report-contract.md +36 -3
  14. agentic_evals-0.9.0/docs/tower-0.7-fleet-operations.md +26 -0
  15. agentic_evals-0.9.0/examples/async_live_target.py +68 -0
  16. agentic_evals-0.9.0/examples/ci/evaluate.yml +30 -0
  17. agentic_evals-0.9.0/examples/ci/failing-samples.jsonl +1 -0
  18. agentic_evals-0.9.0/examples/ci/gate.yaml +2 -0
  19. agentic_evals-0.9.0/examples/ci/samples.jsonl +1 -0
  20. agentic_evals-0.9.0/examples/ci/suite.yaml +8 -0
  21. agentic_evals-0.9.0/examples/judge_reliability.py +116 -0
  22. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/examples/report-contract/evaluation-report.json +11 -3
  23. agentic_evals-0.9.0/examples/report-contract/generate.py +130 -0
  24. agentic_evals-0.9.0/examples/report-contract/judged-live-gate-decision.json +16 -0
  25. agentic_evals-0.9.0/examples/report-contract/judged-live-report.json +219 -0
  26. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/pyproject.toml +2 -1
  27. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/scripts/check_installed_contract.py +3 -0
  28. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/scripts/generate_schemas.py +1 -1
  29. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/__init__.py +15 -0
  30. agentic_evals-0.9.0/src/agentic_evals/_async.py +51 -0
  31. agentic_evals-0.9.0/src/agentic_evals/_version.py +1 -0
  32. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/cli.py +7 -2
  33. agentic_evals-0.9.0/src/agentic_evals/cli_commands.py +168 -0
  34. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/evaluators.py +71 -8
  35. agentic_evals-0.9.0/src/agentic_evals/formatting.py +153 -0
  36. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/gate.py +86 -0
  37. agentic_evals-0.9.0/src/agentic_evals/judging.py +333 -0
  38. agentic_evals-0.9.0/src/agentic_evals/live.py +261 -0
  39. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/models.py +68 -0
  40. agentic_evals-0.9.0/src/agentic_evals/reliability.py +133 -0
  41. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/runner.py +367 -138
  42. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/schemas/evaluation-report-1.0.schema.json +236 -2
  43. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/schemas/gate-decision-1.0.schema.json +1 -1
  44. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/scorers/rubric.py +66 -13
  45. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/debug-a-flaky-llm-judge/SKILL.md +15 -7
  46. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/define-a-release-gate/SKILL.md +13 -0
  47. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/run-a-live-suite/SKILL.md +21 -8
  48. agentic_evals-0.9.0/tests/test_async_evaluation.py +231 -0
  49. agentic_evals-0.9.0/tests/test_cli_declarative.py +212 -0
  50. agentic_evals-0.9.0/tests/test_cli_examples.py +73 -0
  51. agentic_evals-0.9.0/tests/test_judge_reliability.py +402 -0
  52. agentic_evals-0.9.0/tests/test_live_resilience.py +356 -0
  53. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_live_targets.py +42 -3
  54. agentic_evals-0.9.0/tests/test_reliability.py +99 -0
  55. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_report_contract.py +52 -0
  56. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_scorers_rubric.py +7 -3
  57. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/uv.lock +12 -1
  58. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/README.md +17 -0
  59. agentic_evals-0.9.0/voice-agent-evals/export_cli.py +27 -0
  60. agentic_evals-0.7.2/examples/report-contract/generate.py +0 -51
  61. agentic_evals-0.7.2/src/agentic_evals/_version.py +0 -1
  62. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/.github/ISSUE_TEMPLATE/bug_report.md +0 -0
  63. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/.github/ISSUE_TEMPLATE/config.yml +0 -0
  64. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/.github/ISSUE_TEMPLATE/feature_request.md +0 -0
  65. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/.github/PULL_REQUEST_TEMPLATE.md +0 -0
  66. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/.github/workflows/ci.yml +0 -0
  67. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/.github/workflows/pragent.yml +0 -0
  68. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/.github/workflows/release-pypi.yml +0 -0
  69. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/.github/workflows/release-testpypi.yml +0 -0
  70. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/CONTRIBUTING.md +0 -0
  71. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/LICENSE +0 -0
  72. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/SECURITY.md +0 -0
  73. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/docs/control-tower-0.6.md +0 -0
  74. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/docs/sidecar-v0.8.md +0 -0
  75. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/examples/capitals_eval.py +0 -0
  76. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/examples/eval_pack.py +0 -0
  77. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/examples/llm_rubric_judge.py +0 -0
  78. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/examples/near_miss_similarity.py +0 -0
  79. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/examples/quickstart.py +0 -0
  80. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/examples/report-contract/gate-decision.json +0 -0
  81. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/packs/__init__.py +0 -0
  82. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/packs/builtin/factual-qa.yaml +0 -0
  83. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/packs/builtin/json-output-contract.yaml +0 -0
  84. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/packs/builtin/tool-use-correctness.yaml +0 -0
  85. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/packs/library.py +0 -0
  86. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/packs/loader.py +0 -0
  87. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/py.typed +0 -0
  88. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/report_contract.py +0 -0
  89. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/scorers/__init__.py +0 -0
  90. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/scorers/registry.py +0 -0
  91. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/scorers/text.py +0 -0
  92. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/scorers/trajectory.py +0 -0
  93. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/simple.py +0 -0
  94. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/analyze-an-eval-experiment/SKILL.md +0 -0
  95. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/build-an-eval-dataset/SKILL.md +0 -0
  96. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/choose-a-rubric-template/SKILL.md +0 -0
  97. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/define-an-eval-objective/SKILL.md +0 -0
  98. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/design-an-eval-experiment/SKILL.md +0 -0
  99. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/discover-failure-modes/SKILL.md +0 -0
  100. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/elicit-eval-criteria/SKILL.md +0 -0
  101. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/instrument-a-trace/SKILL.md +0 -0
  102. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/monitor-evals-in-production/SKILL.md +0 -0
  103. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/red-team-an-agent-suite/SKILL.md +0 -0
  104. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/report-eval-results/SKILL.md +0 -0
  105. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/size-a-test-suite/SKILL.md +0 -0
  106. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/validate-a-scorer/SKILL.md +0 -0
  107. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/write-a-scorer/SKILL.md +0 -0
  108. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/fixtures/v0.7.1/README.md +0 -0
  109. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/fixtures/v0.7.1/evaluation-report.json +0 -0
  110. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/fixtures/v0.7.1/evaluation-report.schema.json +0 -0
  111. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/fixtures/v0.7.1/gate-decision.json +0 -0
  112. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/fixtures/v0.7.1/gate-decision.schema.json +0 -0
  113. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/live_eval_target.py +0 -0
  114. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_cli.py +0 -0
  115. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_evaluators.py +0 -0
  116. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_gate.py +0 -0
  117. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_models.py +0 -0
  118. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_packs.py +0 -0
  119. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_provenance.py +0 -0
  120. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_report_breakdowns.py +0 -0
  121. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_runner.py +0 -0
  122. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_scorers_registry.py +0 -0
  123. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_scorers_text.py +0 -0
  124. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_scorers_trajectory.py +0 -0
  125. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_simple.py +0 -0
  126. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_skills.py +0 -0
  127. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_tool_expectations.py +0 -0
  128. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/.gitignore +0 -0
  129. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/config.js +0 -0
  130. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/dataset.py +0 -0
  131. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/evaluators.py +0 -0
  132. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/index.html +0 -0
  133. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/run_evals.py +0 -0
  134. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/scenario_runner.py +0 -0
  135. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/scenarios.py +0 -0
  136. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/server.py +0 -0
  137. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/voice_agents.py +0 -0
  138. {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/voice_test_suite.yaml +0 -0
@@ -11,3 +11,4 @@ htmlcov/
11
11
  dist/
12
12
  build/
13
13
  .env
14
+ voice-agent-evals/cli-output/
@@ -4,6 +4,93 @@ All notable changes to this project will be documented here.
4
4
 
5
5
  This project follows [Semantic Versioning](https://semver.org/).
6
6
 
7
+ ## 0.9.0 - 2026-10-19
8
+
9
+ ### Added
10
+
11
+ - Judge cost and latency on every judge score, under the `judge_*` metadata
12
+ keys AgenticLens already reads (`judge_cost_usd`, `judge_cost_source`,
13
+ `judge_input_tokens`, `judge_output_tokens`, `judge_latency_ms`,
14
+ `judge_model`, `judge_provider`), plus `judge_status` and `judge_attempts`.
15
+ A rubric `complete_fn` reports usage by returning a `JudgeResponse`.
16
+ - `EvaluationSummary.judge` (`JudgeSummary`): judge scores, unavailable scores
17
+ and the cases they belong to, retries, cost total (only when every judge score
18
+ is priced), cost sources, tokens, latency and sampling agreement. Judge spend
19
+ stays separate from the evaluated system's `total_cost_usd`.
20
+ - `RetryPolicy`: bounded attempts with capped exponential backoff, used by
21
+ judges (`retry=` on `LLMJudgeEvaluator`/`LLMRubricEvaluator`) and live
22
+ targets (`retry=` on `run_live_suite`/`arun_live_suite`).
23
+ - An explicit unavailable judge outcome: when every attempt fails, the score
24
+ has value 0, never passes at any threshold, and records `judge_error`.
25
+ - Repeated judging with `samples=N`: mean value, most common verdict (ties go
26
+ to the lower verdict), `judge_verdicts` and `judge_agreement` per score, and
27
+ `min_agreement`/`case_agreement` in `summary.judge`.
28
+ - Live-target outcomes per case: `CaseEvaluation.target_status` (`ok`,
29
+ `timed_out`, `error`), `target_attempts` and `target_error`, a `target_call`
30
+ score, and `EvaluationSummary.live` (`LiveSummary`).
31
+ - Python target timeouts when `timeout_seconds` is set explicitly; `async def`
32
+ targets are cancelled, synchronous ones are abandoned in daemon threads.
33
+ - Async evaluation: `aevaluate_suite`, `arun_live_suite`, `aevaluate()` on
34
+ `CallableEvaluator` and its subclasses, the `AsyncEvaluator` protocol, and
35
+ `async def` judges, rubric `complete_fn`s and Python targets (also accepted
36
+ by the sync APIs outside a running event loop).
37
+ - Gate checks: `max_judge_unavailable_scores` (default 0), `max_judge_cost_usd`,
38
+ `min_judge_agreement`, `max_timed_out_cases` and `max_target_error_cases`.
39
+ Reasons name the affected cases.
40
+ - CLI `live --retries`, `--retry-backoff`, and `--timeout-seconds` for Python
41
+ targets. JUnit reports timed-out/errored live cases as `<error>`; Markdown
42
+ shows `TIMEOUT`/`ERROR` and judge/live summaries.
43
+ - `docs/judge-reliability.md`, offline `examples/judge_reliability.py` and
44
+ `examples/async_live_target.py`, a judged live contract example, and
45
+ updated skills for flaky judges, live suites and release gates.
46
+
47
+ ### Changed
48
+
49
+ - A judge that raises, or a rubric completion with no parseable verdict, now
50
+ yields an unavailable score instead of raising out of `evaluate_suite`.
51
+ - A live-target call that raises or times out (including an unreachable HTTP
52
+ target) is reported on its case instead of aborting `run_live_suite`;
53
+ `agentic-evals live` writes the report and exits 0. A case is `timed_out` only
54
+ when every attempt timed out. An invalid target payload, including a non-JSON
55
+ HTTP body, still stops the run without retrying (exit 2).
56
+ - Usage reported by a rubric attempt with no parseable verdict is added to the
57
+ score's judge cost and tokens.
58
+ - The default gate fails when any judge score is unavailable, even for an
59
+ optional evaluator. Set `max_judge_unavailable_scores: null` to opt out.
60
+ - `LLMRubricEvaluator` now subclasses `LLMJudgeEvaluator`.
61
+
62
+ ### Compatibility
63
+
64
+ - Report schema 1.0 gains optional fields only (`summary.judge`, `summary.live`,
65
+ `cases[].target_*`); packaged schema IDs now point at `v0.9.0`. Reports saved
66
+ by earlier versions load unchanged, and 0.9.0 reports validate against the
67
+ frozen 0.7.1 schemas, so readers bundling 0.7.2 schemas keep working.
68
+ - Lens's `agentic-evals>=0.7.1,<1` range already covers 0.9.0 and its tests pass
69
+ unchanged. MCP and Control Tower sync their bundled schemas and test pins to
70
+ 0.9.0 in patch releases. AIOS evidence export remains v0.10.
71
+
72
+ ## 0.8.0 - 2026-10-17
73
+
74
+ ### Added
75
+
76
+ - Declarative `evaluate`, Python/HTTP `live`, and `gate` commands with explicit
77
+ exit codes, custom registry imports, concurrency and JSONL samples.
78
+ - Built-in `pack list` and `pack show`, including plain suite export without
79
+ requiring judge credentials for inspection.
80
+ - Native JSON, JUnit XML and escaped Markdown output for reports and gates.
81
+ - Passing/failing CI fixtures, a GitHub Actions workflow, CLI documentation and
82
+ voice-agent dataset export with the same custom evaluators as the Python path.
83
+ - CLI parity, process exit-code, malformed-input and presentation tests.
84
+
85
+ ### Compatibility
86
+
87
+ - Existing `run` behavior and Python scoring/gating APIs are preserved. Native
88
+ report schema 1.0 and gate wire shapes are unchanged from 0.7.2.
89
+ - `evaluate`/`live` return 0 when evidence is produced; use `gate` to fail CI
90
+ according to policy (0 pass, 1 failed gate, 2 input/usage error).
91
+ - Sibling runtime dependency floors are unchanged; Lens, MCP and Tower consume
92
+ native JSON. AIOS evidence export remains future work.
93
+
7
94
  ## 0.7.2 - 2026-10-17
8
95
 
9
96
  ### Added
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: agentic-evals
3
- Version: 0.7.2
3
+ Version: 0.9.0
4
4
  Summary: A standalone, framework-agnostic evaluation and scoring engine for LLM/agent outputs — extracted from AgenticLens's evaluation module.
5
5
  Project-URL: Homepage, https://github.com/DeepAgentLabs/agentic-evals
6
6
  Project-URL: Repository, https://github.com/DeepAgentLabs/agentic-evals
@@ -46,6 +46,7 @@ Requires-Dist: pyyaml<7,>=6.0
46
46
  Requires-Dist: referencing>=0.35
47
47
  Provides-Extra: dev
48
48
  Requires-Dist: build>=1.2; extra == 'dev'
49
+ Requires-Dist: junitparser>=3; extra == 'dev'
49
50
  Requires-Dist: mypy>=1.10; extra == 'dev'
50
51
  Requires-Dist: pytest-cov>=5.0; extra == 'dev'
51
52
  Requires-Dist: pytest>=8.0; extra == 'dev'
@@ -83,6 +84,55 @@ quickstart API, and the `agentic-evals` CLI) is real, tested code — see the
83
84
  still move between minor versions ahead of a 1.0; pin a version in
84
85
  production and read the changelog before upgrading.
85
86
 
87
+ ## Judge reliability and live resilience (0.9.0)
88
+
89
+ LLM-judged and live evaluations now say what they cost and whether they
90
+ actually happened:
91
+
92
+ ```python
93
+ from agentic_evals import FACTUALITY, JudgeResponse, LLMRubricEvaluator, RetryPolicy
94
+
95
+ judge = LLMRubricEvaluator(
96
+ "factuality",
97
+ FACTUALITY,
98
+ complete_fn, # may return JudgeResponse(text=..., cost_usd=...)
99
+ retry=RetryPolicy(max_attempts=3, backoff_seconds=0.5),
100
+ samples=3, # judge three times, report agreement
101
+ )
102
+ ```
103
+
104
+ - Judge cost, tokens and latency are recorded on each judge `Score` under the
105
+ `judge_*` keys AgenticLens already reads, and totalled in `summary.judge`.
106
+ - A judge that fails after its retries yields an explicit **unavailable** score
107
+ that never passes; the default gate fails and names the cases.
108
+ - `samples=N` reports per-case agreement; `GateConfig(min_judge_agreement=...)`
109
+ catches a flaky judge.
110
+ - Live targets retry with backoff and report `timed_out` separately from failed
111
+ (`CaseEvaluation.target_status`, `summary.live`, JUnit `<error>`).
112
+ - `aevaluate_suite` / `arun_live_suite` await `async def` evaluators and targets.
113
+
114
+ See the [judge reliability guide](docs/judge-reliability.md) and the offline
115
+ [judge](examples/judge_reliability.py) and [async target](examples/async_live_target.py)
116
+ examples. Report schema 1.0 gains optional fields only.
117
+
118
+ ## Declarative CLI (0.8.0)
119
+
120
+ Run saved suites and samples, then apply an explicit release gate:
121
+
122
+ ```bash
123
+ agentic-evals evaluate examples/ci/suite.yaml examples/ci/samples.jsonl --out report.json
124
+ agentic-evals gate report.json --config examples/ci/gate.yaml
125
+ agentic-evals pack list
126
+ agentic-evals live examples/ci/suite.yaml --target my_agent:run --max-workers 4 --out live.json
127
+ ```
128
+
129
+ `evaluate` and `live` exit 0 when a report is produced, including failing cases.
130
+ `gate` exits 0 for pass, 1 for a failed gate, and 2 for input or usage errors.
131
+ Use `--format json|junit|markdown` for native evidence, JUnit or job summaries.
132
+ See the [CLI guide](docs/cli.md), [CI workflow](examples/ci/evaluate.yml), and
133
+ [voice-agent CLI example](voice-agent-evals/README.md#declarative-cli-080).
134
+ The native report and gate contracts remain compatible with 0.7.2 consumers.
135
+
86
136
  ## Why a separate package
87
137
 
88
138
  AgenticLens's evaluation module was never actually AgenticLens-specific —
@@ -367,6 +417,20 @@ A `None` version means the evaluator did not declare one. Version a judge
367
417
  whenever its prompt or model changes, so two reports can be compared only
368
418
  when they were graded by the same judge.
369
419
 
420
+ ### Judge cost, retries and agreement
421
+
422
+ `LLMJudgeEvaluator` and `LLMRubricEvaluator` take `retry=RetryPolicy(...)` and
423
+ `samples=N`. Every judge score records `judge_status` (`ok` or
424
+ `unavailable`), `judge_attempts` and `judge_latency_ms`, plus any usage the
425
+ judge reported (`judge_cost_usd`, `judge_cost_source`, tokens, model,
426
+ provider). A rubric `complete_fn` reports usage by returning a
427
+ `JudgeResponse`. A judge that raises, or a rubric completion that cannot be parsed,
428
+ no longer aborts the suite. After the last attempt it becomes an unavailable score:
429
+ value 0 and never passing, with the error recorded.
430
+ `summary.judge` totals judge cost apart from the agent's cost, and lists
431
+ unavailable cases and per-case agreement. See the
432
+ [judge reliability guide](docs/judge-reliability.md).
433
+
370
434
  ## Release gates
371
435
 
372
436
  ```python
@@ -394,6 +458,18 @@ GateConfig(
394
458
  A tag or metric named in the config but missing from the report fails the
395
459
  gate, so a renamed tag cannot quietly switch a check off.
396
460
 
461
+ Judge and live-target checks (0.9.0):
462
+
463
+ ```python
464
+ GateConfig(
465
+ max_judge_unavailable_scores=0, # the default: an unavailable judge fails the gate
466
+ min_judge_agreement=0.8, # every case judged with samples=N
467
+ max_judge_cost_usd=2.50, # fails when any judge score has no cost
468
+ max_timed_out_cases=0, # live runs: timeouts, apart from wrong answers
469
+ max_target_error_cases=0,
470
+ )
471
+ ```
472
+
397
473
  Never fabricates a value it can't back up: `total_cost_usd` on a summary or
398
474
  gate decision stays `None` unless every case in scope has a known cost —
399
475
  an incomplete cost picture is reported as unavailable, not `$0.00`.
@@ -511,6 +587,15 @@ targets that are safe to call concurrently. Python and HTTP targets go
511
587
  through the same scoring path; the test suite checks that both produce the
512
588
  same report for the same outputs.
513
589
 
590
+ `retry=RetryPolicy(max_attempts=3, backoff_seconds=1)` retries a call that
591
+ raises or times out. A case whose calls all fail is reported on the case
592
+ (`target_status` `timed_out` or `error`, `target_attempts`, `target_error`)
593
+ instead of aborting the run, and `summary.live` counts each outcome. HTTP
594
+ targets always apply `timeout_seconds`; Python targets apply it when it is set
595
+ explicitly, cancelling `async def` targets. `arun_live_suite(...,
596
+ max_concurrency=N)` awaits async targets and evaluators; `aevaluate_suite`
597
+ does the same for saved samples.
598
+
514
599
  Live targets are intentionally powerful developer-facing integrations —
515
600
  Python targets execute local code and HTTP targets can reach arbitrary
516
601
  URLs. Only point them at trusted suite files and trusted target
@@ -649,3 +734,123 @@ Contract `1.x` changes are additive only; readers must tolerate unknown fields.
649
734
  See [contract policy and sibling compatibility](docs/report-contract.md) and
650
735
  [validated JSON examples](examples/report-contract/). The quickstart `EvalResult`
651
736
  is a separate format. Native schema validation does not establish AIOS conformance.
737
+
738
+ ## Native consumer validation
739
+
740
+ MCP 0.3.8 adds `evals.validate_artifact` and validates Lens report inputs and
741
+ produced SLO gates. Control Tower 0.6.1 validates complete reports and supplied
742
+ gates before evidence projection. Both bundle exact Evals 0.7.2 schemas for
743
+ offline, sibling-independent validation. See [the consumer matrix](docs/report-contract.md).
744
+
745
+ ## AgenticLens 0.8.0 suite management
746
+
747
+ AgenticLens 0.8.0 delivers its v0.6 milestone: `agenticlens.suites` and
748
+ `agenticlens suite` build regression suites from CSV/JSON imports, production
749
+ traces and user feedback. It masks PII by default before storage (`--no-mask`
750
+ opts out), detects duplicates and
751
+ conflicting labels, adds synthetic variations and stable splits, records
752
+ reviews, and approves suites into immutable, hash-verified versions that CI
753
+ verifies, exports and evaluates. Suites stay plain agentic-evals `TestSuite`
754
+ files.
755
+
756
+ Evals runs these suites unchanged with `evaluate_suite` and `run_live_suite`.
757
+ Lens keeps management state (lineage, review status, domain, split) in
758
+ `TestCase.metadata` and `tags`, and an exported approved version adds
759
+ `metadata.agenticlens_suite_version` (name, version, SHA-256, approver).
760
+ Suite approval is not a release gate: scoring and gate computation stay in
761
+ Evals. No Evals change or release is needed.
762
+
763
+ See the [suite management guide](https://github.com/DeepAgentLabs/agenticlens/blob/v0.8.0/docs/suite-management.md).
764
+
765
+ ## AgenticLens 0.9.0 ModelFit
766
+
767
+ AgenticLens 0.9.0 delivers its v0.7 milestone: `agenticlens.modelfit` and
768
+ `agenticlens modelfit` recommend a model per task category from recorded
769
+ experiment reports, under hard quality, cost, latency, safety, provider and
770
+ deployment constraints that fail closed. Each recommendation carries a
771
+ confidence level with reasons, an explanation, alternatives, a fallback
772
+ (on a different provider when one is feasible), savings against a reference variant tied to the
773
+ experiment evidence, and quality risks, plus a task-level model matrix.
774
+
775
+ Evals is unchanged. ModelFit reads the native evaluation reports that Lens
776
+ experiments embed per trial (`cases[].passed`, `latency_ms`, `cost_usd`,
777
+ `tags`, `metadata`) and computes no release gate; scoring and gate
778
+ computation stay in Evals. Lens keeps its `agentic-evals>=0.7.1,<1`
779
+ dependency. No Evals change or release is needed.
780
+
781
+ See the [ModelFit guide](https://github.com/DeepAgentLabs/agenticlens/blob/v0.9.0/docs/modelfit.md).
782
+
783
+ ## Chaos 0.7.0 interoperability
784
+
785
+ Chaos 0.7.0 adds sequential fault cascades, deterministic monotonic breaking-point
786
+ search, synthetic response poisoning (including prompt/tool-response instruction
787
+ and policy-bypass payloads), and explicit-price retry cost estimates. Native
788
+ child event IDs and fault types remain intact; cascade IDs/stages and poisoning
789
+ strategy/payload/seed are additive event detail. Existing report readers and
790
+ AIOS exports remain compatible.
791
+
792
+ See the [v0.7 guide and offline synthetic example](https://github.com/DeepAgentLabs/agentic-chaos/blob/v0.7.0/docs/v0.7.md).
793
+ The example includes observed synthetic dispatches and a bypassed negative
794
+ control. Injection or recovery alone does not prove containment. No new shared
795
+ schema, automatic adapter, hosted execution capability or containment guarantee
796
+ is introduced.
797
+
798
+ Scores and release gates remain owned by Evals; Chaos contracts and threshold trials are per-campaign evidence. Convert observations to native EvaluationSamples explicitly. No runtime dependency or patch release is needed.
799
+
800
+ ## Control Tower 0.7.0 fleet operations
801
+
802
+ Tower 0.7.0 adds frozen environment/capability targeting, bulk desired settings,
803
+ separate approval, durable per-target results and bounded stop/MCP-revocation
804
+ acknowledgement through Python, HTTP and CLI. No fleet atomicity or automatic
805
+ rollback/resume is claimed. Timed-out/requested commands need host reconciliation.
806
+
807
+ Evals retains scoring and native report/gate decisions. Bulk execution does not recompute gates or equate passing evaluation with deployment health or containment.
808
+
809
+ Existing optional dependency ranges remain sufficient. No sibling runtime or
810
+ native evidence schema changes are required; this documentation alignment needs
811
+ no patch release. See the [operator guide](https://github.com/DeepAgentLabs/agenticops-control-tower/blob/v0.7.0/docs/fleet-operations.md).
812
+
813
+ See the [local Tower compatibility guide](docs/tower-0.7-fleet-operations.md) for the operator workflow and package boundaries.
814
+
815
+ ## AgenticLens 0.10.0 routing compatibility
816
+
817
+ See the [local integration boundary guide](docs/lens-0.10-routing.md).
818
+
819
+ AgenticLens 0.10.0 implements roadmap v0.8: evidence-based runtime model
820
+ selection, fail-closed routing constraints, bounded provider failover,
821
+ confidence/escalation handoffs, shadow/canary modes and a local routing journal.
822
+ The default is advisory; execution uses explicit caller-owned callbacks.
823
+ MCP 0.3.12 exposes read-only `lens.routing_decide` and pins hosted Lens 0.10.0.
824
+ See the [routing guide](https://github.com/DeepAgentLabs/agenticlens/blob/v0.10.0/docs/routing.md)
825
+ and [offline example](https://github.com/DeepAgentLabs/agenticlens/blob/v0.10.0/examples/routing_demo.py).
826
+
827
+ Lens retains `agentic-evals>=0.7.1,<1`, which covers current Evals 0.8.0.
828
+ Standalone scoring and release-gate computation remain Evals-owned. Routing
829
+ outcomes do not redefine native evaluation reports or gate decisions. No dependency
830
+ change or Evals patch release is required.
831
+
832
+ ## Chaos 0.8.0 interoperability
833
+
834
+ Chaos 0.8.0 adds:
835
+
836
+ - multi-stage chaos workflows: steps, steady-state health checks, parallel
837
+ branches, approval gates that can be resumed, and reusable templates
838
+ - declarative YAML/JSON/TOML experiments, with validation that never executes
839
+ targets and cron schedule previews
840
+ - `Scope` explosion-radius control (steps, providers and exclusions) on every
841
+ built-in fault and `cascade()`; custom faults get it by calling
842
+ `BaseFault.__init__`, setting `scope` or overriding `in_scope()`
843
+ - a topology fuzzer that weights fault placement by fan-out
844
+
845
+ Native event IDs and fault types are unchanged. Workflow provenance
846
+ (`detail.workflow`) and an optional `ChaosReport.workflow` summary are
847
+ additive. A workflow's steps merge into one native report for contracts and
848
+ AIOS export.
849
+
850
+ See the [v0.8 guide](https://github.com/DeepAgentLabs/agentic-chaos/blob/v0.8.0/docs/v0.8.md)
851
+ and the [offline examples](https://github.com/DeepAgentLabs/agentic-chaos/tree/v0.8.0/examples/workflows).
852
+ Workflow statuses, health checks and fuzz outcomes are resilience evidence.
853
+ They are not scores, release gates or proof of containment. No new shared
854
+ schema, automatic adapter or hosted execution capability is introduced.
855
+
856
+ Workflow statuses and per-step contracts are per-run resilience checks, not Evals scores or release gates. Convert observed outputs to native EvaluationSamples explicitly. No adapter, runtime dependency or Evals patch release is needed.
@@ -23,6 +23,55 @@ quickstart API, and the `agentic-evals` CLI) is real, tested code — see the
23
23
  still move between minor versions ahead of a 1.0; pin a version in
24
24
  production and read the changelog before upgrading.
25
25
 
26
+ ## Judge reliability and live resilience (0.9.0)
27
+
28
+ LLM-judged and live evaluations now say what they cost and whether they
29
+ actually happened:
30
+
31
+ ```python
32
+ from agentic_evals import FACTUALITY, JudgeResponse, LLMRubricEvaluator, RetryPolicy
33
+
34
+ judge = LLMRubricEvaluator(
35
+ "factuality",
36
+ FACTUALITY,
37
+ complete_fn, # may return JudgeResponse(text=..., cost_usd=...)
38
+ retry=RetryPolicy(max_attempts=3, backoff_seconds=0.5),
39
+ samples=3, # judge three times, report agreement
40
+ )
41
+ ```
42
+
43
+ - Judge cost, tokens and latency are recorded on each judge `Score` under the
44
+ `judge_*` keys AgenticLens already reads, and totalled in `summary.judge`.
45
+ - A judge that fails after its retries yields an explicit **unavailable** score
46
+ that never passes; the default gate fails and names the cases.
47
+ - `samples=N` reports per-case agreement; `GateConfig(min_judge_agreement=...)`
48
+ catches a flaky judge.
49
+ - Live targets retry with backoff and report `timed_out` separately from failed
50
+ (`CaseEvaluation.target_status`, `summary.live`, JUnit `<error>`).
51
+ - `aevaluate_suite` / `arun_live_suite` await `async def` evaluators and targets.
52
+
53
+ See the [judge reliability guide](docs/judge-reliability.md) and the offline
54
+ [judge](examples/judge_reliability.py) and [async target](examples/async_live_target.py)
55
+ examples. Report schema 1.0 gains optional fields only.
56
+
57
+ ## Declarative CLI (0.8.0)
58
+
59
+ Run saved suites and samples, then apply an explicit release gate:
60
+
61
+ ```bash
62
+ agentic-evals evaluate examples/ci/suite.yaml examples/ci/samples.jsonl --out report.json
63
+ agentic-evals gate report.json --config examples/ci/gate.yaml
64
+ agentic-evals pack list
65
+ agentic-evals live examples/ci/suite.yaml --target my_agent:run --max-workers 4 --out live.json
66
+ ```
67
+
68
+ `evaluate` and `live` exit 0 when a report is produced, including failing cases.
69
+ `gate` exits 0 for pass, 1 for a failed gate, and 2 for input or usage errors.
70
+ Use `--format json|junit|markdown` for native evidence, JUnit or job summaries.
71
+ See the [CLI guide](docs/cli.md), [CI workflow](examples/ci/evaluate.yml), and
72
+ [voice-agent CLI example](voice-agent-evals/README.md#declarative-cli-080).
73
+ The native report and gate contracts remain compatible with 0.7.2 consumers.
74
+
26
75
  ## Why a separate package
27
76
 
28
77
  AgenticLens's evaluation module was never actually AgenticLens-specific —
@@ -307,6 +356,20 @@ A `None` version means the evaluator did not declare one. Version a judge
307
356
  whenever its prompt or model changes, so two reports can be compared only
308
357
  when they were graded by the same judge.
309
358
 
359
+ ### Judge cost, retries and agreement
360
+
361
+ `LLMJudgeEvaluator` and `LLMRubricEvaluator` take `retry=RetryPolicy(...)` and
362
+ `samples=N`. Every judge score records `judge_status` (`ok` or
363
+ `unavailable`), `judge_attempts` and `judge_latency_ms`, plus any usage the
364
+ judge reported (`judge_cost_usd`, `judge_cost_source`, tokens, model,
365
+ provider). A rubric `complete_fn` reports usage by returning a
366
+ `JudgeResponse`. A judge that raises, or a rubric completion that cannot be parsed,
367
+ no longer aborts the suite. After the last attempt it becomes an unavailable score:
368
+ value 0 and never passing, with the error recorded.
369
+ `summary.judge` totals judge cost apart from the agent's cost, and lists
370
+ unavailable cases and per-case agreement. See the
371
+ [judge reliability guide](docs/judge-reliability.md).
372
+
310
373
  ## Release gates
311
374
 
312
375
  ```python
@@ -334,6 +397,18 @@ GateConfig(
334
397
  A tag or metric named in the config but missing from the report fails the
335
398
  gate, so a renamed tag cannot quietly switch a check off.
336
399
 
400
+ Judge and live-target checks (0.9.0):
401
+
402
+ ```python
403
+ GateConfig(
404
+ max_judge_unavailable_scores=0, # the default: an unavailable judge fails the gate
405
+ min_judge_agreement=0.8, # every case judged with samples=N
406
+ max_judge_cost_usd=2.50, # fails when any judge score has no cost
407
+ max_timed_out_cases=0, # live runs: timeouts, apart from wrong answers
408
+ max_target_error_cases=0,
409
+ )
410
+ ```
411
+
337
412
  Never fabricates a value it can't back up: `total_cost_usd` on a summary or
338
413
  gate decision stays `None` unless every case in scope has a known cost —
339
414
  an incomplete cost picture is reported as unavailable, not `$0.00`.
@@ -451,6 +526,15 @@ targets that are safe to call concurrently. Python and HTTP targets go
451
526
  through the same scoring path; the test suite checks that both produce the
452
527
  same report for the same outputs.
453
528
 
529
+ `retry=RetryPolicy(max_attempts=3, backoff_seconds=1)` retries a call that
530
+ raises or times out. A case whose calls all fail is reported on the case
531
+ (`target_status` `timed_out` or `error`, `target_attempts`, `target_error`)
532
+ instead of aborting the run, and `summary.live` counts each outcome. HTTP
533
+ targets always apply `timeout_seconds`; Python targets apply it when it is set
534
+ explicitly, cancelling `async def` targets. `arun_live_suite(...,
535
+ max_concurrency=N)` awaits async targets and evaluators; `aevaluate_suite`
536
+ does the same for saved samples.
537
+
454
538
  Live targets are intentionally powerful developer-facing integrations —
455
539
  Python targets execute local code and HTTP targets can reach arbitrary
456
540
  URLs. Only point them at trusted suite files and trusted target
@@ -589,3 +673,123 @@ Contract `1.x` changes are additive only; readers must tolerate unknown fields.
589
673
  See [contract policy and sibling compatibility](docs/report-contract.md) and
590
674
  [validated JSON examples](examples/report-contract/). The quickstart `EvalResult`
591
675
  is a separate format. Native schema validation does not establish AIOS conformance.
676
+
677
+ ## Native consumer validation
678
+
679
+ MCP 0.3.8 adds `evals.validate_artifact` and validates Lens report inputs and
680
+ produced SLO gates. Control Tower 0.6.1 validates complete reports and supplied
681
+ gates before evidence projection. Both bundle exact Evals 0.7.2 schemas for
682
+ offline, sibling-independent validation. See [the consumer matrix](docs/report-contract.md).
683
+
684
+ ## AgenticLens 0.8.0 suite management
685
+
686
+ AgenticLens 0.8.0 delivers its v0.6 milestone: `agenticlens.suites` and
687
+ `agenticlens suite` build regression suites from CSV/JSON imports, production
688
+ traces and user feedback. It masks PII by default before storage (`--no-mask`
689
+ opts out), detects duplicates and
690
+ conflicting labels, adds synthetic variations and stable splits, records
691
+ reviews, and approves suites into immutable, hash-verified versions that CI
692
+ verifies, exports and evaluates. Suites stay plain agentic-evals `TestSuite`
693
+ files.
694
+
695
+ Evals runs these suites unchanged with `evaluate_suite` and `run_live_suite`.
696
+ Lens keeps management state (lineage, review status, domain, split) in
697
+ `TestCase.metadata` and `tags`, and an exported approved version adds
698
+ `metadata.agenticlens_suite_version` (name, version, SHA-256, approver).
699
+ Suite approval is not a release gate: scoring and gate computation stay in
700
+ Evals. No Evals change or release is needed.
701
+
702
+ See the [suite management guide](https://github.com/DeepAgentLabs/agenticlens/blob/v0.8.0/docs/suite-management.md).
703
+
704
+ ## AgenticLens 0.9.0 ModelFit
705
+
706
+ AgenticLens 0.9.0 delivers its v0.7 milestone: `agenticlens.modelfit` and
707
+ `agenticlens modelfit` recommend a model per task category from recorded
708
+ experiment reports, under hard quality, cost, latency, safety, provider and
709
+ deployment constraints that fail closed. Each recommendation carries a
710
+ confidence level with reasons, an explanation, alternatives, a fallback
711
+ (on a different provider when one is feasible), savings against a reference variant tied to the
712
+ experiment evidence, and quality risks, plus a task-level model matrix.
713
+
714
+ Evals is unchanged. ModelFit reads the native evaluation reports that Lens
715
+ experiments embed per trial (`cases[].passed`, `latency_ms`, `cost_usd`,
716
+ `tags`, `metadata`) and computes no release gate; scoring and gate
717
+ computation stay in Evals. Lens keeps its `agentic-evals>=0.7.1,<1`
718
+ dependency. No Evals change or release is needed.
719
+
720
+ See the [ModelFit guide](https://github.com/DeepAgentLabs/agenticlens/blob/v0.9.0/docs/modelfit.md).
721
+
722
+ ## Chaos 0.7.0 interoperability
723
+
724
+ Chaos 0.7.0 adds sequential fault cascades, deterministic monotonic breaking-point
725
+ search, synthetic response poisoning (including prompt/tool-response instruction
726
+ and policy-bypass payloads), and explicit-price retry cost estimates. Native
727
+ child event IDs and fault types remain intact; cascade IDs/stages and poisoning
728
+ strategy/payload/seed are additive event detail. Existing report readers and
729
+ AIOS exports remain compatible.
730
+
731
+ See the [v0.7 guide and offline synthetic example](https://github.com/DeepAgentLabs/agentic-chaos/blob/v0.7.0/docs/v0.7.md).
732
+ The example includes observed synthetic dispatches and a bypassed negative
733
+ control. Injection or recovery alone does not prove containment. No new shared
734
+ schema, automatic adapter, hosted execution capability or containment guarantee
735
+ is introduced.
736
+
737
+ Scores and release gates remain owned by Evals; Chaos contracts and threshold trials are per-campaign evidence. Convert observations to native EvaluationSamples explicitly. No runtime dependency or patch release is needed.
738
+
739
+ ## Control Tower 0.7.0 fleet operations
740
+
741
+ Tower 0.7.0 adds frozen environment/capability targeting, bulk desired settings,
742
+ separate approval, durable per-target results and bounded stop/MCP-revocation
743
+ acknowledgement through Python, HTTP and CLI. No fleet atomicity or automatic
744
+ rollback/resume is claimed. Timed-out/requested commands need host reconciliation.
745
+
746
+ Evals retains scoring and native report/gate decisions. Bulk execution does not recompute gates or equate passing evaluation with deployment health or containment.
747
+
748
+ Existing optional dependency ranges remain sufficient. No sibling runtime or
749
+ native evidence schema changes are required; this documentation alignment needs
750
+ no patch release. See the [operator guide](https://github.com/DeepAgentLabs/agenticops-control-tower/blob/v0.7.0/docs/fleet-operations.md).
751
+
752
+ See the [local Tower compatibility guide](docs/tower-0.7-fleet-operations.md) for the operator workflow and package boundaries.
753
+
754
+ ## AgenticLens 0.10.0 routing compatibility
755
+
756
+ See the [local integration boundary guide](docs/lens-0.10-routing.md).
757
+
758
+ AgenticLens 0.10.0 implements roadmap v0.8: evidence-based runtime model
759
+ selection, fail-closed routing constraints, bounded provider failover,
760
+ confidence/escalation handoffs, shadow/canary modes and a local routing journal.
761
+ The default is advisory; execution uses explicit caller-owned callbacks.
762
+ MCP 0.3.12 exposes read-only `lens.routing_decide` and pins hosted Lens 0.10.0.
763
+ See the [routing guide](https://github.com/DeepAgentLabs/agenticlens/blob/v0.10.0/docs/routing.md)
764
+ and [offline example](https://github.com/DeepAgentLabs/agenticlens/blob/v0.10.0/examples/routing_demo.py).
765
+
766
+ Lens retains `agentic-evals>=0.7.1,<1`, which covers current Evals 0.8.0.
767
+ Standalone scoring and release-gate computation remain Evals-owned. Routing
768
+ outcomes do not redefine native evaluation reports or gate decisions. No dependency
769
+ change or Evals patch release is required.
770
+
771
+ ## Chaos 0.8.0 interoperability
772
+
773
+ Chaos 0.8.0 adds:
774
+
775
+ - multi-stage chaos workflows: steps, steady-state health checks, parallel
776
+ branches, approval gates that can be resumed, and reusable templates
777
+ - declarative YAML/JSON/TOML experiments, with validation that never executes
778
+ targets and cron schedule previews
779
+ - `Scope` explosion-radius control (steps, providers and exclusions) on every
780
+ built-in fault and `cascade()`; custom faults get it by calling
781
+ `BaseFault.__init__`, setting `scope` or overriding `in_scope()`
782
+ - a topology fuzzer that weights fault placement by fan-out
783
+
784
+ Native event IDs and fault types are unchanged. Workflow provenance
785
+ (`detail.workflow`) and an optional `ChaosReport.workflow` summary are
786
+ additive. A workflow's steps merge into one native report for contracts and
787
+ AIOS export.
788
+
789
+ See the [v0.8 guide](https://github.com/DeepAgentLabs/agentic-chaos/blob/v0.8.0/docs/v0.8.md)
790
+ and the [offline examples](https://github.com/DeepAgentLabs/agentic-chaos/tree/v0.8.0/examples/workflows).
791
+ Workflow statuses, health checks and fuzz outcomes are resilience evidence.
792
+ They are not scores, release gates or proof of containment. No new shared
793
+ schema, automatic adapter or hosted execution capability is introduced.
794
+
795
+ Workflow statuses and per-step contracts are per-run resilience checks, not Evals scores or release gates. Convert observed outputs to native EvaluationSamples explicitly. No adapter, runtime dependency or Evals patch release is needed.