agentic-evals 0.7.2__tar.gz → 0.9.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/.gitignore +1 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/CHANGELOG.md +87 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/PKG-INFO +206 -1
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/README.md +204 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/ROADMAP.md +155 -21
- agentic_evals-0.9.0/docs/chaos-0.7.md +18 -0
- agentic_evals-0.9.0/docs/chaos-0.8.md +27 -0
- agentic_evals-0.9.0/docs/cli.md +121 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/docs/docs/index.html +24 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/docs/index.html +3 -3
- agentic_evals-0.9.0/docs/judge-reliability.md +225 -0
- agentic_evals-0.9.0/docs/lens-0.10-routing.md +16 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/docs/report-contract.md +36 -3
- agentic_evals-0.9.0/docs/tower-0.7-fleet-operations.md +26 -0
- agentic_evals-0.9.0/examples/async_live_target.py +68 -0
- agentic_evals-0.9.0/examples/ci/evaluate.yml +30 -0
- agentic_evals-0.9.0/examples/ci/failing-samples.jsonl +1 -0
- agentic_evals-0.9.0/examples/ci/gate.yaml +2 -0
- agentic_evals-0.9.0/examples/ci/samples.jsonl +1 -0
- agentic_evals-0.9.0/examples/ci/suite.yaml +8 -0
- agentic_evals-0.9.0/examples/judge_reliability.py +116 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/examples/report-contract/evaluation-report.json +11 -3
- agentic_evals-0.9.0/examples/report-contract/generate.py +130 -0
- agentic_evals-0.9.0/examples/report-contract/judged-live-gate-decision.json +16 -0
- agentic_evals-0.9.0/examples/report-contract/judged-live-report.json +219 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/pyproject.toml +2 -1
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/scripts/check_installed_contract.py +3 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/scripts/generate_schemas.py +1 -1
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/__init__.py +15 -0
- agentic_evals-0.9.0/src/agentic_evals/_async.py +51 -0
- agentic_evals-0.9.0/src/agentic_evals/_version.py +1 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/cli.py +7 -2
- agentic_evals-0.9.0/src/agentic_evals/cli_commands.py +168 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/evaluators.py +71 -8
- agentic_evals-0.9.0/src/agentic_evals/formatting.py +153 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/gate.py +86 -0
- agentic_evals-0.9.0/src/agentic_evals/judging.py +333 -0
- agentic_evals-0.9.0/src/agentic_evals/live.py +261 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/models.py +68 -0
- agentic_evals-0.9.0/src/agentic_evals/reliability.py +133 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/runner.py +367 -138
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/schemas/evaluation-report-1.0.schema.json +236 -2
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/schemas/gate-decision-1.0.schema.json +1 -1
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/scorers/rubric.py +66 -13
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/debug-a-flaky-llm-judge/SKILL.md +15 -7
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/define-a-release-gate/SKILL.md +13 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/run-a-live-suite/SKILL.md +21 -8
- agentic_evals-0.9.0/tests/test_async_evaluation.py +231 -0
- agentic_evals-0.9.0/tests/test_cli_declarative.py +212 -0
- agentic_evals-0.9.0/tests/test_cli_examples.py +73 -0
- agentic_evals-0.9.0/tests/test_judge_reliability.py +402 -0
- agentic_evals-0.9.0/tests/test_live_resilience.py +356 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_live_targets.py +42 -3
- agentic_evals-0.9.0/tests/test_reliability.py +99 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_report_contract.py +52 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_scorers_rubric.py +7 -3
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/uv.lock +12 -1
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/README.md +17 -0
- agentic_evals-0.9.0/voice-agent-evals/export_cli.py +27 -0
- agentic_evals-0.7.2/examples/report-contract/generate.py +0 -51
- agentic_evals-0.7.2/src/agentic_evals/_version.py +0 -1
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/.github/ISSUE_TEMPLATE/bug_report.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/.github/ISSUE_TEMPLATE/config.yml +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/.github/ISSUE_TEMPLATE/feature_request.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/.github/PULL_REQUEST_TEMPLATE.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/.github/workflows/ci.yml +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/.github/workflows/pragent.yml +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/.github/workflows/release-pypi.yml +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/.github/workflows/release-testpypi.yml +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/CONTRIBUTING.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/LICENSE +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/SECURITY.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/docs/control-tower-0.6.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/docs/sidecar-v0.8.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/examples/capitals_eval.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/examples/eval_pack.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/examples/llm_rubric_judge.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/examples/near_miss_similarity.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/examples/quickstart.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/examples/report-contract/gate-decision.json +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/packs/__init__.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/packs/builtin/factual-qa.yaml +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/packs/builtin/json-output-contract.yaml +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/packs/builtin/tool-use-correctness.yaml +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/packs/library.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/packs/loader.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/py.typed +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/report_contract.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/scorers/__init__.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/scorers/registry.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/scorers/text.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/scorers/trajectory.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/simple.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/analyze-an-eval-experiment/SKILL.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/build-an-eval-dataset/SKILL.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/choose-a-rubric-template/SKILL.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/define-an-eval-objective/SKILL.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/design-an-eval-experiment/SKILL.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/discover-failure-modes/SKILL.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/elicit-eval-criteria/SKILL.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/instrument-a-trace/SKILL.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/monitor-evals-in-production/SKILL.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/red-team-an-agent-suite/SKILL.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/report-eval-results/SKILL.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/size-a-test-suite/SKILL.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/validate-a-scorer/SKILL.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/src/agentic_evals/skills/write-a-scorer/SKILL.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/fixtures/v0.7.1/README.md +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/fixtures/v0.7.1/evaluation-report.json +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/fixtures/v0.7.1/evaluation-report.schema.json +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/fixtures/v0.7.1/gate-decision.json +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/fixtures/v0.7.1/gate-decision.schema.json +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/live_eval_target.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_cli.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_evaluators.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_gate.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_models.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_packs.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_provenance.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_report_breakdowns.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_runner.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_scorers_registry.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_scorers_text.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_scorers_trajectory.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_simple.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_skills.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/tests/test_tool_expectations.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/.gitignore +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/config.js +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/dataset.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/evaluators.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/index.html +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/run_evals.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/scenario_runner.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/scenarios.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/server.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/voice_agents.py +0 -0
- {agentic_evals-0.7.2 → agentic_evals-0.9.0}/voice-agent-evals/voice_test_suite.yaml +0 -0
|
@@ -4,6 +4,93 @@ All notable changes to this project will be documented here.
|
|
|
4
4
|
|
|
5
5
|
This project follows [Semantic Versioning](https://semver.org/).
|
|
6
6
|
|
|
7
|
+
## 0.9.0 - 2026-10-19
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
|
|
11
|
+
- Judge cost and latency on every judge score, under the `judge_*` metadata
|
|
12
|
+
keys AgenticLens already reads (`judge_cost_usd`, `judge_cost_source`,
|
|
13
|
+
`judge_input_tokens`, `judge_output_tokens`, `judge_latency_ms`,
|
|
14
|
+
`judge_model`, `judge_provider`), plus `judge_status` and `judge_attempts`.
|
|
15
|
+
A rubric `complete_fn` reports usage by returning a `JudgeResponse`.
|
|
16
|
+
- `EvaluationSummary.judge` (`JudgeSummary`): judge scores, unavailable scores
|
|
17
|
+
and the cases they belong to, retries, cost total (only when every judge score
|
|
18
|
+
is priced), cost sources, tokens, latency and sampling agreement. Judge spend
|
|
19
|
+
stays separate from the evaluated system's `total_cost_usd`.
|
|
20
|
+
- `RetryPolicy`: bounded attempts with capped exponential backoff, used by
|
|
21
|
+
judges (`retry=` on `LLMJudgeEvaluator`/`LLMRubricEvaluator`) and live
|
|
22
|
+
targets (`retry=` on `run_live_suite`/`arun_live_suite`).
|
|
23
|
+
- An explicit unavailable judge outcome: when every attempt fails, the score
|
|
24
|
+
has value 0, never passes at any threshold, and records `judge_error`.
|
|
25
|
+
- Repeated judging with `samples=N`: mean value, most common verdict (ties go
|
|
26
|
+
to the lower verdict), `judge_verdicts` and `judge_agreement` per score, and
|
|
27
|
+
`min_agreement`/`case_agreement` in `summary.judge`.
|
|
28
|
+
- Live-target outcomes per case: `CaseEvaluation.target_status` (`ok`,
|
|
29
|
+
`timed_out`, `error`), `target_attempts` and `target_error`, a `target_call`
|
|
30
|
+
score, and `EvaluationSummary.live` (`LiveSummary`).
|
|
31
|
+
- Python target timeouts when `timeout_seconds` is set explicitly; `async def`
|
|
32
|
+
targets are cancelled, synchronous ones are abandoned in daemon threads.
|
|
33
|
+
- Async evaluation: `aevaluate_suite`, `arun_live_suite`, `aevaluate()` on
|
|
34
|
+
`CallableEvaluator` and its subclasses, the `AsyncEvaluator` protocol, and
|
|
35
|
+
`async def` judges, rubric `complete_fn`s and Python targets (also accepted
|
|
36
|
+
by the sync APIs outside a running event loop).
|
|
37
|
+
- Gate checks: `max_judge_unavailable_scores` (default 0), `max_judge_cost_usd`,
|
|
38
|
+
`min_judge_agreement`, `max_timed_out_cases` and `max_target_error_cases`.
|
|
39
|
+
Reasons name the affected cases.
|
|
40
|
+
- CLI `live --retries`, `--retry-backoff`, and `--timeout-seconds` for Python
|
|
41
|
+
targets. JUnit reports timed-out/errored live cases as `<error>`; Markdown
|
|
42
|
+
shows `TIMEOUT`/`ERROR` and judge/live summaries.
|
|
43
|
+
- `docs/judge-reliability.md`, offline `examples/judge_reliability.py` and
|
|
44
|
+
`examples/async_live_target.py`, a judged live contract example, and
|
|
45
|
+
updated skills for flaky judges, live suites and release gates.
|
|
46
|
+
|
|
47
|
+
### Changed
|
|
48
|
+
|
|
49
|
+
- A judge that raises, or a rubric completion with no parseable verdict, now
|
|
50
|
+
yields an unavailable score instead of raising out of `evaluate_suite`.
|
|
51
|
+
- A live-target call that raises or times out (including an unreachable HTTP
|
|
52
|
+
target) is reported on its case instead of aborting `run_live_suite`;
|
|
53
|
+
`agentic-evals live` writes the report and exits 0. A case is `timed_out` only
|
|
54
|
+
when every attempt timed out. An invalid target payload, including a non-JSON
|
|
55
|
+
HTTP body, still stops the run without retrying (exit 2).
|
|
56
|
+
- Usage reported by a rubric attempt with no parseable verdict is added to the
|
|
57
|
+
score's judge cost and tokens.
|
|
58
|
+
- The default gate fails when any judge score is unavailable, even for an
|
|
59
|
+
optional evaluator. Set `max_judge_unavailable_scores: null` to opt out.
|
|
60
|
+
- `LLMRubricEvaluator` now subclasses `LLMJudgeEvaluator`.
|
|
61
|
+
|
|
62
|
+
### Compatibility
|
|
63
|
+
|
|
64
|
+
- Report schema 1.0 gains optional fields only (`summary.judge`, `summary.live`,
|
|
65
|
+
`cases[].target_*`); packaged schema IDs now point at `v0.9.0`. Reports saved
|
|
66
|
+
by earlier versions load unchanged, and 0.9.0 reports validate against the
|
|
67
|
+
frozen 0.7.1 schemas, so readers bundling 0.7.2 schemas keep working.
|
|
68
|
+
- Lens's `agentic-evals>=0.7.1,<1` range already covers 0.9.0 and its tests pass
|
|
69
|
+
unchanged. MCP and Control Tower sync their bundled schemas and test pins to
|
|
70
|
+
0.9.0 in patch releases. AIOS evidence export remains v0.10.
|
|
71
|
+
|
|
72
|
+
## 0.8.0 - 2026-10-17
|
|
73
|
+
|
|
74
|
+
### Added
|
|
75
|
+
|
|
76
|
+
- Declarative `evaluate`, Python/HTTP `live`, and `gate` commands with explicit
|
|
77
|
+
exit codes, custom registry imports, concurrency and JSONL samples.
|
|
78
|
+
- Built-in `pack list` and `pack show`, including plain suite export without
|
|
79
|
+
requiring judge credentials for inspection.
|
|
80
|
+
- Native JSON, JUnit XML and escaped Markdown output for reports and gates.
|
|
81
|
+
- Passing/failing CI fixtures, a GitHub Actions workflow, CLI documentation and
|
|
82
|
+
voice-agent dataset export with the same custom evaluators as the Python path.
|
|
83
|
+
- CLI parity, process exit-code, malformed-input and presentation tests.
|
|
84
|
+
|
|
85
|
+
### Compatibility
|
|
86
|
+
|
|
87
|
+
- Existing `run` behavior and Python scoring/gating APIs are preserved. Native
|
|
88
|
+
report schema 1.0 and gate wire shapes are unchanged from 0.7.2.
|
|
89
|
+
- `evaluate`/`live` return 0 when evidence is produced; use `gate` to fail CI
|
|
90
|
+
according to policy (0 pass, 1 failed gate, 2 input/usage error).
|
|
91
|
+
- Sibling runtime dependency floors are unchanged; Lens, MCP and Tower consume
|
|
92
|
+
native JSON. AIOS evidence export remains future work.
|
|
93
|
+
|
|
7
94
|
## 0.7.2 - 2026-10-17
|
|
8
95
|
|
|
9
96
|
### Added
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: agentic-evals
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.9.0
|
|
4
4
|
Summary: A standalone, framework-agnostic evaluation and scoring engine for LLM/agent outputs — extracted from AgenticLens's evaluation module.
|
|
5
5
|
Project-URL: Homepage, https://github.com/DeepAgentLabs/agentic-evals
|
|
6
6
|
Project-URL: Repository, https://github.com/DeepAgentLabs/agentic-evals
|
|
@@ -46,6 +46,7 @@ Requires-Dist: pyyaml<7,>=6.0
|
|
|
46
46
|
Requires-Dist: referencing>=0.35
|
|
47
47
|
Provides-Extra: dev
|
|
48
48
|
Requires-Dist: build>=1.2; extra == 'dev'
|
|
49
|
+
Requires-Dist: junitparser>=3; extra == 'dev'
|
|
49
50
|
Requires-Dist: mypy>=1.10; extra == 'dev'
|
|
50
51
|
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
|
|
51
52
|
Requires-Dist: pytest>=8.0; extra == 'dev'
|
|
@@ -83,6 +84,55 @@ quickstart API, and the `agentic-evals` CLI) is real, tested code — see the
|
|
|
83
84
|
still move between minor versions ahead of a 1.0; pin a version in
|
|
84
85
|
production and read the changelog before upgrading.
|
|
85
86
|
|
|
87
|
+
## Judge reliability and live resilience (0.9.0)
|
|
88
|
+
|
|
89
|
+
LLM-judged and live evaluations now say what they cost and whether they
|
|
90
|
+
actually happened:
|
|
91
|
+
|
|
92
|
+
```python
|
|
93
|
+
from agentic_evals import FACTUALITY, JudgeResponse, LLMRubricEvaluator, RetryPolicy
|
|
94
|
+
|
|
95
|
+
judge = LLMRubricEvaluator(
|
|
96
|
+
"factuality",
|
|
97
|
+
FACTUALITY,
|
|
98
|
+
complete_fn, # may return JudgeResponse(text=..., cost_usd=...)
|
|
99
|
+
retry=RetryPolicy(max_attempts=3, backoff_seconds=0.5),
|
|
100
|
+
samples=3, # judge three times, report agreement
|
|
101
|
+
)
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
- Judge cost, tokens and latency are recorded on each judge `Score` under the
|
|
105
|
+
`judge_*` keys AgenticLens already reads, and totalled in `summary.judge`.
|
|
106
|
+
- A judge that fails after its retries yields an explicit **unavailable** score
|
|
107
|
+
that never passes; the default gate fails and names the cases.
|
|
108
|
+
- `samples=N` reports per-case agreement; `GateConfig(min_judge_agreement=...)`
|
|
109
|
+
catches a flaky judge.
|
|
110
|
+
- Live targets retry with backoff and report `timed_out` separately from failed
|
|
111
|
+
(`CaseEvaluation.target_status`, `summary.live`, JUnit `<error>`).
|
|
112
|
+
- `aevaluate_suite` / `arun_live_suite` await `async def` evaluators and targets.
|
|
113
|
+
|
|
114
|
+
See the [judge reliability guide](docs/judge-reliability.md) and the offline
|
|
115
|
+
[judge](examples/judge_reliability.py) and [async target](examples/async_live_target.py)
|
|
116
|
+
examples. Report schema 1.0 gains optional fields only.
|
|
117
|
+
|
|
118
|
+
## Declarative CLI (0.8.0)
|
|
119
|
+
|
|
120
|
+
Run saved suites and samples, then apply an explicit release gate:
|
|
121
|
+
|
|
122
|
+
```bash
|
|
123
|
+
agentic-evals evaluate examples/ci/suite.yaml examples/ci/samples.jsonl --out report.json
|
|
124
|
+
agentic-evals gate report.json --config examples/ci/gate.yaml
|
|
125
|
+
agentic-evals pack list
|
|
126
|
+
agentic-evals live examples/ci/suite.yaml --target my_agent:run --max-workers 4 --out live.json
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
`evaluate` and `live` exit 0 when a report is produced, including failing cases.
|
|
130
|
+
`gate` exits 0 for pass, 1 for a failed gate, and 2 for input or usage errors.
|
|
131
|
+
Use `--format json|junit|markdown` for native evidence, JUnit or job summaries.
|
|
132
|
+
See the [CLI guide](docs/cli.md), [CI workflow](examples/ci/evaluate.yml), and
|
|
133
|
+
[voice-agent CLI example](voice-agent-evals/README.md#declarative-cli-080).
|
|
134
|
+
The native report and gate contracts remain compatible with 0.7.2 consumers.
|
|
135
|
+
|
|
86
136
|
## Why a separate package
|
|
87
137
|
|
|
88
138
|
AgenticLens's evaluation module was never actually AgenticLens-specific —
|
|
@@ -367,6 +417,20 @@ A `None` version means the evaluator did not declare one. Version a judge
|
|
|
367
417
|
whenever its prompt or model changes, so two reports can be compared only
|
|
368
418
|
when they were graded by the same judge.
|
|
369
419
|
|
|
420
|
+
### Judge cost, retries and agreement
|
|
421
|
+
|
|
422
|
+
`LLMJudgeEvaluator` and `LLMRubricEvaluator` take `retry=RetryPolicy(...)` and
|
|
423
|
+
`samples=N`. Every judge score records `judge_status` (`ok` or
|
|
424
|
+
`unavailable`), `judge_attempts` and `judge_latency_ms`, plus any usage the
|
|
425
|
+
judge reported (`judge_cost_usd`, `judge_cost_source`, tokens, model,
|
|
426
|
+
provider). A rubric `complete_fn` reports usage by returning a
|
|
427
|
+
`JudgeResponse`. A judge that raises, or a rubric completion that cannot be parsed,
|
|
428
|
+
no longer aborts the suite. After the last attempt it becomes an unavailable score:
|
|
429
|
+
value 0 and never passing, with the error recorded.
|
|
430
|
+
`summary.judge` totals judge cost apart from the agent's cost, and lists
|
|
431
|
+
unavailable cases and per-case agreement. See the
|
|
432
|
+
[judge reliability guide](docs/judge-reliability.md).
|
|
433
|
+
|
|
370
434
|
## Release gates
|
|
371
435
|
|
|
372
436
|
```python
|
|
@@ -394,6 +458,18 @@ GateConfig(
|
|
|
394
458
|
A tag or metric named in the config but missing from the report fails the
|
|
395
459
|
gate, so a renamed tag cannot quietly switch a check off.
|
|
396
460
|
|
|
461
|
+
Judge and live-target checks (0.9.0):
|
|
462
|
+
|
|
463
|
+
```python
|
|
464
|
+
GateConfig(
|
|
465
|
+
max_judge_unavailable_scores=0, # the default: an unavailable judge fails the gate
|
|
466
|
+
min_judge_agreement=0.8, # every case judged with samples=N
|
|
467
|
+
max_judge_cost_usd=2.50, # fails when any judge score has no cost
|
|
468
|
+
max_timed_out_cases=0, # live runs: timeouts, apart from wrong answers
|
|
469
|
+
max_target_error_cases=0,
|
|
470
|
+
)
|
|
471
|
+
```
|
|
472
|
+
|
|
397
473
|
Never fabricates a value it can't back up: `total_cost_usd` on a summary or
|
|
398
474
|
gate decision stays `None` unless every case in scope has a known cost —
|
|
399
475
|
an incomplete cost picture is reported as unavailable, not `$0.00`.
|
|
@@ -511,6 +587,15 @@ targets that are safe to call concurrently. Python and HTTP targets go
|
|
|
511
587
|
through the same scoring path; the test suite checks that both produce the
|
|
512
588
|
same report for the same outputs.
|
|
513
589
|
|
|
590
|
+
`retry=RetryPolicy(max_attempts=3, backoff_seconds=1)` retries a call that
|
|
591
|
+
raises or times out. A case whose calls all fail is reported on the case
|
|
592
|
+
(`target_status` `timed_out` or `error`, `target_attempts`, `target_error`)
|
|
593
|
+
instead of aborting the run, and `summary.live` counts each outcome. HTTP
|
|
594
|
+
targets always apply `timeout_seconds`; Python targets apply it when it is set
|
|
595
|
+
explicitly, cancelling `async def` targets. `arun_live_suite(...,
|
|
596
|
+
max_concurrency=N)` awaits async targets and evaluators; `aevaluate_suite`
|
|
597
|
+
does the same for saved samples.
|
|
598
|
+
|
|
514
599
|
Live targets are intentionally powerful developer-facing integrations —
|
|
515
600
|
Python targets execute local code and HTTP targets can reach arbitrary
|
|
516
601
|
URLs. Only point them at trusted suite files and trusted target
|
|
@@ -649,3 +734,123 @@ Contract `1.x` changes are additive only; readers must tolerate unknown fields.
|
|
|
649
734
|
See [contract policy and sibling compatibility](docs/report-contract.md) and
|
|
650
735
|
[validated JSON examples](examples/report-contract/). The quickstart `EvalResult`
|
|
651
736
|
is a separate format. Native schema validation does not establish AIOS conformance.
|
|
737
|
+
|
|
738
|
+
## Native consumer validation
|
|
739
|
+
|
|
740
|
+
MCP 0.3.8 adds `evals.validate_artifact` and validates Lens report inputs and
|
|
741
|
+
produced SLO gates. Control Tower 0.6.1 validates complete reports and supplied
|
|
742
|
+
gates before evidence projection. Both bundle exact Evals 0.7.2 schemas for
|
|
743
|
+
offline, sibling-independent validation. See [the consumer matrix](docs/report-contract.md).
|
|
744
|
+
|
|
745
|
+
## AgenticLens 0.8.0 suite management
|
|
746
|
+
|
|
747
|
+
AgenticLens 0.8.0 delivers its v0.6 milestone: `agenticlens.suites` and
|
|
748
|
+
`agenticlens suite` build regression suites from CSV/JSON imports, production
|
|
749
|
+
traces and user feedback. It masks PII by default before storage (`--no-mask`
|
|
750
|
+
opts out), detects duplicates and
|
|
751
|
+
conflicting labels, adds synthetic variations and stable splits, records
|
|
752
|
+
reviews, and approves suites into immutable, hash-verified versions that CI
|
|
753
|
+
verifies, exports and evaluates. Suites stay plain agentic-evals `TestSuite`
|
|
754
|
+
files.
|
|
755
|
+
|
|
756
|
+
Evals runs these suites unchanged with `evaluate_suite` and `run_live_suite`.
|
|
757
|
+
Lens keeps management state (lineage, review status, domain, split) in
|
|
758
|
+
`TestCase.metadata` and `tags`, and an exported approved version adds
|
|
759
|
+
`metadata.agenticlens_suite_version` (name, version, SHA-256, approver).
|
|
760
|
+
Suite approval is not a release gate: scoring and gate computation stay in
|
|
761
|
+
Evals. No Evals change or release is needed.
|
|
762
|
+
|
|
763
|
+
See the [suite management guide](https://github.com/DeepAgentLabs/agenticlens/blob/v0.8.0/docs/suite-management.md).
|
|
764
|
+
|
|
765
|
+
## AgenticLens 0.9.0 ModelFit
|
|
766
|
+
|
|
767
|
+
AgenticLens 0.9.0 delivers its v0.7 milestone: `agenticlens.modelfit` and
|
|
768
|
+
`agenticlens modelfit` recommend a model per task category from recorded
|
|
769
|
+
experiment reports, under hard quality, cost, latency, safety, provider and
|
|
770
|
+
deployment constraints that fail closed. Each recommendation carries a
|
|
771
|
+
confidence level with reasons, an explanation, alternatives, a fallback
|
|
772
|
+
(on a different provider when one is feasible), savings against a reference variant tied to the
|
|
773
|
+
experiment evidence, and quality risks, plus a task-level model matrix.
|
|
774
|
+
|
|
775
|
+
Evals is unchanged. ModelFit reads the native evaluation reports that Lens
|
|
776
|
+
experiments embed per trial (`cases[].passed`, `latency_ms`, `cost_usd`,
|
|
777
|
+
`tags`, `metadata`) and computes no release gate; scoring and gate
|
|
778
|
+
computation stay in Evals. Lens keeps its `agentic-evals>=0.7.1,<1`
|
|
779
|
+
dependency. No Evals change or release is needed.
|
|
780
|
+
|
|
781
|
+
See the [ModelFit guide](https://github.com/DeepAgentLabs/agenticlens/blob/v0.9.0/docs/modelfit.md).
|
|
782
|
+
|
|
783
|
+
## Chaos 0.7.0 interoperability
|
|
784
|
+
|
|
785
|
+
Chaos 0.7.0 adds sequential fault cascades, deterministic monotonic breaking-point
|
|
786
|
+
search, synthetic response poisoning (including prompt/tool-response instruction
|
|
787
|
+
and policy-bypass payloads), and explicit-price retry cost estimates. Native
|
|
788
|
+
child event IDs and fault types remain intact; cascade IDs/stages and poisoning
|
|
789
|
+
strategy/payload/seed are additive event detail. Existing report readers and
|
|
790
|
+
AIOS exports remain compatible.
|
|
791
|
+
|
|
792
|
+
See the [v0.7 guide and offline synthetic example](https://github.com/DeepAgentLabs/agentic-chaos/blob/v0.7.0/docs/v0.7.md).
|
|
793
|
+
The example includes observed synthetic dispatches and a bypassed negative
|
|
794
|
+
control. Injection or recovery alone does not prove containment. No new shared
|
|
795
|
+
schema, automatic adapter, hosted execution capability or containment guarantee
|
|
796
|
+
is introduced.
|
|
797
|
+
|
|
798
|
+
Scores and release gates remain owned by Evals; Chaos contracts and threshold trials are per-campaign evidence. Convert observations to native EvaluationSamples explicitly. No runtime dependency or patch release is needed.
|
|
799
|
+
|
|
800
|
+
## Control Tower 0.7.0 fleet operations
|
|
801
|
+
|
|
802
|
+
Tower 0.7.0 adds frozen environment/capability targeting, bulk desired settings,
|
|
803
|
+
separate approval, durable per-target results and bounded stop/MCP-revocation
|
|
804
|
+
acknowledgement through Python, HTTP and CLI. No fleet atomicity or automatic
|
|
805
|
+
rollback/resume is claimed. Timed-out/requested commands need host reconciliation.
|
|
806
|
+
|
|
807
|
+
Evals retains scoring and native report/gate decisions. Bulk execution does not recompute gates or equate passing evaluation with deployment health or containment.
|
|
808
|
+
|
|
809
|
+
Existing optional dependency ranges remain sufficient. No sibling runtime or
|
|
810
|
+
native evidence schema changes are required; this documentation alignment needs
|
|
811
|
+
no patch release. See the [operator guide](https://github.com/DeepAgentLabs/agenticops-control-tower/blob/v0.7.0/docs/fleet-operations.md).
|
|
812
|
+
|
|
813
|
+
See the [local Tower compatibility guide](docs/tower-0.7-fleet-operations.md) for the operator workflow and package boundaries.
|
|
814
|
+
|
|
815
|
+
## AgenticLens 0.10.0 routing compatibility
|
|
816
|
+
|
|
817
|
+
See the [local integration boundary guide](docs/lens-0.10-routing.md).
|
|
818
|
+
|
|
819
|
+
AgenticLens 0.10.0 implements roadmap v0.8: evidence-based runtime model
|
|
820
|
+
selection, fail-closed routing constraints, bounded provider failover,
|
|
821
|
+
confidence/escalation handoffs, shadow/canary modes and a local routing journal.
|
|
822
|
+
The default is advisory; execution uses explicit caller-owned callbacks.
|
|
823
|
+
MCP 0.3.12 exposes read-only `lens.routing_decide` and pins hosted Lens 0.10.0.
|
|
824
|
+
See the [routing guide](https://github.com/DeepAgentLabs/agenticlens/blob/v0.10.0/docs/routing.md)
|
|
825
|
+
and [offline example](https://github.com/DeepAgentLabs/agenticlens/blob/v0.10.0/examples/routing_demo.py).
|
|
826
|
+
|
|
827
|
+
Lens retains `agentic-evals>=0.7.1,<1`, which covers current Evals 0.8.0.
|
|
828
|
+
Standalone scoring and release-gate computation remain Evals-owned. Routing
|
|
829
|
+
outcomes do not redefine native evaluation reports or gate decisions. No dependency
|
|
830
|
+
change or Evals patch release is required.
|
|
831
|
+
|
|
832
|
+
## Chaos 0.8.0 interoperability
|
|
833
|
+
|
|
834
|
+
Chaos 0.8.0 adds:
|
|
835
|
+
|
|
836
|
+
- multi-stage chaos workflows: steps, steady-state health checks, parallel
|
|
837
|
+
branches, approval gates that can be resumed, and reusable templates
|
|
838
|
+
- declarative YAML/JSON/TOML experiments, with validation that never executes
|
|
839
|
+
targets and cron schedule previews
|
|
840
|
+
- `Scope` explosion-radius control (steps, providers and exclusions) on every
|
|
841
|
+
built-in fault and `cascade()`; custom faults get it by calling
|
|
842
|
+
`BaseFault.__init__`, setting `scope` or overriding `in_scope()`
|
|
843
|
+
- a topology fuzzer that weights fault placement by fan-out
|
|
844
|
+
|
|
845
|
+
Native event IDs and fault types are unchanged. Workflow provenance
|
|
846
|
+
(`detail.workflow`) and an optional `ChaosReport.workflow` summary are
|
|
847
|
+
additive. A workflow's steps merge into one native report for contracts and
|
|
848
|
+
AIOS export.
|
|
849
|
+
|
|
850
|
+
See the [v0.8 guide](https://github.com/DeepAgentLabs/agentic-chaos/blob/v0.8.0/docs/v0.8.md)
|
|
851
|
+
and the [offline examples](https://github.com/DeepAgentLabs/agentic-chaos/tree/v0.8.0/examples/workflows).
|
|
852
|
+
Workflow statuses, health checks and fuzz outcomes are resilience evidence.
|
|
853
|
+
They are not scores, release gates or proof of containment. No new shared
|
|
854
|
+
schema, automatic adapter or hosted execution capability is introduced.
|
|
855
|
+
|
|
856
|
+
Workflow statuses and per-step contracts are per-run resilience checks, not Evals scores or release gates. Convert observed outputs to native EvaluationSamples explicitly. No adapter, runtime dependency or Evals patch release is needed.
|
|
@@ -23,6 +23,55 @@ quickstart API, and the `agentic-evals` CLI) is real, tested code — see the
|
|
|
23
23
|
still move between minor versions ahead of a 1.0; pin a version in
|
|
24
24
|
production and read the changelog before upgrading.
|
|
25
25
|
|
|
26
|
+
## Judge reliability and live resilience (0.9.0)
|
|
27
|
+
|
|
28
|
+
LLM-judged and live evaluations now say what they cost and whether they
|
|
29
|
+
actually happened:
|
|
30
|
+
|
|
31
|
+
```python
|
|
32
|
+
from agentic_evals import FACTUALITY, JudgeResponse, LLMRubricEvaluator, RetryPolicy
|
|
33
|
+
|
|
34
|
+
judge = LLMRubricEvaluator(
|
|
35
|
+
"factuality",
|
|
36
|
+
FACTUALITY,
|
|
37
|
+
complete_fn, # may return JudgeResponse(text=..., cost_usd=...)
|
|
38
|
+
retry=RetryPolicy(max_attempts=3, backoff_seconds=0.5),
|
|
39
|
+
samples=3, # judge three times, report agreement
|
|
40
|
+
)
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
- Judge cost, tokens and latency are recorded on each judge `Score` under the
|
|
44
|
+
`judge_*` keys AgenticLens already reads, and totalled in `summary.judge`.
|
|
45
|
+
- A judge that fails after its retries yields an explicit **unavailable** score
|
|
46
|
+
that never passes; the default gate fails and names the cases.
|
|
47
|
+
- `samples=N` reports per-case agreement; `GateConfig(min_judge_agreement=...)`
|
|
48
|
+
catches a flaky judge.
|
|
49
|
+
- Live targets retry with backoff and report `timed_out` separately from failed
|
|
50
|
+
(`CaseEvaluation.target_status`, `summary.live`, JUnit `<error>`).
|
|
51
|
+
- `aevaluate_suite` / `arun_live_suite` await `async def` evaluators and targets.
|
|
52
|
+
|
|
53
|
+
See the [judge reliability guide](docs/judge-reliability.md) and the offline
|
|
54
|
+
[judge](examples/judge_reliability.py) and [async target](examples/async_live_target.py)
|
|
55
|
+
examples. Report schema 1.0 gains optional fields only.
|
|
56
|
+
|
|
57
|
+
## Declarative CLI (0.8.0)
|
|
58
|
+
|
|
59
|
+
Run saved suites and samples, then apply an explicit release gate:
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
agentic-evals evaluate examples/ci/suite.yaml examples/ci/samples.jsonl --out report.json
|
|
63
|
+
agentic-evals gate report.json --config examples/ci/gate.yaml
|
|
64
|
+
agentic-evals pack list
|
|
65
|
+
agentic-evals live examples/ci/suite.yaml --target my_agent:run --max-workers 4 --out live.json
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
`evaluate` and `live` exit 0 when a report is produced, including failing cases.
|
|
69
|
+
`gate` exits 0 for pass, 1 for a failed gate, and 2 for input or usage errors.
|
|
70
|
+
Use `--format json|junit|markdown` for native evidence, JUnit or job summaries.
|
|
71
|
+
See the [CLI guide](docs/cli.md), [CI workflow](examples/ci/evaluate.yml), and
|
|
72
|
+
[voice-agent CLI example](voice-agent-evals/README.md#declarative-cli-080).
|
|
73
|
+
The native report and gate contracts remain compatible with 0.7.2 consumers.
|
|
74
|
+
|
|
26
75
|
## Why a separate package
|
|
27
76
|
|
|
28
77
|
AgenticLens's evaluation module was never actually AgenticLens-specific —
|
|
@@ -307,6 +356,20 @@ A `None` version means the evaluator did not declare one. Version a judge
|
|
|
307
356
|
whenever its prompt or model changes, so two reports can be compared only
|
|
308
357
|
when they were graded by the same judge.
|
|
309
358
|
|
|
359
|
+
### Judge cost, retries and agreement
|
|
360
|
+
|
|
361
|
+
`LLMJudgeEvaluator` and `LLMRubricEvaluator` take `retry=RetryPolicy(...)` and
|
|
362
|
+
`samples=N`. Every judge score records `judge_status` (`ok` or
|
|
363
|
+
`unavailable`), `judge_attempts` and `judge_latency_ms`, plus any usage the
|
|
364
|
+
judge reported (`judge_cost_usd`, `judge_cost_source`, tokens, model,
|
|
365
|
+
provider). A rubric `complete_fn` reports usage by returning a
|
|
366
|
+
`JudgeResponse`. A judge that raises, or a rubric completion that cannot be parsed,
|
|
367
|
+
no longer aborts the suite. After the last attempt it becomes an unavailable score:
|
|
368
|
+
value 0 and never passing, with the error recorded.
|
|
369
|
+
`summary.judge` totals judge cost apart from the agent's cost, and lists
|
|
370
|
+
unavailable cases and per-case agreement. See the
|
|
371
|
+
[judge reliability guide](docs/judge-reliability.md).
|
|
372
|
+
|
|
310
373
|
## Release gates
|
|
311
374
|
|
|
312
375
|
```python
|
|
@@ -334,6 +397,18 @@ GateConfig(
|
|
|
334
397
|
A tag or metric named in the config but missing from the report fails the
|
|
335
398
|
gate, so a renamed tag cannot quietly switch a check off.
|
|
336
399
|
|
|
400
|
+
Judge and live-target checks (0.9.0):
|
|
401
|
+
|
|
402
|
+
```python
|
|
403
|
+
GateConfig(
|
|
404
|
+
max_judge_unavailable_scores=0, # the default: an unavailable judge fails the gate
|
|
405
|
+
min_judge_agreement=0.8, # every case judged with samples=N
|
|
406
|
+
max_judge_cost_usd=2.50, # fails when any judge score has no cost
|
|
407
|
+
max_timed_out_cases=0, # live runs: timeouts, apart from wrong answers
|
|
408
|
+
max_target_error_cases=0,
|
|
409
|
+
)
|
|
410
|
+
```
|
|
411
|
+
|
|
337
412
|
Never fabricates a value it can't back up: `total_cost_usd` on a summary or
|
|
338
413
|
gate decision stays `None` unless every case in scope has a known cost —
|
|
339
414
|
an incomplete cost picture is reported as unavailable, not `$0.00`.
|
|
@@ -451,6 +526,15 @@ targets that are safe to call concurrently. Python and HTTP targets go
|
|
|
451
526
|
through the same scoring path; the test suite checks that both produce the
|
|
452
527
|
same report for the same outputs.
|
|
453
528
|
|
|
529
|
+
`retry=RetryPolicy(max_attempts=3, backoff_seconds=1)` retries a call that
|
|
530
|
+
raises or times out. A case whose calls all fail is reported on the case
|
|
531
|
+
(`target_status` `timed_out` or `error`, `target_attempts`, `target_error`)
|
|
532
|
+
instead of aborting the run, and `summary.live` counts each outcome. HTTP
|
|
533
|
+
targets always apply `timeout_seconds`; Python targets apply it when it is set
|
|
534
|
+
explicitly, cancelling `async def` targets. `arun_live_suite(...,
|
|
535
|
+
max_concurrency=N)` awaits async targets and evaluators; `aevaluate_suite`
|
|
536
|
+
does the same for saved samples.
|
|
537
|
+
|
|
454
538
|
Live targets are intentionally powerful developer-facing integrations —
|
|
455
539
|
Python targets execute local code and HTTP targets can reach arbitrary
|
|
456
540
|
URLs. Only point them at trusted suite files and trusted target
|
|
@@ -589,3 +673,123 @@ Contract `1.x` changes are additive only; readers must tolerate unknown fields.
|
|
|
589
673
|
See [contract policy and sibling compatibility](docs/report-contract.md) and
|
|
590
674
|
[validated JSON examples](examples/report-contract/). The quickstart `EvalResult`
|
|
591
675
|
is a separate format. Native schema validation does not establish AIOS conformance.
|
|
676
|
+
|
|
677
|
+
## Native consumer validation
|
|
678
|
+
|
|
679
|
+
MCP 0.3.8 adds `evals.validate_artifact` and validates Lens report inputs and
|
|
680
|
+
produced SLO gates. Control Tower 0.6.1 validates complete reports and supplied
|
|
681
|
+
gates before evidence projection. Both bundle exact Evals 0.7.2 schemas for
|
|
682
|
+
offline, sibling-independent validation. See [the consumer matrix](docs/report-contract.md).
|
|
683
|
+
|
|
684
|
+
## AgenticLens 0.8.0 suite management
|
|
685
|
+
|
|
686
|
+
AgenticLens 0.8.0 delivers its v0.6 milestone: `agenticlens.suites` and
|
|
687
|
+
`agenticlens suite` build regression suites from CSV/JSON imports, production
|
|
688
|
+
traces and user feedback. It masks PII by default before storage (`--no-mask`
|
|
689
|
+
opts out), detects duplicates and
|
|
690
|
+
conflicting labels, adds synthetic variations and stable splits, records
|
|
691
|
+
reviews, and approves suites into immutable, hash-verified versions that CI
|
|
692
|
+
verifies, exports and evaluates. Suites stay plain agentic-evals `TestSuite`
|
|
693
|
+
files.
|
|
694
|
+
|
|
695
|
+
Evals runs these suites unchanged with `evaluate_suite` and `run_live_suite`.
|
|
696
|
+
Lens keeps management state (lineage, review status, domain, split) in
|
|
697
|
+
`TestCase.metadata` and `tags`, and an exported approved version adds
|
|
698
|
+
`metadata.agenticlens_suite_version` (name, version, SHA-256, approver).
|
|
699
|
+
Suite approval is not a release gate: scoring and gate computation stay in
|
|
700
|
+
Evals. No Evals change or release is needed.
|
|
701
|
+
|
|
702
|
+
See the [suite management guide](https://github.com/DeepAgentLabs/agenticlens/blob/v0.8.0/docs/suite-management.md).
|
|
703
|
+
|
|
704
|
+
## AgenticLens 0.9.0 ModelFit
|
|
705
|
+
|
|
706
|
+
AgenticLens 0.9.0 delivers its v0.7 milestone: `agenticlens.modelfit` and
|
|
707
|
+
`agenticlens modelfit` recommend a model per task category from recorded
|
|
708
|
+
experiment reports, under hard quality, cost, latency, safety, provider and
|
|
709
|
+
deployment constraints that fail closed. Each recommendation carries a
|
|
710
|
+
confidence level with reasons, an explanation, alternatives, a fallback
|
|
711
|
+
(on a different provider when one is feasible), savings against a reference variant tied to the
|
|
712
|
+
experiment evidence, and quality risks, plus a task-level model matrix.
|
|
713
|
+
|
|
714
|
+
Evals is unchanged. ModelFit reads the native evaluation reports that Lens
|
|
715
|
+
experiments embed per trial (`cases[].passed`, `latency_ms`, `cost_usd`,
|
|
716
|
+
`tags`, `metadata`) and computes no release gate; scoring and gate
|
|
717
|
+
computation stay in Evals. Lens keeps its `agentic-evals>=0.7.1,<1`
|
|
718
|
+
dependency. No Evals change or release is needed.
|
|
719
|
+
|
|
720
|
+
See the [ModelFit guide](https://github.com/DeepAgentLabs/agenticlens/blob/v0.9.0/docs/modelfit.md).
|
|
721
|
+
|
|
722
|
+
## Chaos 0.7.0 interoperability
|
|
723
|
+
|
|
724
|
+
Chaos 0.7.0 adds sequential fault cascades, deterministic monotonic breaking-point
|
|
725
|
+
search, synthetic response poisoning (including prompt/tool-response instruction
|
|
726
|
+
and policy-bypass payloads), and explicit-price retry cost estimates. Native
|
|
727
|
+
child event IDs and fault types remain intact; cascade IDs/stages and poisoning
|
|
728
|
+
strategy/payload/seed are additive event detail. Existing report readers and
|
|
729
|
+
AIOS exports remain compatible.
|
|
730
|
+
|
|
731
|
+
See the [v0.7 guide and offline synthetic example](https://github.com/DeepAgentLabs/agentic-chaos/blob/v0.7.0/docs/v0.7.md).
|
|
732
|
+
The example includes observed synthetic dispatches and a bypassed negative
|
|
733
|
+
control. Injection or recovery alone does not prove containment. No new shared
|
|
734
|
+
schema, automatic adapter, hosted execution capability or containment guarantee
|
|
735
|
+
is introduced.
|
|
736
|
+
|
|
737
|
+
Scores and release gates remain owned by Evals; Chaos contracts and threshold trials are per-campaign evidence. Convert observations to native EvaluationSamples explicitly. No runtime dependency or patch release is needed.
|
|
738
|
+
|
|
739
|
+
## Control Tower 0.7.0 fleet operations
|
|
740
|
+
|
|
741
|
+
Tower 0.7.0 adds frozen environment/capability targeting, bulk desired settings,
|
|
742
|
+
separate approval, durable per-target results and bounded stop/MCP-revocation
|
|
743
|
+
acknowledgement through Python, HTTP and CLI. No fleet atomicity or automatic
|
|
744
|
+
rollback/resume is claimed. Timed-out/requested commands need host reconciliation.
|
|
745
|
+
|
|
746
|
+
Evals retains scoring and native report/gate decisions. Bulk execution does not recompute gates or equate passing evaluation with deployment health or containment.
|
|
747
|
+
|
|
748
|
+
Existing optional dependency ranges remain sufficient. No sibling runtime or
|
|
749
|
+
native evidence schema changes are required; this documentation alignment needs
|
|
750
|
+
no patch release. See the [operator guide](https://github.com/DeepAgentLabs/agenticops-control-tower/blob/v0.7.0/docs/fleet-operations.md).
|
|
751
|
+
|
|
752
|
+
See the [local Tower compatibility guide](docs/tower-0.7-fleet-operations.md) for the operator workflow and package boundaries.
|
|
753
|
+
|
|
754
|
+
## AgenticLens 0.10.0 routing compatibility
|
|
755
|
+
|
|
756
|
+
See the [local integration boundary guide](docs/lens-0.10-routing.md).
|
|
757
|
+
|
|
758
|
+
AgenticLens 0.10.0 implements roadmap v0.8: evidence-based runtime model
|
|
759
|
+
selection, fail-closed routing constraints, bounded provider failover,
|
|
760
|
+
confidence/escalation handoffs, shadow/canary modes and a local routing journal.
|
|
761
|
+
The default is advisory; execution uses explicit caller-owned callbacks.
|
|
762
|
+
MCP 0.3.12 exposes read-only `lens.routing_decide` and pins hosted Lens 0.10.0.
|
|
763
|
+
See the [routing guide](https://github.com/DeepAgentLabs/agenticlens/blob/v0.10.0/docs/routing.md)
|
|
764
|
+
and [offline example](https://github.com/DeepAgentLabs/agenticlens/blob/v0.10.0/examples/routing_demo.py).
|
|
765
|
+
|
|
766
|
+
Lens retains `agentic-evals>=0.7.1,<1`, which covers current Evals 0.8.0.
|
|
767
|
+
Standalone scoring and release-gate computation remain Evals-owned. Routing
|
|
768
|
+
outcomes do not redefine native evaluation reports or gate decisions. No dependency
|
|
769
|
+
change or Evals patch release is required.
|
|
770
|
+
|
|
771
|
+
## Chaos 0.8.0 interoperability
|
|
772
|
+
|
|
773
|
+
Chaos 0.8.0 adds:
|
|
774
|
+
|
|
775
|
+
- multi-stage chaos workflows: steps, steady-state health checks, parallel
|
|
776
|
+
branches, approval gates that can be resumed, and reusable templates
|
|
777
|
+
- declarative YAML/JSON/TOML experiments, with validation that never executes
|
|
778
|
+
targets and cron schedule previews
|
|
779
|
+
- `Scope` explosion-radius control (steps, providers and exclusions) on every
|
|
780
|
+
built-in fault and `cascade()`; custom faults get it by calling
|
|
781
|
+
`BaseFault.__init__`, setting `scope` or overriding `in_scope()`
|
|
782
|
+
- a topology fuzzer that weights fault placement by fan-out
|
|
783
|
+
|
|
784
|
+
Native event IDs and fault types are unchanged. Workflow provenance
|
|
785
|
+
(`detail.workflow`) and an optional `ChaosReport.workflow` summary are
|
|
786
|
+
additive. A workflow's steps merge into one native report for contracts and
|
|
787
|
+
AIOS export.
|
|
788
|
+
|
|
789
|
+
See the [v0.8 guide](https://github.com/DeepAgentLabs/agentic-chaos/blob/v0.8.0/docs/v0.8.md)
|
|
790
|
+
and the [offline examples](https://github.com/DeepAgentLabs/agentic-chaos/tree/v0.8.0/examples/workflows).
|
|
791
|
+
Workflow statuses, health checks and fuzz outcomes are resilience evidence.
|
|
792
|
+
They are not scores, release gates or proof of containment. No new shared
|
|
793
|
+
schema, automatic adapter or hosted execution capability is introduced.
|
|
794
|
+
|
|
795
|
+
Workflow statuses and per-step contracts are per-run resilience checks, not Evals scores or release gates. Convert observed outputs to native EvaluationSamples explicitly. No adapter, runtime dependency or Evals patch release is needed.
|