agentme 0.26.0 → 0.27.0

package/.xdrs/agentme/edrs/application/018-ai-llm-development-standards.md

@@ -218,4 +218,6 @@ Return a JSON object with:
 - [agentme-edr-004](../principles/004-unit-test-requirements.md) — Unit test requirements including external API mocking guidance
 - [agentme-edr-014](014-python-project-tooling.md) — Python project tooling and structure
 - [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards including AI-tier testing requirements (rule `09-ai-project-testing-requirements`)
-- [agentme-edr-028](028-ai-eval-standards.md) — AI eval standards: folder structure, script requirements, and MLflow tracking
+- [agentme-edr-028](028-ai-eval-core-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract (rule `02`)
+- [agentme-edr-031](031-ai-eval-script.md) — AI eval script: entry-first loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
+- [agentme-edr-032](032-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis

package/.xdrs/agentme/edrs/application/020-ai-agents-quality-standards.md

@@ -179,4 +179,6 @@ def test_workflow_calls_subagent(mocker):
 - [agentme-edr-018](018-ai-llm-development-standards.md) — LLM development standards (LangChain configuration, mocking patterns)
 - [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) — Hexagonal architecture (tool placement in adapters/connectors)
 - [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards including AI-tier testing requirements (rule `09-ai-project-testing-requirements`)
-- [agentme-edr-028](028-ai-eval-standards.md) — AI eval standards: folder structure, script requirements, and MLflow tracking
+- [agentme-edr-028](028-ai-eval-core-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract (rule `02`)
+- [agentme-edr-031](031-ai-eval-script.md) — AI eval script: entry-first loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
+- [agentme-edr-032](032-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis

package/.xdrs/agentme/edrs/application/021-ai-workflow-development-standards.md

@@ -37,7 +37,7 @@ Use **MLflow** for all workflow observability and evaluation:
 
 #### 04-dataset-driven-accuracy-measurement
 
-Projects MUST follow the eval dataset and implementation requirements defined in [agentme-edr-028](028-ai-eval-standards.md). Testing requirements (when evals are required, release gates) are defined in [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
+Projects MUST follow the eval dataset and implementation requirements defined in [agentme-edr-028](028-ai-eval-core-standards.md) and [agentme-edr-031](031-ai-eval-script.md). Testing requirements (when evals are required, release gates) are defined in [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
 
 #### 05-flow-documentation
 
@@ -101,7 +101,7 @@ lib/src/<package_name>/
 
 #### 08-workflow-evals
 
-Projects MUST follow the eval folder structure and script requirements defined in [agentme-edr-028](028-ai-eval-standards.md).
+Projects MUST follow the eval folder structure defined in [agentme-edr-028](028-ai-eval-core-standards.md) rule `01` and the eval script requirements defined in [agentme-edr-031](031-ai-eval-script.md). For LLM-as-judge scoring used in workflow verification nodes and evals, see [agentme-edr-028](028-ai-eval-core-standards.md) rule `02`.
 
 #### 09-node-naming-conventions
 
@@ -207,5 +207,8 @@ All workflow elements MUST maintain naming coherence as defined in [agentme-edr-
 - [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) — Adapter/application layer separation that defines the project layout
 - [agentme-edr-014](014-python-project-tooling.md) — Python project tooling and structure
 - [agentme-edr-024](024-ml-dataset-structure.md) — ML dataset structure for eval datasets
-- [agentme-edr-028](028-ai-eval-standards.md) — AI eval standards: folder structure, script requirements, and MLflow tracking
+- [agentme-edr-028](028-ai-eval-core-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract for all tiers (rule `02`)
+- [agentme-edr-031](031-ai-eval-script.md) — AI eval script: entry-first loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
+- [agentme-edr-032](032-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
+- [agentme-edr-033](033-ai-eval-repeatability.md) — AI eval repeatability: REPEAT_COUNT loop, scoring methods, and release cadence
 - [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards including AI-tier testing requirements (rule `09-ai-project-testing-requirements`)

package/.xdrs/agentme/edrs/application/028-ai-eval-core-standards.md

@@ -0,0 +1,120 @@
+---
+name: agentme-edr-policy-028-ai-eval-core-standards
+description: Defines the eval folder structure and Makefile interface for AI projects, and the LLM-as-judge binary scoring contract applicable to all AI tiers and test types. Use when scaffolding evals or implementing an LLM judge. For eval script requirements see agentme-edr-031. For report format see agentme-edr-032. For repeatability see agentme-edr-033. For when evals are required see agentme-edr-007 rule 09-ai-project-testing-requirements. For the test type taxonomy see agentme-edr-030.
+apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
+valid-from: 2026-06-05
+---
+
+# agentme-edr-policy-028: AI eval core standards
+
+## Context and Problem Statement
+
+Eval tests measure AI component accuracy against expected outputs using real LLM providers. Without a shared folder layout and a common LLM judge contract, eval setups diverge across LLM, Agent, and Workflow projects, making them hard to scaffold, run, and compare.
+
+How should eval tests be structured, and how must LLM judges produce scores across all AI tiers?
+
+## Decision Outcome
+
+**Use a per-component folder structure under `evals/` with a standardized Makefile interface, and require all LLM judges to produce binary (`0`/`1`) output compatible with classification metrics.**
+
+For when evals are required per AI tier, see [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
+
+### Details
+
+#### 01-eval-folder-structure
+
+Evals are grouped first by the component being evaluated, then by the specific evaluation scenario. Create one directory per component under `evals/`, and one directory per eval scenario inside it. Place `evals/` at the same level as `lib/` and `examples/`:
+
+```text
+evals/
+  <component>/           # the component being evaluated (e.g., workflow-x, agent-y, model-z)
+    eval-<name>/
+      golden_dataset/    # EDR-024 + EDR-030 compliant golden dataset (README.md, dataset.schema.json, data/)
+      eval.py            # evaluation script
+      report-<type>.md   # generated report, one per evaluated test type (overwritten on each run — see agentme-edr-032 rule 01)
+      Makefile           # lint, eval, run, and eval-<type> targets
+    eval-<name2>/
+      ...
+  <component2>/
+    ...
+```
+
+`<component>` MUST match the name of the component under evaluation and use lowercase hyphen-separated words (e.g., `workflow-document-review`, `agent-support`, `model-classifier`).
+
+`<name>` identifies the specific evaluation scenario using lowercase hyphen-separated words (e.g., `eval-basic`, `eval-complex`, `eval-edge-cases`). A scenario's `golden_dataset` MAY mix multiple test types across its entries: label each entry with its applicable `test_types` ([agentme-edr-030](030-ai-test-types-taxonomy.md) rule `04`) and use the `eval-<type>` targets below to run one type at a time.
+
+The `golden_dataset/` subfolder MUST be a valid [agentme-edr-024](024-ml-dataset-structure.md) dataset (`README.md`, `dataset.schema.json`, one JSON file per entry under `data/` per rule `04-complex-structured-datasets-must-use-per-entry-json-files`, lint-validated per rule `06`) whose entries follow the golden dataset envelope defined in [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02`.
+
+Each `evals/<component>/eval-<name>/Makefile` MUST declare a `TEST_TYPES` variable listing the `test_types` values present in its golden dataset, and define:
+
+| Target | Behaviour |
+|---|---|
+| `lint` | Validates every `golden_dataset/data/*.json` file against `golden_dataset/dataset.schema.json` per [agentme-edr-024](024-ml-dataset-structure.md) rule `06` |
+| `eval` | Depends on `lint`; runs `eval.py --type=all` with threshold enforcement; exits non-zero on failure (CI-safe) |
+| `run` | Depends on `lint`; runs `eval.py --type=all` with threshold enforcement; same as `eval` but intended for local exploration |
+| `eval-<type>` | Depends on `lint`; runs `eval.py --type=<type>` for one declared test type, following [agentme-edr-008](../devops/008-common-targets.md) rule `03`'s `eval-<qualifier>` convention |
+
+```makefile
+TEST_TYPES := smoke functional safety
+
+lint:
+	mise exec -- uv run --project . python lint_dataset.py golden_dataset/
+
+eval: lint
+	mise exec -- uv run --project . python eval.py --type=all
+
+run: lint
+	mise exec -- uv run --project . python eval.py --type=all
+
+eval-%: lint
+	mise exec -- uv run --project . python eval.py --type=$*
+```
+
+The module root Makefile MUST expose `make eval` and `make lint` targets that delegate to `eval` and `lint` respectively in every `evals/<component>/eval-<name>/Makefile`:
+
+```makefile
+eval:
+	$(MAKE) -C evals/workflow-document-review/eval-basic eval
+	$(MAKE) -C evals/workflow-document-review/eval-complex eval
+
+lint:
+	$(MAKE) -C evals/workflow-document-review/eval-basic lint
+	$(MAKE) -C evals/workflow-document-review/eval-complex lint
+```
+
+#### 02-llm-as-judge-binary-output
+
+LLM judges scoring component outputs MUST produce binary output: `0` (fail) or `1` (success). This rule applies to all AI tiers (LLM, Agent, Workflow) and to all eval test types that use an LLM judge (functional, quality, safety, repeatability, or any other).
+
+**Requirements:**
+
+- Judge prompts MUST instruct the model to output exactly `0` or `1`
+- Scoring logic MUST parse the response and map to binary. Ambiguous/invalid responses: score as `0` or raise error
+- Reports using LLM judges MUST use classification metrics (Accuracy, F1, Precision, Recall), not regression metrics (RMSE, R2, MAE)
+- Multi-class classification not supported. For multiple quality levels, use multiple binary judges (e.g., one for "factually correct", another for "tone appropriate")
+
+**Rationale:** Binary output makes LLM judges compatible with classification metrics infrastructure (Accuracy, F1, Wilson CI, convergence analysis) defined in [agentme-edr-032](032-ai-eval-report-format.md) rule `01`.
+
+**Example LLM judge prompt:**
+
+```
+Evaluate whether the document review decision is correct.
+
+Input: {input_summary}
+Expected: {expected_decision}
+Actual: {actual_decision}
+
+Output exactly "1" if the actual decision matches the expected decision and reasoning, or "0" if it does not.
+
+Output:
+```
+
+## References
+
+- [agentme-edr-031](031-ai-eval-script.md) — AI eval script: entry-first eval loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
+- [agentme-edr-032](032-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
+- [agentme-edr-033](033-ai-eval-repeatability.md) — AI eval repeatability: loop exception, scoring methods (including LLM-as-judge per rule `02`), and cadence
+- [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09`) and statistical model eval targets (rule `07`)
+- [agentme-edr-030](030-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum and golden dataset entry envelope
+- [agentme-edr-024](024-ml-dataset-structure.md) — ML dataset structure, per-entry JSON format, and schema-lint validation for golden datasets
+- [agentme-edr-008](../devops/008-common-targets.md) — `eval-<qualifier>` Makefile convention (rule `03`)

package/.xdrs/agentme/edrs/application/030-ai-test-types-taxonomy.md

@@ -21,7 +21,7 @@ Each test type is named with its group, objective, mocking constraint, applicabi
 
 #### 01-golden-dataset-concept
 
-Projects MUST use a golden dataset to test AI components. A **golden dataset** comprises all eval case entries used to test an AI component (LLM, Agent, or Workflow tier); each entry is labeled with the `test_types` (rule `04`) it applies to. It is the dataset consumed by [agentme-edr-028](028-ai-eval-standards.md) evals and stored as one JSON file per entry per [agentme-edr-024](024-ml-dataset-structure.md) rule `04`, at `evals/<component>/eval-<name>/golden_dataset/`.
+Projects MUST use a golden dataset to test AI components. A **golden dataset** comprises all eval case entries used to test an AI component (LLM, Agent, or Workflow tier); each entry is labeled with the `test_types` (rule `04`) it applies to. It is the dataset consumed by [agentme-edr-031](031-ai-eval-script.md) evals and stored as one JSON file per entry per [agentme-edr-024](024-ml-dataset-structure.md) rule `04`, at `evals/<component>/eval-<name>/golden_dataset/`.
 
 #### 02-golden-dataset-entry-envelope
 
@@ -47,52 +47,70 @@ Every golden dataset entry (a JSON file in `golden_dataset/data/`) MUST have thi
 
 #### 03-mocks-allowed-values
 
-The taxonomy in rule `05` rates each test type using one of three values:
+The taxonomy in rule `05` rates each test type using one of three values under the `Mock Constraint` column:
 
 | Value | Meaning |
 |---|---|
-| `mocks allowed` | Fully offline; fakes may replace every dependency (e.g. `FakeListChatModel` per [agentme-edr-018](018-ai-llm-development-standards.md) rule `04`). |
-| `mocks disallowed` | No mocking of any dependency — real external systems required. |
-| `mocks disallowed for LLM calls` | Tools and other external/dependency calls MAY be mocked; only the LLM call itself MUST be real for the test to be meaningful. |
+| `mocks allowed` | Fully offline; fakes may replace every dependency including the LLM (e.g. `FakeListChatModel` per [agentme-edr-018](018-ai-llm-development-standards.md) rule `04`). Used only for code-level unit tests. |
+| `mocks disallowed` | No mocking of any dependency — all real external systems required. Used for integration tests. |
+| `mocks disallowed for LLM calls` | **The LLM call MUST be real; all other external dependencies (databases, APIs, external services) MAY and SHOULD be mocked via `mock_fixtures`.** `mock_fixtures` keys MUST NOT reference LLM adapters. This is the standard constraint for every golden-dataset eval test type. See rule `08` for rationale. |
 
 #### 04-test-types-enum
 
-A golden dataset entry's `test_types` array MUST only use these values: `safety`, `adversarial`, `fairness`, `bias`, `robustness`, `explainability`, `groundedness`, `functional`, `prompt`, `smoke`, `human`. These correspond to the dataset-driven rows of rule `05`. **Unit test** and **Integration test** (the two Code-level rows) are NOT part of this enum — they have no golden dataset entries and remain governed entirely by [agentme-edr-004](../principles/004-unit-test-requirements.md) and [agentme-edr-007](../principles/007-project-quality-standards.md) rule `08`.
+A golden dataset entry's `test_types` array MUST only use these values: `safety`, `adversarial`, `fairness`, `bias`, `robustness`, `explainability`, `groundedness`, `functional`, `prompt`, `smoke`, `human`, `repeatability`. These correspond to the dataset-driven rows of rule `05`. **Unit test** and **Integration test** (the two Code-level rows) are NOT part of this enum — they have no golden dataset entries and remain governed entirely by [agentme-edr-004](../principles/004-unit-test-requirements.md) and [agentme-edr-007](../principles/007-project-quality-standards.md) rule `08`.
 
 #### 05-test-type-taxonomy
 
 Test types MUST be selected from this taxonomy. Each test type is named with its group, objective, mocking constraint, applicability, and relevance:
 
-| Test Type Name | Group | Test Objective | Mocks Allowed | When to Apply | Relevance – Business | Relevance – Development Team | Priority (1-5) |
-|---|---|---|---|---|---|---|---|
-| Safety/content eval | Safety & adversarial | Detect harmful, biased, or policy-violating output | mocks disallowed for LLM calls | Any user-facing release | Avoids reputational harm; acceptable-use compliance | Automated content gate before merge/release | 5 |
-| Adversarial/red-team test | Safety & adversarial | Probe for prompt injection, jailbreaks, unsafe tool use | mocks disallowed for LLM calls | System exposes tool-invocation or agent loops | Reduces security-incident/breach liability | Finds exploitable tool-loop paths before attackers do | 5 |
-| Fairness test | Responsible AI | Verify equitable outcomes across user groups | mocks disallowed for LLM calls | Output affects decisions about individuals/groups | Regulatory requirement; protects equitable access | Surfaces uneven outcomes before release | 4 |
-| Bias test | Responsible AI | Detect skewed or stereotyped associations | mocks disallowed for LLM calls | User-facing content generation | Lowers legal/reputational exposure | Catches bias introduced by data/prompts/fine-tuning | 3 |
-| Robustness test | Responsible AI | Verify stable behavior under noisy/out-of-distribution input | mocks disallowed for LLM calls | Inputs come from untrusted/variable sources | Protects reliability/SLAs | Confirms graceful degradation, guides input validation | 3 |
-| Explainability test | Responsible AI | Verify output is justifiable with a faithful rationale | mocks disallowed for LLM calls | Output must be justified to users/auditors/regulators | Required for auditability; builds user trust | Gives rationale trace for debugging wrong answers | 2 |
-| Groundedness (RAG) eval | Quality eval | Verify the answer is supported by retrieved context | mocks disallowed for LLM calls | System uses retrieval-augmented generation | Avoids confidently-wrong answers reaching customers | Pinpoints retrieval/prompt bugs | 4 |
-| Human evaluation | Quality eval | Manually verify aspects automated scoring can't (ethics, side effects, external state) | mocks disallowed for LLM calls | Before major releases; periodic spot-check | Defensible, human-reviewed sign-off | Catches what automated metrics miss | 3 |
-| Functional eval (golden-dataset accuracy / LLM-as-judge) | Quality eval | Measure output correctness against the golden dataset | mocks disallowed for LLM calls | Required before every Workflow release ([agentme-edr-007](../principles/007-project-quality-standards.md) rule `09`); advised elsewhere | Auditable evidence of business correctness before release | Detects regressions from model/provider/prompt changes | 5 |
-| Smoke test | Quality eval | Fast pass/fail check on a small, critical subset before running fuller suites | mocks disallowed for LLM calls | Every commit/PR, before functional/responsible-AI evals run | Cheap early warning before slower evals run | Fast, cheap feedback loop | 4 |
-| Prompt regression test | Prompt/LLM | Detect behavior change when a prompt or model version changes | mocks disallowed for LLM calls | Whenever a prompt template or model version changes | Prevents shipping a worse experience via a "small" tweak | Fast check on every prompt edit | 3 |
-| Integration test | Code-level | Verify real interaction with external systems | mocks disallowed | Component depends on external systems | Reduces production outages from integration mismatches | Catches wiring bugs unit tests can't see | 2 |
-| Unit test (offline, mocked) | Code-level | Verify deterministic logic in isolation, offline | mocks allowed | Required for Workflow tier every commit ([agentme-edr-007](../principles/007-project-quality-standards.md) rule `09`) | Lowest-cost point to catch defects | Fastest, fully offline feedback on every commit | 5 |
+| Test Type Name | `test_types` value | Group | Test Objective | Mock Constraint | When to Apply | Relevance – Business | Relevance – Development Team | Priority (1-5) |
+|---|---|---|---|---|---|---|---|---|
+| Safety/content eval | `safety` | Safety & adversarial | Detect harmful, biased, or policy-violating output | mocks disallowed for LLM calls | Any user-facing release | Avoids reputational harm; acceptable-use compliance | Automated content gate before merge/release | 5 |
+| Adversarial/red-team test | `adversarial` | Safety & adversarial | Probe for prompt injection, jailbreaks, unsafe tool use | mocks disallowed for LLM calls | System exposes tool-invocation or agent loops | Reduces security-incident/breach liability | Finds exploitable tool-loop paths before attackers do | 5 |
+| Fairness test | `fairness` | Responsible AI | Verify equitable outcomes across user groups | mocks disallowed for LLM calls | Output affects decisions about individuals/groups | Regulatory requirement; protects equitable access | Surfaces uneven outcomes before release | 4 |
+| Bias test | `bias` | Responsible AI | Detect skewed or stereotyped associations | mocks disallowed for LLM calls | User-facing content generation | Lowers legal/reputational exposure | Catches bias introduced by data/prompts/fine-tuning | 3 |
+| Robustness test | `robustness` | Responsible AI | Verify stable behavior under noisy/out-of-distribution input | mocks disallowed for LLM calls | Inputs come from untrusted/variable sources | Protects reliability/SLAs | Confirms graceful degradation, guides input validation | 3 |
+| Explainability test | `explainability` | Responsible AI | Verify output is justifiable with a faithful rationale | mocks disallowed for LLM calls | Output must be justified to users/auditors/regulators | Required for auditability; builds user trust | Gives rationale trace for debugging wrong answers | 2 |
+| Groundedness (RAG) eval | `groundedness` | Quality eval | Verify the answer is supported by retrieved context | mocks disallowed for LLM calls | System uses retrieval-augmented generation | Avoids confidently-wrong answers reaching customers | Pinpoints retrieval/prompt bugs | 4 |
+| Repeatability test | `repeatability` | Quality eval | Verify output stability/variance across N repeated invocations of the same input under fixed configuration | mocks disallowed for LLM calls | Non-deterministic components (temperature > 0, agentic tool-selection loops, sampling-based decoding) used in decision-critical or user-facing flows | Protects against silently flaky behavior reaching production; supports consistency SLAs | Detects prompt/agent designs too sensitive to sampling noise; informs temperature/seed tuning | 3 |
+| Human evaluation | `human` | Quality eval | Manually verify aspects automated scoring can't (ethics, side effects, external state) | mocks disallowed for LLM calls | Before major releases; periodic spot-check | Defensible, human-reviewed sign-off | Catches what automated metrics miss | 3 |
+| Functional eval (golden-dataset accuracy / LLM-as-judge) | `functional` | Quality eval | Measure output correctness against the golden dataset | mocks disallowed for LLM calls | Required before every Workflow release ([agentme-edr-007](../principles/007-project-quality-standards.md) rule `09`); advised elsewhere | Auditable evidence of business correctness before release | Detects regressions from model/provider/prompt changes | 5 |
+| Smoke test | `smoke` | Quality eval | Fast pass/fail check on a small, critical subset before running fuller suites | mocks disallowed for LLM calls | Every commit/PR, before functional/responsible-AI evals run | Cheap early warning before slower evals run | Fast, cheap feedback loop | 4 |
+| Prompt regression test | `prompt` | Prompt/LLM | Detect behavior change when a prompt or model version changes | mocks disallowed for LLM calls | Whenever a prompt template or model version changes | Prevents shipping a worse experience via a "small" tweak | Fast check on every prompt edit | 3 |
+| Integration test | n/a — code-level only (see rule `04`) | Code-level | Verify real interaction with external systems | mocks disallowed | Component depends on external systems | Reduces production outages from integration mismatches | Catches wiring bugs unit tests can't see | 2 |
+| Unit test (offline, mocked) | n/a — code-level only (see rule `04`) | Code-level | Verify deterministic logic in isolation, offline | mocks allowed | Required for Workflow tier every commit ([agentme-edr-007](../principles/007-project-quality-standards.md) rule `09`) | Lowest-cost point to catch defects | Fastest, fully offline feedback on every commit | 5 |
 
 #### 06-priority-and-relevance-are-descriptive-only
 
-Priority, Relevance, and When to Apply in rule `05` are guidance for prioritization conversations — they MUST NOT be treated as mandating which test types a project must implement, nor their thresholds. [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09` remains the only tier-level testing requirement in force (Workflow unit tests + functional evals). Once a project chooses to implement and threshold a test type, [agentme-edr-028](028-ai-eval-standards.md) rule `02`'s failing-threshold behavior applies uniformly, regardless of this table's priority rating — a project may enforce fairness at 70% and functional at 90%, or skip fairness entirely; that choice is a project/business decision, not one this Policy makes.
+Priority, Relevance, and When to Apply in rule `05` are guidance for prioritization conversations — they MUST NOT be treated as mandating which test types a project must implement, nor their thresholds. [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09` remains the only tier-level testing requirement in force (Workflow unit tests + functional evals). Once a project chooses to implement and threshold a test type, [agentme-edr-031](031-ai-eval-script.md) rule `01`'s failing-threshold behavior applies uniformly, regardless of this table's priority rating — a project may enforce fairness at 70% and functional at 90%, or skip fairness entirely; that choice is a project/business decision, not one this Policy makes.
 
 #### 07-smoke-is-distinct-from-test-smoke
 
 The `smoke` test type (surfaced as the `eval-smoke` Makefile target, a fast subset of the golden-dataset functional eval) is a different concept from [agentme-edr-008](../devops/008-common-targets.md)'s existing `test-smoke` target (a fast subset of code-level tests). Both MAY exist in the same project; teams MUST NOT conflate them.
 
+#### 08-eval-mocking-constraint
+
+For every golden-dataset eval test type: **the LLM call MUST be real; all other external dependencies MUST be mocked via `mock_fixtures`.** This applies equally to `human` entries — the component is invoked to capture `actual_output` and external dependencies must be deterministic. `mock_fixtures` keys MUST NOT reference LLM adapters. Code-level unit tests are the correct place for fully offline, LLM-mocked testing (see [agentme-edr-018](018-ai-llm-development-standards.md) rule `04`).
+
+#### 09-repeatability-vs-reproducibility
+
+| Property | Definition | What varies | Measured by |
+|---|---|---|---|
+| **Repeatability** | Output stability across N invocations at non-zero temperature | Model sampling variance | `repeatability` test type per [agentme-edr-033](033-ai-eval-repeatability.md) |
+| **Reproducibility** | Deterministic output at temperature = 0 with fixed seed | Nothing — any variance is a config bug | Not a golden-dataset type; verified via config, documented in [agentme-edr-027](../devops/027-environment-variable-configuration.md) |
+
+A component may satisfy reproducibility (temperature = 0) yet still need repeatability tests for its production configuration (temperature > 0). The `repeatability` test type MUST NOT be applied to components with intentionally diverse output (brainstorming, creative generation) — variance is correct behavior there.
+
 ## References
 
 - [agentme-edr-024](024-ml-dataset-structure.md) — Golden dataset file layout, per-entry JSON format, `$schema` pointer, and schema-lint validation
-- [agentme-edr-028](028-ai-eval-standards.md) — Eval folder structure, `--type` filtering, per-type Makefile targets, and per-type reports that consume this taxonomy
+- [agentme-edr-028](028-ai-eval-core-standards.md) — AI eval core standards: eval folder structure and Makefile targets (rule `01`); LLM-as-judge binary scoring contract (rule `02`)
+- [agentme-edr-031](031-ai-eval-script.md) — AI eval script: `--type` filtering, entry-first loop, `mock_fixtures`, threshold enforcement, and MLflow conventions
+- [agentme-edr-032](032-ai-eval-report-format.md) — AI eval report format: per-type `report-<type>.md` that consumes this taxonomy's test types
+- [agentme-edr-033](033-ai-eval-repeatability.md) — AI eval repeatability: `REPEAT_COUNT` loop exception, scoring constants (`EVAL_MIN_ACCURACY_REPEATABILITY`, `REPEAT_SEMANTIC_SIMILARITY_SCORE`), scoring methods, `repeatability_accuracy` MLflow metric, report shape, and run cadence
 - [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) — Rule `10`: `_mock` file naming and placement convention for mock adapters referenced by `mock_fixtures`
 - [agentme-edr-007](../principles/007-project-quality-standards.md) — Rule `09` tier-level testing requirements (the only mandated AI testing baseline)
 - [agentme-edr-008](../devops/008-common-targets.md) — Rule `03` `eval-<qualifier>` Makefile convention; rule `03`'s `test-smoke` (distinguished in rule `07`)
+- [agentme-edr-027](../devops/027-environment-variable-configuration.md) — Environment-configuration conventions referenced in rule `09`'s reproducibility disambiguation
 - [agentme-edr-018](018-ai-llm-development-standards.md) — LLM tier definition and mocking utilities referenced by the `mocks allowed` value
 - [agentme-edr-004](../principles/004-unit-test-requirements.md) — Unit test requirements underlying the Code-level rows

package/.xdrs/agentme/edrs/application/031-ai-eval-script.md

@@ -0,0 +1,136 @@
+---
+name: agentme-edr-policy-031-ai-eval-script
+description: Defines eval.py script requirements for AI projects — entry-first eval loop, --type test-type filtering, mock_fixtures wiring, human entries, threshold enforcement, and MLflow experiment naming and port assignment. Use when implementing eval scripts. For eval folder structure see agentme-edr-028 rule 01. For the test type taxonomy and mock_fixtures envelope see agentme-edr-030. For mock file naming see agentme-edr-026 rule 10. For report format see agentme-edr-032. For repeatability loop exception see agentme-edr-033.
+apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
+valid-from: 2026-07-07
+---
+
+# agentme-edr-policy-031: AI eval script
+
+## Context and Problem Statement
+
+Eval scripts execute entries from a golden dataset against a real AI component and measure output quality. Without a shared script contract, eval implementations diverge: some invoke components multiple times per entry (wasting LLM cost), some skip mock isolation, and some omit threshold enforcement — making results inconsistent and hard to trust across projects.
+
+How should eval scripts load datasets, iterate entries, handle mocking, and produce metrics?
+
+## Decision Outcome
+
+**Use an entry-first eval loop with `--type` filtering, fresh mock isolation per entry, real LLM providers, and MLflow-backed metrics with explicit per-type thresholds.**
+
+For when evals are required per AI tier, see [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
+
+### Details
+
+#### 01-eval-script-requirements
+
+Each `eval.py` script MUST:
+
+- Load the golden dataset from `golden_dataset/` in the same eval folder, following [agentme-edr-024](024-ml-dataset-structure.md) and the entry envelope in [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02` (one JSON file per entry, `test_types` array, `input`, `expected_output`, optional `mock_fixtures`).
+- Accept a required `--type=<test_type>|all` CLI argument and filter entries whose `test_types` array contains the requested value; `--type=all` includes every entry.
+- Iterate **entry-first**: for each entry in the filtered set, invoke the real component exactly once; then score that single `actual_output` for every `test_types` value the entry carries that falls within the current `--type` scope — MUST NOT invoke the component more than once per entry per run.
+- When an entry contains `mock_fixtures` ([agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02`), configure each named mock adapter with its fixture data BEFORE invoking the component for that entry. Each entry MUST use fresh mock instances so fixture state does not bleed across entries. `mock_fixtures` applies to all test types including `human`. `mock_fixtures` MUST NOT configure LLM adapters — the LLM call MUST be real (see [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `03`). How mock adapters are discovered and instantiated is left to the project; see [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
+- Run every component invocation against **real LLM providers** (not mocked responses), to capture model drift.
+- For `human` entries: invoke the component to capture `actual_output`, export each entry's `input`, `expected_output.human_test` instructions, and `actual_output` into a manual-review checklist (`report-human.md`). MUST NOT invoke an automated scorer and MUST NOT enforce a pass/fail threshold for it. Other `test_types` on the same entry (e.g. `functional`) are still scored automatically.
+- After all entries are processed, compute aggregate metrics per test type, log them to a local MLflow experiment (see rule `02`), write one `report-<type>.md` per evaluated test type ([agentme-edr-032](032-ai-eval-report-format.md) rule `01`), and exit with a non-zero status when any metric falls below its defined threshold per [agentme-edr-007](../principles/007-project-quality-standards.md) rule `07-statistical-models-must-have-eval-targets`. The `human` type has no threshold and does not trigger a non-zero exit.
+- Compare outputs to expected values using project-defined quality thresholds per test type. Thresholds and all other scoring parameters MUST be declared as constants in `eval.py` — they are design decisions about what constitutes acceptable quality for the component under test, not runtime configuration, and MUST NOT be passed via Makefile variables or CLI flags. Use one of two naming conventions, chosen consistently within an `eval.py`: (a) **per-type constants** — `EVAL_MIN_<METRIC>_<TYPE>` for each test type (e.g. `EVAL_MIN_ACCURACY_FUNCTIONAL = 0.85`, `EVAL_MIN_ACCURACY_REPEATABILITY = 0.8`); or (b) **dict constant** — `EVAL_MIN_<METRIC> = {<type>: <value>}` (e.g. `EVAL_MIN_ACCURACY = {"functional": 0.85, "smoke": 0.85}`). Per-type constants are preferred when each test type has a dedicated `eval.py`; the dict form is preferred when a single `eval.py` handles multiple types. In either convention, `EVAL_MIN_ACCURACY` (as a scalar) MAY be declared as a project-wide default and MUST be used as fallback when no per-type override is defined for the current test type. This Policy does not mandate which test types a project must threshold or what value to use (see [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `06`).
+
+**Example:**
+
+```python
+import argparse
+from collections import defaultdict
+import mlflow
+from my_package.app.workflows.document_review_workflow.graph import graph
+
+# Per-type constants; fall back to EVAL_MIN_ACCURACY default when no override is defined
+EVAL_MIN_ACCURACY = 0.85  # project-wide default
+EVAL_MIN_ACCURACY_FUNCTIONAL = 0.85
+EVAL_MIN_ACCURACY_SMOKE = 0.90
+EVAL_MIN_ACCURACY_PER_TYPE = {"functional": EVAL_MIN_ACCURACY_FUNCTIONAL, "smoke": EVAL_MIN_ACCURACY_SMOKE}
+
+def get_min_accuracy(test_type: str) -> float:
+    return EVAL_MIN_ACCURACY_PER_TYPE.get(test_type, EVAL_MIN_ACCURACY)
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--type", required=True)
+args = parser.parse_args()
+
+entries = load_golden_dataset("golden_dataset/", test_type=args.type)  # "all" loads every entry
+resolved_types = resolve_types(args.type, entries)
+
+mlflow.set_experiment("document-review/eval-basic")
+
+with mlflow.start_run():
+    mlflow.set_tag("test_types", ",".join(sorted(resolved_types)))
+
+    results = defaultdict(list)
+    cumulative_metrics = defaultdict(lambda: {"accuracy": [], "f1": []})  # Track cumulative metrics
+
+    # Entry-first loop: invoke each entry exactly once
+    for idx, entry in enumerate(entries, start=1):
+        # Configure mock adapters from mock_fixtures before invocation
+        # (implementation left to the project — see agentme-edr-026 rule 10)
+        if entry.get("mock_fixtures"):
+            configure_mocks(entry["mock_fixtures"])  # project-defined helper
+
+        actual_output = invoke_component(entry, graph)
+
+        for test_type in [t for t in entry["test_types"] if t in resolved_types]:
+            if test_type == "human":
+                export_human_review(entry, actual_output)
+                continue
+            
+            score_val = score(test_type, actual_output, entry["expected_output"])
+            results[test_type].append(score_val)
+            
+            # Track cumulative metrics for convergence analysis
+            cumulative_accuracy = sum(results[test_type]) / len(results[test_type])
+            cumulative_f1 = compute_f1(results[test_type])  # project-defined
+            cumulative_metrics[test_type]["accuracy"].append(cumulative_accuracy)
+            cumulative_metrics[test_type]["f1"].append(cumulative_f1)
+
+    # Aggregate, report, and enforce thresholds per test type
+    for test_type in resolved_types:
+        if test_type == "human":
+            continue
+
+        accuracy = sum(results[test_type]) / len(results[test_type])
+        mlflow.log_metric(f"{test_type}_accuracy", accuracy)
+        
+        # Generate convergence analysis
+        stability_window = min(10, len(results[test_type]))
+        acc_change = abs(cumulative_metrics[test_type]["accuracy"][-1] - 
+                        cumulative_metrics[test_type]["accuracy"][-stability_window])
+        f1_change = abs(cumulative_metrics[test_type]["f1"][-1] - 
+                       cumulative_metrics[test_type]["f1"][-stability_window])
+        
+        write_eval_report(
+            test_type, 
+            results[test_type], 
+            cumulative_metrics=cumulative_metrics[test_type],
+            stability_window=stability_window,
+            thresholds={"accuracy": get_min_accuracy(test_type)}
+        )
+
+        if accuracy < get_min_accuracy(test_type):
+            raise SystemExit(f"Eval failed: {test_type} accuracy {accuracy:.2f} < {get_min_accuracy(test_type)}")
+```
+
+#### 02-eval-mlflow-unique-port
+
+Each `evals/<component>/eval-<name>/Makefile` MUST start its MLflow tracking server on a **unique port** to prevent conflicts when multiple eval Makefiles are run concurrently or in parallel (e.g., in CI or across multiple terminal sessions).
+
+Ports MUST be statically assigned per eval scenario (not per test type) and MUST NOT reuse the default `5000` port (reserved for `dev-mlflow` per [agentme-edr-008](../devops/008-common-targets.md) rule `09-ai-project-dev-targets`). Assign ports starting at `5100` and incrementing by 1 for each additional eval scenario across the entire project.
+
+The MLflow **experiment** is scoped to the eval scenario: `<component>/<eval-name>` (e.g. `document-review/eval-basic`). Each `mlflow.start_run()` call MUST set a `test_types` tag listing the test types evaluated in that invocation (comma-separated, e.g. `"functional,smoke"` for `--type=all`, `"smoke"` for `--type=smoke`). A remote MLflow server MUST NOT be required — all tracking is local.
+
+## References
+
+- [agentme-edr-028](028-ai-eval-core-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract (rule `02`)
+- [agentme-edr-032](032-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
+- [agentme-edr-033](033-ai-eval-repeatability.md) — AI eval repeatability: loop exception to rule `01`'s entry-first constraint, scoring methods, and cadence
+- [agentme-edr-030](030-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum, golden dataset entry envelope (including `mock_fixtures`), and mocking constraints per type
+- [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) — Rule `10`: `_mock` file naming and placement convention for mock adapters used in `mock_fixtures`
+- [agentme-edr-024](024-ml-dataset-structure.md) — ML dataset structure, per-entry JSON format, and schema-lint validation for golden datasets
+- [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09`) and statistical model eval targets (rule `07`)
+- [agentme-edr-008](../devops/008-common-targets.md) — `eval-<qualifier>` Makefile convention (rule `03`) and reserved MLflow port `5000` (rule `09`)

package/.xdrs/agentme/edrs/application/032-ai-eval-report-format.md

@@ -0,0 +1,171 @@
+---
+name: agentme-edr-policy-032-ai-eval-report-format
+description: Defines the eval report format for AI projects — report-<type>.md template, Wilson score confidence interval, convergence analysis, and the human-type checklist artifact. Use when generating eval reports from eval scripts. For the eval script that produces these reports see agentme-edr-031. For repeatability report shape see agentme-edr-033 rule 02.
+apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
+valid-from: 2026-07-07
+---
+
+# agentme-edr-policy-032: AI eval report format
+
+## Context and Problem Statement
+
+Eval scripts produce output reports, but without a shared format these vary across projects: some omit confidence intervals, some skip convergence analysis, and some allow LLMs to write sections — making reports unreliable and non-comparable.
+
+What format should eval reports follow, and what constraints apply to how they are generated?
+
+## Decision Outcome
+
+**Use a standardized `report-<type>.md` template with a Wilson score confidence interval, a Mermaid convergence chart, and a strict no-LLM generation constraint for all metric values.**
+
+### Details
+
+#### 01-eval-report-file
+
+Each eval script MUST produce one `report-<type>.md` per evaluated test type in the same `evals/<component>/eval-<name>/` folder and overwrite each on every run — only the types included in the current `--type` invocation are (re)written; report files for other types are left untouched. The `human` type does not produce a metrics report (see below).
+
+**Generation constraint:** The report MUST be produced programmatically, reading raw metric values directly from MLflow. No LLM or generative model may write, summarize, or paraphrase any section of the report, to prevent hallucinated metric values. This constraint applies to all report sections including Overall Results, Convergence Analysis, and Per-item Results — all metric values and convergence chart data points MUST be computed from actual evaluation results.
+
+The report MUST follow this template:
+
+```markdown
+# Eval Report: <name> — <type>
+
+**Date:** <ISO date>
+**Dataset:** golden_dataset/
+**Script:** eval.py --type=<type>
+**Thresholds:** accuracy ≥ <value>, F1 ≥ <value>
+
+## Overall Results
+
+| Metric    | Value  | 95% CI         | Threshold | Status  |
+|-----------|--------|----------------|-----------|---------|
+| Accuracy  | <val>  | [<low>, <high>]| ≥ <thr>   | ✓/✗ PASS/FAIL |
+| F1 Score  | <val>  | —              | ≥ <thr>   | ✓/✗ PASS/FAIL |
+| Precision | <val>  | —              | —         | —       |
+| Recall    | <val>  | —              | —         | —       |
+| Samples   | <n>    | —              | —         | —       |
+
+**Overall: PASS / FAIL**
+
+## Convergence Analysis
+
+```mermaid
+xychart-beta
+  title "Metric Evolution vs Sample Count"
+  x-axis "Samples" [<sample_points>]
+  y-axis "Score" 0.0 --> 1.0
+  line "Accuracy" [<accuracy_values>]
+  line "F1 Score" [<f1_values>]
+```
+
+**Stability Analysis:**
+- Accuracy change over last <window> samples: <change> percentage points
+- F1 change over last <window> samples: <change> percentage points
+
+**Recommendation:** <"Dataset appears sufficient for confident evaluation" | "Add more samples — metrics have not yet stabilized">
+
+## Per-item Results
+
+| ID  | Input Summary | Expected | Actual | Correct |
+|-----|---------------|----------|--------|---------|  
+| 001 | <summary>     | <label>  | <label>| ✓       |
+| 002 | <summary>     | <label>  | <label>| ✗       |
+
+## Notes
+
+- <observations, failure patterns, MLflow run link>
+```
+
+**Confidence interval:** The 95% CI for accuracy MUST be computed using the **Wilson score interval** (preferred over the normal approximation for small $n$). A wide interval signals that the dataset is too small to support confident conclusions and the sample count should be increased.
+
+The Wilson score bounds at 95% confidence ($z = 1.96$) are:
+
+$$\frac{\hat{p} + \frac{z^2}{2n} \pm z\sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}$$
+
+Where $\hat{p}$ is observed accuracy and $n$ is sample count. Accuracy and F1 are required; precision and recall are recommended.
+
+**Convergence analysis:** The Convergence Analysis section shows whether adding more samples would likely change measured metrics. The section MUST include:
+
+1. **Mermaid xychart-beta** showing cumulative Accuracy and F1 evolution:
+   - X-axis: absolute cumulative sample count; Y-axis: metric value (0.0 to 1.0)
+   - Two lines: Accuracy and F1
+   - For datasets > 50 samples: sample at `floor(dataset_size / 10)` intervals (minimum 5), always include first and last points
+   - For datasets ≤ 50 samples: show all points
+
+2. **Stability analysis**: compute absolute change (percentage points) for both Accuracy and F1 over last `min(10, dataset_size)` samples. Example: Accuracy from 0.85 to 0.87 = 0.02 = 2 percentage points.
+
+3. **Recommendation**:
+   - Default threshold: both Accuracy AND F1 change ≤ 2 percentage points
+   - If both meet threshold: "Dataset appears sufficient for confident evaluation"
+   - If either exceeds: "Add more samples — metrics have not yet stabilized"
+   - Projects MAY customize threshold (document in Makefile/README)
+
+Exclude from `report-human.md` (no automated metrics).
+
+**Filled-in example** (`evals/workflow-document-review/eval-basic/report-functional.md` for a document review workflow):
+
+```markdown
+# Eval Report: eval-basic — functional
+
+**Date:** 2026-06-12
+**Dataset:** golden_dataset/
+**Script:** eval.py --type=functional
+**Thresholds:** accuracy ≥ 0.85, F1 ≥ 0.80
+
+## Overall Results
+
+| Metric    | Value | 95% CI       | Threshold | Status      |
+|-----------|-------|--------------|-----------|-------------|
+| Accuracy  | 0.88  | [0.69, 0.97] | ≥ 0.85    | ✓ PASS      |
+| F1 Score  | 0.86  | —            | ≥ 0.80    | ✓ PASS      |
+| Precision | 0.89  | —            | —         | —           |
+| Recall    | 0.84  | —            | —         | —           |
+| Samples   | 25    | —            | —         | —           |
+
+**Overall: PASS**
+
+> Note: CI [0.69, 0.97] is wide — 25 samples may be insufficient for high confidence. Consider expanding the dataset.
+
+## Convergence Analysis
+
+```mermaid
+xychart-beta
+  title "Metric Evolution vs Sample Count"
+  x-axis "Samples" [1, 5, 10, 15, 20, 25]
+  y-axis "Score" 0.0 --> 1.0
+  line "Accuracy" [0.60, 0.80, 0.85, 0.87, 0.88, 0.88]
+  line "F1 Score" [0.55, 0.78, 0.83, 0.85, 0.86, 0.86]
+```
+
+**Stability Analysis:**
+- Accuracy change over last 10 samples: 0.01 percentage points
+- F1 change over last 10 samples: 0.01 percentage points
+
+**Recommendation:** Dataset appears sufficient for confident evaluation
+
+> Metrics stabilized after ~15 samples. Changes over last 10 samples are well below the 2 percentage point threshold.
+
+## Per-item Results
+
+| ID  | Input Summary                       | Expected | Actual   | Correct |
+|-----|--------------------------------------|----------|----------|---------|
+| 001 | Contract renewal, 3 pages, standard | approve  | approve  | ✓       |
+| 002 | NDA with unusual liability clause   | escalate | escalate | ✓       |
+| 003 | Vendor invoice, missing PO number   | reject   | reject   | ✓       |
+| 004 | Employment agreement, standard terms| approve  | approve  | ✓       |
+| 005 | Amendment with redlined IP clause   | escalate | approve  | ✗       |
+
+## Notes
+
+- Sample 005 misclassified: redlined IP clause not flagged as escalation trigger. Possible model drift.
+- MLflow run: experiment `workflow-document-review/eval-basic`, tag `test_types=functional` — view with `mlflow ui`
+```
+```
+
+**`human` type artifact:** instead of `report-human.md` with metrics, `--type=human` produces a checklist artifact (still named `report-human.md`) listing, per entry, its `input`, `expected_output.human_test` instructions, and the captured `actual_output` — with no Overall Results table, threshold, or PASS/FAIL section, since this type MUST NOT be auto-scored.
+
+## References
+
+- [agentme-edr-031](031-ai-eval-script.md) — AI eval script: the script that produces these reports (rule `01`)
+- [agentme-edr-033](033-ai-eval-repeatability.md) — AI eval repeatability: rule `02` defines the adapted report shape for `report-repeatability.md`
+- [agentme-edr-028](028-ai-eval-core-standards.md) — AI eval core standards: folder structure (rule `01`) and LLM-as-judge binary scoring (rule `02`)

package/.xdrs/agentme/edrs/application/033-ai-eval-repeatability.md

@@ -0,0 +1,75 @@
+---
+name: agentme-edr-policy-033-ai-eval-repeatability
+description: Defines the repeatability test type for AI evals — REPEAT_COUNT loop exception to the entry-first constraint, semantic-similarity and LLM-as-judge scoring methods, repeatability_accuracy metric, and the repeatability report format and run cadence. Use when implementing repeatability evals. For the entry-first constraint see agentme-edr-031 rule 01. For LLM-as-judge binary output see agentme-edr-028 rule 02. For the base report template see agentme-edr-032 rule 01. For the repeatability test type definition see agentme-edr-030.
+apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement repeatability eval testing
+valid-from: 2026-07-07
+---
+
+# agentme-edr-policy-033: AI eval repeatability
+
+## Context and Problem Statement
+
+Some AI components must be tested for output consistency across multiple invocations with the same input — a property the standard entry-first eval loop cannot capture because it invokes each entry only once. Without a shared repeatability protocol, teams either skip this test or implement ad-hoc variations that produce incomparable results.
+
+How should repeatability be measured in evals, and how should its results be reported?
+
+## Decision Outcome
+
+**Exempt `repeatability` entries from the entry-first constraint, invoking each `REPEAT_COUNT` times and scoring via semantic-similarity or LLM-as-judge; schedule at release cadence rather than per-commit.**
+
+For the `repeatability` test type definition and its disambiguation from `reproducibility`, see [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `09`.
+
+### Details
+
+#### 01-repeatability-eval-loop-exception
+
+Entries whose `test_types` includes `repeatability` are exempt from [agentme-edr-031](031-ai-eval-script.md) rule `01`'s "invoke exactly once per entry" constraint. The following constants MUST be declared in `eval.py` and MUST NOT be exposed as Makefile variables, CLI flags, or stored as per-entry dataset fields:
+
+- `REPEAT_COUNT` — number of times each repeatability entry is invoked. SHOULD default to 3-5 for routine CI runs and 10-20 for focused passes on decision-critical or previously-flagged components. Projects SHOULD calibrate the value once per component by plotting cumulative pass rate against repeat count for a few representative entries and picking the point where it plateaus, rather than guessing.
+- `EVAL_MIN_ACCURACY_REPEATABILITY` — minimum fraction of repeatability entries that must PASS for the eval to exit 0.
+- `REPEAT_SEMANTIC_SIMILARITY_SCORE` — minimum average pairwise cosine similarity for a single entry to PASS; declared only when using semantic-similarity scoring.
+
+`eval.py` MUST invoke the component `REPEAT_COUNT` times for every repeatability entry and score the resulting outputs by comparing them to each other. `expected_output` is unused for repeatability entries and SHOULD be omitted or set to `null` in the dataset.
+
+**Choosing the scoring method:** Two approaches are supported, declared as a constant in `eval.py`:
+
+- **Semantic-similarity:** Embed all `REPEAT_COUNT` outputs for an entry into vectors and compute the average pairwise cosine similarity. The entry passes (score = 1) if the average similarity ≥ `REPEAT_SEMANTIC_SIMILARITY_SCORE`; otherwise it fails (score = 0). Use for classification or short structured outputs.
+- **LLM-as-judge:** Provide all `REPEAT_COUNT` outputs for an entry to an LLM judge (kept at low/zero temperature) that returns 0 (fail) or 1 (pass) directly, based on whether the outputs are sufficiently consistent. No `REPEAT_SEMANTIC_SIMILARITY_SCORE` constant is needed. Use for free-text or complex structured outputs where vector distance is an unreliable proxy for agreement. The judge MUST follow [agentme-edr-028](028-ai-eval-core-standards.md) rule `02`'s binary output contract.
+
+**`repeatability_accuracy`:** the fraction of repeatability entries that received PASS (score = 1), logged to MLflow as `repeatability_accuracy`. The eval exits non-zero if `repeatability_accuracy` < `EVAL_MIN_ACCURACY_REPEATABILITY`. Both `repeatability_accuracy` and `repeat_count` MUST be logged to MLflow and included in `report-repeatability.md` (rule `02`).
+
+```python
+REPEAT_COUNT = 5
+EVAL_MIN_ACCURACY_REPEATABILITY = 0.8
+REPEAT_SEMANTIC_SIMILARITY_SCORE = 0.9  # only when using semantic-similarity scoring
+
+for entry in repeatability_entries:
+    outputs = [invoke_component(entry, graph) for _ in range(REPEAT_COUNT)]
+    entry_pass = score_agreement(outputs)  # returns 0 or 1; expected_output is not used
+    results["repeatability"].append(entry_pass)  # 1 = PASS, 0 = FAIL
+
+repeatability_accuracy = sum(results["repeatability"]) / len(results["repeatability"])
+mlflow.log_metric("repeatability_accuracy", repeatability_accuracy)
+mlflow.log_metric("repeat_count", REPEAT_COUNT)
+
+if repeatability_accuracy < EVAL_MIN_ACCURACY_REPEATABILITY:
+    raise SystemExit(f"Eval failed: repeatability_accuracy {repeatability_accuracy:.2f} < {EVAL_MIN_ACCURACY_REPEATABILITY}")
+```
+
+`mock_fixtures` configuration per [agentme-edr-031](031-ai-eval-script.md) rule `01` applies to each of the `REPEAT_COUNT` invocations. Any prompt or response caching (provider-side or gateway-side) MUST be bypassed for these invocations — a cache hit would return an identical cached response and falsely report perfect stability instead of measuring the model's actual variance.
+
+**Scoping:** this test type MUST NOT be applied to components whose intended behavior is diverse or creative output (e.g. brainstorming, creative writing) — low agreement there is correct behavior, not a defect.
+
+#### 02-repeatability-report-and-cadence
+
+`--type=repeatability` MUST produce `report-repeatability.md` with a shape adapted from [agentme-edr-032](032-ai-eval-report-format.md) rule `01`'s template: the header MUST include a **Repeat count:** line stating the `REPEAT_COUNT` value used for the run, alongside the usual Date/Dataset/Script/Thresholds lines. The body MUST have an aggregate row reporting `repeatability_accuracy` (the fraction of entries that PASS — see rule `01`) with a Wilson score interval computed over the number of `repeatability` entries, plus a per-item table listing each entry's individual pass/fail result and, when using semantic-similarity, its computed average pairwise cosine similarity — instead of the `Expected | Actual | Correct` columns used by other types.
+
+Because `repeatability` entries multiply real LLM-provider calls by `REPEAT_COUNT`, projects SHOULD schedule `make eval-repeatability` at release cadence rather than on every commit, aligned with the Workflow eval cadence in [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09`, rather than treating it as a mandatory per-commit gate.
+
+## References
+
+- [agentme-edr-031](031-ai-eval-script.md) — AI eval script: rule `01` defines the entry-first constraint this policy exempts for repeatability entries
+- [agentme-edr-028](028-ai-eval-core-standards.md) — AI eval core standards: rule `02` defines the LLM-as-judge binary output contract used by the LLM-as-judge scoring method in rule `01`
+- [agentme-edr-032](032-ai-eval-report-format.md) — AI eval report format: rule `01` defines the base report template that `report-repeatability.md` adapts
+- [agentme-edr-030](030-ai-test-types-taxonomy.md) — AI test types taxonomy: `repeatability` test type definition and disambiguation from `reproducibility` (rule `09`)
+- [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards: Workflow eval cadence (rule `09`) that repeatability runs align with

package/.xdrs/agentme/edrs/index.md

@@ -27,21 +27,36 @@ Synthetic views combining agentme XDRs and skills around a specific topic.
 
 Language and framework-specific tooling and project structure.
 
+### Language and framework tooling
+
 - [agentme-edr-003](application/003-javascript-project-tooling.md) - **JavaScript project tooling and structure** - Scaffold JavaScript libraries with the standard toolchain *(includes skill: [001-create-javascript-project](application/skills/001-create-javascript-project/SKILL.md))*
 - [agentme-edr-010](application/010-golang-project-tooling.md) - **Go project tooling and structure** - Scaffold Go CLIs and libraries with the standard layout *(includes skill: [003-create-golang-project](application/skills/003-create-golang-project/SKILL.md))*
 - [agentme-edr-014](application/014-python-project-tooling.md) - **Python project tooling and structure** - Scaffold Python packages and CLIs with the standard layout *(includes skill: [005-create-python-project](application/skills/005-create-python-project/SKILL.md))*
 - [agentme-edr-015](application/015-cli-tool-standards.md) - **CLI tool standards** - Define command UX and behavior for CLI tools
+- [agentme-edr-026](application/026-pragmatic-hexagonal-architecture.md) - **Pragmatic hexagonal architecture** - Organize application layers as External/Adapters/Application with practical coupling rules
+- [004-select-relevant-xdrs](application/skills/004-select-relevant-xdrs/SKILL.md) - **Select relevant XDRs**
+
+### AI development
+
+Standards for building LLM, Agent, and Workflow components.
+
 - [agentme-edr-018](application/018-ai-llm-development-standards.md) - **AI LLM development standards** - Standard framework (LangChain) and patterns for simple LLM calls with explicit configuration (no environment variables)
 - [agentme-edr-019](application/019-ai-agents-development-standards.md) - **AI agents development standards** - Structural patterns for agents: framework selection, sandbox setup, naming conventions, composition, and system prompt structure
 - [agentme-edr-020](application/020-ai-agents-quality-standards.md) - **AI agents implementation quality standards** - Tool definition patterns, error handling, observability, and unit testing for agents
 - [agentme-edr-021](application/021-ai-workflow-development-standards.md) - **AI workflow development standards** - Standard toolchain (LangGraph), evaluation, and testing patterns for workflow projects
 - [agentme-edr-029](application/029-ai-workflow-naming-conventions.md) - **AI workflow naming conventions** - Node suffix/prefix roles, state type and attribute naming, judge output schema, workflow class names, and cross-element coherence rules
-- [agentme-edr-028](application/028-ai-eval-standards.md) - **AI eval standards** - Folder structure, script requirements, and MLflow tracking for eval tests across LLM, Agent, and Workflow tiers
-- [agentme-edr-030](application/030-ai-test-types-taxonomy.md) - **AI test types taxonomy** - Names AI test types (safety, responsible-AI, quality-eval, prompt, code-level) with group, objective, mocking constraint, and relevance, and defines the shared golden dataset entry envelope
-- [agentme-edr-024](application/024-ml-dataset-structure.md) - **ML dataset structure** - Standard folder layout and file conventions for ML datasets
 - [agentme-edr-025](application/025-ai-agent-xdrs-knowledge-layer.md) - **AI agent XDRS knowledge layer** - How to integrate XDRS as the runtime source of truth for policies and skills in AI agents (apply only when the project explicitly uses XDRS)
-- [agentme-edr-026](application/026-pragmatic-hexagonal-architecture.md) - **Pragmatic hexagonal architecture** - Organize application layers as External/Adapters/Application with practical coupling rules
-- [004-select-relevant-xdrs](application/skills/004-select-relevant-xdrs/SKILL.md) - **Select relevant XDRs**
+
+### AI evaluation and testing
+
+Standards for eval datasets, scripts, reports, and test type taxonomy.
+
+- [agentme-edr-030](application/030-ai-test-types-taxonomy.md) - **AI test types taxonomy** - Names AI test types (`functional`, `safety`, `smoke`, `repeatability`, `adversarial`, `fairness`, and 6 others) with group, objective, mocking constraint, and relevance, and defines the shared golden dataset entry envelope
+- [agentme-edr-028](application/028-ai-eval-core-standards.md) - **AI eval core standards** - Eval folder structure and Makefile interface; LLM-as-judge binary scoring contract applicable to all AI tiers and test types
+- [agentme-edr-031](application/031-ai-eval-script.md) - **AI eval script** - eval.py requirements: entry-first loop, --type filtering, mock_fixtures wiring, human entries, threshold enforcement, and MLflow experiment conventions
+- [agentme-edr-032](application/032-ai-eval-report-format.md) - **AI eval report format** - report-<type>.md template, Wilson score confidence interval, convergence analysis, and human-type checklist artifact
+- [agentme-edr-033](application/033-ai-eval-repeatability.md) - **AI eval repeatability** - Repeatability test type: REPEAT_COUNT loop exception, semantic-similarity and LLM-as-judge scoring, repeatability_accuracy metric, report shape, and run cadence
+- [agentme-edr-024](application/024-ml-dataset-structure.md) - **ML dataset structure** - Standard folder layout and file conventions for ML datasets
 
 ## Devops
 

package/.xdrs/agentme/edrs/principles/007-project-quality-standards.md

@@ -175,23 +175,28 @@ Projects that contain statistical models (e.g., ML models, LLM-based evaluators,
 **Requirements:**
 - A `make eval` target MUST exist and execute all performance evaluations
 - Each evaluation MUST have a **documented minimum performance threshold** (e.g., accuracy ≥ 0.85, F1 ≥ 0.80, BLEU ≥ 0.70)
-- Thresholds MUST be declared explicitly in the project (e.g., in a config file, `Makefile` variable, or documented in `README.md`)
+- Thresholds and all scoring parameters MUST be declared as constants in `eval.py` — they are design decisions about acceptable quality for the component under test, not runtime configuration, and MUST NOT be passed as Makefile variables or CLI flags. See [agentme-edr-031](../application/031-ai-eval-script.md) rule `01`.
 - `make eval` MUST **exit with a non-zero status** (fail) if:
   - The evaluation cannot be executed (missing data, environment errors, model load failures)
   - Any metric falls below its defined minimum threshold
 - CI/CD MUST invoke `make eval` before releasing any version that changes model weights, prompts, or evaluation logic
 
-**Threshold declaration example (Makefile):**
+**Threshold declaration example (eval.py):**
 
-```makefile
-EVAL_MIN_ACCURACY := 0.85
-EVAL_MIN_F1       := 0.80
+```python
+EVAL_MIN_ACCURACY = {"functional": 0.85, "smoke": 0.90}
+EVAL_MIN_F1 = {"functional": 0.80}
+
+# Thresholds are declared here and enforced by the script:
+if accuracy < EVAL_MIN_ACCURACY.get(test_type, 0):
+    raise SystemExit(f"Eval failed: {test_type} accuracy {accuracy:.2f} < {EVAL_MIN_ACCURACY[test_type]}")
+```
 
+**Makefile target (delegates to eval.py — no threshold variables here):**
+
+```makefile
 eval:
-	python eval.py \
-	  --min-accuracy $(EVAL_MIN_ACCURACY) \
-	  --min-f1 $(EVAL_MIN_F1) \
-	  || (echo "Evaluation failed: metrics below threshold"; exit 1)
+	mise exec -- uv run --project . python eval.py --type=all
 ```
 
 ---
@@ -261,5 +266,5 @@ AI projects are classified into three tiers — LLM, Agent, and Workflow — def
 - Evals MUST be executed before every release.
 - Accuracy below project-defined thresholds MUST block the release. Thresholds MUST be documented in the eval Makefile or README.
 - Evals MUST run against real LLM providers (not mocks) to capture model drift.
-- For eval folder structure and script requirements, see [agentme-edr-028](../application/028-ai-eval-standards.md).
+- For eval folder structure and LLM-as-judge scoring, see [agentme-edr-028](../application/028-ai-eval-core-standards.md). For eval script requirements, see [agentme-edr-031](../application/031-ai-eval-script.md).
 - For the taxonomy of AI test types (safety, responsible-AI, quality-eval, prompt, code-level) and the golden dataset entry format, see [agentme-edr-030](../application/030-ai-test-types-taxonomy.md).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agentme",
-  "version": "0.26.0",
+  "version": "0.27.0",
   "description": "",
   "dependencies": {
     "filedist": "^0.39.0"

package/.xdrs/agentme/edrs/application/028-ai-eval-standards.md

@@ -1,257 +0,0 @@
----
-name: agentme-edr-policy-028-ai-eval-standards
-description: Defines how to structure, write, and run eval tests for AI projects — folder layout, golden dataset, --type test-type filtering, mock_fixtures wiring, entry-first eval loop, per-type Makefile targets and reports, and MLflow tracking. Use when implementing evals for LLM, Agent, or Workflow projects. For when evals are required see agentme-edr-007 rule 09-ai-project-testing-requirements. For the test type taxonomy and mock_fixtures envelope see agentme-edr-030. For mock file naming see agentme-edr-026 rule 10.
-apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
-valid-from: 2026-06-05
----
-
-# agentme-edr-policy-028: AI eval standards
-
-## Context and Problem Statement
-
-Eval tests measure AI component accuracy against expected outputs using real LLM providers. Without a shared folder layout and script convention, eval setups diverge across LLM, Agent, and Workflow projects, making them hard to run, compare, and integrate into CI/CD pipelines.
-
-How should eval tests be structured and run across all AI tiers?
-
-## Decision Outcome
-
-**Use a per-component folder structure under `evals/` with a standardized Makefile interface and MLflow-backed scripts, applicable to LLM, Agent, and Workflow components.**
-
-For when evals are required per AI tier, see [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
-
-### Details
-
-#### 01-eval-folder-structure
-
-Evals are grouped first by the component being evaluated, then by the specific evaluation scenario. Create one directory per component under `evals/`, and one directory per eval scenario inside it. Place `evals/` at the same level as `lib/` and `examples/`:
-
-```text
-evals/
-  <component>/           # the component being evaluated (e.g., workflow-x, agent-y, model-z)
-    eval-<name>/
-      golden_dataset/    # EDR-024 + EDR-030 compliant golden dataset (README.md, dataset.schema.json, data/)
-      eval.py            # evaluation script
-      report-<type>.md   # generated report, one per evaluated test type (overwritten on each run — see rule 03)
-      Makefile           # lint, eval, run, and eval-<type> targets
-    eval-<name2>/
-      ...
-  <component2>/
-    ...
-```
-
-`<component>` MUST match the name of the component under evaluation and use lowercase hyphen-separated words (e.g., `workflow-document-review`, `agent-support`, `model-classifier`).
-
-`<name>` identifies the specific evaluation scenario using lowercase hyphen-separated words (e.g., `eval-basic`, `eval-complex`, `eval-edge-cases`). A scenario's `golden_dataset` MAY mix multiple test types across its entries: label each entry with its applicable `test_types` ([agentme-edr-030](030-ai-test-types-taxonomy.md) rule `04`) and use the `eval-<type>` targets below to run one type at a time.
-
-The `golden_dataset/` subfolder MUST be a valid [agentme-edr-024](024-ml-dataset-structure.md) dataset (`README.md`, `dataset.schema.json`, one JSON file per entry under `data/` per rule `04-complex-structured-datasets-must-use-per-entry-json-files`, lint-validated per rule `06`) whose entries follow the golden dataset envelope defined in [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02`.
-
-Each `evals/<component>/eval-<name>/Makefile` MUST declare a `TEST_TYPES` variable listing the `test_types` values present in its golden dataset, and define:
-
-| Target | Behaviour |
-|---|---|
-| `lint` | Validates every `golden_dataset/data/*.json` file against `golden_dataset/dataset.schema.json` per [agentme-edr-024](024-ml-dataset-structure.md) rule `06` |
-| `eval` | Depends on `lint`; runs `eval.py --type=all` with threshold enforcement; exits non-zero on failure (CI-safe) |
-| `run` | Depends on `lint`; runs `eval.py --type=all` without threshold enforcement (exploration / debugging) |
-| `eval-<type>` | Depends on `lint`; runs `eval.py --type=<type>` for one declared test type, following [agentme-edr-008](../devops/008-common-targets.md) rule `03`'s `eval-<qualifier>` convention |
-
-```makefile
-TEST_TYPES := smoke functional safety
-
-lint:
-	mise exec -- uv run --project . python lint_dataset.py golden_dataset/
-
-eval: lint
-	mise exec -- uv run --project . python eval.py --type=all
-
-run: lint
-	mise exec -- uv run --project . python eval.py --type=all --no-threshold
-
-eval-%: lint
-	mise exec -- uv run --project . python eval.py --type=$*
-```
-
-The module root Makefile MUST expose `make eval` and `make lint` targets that delegate to `eval` and `lint` respectively in every `evals/<component>/eval-<name>/Makefile`:
-
-```makefile
-eval:
-	$(MAKE) -C evals/workflow-document-review/eval-basic eval
-	$(MAKE) -C evals/workflow-document-review/eval-complex eval
-
-lint:
-	$(MAKE) -C evals/workflow-document-review/eval-basic lint
-	$(MAKE) -C evals/workflow-document-review/eval-complex lint
-```
-
-#### 02-eval-script-requirements
-
-Each `eval.py` script MUST:
-
-- Load the golden dataset from `golden_dataset/` in the same eval folder, following [agentme-edr-024](024-ml-dataset-structure.md) and the entry envelope in [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02` (one JSON file per entry, `test_types` array, `input`, `expected_output`, optional `mock_fixtures`).
-- Accept a required `--type=<test_type>|all` CLI argument and filter entries whose `test_types` array contains the requested value; `--type=all` includes every entry.
-- Iterate **entry-first**: for each entry in the filtered set, invoke the real component exactly once; then score that single `actual_output` for every `test_types` value the entry carries that falls within the current `--type` scope — MUST NOT invoke the component more than once per entry per run.
-- When an entry contains `mock_fixtures` ([agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02`), configure each named mock adapter with its fixture data BEFORE invoking the component for that entry. Each entry MUST use fresh mock instances so fixture state does not bleed across entries. `mock_fixtures` applies to all test types including `human`. `mock_fixtures` MUST NOT configure LLM adapters — the LLM call MUST be real (see [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `03`). How mock adapters are discovered and instantiated is left to the project; see [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
-- Run every component invocation against **real LLM providers** (not mocked responses), to capture model drift.
-- For `human` entries: invoke the component to capture `actual_output`, export each entry's `input`, `expected_output.human_test` instructions, and `actual_output` into a manual-review checklist (`report-human.md`). MUST NOT invoke an automated scorer and MUST NOT enforce a pass/fail threshold for it. Other `test_types` on the same entry (e.g. `functional`) are still scored automatically.
-- After all entries are processed, compute aggregate metrics per test type, log them to a local MLflow experiment (see rule `04`), write one `report-<type>.md` per evaluated test type (rule `03`), and exit with a non-zero status when any metric falls below its defined threshold per [agentme-edr-007](../principles/007-project-quality-standards.md) rule `07-statistical-models-must-have-eval-targets`. The `human` type has no threshold and does not trigger a non-zero exit.
-- Compare outputs to expected values using project-defined quality thresholds per test type. Thresholds MUST be declared explicitly (e.g., in a Makefile variable or README) — this Policy does not mandate which test types a project must threshold or what value to use (see [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `06`).
-
-**Example:**
-
-```python
-import argparse
-from collections import defaultdict
-import mlflow
-from my_package.app.workflows.document_review_workflow.graph import graph
-
-EVAL_MIN_ACCURACY = {"functional": 0.85, "smoke": 0.85}
-
-parser = argparse.ArgumentParser()
-parser.add_argument("--type", required=True)
-args = parser.parse_args()
-
-entries = load_golden_dataset("golden_dataset/", test_type=args.type)  # "all" loads every entry
-resolved_types = resolve_types(args.type, entries)
-
-mlflow.set_experiment("document-review/eval-basic")
-
-with mlflow.start_run():
-    mlflow.set_tag("test_types", ",".join(sorted(resolved_types)))
-
-    results = defaultdict(list)
-
-    # Entry-first loop: invoke each entry exactly once
-    for entry in entries:
-        # Configure mock adapters from mock_fixtures before invocation
-        # (implementation left to the project — see agentme-edr-026 rule 10)
-        if entry.get("mock_fixtures"):
-            configure_mocks(entry["mock_fixtures"])  # project-defined helper
-
-        actual_output = invoke_component(entry, graph)
-
-        for test_type in [t for t in entry["test_types"] if t in resolved_types]:
-            if test_type == "human":
-                export_human_review(entry, actual_output)
-                continue
-            results[test_type].append(score(test_type, actual_output, entry["expected_output"]))
-
-    # Aggregate, report, and enforce thresholds per test type
-    for test_type in resolved_types:
-        if test_type == "human":
-            continue
-
-        accuracy = sum(results[test_type]) / len(results[test_type])
-        mlflow.log_metric(f"{test_type}_accuracy", accuracy)
-        write_eval_report(test_type, results[test_type], thresholds={"accuracy": EVAL_MIN_ACCURACY[test_type]})
-
-        if accuracy < EVAL_MIN_ACCURACY[test_type]:
-            raise SystemExit(f"Eval failed: {test_type} accuracy {accuracy:.2f} < {EVAL_MIN_ACCURACY[test_type]}")
-```
-
-#### 03-eval-report-file
-
-Each eval script MUST produce one `report-<type>.md` per evaluated test type in the same `evals/<component>/eval-<name>/` folder and overwrite each on every run — only the types included in the current `--type` invocation are (re)written; report files for other types are left untouched. The `human` type does not produce a metrics report (see below).
-
-**Generation constraint:** The report MUST be produced programmatically, reading raw metric values directly from MLflow. No LLM or generative model may write, summarize, or paraphrase any section of the report, to prevent hallucinated metric values.
-
-The report MUST follow this template:
-
-```markdown
-# Eval Report: <name> — <type>
-
-**Date:** <ISO date>
-**Dataset:** golden_dataset/
-**Script:** eval.py --type=<type>
-**Thresholds:** accuracy ≥ <value>, F1 ≥ <value>
-
-## Overall Results
-
-| Metric    | Value  | 95% CI         | Threshold | Status  |
-|-----------|--------|----------------|-----------|---------|
-| Accuracy  | <val>  | [<low>, <high>]| ≥ <thr>   | ✓/✗ PASS/FAIL |
-| F1 Score  | <val>  | —              | ≥ <thr>   | ✓/✗ PASS/FAIL |
-| Precision | <val>  | —              | —         | —       |
-| Recall    | <val>  | —              | —         | —       |
-| Samples   | <n>    | —              | —         | —       |
-
-**Overall: PASS / FAIL**
-
-## Per-item Results
-
-| ID  | Input Summary | Expected | Actual | Correct |
-|-----|---------------|----------|--------|---------|
-| 001 | <summary>     | <label>  | <label>| ✓       |
-| 002 | <summary>     | <label>  | <label>| ✗       |
-
-## Notes
-
-- <observations, failure patterns, MLflow run link>
-```
-
-**Confidence interval:** The 95% CI for accuracy MUST be computed using the **Wilson score interval** (preferred over the normal approximation for small $n$). A wide interval signals that the dataset is too small to support confident conclusions and the sample count should be increased.
-
-The Wilson score bounds at 95% confidence ($z = 1.96$) are:
-
-$$\frac{\hat{p} + \frac{z^2}{2n} \pm z\sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}$$
-
-Where $\hat{p}$ is observed accuracy and $n$ is sample count. Accuracy and F1 are required; precision and recall are recommended.
-
-**Filled-in example** (`evals/workflow-document-review/eval-basic/report-functional.md` for a document review workflow):
-
-```markdown
-# Eval Report: eval-basic — functional
-
-**Date:** 2026-06-12
-**Dataset:** golden_dataset/
-**Script:** eval.py --type=functional
-**Thresholds:** accuracy ≥ 0.85, F1 ≥ 0.80
-
-## Overall Results
-
-| Metric    | Value | 95% CI       | Threshold | Status      |
-|-----------|-------|--------------|-----------|-------------|
-| Accuracy  | 0.88  | [0.69, 0.97] | ≥ 0.85    | ✓ PASS      |
-| F1 Score  | 0.86  | —            | ≥ 0.80    | ✓ PASS      |
-| Precision | 0.89  | —            | —         | —           |
-| Recall    | 0.84  | —            | —         | —           |
-| Samples   | 25    | —            | —         | —           |
-
-**Overall: PASS**
-
-> Note: CI [0.69, 0.97] is wide — 25 samples may be insufficient for high confidence. Consider expanding the dataset.
-
-## Per-item Results
-
-| ID  | Input Summary                       | Expected | Actual   | Correct |
-|-----|--------------------------------------|----------|----------|---------|
-| 001 | Contract renewal, 3 pages, standard | approve  | approve  | ✓       |
-| 002 | NDA with unusual liability clause   | escalate | escalate | ✓       |
-| 003 | Vendor invoice, missing PO number   | reject   | reject   | ✓       |
-| 004 | Employment agreement, standard terms| approve  | approve  | ✓       |
-| 005 | Amendment with redlined IP clause   | escalate | approve  | ✗       |
-
-## Notes
-
-- Sample 005 misclassified: redlined IP clause not flagged as escalation trigger. Possible model drift.
-- MLflow run: experiment `workflow-document-review/eval-basic`, tag `test_types=functional` — view with `mlflow ui`
-```
-
-**`human` type artifact:** instead of `report-human.md` with metrics, `--type=human` produces a checklist artifact (still named `report-human.md`) listing, per entry, its `input`, `expected_output.human_test` instructions, and the captured `actual_output` — with no Overall Results table, threshold, or PASS/FAIL section, since this type MUST NOT be auto-scored.
-
-#### 04-eval-mlflow-unique-port
-
-Each `evals/<component>/eval-<name>/Makefile` MUST start its MLflow tracking server on a **unique port** to prevent conflicts when multiple eval Makefiles are run concurrently or in parallel (e.g., in CI or across multiple terminal sessions).
-
-Ports MUST be statically assigned per eval scenario (not per test type) and MUST NOT reuse the default `5000` port (reserved for `dev-mlflow` per [agentme-edr-008](../devops/008-common-targets.md) rule `09-ai-project-dev-targets`). Assign ports starting at `5100` and incrementing by 1 for each additional eval scenario across the entire project.
-
-The MLflow **experiment** is scoped to the eval scenario: `<component>/<eval-name>` (e.g. `document-review/eval-basic`). Each `mlflow.start_run()` call MUST set a `test_types` tag listing the test types evaluated in that invocation (comma-separated, e.g. `"functional,smoke"` for `--type=all`, `"smoke"` for `--type=smoke`). A remote MLflow server MUST NOT be required — all tracking is local.
-
-## References
-
-- [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09-ai-project-testing-requirements`) and statistical model eval targets (rule `07-statistical-models-must-have-eval-targets`)
-- [agentme-edr-030](030-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum, golden dataset entry envelope (including `mock_fixtures`), and mocking constraints per type
-- [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) — Rule `10`: `_mock` file naming and placement convention for mock adapters used in `mock_fixtures`
-- [agentme-edr-018](018-ai-llm-development-standards.md) — LLM development standards: LangChain framework and observability
-- [agentme-edr-019](019-ai-agents-development-standards.md) — Agent development standards
-- [agentme-edr-021](021-ai-workflow-development-standards.md) — Workflow development standards
-- [agentme-edr-024](024-ml-dataset-structure.md) — ML dataset structure, per-entry JSON format, and schema-lint validation for golden datasets
-- [agentme-edr-008](../devops/008-common-targets.md) — `eval-<qualifier>` Makefile convention (rule `03`) and Mise tool-execution flow (rule `02`)