agentme 0.25.2 → 0.27.0

package/.xdrs/agentme/edrs/application/031-ai-eval-script.md

@@ -0,0 +1,136 @@
+---
+name: agentme-edr-policy-031-ai-eval-script
+description: Defines eval.py script requirements for AI projects — entry-first eval loop, --type test-type filtering, mock_fixtures wiring, human entries, threshold enforcement, and MLflow experiment naming and port assignment. Use when implementing eval scripts. For eval folder structure see agentme-edr-028 rule 01. For the test type taxonomy and mock_fixtures envelope see agentme-edr-030. For mock file naming see agentme-edr-026 rule 10. For report format see agentme-edr-032. For repeatability loop exception see agentme-edr-033.
+apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
+valid-from: 2026-07-07
+---
+
+# agentme-edr-policy-031: AI eval script
+
+## Context and Problem Statement
+
+Eval scripts execute entries from a golden dataset against a real AI component and measure output quality. Without a shared script contract, eval implementations diverge: some invoke components multiple times per entry (wasting LLM cost), some skip mock isolation, and some omit threshold enforcement — making results inconsistent and hard to trust across projects.
+
+How should eval scripts load datasets, iterate entries, handle mocking, and produce metrics?
+
+## Decision Outcome
+
+**Use an entry-first eval loop with `--type` filtering, fresh mock isolation per entry, real LLM providers, and MLflow-backed metrics with explicit per-type thresholds.**
+
+For when evals are required per AI tier, see [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
+
+### Details
+
+#### 01-eval-script-requirements
+
+Each `eval.py` script MUST:
+
+- Load the golden dataset from `golden_dataset/` in the same eval folder, following [agentme-edr-024](024-ml-dataset-structure.md) and the entry envelope in [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02` (one JSON file per entry, `test_types` array, `input`, `expected_output`, optional `mock_fixtures`).
+- Accept a required `--type=<test_type>|all` CLI argument and filter entries whose `test_types` array contains the requested value; `--type=all` includes every entry.
+- Iterate **entry-first**: for each entry in the filtered set, invoke the real component exactly once; then score that single `actual_output` for every `test_types` value the entry carries that falls within the current `--type` scope — MUST NOT invoke the component more than once per entry per run.
+- When an entry contains `mock_fixtures` ([agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02`), configure each named mock adapter with its fixture data BEFORE invoking the component for that entry. Each entry MUST use fresh mock instances so fixture state does not bleed across entries. `mock_fixtures` applies to all test types including `human`. `mock_fixtures` MUST NOT configure LLM adapters — the LLM call MUST be real (see [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `03`). How mock adapters are discovered and instantiated is left to the project; see [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
+- Run every component invocation against **real LLM providers** (not mocked responses), to capture model drift.
+- For `human` entries: invoke the component to capture `actual_output`, export each entry's `input`, `expected_output.human_test` instructions, and `actual_output` into a manual-review checklist (`report-human.md`). MUST NOT invoke an automated scorer and MUST NOT enforce a pass/fail threshold for it. Other `test_types` on the same entry (e.g. `functional`) are still scored automatically.
+- After all entries are processed, compute aggregate metrics per test type, log them to a local MLflow experiment (see rule `02`), write one `report-<type>.md` per evaluated test type ([agentme-edr-032](032-ai-eval-report-format.md) rule `01`), and exit with a non-zero status when any metric falls below its defined threshold per [agentme-edr-007](../principles/007-project-quality-standards.md) rule `07-statistical-models-must-have-eval-targets`. The `human` type has no threshold and does not trigger a non-zero exit.
+- Compare outputs to expected values using project-defined quality thresholds per test type. Thresholds and all other scoring parameters MUST be declared as constants in `eval.py` — they are design decisions about what constitutes acceptable quality for the component under test, not runtime configuration, and MUST NOT be passed via Makefile variables or CLI flags. Use one of two naming conventions, chosen consistently within an `eval.py`: (a) **per-type constants** — `EVAL_MIN_<METRIC>_<TYPE>` for each test type (e.g. `EVAL_MIN_ACCURACY_FUNCTIONAL = 0.85`, `EVAL_MIN_ACCURACY_REPEATABILITY = 0.8`); or (b) **dict constant** — `EVAL_MIN_<METRIC> = {<type>: <value>}` (e.g. `EVAL_MIN_ACCURACY = {"functional": 0.85, "smoke": 0.85}`). Per-type constants are preferred when each test type has a dedicated `eval.py`; the dict form is preferred when a single `eval.py` handles multiple types. In either convention, `EVAL_MIN_ACCURACY` (as a scalar) MAY be declared as a project-wide default and MUST be used as fallback when no per-type override is defined for the current test type. This Policy does not mandate which test types a project must threshold or what value to use (see [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `06`).
+
+**Example:**
+
+```python
+import argparse
+from collections import defaultdict
+import mlflow
+from my_package.app.workflows.document_review_workflow.graph import graph
+
+# Per-type constants; fall back to EVAL_MIN_ACCURACY default when no override is defined
+EVAL_MIN_ACCURACY = 0.85  # project-wide default
+EVAL_MIN_ACCURACY_FUNCTIONAL = 0.85
+EVAL_MIN_ACCURACY_SMOKE = 0.90
+EVAL_MIN_ACCURACY_PER_TYPE = {"functional": EVAL_MIN_ACCURACY_FUNCTIONAL, "smoke": EVAL_MIN_ACCURACY_SMOKE}
+
+def get_min_accuracy(test_type: str) -> float:
+    return EVAL_MIN_ACCURACY_PER_TYPE.get(test_type, EVAL_MIN_ACCURACY)
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--type", required=True)
+args = parser.parse_args()
+
+entries = load_golden_dataset("golden_dataset/", test_type=args.type)  # "all" loads every entry
+resolved_types = resolve_types(args.type, entries)
+
+mlflow.set_experiment("document-review/eval-basic")
+
+with mlflow.start_run():
+    mlflow.set_tag("test_types", ",".join(sorted(resolved_types)))
+
+    results = defaultdict(list)
+    cumulative_metrics = defaultdict(lambda: {"accuracy": [], "f1": []})  # Track cumulative metrics
+
+    # Entry-first loop: invoke each entry exactly once
+    for idx, entry in enumerate(entries, start=1):
+        # Configure mock adapters from mock_fixtures before invocation
+        # (implementation left to the project — see agentme-edr-026 rule 10)
+        if entry.get("mock_fixtures"):
+            configure_mocks(entry["mock_fixtures"])  # project-defined helper
+
+        actual_output = invoke_component(entry, graph)
+
+        for test_type in [t for t in entry["test_types"] if t in resolved_types]:
+            if test_type == "human":
+                export_human_review(entry, actual_output)
+                continue
+            
+            score_val = score(test_type, actual_output, entry["expected_output"])
+            results[test_type].append(score_val)
+            
+            # Track cumulative metrics for convergence analysis
+            cumulative_accuracy = sum(results[test_type]) / len(results[test_type])
+            cumulative_f1 = compute_f1(results[test_type])  # project-defined
+            cumulative_metrics[test_type]["accuracy"].append(cumulative_accuracy)
+            cumulative_metrics[test_type]["f1"].append(cumulative_f1)
+
+    # Aggregate, report, and enforce thresholds per test type
+    for test_type in resolved_types:
+        if test_type == "human":
+            continue
+
+        accuracy = sum(results[test_type]) / len(results[test_type])
+        mlflow.log_metric(f"{test_type}_accuracy", accuracy)
+        
+        # Generate convergence analysis
+        stability_window = min(10, len(results[test_type]))
+        acc_change = abs(cumulative_metrics[test_type]["accuracy"][-1] - 
+                        cumulative_metrics[test_type]["accuracy"][-stability_window])
+        f1_change = abs(cumulative_metrics[test_type]["f1"][-1] - 
+                       cumulative_metrics[test_type]["f1"][-stability_window])
+        
+        write_eval_report(
+            test_type, 
+            results[test_type], 
+            cumulative_metrics=cumulative_metrics[test_type],
+            stability_window=stability_window,
+            thresholds={"accuracy": get_min_accuracy(test_type)}
+        )
+
+        if accuracy < get_min_accuracy(test_type):
+            raise SystemExit(f"Eval failed: {test_type} accuracy {accuracy:.2f} < {get_min_accuracy(test_type)}")
+```
+
+#### 02-eval-mlflow-unique-port
+
+Each `evals/<component>/eval-<name>/Makefile` MUST start its MLflow tracking server on a **unique port** to prevent conflicts when multiple eval Makefiles are run concurrently or in parallel (e.g., in CI or across multiple terminal sessions).
+
+Ports MUST be statically assigned per eval scenario (not per test type) and MUST NOT reuse the default `5000` port (reserved for `dev-mlflow` per [agentme-edr-008](../devops/008-common-targets.md) rule `09-ai-project-dev-targets`). Assign ports starting at `5100` and incrementing by 1 for each additional eval scenario across the entire project.
+
+The MLflow **experiment** is scoped to the eval scenario: `<component>/<eval-name>` (e.g. `document-review/eval-basic`). Each `mlflow.start_run()` call MUST set a `test_types` tag listing the test types evaluated in that invocation (comma-separated, e.g. `"functional,smoke"` for `--type=all`, `"smoke"` for `--type=smoke`). A remote MLflow server MUST NOT be required — all tracking is local.
+
+## References
+
+- [agentme-edr-028](028-ai-eval-core-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract (rule `02`)
+- [agentme-edr-032](032-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
+- [agentme-edr-033](033-ai-eval-repeatability.md) — AI eval repeatability: loop exception to rule `01`'s entry-first constraint, scoring methods, and cadence
+- [agentme-edr-030](030-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum, golden dataset entry envelope (including `mock_fixtures`), and mocking constraints per type
+- [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) — Rule `10`: `_mock` file naming and placement convention for mock adapters used in `mock_fixtures`
+- [agentme-edr-024](024-ml-dataset-structure.md) — ML dataset structure, per-entry JSON format, and schema-lint validation for golden datasets
+- [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09`) and statistical model eval targets (rule `07`)
+- [agentme-edr-008](../devops/008-common-targets.md) — `eval-<qualifier>` Makefile convention (rule `03`) and reserved MLflow port `5000` (rule `09`)

package/.xdrs/agentme/edrs/application/032-ai-eval-report-format.md

@@ -0,0 +1,171 @@
+---
+name: agentme-edr-policy-032-ai-eval-report-format
+description: Defines the eval report format for AI projects — report-<type>.md template, Wilson score confidence interval, convergence analysis, and the human-type checklist artifact. Use when generating eval reports from eval scripts. For the eval script that produces these reports see agentme-edr-031. For repeatability report shape see agentme-edr-033 rule 02.
+apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
+valid-from: 2026-07-07
+---
+
+# agentme-edr-policy-032: AI eval report format
+
+## Context and Problem Statement
+
+Eval scripts produce output reports, but without a shared format these vary across projects: some omit confidence intervals, some skip convergence analysis, and some allow LLMs to write sections — making reports unreliable and non-comparable.
+
+What format should eval reports follow, and what constraints apply to how they are generated?
+
+## Decision Outcome
+
+**Use a standardized `report-<type>.md` template with a Wilson score confidence interval, a Mermaid convergence chart, and a strict no-LLM generation constraint for all metric values.**
+
+### Details
+
+#### 01-eval-report-file
+
+Each eval script MUST produce one `report-<type>.md` per evaluated test type in the same `evals/<component>/eval-<name>/` folder and overwrite each on every run — only the types included in the current `--type` invocation are (re)written; report files for other types are left untouched. The `human` type does not produce a metrics report (see below).
+
+**Generation constraint:** The report MUST be produced programmatically, reading raw metric values directly from MLflow. No LLM or generative model may write, summarize, or paraphrase any section of the report, to prevent hallucinated metric values. This constraint applies to all report sections including Overall Results, Convergence Analysis, and Per-item Results — all metric values and convergence chart data points MUST be computed from actual evaluation results.
+
+The report MUST follow this template:
+
+```markdown
+# Eval Report: <name> — <type>
+
+**Date:** <ISO date>
+**Dataset:** golden_dataset/
+**Script:** eval.py --type=<type>
+**Thresholds:** accuracy ≥ <value>, F1 ≥ <value>
+
+## Overall Results
+
+| Metric    | Value  | 95% CI         | Threshold | Status  |
+|-----------|--------|----------------|-----------|---------|
+| Accuracy  | <val>  | [<low>, <high>]| ≥ <thr>   | ✓/✗ PASS/FAIL |
+| F1 Score  | <val>  | —              | ≥ <thr>   | ✓/✗ PASS/FAIL |
+| Precision | <val>  | —              | —         | —       |
+| Recall    | <val>  | —              | —         | —       |
+| Samples   | <n>    | —              | —         | —       |
+
+**Overall: PASS / FAIL**
+
+## Convergence Analysis
+
+```mermaid
+xychart-beta
+  title "Metric Evolution vs Sample Count"
+  x-axis "Samples" [<sample_points>]
+  y-axis "Score" 0.0 --> 1.0
+  line "Accuracy" [<accuracy_values>]
+  line "F1 Score" [<f1_values>]
+```
+
+**Stability Analysis:**
+- Accuracy change over last <window> samples: <change> percentage points
+- F1 change over last <window> samples: <change> percentage points
+
+**Recommendation:** <"Dataset appears sufficient for confident evaluation" | "Add more samples — metrics have not yet stabilized">
+
+## Per-item Results
+
+| ID  | Input Summary | Expected | Actual | Correct |
+|-----|---------------|----------|--------|---------|  
+| 001 | <summary>     | <label>  | <label>| ✓       |
+| 002 | <summary>     | <label>  | <label>| ✗       |
+
+## Notes
+
+- <observations, failure patterns, MLflow run link>
+```
+
+**Confidence interval:** The 95% CI for accuracy MUST be computed using the **Wilson score interval** (preferred over the normal approximation for small $n$). A wide interval signals that the dataset is too small to support confident conclusions and the sample count should be increased.
+
+The Wilson score bounds at 95% confidence ($z = 1.96$) are:
+
+$$\frac{\hat{p} + \frac{z^2}{2n} \pm z\sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}$$
+
+Where $\hat{p}$ is observed accuracy and $n$ is sample count. Accuracy and F1 are required; precision and recall are recommended.
+
+**Convergence analysis:** The Convergence Analysis section shows whether adding more samples would likely change measured metrics. The section MUST include:
+
+1. **Mermaid xychart-beta** showing cumulative Accuracy and F1 evolution:
+   - X-axis: absolute cumulative sample count; Y-axis: metric value (0.0 to 1.0)
+   - Two lines: Accuracy and F1
+   - For datasets > 50 samples: sample at `floor(dataset_size / 10)` intervals (minimum 5), always include first and last points
+   - For datasets ≤ 50 samples: show all points
+
+2. **Stability analysis**: compute absolute change (percentage points) for both Accuracy and F1 over last `min(10, dataset_size)` samples. Example: Accuracy from 0.85 to 0.87 = 0.02 = 2 percentage points.
+
+3. **Recommendation**:
+   - Default threshold: both Accuracy AND F1 change ≤ 2 percentage points
+   - If both meet threshold: "Dataset appears sufficient for confident evaluation"
+   - If either exceeds: "Add more samples — metrics have not yet stabilized"
+   - Projects MAY customize threshold (document in Makefile/README)
+
+Exclude from `report-human.md` (no automated metrics).
+
+**Filled-in example** (`evals/workflow-document-review/eval-basic/report-functional.md` for a document review workflow):
+
+```markdown
+# Eval Report: eval-basic — functional
+
+**Date:** 2026-06-12
+**Dataset:** golden_dataset/
+**Script:** eval.py --type=functional
+**Thresholds:** accuracy ≥ 0.85, F1 ≥ 0.80
+
+## Overall Results
+
+| Metric    | Value | 95% CI       | Threshold | Status      |
+|-----------|-------|--------------|-----------|-------------|
+| Accuracy  | 0.88  | [0.69, 0.97] | ≥ 0.85    | ✓ PASS      |
+| F1 Score  | 0.86  | —            | ≥ 0.80    | ✓ PASS      |
+| Precision | 0.89  | —            | —         | —           |
+| Recall    | 0.84  | —            | —         | —           |
+| Samples   | 25    | —            | —         | —           |
+
+**Overall: PASS**
+
+> Note: CI [0.69, 0.97] is wide — 25 samples may be insufficient for high confidence. Consider expanding the dataset.
+
+## Convergence Analysis
+
+```mermaid
+xychart-beta
+  title "Metric Evolution vs Sample Count"
+  x-axis "Samples" [1, 5, 10, 15, 20, 25]
+  y-axis "Score" 0.0 --> 1.0
+  line "Accuracy" [0.60, 0.80, 0.85, 0.87, 0.88, 0.88]
+  line "F1 Score" [0.55, 0.78, 0.83, 0.85, 0.86, 0.86]
+```
+
+**Stability Analysis:**
+- Accuracy change over last 10 samples: 0.01 percentage points
+- F1 change over last 10 samples: 0.01 percentage points
+
+**Recommendation:** Dataset appears sufficient for confident evaluation
+
+> Metrics stabilized after ~15 samples. Changes over last 10 samples are well below the 2 percentage point threshold.
+
+## Per-item Results
+
+| ID  | Input Summary                       | Expected | Actual   | Correct |
+|-----|--------------------------------------|----------|----------|---------|
+| 001 | Contract renewal, 3 pages, standard | approve  | approve  | ✓       |
+| 002 | NDA with unusual liability clause   | escalate | escalate | ✓       |
+| 003 | Vendor invoice, missing PO number   | reject   | reject   | ✓       |
+| 004 | Employment agreement, standard terms| approve  | approve  | ✓       |
+| 005 | Amendment with redlined IP clause   | escalate | approve  | ✗       |
+
+## Notes
+
+- Sample 005 misclassified: redlined IP clause not flagged as escalation trigger. Possible model drift.
+- MLflow run: experiment `workflow-document-review/eval-basic`, tag `test_types=functional` — view with `mlflow ui`
+```
+```
+
+**`human` type artifact:** instead of `report-human.md` with metrics, `--type=human` produces a checklist artifact (still named `report-human.md`) listing, per entry, its `input`, `expected_output.human_test` instructions, and the captured `actual_output` — with no Overall Results table, threshold, or PASS/FAIL section, since this type MUST NOT be auto-scored.
+
+## References
+
+- [agentme-edr-031](031-ai-eval-script.md) — AI eval script: the script that produces these reports (rule `01`)
+- [agentme-edr-033](033-ai-eval-repeatability.md) — AI eval repeatability: rule `02` defines the adapted report shape for `report-repeatability.md`
+- [agentme-edr-028](028-ai-eval-core-standards.md) — AI eval core standards: folder structure (rule `01`) and LLM-as-judge binary scoring (rule `02`)

package/.xdrs/agentme/edrs/application/033-ai-eval-repeatability.md

@@ -0,0 +1,75 @@
+---
+name: agentme-edr-policy-033-ai-eval-repeatability
+description: Defines the repeatability test type for AI evals — REPEAT_COUNT loop exception to the entry-first constraint, semantic-similarity and LLM-as-judge scoring methods, repeatability_accuracy metric, and the repeatability report format and run cadence. Use when implementing repeatability evals. For the entry-first constraint see agentme-edr-031 rule 01. For LLM-as-judge binary output see agentme-edr-028 rule 02. For the base report template see agentme-edr-032 rule 01. For the repeatability test type definition see agentme-edr-030.
+apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement repeatability eval testing
+valid-from: 2026-07-07
+---
+
+# agentme-edr-policy-033: AI eval repeatability
+
+## Context and Problem Statement
+
+Some AI components must be tested for output consistency across multiple invocations with the same input — a property the standard entry-first eval loop cannot capture because it invokes each entry only once. Without a shared repeatability protocol, teams either skip this test or implement ad-hoc variations that produce incomparable results.
+
+How should repeatability be measured in evals, and how should its results be reported?
+
+## Decision Outcome
+
+**Exempt `repeatability` entries from the entry-first constraint, invoking each `REPEAT_COUNT` times and scoring via semantic-similarity or LLM-as-judge; schedule at release cadence rather than per-commit.**
+
+For the `repeatability` test type definition and its disambiguation from `reproducibility`, see [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `09`.
+
+### Details
+
+#### 01-repeatability-eval-loop-exception
+
+Entries whose `test_types` includes `repeatability` are exempt from [agentme-edr-031](031-ai-eval-script.md) rule `01`'s "invoke exactly once per entry" constraint. The following constants MUST be declared in `eval.py` and MUST NOT be exposed as Makefile variables, CLI flags, or stored as per-entry dataset fields:
+
+- `REPEAT_COUNT` — number of times each repeatability entry is invoked. SHOULD default to 3-5 for routine CI runs and 10-20 for focused passes on decision-critical or previously-flagged components. Projects SHOULD calibrate the value once per component by plotting cumulative pass rate against repeat count for a few representative entries and picking the point where it plateaus, rather than guessing.
+- `EVAL_MIN_ACCURACY_REPEATABILITY` — minimum fraction of repeatability entries that must PASS for the eval to exit 0.
+- `REPEAT_SEMANTIC_SIMILARITY_SCORE` — minimum average pairwise cosine similarity for a single entry to PASS; declared only when using semantic-similarity scoring.
+
+`eval.py` MUST invoke the component `REPEAT_COUNT` times for every repeatability entry and score the resulting outputs by comparing them to each other. `expected_output` is unused for repeatability entries and SHOULD be omitted or set to `null` in the dataset.
+
+**Choosing the scoring method:** Two approaches are supported, declared as a constant in `eval.py`:
+
+- **Semantic-similarity:** Embed all `REPEAT_COUNT` outputs for an entry into vectors and compute the average pairwise cosine similarity. The entry passes (score = 1) if the average similarity ≥ `REPEAT_SEMANTIC_SIMILARITY_SCORE`; otherwise it fails (score = 0). Use for classification or short structured outputs.
+- **LLM-as-judge:** Provide all `REPEAT_COUNT` outputs for an entry to an LLM judge (kept at low/zero temperature) that returns 0 (fail) or 1 (pass) directly, based on whether the outputs are sufficiently consistent. No `REPEAT_SEMANTIC_SIMILARITY_SCORE` constant is needed. Use for free-text or complex structured outputs where vector distance is an unreliable proxy for agreement. The judge MUST follow [agentme-edr-028](028-ai-eval-core-standards.md) rule `02`'s binary output contract.
+
+**`repeatability_accuracy`:** the fraction of repeatability entries that received PASS (score = 1), logged to MLflow as `repeatability_accuracy`. The eval exits non-zero if `repeatability_accuracy` < `EVAL_MIN_ACCURACY_REPEATABILITY`. Both `repeatability_accuracy` and `repeat_count` MUST be logged to MLflow and included in `report-repeatability.md` (rule `02`).
+
+```python
+REPEAT_COUNT = 5
+EVAL_MIN_ACCURACY_REPEATABILITY = 0.8
+REPEAT_SEMANTIC_SIMILARITY_SCORE = 0.9  # only when using semantic-similarity scoring
+
+for entry in repeatability_entries:
+    outputs = [invoke_component(entry, graph) for _ in range(REPEAT_COUNT)]
+    entry_pass = score_agreement(outputs)  # returns 0 or 1; expected_output is not used
+    results["repeatability"].append(entry_pass)  # 1 = PASS, 0 = FAIL
+
+repeatability_accuracy = sum(results["repeatability"]) / len(results["repeatability"])
+mlflow.log_metric("repeatability_accuracy", repeatability_accuracy)
+mlflow.log_metric("repeat_count", REPEAT_COUNT)
+
+if repeatability_accuracy < EVAL_MIN_ACCURACY_REPEATABILITY:
+    raise SystemExit(f"Eval failed: repeatability_accuracy {repeatability_accuracy:.2f} < {EVAL_MIN_ACCURACY_REPEATABILITY}")
+```
+
+`mock_fixtures` configuration per [agentme-edr-031](031-ai-eval-script.md) rule `01` applies to each of the `REPEAT_COUNT` invocations. Any prompt or response caching (provider-side or gateway-side) MUST be bypassed for these invocations — a cache hit would return an identical cached response and falsely report perfect stability instead of measuring the model's actual variance.
+
+**Scoping:** this test type MUST NOT be applied to components whose intended behavior is diverse or creative output (e.g. brainstorming, creative writing) — low agreement there is correct behavior, not a defect.
+
+#### 02-repeatability-report-and-cadence
+
+`--type=repeatability` MUST produce `report-repeatability.md` with a shape adapted from [agentme-edr-032](032-ai-eval-report-format.md) rule `01`'s template: the header MUST include a **Repeat count:** line stating the `REPEAT_COUNT` value used for the run, alongside the usual Date/Dataset/Script/Thresholds lines. The body MUST have an aggregate row reporting `repeatability_accuracy` (the fraction of entries that PASS — see rule `01`) with a Wilson score interval computed over the number of `repeatability` entries, plus a per-item table listing each entry's individual pass/fail result and, when using semantic-similarity, its computed average pairwise cosine similarity — instead of the `Expected | Actual | Correct` columns used by other types.
+
+Because `repeatability` entries multiply real LLM-provider calls by `REPEAT_COUNT`, projects SHOULD schedule `make eval-repeatability` at release cadence rather than on every commit, aligned with the Workflow eval cadence in [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09`, rather than treating it as a mandatory per-commit gate.
+
+## References
+
+- [agentme-edr-031](031-ai-eval-script.md) — AI eval script: rule `01` defines the entry-first constraint this policy exempts for repeatability entries
+- [agentme-edr-028](028-ai-eval-core-standards.md) — AI eval core standards: rule `02` defines the LLM-as-judge binary output contract used by the LLM-as-judge scoring method in rule `01`
+- [agentme-edr-032](032-ai-eval-report-format.md) — AI eval report format: rule `01` defines the base report template that `report-repeatability.md` adapts
+- [agentme-edr-030](030-ai-test-types-taxonomy.md) — AI test types taxonomy: `repeatability` test type definition and disambiguation from `reproducibility` (rule `09`)
+- [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards: Workflow eval cadence (rule `09`) that repeatability runs align with

package/.xdrs/agentme/edrs/devops/005-monorepo-structure.md

@@ -53,60 +53,60 @@ Module folder responsibilities, artifact locations, and test-folder conventions
 #### 02-application-folders
 
 - Represent a cohesive unit with its own lifecycle (e.g., `mymobileapp`, `graph-visualizer`).
-- **MUST** depend only on resources in `/shared/`. Direct cross-application dependencies are forbidden; use published artifacts (container images, published libraries) instead.
-- **MUST** contain a `README.md` with: purpose, architecture overview, how to build, and how to run.
-- **MAY** contain `examples/`, `tests_integration/`, and `tests_benchmark/` when those artifacts apply to multiple modules inside the application.
+- MUST depend only on resources in `/shared/`. Direct cross-application dependencies are forbidden; use published artifacts (container images, published libraries) instead.
+- MUST contain a `README.md` with: purpose, architecture overview, how to build, and how to run.
+- MAY contain `examples/`, `tests_integration/`, and `tests_benchmark/` when those artifacts apply to multiple modules inside the application.
 
 #### 03-module-folders
 
 - A module is a subfolder inside an application that is independently compilable and produces a build artifact.
 - May depend on sibling modules within the same application or on `/shared/` resources.
-- **MUST NOT** depend on modules from other applications.
-- **MUST** contain its own `Makefile`, `README.md`, and language/tooling configuration.
-- **MUST** keep build outputs under `dist/` and persistent caches under `.cache/`, following [agentme-edr-016](../principles/016-cross-language-module-structure.md).
-- **MUST NOT** keep consumer examples inside the module folder; those belong in a sibling `examples/` folder at the nearest parent aggregation root.
+- MUST NOT depend on modules from other applications.
+- MUST contain its own `Makefile`, `README.md`, and language/tooling configuration.
+- MUST keep build outputs under `dist/` and persistent caches under `.cache/`, following [agentme-edr-016](../principles/016-cross-language-module-structure.md).
+- MUST NOT keep consumer examples inside the module folder; those belong in a sibling `examples/` folder at the nearest parent aggregation root.
 
 #### 04-naming-conventions
 
-- All folder and file names **MUST** be **lowercase**.
+- All folder and file names MUST be lowercase.
 - Use hyphens (`-`) to separate words (e.g., `data-loader`, `graph-visualizer`).
 - Avoid abbreviations unless universally understood in the domain (e.g., `cli`, `api`).
 
 #### 05-makefiles-at-every-level
 
-A `Makefile` **MUST** be present at the repository root, in every application folder, and in every module folder.
+A `Makefile` MUST be present at the repository root, in every application folder, and in every module folder.
 
-All Makefiles **MUST** use the shared target vocabulary from [agentme-edr-008](008-common-targets.md).
+All Makefiles MUST use the shared target vocabulary from [agentme-edr-008](008-common-targets.md).
 
-Repository, application, and module Makefiles **MUST** define at minimum: `all`, `build`, `lint`, `test`, and `clean`.
+Repository, application, and module Makefiles MUST define at minimum: `all`, `build`, `lint`, `test`, and `clean`.
 
-Module Makefiles **SHOULD** also provide `lint-fix` and `install` when the underlying tooling supports them.
+Module Makefiles SHOULD also provide `lint-fix` and `install` when the underlying tooling supports them.
 
-The root `Makefile` **MUST** also define a `setup` target that guides a new contributor to prepare their machine.
-The root `setup` target **MUST** run `mise install` and any small repository bootstrap required before routine targets work.
+The root `Makefile` MUST also define a `setup` target that guides a new contributor to prepare their machine.
+The root `setup` target MUST run `mise install` and any small repository bootstrap required before routine targets work.
 
 #### 06-mise-for-tooling-management
 
-- [Mise](https://mise.jdx.dev/) **MUST** be used to pin all tool versions (compilers, runtimes, CLI tools).
-- A `.mise.toml` **MUST** exist at the repository root.
-- Every language runtime or CLI referenced by any module `Makefile`, CI workflow, or README command **MUST** be pinned in `.mise.toml`.
-- Contributors and CI run `make setup` after cloning or checkout; this target must call `mise install`.
-- Agents and contributors **MUST** check `.mise.toml` before using a system-installed compiler, runtime, or CLI.
-- When `.mise.toml` exists, all build, test, lint, and code-generation commands **MUST** run through `make <target>`, and the Makefile recipes **MUST** execute the underlying tools via `mise exec -- <command>`, following [agentme-edr-017](017-tool-execution-and-scripting.md).
-- If a required tool is missing, the first remediation step **MUST** be to update `.mise.toml` or run `mise install`, not to install ad-hoc global tools with language-specific installers such as `go install`, `npm install -g`, `pip install --user`, or `cargo install`.
-- Root and module `Makefile` targets **MUST** work when invoked as plain `make <target>` after `make setup`.
+- [Mise](https://mise.jdx.dev/) MUST be used to pin all tool versions (compilers, runtimes, CLI tools).
+- A `.mise.toml` MUST exist at the repository root.
+- Every language runtime or CLI referenced by any module `Makefile`, CI workflow, or README command MUST be pinned in `.mise.toml`.
+- Contributors and CI run `make setup` after cloning or checkout; this target MUST call `mise install`.
+- Agents and contributors MUST check `.mise.toml` before using a system-installed compiler, runtime, or CLI.
+- When `.mise.toml` exists, all build, test, lint, and code-generation commands MUST run through `make <target>`, and the Makefile recipes MUST execute the underlying tools via `mise exec -- <command>`, following [agentme-edr-017](017-tool-execution-and-scripting.md).
+- If a required tool is missing, the first remediation step MUST be to update `.mise.toml` or run `mise install`, not to install ad-hoc global tools with language-specific installers such as `go install`, `npm install -g`, `pip install --user`, or `cargo install`.
+- Root and module `Makefile` targets MUST work when invoked as plain `make <target>` after `make setup`.
 
 #### 07-root-readme
 
-The root `README.md` **MUST** include: overview, machine setup, quickstart, and a repository map.
+The root `README.md` MUST include: overview, machine setup, quickstart, and a repository map.
 
 #### 08-root-gitignore
 
-The repository root **MUST** ignore `dist/` and `.cache/` so module artifacts and tool caches are never committed accidentally.
+The repository root MUST ignore `dist/` and `.cache/` so module artifacts and tool caches MUST NOT be committed accidentally.
 
 #### 09-git-tagging-and-artifact-versioning
 
-All releases **MUST** be tagged using the format `<module-name>/<semver>` (e.g., `graphvisualizer/renderer/1.0.0`, `shared/libs/mylib/2.1.0`).
+All releases MUST be tagged using the format `<module-name>/<semver>` (e.g., `graphvisualizer/renderer/1.0.0`, `shared/libs/mylib/2.1.0`).
 
 `<module-name>` is preferably the path-like identifier of the module being released. A custom name is allowed but the folder name is strongly preferred.
 

package/.xdrs/agentme/edrs/devops/006-github-pipelines.md

@@ -29,7 +29,7 @@ Separating these concerns eliminates accidental publishes from CI runs, ensures
 | `release.yml` | `workflow_dispatch` | Tag the next version using monotag |
 | `publish.yml` | `push` of tags matching `*` | Publish artifacts for the tagged version |
 
-All workflows run on `ubuntu-latest`. Tool versions MUST be managed by Mise via `jdx/mise-action`. Projects should have a .mise.toml file to configure it
+All workflows run on `ubuntu-latest`. Tool versions MUST be managed by Mise via `jdx/mise-action`. Projects SHOULD have a `.mise.toml` file to configure it
 
 ---
 
@@ -69,7 +69,7 @@ File: `.github/workflows/release.yml`
 
 Projects MUST use this manually dispatched (`workflow_dispatch`) workflow. It calculates the next semantic version tag using **monotag** and pushes that tag to the repository. Pushing the tag then automatically triggers the publish workflow.
 
-The checkout step **must** use `fetch-depth: 0` so monotag can traverse the full commit history to determine the correct next version.
+The checkout step MUST use `fetch-depth: 0` so monotag can traverse the full commit history to determine the correct next version.
 
 ```yaml
 name: release

package/.xdrs/agentme/edrs/devops/008-common-targets.md

@@ -23,18 +23,18 @@ Standardizing both the target names and the execution chain removes per-project
 
 #### 01-every-project-must-have-root-makefile
 
-The project root **MUST** contain a single authoritative `Makefile` that exposes the standard target names defined in rule 3. Developers and CI pipelines **MUST** invoke routine actions through this `Makefile`, **NEVER** by calling underlying tools directly in documentation, CI, or daily workflow commands.
+The project root MUST contain a single authoritative `Makefile` that exposes the standard target names defined in rule 3. Developers and CI pipelines MUST invoke routine actions through this `Makefile`, MUST NOT call underlying tools directly in documentation, CI, or daily workflow commands.
 
 `make <target>` is the shared contract across projects and languages.
 
-- The root `Makefile` **MUST** be the entry point for both developers and pipelines.
-- The root `Makefile` **MUST** expose at minimum the common targets defined in this XDR.
-- Reverse-compatibility wrappers are allowed when an ecosystem expects them, but they **MUST** stay trivial.
+- The root `Makefile` MUST be the entry point for both developers and pipelines.
+- The root `Makefile` MUST expose at minimum the common targets defined in this XDR.
+- Reverse-compatibility wrappers are allowed when an ecosystem expects them, but they MUST stay trivial.
 	- Allowed: `package.json` script `"test": "make test"`
 	- Not allowed: `make test` -> `npm run test` -> tool command
-- Project logic **MUST NOT** live in npm scripts, Mise tasks, shell wrappers, or other secondary runners when the same logic belongs in the `Makefile`.
+- Project logic MUST NOT live in npm scripts, Mise tasks, shell wrappers, or other secondary runners when the same logic belongs in the `Makefile`.
 
-*Why:* The project entry point **MUST** stay language-agnostic and obvious. A developer **SHOULD** be able to inspect the `Makefile` and immediately see which real tool commands will run.
+*Why:* The project entry point MUST stay language-agnostic and obvious. A developer SHOULD be able to inspect the `Makefile` and immediately see which real tool commands will run.
 
 #### 02-makefile-recipes-must-use-mise
 
@@ -47,10 +47,10 @@ make <target>
 			-> explicit tool command
 ```
 
-- The `setup` target **MUST** run `mise install` and any small project-specific bootstrap needed before normal targets work.
-- Routine targets such as `build`, `lint`, `test`, `run`, and `publish` **MUST** be invoked as `make <target>` by both contributors and CI.
-- Each Makefile recipe **MUST** call the real underlying command through `mise exec --`, following [agentme-edr-017](017-tool-execution-and-scripting.md).
-- Makefile recipes **MUST NOT** add extra script layers such as `npm run`, `pnpm run`, `yarn run`, `mise run`, `mise tasks`, or shell aliases when those layers only forward to another command.
+- The `setup` target MUST run `mise install` and any small project-specific bootstrap needed before normal targets work.
+- Routine targets such as `build`, `lint`, `test`, `run`, and `publish` MUST be invoked as `make <target>` by both contributors and CI.
+- Each Makefile recipe MUST call the real underlying command through `mise exec --`, following [agentme-edr-017](017-tool-execution-and-scripting.md).
+- Makefile recipes MUST NOT add extra script layers such as `npm run`, `pnpm run`, `yarn run`, `mise run`, `mise tasks`, or shell aliases when those layers only forward to another command.
 - Calling the actual tool is allowed even when that tool itself launches another program as part of its normal interface.
 	- Allowed: `mise exec -- pnpm exec eslint ./src`
 	- Allowed: `mise exec -- go test -cover ./...`
@@ -66,7 +66,7 @@ make <target>
 
 #### 03-standard-target-groups-and-names
 
-Targets are organized into five lifecycle groups. Projects **MUST** use these names unchanged. Extensions are allowed (see rule 5) but the core names **MUST NOT** be repurposed.
+Targets are organized into five lifecycle groups. Projects MUST use these names unchanged. Extensions are allowed (see rule 5) but the core names MUST NOT be repurposed.
 
 ##### Developer group
 
@@ -150,28 +150,6 @@ The prefix convention ensures developers can infer the purpose of any target wit
 
 ---
 
-#### 09-ai-project-dev-targets
-
-AI-based projects (LLM, Agent, and Workflow tiers as defined in [agentme-edr-018](../application/018-ai-llm-development-standards.md)) MUST expose a `dev-mlflow` target that starts a local MLflow tracking server for development inspection.
-
-**Example implementation:**
-
-```makefile
-dev-mlflow:
-	mise exec -- mlflow ui --host 0.0.0.0 --port 5000
-	open http://localhost:5000/
-```
-
----
-
-#### 08-default-targets-must-only-include-offline-subtargets
-
-`make all`, `make test`, and `make lint` **MUST** include every subtarget that runs **offline** — meaning it requires no external credentials, no running servers, no paid APIs, and no environment-specific configuration outside the repository.
-
-Subtargets that require external dependencies (e.g., `test-integration` against a live database, `test-e2e` against a staging environment, `lint-api` against a remote schema registry) **MUST** exist as named targets so developers can invoke them explicitly, but **MUST NOT** be invoked from `all`, `test`, or `lint`.
-
----
-
 #### 06-monorepo-usage
 
 In a monorepo, each module has its own `Makefile` with its own `build`, `lint`, `test`, and `deploy` targets scoped to that module. Parent-level Makefiles (at the application or repo root) delegate to child Makefiles in sequence. The parent Makefile **SHOULD** call `$(MAKE) -C <child> <target>` directly, while each child `Makefile` runs its actual tool commands through `mise exec --`.
@@ -238,6 +216,28 @@ make clean
 make all
 ```
 
+---
+
+#### 08-default-targets-must-only-include-offline-subtargets
+
+`make all`, `make test`, and `make lint` **MUST** include every subtarget that runs **offline** — meaning it requires no external credentials, no running servers, no paid APIs, and no environment-specific configuration outside the repository.
+
+Subtargets that require external dependencies (e.g., `test-integration` against a live database, `test-e2e` against a staging environment, `lint-api` against a remote schema registry) **MUST** exist as named targets so developers can invoke them explicitly, but **MUST NOT** be invoked from `all`, `test`, or `lint`.
+
+---
+
+#### 09-ai-project-dev-targets
+
+AI-based projects (LLM, Agent, and Workflow tiers as defined in [agentme-edr-018](../application/018-ai-llm-development-standards.md)) MUST expose a `dev-mlflow` target that starts a local MLflow tracking server for development inspection.
+
+**Example implementation:**
+
+```makefile
+dev-mlflow:
+	mise exec -- mlflow ui --host 0.0.0.0 --port 5000
+	open http://localhost:5000/
+```
+
 ## Considered Options
 
 * (REJECTED) **Language-native entry points only** - Use `npm run`, `python -m`, `go run`, and similar tool-specific commands directly as the standard surface

package/.xdrs/agentme/edrs/devops/017-tool-execution-and-scripting.md

@@ -24,7 +24,7 @@ This keeps local development and CI aligned, reduces indirection, and lets contr
 - Every project MUST use a root `Makefile` as the authoritative entry point for developer and pipeline commands.
 - The target names in that `Makefile` MUST follow [agentme-edr-008](008-common-targets.md).
 - CI pipelines MUST run `make <target>` from the relevant root instead of calling language-specific scripts such as `npm run`, `pnpm run`, shell wrappers, or secondary task runners.
-- A Makefile target MUST execute the real operation through `mise exec --` before invoking the tool itself, so it always uses the version pinned in `.mise.toml`. Avoid intermediate script layers that hide the actual command.
+- A Makefile target MUST execute the real operation through `mise exec --` before invoking the tool itself, so it MUST use the version pinned in `.mise.toml`. Avoid intermediate script layers that hide the actual command.
 - Every Makefile target MUST start by echoing a concise summary of the target and folder or context, using fewer than 10 words. When delegating to another Makefile, echo the child path and delegated target before invoking it.
 - Direct delegation to another Makefile is allowed when traversing repo, app, or module boundaries, for example `$(MAKE) -C lib build`.
 - Calling the actual tool binary through its native executable launcher is allowed when that is the direct command under `mise exec --`, for example `mise exec -- pnpm exec eslint ./src`, `mise exec -- uv run ty check`, `mise exec -- go test`, or `mise exec -- npx -y monotag`.

package/.xdrs/agentme/edrs/devops/027-environment-variable-configuration.md

@@ -17,9 +17,9 @@ How should projects manage environment variable configuration and CLI invocation
 
 ## Decision Outcome
 
-**Use YAML config files for CLI invocation configuration with multiple attributes; use `.env` files to supply environment variables to spawned processes and to hold uncommitted values referenced by config files. Load `.env` exclusively at process launch time — never inside application code.**
+**Use YAML config files for CLI invocation configuration with multiple attributes; use `.env` files to supply environment variables to spawned processes and to hold uncommitted values referenced by config files. Load `.env` exclusively at process launch time — MUST NOT be loaded inside application code.**
 
-Secrets (API keys, passwords, tokens) must never be placed in `.env` files. Those are handled by [agentme-edr-022](../principles/022-secrets-management.md).
+Secrets (API keys, passwords, tokens) MUST NOT be placed in `.env` files. Those are handled by [agentme-edr-022](../principles/022-secrets-management.md).
 
 ### Details
 
@@ -43,7 +43,7 @@ FEATURE_FLAG_NEW_UI=false
 
 #### 02-dotenv-not-committed
 
-`.env` MUST be listed in `.gitignore` and must never be committed to the repository. It is intended for local use in standalone projects and libraries that do not have a formal deployment pipeline.
+`.env` MUST be listed in `.gitignore` and MUST NOT be committed to the repository. It is intended for local use in standalone projects and libraries that do not have a formal deployment pipeline.
 
 ---