agentme 0.26.0 → 0.28.1

package/.xdrs/agentme/edrs/application/028-ai-eval-standards.md

@@ -1,257 +0,0 @@
----
-name: agentme-edr-policy-028-ai-eval-standards
-description: Defines how to structure, write, and run eval tests for AI projects — folder layout, golden dataset, --type test-type filtering, mock_fixtures wiring, entry-first eval loop, per-type Makefile targets and reports, and MLflow tracking. Use when implementing evals for LLM, Agent, or Workflow projects. For when evals are required see agentme-edr-007 rule 09-ai-project-testing-requirements. For the test type taxonomy and mock_fixtures envelope see agentme-edr-030. For mock file naming see agentme-edr-026 rule 10.
-apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
-valid-from: 2026-06-05
----
-
-# agentme-edr-policy-028: AI eval standards
-
-## Context and Problem Statement
-
-Eval tests measure AI component accuracy against expected outputs using real LLM providers. Without a shared folder layout and script convention, eval setups diverge across LLM, Agent, and Workflow projects, making them hard to run, compare, and integrate into CI/CD pipelines.
-
-How should eval tests be structured and run across all AI tiers?
-
-## Decision Outcome
-
-**Use a per-component folder structure under `evals/` with a standardized Makefile interface and MLflow-backed scripts, applicable to LLM, Agent, and Workflow components.**
-
-For when evals are required per AI tier, see [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
-
-### Details
-
-#### 01-eval-folder-structure
-
-Evals are grouped first by the component being evaluated, then by the specific evaluation scenario. Create one directory per component under `evals/`, and one directory per eval scenario inside it. Place `evals/` at the same level as `lib/` and `examples/`:
-
-```text
-evals/
-  <component>/           # the component being evaluated (e.g., workflow-x, agent-y, model-z)
-    eval-<name>/
-      golden_dataset/    # EDR-024 + EDR-030 compliant golden dataset (README.md, dataset.schema.json, data/)
-      eval.py            # evaluation script
-      report-<type>.md   # generated report, one per evaluated test type (overwritten on each run — see rule 03)
-      Makefile           # lint, eval, run, and eval-<type> targets
-    eval-<name2>/
-      ...
-  <component2>/
-    ...
-```
-
-`<component>` MUST match the name of the component under evaluation and use lowercase hyphen-separated words (e.g., `workflow-document-review`, `agent-support`, `model-classifier`).
-
-`<name>` identifies the specific evaluation scenario using lowercase hyphen-separated words (e.g., `eval-basic`, `eval-complex`, `eval-edge-cases`). A scenario's `golden_dataset` MAY mix multiple test types across its entries: label each entry with its applicable `test_types` ([agentme-edr-030](030-ai-test-types-taxonomy.md) rule `04`) and use the `eval-<type>` targets below to run one type at a time.
-
-The `golden_dataset/` subfolder MUST be a valid [agentme-edr-024](024-ml-dataset-structure.md) dataset (`README.md`, `dataset.schema.json`, one JSON file per entry under `data/` per rule `04-complex-structured-datasets-must-use-per-entry-json-files`, lint-validated per rule `06`) whose entries follow the golden dataset envelope defined in [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02`.
-
-Each `evals/<component>/eval-<name>/Makefile` MUST declare a `TEST_TYPES` variable listing the `test_types` values present in its golden dataset, and define:
-
-| Target | Behaviour |
-|---|---|
-| `lint` | Validates every `golden_dataset/data/*.json` file against `golden_dataset/dataset.schema.json` per [agentme-edr-024](024-ml-dataset-structure.md) rule `06` |
-| `eval` | Depends on `lint`; runs `eval.py --type=all` with threshold enforcement; exits non-zero on failure (CI-safe) |
-| `run` | Depends on `lint`; runs `eval.py --type=all` without threshold enforcement (exploration / debugging) |
-| `eval-<type>` | Depends on `lint`; runs `eval.py --type=<type>` for one declared test type, following [agentme-edr-008](../devops/008-common-targets.md) rule `03`'s `eval-<qualifier>` convention |
-
-```makefile
-TEST_TYPES := smoke functional safety
-
-lint:
-	mise exec -- uv run --project . python lint_dataset.py golden_dataset/
-
-eval: lint
-	mise exec -- uv run --project . python eval.py --type=all
-
-run: lint
-	mise exec -- uv run --project . python eval.py --type=all --no-threshold
-
-eval-%: lint
-	mise exec -- uv run --project . python eval.py --type=$*
-```
-
-The module root Makefile MUST expose `make eval` and `make lint` targets that delegate to `eval` and `lint` respectively in every `evals/<component>/eval-<name>/Makefile`:
-
-```makefile
-eval:
-	$(MAKE) -C evals/workflow-document-review/eval-basic eval
-	$(MAKE) -C evals/workflow-document-review/eval-complex eval
-
-lint:
-	$(MAKE) -C evals/workflow-document-review/eval-basic lint
-	$(MAKE) -C evals/workflow-document-review/eval-complex lint
-```
-
-#### 02-eval-script-requirements
-
-Each `eval.py` script MUST:
-
-- Load the golden dataset from `golden_dataset/` in the same eval folder, following [agentme-edr-024](024-ml-dataset-structure.md) and the entry envelope in [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02` (one JSON file per entry, `test_types` array, `input`, `expected_output`, optional `mock_fixtures`).
-- Accept a required `--type=<test_type>|all` CLI argument and filter entries whose `test_types` array contains the requested value; `--type=all` includes every entry.
-- Iterate **entry-first**: for each entry in the filtered set, invoke the real component exactly once; then score that single `actual_output` for every `test_types` value the entry carries that falls within the current `--type` scope — MUST NOT invoke the component more than once per entry per run.
-- When an entry contains `mock_fixtures` ([agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02`), configure each named mock adapter with its fixture data BEFORE invoking the component for that entry. Each entry MUST use fresh mock instances so fixture state does not bleed across entries. `mock_fixtures` applies to all test types including `human`. `mock_fixtures` MUST NOT configure LLM adapters — the LLM call MUST be real (see [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `03`). How mock adapters are discovered and instantiated is left to the project; see [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
-- Run every component invocation against **real LLM providers** (not mocked responses), to capture model drift.
-- For `human` entries: invoke the component to capture `actual_output`, export each entry's `input`, `expected_output.human_test` instructions, and `actual_output` into a manual-review checklist (`report-human.md`). MUST NOT invoke an automated scorer and MUST NOT enforce a pass/fail threshold for it. Other `test_types` on the same entry (e.g. `functional`) are still scored automatically.
-- After all entries are processed, compute aggregate metrics per test type, log them to a local MLflow experiment (see rule `04`), write one `report-<type>.md` per evaluated test type (rule `03`), and exit with a non-zero status when any metric falls below its defined threshold per [agentme-edr-007](../principles/007-project-quality-standards.md) rule `07-statistical-models-must-have-eval-targets`. The `human` type has no threshold and does not trigger a non-zero exit.
-- Compare outputs to expected values using project-defined quality thresholds per test type. Thresholds MUST be declared explicitly (e.g., in a Makefile variable or README) — this Policy does not mandate which test types a project must threshold or what value to use (see [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `06`).
-
-**Example:**
-
-```python
-import argparse
-from collections import defaultdict
-import mlflow
-from my_package.app.workflows.document_review_workflow.graph import graph
-
-EVAL_MIN_ACCURACY = {"functional": 0.85, "smoke": 0.85}
-
-parser = argparse.ArgumentParser()
-parser.add_argument("--type", required=True)
-args = parser.parse_args()
-
-entries = load_golden_dataset("golden_dataset/", test_type=args.type)  # "all" loads every entry
-resolved_types = resolve_types(args.type, entries)
-
-mlflow.set_experiment("document-review/eval-basic")
-
-with mlflow.start_run():
-    mlflow.set_tag("test_types", ",".join(sorted(resolved_types)))
-
-    results = defaultdict(list)
-
-    # Entry-first loop: invoke each entry exactly once
-    for entry in entries:
-        # Configure mock adapters from mock_fixtures before invocation
-        # (implementation left to the project — see agentme-edr-026 rule 10)
-        if entry.get("mock_fixtures"):
-            configure_mocks(entry["mock_fixtures"])  # project-defined helper
-
-        actual_output = invoke_component(entry, graph)
-
-        for test_type in [t for t in entry["test_types"] if t in resolved_types]:
-            if test_type == "human":
-                export_human_review(entry, actual_output)
-                continue
-            results[test_type].append(score(test_type, actual_output, entry["expected_output"]))
-
-    # Aggregate, report, and enforce thresholds per test type
-    for test_type in resolved_types:
-        if test_type == "human":
-            continue
-
-        accuracy = sum(results[test_type]) / len(results[test_type])
-        mlflow.log_metric(f"{test_type}_accuracy", accuracy)
-        write_eval_report(test_type, results[test_type], thresholds={"accuracy": EVAL_MIN_ACCURACY[test_type]})
-
-        if accuracy < EVAL_MIN_ACCURACY[test_type]:
-            raise SystemExit(f"Eval failed: {test_type} accuracy {accuracy:.2f} < {EVAL_MIN_ACCURACY[test_type]}")
-```
-
-#### 03-eval-report-file
-
-Each eval script MUST produce one `report-<type>.md` per evaluated test type in the same `evals/<component>/eval-<name>/` folder and overwrite each on every run — only the types included in the current `--type` invocation are (re)written; report files for other types are left untouched. The `human` type does not produce a metrics report (see below).
-
-**Generation constraint:** The report MUST be produced programmatically, reading raw metric values directly from MLflow. No LLM or generative model may write, summarize, or paraphrase any section of the report, to prevent hallucinated metric values.
-
-The report MUST follow this template:
-
-```markdown
-# Eval Report: <name> — <type>
-
-**Date:** <ISO date>
-**Dataset:** golden_dataset/
-**Script:** eval.py --type=<type>
-**Thresholds:** accuracy ≥ <value>, F1 ≥ <value>
-
-## Overall Results
-
-| Metric    | Value  | 95% CI         | Threshold | Status  |
-|-----------|--------|----------------|-----------|---------|
-| Accuracy  | <val>  | [<low>, <high>]| ≥ <thr>   | ✓/✗ PASS/FAIL |
-| F1 Score  | <val>  | —              | ≥ <thr>   | ✓/✗ PASS/FAIL |
-| Precision | <val>  | —              | —         | —       |
-| Recall    | <val>  | —              | —         | —       |
-| Samples   | <n>    | —              | —         | —       |
-
-**Overall: PASS / FAIL**
-
-## Per-item Results
-
-| ID  | Input Summary | Expected | Actual | Correct |
-|-----|---------------|----------|--------|---------|
-| 001 | <summary>     | <label>  | <label>| ✓       |
-| 002 | <summary>     | <label>  | <label>| ✗       |
-
-## Notes
-
-- <observations, failure patterns, MLflow run link>
-```
-
-**Confidence interval:** The 95% CI for accuracy MUST be computed using the **Wilson score interval** (preferred over the normal approximation for small $n$). A wide interval signals that the dataset is too small to support confident conclusions and the sample count should be increased.
-
-The Wilson score bounds at 95% confidence ($z = 1.96$) are:
-
-$$\frac{\hat{p} + \frac{z^2}{2n} \pm z\sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}$$
-
-Where $\hat{p}$ is observed accuracy and $n$ is sample count. Accuracy and F1 are required; precision and recall are recommended.
-
-**Filled-in example** (`evals/workflow-document-review/eval-basic/report-functional.md` for a document review workflow):
-
-```markdown
-# Eval Report: eval-basic — functional
-
-**Date:** 2026-06-12
-**Dataset:** golden_dataset/
-**Script:** eval.py --type=functional
-**Thresholds:** accuracy ≥ 0.85, F1 ≥ 0.80
-
-## Overall Results
-
-| Metric    | Value | 95% CI       | Threshold | Status      |
-|-----------|-------|--------------|-----------|-------------|
-| Accuracy  | 0.88  | [0.69, 0.97] | ≥ 0.85    | ✓ PASS      |
-| F1 Score  | 0.86  | —            | ≥ 0.80    | ✓ PASS      |
-| Precision | 0.89  | —            | —         | —           |
-| Recall    | 0.84  | —            | —         | —           |
-| Samples   | 25    | —            | —         | —           |
-
-**Overall: PASS**
-
-> Note: CI [0.69, 0.97] is wide — 25 samples may be insufficient for high confidence. Consider expanding the dataset.
-
-## Per-item Results
-
-| ID  | Input Summary                       | Expected | Actual   | Correct |
-|-----|--------------------------------------|----------|----------|---------|
-| 001 | Contract renewal, 3 pages, standard | approve  | approve  | ✓       |
-| 002 | NDA with unusual liability clause   | escalate | escalate | ✓       |
-| 003 | Vendor invoice, missing PO number   | reject   | reject   | ✓       |
-| 004 | Employment agreement, standard terms| approve  | approve  | ✓       |
-| 005 | Amendment with redlined IP clause   | escalate | approve  | ✗       |
-
-## Notes
-
-- Sample 005 misclassified: redlined IP clause not flagged as escalation trigger. Possible model drift.
-- MLflow run: experiment `workflow-document-review/eval-basic`, tag `test_types=functional` — view with `mlflow ui`
-```
-
-**`human` type artifact:** instead of `report-human.md` with metrics, `--type=human` produces a checklist artifact (still named `report-human.md`) listing, per entry, its `input`, `expected_output.human_test` instructions, and the captured `actual_output` — with no Overall Results table, threshold, or PASS/FAIL section, since this type MUST NOT be auto-scored.
-
-#### 04-eval-mlflow-unique-port
-
-Each `evals/<component>/eval-<name>/Makefile` MUST start its MLflow tracking server on a **unique port** to prevent conflicts when multiple eval Makefiles are run concurrently or in parallel (e.g., in CI or across multiple terminal sessions).
-
-Ports MUST be statically assigned per eval scenario (not per test type) and MUST NOT reuse the default `5000` port (reserved for `dev-mlflow` per [agentme-edr-008](../devops/008-common-targets.md) rule `09-ai-project-dev-targets`). Assign ports starting at `5100` and incrementing by 1 for each additional eval scenario across the entire project.
-
-The MLflow **experiment** is scoped to the eval scenario: `<component>/<eval-name>` (e.g. `document-review/eval-basic`). Each `mlflow.start_run()` call MUST set a `test_types` tag listing the test types evaluated in that invocation (comma-separated, e.g. `"functional,smoke"` for `--type=all`, `"smoke"` for `--type=smoke`). A remote MLflow server MUST NOT be required — all tracking is local.
-
-## References
-
-- [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09-ai-project-testing-requirements`) and statistical model eval targets (rule `07-statistical-models-must-have-eval-targets`)
-- [agentme-edr-030](030-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum, golden dataset entry envelope (including `mock_fixtures`), and mocking constraints per type
-- [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) — Rule `10`: `_mock` file naming and placement convention for mock adapters used in `mock_fixtures`
-- [agentme-edr-018](018-ai-llm-development-standards.md) — LLM development standards: LangChain framework and observability
-- [agentme-edr-019](019-ai-agents-development-standards.md) — Agent development standards
-- [agentme-edr-021](021-ai-workflow-development-standards.md) — Workflow development standards
-- [agentme-edr-024](024-ml-dataset-structure.md) — ML dataset structure, per-entry JSON format, and schema-lint validation for golden datasets
-- [agentme-edr-008](../devops/008-common-targets.md) — `eval-<qualifier>` Makefile convention (rule `03`) and Mise tool-execution flow (rule `02`)

package/.xdrs/agentme/edrs/application/030-ai-test-types-taxonomy.md

@@ -1,98 +0,0 @@
----
-name: agentme-edr-policy-030-ai-test-types-taxonomy
-description: Names AI-application test types (safety, responsible-AI, quality-eval, prompt, code-level) with their group, objective, mocking constraint, and relevance, and defines the shared "golden dataset" entry envelope that agentme-edr-028's eval tooling filters by test_types. Use when deciding which AI test types to implement or when authoring a golden dataset entry.
-apply-to: AI projects (LLM, Agent, or Workflow tier) implementing AI-specific test types beyond generic code-level unit/integration tests
-valid-from: 2026-07-05
----
-
-# agentme-edr-policy-030: AI test types taxonomy
-
-## Context and Problem Statement
-
-AI components need test types beyond generic unit/integration tests (safety, fairness, groundedness, functional accuracy, etc.). Which test types should be named, and how should their datasets and eval tooling work?
-
-## Decision Outcome
-
-**Adopt a named taxonomy of AI test types plus a shared "golden dataset" entry envelope that agentme-edr-028's eval tooling filters by `test_types`.**
-
-Each test type is named with its group, objective, mocking constraint, applicability, and relevance; every golden dataset entry is labeled with the test types it applies to.
-
-### Details
-
-#### 01-golden-dataset-concept
-
-Projects MUST use a golden dataset to test AI components. A **golden dataset** comprises all eval case entries used to test an AI component (LLM, Agent, or Workflow tier); each entry is labeled with the `test_types` (rule `04`) it applies to. It is the dataset consumed by [agentme-edr-028](028-ai-eval-standards.md) evals and stored as one JSON file per entry per [agentme-edr-024](024-ml-dataset-structure.md) rule `04`, at `evals/<component>/eval-<name>/golden_dataset/`.
-
-#### 02-golden-dataset-entry-envelope
-
-Every golden dataset entry (a JSON file in `golden_dataset/data/`) MUST have this shape, in addition to any project-specific fields:
-
-```json
-{
-  "$schema": "../dataset.schema.json",
-  "test_types": ["functional"],
-  "input": "...",
-  "expected_output": "...",
-  "mock_fixtures": {
-    "system_y": [{"123": {"name": "Flavio"}}, {"456": {"name": "Andrew"}}]
-  }
-}
-```
-
-- `test_types` — array, values MUST come from rule `04`'s enum, MUST contain at least one value. An entry MAY carry more than one value additively (e.g. `["functional", "smoke", "human"]`) — no test type excludes another.
-- `input` — for Prompt-tier components, a raw prompt string or the prompt template's input parameters object; for Agent/Workflow-tier components, the input attributes object passed to the component.
-- `expected_output` — the fields used to score the entry under each of its automated `test_types`: output attributes for an LLM-as-judge rubric, a target for vector-similarity scoring, or exact attribute values for strict comparison. When `human` is one of the entry's `test_types`, `expected_output` MUST additionally include a `human_test` string field with manual-verification instructions (e.g. `"check for ethical issues, verify record change in system X"`) — this supplements, and MUST NOT replace, the entry's automated scoring fields.
-- `mock_fixtures` — optional object; keys identify the adapter or external system to mock (SHOULD match the connector folder name under `adapters/connectors/<name>` for readability, though not enforced), values are any valid JSON interpreted by the mock implementation. When present, eval.py MUST configure each named mock adapter with its fixture data BEFORE invoking the component for that entry; each entry MUST use fresh mock instances to prevent state from bleeding across entries. `mock_fixtures` applies to all `test_types` including `human` — the component is still invoked for human entries to capture `actual_output`. `mock_fixtures` MUST NOT include keys for LLM adapters: all golden dataset test types are rated `mocks disallowed for LLM calls` (rule `03`), so the LLM call MUST be real; LLM provider mocking belongs exclusively to unit tests via [agentme-edr-018](018-ai-llm-development-standards.md) rule `04`. See [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
-- The dataset's `dataset.schema.json` MUST require `test_types`, `input`, and `expected_output`, and SHOULD declare `mock_fixtures` as optional (`"type": "object", "additionalProperties": {}`), per [agentme-edr-024](024-ml-dataset-structure.md) rule `04`.
-
-#### 03-mocks-allowed-values
-
-The taxonomy in rule `05` rates each test type using one of three values:
-
-| Value | Meaning |
-|---|---|
-| `mocks allowed` | Fully offline; fakes may replace every dependency (e.g. `FakeListChatModel` per [agentme-edr-018](018-ai-llm-development-standards.md) rule `04`). |
-| `mocks disallowed` | No mocking of any dependency — real external systems required. |
-| `mocks disallowed for LLM calls` | Tools and other external/dependency calls MAY be mocked; only the LLM call itself MUST be real for the test to be meaningful. |
-
-#### 04-test-types-enum
-
-A golden dataset entry's `test_types` array MUST only use these values: `safety`, `adversarial`, `fairness`, `bias`, `robustness`, `explainability`, `groundedness`, `functional`, `prompt`, `smoke`, `human`. These correspond to the dataset-driven rows of rule `05`. **Unit test** and **Integration test** (the two Code-level rows) are NOT part of this enum — they have no golden dataset entries and remain governed entirely by [agentme-edr-004](../principles/004-unit-test-requirements.md) and [agentme-edr-007](../principles/007-project-quality-standards.md) rule `08`.
-
-#### 05-test-type-taxonomy
-
-Test types MUST be selected from this taxonomy. Each test type is named with its group, objective, mocking constraint, applicability, and relevance:
-
-| Test Type Name | Group | Test Objective | Mocks Allowed | When to Apply | Relevance – Business | Relevance – Development Team | Priority (1-5) |
-|---|---|---|---|---|---|---|---|
-| Safety/content eval | Safety & adversarial | Detect harmful, biased, or policy-violating output | mocks disallowed for LLM calls | Any user-facing release | Avoids reputational harm; acceptable-use compliance | Automated content gate before merge/release | 5 |
-| Adversarial/red-team test | Safety & adversarial | Probe for prompt injection, jailbreaks, unsafe tool use | mocks disallowed for LLM calls | System exposes tool-invocation or agent loops | Reduces security-incident/breach liability | Finds exploitable tool-loop paths before attackers do | 5 |
-| Fairness test | Responsible AI | Verify equitable outcomes across user groups | mocks disallowed for LLM calls | Output affects decisions about individuals/groups | Regulatory requirement; protects equitable access | Surfaces uneven outcomes before release | 4 |
-| Bias test | Responsible AI | Detect skewed or stereotyped associations | mocks disallowed for LLM calls | User-facing content generation | Lowers legal/reputational exposure | Catches bias introduced by data/prompts/fine-tuning | 3 |
-| Robustness test | Responsible AI | Verify stable behavior under noisy/out-of-distribution input | mocks disallowed for LLM calls | Inputs come from untrusted/variable sources | Protects reliability/SLAs | Confirms graceful degradation, guides input validation | 3 |
-| Explainability test | Responsible AI | Verify output is justifiable with a faithful rationale | mocks disallowed for LLM calls | Output must be justified to users/auditors/regulators | Required for auditability; builds user trust | Gives rationale trace for debugging wrong answers | 2 |
-| Groundedness (RAG) eval | Quality eval | Verify the answer is supported by retrieved context | mocks disallowed for LLM calls | System uses retrieval-augmented generation | Avoids confidently-wrong answers reaching customers | Pinpoints retrieval/prompt bugs | 4 |
-| Human evaluation | Quality eval | Manually verify aspects automated scoring can't (ethics, side effects, external state) | mocks disallowed for LLM calls | Before major releases; periodic spot-check | Defensible, human-reviewed sign-off | Catches what automated metrics miss | 3 |
-| Functional eval (golden-dataset accuracy / LLM-as-judge) | Quality eval | Measure output correctness against the golden dataset | mocks disallowed for LLM calls | Required before every Workflow release ([agentme-edr-007](../principles/007-project-quality-standards.md) rule `09`); advised elsewhere | Auditable evidence of business correctness before release | Detects regressions from model/provider/prompt changes | 5 |
-| Smoke test | Quality eval | Fast pass/fail check on a small, critical subset before running fuller suites | mocks disallowed for LLM calls | Every commit/PR, before functional/responsible-AI evals run | Cheap early warning before slower evals run | Fast, cheap feedback loop | 4 |
-| Prompt regression test | Prompt/LLM | Detect behavior change when a prompt or model version changes | mocks disallowed for LLM calls | Whenever a prompt template or model version changes | Prevents shipping a worse experience via a "small" tweak | Fast check on every prompt edit | 3 |
-| Integration test | Code-level | Verify real interaction with external systems | mocks disallowed | Component depends on external systems | Reduces production outages from integration mismatches | Catches wiring bugs unit tests can't see | 2 |
-| Unit test (offline, mocked) | Code-level | Verify deterministic logic in isolation, offline | mocks allowed | Required for Workflow tier every commit ([agentme-edr-007](../principles/007-project-quality-standards.md) rule `09`) | Lowest-cost point to catch defects | Fastest, fully offline feedback on every commit | 5 |
-
-#### 06-priority-and-relevance-are-descriptive-only
-
-Priority, Relevance, and When to Apply in rule `05` are guidance for prioritization conversations — they MUST NOT be treated as mandating which test types a project must implement, nor their thresholds. [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09` remains the only tier-level testing requirement in force (Workflow unit tests + functional evals). Once a project chooses to implement and threshold a test type, [agentme-edr-028](028-ai-eval-standards.md) rule `02`'s failing-threshold behavior applies uniformly, regardless of this table's priority rating — a project may enforce fairness at 70% and functional at 90%, or skip fairness entirely; that choice is a project/business decision, not one this Policy makes.
-
-#### 07-smoke-is-distinct-from-test-smoke
-
-The `smoke` test type (surfaced as the `eval-smoke` Makefile target, a fast subset of the golden-dataset functional eval) is a different concept from [agentme-edr-008](../devops/008-common-targets.md)'s existing `test-smoke` target (a fast subset of code-level tests). Both MAY exist in the same project; teams MUST NOT conflate them.
-
-## References
-
-- [agentme-edr-024](024-ml-dataset-structure.md) — Golden dataset file layout, per-entry JSON format, `$schema` pointer, and schema-lint validation
-- [agentme-edr-028](028-ai-eval-standards.md) — Eval folder structure, `--type` filtering, per-type Makefile targets, and per-type reports that consume this taxonomy
-- [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) — Rule `10`: `_mock` file naming and placement convention for mock adapters referenced by `mock_fixtures`
-- [agentme-edr-007](../principles/007-project-quality-standards.md) — Rule `09` tier-level testing requirements (the only mandated AI testing baseline)
-- [agentme-edr-008](../devops/008-common-targets.md) — Rule `03` `eval-<qualifier>` Makefile convention; rule `03`'s `test-smoke` (distinguished in rule `07`)
-- [agentme-edr-018](018-ai-llm-development-standards.md) — LLM tier definition and mocking utilities referenced by the `mocks allowed` value
-- [agentme-edr-004](../principles/004-unit-test-requirements.md) — Unit test requirements underlying the Code-level rows