npm - agentme - Versions diffs - 0.27.0 → 0.28.1 - Mend

agentme 0.27.0 → 0.28.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (36) hide show

package/.xdrs/agentme/edrs/application/{028-ai-eval-core-standards.md → 051-ai-eval-core-standards.md} RENAMED Viewed

@@ -1,11 +1,11 @@
 ---
-name: agentme-edr-policy-028-ai-eval-core-standards
-description: Defines the eval folder structure and Makefile interface for AI projects, and the LLM-as-judge binary scoring contract applicable to all AI tiers and test types. Use when scaffolding evals or implementing an LLM judge. For eval script requirements see agentme-edr-031. For report format see agentme-edr-032. For repeatability see agentme-edr-033. For when evals are required see agentme-edr-007 rule 09-ai-project-testing-requirements. For the test type taxonomy see agentme-edr-030.
+name: agentme-edr-policy-051-ai-eval-core-standards
+description: Defines the eval folder structure and Makefile interface for AI projects, and the LLM-as-judge binary scoring contract applicable to all AI tiers and test types. Use when scaffolding evals or implementing an LLM judge. For eval script requirements see agentme-edr-053. For report format see agentme-edr-054. For repeatability see agentme-edr-055. For when evals are required see agentme-edr-007 rule 09-ai-project-testing-requirements. For the test type taxonomy see agentme-edr-052.
 apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
 valid-from: 2026-06-05
 ---
-# agentme-edr-policy-028: AI eval core standards
+# agentme-edr-policy-051: AI eval core standards
 ## Context and Problem Statement
@@ -17,7 +17,7 @@ How should eval tests be structured, and how must LLM judges produce scores acro
 **Use a per-component folder structure under `evals/` with a standardized Makefile interface, and require all LLM judges to produce binary (`0`/`1`) output compatible with classification metrics.**
-For when evals are required per AI tier, see [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
+For when evals are required per AI tier, see [agentme-edr-007](../governance/007-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
 ### Details
@@ -31,7 +31,7 @@ evals/
     eval-<name>/
       golden_dataset/    # EDR-024 + EDR-030 compliant golden dataset (README.md, dataset.schema.json, data/)
       eval.py            # evaluation script
-      report-<type>.md   # generated report, one per evaluated test type (overwritten on each run — see agentme-edr-032 rule 01)
+      report-<type>.md   # generated report, one per evaluated test type (overwritten on each run — see agentme-edr-054 rule 01)
       Makefile           # lint, eval, run, and eval-<type> targets
     eval-<name2>/
       ...
@@ -41,18 +41,18 @@ evals/
 `<component>` MUST match the name of the component under evaluation and use lowercase hyphen-separated words (e.g., `workflow-document-review`, `agent-support`, `model-classifier`).
-`<name>` identifies the specific evaluation scenario using lowercase hyphen-separated words (e.g., `eval-basic`, `eval-complex`, `eval-edge-cases`). A scenario's `golden_dataset` MAY mix multiple test types across its entries: label each entry with its applicable `test_types` ([agentme-edr-030](030-ai-test-types-taxonomy.md) rule `04`) and use the `eval-<type>` targets below to run one type at a time.
+`<name>` identifies the specific evaluation scenario using lowercase hyphen-separated words (e.g., `eval-basic`, `eval-complex`, `eval-edge-cases`). A scenario's `golden_dataset` MAY mix multiple test types across its entries: label each entry with its applicable `test_types` ([agentme-edr-052](052-ai-test-types-taxonomy.md) rule `04`) and use the `eval-<type>` targets below to run one type at a time.
-The `golden_dataset/` subfolder MUST be a valid [agentme-edr-024](024-ml-dataset-structure.md) dataset (`README.md`, `dataset.schema.json`, one JSON file per entry under `data/` per rule `04-complex-structured-datasets-must-use-per-entry-json-files`, lint-validated per rule `06`) whose entries follow the golden dataset envelope defined in [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02`.
+The `golden_dataset/` subfolder MUST be a valid [agentme-edr-050](../data/050-ml-dataset-structure.md) dataset (`README.md`, `dataset.schema.json`, one JSON file per entry under `data/` per rule `04-complex-structured-datasets-must-use-per-entry-json-files`, lint-validated per rule `06`) whose entries follow the golden dataset envelope defined in [agentme-edr-052](052-ai-test-types-taxonomy.md) rule `02`.
 Each `evals/<component>/eval-<name>/Makefile` MUST declare a `TEST_TYPES` variable listing the `test_types` values present in its golden dataset, and define:
 | Target | Behaviour |
 |---|---|
-| `lint` | Validates every `golden_dataset/data/*.json` file against `golden_dataset/dataset.schema.json` per [agentme-edr-024](024-ml-dataset-structure.md) rule `06` |
+| `lint` | Validates every `golden_dataset/data/*.json` file against `golden_dataset/dataset.schema.json` per [agentme-edr-050](../data/050-ml-dataset-structure.md) rule `06` |
 | `eval` | Depends on `lint`; runs `eval.py --type=all` with threshold enforcement; exits non-zero on failure (CI-safe) |
 | `run` | Depends on `lint`; runs `eval.py --type=all` with threshold enforcement; same as `eval` but intended for local exploration |
-| `eval-<type>` | Depends on `lint`; runs `eval.py --type=<type>` for one declared test type, following [agentme-edr-008](../devops/008-common-targets.md) rule `03`'s `eval-<qualifier>` convention |
+| `eval-<type>` | Depends on `lint`; runs `eval.py --type=<type>` for one declared test type, following [agentme-edr-008](../platform/008-common-targets.md) rule `03`'s `eval-<qualifier>` convention |
 ```makefile
 TEST_TYPES := smoke functional safety
@@ -91,9 +91,10 @@ LLM judges scoring component outputs MUST produce binary output: `0` (fail) or `
 - Judge prompts MUST instruct the model to output exactly `0` or `1`
 - Scoring logic MUST parse the response and map to binary. Ambiguous/invalid responses: score as `0` or raise error
 - Reports using LLM judges MUST use classification metrics (Accuracy, F1, Precision, Recall), not regression metrics (RMSE, R2, MAE)
+- **F1 computation:** F1 MUST be computed from a binary confusion matrix treating score `1` as the positive class. In a standard golden dataset where every entry is expected to pass, all entries are true positives or false negatives (no true negatives exist), so Recall = TP / (TP + FN) and Precision = TP / (TP + FP) where FP = entries the judge scores `1` that the entry-level expected outcome marks as expected-fail. When all entries are expected-pass, Recall = Accuracy and Precision = 1, making F1 a conservative lower-bound on accuracy. Projects with adversarial or expected-fail entries MUST annotate each golden dataset entry with its expected binary outcome (`1` = should pass, `0` = should fail) in `expected_output` to enable a correct confusion matrix.
 - Multi-class classification not supported. For multiple quality levels, use multiple binary judges (e.g., one for "factually correct", another for "tone appropriate")
-**Rationale:** Binary output makes LLM judges compatible with classification metrics infrastructure (Accuracy, F1, Wilson CI, convergence analysis) defined in [agentme-edr-032](032-ai-eval-report-format.md) rule `01`.
+**Rationale:** Binary output makes LLM judges compatible with classification metrics infrastructure (Accuracy, F1, Wilson CI, convergence analysis) defined in [agentme-edr-054](054-ai-eval-report-format.md) rule `01`.
 **Example LLM judge prompt:**
@@ -111,10 +112,10 @@ Output:
 ## References
-- [agentme-edr-031](031-ai-eval-script.md) — AI eval script: entry-first eval loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
-- [agentme-edr-032](032-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
-- [agentme-edr-033](033-ai-eval-repeatability.md) — AI eval repeatability: loop exception, scoring methods (including LLM-as-judge per rule `02`), and cadence
-- [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09`) and statistical model eval targets (rule `07`)
-- [agentme-edr-030](030-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum and golden dataset entry envelope
-- [agentme-edr-024](024-ml-dataset-structure.md) — ML dataset structure, per-entry JSON format, and schema-lint validation for golden datasets
-- [agentme-edr-008](../devops/008-common-targets.md) — `eval-<qualifier>` Makefile convention (rule `03`)
+- [agentme-edr-053](053-ai-eval-script.md) — AI eval script: entry-first eval loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
+- [agentme-edr-054](054-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
+- [agentme-edr-055](055-ai-eval-repeatability.md) — AI eval repeatability: loop exception, scoring methods (including LLM-as-judge per rule `02`), and cadence
+- [agentme-edr-007](../governance/007-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09`) and statistical model eval targets (rule `07`)
+- [agentme-edr-052](052-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum and golden dataset entry envelope
+- [agentme-edr-050](../data/050-ml-dataset-structure.md) — ML dataset structure, per-entry JSON format, and schema-lint validation for golden datasets
+- [agentme-edr-008](../platform/008-common-targets.md) — `eval-<qualifier>` Makefile convention (rule `03`)

package/.xdrs/agentme/edrs/application/{030-ai-test-types-taxonomy.md → 052-ai-test-types-taxonomy.md} RENAMED Viewed

@@ -1,11 +1,11 @@
 ---
-name: agentme-edr-policy-030-ai-test-types-taxonomy
-description: Names AI-application test types (safety, responsible-AI, quality-eval, prompt, code-level) with their group, objective, mocking constraint, and relevance, and defines the shared "golden dataset" entry envelope that agentme-edr-028's eval tooling filters by test_types. Use when deciding which AI test types to implement or when authoring a golden dataset entry.
+name: agentme-edr-policy-052-ai-test-types-taxonomy
+description: Names AI-application test types grouped as "Safety & adversarial", "Responsible AI", "Quality eval", "Prompt/LLM", and "Code-level" with their objective, mocking constraint, and relevance, and defines the shared "golden dataset" entry envelope that agentme-edr-051's eval tooling filters by test_types. Use when deciding which AI test types to implement or when authoring a golden dataset entry.
 apply-to: AI projects (LLM, Agent, or Workflow tier) implementing AI-specific test types beyond generic code-level unit/integration tests
 valid-from: 2026-07-05
 ---
-# agentme-edr-policy-030: AI test types taxonomy
+# agentme-edr-policy-052: AI test types taxonomy
 ## Context and Problem Statement
@@ -13,7 +13,7 @@ AI components need test types beyond generic unit/integration tests (safety, fai
 ## Decision Outcome
-**Adopt a named taxonomy of AI test types plus a shared "golden dataset" entry envelope that agentme-edr-028's eval tooling filters by `test_types`.**
+**Adopt a named taxonomy of AI test types plus a shared "golden dataset" entry envelope that agentme-edr-051's eval tooling filters by `test_types`.**
 Each test type is named with its group, objective, mocking constraint, applicability, and relevance; every golden dataset entry is labeled with the test types it applies to.
@@ -21,7 +21,7 @@ Each test type is named with its group, objective, mocking constraint, applicabi
 #### 01-golden-dataset-concept
-Projects MUST use a golden dataset to test AI components. A **golden dataset** comprises all eval case entries used to test an AI component (LLM, Agent, or Workflow tier); each entry is labeled with the `test_types` (rule `04`) it applies to. It is the dataset consumed by [agentme-edr-031](031-ai-eval-script.md) evals and stored as one JSON file per entry per [agentme-edr-024](024-ml-dataset-structure.md) rule `04`, at `evals/<component>/eval-<name>/golden_dataset/`.
+Projects MUST use a golden dataset to test AI components. A **golden dataset** comprises all eval case entries used to test an AI component (LLM, Agent, or Workflow tier); each entry is labeled with the `test_types` (rule `04`) it applies to. It is the dataset consumed by [agentme-edr-053](053-ai-eval-script.md) evals and stored as one JSON file per entry per [agentme-edr-050](../data/050-ml-dataset-structure.md) rule `04`, at `evals/<component>/eval-<name>/golden_dataset/`.
 #### 02-golden-dataset-entry-envelope
@@ -42,8 +42,8 @@ Every golden dataset entry (a JSON file in `golden_dataset/data/`) MUST have thi
 - `test_types` — array, values MUST come from rule `04`'s enum, MUST contain at least one value. An entry MAY carry more than one value additively (e.g. `["functional", "smoke", "human"]`) — no test type excludes another.
 - `input` — for Prompt-tier components, a raw prompt string or the prompt template's input parameters object; for Agent/Workflow-tier components, the input attributes object passed to the component.
 - `expected_output` — the fields used to score the entry under each of its automated `test_types`: output attributes for an LLM-as-judge rubric, a target for vector-similarity scoring, or exact attribute values for strict comparison. When `human` is one of the entry's `test_types`, `expected_output` MUST additionally include a `human_test` string field with manual-verification instructions (e.g. `"check for ethical issues, verify record change in system X"`) — this supplements, and MUST NOT replace, the entry's automated scoring fields.
-- `mock_fixtures` — optional object; keys identify the adapter or external system to mock (SHOULD match the connector folder name under `adapters/connectors/<name>` for readability, though not enforced), values are any valid JSON interpreted by the mock implementation. When present, eval.py MUST configure each named mock adapter with its fixture data BEFORE invoking the component for that entry; each entry MUST use fresh mock instances to prevent state from bleeding across entries. `mock_fixtures` applies to all `test_types` including `human` — the component is still invoked for human entries to capture `actual_output`. `mock_fixtures` MUST NOT include keys for LLM adapters: all golden dataset test types are rated `mocks disallowed for LLM calls` (rule `03`), so the LLM call MUST be real; LLM provider mocking belongs exclusively to unit tests via [agentme-edr-018](018-ai-llm-development-standards.md) rule `04`. See [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
-- The dataset's `dataset.schema.json` MUST require `test_types`, `input`, and `expected_output`, and SHOULD declare `mock_fixtures` as optional (`"type": "object", "additionalProperties": {}`), per [agentme-edr-024](024-ml-dataset-structure.md) rule `04`.
+- `mock_fixtures` — optional object; keys identify the adapter or external system to mock (SHOULD match the connector folder name under `adapters/connectors/<name>` for readability, though not enforced), values are any valid JSON interpreted by the mock implementation. When present, eval.py MUST configure each named mock adapter with its fixture data BEFORE invoking the component for that entry; each entry MUST use fresh mock instances to prevent state from bleeding across entries. `mock_fixtures` applies to all `test_types` including `human` — the component is still invoked for human entries to capture `actual_output`. `mock_fixtures` MUST NOT include keys for LLM adapters: all golden dataset test types are rated `mocks disallowed for LLM calls` (rule `03`), so the LLM call MUST be real; LLM provider mocking belongs exclusively to unit tests via [agentme-edr-040](040-ai-llm-development-standards.md) rule `04`. See [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
+- The dataset's `dataset.schema.json` MUST require `test_types`, `input`, and `expected_output`, and SHOULD declare `mock_fixtures` as optional (`"type": "object", "additionalProperties": {}`), per [agentme-edr-050](../data/050-ml-dataset-structure.md) rule `04`.
 #### 03-mocks-allowed-values
@@ -51,13 +51,13 @@ The taxonomy in rule `05` rates each test type using one of three values under t
 | Value | Meaning |
 |---|---|
-| `mocks allowed` | Fully offline; fakes may replace every dependency including the LLM (e.g. `FakeListChatModel` per [agentme-edr-018](018-ai-llm-development-standards.md) rule `04`). Used only for code-level unit tests. |
+| `mocks allowed` | Fully offline; fakes may replace every dependency including the LLM (e.g. `FakeListChatModel` per [agentme-edr-040](040-ai-llm-development-standards.md) rule `04`). Used only for code-level unit tests. |
 | `mocks disallowed` | No mocking of any dependency — all real external systems required. Used for integration tests. |
 | `mocks disallowed for LLM calls` | **The LLM call MUST be real; all other external dependencies (databases, APIs, external services) MAY and SHOULD be mocked via `mock_fixtures`.** `mock_fixtures` keys MUST NOT reference LLM adapters. This is the standard constraint for every golden-dataset eval test type. See rule `08` for rationale. |
 #### 04-test-types-enum
-A golden dataset entry's `test_types` array MUST only use these values: `safety`, `adversarial`, `fairness`, `bias`, `robustness`, `explainability`, `groundedness`, `functional`, `prompt`, `smoke`, `human`, `repeatability`. These correspond to the dataset-driven rows of rule `05`. **Unit test** and **Integration test** (the two Code-level rows) are NOT part of this enum — they have no golden dataset entries and remain governed entirely by [agentme-edr-004](../principles/004-unit-test-requirements.md) and [agentme-edr-007](../principles/007-project-quality-standards.md) rule `08`.
+A golden dataset entry's `test_types` array MUST only use these values: `safety`, `adversarial`, `fairness`, `bias`, `robustness`, `explainability`, `groundedness`, `functional`, `prompt`, `smoke`, `human`, `repeatability`. These correspond to the dataset-driven rows of rule `05`. **Unit test** and **Integration test** (the two Code-level rows) are NOT part of this enum — they have no golden dataset entries and remain governed entirely by [agentme-edr-004](004-unit-test-requirements.md) and [agentme-edr-007](../governance/007-project-quality-standards.md) rule `08`.
 #### 05-test-type-taxonomy
@@ -74,43 +74,43 @@ Test types MUST be selected from this taxonomy. Each test type is named with its
 | Groundedness (RAG) eval | `groundedness` | Quality eval | Verify the answer is supported by retrieved context | mocks disallowed for LLM calls | System uses retrieval-augmented generation | Avoids confidently-wrong answers reaching customers | Pinpoints retrieval/prompt bugs | 4 |
 | Repeatability test | `repeatability` | Quality eval | Verify output stability/variance across N repeated invocations of the same input under fixed configuration | mocks disallowed for LLM calls | Non-deterministic components (temperature > 0, agentic tool-selection loops, sampling-based decoding) used in decision-critical or user-facing flows | Protects against silently flaky behavior reaching production; supports consistency SLAs | Detects prompt/agent designs too sensitive to sampling noise; informs temperature/seed tuning | 3 |
 | Human evaluation | `human` | Quality eval | Manually verify aspects automated scoring can't (ethics, side effects, external state) | mocks disallowed for LLM calls | Before major releases; periodic spot-check | Defensible, human-reviewed sign-off | Catches what automated metrics miss | 3 |
-| Functional eval (golden-dataset accuracy / LLM-as-judge) | `functional` | Quality eval | Measure output correctness against the golden dataset | mocks disallowed for LLM calls | Required before every Workflow release ([agentme-edr-007](../principles/007-project-quality-standards.md) rule `09`); advised elsewhere | Auditable evidence of business correctness before release | Detects regressions from model/provider/prompt changes | 5 |
+| Functional eval (golden-dataset accuracy / LLM-as-judge) | `functional` | Quality eval | Measure output correctness against the golden dataset | mocks disallowed for LLM calls | Required before every Workflow release ([agentme-edr-007](../governance/007-project-quality-standards.md) rule `09`); advised elsewhere | Auditable evidence of business correctness before release | Detects regressions from model/provider/prompt changes | 5 |
 | Smoke test | `smoke` | Quality eval | Fast pass/fail check on a small, critical subset before running fuller suites | mocks disallowed for LLM calls | Every commit/PR, before functional/responsible-AI evals run | Cheap early warning before slower evals run | Fast, cheap feedback loop | 4 |
 | Prompt regression test | `prompt` | Prompt/LLM | Detect behavior change when a prompt or model version changes | mocks disallowed for LLM calls | Whenever a prompt template or model version changes | Prevents shipping a worse experience via a "small" tweak | Fast check on every prompt edit | 3 |
 | Integration test | n/a — code-level only (see rule `04`) | Code-level | Verify real interaction with external systems | mocks disallowed | Component depends on external systems | Reduces production outages from integration mismatches | Catches wiring bugs unit tests can't see | 2 |
-| Unit test (offline, mocked) | n/a — code-level only (see rule `04`) | Code-level | Verify deterministic logic in isolation, offline | mocks allowed | Required for Workflow tier every commit ([agentme-edr-007](../principles/007-project-quality-standards.md) rule `09`) | Lowest-cost point to catch defects | Fastest, fully offline feedback on every commit | 5 |
+| Unit test (offline, mocked) | n/a — code-level only (see rule `04`) | Code-level | Verify deterministic logic in isolation, offline | mocks allowed | Required for Workflow tier every commit ([agentme-edr-007](../governance/007-project-quality-standards.md) rule `09`) | Lowest-cost point to catch defects | Fastest, fully offline feedback on every commit | 5 |
 #### 06-priority-and-relevance-are-descriptive-only
-Priority, Relevance, and When to Apply in rule `05` are guidance for prioritization conversations — they MUST NOT be treated as mandating which test types a project must implement, nor their thresholds. [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09` remains the only tier-level testing requirement in force (Workflow unit tests + functional evals). Once a project chooses to implement and threshold a test type, [agentme-edr-031](031-ai-eval-script.md) rule `01`'s failing-threshold behavior applies uniformly, regardless of this table's priority rating — a project may enforce fairness at 70% and functional at 90%, or skip fairness entirely; that choice is a project/business decision, not one this Policy makes.
+Priority, Relevance, and When to Apply in rule `05` are guidance for prioritization conversations — they MUST NOT be treated as mandating which test types a project must implement, nor their thresholds. [agentme-edr-007](../governance/007-project-quality-standards.md) rule `09` remains the only tier-level testing requirement in force (Workflow unit tests + functional evals). Once a project chooses to implement and threshold a test type, [agentme-edr-053](053-ai-eval-script.md) rule `01`'s failing-threshold behavior applies uniformly, regardless of this table's priority rating — a project may enforce fairness at 70% and functional at 90%, or skip fairness entirely; that choice is a project/business decision, not one this Policy makes.
 #### 07-smoke-is-distinct-from-test-smoke
-The `smoke` test type (surfaced as the `eval-smoke` Makefile target, a fast subset of the golden-dataset functional eval) is a different concept from [agentme-edr-008](../devops/008-common-targets.md)'s existing `test-smoke` target (a fast subset of code-level tests). Both MAY exist in the same project; teams MUST NOT conflate them.
+The `smoke` test type (surfaced as the `eval-smoke` Makefile target, a fast subset of the golden-dataset functional eval) is a different concept from [agentme-edr-008](../platform/008-common-targets.md)'s existing `test-smoke` target (a fast subset of code-level tests). Both MAY exist in the same project; teams MUST NOT conflate them.
 #### 08-eval-mocking-constraint
-For every golden-dataset eval test type: **the LLM call MUST be real; all other external dependencies MUST be mocked via `mock_fixtures`.** This applies equally to `human` entries — the component is invoked to capture `actual_output` and external dependencies must be deterministic. `mock_fixtures` keys MUST NOT reference LLM adapters. Code-level unit tests are the correct place for fully offline, LLM-mocked testing (see [agentme-edr-018](018-ai-llm-development-standards.md) rule `04`).
+For every golden-dataset eval test type: **the LLM call MUST be real; all other external dependencies MUST be mocked via `mock_fixtures`.** This applies equally to `human` entries — the component is invoked to capture `actual_output` and external dependencies must be deterministic. `mock_fixtures` keys MUST NOT reference LLM adapters. Code-level unit tests are the correct place for fully offline, LLM-mocked testing (see [agentme-edr-040](040-ai-llm-development-standards.md) rule `04`).
 #### 09-repeatability-vs-reproducibility
 | Property | Definition | What varies | Measured by |
 |---|---|---|---|
-| **Repeatability** | Output stability across N invocations at non-zero temperature | Model sampling variance | `repeatability` test type per [agentme-edr-033](033-ai-eval-repeatability.md) |
-| **Reproducibility** | Deterministic output at temperature = 0 with fixed seed | Nothing — any variance is a config bug | Not a golden-dataset type; verified via config, documented in [agentme-edr-027](../devops/027-environment-variable-configuration.md) |
+| **Repeatability** | Output stability across N invocations at non-zero temperature | Model sampling variance | `repeatability` test type per [agentme-edr-055](055-ai-eval-repeatability.md) |
+| **Reproducibility** | Deterministic output at temperature = 0 with fixed seed | Nothing — any variance is a config bug | Not a golden-dataset type; verified via config, documented in [agentme-edr-027](../platform/027-environment-variable-configuration.md) |
 A component may satisfy reproducibility (temperature = 0) yet still need repeatability tests for its production configuration (temperature > 0). The `repeatability` test type MUST NOT be applied to components with intentionally diverse output (brainstorming, creative generation) — variance is correct behavior there.
 ## References
-- [agentme-edr-024](024-ml-dataset-structure.md) — Golden dataset file layout, per-entry JSON format, `$schema` pointer, and schema-lint validation
-- [agentme-edr-028](028-ai-eval-core-standards.md) — AI eval core standards: eval folder structure and Makefile targets (rule `01`); LLM-as-judge binary scoring contract (rule `02`)
-- [agentme-edr-031](031-ai-eval-script.md) — AI eval script: `--type` filtering, entry-first loop, `mock_fixtures`, threshold enforcement, and MLflow conventions
-- [agentme-edr-032](032-ai-eval-report-format.md) — AI eval report format: per-type `report-<type>.md` that consumes this taxonomy's test types
-- [agentme-edr-033](033-ai-eval-repeatability.md) — AI eval repeatability: `REPEAT_COUNT` loop exception, scoring constants (`EVAL_MIN_ACCURACY_REPEATABILITY`, `REPEAT_SEMANTIC_SIMILARITY_SCORE`), scoring methods, `repeatability_accuracy` MLflow metric, report shape, and run cadence
+- [agentme-edr-050](../data/050-ml-dataset-structure.md) — Golden dataset file layout, per-entry JSON format, `$schema` pointer, and schema-lint validation
+- [agentme-edr-051](051-ai-eval-core-standards.md) — AI eval core standards: eval folder structure and Makefile targets (rule `01`); LLM-as-judge binary scoring contract (rule `02`)
+- [agentme-edr-053](053-ai-eval-script.md) — AI eval script: `--type` filtering, entry-first loop, `mock_fixtures`, threshold enforcement, and MLflow conventions
+- [agentme-edr-054](054-ai-eval-report-format.md) — AI eval report format: per-type `report-<type>.md` that consumes this taxonomy's test types
+- [agentme-edr-055](055-ai-eval-repeatability.md) — AI eval repeatability: `REPEAT_COUNT` loop exception, scoring constants (`EVAL_MIN_ACCURACY_REPEATABILITY`, `REPEAT_SEMANTIC_SIMILARITY_SCORE`), scoring methods, `repeatability_accuracy` MLflow metric, report shape, and run cadence
 - [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) — Rule `10`: `_mock` file naming and placement convention for mock adapters referenced by `mock_fixtures`
-- [agentme-edr-007](../principles/007-project-quality-standards.md) — Rule `09` tier-level testing requirements (the only mandated AI testing baseline)
-- [agentme-edr-008](../devops/008-common-targets.md) — Rule `03` `eval-<qualifier>` Makefile convention; rule `03`'s `test-smoke` (distinguished in rule `07`)
-- [agentme-edr-027](../devops/027-environment-variable-configuration.md) — Environment-configuration conventions referenced in rule `09`'s reproducibility disambiguation
-- [agentme-edr-018](018-ai-llm-development-standards.md) — LLM tier definition and mocking utilities referenced by the `mocks allowed` value
-- [agentme-edr-004](../principles/004-unit-test-requirements.md) — Unit test requirements underlying the Code-level rows
+- [agentme-edr-007](../governance/007-project-quality-standards.md) — Rule `09` tier-level testing requirements (the only mandated AI testing baseline)
+- [agentme-edr-008](../platform/008-common-targets.md) — Rule `03` `eval-<qualifier>` Makefile convention; rule `03`'s `test-smoke` (distinguished in rule `07`)
+- [agentme-edr-027](../platform/027-environment-variable-configuration.md) — Environment-configuration conventions referenced in rule `09`'s reproducibility disambiguation
+- [agentme-edr-040](040-ai-llm-development-standards.md) — LLM tier definition and mocking utilities referenced by the `mocks allowed` value
+- [agentme-edr-004](004-unit-test-requirements.md) — Unit test requirements underlying the Code-level rows

package/.xdrs/agentme/edrs/application/{031-ai-eval-script.md → 053-ai-eval-script.md} RENAMED Viewed

@@ -1,11 +1,11 @@
 ---
-name: agentme-edr-policy-031-ai-eval-script
-description: Defines eval.py script requirements for AI projects — entry-first eval loop, --type test-type filtering, mock_fixtures wiring, human entries, threshold enforcement, and MLflow experiment naming and port assignment. Use when implementing eval scripts. For eval folder structure see agentme-edr-028 rule 01. For the test type taxonomy and mock_fixtures envelope see agentme-edr-030. For mock file naming see agentme-edr-026 rule 10. For report format see agentme-edr-032. For repeatability loop exception see agentme-edr-033.
+name: agentme-edr-policy-053-ai-eval-script
+description: Defines eval.py script requirements for AI projects — entry-first eval loop, --type test-type filtering, mock_fixtures wiring, human entries, threshold enforcement, and MLflow experiment naming and port assignment. Use when implementing eval scripts. For eval folder structure see agentme-edr-051 rule 01. For the test type taxonomy and mock_fixtures envelope see agentme-edr-052. For mock file naming see agentme-edr-026 rule 10. For report format see agentme-edr-054. For repeatability loop exception see agentme-edr-055.
 apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
 valid-from: 2026-07-07
 ---
-# agentme-edr-policy-031: AI eval script
+# agentme-edr-policy-053: AI eval script
 ## Context and Problem Statement
@@ -17,7 +17,7 @@ How should eval scripts load datasets, iterate entries, handle mocking, and prod
 **Use an entry-first eval loop with `--type` filtering, fresh mock isolation per entry, real LLM providers, and MLflow-backed metrics with explicit per-type thresholds.**
-For when evals are required per AI tier, see [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
+For when evals are required per AI tier, see [agentme-edr-007](../governance/007-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
 ### Details
@@ -25,14 +25,14 @@ For when evals are required per AI tier, see [agentme-edr-007](../principles/007
 Each `eval.py` script MUST:
-- Load the golden dataset from `golden_dataset/` in the same eval folder, following [agentme-edr-024](024-ml-dataset-structure.md) and the entry envelope in [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02` (one JSON file per entry, `test_types` array, `input`, `expected_output`, optional `mock_fixtures`).
+- Load the golden dataset from `golden_dataset/` in the same eval folder, following [agentme-edr-050](../data/050-ml-dataset-structure.md) and the entry envelope in [agentme-edr-052](052-ai-test-types-taxonomy.md) rule `02` (one JSON file per entry, `test_types` array, `input`, `expected_output`, optional `mock_fixtures`).
 - Accept a required `--type=<test_type>|all` CLI argument and filter entries whose `test_types` array contains the requested value; `--type=all` includes every entry.
 - Iterate **entry-first**: for each entry in the filtered set, invoke the real component exactly once; then score that single `actual_output` for every `test_types` value the entry carries that falls within the current `--type` scope — MUST NOT invoke the component more than once per entry per run.
-- When an entry contains `mock_fixtures` ([agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02`), configure each named mock adapter with its fixture data BEFORE invoking the component for that entry. Each entry MUST use fresh mock instances so fixture state does not bleed across entries. `mock_fixtures` applies to all test types including `human`. `mock_fixtures` MUST NOT configure LLM adapters — the LLM call MUST be real (see [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `03`). How mock adapters are discovered and instantiated is left to the project; see [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
+- When an entry contains `mock_fixtures` ([agentme-edr-052](052-ai-test-types-taxonomy.md) rule `02`), configure each named mock adapter with its fixture data BEFORE invoking the component for that entry. Each entry MUST use fresh mock instances so fixture state does not bleed across entries. `mock_fixtures` applies to all test types including `human`. `mock_fixtures` MUST NOT configure LLM adapters — the LLM call MUST be real (see [agentme-edr-052](052-ai-test-types-taxonomy.md) rule `03`). How mock adapters are discovered and instantiated is left to the project; see [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
 - Run every component invocation against **real LLM providers** (not mocked responses), to capture model drift.
 - For `human` entries: invoke the component to capture `actual_output`, export each entry's `input`, `expected_output.human_test` instructions, and `actual_output` into a manual-review checklist (`report-human.md`). MUST NOT invoke an automated scorer and MUST NOT enforce a pass/fail threshold for it. Other `test_types` on the same entry (e.g. `functional`) are still scored automatically.
-- After all entries are processed, compute aggregate metrics per test type, log them to a local MLflow experiment (see rule `02`), write one `report-<type>.md` per evaluated test type ([agentme-edr-032](032-ai-eval-report-format.md) rule `01`), and exit with a non-zero status when any metric falls below its defined threshold per [agentme-edr-007](../principles/007-project-quality-standards.md) rule `07-statistical-models-must-have-eval-targets`. The `human` type has no threshold and does not trigger a non-zero exit.
-- Compare outputs to expected values using project-defined quality thresholds per test type. Thresholds and all other scoring parameters MUST be declared as constants in `eval.py` — they are design decisions about what constitutes acceptable quality for the component under test, not runtime configuration, and MUST NOT be passed via Makefile variables or CLI flags. Use one of two naming conventions, chosen consistently within an `eval.py`: (a) **per-type constants** — `EVAL_MIN_<METRIC>_<TYPE>` for each test type (e.g. `EVAL_MIN_ACCURACY_FUNCTIONAL = 0.85`, `EVAL_MIN_ACCURACY_REPEATABILITY = 0.8`); or (b) **dict constant** — `EVAL_MIN_<METRIC> = {<type>: <value>}` (e.g. `EVAL_MIN_ACCURACY = {"functional": 0.85, "smoke": 0.85}`). Per-type constants are preferred when each test type has a dedicated `eval.py`; the dict form is preferred when a single `eval.py` handles multiple types. In either convention, `EVAL_MIN_ACCURACY` (as a scalar) MAY be declared as a project-wide default and MUST be used as fallback when no per-type override is defined for the current test type. This Policy does not mandate which test types a project must threshold or what value to use (see [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `06`).
+- After all entries are processed, compute aggregate metrics per test type, log them to a local MLflow experiment (see rule `02`), write one `report-<type>.md` per evaluated test type ([agentme-edr-054](054-ai-eval-report-format.md) rule `01`), and exit with a non-zero status when any metric falls below its defined threshold per [agentme-edr-007](../governance/007-project-quality-standards.md) rule `07-statistical-models-must-have-eval-targets`. The `human` type has no threshold and does not trigger a non-zero exit.
+- Compare outputs to expected values using project-defined quality thresholds per test type. Thresholds and all other scoring parameters MUST be declared as constants in `eval.py` — they are design decisions about what constitutes acceptable quality for the component under test, not runtime configuration, and MUST NOT be passed via Makefile variables or CLI flags. Use one of two naming conventions, chosen consistently within an `eval.py`: (a) **per-type constants** — `EVAL_MIN_<METRIC>_<TYPE>` for each test type (e.g. `EVAL_MIN_ACCURACY_FUNCTIONAL = 0.85`, `EVAL_MIN_ACCURACY_REPEATABILITY = 0.8`); or (b) **dict constant** — `EVAL_MIN_<METRIC> = {<type>: <value>}` (e.g. `EVAL_MIN_ACCURACY = {"functional": 0.85, "smoke": 0.85}`). Per-type constants are preferred when each test type has a dedicated `eval.py`; the dict form is preferred when a single `eval.py` handles multiple types. In either convention, `EVAL_MIN_ACCURACY` (as a scalar) MAY be declared as a project-wide default and MUST be used as fallback when no per-type override is defined for the current test type. This Policy does not mandate which test types a project must threshold or what value to use (see [agentme-edr-052](052-ai-test-types-taxonomy.md) rule `06`).
 **Example:**
@@ -120,17 +120,17 @@ with mlflow.start_run():
 Each `evals/<component>/eval-<name>/Makefile` MUST start its MLflow tracking server on a **unique port** to prevent conflicts when multiple eval Makefiles are run concurrently or in parallel (e.g., in CI or across multiple terminal sessions).
-Ports MUST be statically assigned per eval scenario (not per test type) and MUST NOT reuse the default `5000` port (reserved for `dev-mlflow` per [agentme-edr-008](../devops/008-common-targets.md) rule `09-ai-project-dev-targets`). Assign ports starting at `5100` and incrementing by 1 for each additional eval scenario across the entire project.
+Ports MUST be statically assigned per eval scenario (not per test type) and MUST NOT reuse the default `5000` port (reserved for `dev-mlflow` per [agentme-edr-008](../platform/008-common-targets.md) rule `09-ai-project-dev-targets`). Assign ports starting at `5100` and incrementing by 1 for each additional eval scenario across the entire project.
 The MLflow **experiment** is scoped to the eval scenario: `<component>/<eval-name>` (e.g. `document-review/eval-basic`). Each `mlflow.start_run()` call MUST set a `test_types` tag listing the test types evaluated in that invocation (comma-separated, e.g. `"functional,smoke"` for `--type=all`, `"smoke"` for `--type=smoke`). A remote MLflow server MUST NOT be required — all tracking is local.
 ## References
-- [agentme-edr-028](028-ai-eval-core-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract (rule `02`)
-- [agentme-edr-032](032-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
-- [agentme-edr-033](033-ai-eval-repeatability.md) — AI eval repeatability: loop exception to rule `01`'s entry-first constraint, scoring methods, and cadence
-- [agentme-edr-030](030-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum, golden dataset entry envelope (including `mock_fixtures`), and mocking constraints per type
+- [agentme-edr-051](051-ai-eval-core-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract (rule `02`)
+- [agentme-edr-054](054-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
+- [agentme-edr-055](055-ai-eval-repeatability.md) — AI eval repeatability: loop exception to rule `01`'s entry-first constraint, scoring methods, and cadence
+- [agentme-edr-052](052-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum, golden dataset entry envelope (including `mock_fixtures`), and mocking constraints per type
 - [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) — Rule `10`: `_mock` file naming and placement convention for mock adapters used in `mock_fixtures`
-- [agentme-edr-024](024-ml-dataset-structure.md) — ML dataset structure, per-entry JSON format, and schema-lint validation for golden datasets
-- [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09`) and statistical model eval targets (rule `07`)
-- [agentme-edr-008](../devops/008-common-targets.md) — `eval-<qualifier>` Makefile convention (rule `03`) and reserved MLflow port `5000` (rule `09`)
+- [agentme-edr-050](../data/050-ml-dataset-structure.md) — ML dataset structure, per-entry JSON format, and schema-lint validation for golden datasets
+- [agentme-edr-007](../governance/007-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09`) and statistical model eval targets (rule `07`)
+- [agentme-edr-008](../platform/008-common-targets.md) — `eval-<qualifier>` Makefile convention (rule `03`) and reserved MLflow port `5000` (rule `09`)

package/.xdrs/agentme/edrs/application/{032-ai-eval-report-format.md → 054-ai-eval-report-format.md} RENAMED Viewed

@@ -1,11 +1,11 @@
 ---
-name: agentme-edr-policy-032-ai-eval-report-format
-description: Defines the eval report format for AI projects — report-<type>.md template, Wilson score confidence interval, convergence analysis, and the human-type checklist artifact. Use when generating eval reports from eval scripts. For the eval script that produces these reports see agentme-edr-031. For repeatability report shape see agentme-edr-033 rule 02.
+name: agentme-edr-policy-054-ai-eval-report-format
+description: Defines the eval report format for AI projects — report-<type>.md template, Wilson score confidence interval, convergence analysis, and the human-type checklist artifact. Use when generating eval reports from eval scripts. For the eval script that produces these reports see agentme-edr-053. For repeatability report shape see agentme-edr-055 rule 02.
 apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
 valid-from: 2026-07-07
 ---
-# agentme-edr-policy-032: AI eval report format
+# agentme-edr-policy-054: AI eval report format
 ## Context and Problem Statement
@@ -166,6 +166,6 @@ xychart-beta
 ## References
-- [agentme-edr-031](031-ai-eval-script.md) — AI eval script: the script that produces these reports (rule `01`)
-- [agentme-edr-033](033-ai-eval-repeatability.md) — AI eval repeatability: rule `02` defines the adapted report shape for `report-repeatability.md`
-- [agentme-edr-028](028-ai-eval-core-standards.md) — AI eval core standards: folder structure (rule `01`) and LLM-as-judge binary scoring (rule `02`)
+- [agentme-edr-053](053-ai-eval-script.md) — AI eval script: the script that produces these reports (rule `01`)
+- [agentme-edr-055](055-ai-eval-repeatability.md) — AI eval repeatability: rule `02` defines the adapted report shape for `report-repeatability.md`
+- [agentme-edr-051](051-ai-eval-core-standards.md) — AI eval core standards: folder structure (rule `01`) and LLM-as-judge binary scoring (rule `02`)

package/.xdrs/agentme/edrs/application/{033-ai-eval-repeatability.md → 055-ai-eval-repeatability.md} RENAMED Viewed

@@ -1,11 +1,11 @@
 ---
-name: agentme-edr-policy-033-ai-eval-repeatability
-description: Defines the repeatability test type for AI evals — REPEAT_COUNT loop exception to the entry-first constraint, semantic-similarity and LLM-as-judge scoring methods, repeatability_accuracy metric, and the repeatability report format and run cadence. Use when implementing repeatability evals. For the entry-first constraint see agentme-edr-031 rule 01. For LLM-as-judge binary output see agentme-edr-028 rule 02. For the base report template see agentme-edr-032 rule 01. For the repeatability test type definition see agentme-edr-030.
+name: agentme-edr-policy-055-ai-eval-repeatability
+description: Defines the repeatability test type for AI evals — REPEAT_COUNT loop exception to the entry-first constraint, semantic-similarity and LLM-as-judge scoring methods, repeatability_accuracy metric, and the repeatability report format and run cadence. Use when implementing repeatability evals. For the entry-first constraint see agentme-edr-053 rule 01. For LLM-as-judge binary output see agentme-edr-051 rule 02. For the base report template see agentme-edr-054 rule 01. For the repeatability test type definition see agentme-edr-052.
 apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement repeatability eval testing
 valid-from: 2026-07-07
 ---
-# agentme-edr-policy-033: AI eval repeatability
+# agentme-edr-policy-055: AI eval repeatability
 ## Context and Problem Statement
@@ -17,13 +17,13 @@ How should repeatability be measured in evals, and how should its results be rep
 **Exempt `repeatability` entries from the entry-first constraint, invoking each `REPEAT_COUNT` times and scoring via semantic-similarity or LLM-as-judge; schedule at release cadence rather than per-commit.**
-For the `repeatability` test type definition and its disambiguation from `reproducibility`, see [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `09`.
+For the `repeatability` test type definition and its disambiguation from `reproducibility`, see [agentme-edr-052](052-ai-test-types-taxonomy.md) rule `09`.
 ### Details
 #### 01-repeatability-eval-loop-exception
-Entries whose `test_types` includes `repeatability` are exempt from [agentme-edr-031](031-ai-eval-script.md) rule `01`'s "invoke exactly once per entry" constraint. The following constants MUST be declared in `eval.py` and MUST NOT be exposed as Makefile variables, CLI flags, or stored as per-entry dataset fields:
+Entries whose `test_types` includes `repeatability` are exempt from [agentme-edr-053](053-ai-eval-script.md) rule `01`'s "invoke exactly once per entry" constraint. The following constants MUST be declared in `eval.py` and MUST NOT be exposed as Makefile variables, CLI flags, or stored as per-entry dataset fields:
 - `REPEAT_COUNT` — number of times each repeatability entry is invoked. SHOULD default to 3-5 for routine CI runs and 10-20 for focused passes on decision-critical or previously-flagged components. Projects SHOULD calibrate the value once per component by plotting cumulative pass rate against repeat count for a few representative entries and picking the point where it plateaus, rather than guessing.
 - `EVAL_MIN_ACCURACY_REPEATABILITY` — minimum fraction of repeatability entries that must PASS for the eval to exit 0.
@@ -34,7 +34,7 @@ Entries whose `test_types` includes `repeatability` are exempt from [agentme-edr
 **Choosing the scoring method:** Two approaches are supported, declared as a constant in `eval.py`:
 - **Semantic-similarity:** Embed all `REPEAT_COUNT` outputs for an entry into vectors and compute the average pairwise cosine similarity. The entry passes (score = 1) if the average similarity ≥ `REPEAT_SEMANTIC_SIMILARITY_SCORE`; otherwise it fails (score = 0). Use for classification or short structured outputs.
-- **LLM-as-judge:** Provide all `REPEAT_COUNT` outputs for an entry to an LLM judge (kept at low/zero temperature) that returns 0 (fail) or 1 (pass) directly, based on whether the outputs are sufficiently consistent. No `REPEAT_SEMANTIC_SIMILARITY_SCORE` constant is needed. Use for free-text or complex structured outputs where vector distance is an unreliable proxy for agreement. The judge MUST follow [agentme-edr-028](028-ai-eval-core-standards.md) rule `02`'s binary output contract.
+- **LLM-as-judge:** Provide all `REPEAT_COUNT` outputs for an entry to an LLM judge (kept at low/zero temperature) that returns 0 (fail) or 1 (pass) directly, based on whether the outputs are sufficiently consistent. No `REPEAT_SEMANTIC_SIMILARITY_SCORE` constant is needed. Use for free-text or complex structured outputs where vector distance is an unreliable proxy for agreement. The judge MUST follow [agentme-edr-051](051-ai-eval-core-standards.md) rule `02`'s binary output contract.
 **`repeatability_accuracy`:** the fraction of repeatability entries that received PASS (score = 1), logged to MLflow as `repeatability_accuracy`. The eval exits non-zero if `repeatability_accuracy` < `EVAL_MIN_ACCURACY_REPEATABILITY`. Both `repeatability_accuracy` and `repeat_count` MUST be logged to MLflow and included in `report-repeatability.md` (rule `02`).
@@ -56,20 +56,20 @@ if repeatability_accuracy < EVAL_MIN_ACCURACY_REPEATABILITY:
     raise SystemExit(f"Eval failed: repeatability_accuracy {repeatability_accuracy:.2f} < {EVAL_MIN_ACCURACY_REPEATABILITY}")
 ```
-`mock_fixtures` configuration per [agentme-edr-031](031-ai-eval-script.md) rule `01` applies to each of the `REPEAT_COUNT` invocations. Any prompt or response caching (provider-side or gateway-side) MUST be bypassed for these invocations — a cache hit would return an identical cached response and falsely report perfect stability instead of measuring the model's actual variance.
+`mock_fixtures` configuration per [agentme-edr-053](053-ai-eval-script.md) rule `01` applies to each of the `REPEAT_COUNT` invocations. Any prompt or response caching (provider-side or gateway-side) MUST be bypassed for these invocations — a cache hit would return an identical cached response and falsely report perfect stability instead of measuring the model's actual variance.
 **Scoping:** this test type MUST NOT be applied to components whose intended behavior is diverse or creative output (e.g. brainstorming, creative writing) — low agreement there is correct behavior, not a defect.
 #### 02-repeatability-report-and-cadence
-`--type=repeatability` MUST produce `report-repeatability.md` with a shape adapted from [agentme-edr-032](032-ai-eval-report-format.md) rule `01`'s template: the header MUST include a **Repeat count:** line stating the `REPEAT_COUNT` value used for the run, alongside the usual Date/Dataset/Script/Thresholds lines. The body MUST have an aggregate row reporting `repeatability_accuracy` (the fraction of entries that PASS — see rule `01`) with a Wilson score interval computed over the number of `repeatability` entries, plus a per-item table listing each entry's individual pass/fail result and, when using semantic-similarity, its computed average pairwise cosine similarity — instead of the `Expected | Actual | Correct` columns used by other types.
+`--type=repeatability` MUST produce `report-repeatability.md` with a shape adapted from [agentme-edr-054](054-ai-eval-report-format.md) rule `01`'s template: the header MUST include a **Repeat count:** line stating the `REPEAT_COUNT` value used for the run, alongside the usual Date/Dataset/Script/Thresholds lines. The body MUST have an aggregate row reporting `repeatability_accuracy` (the fraction of entries that PASS — see rule `01`) with a Wilson score interval computed over the number of `repeatability` entries, plus a per-item table listing each entry's individual pass/fail result and, when using semantic-similarity, its computed average pairwise cosine similarity — instead of the `Expected | Actual | Correct` columns used by other types.
-Because `repeatability` entries multiply real LLM-provider calls by `REPEAT_COUNT`, projects SHOULD schedule `make eval-repeatability` at release cadence rather than on every commit, aligned with the Workflow eval cadence in [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09`, rather than treating it as a mandatory per-commit gate.
+Because `repeatability` entries multiply real LLM-provider calls by `REPEAT_COUNT`, projects SHOULD schedule `make eval-repeatability` at release cadence rather than on every commit, aligned with the Workflow eval cadence in [agentme-edr-007](../governance/007-project-quality-standards.md) rule `09`, rather than treating it as a mandatory per-commit gate.
 ## References
-- [agentme-edr-031](031-ai-eval-script.md) — AI eval script: rule `01` defines the entry-first constraint this policy exempts for repeatability entries
-- [agentme-edr-028](028-ai-eval-core-standards.md) — AI eval core standards: rule `02` defines the LLM-as-judge binary output contract used by the LLM-as-judge scoring method in rule `01`
-- [agentme-edr-032](032-ai-eval-report-format.md) — AI eval report format: rule `01` defines the base report template that `report-repeatability.md` adapts
-- [agentme-edr-030](030-ai-test-types-taxonomy.md) — AI test types taxonomy: `repeatability` test type definition and disambiguation from `reproducibility` (rule `09`)
-- [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards: Workflow eval cadence (rule `09`) that repeatability runs align with
+- [agentme-edr-053](053-ai-eval-script.md) — AI eval script: rule `01` defines the entry-first constraint this policy exempts for repeatability entries
+- [agentme-edr-051](051-ai-eval-core-standards.md) — AI eval core standards: rule `02` defines the LLM-as-judge binary output contract used by the LLM-as-judge scoring method in rule `01`
+- [agentme-edr-054](054-ai-eval-report-format.md) — AI eval report format: rule `01` defines the base report template that `report-repeatability.md` adapts
+- [agentme-edr-052](052-ai-test-types-taxonomy.md) — AI test types taxonomy: `repeatability` test type definition and disambiguation from `reproducibility` (rule `09`)
+- [agentme-edr-007](../governance/007-project-quality-standards.md) — Project quality standards: Workflow eval cadence (rule `09`) that repeatability runs align with

package/.xdrs/agentme/edrs/application/skills/004-select-relevant-xdrs/SKILL.md CHANGED Viewed

@@ -61,8 +61,8 @@ extraction.
 2. Reduce that set only by excluding the XDRs that clearly do not fit the repository. Use
    path-stable identifiers so the extraction command is auditable, for example:
    - `.xdrs/agentme/edrs/application/010-golang-project-tooling.md` for non-Go projects
-   - `.xdrs/agentme/edrs/devops/005-monorepo-structure.md` for non-monorepos
-   - `.xdrs/agentme/edrs/observability/011-service-health-check-endpoint.md` for projects without
+   - `.xdrs/agentme/edrs/platform/005-monorepo-structure.md` for non-monorepos
+   - `.xdrs/agentme/edrs/operations/011-service-health-check-endpoint.md` for projects without
      a long-running service surface
 3. If the repository does not want agentme XDRs, stop and explain why instead of forcing an
    installation.
@@ -109,8 +109,8 @@ extraction.
 Input: "Install the right agentme XDR presets for this Node.js library"
 - Inventory the shipped agentme XDR files
 - Analyze the repository and detect a JavaScript library with Makefiles
-- Exclude `.xdrs/agentme/edrs/application/010-golang-project-tooling.md` and `.xdrs/agentme/edrs/observability/011-service-health-check-endpoint.md`
-- Run `npx -y agentme extract --output . --all --exclude .xdrs/agentme/edrs/application/010-golang-project-tooling.md --exclude .xdrs/agentme/edrs/observability/011-service-health-check-endpoint.md`
+- Exclude `.xdrs/agentme/edrs/application/010-golang-project-tooling.md` and `.xdrs/agentme/edrs/operations/011-service-health-check-endpoint.md`
+- Run `npx -y agentme extract --output . --all --exclude .xdrs/agentme/edrs/application/010-golang-project-tooling.md --exclude .xdrs/agentme/edrs/operations/011-service-health-check-endpoint.md`
 Input: "Set up agentme for this repo"
 - Inventory the shipped agentme XDR files
@@ -135,7 +135,7 @@ Input: "Set up agentme for this repo"
 - [agentme README](../../../../../../README.md)
 - [agentme-edr-003 - JavaScript project tooling and structure](../../003-javascript-project-tooling.md)
-- [agentme-edr-005 - Monorepo structure](../../../devops/005-monorepo-structure.md)
-- [agentme-edr-007 - Project quality standards](../../../principles/007-project-quality-standards.md)
-- [agentme-edr-008 - Common development script names](../../../devops/008-common-targets.md)
+- [agentme-edr-005 - Monorepo structure](../../../platform/005-monorepo-structure.md)
+- [agentme-edr-007 - Project quality standards](../../../governance/007-project-quality-standards.md)
+- [agentme-edr-008 - Common development script names](../../../platform/008-common-targets.md)
 - [_core-adr-003 - Skill standards](../../../../../_core/adrs/principles/003-skill-standards.md)

package/.xdrs/agentme/edrs/{application/024-ml-dataset-structure.md → data/050-ml-dataset-structure.md} RENAMED Viewed

@@ -1,11 +1,11 @@
 ---
-name: agentme-edr-policy-024-ml-dataset-structure
+name: agentme-edr-policy-050-ml-dataset-structure
 description: Defines the standard folder layout and file conventions for ML datasets used in AI/ML projects. Use when creating, organizing, or consuming datasets for machine learning tasks such as image labeling, document extraction, tabular data, LLM evaluation, and Q&A sets.
 apply-to: ML and AI projects that produce or consume datasets
 valid-from: 2026-05-27
 ---
-# agentme-edr-policy-024: ML dataset structure
+# agentme-edr-policy-050: ML dataset structure
 ## Context and Problem Statement
@@ -102,10 +102,10 @@ Every dataset MUST expose a `make lint` target (in the Makefile of the project/c
 - Per-entry JSON files (rule `04`) and annotation-pair `.json` siblings (rule `02`) MUST each be validated against `dataset.schema.json`, and their `$schema` property MUST be present and resolve to the dataset's actual schema file.
 - CSV rows (rule `03`) MUST each be converted to a JSON object (column header → value) and validated against the same `dataset.schema.json`.
 - `make lint` MUST list every violation found across all files/rows before exiting with a non-zero status (not fail-fast on the first violation).
-- `jsonschema` MUST be declared as a normal project dependency per [agentme-edr-014](014-python-project-tooling.md); no special-casing.
+- `jsonschema` MUST be declared as a normal project dependency per [agentme-edr-014](../application/014-python-project-tooling.md); no special-casing.
 ## References
 - [JSON Schema specification](https://json-schema.org/)
 - [jsonschema (Python library)](https://pypi.org/project/jsonschema/)
-- [agentme-edr-014](014-python-project-tooling.md) — Python project tooling and dependency conventions
+- [agentme-edr-014](../application/014-python-project-tooling.md) — Python project tooling and dependency conventions

package/.xdrs/agentme/edrs/{principles → governance}/007-project-quality-standards.md RENAMED Viewed

@@ -61,7 +61,7 @@ A unit test suite MUST run automatically before every release. Failing tests MUS
 **Exception:** Projects with fewer than 100 lines of code, or whose `README.md` prominently marks them as a **Spike** or **Experiment**, are exempt from this requirement. Such projects MUST NOT be deployed to production.
-**Reference:** [agentme-edr-004](004-unit-test-requirements.md) for detailed unit test requirements.
+**Reference:** [agentme-edr-004](../application/004-unit-test-requirements.md) for detailed unit test requirements.
 ---
@@ -156,7 +156,7 @@ test-examples:
 	$(MAKE) -C examples
 ```
-If examples require live services or credentials, remove `test-examples` from the `test` dependency list and keep it as a standalone named target only. See [agentme-edr-008](../devops/008-common-targets.md) rule 08 for the full offline/online decision table.
+If examples require live services or credentials, remove `test-examples` from the `test` dependency list and keep it as a standalone named target only. See [agentme-edr-008](../platform/008-common-targets.md) rule 08 for the full offline/online decision table.
 **Examples Makefile:**
@@ -175,7 +175,7 @@ Projects that contain statistical models (e.g., ML models, LLM-based evaluators,
 **Requirements:**
 - A `make eval` target MUST exist and execute all performance evaluations
 - Each evaluation MUST have a **documented minimum performance threshold** (e.g., accuracy ≥ 0.85, F1 ≥ 0.80, BLEU ≥ 0.70)
-- Thresholds and all scoring parameters MUST be declared as constants in `eval.py` — they are design decisions about acceptable quality for the component under test, not runtime configuration, and MUST NOT be passed as Makefile variables or CLI flags. See [agentme-edr-031](../application/031-ai-eval-script.md) rule `01`.
+- Thresholds and all scoring parameters MUST be declared as constants in `eval.py` — they are design decisions about acceptable quality for the component under test, not runtime configuration, and MUST NOT be passed as Makefile variables or CLI flags. See [agentme-edr-053](../application/053-ai-eval-script.md) rule `01`.
 - `make eval` MUST **exit with a non-zero status** (fail) if:
   - The evaluation cannot be executed (missing data, environment errors, model load failures)
   - Any metric falls below its defined minimum threshold
@@ -244,27 +244,27 @@ Projects are not required to implement integration tests, but when present, they
 #### 09-ai-project-testing-requirements
-AI projects are classified into three tiers — LLM, Agent, and Workflow — defined in [agentme-edr-018](../application/018-ai-llm-development-standards.md). Testing requirements differ per tier:
+AI projects are classified into three tiers — LLM, Agent, and Workflow — defined in [agentme-edr-040](../application/040-ai-llm-development-standards.md). Testing requirements differ per tier:
 | Tier | Unit tests | Evals | Integration tests |
 |---|---|---|---|
-| **LLM** ([agentme-edr-018](../application/018-ai-llm-development-standards.md)) | Not required | Not required; SHOULD be used when critical prompts are in use to measure accuracy and detect model drift | Not required |
-| **Agent** ([agentme-edr-019](../application/019-ai-agents-development-standards.md)) | Not required | Not required; MAY be used | Not required |
-| **Workflow** ([agentme-edr-021](../application/021-ai-workflow-development-standards.md)) | **Required** — see below | **Required** before every release; failed evals block release | Advised |
+| **LLM** ([agentme-edr-040](../application/040-ai-llm-development-standards.md)) | Not required | Not required; SHOULD be used when critical prompts are in use to measure accuracy and detect model drift | Not required |
+| **Agent** ([agentme-edr-041](../application/041-ai-agents-development-standards.md)) | Not required | Not required; MAY be used | Not required |
+| **Workflow** ([agentme-edr-043](../application/043-ai-workflow-development-standards.md)) | Required — see below | Required before every release; failed evals block release | Advised |
 **Workflow unit test requirements:**
-- MUST use mocked LLM providers. See [agentme-edr-018](../application/018-ai-llm-development-standards.md) rule `04-unit-test-mocking` for the mocking pattern.
-- MUST run offline with no external dependencies per [agentme-edr-004](004-unit-test-requirements.md) rule `02-must-run-offline`.
-- MUST achieve 80% code coverage per [agentme-edr-004](004-unit-test-requirements.md) rule `03-must-maintain-80-percent-coverage`.
+- MUST use mocked LLM providers. See [agentme-edr-040](../application/040-ai-llm-development-standards.md) rule `04-unit-test-mocking` for the mocking pattern.
+- MUST run offline with no external dependencies per [agentme-edr-004](../application/004-unit-test-requirements.md) rule `02-must-run-offline`.
+- MUST achieve 80% code coverage per [agentme-edr-004](../application/004-unit-test-requirements.md) rule `03-must-maintain-80-percent-coverage`.
 - MUST test workflow routing logic, conditional edges, state transformations, and error handling.
 - MUST achieve **80% coverage of LangGraph graph edges and branches**: every conditional edge MUST have test cases covering each possible branch, and every node→node transition MUST be exercised by at least one test.
-- Files MUST be named `<name>_test.py` and placed alongside the source file per [agentme-edr-004](004-unit-test-requirements.md) rule `04-must-place-test-files-alongside-source`.
+- Files MUST be named `<name>_test.py` and placed alongside the source file per [agentme-edr-004](../application/004-unit-test-requirements.md) rule `04-must-place-test-files-alongside-source`.
 **Workflow eval requirements:**
 - Evals MUST be executed before every release.
 - Accuracy below project-defined thresholds MUST block the release. Thresholds MUST be documented in the eval Makefile or README.
 - Evals MUST run against real LLM providers (not mocks) to capture model drift.
-- For eval folder structure and LLM-as-judge scoring, see [agentme-edr-028](../application/028-ai-eval-core-standards.md). For eval script requirements, see [agentme-edr-031](../application/031-ai-eval-script.md).
-- For the taxonomy of AI test types (safety, responsible-AI, quality-eval, prompt, code-level) and the golden dataset entry format, see [agentme-edr-030](../application/030-ai-test-types-taxonomy.md).
+- For eval folder structure and LLM-as-judge scoring, see [agentme-edr-051](../application/051-ai-eval-core-standards.md). For eval script requirements, see [agentme-edr-053](../application/053-ai-eval-script.md).
+- For the taxonomy of AI test types (safety, responsible-AI, quality-eval, prompt, code-level) and the golden dataset entry format, see [agentme-edr-052](../application/052-ai-test-types-taxonomy.md).

package/.xdrs/agentme/edrs/governance/013-contributing-guide-requirements.md CHANGED Viewed

@@ -66,5 +66,5 @@ The content SHOULD stay concise and practical; do not turn `CONTRIBUTING.md` int
 ## References
-- [agentme-edr-005 - Monorepo structure](../devops/005-monorepo-structure.md)
-- [002-monorepo-setup skill](../devops/skills/002-monorepo-setup/SKILL.md)
+- [agentme-edr-005 - Monorepo structure](../platform/005-monorepo-structure.md)
+- [002-monorepo-setup skill](../platform/skills/002-monorepo-setup/SKILL.md)