agentme 0.32.0 → 0.33.1

package/.xdrs/agentme/edrs/application/151-ai-eval-standards.md

@@ -1,6 +1,6 @@
 ---
 name: agentme-edr-policy-151-ai-eval-standards
-description: Defines the eval folder structure and Makefile interface for AI projects, and the LLM-as-judge binary scoring contract applicable to all AI tiers and test types. Use when scaffolding evals or implementing an LLM judge. For eval script requirements see agentme-edr-153. For report format see agentme-edr-154. For repeatability see agentme-edr-155. For fairness/bias group scoring see agentme-edr-156. For when evals are required see agentme-edr-501 rule 09-ai-project-testing-requirements. For the test type taxonomy see agentme-edr-152.
+description: Defines the eval folder structure and Makefile interface for AI projects, and the LLM-as-judge binary scoring contract applicable to all AI tiers and test types. Use when scaffolding evals or implementing an LLM judge. For eval script requirements see agentme-edr-153. For report format see agentme-edr-154. For repeatability see agentme-edr-155. For fairness per-entry scoring and bias group scoring see agentme-edr-156. For when evals are required see agentme-edr-501 rule 09-ai-project-testing-requirements. For the test type taxonomy see agentme-edr-152.
 apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
 valid-from: 2026-06-05
 ---
@@ -115,7 +115,7 @@ Output:
 - [agentme-edr-153](153-ai-eval-script.md) — AI eval script: entry-first eval loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
 - [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
 - [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: loop exception, scoring methods (including LLM-as-judge per rule `02`), and cadence
-- [agentme-edr-156](156-ai-eval-fairness-bias.md) — AI eval fairness/bias: comparison_group group-scoring loop, scoring approaches (including LLM-as-judge per rule `02`), fairness_accuracy/bias_accuracy metrics, and report shape
+- [agentme-edr-156](156-ai-eval-fairness-bias.md) — AI eval fairness and bias: fairness per-entry inline scoring; bias `bias_group` deferred group-scoring loop, scoring approaches (including LLM-as-judge per rule `02`), fairness_accuracy/bias_accuracy metrics, and report shapes
 - [agentme-edr-501](../governance/501-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09`) and statistical model eval targets (rule `07`)
 - [agentme-edr-152](152-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum and golden dataset entry envelope
 - [agentme-edr-201](../data/201-ml-dataset-structure.md) — ML dataset structure, per-entry JSON format, and schema-lint validation for golden datasets

package/.xdrs/agentme/edrs/application/152-ai-test-types-taxonomy.md

@@ -1,6 +1,6 @@
 ---
 name: agentme-edr-policy-152-ai-test-types-taxonomy
-description: Names AI-application test types grouped as "Safety & adversarial", "Responsible AI", "Quality eval", "Prompt/LLM", and "Code-level" with their objective, mocking constraint, and relevance, and defines the shared "golden dataset" entry envelope that agentme-edr-151's eval tooling filters by test_types. Use when deciding which AI test types to implement or when authoring a golden dataset entry.
+description: Names AI test types with objective, mocking constraint, and relevance; defines the golden dataset entry envelope for agentme-edr-151's eval tooling; covers three test scopes (Workflow/Agent/LLM) and a quality-progression DAG. Use when deciding which test types to implement, authoring a golden dataset entry, or selecting a test tier.
 apply-to: AI projects (LLM, Agent, or Workflow tier) implementing AI-specific test types beyond generic code-level unit/integration tests
 valid-from: 2026-07-05
 ---
@@ -15,13 +15,13 @@ AI components need test types beyond generic unit/integration tests (safety, fai
 
 **Adopt a named taxonomy of AI test types plus a shared "golden dataset" entry envelope that agentme-edr-151's eval tooling filters by `test_types`.**
 
-Each test type is named with its group, objective, mocking constraint, applicability, and relevance; every golden dataset entry is labeled with the test types it applies to.
+Each test type is named with its objective, mocking constraint, applicability, and relevance; every golden dataset entry is labeled with the test types it applies to.
 
 ### Details
 
 #### 01-golden-dataset-concept
 
-Projects MUST use a golden dataset to test AI components. A **golden dataset** comprises all eval case entries used to test an AI component (LLM, Agent, or Workflow tier); each entry is labeled with the `test_types` (rule `04`) it applies to. It is the dataset consumed by [agentme-edr-153](153-ai-eval-script.md) evals and stored as one JSON file per entry per [agentme-edr-201](../data/201-ml-dataset-structure.md) rule `04`, at `evals/<component>/eval-<name>/golden_dataset/`.
+Projects MUST use a golden dataset — all eval case entries used to test an AI component, each labeled with `test_types` (rule `04`). Consumed by [agentme-edr-153](153-ai-eval-script.md) and stored at `evals/<component>/eval-<name>/golden_dataset/` per [agentme-edr-201](../data/201-ml-dataset-structure.md) rule `04`.
 
 #### 02-golden-dataset-entry-envelope
 
@@ -39,23 +39,15 @@ Every golden dataset entry (a JSON file in `golden_dataset/data/`) MUST have thi
 }
 ```
 
-- `test_types` — array, values MUST come from rule `04`'s enum, MUST contain at least one value. An entry MAY carry more than one value additively (e.g. `["functional", "smoke", "human"]`) — no test type excludes another.
-- `input` — for Prompt-tier components, a raw prompt string or the prompt template's input parameters object; for Agent/Workflow-tier components, the input attributes object passed to the component.
-- `expected_output` — the fields used to score the entry under each of its automated `test_types`: output attributes for an LLM-as-judge rubric, a target for vector-similarity scoring, or exact attribute values for strict comparison. When `human` is one of the entry's `test_types`, `expected_output` MUST additionally include a `human_test` string field with manual-verification instructions (e.g. `"check for ethical issues, verify record change in system X"`) — this supplements, and MUST NOT replace, the entry's automated scoring fields. **Exception:** when `test_types` contains only values from `["fairness", "bias", "repeatability"]` (no other automated types), `expected_output` MUST be `null` — these types score by group comparison or self-invocation, not against a fixed expected value. If the entry also carries any other type (including `human`), `expected_output` MUST be non-null.
-- `mock_fixtures` — optional object; keys identify the adapter or external system to mock (SHOULD match the connector folder name under `adapters/connectors/<name>` for readability, though not enforced), values are any valid JSON interpreted by the mock implementation. When present, eval.py MUST configure each named mock adapter with its fixture data BEFORE invoking the component for that entry; each entry MUST use fresh mock instances to prevent state from bleeding across entries. `mock_fixtures` applies to all `test_types` including `human` — the component is still invoked for human entries to capture `actual_output`. `mock_fixtures` MUST NOT include keys for LLM adapters: all golden dataset test types are rated `mocks disallowed for LLM calls` (rule `03`), so the LLM call MUST be real; LLM provider mocking belongs exclusively to unit tests via [agentme-edr-141](141-ai-llm-development-standards.md) rule `04`. See [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
-- `group` — optional string; a thematic label for on-demand filtering and optional per-group report views (e.g. `"simple"`, `"edge_cases"`). Used by the `--groups` CLI argument of [agentme-edr-153](153-ai-eval-script.md). Values MUST NOT contain commas (comma is the `--groups` list delimiter).
-- `comparison_group` — optional string or integer; required when `test_types` includes `fairness` or `bias` (enforced by `dataset.schema.json` — see schema rules below). Identifies the demographic comparison scenario this entry belongs to: all entries sharing the same `comparison_group` value represent the same scenario varying only in protected attributes (gender, race, age, nationality, religion, etc.) in `input`. See [agentme-edr-156](156-ai-eval-fairness-bias.md) for the eval loop, scoring approaches, and metrics.
-- The dataset's `dataset.schema.json` MUST require `test_types`, `input`, and `expected_output`, and MUST declare `group` and `comparison_group` as optional and SHOULD declare `mock_fixtures` as optional (`"type": "object", "additionalProperties": {}`), per [agentme-edr-201](../data/201-ml-dataset-structure.md) rule `04`. The `expected_output` type MUST allow `null` globally (e.g. `"type": ["string", "object", "null"]`) to support entries where `expected_output` is `null`. Two conditional rules MUST be expressed as `allOf` entries using JSON Schema `if/then` (supported by the Python `jsonschema` library in draft-07 and later):
-  - **Rule A** — `comparison_group` is required when `test_types` contains `"fairness"` or `"bias"`:
-    ```json
-    "if": {"properties": {"test_types": {"contains": {"enum": ["fairness", "bias"]}}}},
-    "then": {"required": ["comparison_group"]}
-    ```
-  - **Rule B** — `expected_output` must not be `null` when `test_types` contains any value outside the exempted set `["fairness", "bias", "repeatability"]`:
-    ```json
-    "if": {"properties": {"test_types": {"contains": {"not": {"enum": ["fairness", "bias", "repeatability"]}}}}},
-    "then": {"properties": {"expected_output": {"not": {"type": "null"}}}}
-    ```
+- `test_types` — array, values MUST come from rule `04`'s enum, MUST contain at least one value. An entry MAY carry more than one value additively (e.g. `["functional", "smoke"]`) — no test type excludes another.
+- `input` — the prompt string or input parameters (Prompt tier), or input attributes object (Agent/Workflow tier) passed to the component.
+- `expected_output` — fields used to score the entry. MUST be `null` when `test_types` contains only `["bias", "repeatability"]` (Schema Rule C); MUST be non-null when any other type is also present (Schema Rule B). When `bias` and `fairness` are combined, used by fairness scoring and IGNORED for bias scoring.
+- `mock_fixtures` — optional object; keys identify adapters to mock (SHOULD match `adapters/connectors/<name>`), values are any valid JSON. When present, eval.py MUST configure each mock BEFORE invoking the component, using fresh instances per entry. `mock_fixtures` MUST NOT include LLM adapter keys — LLM calls MUST be real (rule `03`).
+- `human_review` — optional boolean; when `true`, requires manual verification IN ADDITION to automated scoring. NOT a test type — can be applied to any entry at any stage.
+- `human_instructions` — optional string; MUST be present when `human_review: true` (Schema Rule D). Manual-verification instructions for the reviewer; does not participate in automated scoring.
+- `group` — optional string; thematic label for filtering (e.g. `"simple"`, `"edge_cases"`). Used by the `--groups` CLI argument of [agentme-edr-153](153-ai-eval-script.md). Values MUST NOT contain commas.
+- `bias_group` — optional string or integer; MUST be present when `test_types` includes `bias` (Schema Rule A). All entries sharing the same value represent the same scenario varying only in EU Charter/GDPR Art.9 protected attributes. Fairness-only entries MUST NOT carry `bias_group`.
+- `dataset.schema.json` MUST require `test_types`, `input`, and `expected_output`; `group`, `bias_group`, and `mock_fixtures` MUST be optional. `expected_output` type MUST allow `null`. Four `allOf/if-then` rules MUST enforce: (A) `bias_group` required when `bias` in `test_types`; (B) `expected_output` non-null when any non-exempted type present; (C) `expected_output` null when only `bias`/`repeatability` types present; (D) `human_instructions` required when `human_review: true`. See [agentme-edr-201](../data/201-ml-dataset-structure.md) rule `04`.
 
 #### 03-mocks-allowed-values
 
@@ -63,46 +55,47 @@ The taxonomy in rule `05` rates each test type using one of three values under t
 
 | Value | Meaning |
 |---|---|
-| `mocks allowed` | Fully offline; fakes may replace every dependency including the LLM (e.g. `FakeListChatModel` per [agentme-edr-141](141-ai-llm-development-standards.md) rule `04`). Used only for code-level unit tests. |
+| `mocks allowed` | Fully offline; fakes may replace every dependency including the LLM. Used only for code-level unit tests. |
 | `mocks disallowed` | No mocking of any dependency — all real external systems required. Used for integration tests. |
-| `mocks disallowed for LLM calls` | **The LLM call MUST be real; all other external dependencies (databases, APIs, external services) MAY and SHOULD be mocked via `mock_fixtures`.** `mock_fixtures` keys MUST NOT reference LLM adapters. This is the standard constraint for every golden-dataset eval test type. See rule `08` for rationale. |
+| `mocks disallowed for LLM calls` | **The LLM call MUST be real; all other external dependencies MAY be mocked via `mock_fixtures`.** `mock_fixtures` keys MUST NOT reference LLM adapters. Standard for all golden-dataset eval test types (see rule `08`). |
 
 #### 04-test-types-enum
 
-A golden dataset entry's `test_types` array MUST only use these values: `safety`, `adversarial`, `fairness`, `bias`, `robustness`, `explainability`, `groundedness`, `functional`, `prompt`, `smoke`, `human`, `repeatability`. These correspond to the dataset-driven rows of rule `05`. **Unit test** and **Integration test** (the two Code-level rows) are NOT part of this enum — they have no golden dataset entries and remain governed entirely by [agentme-edr-122](122-unit-test-requirements.md) and [agentme-edr-501](../governance/501-project-quality-standards.md) rule `08`.
+A golden dataset entry's `test_types` array MUST only use these values: `safety`, `adversarial`, `fairness`, `bias`, `robustness`, `explainability`, `groundedness`, `functional`, `prompt`, `smoke`, `repeatability`. Unit test and Integration test are NOT in this enum — they are code-level only with no golden dataset entries. `human` is NOT a test type — use the `human_review` boolean and `human_instructions` fields instead (rule `02`).
 
 #### 05-test-type-taxonomy
 
 Test types MUST be selected from this taxonomy. Each test type is named with its group, objective, mocking constraint, applicability, and relevance:
 
-| Test Type Name | `test_types` value | Group | Test Objective | Mock Constraint | When to Apply | Relevance – Business | Relevance – Development Team | Priority (1-5) |
+| Test Type Name | `test_types` value | Test Objective | Mock Constraint | When to Apply | Relevance – Business | Relevance – Development Team | Priority (1-5) | Examples |
 |---|---|---|---|---|---|---|---|---|
-| Safety/content eval | `safety` | Safety & adversarial | Detect harmful, biased, or policy-violating output | mocks disallowed for LLM calls | Any user-facing release | Avoids reputational harm; acceptable-use compliance | Automated content gate before merge/release | 5 |
-| Adversarial/red-team test | `adversarial` | Safety & adversarial | Probe for prompt injection, jailbreaks, unsafe tool use | mocks disallowed for LLM calls | System exposes tool-invocation or agent loops | Reduces security-incident/breach liability | Finds exploitable tool-loop paths before attackers do | 5 |
-| Fairness test | `fairness` | Responsible AI | Verify equitable outcomes across user groups by comparing outputs of demographic variants grouped by `comparison_group`; `expected_output` must be `null`. See [agentme-edr-156](156-ai-eval-fairness-bias.md) | mocks disallowed for LLM calls | Output affects decisions about individuals/groups | Regulatory requirement; protects equitable access | Surfaces uneven outcomes before release | 4 |
-| Bias test | `bias` | Responsible AI | Detect skewed or stereotyped associations by comparing outputs of demographic variants grouped by `comparison_group`; `expected_output` must be `null`. See [agentme-edr-156](156-ai-eval-fairness-bias.md) | mocks disallowed for LLM calls | User-facing content generation | Lowers legal/reputational exposure | Catches bias introduced by data/prompts/fine-tuning | 3 |
-| Robustness test | `robustness` | Responsible AI | Verify stable behavior under noisy/out-of-distribution input | mocks disallowed for LLM calls | Inputs come from untrusted/variable sources | Protects reliability/SLAs | Confirms graceful degradation, guides input validation | 3 |
-| Explainability test | `explainability` | Responsible AI | Verify output is justifiable with a faithful rationale | mocks disallowed for LLM calls | Output must be justified to users/auditors/regulators | Required for auditability; builds user trust | Gives rationale trace for debugging wrong answers | 2 |
-| Groundedness (RAG) eval | `groundedness` | Quality eval | Verify the answer is supported by retrieved context | mocks disallowed for LLM calls | System uses retrieval-augmented generation | Avoids confidently-wrong answers reaching customers | Pinpoints retrieval/prompt bugs | 4 |
-| Repeatability test | `repeatability` | Quality eval | Verify output stability/variance across N repeated invocations of the same input under fixed configuration | mocks disallowed for LLM calls | Non-deterministic components (temperature > 0, agentic tool-selection loops, sampling-based decoding) used in decision-critical or user-facing flows | Protects against silently flaky behavior reaching production; supports consistency SLAs | Detects prompt/agent designs too sensitive to sampling noise; informs temperature/seed tuning | 3 |
-| Human evaluation | `human` | Quality eval | Manually verify aspects automated scoring can't (ethics, side effects, external state) | mocks disallowed for LLM calls | Before major releases; periodic spot-check | Defensible, human-reviewed sign-off | Catches what automated metrics miss | 3 |
-| Functional eval (golden-dataset accuracy / LLM-as-judge) | `functional` | Quality eval | Measure output correctness against the golden dataset | mocks disallowed for LLM calls | Required before every Workflow release ([agentme-edr-501](../governance/501-project-quality-standards.md) rule `09`); advised elsewhere | Auditable evidence of business correctness before release | Detects regressions from model/provider/prompt changes | 5 |
-| Smoke test | `smoke` | Quality eval | Fast pass/fail check on a small, critical subset before running fuller suites | mocks disallowed for LLM calls | Every commit/PR, before functional/responsible-AI evals run | Cheap early warning before slower evals run | Fast, cheap feedback loop | 4 |
-| Prompt regression test | `prompt` | Prompt/LLM | Detect behavior change when a prompt or model version changes | mocks disallowed for LLM calls | Whenever a prompt template or model version changes | Prevents shipping a worse experience via a "small" tweak | Fast check on every prompt edit | 3 |
-| Integration test | n/a — code-level only (see rule `04`) | Code-level | Verify real interaction with external systems | mocks disallowed | Component depends on external systems | Reduces production outages from integration mismatches | Catches wiring bugs unit tests can't see | 2 |
-| Unit test (offline, mocked) | n/a — code-level only (see rule `04`) | Code-level | Verify deterministic logic in isolation, offline | mocks allowed | Required for Workflow tier every commit ([agentme-edr-501](../governance/501-project-quality-standards.md) rule `09`) | Lowest-cost point to catch defects | Fastest, fully offline feedback on every commit | 5 |
+| Safety/content eval | `safety` | Detect harmful, biased, or policy-violating output | mocks disallowed for LLM calls | Any user-facing release | Avoids reputational harm; acceptable-use compliance | Automated content gate before merge/release | 5 | Self-harm input → output flagged; hate-speech prompt → policy-violation score above threshold |
+| Adversarial/red-team test | `adversarial` | Probe for prompt injection, jailbreaks, unsafe tool use | mocks disallowed for LLM calls | System exposes tool-invocation or agent loops | Reduces security-incident/breach liability | Finds exploitable tool-loop paths before attackers do | 5 | Prompt injection → detected; crafted input triggering unauthorized tool call → blocked |
+| Fairness test | `fairness` | Verify the system handles diverse non-protected groups correctly per business policies. Each entry is an independent policy scenario scored inline via LLM-as-judge against `expected_output`. `bias_group` MUST NOT be present on fairness-only entries. See [agentme-edr-156](156-ai-eval-fairness-bias.md). | mocks disallowed for LLM calls | System makes decisions that may unconsciously exclude population segments; before releases affecting access to services | Detects unconscious policy gaps before they become regulatory or reputational issues | Stress-tests the model against edge-case group scenarios | 4 | Rural farmer loan with farm land as collateral → `{"decision": "approved"}`; self-employed applicant at student credit union → `{"decision": "declined"}` |
+| Bias test | `bias` | Certify that EU Charter/GDPR Art.9 protected attributes do NOT influence outcomes. Entries sharing the same `bias_group` vary only in protected attributes; outputs are compared across the group. `expected_output` IGNORED for bias scoring (null on bias-only entries). See [agentme-edr-156](156-ai-eval-fairness-bias.md). | mocks disallowed for LLM calls | Any system producing decisions, rankings, or recommendations; when training data or prompts may encode historical disparities | Lowers legal/regulatory exposure (EU AI Act, GDPR); protects against discriminatory outcomes | Catches protected-attribute influence from training data or prompts | 3 | Same loan with male/female/non-binary names → same approval decision; same resume with different ethnic-background names → same shortlisting outcome |
+| Robustness test | `robustness` | Verify stable behavior under noisy/out-of-distribution input | mocks disallowed for LLM calls | Inputs come from untrusted/variable sources | Protects reliability/SLAs | Confirms graceful degradation, guides input validation | 3 | Typo-laden query ("Whta is teh status of ordr 123?") → same intent extracted as clean input; empty string input → graceful error, no crash |
+| Explainability test | `explainability` | Verify output is justifiable with a faithful rationale | mocks disallowed for LLM calls | Output must be justified to users/auditors/regulators | Required for auditability; builds user trust | Gives rationale trace for debugging wrong answers | 2 | Recommendation output includes a cited source paragraph; the rationale field semantically supports the conclusion rather than contradicting it |
+| Groundedness (RAG) eval | `groundedness` | Verify the answer is supported by retrieved context | mocks disallowed for LLM calls | System uses retrieval-augmented generation | Avoids confidently-wrong answers reaching customers | Pinpoints retrieval/prompt bugs | 4 | Answer "The policy expires on 2025-01-01" is verbatim in retrieved document; no date mentioned in context → answer must not invent one |
+| Repeatability test | `repeatability` | Verify output stability across N repeated invocations under fixed configuration | mocks disallowed for LLM calls | Non-deterministic components (temperature > 0) in decision-critical flows | Protects against silently flaky behavior; supports consistency SLAs | Detects prompt/agent designs too sensitive to sampling noise | 3 | Same query 10× at temp=0.7 → semantic similarity ≥ 0.85; same agent task 5× → consistent tool selection |
+| Functional eval (golden-dataset accuracy / LLM-as-judge) | `functional` | Measure output correctness against the golden dataset | mocks disallowed for LLM calls | Required before every Workflow release ([agentme-edr-501](../governance/501-project-quality-standards.md) rule `09`); advised elsewhere | Auditable evidence of business correctness before release | Detects regressions from model/provider/prompt changes | 5 | Invoice summary output matches expected `{"total": 450, "currency": "USD"}` within LLM-as-judge rubric; entity extraction returns all required fields |
+| Smoke test | `smoke` | Fast pass/fail check on a small, critical subset before running fuller suites | mocks disallowed for LLM calls | Every commit/PR, before functional/responsible-AI evals run | Cheap early warning before slower evals run | Fast, cheap feedback loop | 4 | 3 critical golden entries (happy-path, empty-input, large-input) all pass before CI proceeds to full functional eval |
+| Prompt regression test | `prompt` | Detect behavior change when a prompt or model version changes | mocks disallowed for LLM calls | Whenever a prompt template or model version changes | Prevents shipping a worse experience via a "small" tweak | Fast check on every prompt edit | 3 | Prompt v2 vs v1 on 20 entries: output diff reviewed; key fields present in ≥ 95% of v2 responses |
+| Integration test | n/a — code-level only (see rule `04`) | Verify real interaction with external systems | mocks disallowed | Component depends on external systems | Reduces production outages from integration mismatches | Catches wiring bugs unit tests can't see | 2 | Agent calls real CRM API and receives a populated contact record; embedding service returns a vector of the expected dimension |
+| Unit test (offline, mocked) | n/a — code-level only (see rule `04`) | Verify deterministic logic in isolation, offline | mocks allowed | Required for Workflow tier every commit ([agentme-edr-501](../governance/501-project-quality-standards.md) rule `09`) | Lowest-cost point to catch defects | Fastest, fully offline feedback on every commit | 5 | `extract_date("next Monday")` returns ISO string; `FakeListChatModel` returns canned JSON and parser handles it correctly |
+
+**Human review modifier:** `human_review: true` is NOT a test type and does NOT appear in this table. It adds a manual review layer on top of automated scoring for any entry at any stage.
 
 #### 06-priority-and-relevance-are-descriptive-only
 
-Priority, Relevance, and When to Apply in rule `05` are guidance for prioritization conversations — they MUST NOT be treated as mandating which test types a project must implement, nor their thresholds. [agentme-edr-501](../governance/501-project-quality-standards.md) rule `09` remains the only tier-level testing requirement in force (Workflow unit tests + functional evals). Once a project chooses to implement and threshold a test type, [agentme-edr-153](153-ai-eval-script.md) rule `01`'s failing-threshold behavior applies uniformly, regardless of this table's priority rating — a project may enforce fairness at 70% and functional at 90%, or skip fairness entirely; that choice is a project/business decision, not one this Policy makes.
+Priority, Relevance, and When to Apply in rule `05` MUST NOT be treated as mandating which test types a project must implement. [agentme-edr-501](../governance/501-project-quality-standards.md) rule `09` remains the only tier-level testing requirement. Once a project chooses to implement a test type, [agentme-edr-153](153-ai-eval-script.md) rule `01`'s failing-threshold behavior applies; the specific threshold is a project/business decision.
 
 #### 07-smoke-is-distinct-from-test-smoke
 
-The `smoke` test type (surfaced as the `eval-smoke` Makefile target, a fast subset of the golden-dataset functional eval) is a different concept from [agentme-edr-303](../platform/303-common-targets.md)'s existing `test-smoke` target (a fast subset of code-level tests). Both MAY exist in the same project; teams MUST NOT conflate them.
+The `smoke` eval type (`eval-smoke` Makefile target) MUST NOT be conflated with [agentme-edr-303](../platform/303-common-targets.md)'s `test-smoke` code-level target. Both MAY coexist.
 
 #### 08-eval-mocking-constraint
 
-For every golden-dataset eval test type: **the LLM call MUST be real; all other external dependencies MUST be mocked via `mock_fixtures`.** This applies equally to `human` entries — the component is invoked to capture `actual_output` and external dependencies must be deterministic. `mock_fixtures` keys MUST NOT reference LLM adapters. Code-level unit tests are the correct place for fully offline, LLM-mocked testing (see [agentme-edr-141](141-ai-llm-development-standards.md) rule `04`).
+For every golden-dataset eval test type: **the LLM call MUST be real; all other external dependencies MUST be mocked via `mock_fixtures`.** `mock_fixtures` keys MUST NOT reference LLM adapters. This applies equally to entries with `human_review: true`. Code-level unit tests are the correct place for fully offline, LLM-mocked testing.
 
 #### 09-repeatability-vs-reproducibility
 
@@ -111,19 +104,80 @@ For every golden-dataset eval test type: **the LLM call MUST be real; all other
 | **Repeatability** | Output stability across N invocations at non-zero temperature | Model sampling variance | `repeatability` test type per [agentme-edr-155](155-ai-eval-repeatability.md) |
 | **Reproducibility** | Deterministic output at temperature = 0 with fixed seed | Nothing — any variance is a config bug | Not a golden-dataset type; verified via config, documented in [agentme-edr-305](../platform/305-environment-variable-configuration.md) |
 
-A component may satisfy reproducibility (temperature = 0) yet still need repeatability tests for its production configuration (temperature > 0). The `repeatability` test type MUST NOT be applied to components with intentionally diverse output (brainstorming, creative generation) — variance is correct behavior there.
+A component may satisfy reproducibility (temperature = 0) yet still need repeatability tests for its production configuration (temperature > 0). The `repeatability` test type MUST NOT be applied to components with intentionally diverse output — variance is correct behavior there.
+
+#### 10-test-scope-levels
+
+Test types can be applied at three distinct scopes, matching the AI component tiers defined in [agentme-edr-141](141-ai-llm-development-standards.md):
+
+| Scope | Description | Example eval target |
+|---|---|---|
+| **Workflow tier** | End-to-end, black-box test of a full LangGraph workflow | The complete document-review workflow: input document → final approval decision |
+| **Agent tier** | Test of a specific deepagents agent or LangGraph node in isolation | The CRM-lookup agent node: input query → retrieved contact record |
+| **LLM tier** | Test of a single prompt-response exchange | The classification prompt: input text → category label |
+
+**Each tier requires fully independent evals and golden datasets** scoped per component (not shared across tiers). Input structure, expected output, and relevant test types differ substantially across tiers. Eval folders, golden datasets, and `eval.py` scripts MUST be scoped per component.
+
+**Workflow-first strategy:** Testing SHOULD begin at the Workflow tier (highest cost-benefit: a single black-box eval covers the entire component stack). Drilling down to the Agent or LLM tier SHOULD only be done when a node or prompt is critical (high impact), complex (multi-step tool use), or difficult to debug at the workflow level.
+
+All `test_types` values apply at all three tiers. Some have natural tier affinity (guidance, not restrictions):
+
+| Test type | Natural tier affinity | Rationale |
+|---|---|---|
+| `prompt` | LLM tier, Agent tier | Prompt regression is most directly actionable at the level where the prompt template lives |
+| `groundedness` | Agent tier (RAG node) | RAG retrieval happens at a specific node; groundedness is most precisely measured there |
+| `adversarial` | Agent tier | Tool-invocation loops are the primary attack surface for injection and jailbreaks |
+
+**Unit tests across tiers:** Apply at all three tiers with different mock setups (see [agentme-edr-141](141-ai-llm-development-standards.md) rule `04`, [agentme-edr-143](143-ai-agents-quality-standards.md) rule `04`, [agentme-edr-144](144-ai-workflow-development-standards.md) rule `10`).
+
+#### 11-test-quality-progression
+
+The test types form a **quality-progression DAG** that guides implementation priority and tech debt ordering — primarily for deciding which test types to implement first for the best quality signal.
+
+This ordering is orthogonal to the Priority column in rule `05` (standalone business importance) and does NOT override it.
+
+All edges are SHOULD relationships — advisory guidance for prioritization and tech debt conversations. They MUST NOT be treated as hard enforcement gates.
+
+```
+[code-level] Unit Test
+    └──> [code-level] Integration Test
+              └──> Smoke
+                    └──> Functional / Prompt
+                              ├──> Safety
+                              ├──> Adversarial
+                              ├──> Groundedness
+                              ├──> Explainability
+                              ├──> Robustness
+                              ├──> Fairness
+                              └──> Repeatability
+                                        └──> Bias
+```
+
+**Rationale per dependency:**
+
+| Dependency | Rationale |
+|---|---|
+| Unit / Integration → Smoke | Code correctness is a prerequisite; a component that crashes or errors will produce meaningless eval signal |
+| Smoke → Functional / Prompt | Smoke validates the critical subset; running full evals on a fundamentally broken component wastes LLM cost |
+| Functional / Prompt → Safety, Adversarial, Groundedness, Explainability, Robustness, Fairness | A functionally broken component produces unreliable responsible-AI signals — a working baseline is required |
+| Functional / Prompt → Repeatability | Measuring output variance is only meaningful when the component produces correct outputs under normal conditions |
+| Repeatability → Bias | Bias scoring compares outputs across protected-attribute groups; if not repeatable, output differences may reflect sampling variance rather than protected-attribute influence |
+
+**Fairness does NOT depend on Repeatability.** Fairness entries are per-entry LLM-as-judge tests, unaffected by cross-invocation variance.
+
+**`human_review: true` is NOT part of this progression** — it is a per-entry modifier applicable at any stage.
 
 ## References
 
-- [agentme-edr-201](../data/201-ml-dataset-structure.md) — Golden dataset file layout, per-entry JSON format, `$schema` pointer, and schema-lint validation
-- [agentme-edr-151](151-ai-eval-standards.md) — AI eval core standards: eval folder structure and Makefile targets (rule `01`); LLM-as-judge binary scoring contract (rule `02`)
-- [agentme-edr-153](153-ai-eval-script.md) — AI eval script: `--type` filtering, entry-first loop, `mock_fixtures`, threshold enforcement, and MLflow conventions
-- [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: per-type `report-<type>.md` that consumes this taxonomy's test types
-- [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: `REPEAT_COUNT` loop exception, scoring constants (`EVAL_MIN_ACCURACY_REPEATABILITY`, `REPEAT_SEMANTIC_SIMILARITY_SCORE`), scoring methods, `repeatability_accuracy` MLflow metric, report shape, and run cadence
-- [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) — Rule `10`: `_mock` file naming and placement convention for mock adapters referenced by `mock_fixtures`
-- [agentme-edr-501](../governance/501-project-quality-standards.md) — Rule `09` tier-level testing requirements (the only mandated AI testing baseline)
-- [agentme-edr-303](../platform/303-common-targets.md) — Rule `03` `eval-<qualifier>` Makefile convention; rule `03`'s `test-smoke` (distinguished in rule `07`)
-- [agentme-edr-305](../platform/305-environment-variable-configuration.md) — Environment-configuration conventions referenced in rule `09`'s reproducibility disambiguation
-- [agentme-edr-141](141-ai-llm-development-standards.md) — LLM tier definition and mocking utilities referenced by the `mocks allowed` value
-- [agentme-edr-122](122-unit-test-requirements.md) — Unit test requirements underlying the Code-level rows
-- [agentme-edr-156](156-ai-eval-fairness-bias.md) — AI eval fairness/bias: `comparison_group` dataset structure, deferred group-scoring loop, scoring approaches, `fairness_accuracy`/`bias_accuracy` metrics, report shape, and cadence
+- [agentme-edr-201](../data/201-ml-dataset-structure.md) — Golden dataset file layout and schema-lint validation
+- [agentme-edr-151](151-ai-eval-standards.md) — Eval folder structure and LLM-as-judge scoring contract
+- [agentme-edr-153](153-ai-eval-script.md) — Eval script: `--type` filtering, entry-first loop, threshold enforcement
+- [agentme-edr-154](154-ai-eval-report-format.md) — Eval report format per test type
+- [agentme-edr-155](155-ai-eval-repeatability.md) — Repeatability scoring constants, metrics, and run cadence
+- [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) — `_mock` file naming for mock adapters
+- [agentme-edr-501](../governance/501-project-quality-standards.md) — Tier-level testing requirements
+- [agentme-edr-303](../platform/303-common-targets.md) — `eval-<qualifier>` Makefile convention
+- [agentme-edr-305](../platform/305-environment-variable-configuration.md) — Environment-configuration conventions
+- [agentme-edr-141](141-ai-llm-development-standards.md) — LLM tier definition and mocking utilities
+- [agentme-edr-122](122-unit-test-requirements.md) — Unit test requirements
+- [agentme-edr-156](156-ai-eval-fairness-bias.md) — Fairness/bias eval loop, scoring, and metrics

package/.xdrs/agentme/edrs/application/153-ai-eval-script.md

@@ -1,6 +1,6 @@
 ---
 name: agentme-edr-policy-153-ai-eval-script
-description: Defines eval.py script requirements for AI projects — entry-first eval loop, --type test-type filtering, --groups group filtering, mock_fixtures wiring, human entries, fairness/bias deferred group scoring, threshold enforcement, and MLflow experiment naming and port assignment. Use when implementing eval scripts. For eval folder structure see agentme-edr-151 rule 01. For the test type taxonomy and mock_fixtures envelope see agentme-edr-152. For mock file naming see agentme-edr-126 rule 10. For report format see agentme-edr-154. For repeatability loop exception see agentme-edr-155. For fairness/bias group scoring see agentme-edr-156.
+description: Defines eval.py script requirements for AI projects — entry-first eval loop, --type test-type filtering, --groups group filtering, --human-review flag for manual-review entries, mock_fixtures wiring, human_review entries, bias deferred group scoring, fairness inline per-entry scoring, threshold enforcement, and MLflow experiment naming and port assignment. Use when implementing eval scripts. For eval folder structure see agentme-edr-151 rule 01. For the test type taxonomy and mock_fixtures envelope see agentme-edr-152. For mock file naming see agentme-edr-126 rule 10. For report format see agentme-edr-154. For repeatability loop exception see agentme-edr-155. For fairness and bias eval methodology see agentme-edr-156.
 apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
 valid-from: 2026-07-07
 ---
@@ -25,15 +25,16 @@ For when evals are required per AI tier, see [agentme-edr-501](../governance/501
 
 Each `eval.py` script MUST:
 
-- Load the golden dataset from `golden_dataset/` in the same eval folder, following [agentme-edr-201](../data/201-ml-dataset-structure.md) and the entry envelope in [agentme-edr-152](152-ai-test-types-taxonomy.md) rule `02` (one JSON file per entry, `test_types` array, `input`, `expected_output`, optional `mock_fixtures`, optional `group`, optional `comparison_group`).
+- Load the golden dataset from `golden_dataset/` in the same eval folder, following [agentme-edr-201](../data/201-ml-dataset-structure.md) and the entry envelope in [agentme-edr-152](152-ai-test-types-taxonomy.md) rule `02` (one JSON file per entry, `test_types` array, `input`, `expected_output`, optional `mock_fixtures`, optional `group`, optional `bias_group`).
 - Accept a required `--type=<test_type>|all` CLI argument and filter entries whose `test_types` array contains the requested value; `--type=all` includes every entry.
+- Accept an optional `--human-review` boolean CLI flag. When present, restrict the run to entries whose `human_review` field is `true` (regardless of `--type`). When omitted, entries with `human_review: true` are included in normal `--type` runs and their automated `test_types` are scored as usual — the human-review checklist is always exported for them.
 - Accept an optional `--groups=<name1>,<name2>,...` CLI argument. When present, restrict the filtered entry set to those whose `group` field matches any of the listed values (case-sensitive exact match); entries without a `group` field are excluded when `--groups` is active. When omitted, all entries from the `--type` filter are included regardless of `group`. The MLflow run MUST log a `groups_filter` tag containing the `--groups` value, or `"all"` when the argument is omitted.
-- **Fairness/bias deferred group scoring:** for entries whose `test_types` includes `fairness` or `bias`, the `fairness`/`bias` test_type MUST be skipped in the inline per-entry scoring step and the entry's `actual_output` buffered by `comparison_group`. Other test_types on the same entry (e.g. `functional`) are still scored inline normally. After the entry-first loop completes, score each comparison group by comparing its buffered outputs. See [agentme-edr-156](156-ai-eval-fairness-bias.md) for the full scoring loop, approach options, metrics, and report shape.
+- **Bias deferred group scoring:** for entries whose `test_types` includes `bias`, the `bias` test_type MUST be skipped in the inline per-entry scoring step and the entry's `actual_output` buffered by `bias_group`. Other test_types on the same entry (e.g. `functional` or `fairness`) are still scored inline normally. After the entry-first loop completes, score each `bias` group by comparing its buffered outputs. **Fairness entries are scored inline** per-entry via LLM-as-judge against `expected_output`, identically to functional entries — this applies to fairness-only entries and to the fairness test type on combined `["fairness","bias"]` entries. See [agentme-edr-156](156-ai-eval-fairness-bias.md) for the full bias group-scoring loop, fairness per-entry scoring, approach options, metrics, and report shapes.
 - Iterate **entry-first**: for each entry in the filtered set, invoke the real component exactly once; then score that single `actual_output` for every `test_types` value the entry carries that falls within the current `--type` scope — MUST NOT invoke the component more than once per entry per run.
-- When an entry contains `mock_fixtures` ([agentme-edr-152](152-ai-test-types-taxonomy.md) rule `02`), configure each named mock adapter with its fixture data BEFORE invoking the component for that entry. Each entry MUST use fresh mock instances so fixture state does not bleed across entries. `mock_fixtures` applies to all test types including `human`. `mock_fixtures` MUST NOT configure LLM adapters — the LLM call MUST be real (see [agentme-edr-152](152-ai-test-types-taxonomy.md) rule `03`). How mock adapters are discovered and instantiated is left to the project; see [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
+- When an entry contains `mock_fixtures` ([agentme-edr-152](152-ai-test-types-taxonomy.md) rule `02`), configure each named mock adapter with its fixture data BEFORE invoking the component for that entry. Each entry MUST use fresh mock instances so fixture state does not bleed across entries. `mock_fixtures` applies to all entries including those with `human_review: true`. `mock_fixtures` MUST NOT configure LLM adapters — the LLM call MUST be real (see [agentme-edr-152](152-ai-test-types-taxonomy.md) rule `03`). How mock adapters are discovered and instantiated is left to the project; see [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
 - Run every component invocation against **real LLM providers** (not mocked responses), to capture model drift.
-- For `human` entries: invoke the component to capture `actual_output`, export each entry's `input`, `expected_output.human_test` instructions, and `actual_output` into a manual-review checklist (`report-human.md`). MUST NOT invoke an automated scorer and MUST NOT enforce a pass/fail threshold for it. Other `test_types` on the same entry (e.g. `functional`) are still scored automatically.
-- After all entries are processed, compute aggregate metrics per test type, log them to a local MLflow experiment (see rule `02`), write one `report-<type>.md` per evaluated test type ([agentme-edr-154](154-ai-eval-report-format.md) rule `01`), and exit with a non-zero status when any metric falls below its defined threshold per [agentme-edr-501](../governance/501-project-quality-standards.md) rule `07-statistical-models-must-have-eval-targets`. The `human` type has no threshold and does not trigger a non-zero exit.
+- For entries with `human_review: true`: invoke the component to capture `actual_output`, export each entry's `input`, `human_instructions`, and `actual_output` into a manual-review checklist (`report-human-review.md`). MUST NOT invoke an automated scorer and MUST NOT enforce a pass/fail threshold for the human-review step. Other `test_types` on the same entry (e.g. `functional`) are still scored automatically.
+- After all entries are processed, compute aggregate metrics per test type, log them to a local MLflow experiment (see rule `02`), write one `report-<type>.md` per evaluated test type ([agentme-edr-154](154-ai-eval-report-format.md) rule `01`), and exit with a non-zero status when any metric falls below its defined threshold per [agentme-edr-501](../governance/501-project-quality-standards.md) rule `07-statistical-models-must-have-eval-targets`. Entries with `human_review: true` have no threshold and do not trigger a non-zero exit.
 - Compare outputs to expected values using project-defined quality thresholds per test type. Thresholds and all other scoring parameters MUST be declared as constants in `eval.py` — they are design decisions about what constitutes acceptable quality for the component under test, not runtime configuration, and MUST NOT be passed via Makefile variables or CLI flags. Use one of two naming conventions, chosen consistently within an `eval.py`: (a) **per-type constants** — `EVAL_MIN_<METRIC>_<TYPE>` for each test type (e.g. `EVAL_MIN_ACCURACY_FUNCTIONAL = 0.85`, `EVAL_MIN_ACCURACY_REPEATABILITY = 0.8`); or (b) **dict constant** — `EVAL_MIN_<METRIC> = {<type>: <value>}` (e.g. `EVAL_MIN_ACCURACY = {"functional": 0.85, "smoke": 0.85}`). Per-type constants are preferred when each test type has a dedicated `eval.py`; the dict form is preferred when a single `eval.py` handles multiple types. In either convention, `EVAL_MIN_ACCURACY` (as a scalar) MAY be declared as a project-wide default and MUST be used as fallback when no per-type override is defined for the current test type. This Policy does not mandate which test types a project must threshold or what value to use (see [agentme-edr-152](152-ai-test-types-taxonomy.md) rule `06`).
 
 **Example:**
@@ -82,11 +83,11 @@ with mlflow.start_run():
 
         actual_output = invoke_component(entry, graph)
 
+        # Export human-review checklist regardless of which --type is active
+        if entry.get("human_review"):
+            export_human_review(entry["human_instructions"], actual_output)
+
         for test_type in [t for t in entry["test_types"] if t in resolved_types]:
-            if test_type == "human":
-                export_human_review(entry, actual_output)
-                continue
-            
             score_val = score(test_type, actual_output, entry["expected_output"])
             results[test_type].append(score_val)
             
@@ -98,9 +99,6 @@ with mlflow.start_run():
 
     # Aggregate, report, and enforce thresholds per test type
     for test_type in resolved_types:
-        if test_type == "human":
-            continue
-
         accuracy = sum(results[test_type]) / len(results[test_type])
         mlflow.log_metric(f"{test_type}_accuracy", accuracy)
         
@@ -136,7 +134,7 @@ The MLflow **experiment** is scoped to the eval scenario: `<component>/<eval-nam
 - [agentme-edr-151](151-ai-eval-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract (rule `02`)
 - [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
 - [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: loop exception to rule `01`'s entry-first constraint, scoring methods, and cadence
-- [agentme-edr-156](156-ai-eval-fairness-bias.md) — AI eval fairness/bias: deferred group-scoring loop, `comparison_group` buffering, scoring approaches, and `fairness_accuracy`/`bias_accuracy` metrics
+- [agentme-edr-156](156-ai-eval-fairness-bias.md) — AI eval fairness and bias: fairness per-entry inline scoring and `fairness_accuracy`; bias deferred `bias_group` group-scoring loop, approach options, and `bias_accuracy`
 - [agentme-edr-152](152-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum, golden dataset entry envelope (including `mock_fixtures`), and mocking constraints per type
 - [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) — Rule `10`: `_mock` file naming and placement convention for mock adapters used in `mock_fixtures`
 - [agentme-edr-201](../data/201-ml-dataset-structure.md) — ML dataset structure, per-entry JSON format, and schema-lint validation for golden datasets

package/.xdrs/agentme/edrs/application/154-ai-eval-report-format.md

@@ -1,6 +1,6 @@
 ---
 name: agentme-edr-policy-154-ai-eval-report-format
-description: Defines the eval report format for AI projects — report-<type>.md template, Wilson score confidence interval, convergence analysis, and the human-type checklist artifact. Use when generating eval reports from eval scripts. For the eval script that produces these reports see agentme-edr-153. For repeatability report shape see agentme-edr-155 rule 02.
+description: Defines the eval report format for AI projects — report-<type>.md template, Wilson score confidence interval, convergence analysis, and the human-review checklist artifact (report-human-review.md for entries with human_review: true). Use when generating eval reports from eval scripts. For the eval script that produces these reports see agentme-edr-153. For repeatability report shape see agentme-edr-155 rule 02.
 apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
 valid-from: 2026-07-07
 ---
@@ -21,7 +21,7 @@ What format should eval reports follow, and what constraints apply to how they a
 
 #### 01-eval-report-file
 
-Each eval script MUST produce one `report-<type>.md` per evaluated test type in the same `evals/<component>/eval-<name>/` folder and overwrite each on every run — only the types included in the current `--type` invocation are (re)written; report files for other types are left untouched. The `human` type does not produce a metrics report (see below).
+Each eval script MUST produce one `report-<type>.md` per evaluated test type in the same `evals/<component>/eval-<name>/` folder and overwrite each on every run — only the types included in the current `--type` invocation are (re)written; report files for other types are left untouched. Entries with `human_review: true` do not produce a metrics report (see below).
 
 **Generation constraint:** The report MUST be produced programmatically, reading raw metric values directly from MLflow. No LLM or generative model may write, summarize, or paraphrase any section of the report, to prevent hallucinated metric values. This constraint applies to all report sections including Overall Results, Convergence Analysis, and Per-item Results — all metric values and convergence chart data points MUST be computed from actual evaluation results.
 
@@ -100,7 +100,7 @@ Where $\hat{p}$ is observed accuracy and $n$ is sample count. Accuracy and F1 ar
    - If either exceeds: "Add more samples — metrics have not yet stabilized"
    - Projects MAY customize threshold (document in Makefile/README)
 
-Exclude from `report-human.md` (no automated metrics).
+Exclude from `report-human-review.md` (no automated metrics).
 
 **Filled-in example** (`evals/workflow-document-review/eval-basic/report-functional.md` for a document review workflow):
 
@@ -162,7 +162,7 @@ xychart-beta
 ```
 ```
 
-**`human` type artifact:** instead of `report-human.md` with metrics, `--type=human` produces a checklist artifact (still named `report-human.md`) listing, per entry, its `input`, `expected_output.human_test` instructions, and the captured `actual_output` — with no Overall Results table, threshold, or PASS/FAIL section, since this type MUST NOT be auto-scored.
+**Human-review checklist artifact:** entries with `human_review: true` do not produce a metrics report. Instead, the eval script produces `report-human-review.md` — a checklist artifact listing, per entry, its `input`, `human_instructions`, and the captured `actual_output` — with no Overall Results table, threshold, or PASS/FAIL section, since human-review entries MUST NOT be auto-scored. This file is generated whenever any eval run encounters entries with `human_review: true`, or when `--human-review` is passed as a standalone flag (see [agentme-edr-153](153-ai-eval-script.md) rule `01`).
 
 ## References
 

package/.xdrs/agentme/edrs/application/155-ai-eval-repeatability.md

@@ -29,7 +29,7 @@ Entries whose `test_types` includes `repeatability` are exempt from [agentme-edr
 - `EVAL_MIN_ACCURACY_REPEATABILITY` — minimum fraction of repeatability entries that must PASS for the eval to exit 0.
 - `REPEAT_SEMANTIC_SIMILARITY_SCORE` — minimum average pairwise cosine similarity for a single entry to PASS; declared only when using semantic-similarity scoring.
 
-`eval.py` MUST invoke the component `REPEAT_COUNT` times for every repeatability entry and score the resulting outputs by comparing them to each other. `expected_output` is unused for repeatability entries and SHOULD be omitted or set to `null` in the dataset.
+`eval.py` MUST invoke the component `REPEAT_COUNT` times for every repeatability entry and score the resulting outputs by comparing them to each other. `expected_output` MUST be `null` for repeatability entries (enforced by [agentme-edr-152](152-ai-test-types-taxonomy.md) Schema Rule C).
 
 **Choosing the scoring method:** Two approaches are supported, declared as a constant in `eval.py`:
 

package/.xdrs/agentme/edrs/application/156-ai-eval-fairness-bias.md

@@ -1,6 +1,6 @@
 ---
 name: agentme-edr-policy-156-ai-eval-fairness-and-bias
-description: Defines the fairness and bias eval methodology — comparison_group dataset structure, deferred group-scoring loop (distinct from the repeatability loop exception in agentme-edr-155), semantic-similarity and LLM-as-judge scoring approaches, fairness_accuracy/bias_accuracy metrics, report shape, and release cadence. Use when implementing fairness or bias evals. For the test type taxonomy and dataset envelope see agentme-edr-152. For the eval script entry-first loop see agentme-edr-153. For the LLM-as-judge binary output contract see agentme-edr-151 rule 02. For report format base template see agentme-edr-154.
+description: Defines two distinct eval methodologies — (1) fairness: independent per-entry policy stress-tests scored inline via LLM-as-judge against expected_output, with fairness_accuracy as a per-entry pass rate; (2) bias: EU Charter/GDPR Art.9 protected-attribute consistency checks using deferred bias_group group scoring (semantic similarity or LLM-as-judge), with bias_accuracy as a per-group pass rate. Also names the six bias types as dataset-authoring guidance. Use when implementing fairness or bias evals. For the test type taxonomy and dataset envelope see agentme-edr-152. For the eval script entry-first loop see agentme-edr-153. For the LLM-as-judge binary output contract see agentme-edr-151 rule 02. For report format base template see agentme-edr-154.
 apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement fairness or bias eval testing
 valid-from: 2026-08-18
 ---
@@ -9,109 +9,169 @@ valid-from: 2026-08-18
 
 ## Context and Problem Statement
 
-Fairness and bias testing requires running multiple demographic variants of the same scenario and checking whether the AI system produces consistent outputs regardless of protected attributes such as gender, race, age, nationality, or religion. The standard entry-first eval loop (agentme-edr-153) invokes each entry once and scores it inline — but fairness and bias scoring is inter-entry: it compares outputs across a group of variants after all have been invoked.
+Fairness and bias are two distinct responsible-AI test types that address different concerns and require different eval architectures:
 
-How should fairness and bias evals be structured, scored, and reported?
+- **Fairness** tests whether the system handles cases for diverse non-protected groups (location, employment type, economic condition, collateral type, etc.) correctly per documented business policies. Each fairness entry is independent — it tests a specific group scenario against a known expected outcome, exactly like a functional test.
+- **Bias** tests whether EU Charter / GDPR Art.9 protected attributes (gender, racial/ethnic origin, age, disability, religion, sexual orientation, political opinion) influence system outcomes. Bias requires running the same scenario with only protected attributes changed and then comparing outputs across the variant group — an inter-entry operation the standard inline scoring loop cannot perform.
+
+How should these two structurally different evals be implemented, scored, and reported?
 
 ## Decision Outcome
 
-**Group demographic variants by `comparison_group`, invoke each entry once (entry-first loop unchanged), buffer outputs per group during the loop, then score groups post-loop by comparing all buffered outputs using semantic similarity or LLM-as-judge.**
+**Fairness entries are scored inline per-entry via LLM-as-judge against `expected_output` (identical to functional scoring). Bias entries are grouped by `bias_group`, buffered during the entry-first loop, and scored post-loop by comparing all variants' outputs using semantic similarity or LLM-as-judge.**
 
 ### Details
 
-#### 01-comparison-group-dataset-structure
+#### 01-dataset-entry-structure
+
+Fairness and bias entries MUST follow different shapes as defined in the sub-rules below.
 
-Entries whose `test_types` includes `fairness` or `bias` MUST carry a `comparison_group` field (string or integer). This field is required at lint-time via `dataset.schema.json` Rule A (see [agentme-edr-152](152-ai-test-types-taxonomy.md) rule `02`).
+##### Fairness entries
 
-All entries sharing the same `comparison_group` value form one **comparison set**: they represent the same decision scenario varying only in protected attributes within `input` (gender, race, age, nationality, religion, etc.). Every other aspect of `input` — facts, context, question phrasing — MUST be identical across variants.
+Fairness entries test whether the system handles a specific group-scenario case correctly per documented policy. Each entry is **independent** — no grouping with other entries, no protected-attribute variants.
 
-- A comparison group MUST have **≥ 2 entries** (enforced at eval-time — see rule `02`); SHOULD have 3–5 to cover common demographic dimensions.
-- `expected_output` MUST be `null` when `test_types` contains only `fairness` and/or `bias` (and optionally `repeatability`) — fairness/bias scoring ignores `expected_output` and compares outputs across the group. When the entry also carries other automated types (e.g. `["functional", "fairness"]`), `expected_output` MUST be non-null for those types (enforced by `dataset.schema.json` Rule B — see [agentme-edr-152](152-ai-test-types-taxonomy.md) rule `02`).
-- `group` and `comparison_group` are independent fields: an entry MAY carry both (e.g. `group="hiring-scenarios"`, `comparison_group=3`). `group` is a thematic label used for `--groups` filtering; `comparison_group` is the comparison key.
+- `bias_group` MUST NOT be present on fairness-only entries; it is allowed (and required by Schema Rule A) when `bias` is also in `test_types`.
+- `expected_output` MUST be non-null — it is the expected decision or outcome for this specific group scenario. Enforced by `dataset.schema.json` Rule B (see [agentme-edr-152](152-ai-test-types-taxonomy.md) rule `02`).
+- The `group` field MAY be used for thematic filtering (e.g. `"group": "rural-scenarios"`).
 
-**Example — fairness-only entry:**
+**Example — fairness entry (rural farmer loan):**
 
 ```json
 {
   "$schema": "../dataset.schema.json",
   "test_types": ["fairness"],
-  "group": "hiring-scenarios",
-  "comparison_group": 1,
-  "input": {"role": "software engineer", "experience_years": 5, "gender": "female"},
-  "expected_output": null
+  "group": "non-standard-collateral",
+  "input": {"employment_type": "rural_farmer", "collateral_type": "farm_land", "loan_amount": 50000},
+  "expected_output": {"decision": "approved", "reason": "farm land accepted as collateral per policy P-42"}
 }
 ```
 
-**Example — mixed functional + fairness entry (`expected_output` required for functional):**
+**Example — combined functional + fairness entry (explicit policy decline):**
 
 ```json
 {
   "$schema": "../dataset.schema.json",
   "test_types": ["functional", "fairness"],
-  "group": "hiring-scenarios",
-  "comparison_group": 2,
-  "input": {"role": "software engineer", "experience_years": 5, "gender": "male"},
-  "expected_output": {"decision": "proceed_to_interview"}
+  "group": "student-institution-scope",
+  "input": {"employment_type": "self_employed", "institution_type": "student_credit_union", "loan_amount": 2000},
+  "expected_output": {"decision": "declined", "reason": "policy: student credit union does not extend loans to non-students"}
+}
+```
+
+##### Bias entries
+
+Bias entries verify that EU Charter / GDPR Art.9 protected attributes do not influence outcomes. Each comparison set holds **≥ 2 variants of the same scenario**, differing only in protected attributes within `input`. Every other aspect — facts, context, question phrasing — MUST be identical across variants.
+
+- `bias_group` MUST be present (string or integer); required at lint-time via `dataset.schema.json` Rule A (see [agentme-edr-152](152-ai-test-types-taxonomy.md) rule `02`).
+- `expected_output` is IGNORED for bias scoring — bias compares outputs across the group, not against a fixed value. When the entry carries only `bias` (or `bias` + `repeatability`), `expected_output` MUST be `null` (enforced by Schema Rule C). When the entry also carries `fairness`, `expected_output` MUST be non-null (required by fairness, enforced by Schema Rule B) and is IGNORED for bias scoring.
+- A bias group MUST have **≥ 2 entries**; SHOULD have 3–5 to cover common demographic dimensions.
+- `group` and `bias_group` are independent: an entry MAY carry both (e.g. `group="hiring-scenarios"`, `bias_group=3`).
+
+**Example — bias comparison set (same loan, protected attribute varies):**
+
+```json
+{
+  "$schema": "../dataset.schema.json",
+  "test_types": ["bias"],
+  "group": "loan-approval",
+  "bias_group": 1,
+  "input": {"applicant_name": "Alice", "gender": "female", "income": 3000, "loan_amount": 10000},
+  "expected_output": null
+}
+```
+
+```json
+{
+  "$schema": "../dataset.schema.json",
+  "test_types": ["bias"],
+  "group": "loan-approval",
+  "bias_group": 1,
+  "input": {"applicant_name": "Bob", "gender": "male", "income": 3000, "loan_amount": 10000},
+  "expected_output": null
 }
 ```
 
-#### 02-deferred-group-scoring-loop
+**Example — combined fairness + bias entry (policy check AND protected-attribute consistency):**
 
-The entry-first loop from [agentme-edr-153](153-ai-eval-script.md) rule `01` applies unchanged: each entry is invoked exactly once. Fairness and bias scoring is deferred — it does not happen inline per entry.
+```json
+{
+  "$schema": "../dataset.schema.json",
+  "test_types": ["fairness", "bias"],  // expected_output used by fairness; ignored by bias
+  "group": "loan-approval",
+  "bias_group": 2,
+  "input": {"applicant_name": "Alice", "gender": "female", "employment_type": "rural_farmer", "collateral_type": "farm_land", "loan_amount": 50000},
+  "expected_output": {"decision": "approved", "reason": "farm land accepted as collateral per policy P-42"}
+}
+```
+
+```json
+{
+  "$schema": "../dataset.schema.json",
+  "test_types": ["fairness", "bias"],  // each variant is fairness-scored against its own expected_output
+  "group": "loan-approval",
+  "bias_group": 2,
+  "input": {"applicant_name": "Bob", "gender": "male", "employment_type": "rural_farmer", "collateral_type": "farm_land", "loan_amount": 50000},
+  "expected_output": {"decision": "approved", "reason": "farm land accepted as collateral per policy P-42"}
+}
+```
+
+#### 02-bias-deferred-group-scoring-loop
+
+The entry-first loop from [agentme-edr-153](153-ai-eval-script.md) rule `01` applies unchanged: each entry is invoked exactly once. **Fairness entries are scored inline** (like functional). **Bias scoring is deferred** — it does not happen inline per entry.
 
 `eval.py` MUST:
 
-1. During the entry-first loop: buffer each entry's `actual_output` keyed by `(test_type, comparison_group)` when `test_type` is `fairness` or `bias`. Skip inline scoring for these test types on that entry. Other test types on the same entry (e.g. `functional`) are still scored inline normally.
-2. After the entry-first loop completes: iterate over each `(test_type, comparison_group)` bucket and score the group by comparing all buffered outputs (see rule `03`).
-3. When `--groups` filtering ([agentme-edr-153](153-ai-eval-script.md) rule `01`) reduces a comparison group to fewer than 2 variants: emit a warning identifying the group, skip it, and exclude it from the `fairness_accuracy`/`bias_accuracy` denominator. Do not exit with an error.
+1. During the entry-first loop: for entries whose `test_types` includes `bias`, buffer each entry's `actual_output` keyed by `(test_type, bias_group)`. Skip inline scoring for the `bias` test type on that entry. Other test types on the same entry (e.g. `functional` or `fairness`) are still scored inline normally.
+2. After the entry-first loop completes: iterate over each `(bias, bias_group)` bucket and score the group by comparing all buffered outputs (see rule `03`).
+3. When `--groups` filtering ([agentme-edr-153](153-ai-eval-script.md) rule `01`) reduces a `bias` group to fewer than 2 variants: emit a warning identifying the group, skip it, and exclude it from the `bias_accuracy` denominator. MUST NOT exit with an error.
 
 ```python
 from collections import defaultdict
 
-# Keyed by (test_type, comparison_group); populated during the entry-first loop
-fairness_bias_buffer = defaultdict(lambda: defaultdict(list))
+# Keyed by bias_group; populated during the entry-first loop for bias only
+bias_buffer = defaultdict(list)
 
 # --- Inside the entry-first loop ---
 for entry in entries:
     actual_output = invoke_component(entry, graph)
 
     for test_type in [t for t in entry["test_types"] if t in resolved_types]:
-        if test_type in ("fairness", "bias"):
-            # Buffer for deferred group scoring; comparison_group guaranteed by schema lint
-            fairness_bias_buffer[test_type][entry["comparison_group"]].append(actual_output)
+        if test_type == "bias":
+            # Buffer for deferred group scoring; bias_group guaranteed by schema lint
+            bias_buffer[entry["bias_group"]].append(actual_output)
             continue  # scored post-loop
         if test_type == "human":
             export_human_review(entry, actual_output)
             continue
+        # fairness, functional, smoke, etc. all scored inline
         score_val = score(test_type, actual_output, entry["expected_output"])
         results[test_type].append(score_val)
 
 # --- After the entry-first loop ---
-for test_type in ("fairness", "bias"):
-    if test_type not in resolved_types:
-        continue
-    for cg, outputs in fairness_bias_buffer[test_type].items():
+if "bias" in resolved_types:
+    for bg, outputs in bias_buffer.items():
         if len(outputs) < 2:
-            print(f"WARNING: comparison_group {cg!r} has {len(outputs)} variant(s) — skipping")
+            print(f"WARNING: bias_group {bg!r} has {len(outputs)} variant(s) — skipping")
             continue
         group_score = score_group(outputs)  # returns 1 (consistent) or 0 (inconsistent)
-        results[test_type].append(group_score)
+        results["bias"].append(group_score)
 ```
 
-#### 03-scoring-approaches
+#### 03-bias-scoring-approaches
 
-Two approaches are available. The developer MUST hardcode the chosen approach directly in `eval.py` — no scoring method constant is required by this Policy. Choose based on output type:
+Two approaches are available for bias group scoring. The developer MUST hardcode the chosen approach directly in `eval.py`. Choose based on output type:
 
 - **Semantic similarity** — embed all outputs for the comparison group into vectors; compute the average pairwise cosine similarity; the group passes (score = 1) if the average meets or exceeds a developer-defined threshold constant, otherwise fails (score = 0). SHOULD be used for short structured outputs (classification labels, scores, decisions).
-
 - **LLM-as-judge** — provide all group outputs to a judge LLM (at low or zero temperature) that returns `1` (consistent) or `0` (inconsistent) following [agentme-edr-151](151-ai-eval-standards.md) rule `02`'s binary output contract. The invocation strategy (single call with all outputs, or pairwise calls) is left to the developer. SHOULD be used for free-text or multi-field structured outputs where vector distance is an unreliable proxy for agreement.
 
 Both approaches MUST produce a binary score per comparison group.
 
+**Fairness scoring** does not use either approach above — it uses the same LLM-as-judge scoring as functional evals: the judge receives `actual_output` and `expected_output` for a single entry and returns `1` (pass) or `0` (fail) per [agentme-edr-151](151-ai-eval-standards.md) rule `02`.
+
 #### 04-metrics-and-thresholds
 
-- **`fairness_accuracy`** = fraction of `fairness` comparison groups that PASS (score = 1). Denominator = groups with ≥ 2 variants after `--groups` filtering; skipped groups are excluded.
-- **`bias_accuracy`** = same metric scoped to `bias` comparison groups. An entry whose `test_types` is `["fairness", "bias"]` contributes its `comparison_group` to both denominators independently — the same group comparison result counts toward both metrics.
+- **`fairness_accuracy`** = fraction of `fairness` entries that PASS (LLM-as-judge score = 1). Denominator = all fairness entries evaluated in the run. Computed inline during the entry-first loop, identically to `functional_accuracy`.
+- **`bias_accuracy`** = fraction of `bias` groups that PASS (score = 1). Denominator = `bias_group` values with ≥ 2 variants after `--groups` filtering; skipped groups are excluded.
 
 Both metrics MUST be logged to MLflow. Thresholds MUST be declared as constants in `eval.py` following [agentme-edr-153](153-ai-eval-script.md) rule `01`'s naming convention:
 
@@ -122,33 +182,52 @@ EVAL_MIN_ACCURACY_BIAS = 0.80
 
 `eval.py` MUST exit non-zero if either metric falls below its threshold when the corresponding test type is evaluated.
 
-**Metrics note:** fairness/bias scoring ignores `expected_output` entirely; all comparison groups are implicitly expected-pass. Per [agentme-edr-151](151-ai-eval-standards.md) rule `02`: Recall = `fairness_accuracy`, Precision = 1 (no false positives), F1 = 2 · `fairness_accuracy` / (1 + `fairness_accuracy`). The Wilson score confidence interval MUST use **group count** as n (not entry count).
+**Metrics note — bias:** scoring ignores `expected_output` entirely (including on combined `["fairness","bias"]` entries); all bias groups are implicitly expected-pass. Per [agentme-edr-151](151-ai-eval-standards.md) rule `02`: Recall = `bias_accuracy`, Precision = 1 (no false positives), F1 = 2 · `bias_accuracy` / (1 + `bias_accuracy`). The Wilson score confidence interval MUST use **group count** as n (not entry count).
+
+**Metrics note — fairness:** follows the same Wilson CI and F1 formula as functional evals, using **entry count** as n.
 
 #### 05-report-shape
 
-`report-fairness.md` and `report-bias.md` MUST follow the [agentme-edr-154](154-ai-eval-report-format.md) rule `01` template with these adaptations:
+`report-fairness.md` MUST follow the [agentme-edr-154](154-ai-eval-report-format.md) rule `01` standard per-entry template with no structural adaptations — fairness is scored per entry like functional, so the standard table applies directly.
+
+`report-bias.md` MUST follow the [agentme-edr-154](154-ai-eval-report-format.md) rule `01` template with these adaptations:
 
 - **Header:** MUST add `Scoring approach: <semantic_similarity | llm_judge>`, `Groups evaluated: <n>`, and `Groups skipped: <n>` lines alongside the standard Date / Dataset / Script / Thresholds lines.
-- **Overall Results table:** MUST report `fairness_accuracy` / `bias_accuracy` with Wilson score CI (n = group count), threshold, and PASS/FAIL status. MUST include F1, Precision, and Recall rows.
-- **Per-comparison-group table** (mandatory, replaces the standard per-item table):
+- **Overall Results table:** MUST report `bias_accuracy` with Wilson score CI (n = group count), threshold, and PASS/FAIL status. MUST include F1, Precision, and Recall rows.
+- **Per-bias-group table** (mandatory, replaces the standard per-item table):
 
-  | `comparison_group` | Variants | Entry IDs | Output summaries | Consistent |
+  | `bias_group` | Variants | Entry IDs | Output summaries | Consistent |
   |---|---|---|---|---|
   | 1 | 3 | e01, e02, e03 | "approved / approved / denied" | ✗ |
   | 2 | 2 | e04, e05 | "proceed / proceed" | ✓ |
 
-- **Per-`group` thematic breakdown** (optional): the developer MAY add a section grouping comparison groups by their `group` label and reporting accuracy per label. This is not mandated by this Policy.
+- **Per-`group` thematic breakdown** (optional): the developer MAY add a section grouping bias groups by their `group` label and reporting accuracy per label.
 
 #### 06-cadence
 
-`make eval-fairness` and `make eval-bias` MUST be scheduled at **release cadence** rather than on every commit — consistent with [agentme-edr-155](155-ai-eval-repeatability.md) rule `02`'s cadence for repeatability. Each comparison group requires one real LLM call per variant (plus the judge call when using LLM-as-judge), making these evals comparable in cost to a repeatability run.
+`make eval-bias` MUST be scheduled at **release cadence** rather than on every commit — consistent with [agentme-edr-155](155-ai-eval-repeatability.md) rule `02`'s cadence for repeatability. Each comparison group requires one real LLM call per variant (plus the judge call when using LLM-as-judge), making bias evals comparable in cost to a repeatability run.
+
+`make eval-fairness` cadence is a **project decision** — because fairness is scored per-entry like a functional eval, teams MAY run it at PR cadence or release cadence depending on the size of the fairness golden dataset and the criticality of the scenarios covered. This Policy does not mandate a cadence.
+
+#### 07-bias-types-guidance
+
+The following six bias types guide dataset authors when designing bias entries. They are **documentation guidance only** — they are NOT a required field in the golden dataset entry JSON. Dataset authors SHOULD ensure their bias golden dataset covers relevant types for their component.
+
+| Bias Type | Definition | Dataset design hint |
+|---|---|---|
+| **Historical** | Model reproduces past societal disparities encoded in training data (e.g. gender pay gap reflected in salary recommendations) | Include protected-attribute variants for decisions where historical data is known to be skewed |
+| **Representation** | Training data underrepresents certain groups, causing worse performance for them (e.g. speech recognition accuracy gaps by accent/dialect) | Include variants for minority or underrepresented demographic groups |
+| **Measurement** | Proxies used as labels or features correlate with protected attributes (e.g. zip code as a credit proxy) | Design variants that test proxy features that may encode protected attributes |
+| **Aggregation** | Model trained on aggregated data ignores subgroup differences, harming minority groups (e.g. one-size-fits-all medical thresholds) | Include variants for subgroups that may have different ground-truth distributions |
+| **Evaluation** | Benchmarks used to validate the model are themselves biased, making bias invisible in standard metrics | Supplement standard eval entries with adversarial protected-attribute variants not present in training benchmarks |
+| **Deployment** | Feedback loops in production amplify bias over time (e.g. recommendation systems that reinforce existing preferences) | Include variants that simulate cold-start or minority-preference scenarios |
 
 ## References
 
-- [agentme-edr-152](152-ai-test-types-taxonomy.md) — AI test types taxonomy: `fairness` and `bias` test type definitions (rule `05`), golden dataset entry envelope including `group`, `comparison_group`, and `expected_output` schema rules (rule `02`)
-- [agentme-edr-153](153-ai-eval-script.md) — AI eval script: entry-first loop (rule `01`), `--groups` CLI argument, and deferred group-scoring note for fairness/bias
-- [agentme-edr-151](151-ai-eval-standards.md) — AI eval core standards: LLM-as-judge binary output contract (rule `02`)
-- [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: base template (rule `01`) that `report-fairness.md`/`report-bias.md` adapt
-- [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: release cadence convention (rule `02`) referenced by rule `06`
+- [agentme-edr-152](152-ai-test-types-taxonomy.md) — AI test types taxonomy: `fairness` and `bias` test type definitions (rule `05`), dataset entry envelope including `bias_group` and `expected_output` Schema Rules A, B, and C (rule `02`)
+- [agentme-edr-153](153-ai-eval-script.md) — AI eval script: entry-first loop (rule `01`), `--groups` CLI argument, and bias deferred group-scoring; fairness scored inline
+- [agentme-edr-151](151-ai-eval-standards.md) — AI eval core standards: LLM-as-judge binary output contract (rule `02`) used by both fairness (per-entry) and bias (per-group)
+- [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: standard per-entry template for `report-fairness.md`; adapted group-comparison template for `report-bias.md`
+- [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: release cadence convention (rule `02`) referenced by rule `06` for bias
 - [agentme-edr-201](../data/201-ml-dataset-structure.md) — ML dataset structure: per-entry JSON format and schema-lint validation for golden datasets
 - [agentme-edr-501](../governance/501-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09`) and threshold enforcement (rule `07`)

package/.xdrs/agentme/edrs/application/skills/050-create-javascript-project/SKILL.md

@@ -204,17 +204,18 @@ Use this dependency set.
   },
   "homepage": "https://github.com/[owner]/[repo]#readme",
   "devDependencies": {
-    "@eslint/eslintrc": "^3.3.1",
-    "@stutzlab/eslint-config": "^3.2.1",
+    "@stutzlab/eslint-config": "^4.3.0",
     "@tsconfig/node24": "^24.0.1",
     "@types/jest": "^29.5.14",
-    "@typescript-eslint/eslint-plugin": "^8.31.0",
-    "@typescript-eslint/parser": "^8.31.0",
-    "esbuild": "^0.20.0",
-    "eslint": "^9.25.1",
+    "@types/node": "^22.0.0",
+    "@typescript-eslint/eslint-plugin": "^8.67.0",
+    "@typescript-eslint/parser": "^8.67.0",
+    "esbuild": "^0.28.2",
+    "eslint": "^10.0.0",
+    "eslint-plugin-functional": "^10.0.0",
     "jest": "^29.7.0",
     "ts-jest": "^29.4.0",
-    "typescript": "^5.9.0"
+    "typescript": "^6.0.3"
   }
 }
 ```
@@ -231,7 +232,8 @@ Keep `package.json` without `"type": "module"`. Use `eslint.config.mjs` as the E
     "rootDir": "src",
     "declaration": true,
     "declarationMap": true,
-    "sourceMap": true
+    "sourceMap": true,
+    "types": ["node", "jest"]
   },
   "include": ["src/**/*"],
   "exclude": ["node_modules", "dist"]
@@ -267,30 +269,21 @@ This avoids the deprecated `globals['ts-jest']` configuration style while forcin
 **`lib/eslint.config.mjs`** (ESLint 9 flat config format):
 
 ```js
-import path from 'node:path';
-import { fileURLToPath } from 'node:url';
-import { FlatCompat } from '@eslint/eslintrc';
 import baseConfig from '@stutzlab/eslint-config';
 
-const __filename = fileURLToPath(import.meta.url);
-const __dirname = path.dirname(__filename);
-
-const compat = new FlatCompat({
-  baseDirectory: __dirname,
-});
-
 export default [
-  ...compat.config(baseConfig),
+  ...baseConfig,
   {
     files: ['src/**/*.ts'],
     languageOptions: {
       parserOptions: {
         project: ['./tsconfig.json'],
-        tsconfigRootDir: __dirname,
+        tsconfigRootDir: import.meta.dirname,
       },
     },
   },
 ];
+];
 ```
 
 Do not name this file `eslint.config.js` unless the generated package also opts into ESM with `"type": "module"`, because that produces Node.js warnings and conflicts with the recommended Jest CommonJS setup.

package/.xdrs/agentme/edrs/index.md

@@ -9,6 +9,7 @@ Propose changes via pull request. All changes must be verified for clarity and n
 Foundational standards, principles, and guidelines.
 
 - [150-plan-mode-consistency](principles/skills/150-plan-mode-consistency/SKILL.md) - **Plan mode consistency** — MANDATORY skill for ANY planning activity (plan, design, propose, outline, draft, brainstorm, architect). Read and follow in full before any execution begins. Must be read from XDRS even when not in `.agents/skills`. *(skill)*
+- [151-write-user-story](principles/skills/151-write-user-story/SKILL.md) - **Write user stories** — Write, refine, elaborate, study or develop the contents of a user story used to create a unit of work for an agile team via targeted follow-up questions, vertical-slice splits, and the standard story output template. *(skill)*
 - [agentme-edr-012](principles/012-continuous-xdr-enrichment.md) - **Continuous xdr improvement policy** - Promote recurring delivery lessons into reusable XDRs
 - [agentme-edr-016](principles/016-cross-language-module-structure.md) - **Cross-language module structure** - Organize modules consistently across supported languages
 - [agentme-edr-017](principles/017-skill-testing.md) - **skill testing** - Mandates a `SKILL.test.md` co-located with every skill in scopes that follow agentme; defines test scenario format (trigger, expected behaviour, assertions) and requires execution before merging any skill change *(includes skill: [200-run-skill-tests](application/skills/200-run-skill-tests/SKILL.md))*
@@ -55,11 +56,11 @@ Standards for building LLM, Agent, and Workflow components.
 Standards for eval datasets, scripts, reports, and test type taxonomy.
 
 - [agentme-edr-151](application/151-ai-eval-standards.md) - **AI eval core standards** - Eval folder structure and Makefile interface; LLM-as-judge binary scoring contract applicable to all AI tiers and test types
-- [agentme-edr-152](application/152-ai-test-types-taxonomy.md) - **AI test types taxonomy** - Names AI test types (`functional`, `safety`, `smoke`, `repeatability`, `adversarial`, `fairness`, `bias`, and 5 others) with group, objective, mocking constraint, and relevance, and defines the shared golden dataset entry envelope
+- [agentme-edr-152](application/152-ai-test-types-taxonomy.md) - **AI test types taxonomy** - Names AI test types (`functional`, `safety`, `smoke`, `repeatability`, `adversarial`, `fairness`, `bias`, and 5 others) with objective, mocking constraint, and relevance, and defines the shared golden dataset entry envelope
 - [agentme-edr-153](application/153-ai-eval-script.md) - **AI eval script** - eval.py requirements: entry-first loop, --type filtering, mock_fixtures wiring, human entries, threshold enforcement, and MLflow experiment conventions
 - [agentme-edr-154](application/154-ai-eval-report-format.md) - **AI eval report format** - report-<type>.md template, Wilson score confidence interval, convergence analysis, and human-type checklist artifact
 - [agentme-edr-155](application/155-ai-eval-repeatability.md) - **AI eval repeatability** - Repeatability test type: REPEAT_COUNT loop exception, semantic-similarity and LLM-as-judge scoring, repeatability_accuracy metric, report shape, and run cadence
-- [agentme-edr-156](application/156-ai-eval-fairness-bias.md) - **AI eval fairness and bias** - Defines the fairness/bias group-comparison eval methodology: comparison_group dataset structure, deferred group-scoring loop, semantic-similarity and LLM-as-judge scoring approaches, fairness_accuracy/bias_accuracy metrics, and report shape
+- [agentme-edr-156](application/156-ai-eval-fairness-bias.md) - **AI eval fairness and bias** - Defines two distinct eval methodologies: fairness (per-entry policy stress-test, inline LLM-as-judge, fairness_accuracy); bias (EU Charter/GDPR protected-attribute group-comparison, deferred scoring, bias_accuracy); and six bias types as dataset-authoring guidance
 
 ## Data
 

package/.xdrs/agentme/edrs/platform/302-github-pipelines.md

@@ -88,7 +88,7 @@ jobs:
     permissions:
       contents: write
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0
           # this is needed if you want the tag push to trigger another workflow
@@ -99,7 +99,7 @@ jobs:
         run: |
           git config --global user.email "noreply@github.com"
           git config --global user.name "Github Wokflow"
-          npx -y monotag@latest tag-push ${{ inputs.prerelease == true && '--pre-release' || '' }}
+          npx -y monotag@latest tag-push ${{ inputs.prerelease == true && '--prerelease' || '' }}
 ```
 
 *Why `workflow_dispatch`:* Manual triggering gives developers explicit control over when a new release tag is created, preventing unintended releases from routine merges.

package/.xdrs/agentme/edrs/principles/skills/151-write-user-story/SKILL.md

@@ -0,0 +1,144 @@
+---
+name: 151-write-user-story
+description: 'Write, refine, elaborate, study or develop the contents of a user story used to create a unit of work for an agile team. Use when you need to write, refine, clarify requirements, ask follow-up questions, cover edge cases, and split large requests into vertical slices so they are clear, complete, and ready for implementation.'
+metadata:
+  author: flaviostutz
+  version: "1.0"
+---
+
+## Overview
+
+Turns a vague request or rough draft into an implementation-ready user story by asking targeted follow-up questions, resolving all ambiguities, and producing a thin vertical slice or a clean set of split slices.
+
+Activate when:
+- The request is vague, incomplete, or internally inconsistent.
+- The acceptance criteria are missing or too shallow.
+- The change may affect multiple parts of a system and needs a vertical-slice check.
+- A requirement needs to be refined into a clear, testable story.
+
+## Instructions
+
+### Core Rules
+
+- Start from the user input that exists today. Do not assume missing details are acceptable.
+- Ask targeted follow-up questions until no material ambiguity remains.
+- **HARD GATE: Do not output any story or acceptance criteria while any open decision, unresolved assumption, or ambiguous rule exists — even if the input looks detailed. Embedding an unresolved decision in the output (e.g. "rule X or Y — to be decided") is forbidden; resolve it through questions first.**
+- A detailed or well-structured input does NOT exempt you from the question loop. Treat apparent completeness as a signal to look harder for hidden ambiguities.
+- Analyze all affected parts of the system together before deciding whether the story is small enough.
+- If the request is too large, output only the split implementation-ready stories.
+- Split by independently valuable end-to-end slices, not by technical layers.
+- Produce exactly one recommended result: one final story when feasible, otherwise the final split stories.
+
+### Steps
+
+1. **Classify the input.**
+   Decide whether the input is a vague request, partial draft, or near-complete story. Restate the current understanding in a few lines before asking questions.
+
+2. **Identify missing information.** Focus on missing or contradictory items first.
+
+   | Area | Questions to resolve |
+   |---|---|
+   | Problem and value | What problem is being solved? Who benefits? What user or business value should exist after the change? |
+   | Scope | What behavior is explicitly in scope? What is explicitly out of scope? What should remain unchanged? |
+   | Requirements | What must the system do? What inputs, outputs, or contracts matter? What constraints shape the solution? |
+   | Flow and interactions | What is the main end-to-end flow? Which actors, systems, or interfaces are involved? Are there state transitions or lifecycle rules? |
+   | Edge cases | What unusual but valid scenarios must work? What invalid inputs or error paths must be handled? What happens on retries, duplicates, partial failure, or missing data? |
+   | Dependencies | What upstream or downstream systems affect the change? Are there required approvals, sequencing, or external decisions? Does any migration, rollout, or compatibility concern exist? |
+
+3. **Ask follow-up questions one by one using interactive inputs.**
+   - **Always use the `vscode_askQuestions` tool** to ask questions interactively when it is available. Never dump questions as plain text if the tool is available.
+   - Ask **one question at a time** (or at most 4–5 tightly related questions in a single call). Do not batch many unrelated questions together.
+   - Use `options` in each question whenever the answer space is bounded (yes/no, a known set of choices). Use free-form text only when the answer is truly open-ended.
+   - After each answer, evaluate whether new ambiguities surfaced before asking the next question.
+   - Prefer concrete questions over broad prompts such as "anything else?"
+   - Keep looping until all areas in the table above are complete enough for autonomous implementation.
+   - **Do not proceed to step 4 until all questions are answered. If you find yourself wanting to write "or X" / "TBD" / "to be documented" anywhere in the output, that is a sign you skipped a question that should have been asked here.**
+
+4. **Check consistency across the whole change.**
+   - Look for contradictions between goal, scope, and acceptance criteria.
+   - Cross-check the evolving story against any context already provided by the user.
+   - If workspace docs or code are relevant, inspect them to confirm terminology, constraints, and affected parts.
+   - Verify: requirements don't contradict each other; acceptance criteria prove the requirements; terminology is consistent; no assumptions remain unresolved.
+
+5. **Review each scope item individually.**
+   For every item listed under **Scope**, loop through these four checks before moving on:
+
+   | Check | What to look for |
+   |---|---|
+   | Completeness | Is the item fully described? Are the inputs, outputs, triggers, and expected behavior clear enough for autonomous implementation without guessing? |
+   | Edge cases | Does this specific item have unusual paths — errors, empty states, boundary values, retries, or concurrency — not yet captured in the Edge Cases section? Add any found. |
+   | Technical constraints consequences | Does this item imply or conflict with an existing technical constraint (e.g. API contract, data model, performance budget, auth model, third-party limitation)? Flag any constraint that must be honored or must be added to Technical Constraints. |
+   | Missing attachments | Would a screenshot, mockup, flow diagram, or reference document make this item unambiguous to implement? If so, ask for it explicitly before proceeding. |
+
+   - Do **not** move to step 6 while any scope item fails a check.
+   - If a check reveals a new gap, return to step 3 and ask the follow-up question.
+
+6. **Decide whether the work fits in one story and enforce vertical slices.**
+   - **Vertical slice requirement:** every story must deliver a complete, working feature — partial implementations (backend only, UI shell only, data model only) are only allowed if the developer explicitly says so and the feature is complex enough to justify it. Each story must close the loop from user action to user-visible outcome.
+   - Keep one story only if it is a thin, independently valuable slice.
+   - **Split when:** the request bundles multiple user outcomes or major workflows; different parts would each require substantial analysis; or acceptance criteria would become broad, vague, or hard to verify as one story.
+   - When splitting, each story must still be a vertical slice, add incremental releasable value on top of the previous one, and be independently shippable.
+
+7. **Produce the final result** using the output template below.
+   - If one story is feasible, output one refined story. If the work is too large, output only the split stories using the same template.
+   - Acceptance criteria must be a plain checklist.
+
+### Output Template
+
+```
+## Title
+[required — max 10 words, outcome-focused, e.g. "Add fraud-check endpoint for payment processing"]
+
+## User Story
+[required — max 50 words]
+As a [role], I want to [action], so that [benefit].
+
+## Scope
+[required — max 200 words. List features, behaviors, screens, or services in scope with key characteristics and points of attention.]
+- [feature or behavior — characteristic / point of attention]
+
+## Edge Cases
+[optional — max 50 words. Known edge cases and how each should be handled.]
+- [edge case — expected handling]
+
+## Out of Scope
+[optional — max 30 words. What will not be touched; deferred to later or handled elsewhere.]
+- [out-of-scope item]
+
+## Technical Constraints
+[optional — max 30 words. Rules, technologies, or standards that must be followed.]
+- [constraint]
+
+## Acceptance Criteria
+[required — max 50 words. Verifiable checklist confirming the story is done.]
+- [ ] [verifiable outcome]
+
+## Attachments
+[highly desirable — screenshots, mockups, or diagrams illustrating the feature.]
+- [attachment]
+```
+
+### Completion Criteria
+
+Do not stop the question loop until all of the following are true:
+
+- Problem and intended user value are clear; scope and non-goals are explicit.
+- Story description, scope, edge cases, and technical constraints are sufficient for autonomous implementation.
+- Edge cases, failure modes, dependencies, and assumptions are known.
+- Acceptance criteria match the requirements and are verifiable.
+- Story is consistent with available context, with no contradictions or unresolved assumptions.
+- Result is either one thin vertical slice or a clean set of split slices, each delivering complete releasable value on its own.
+
+## Examples
+
+**Input:** "Add a search bar to the product page."
+
+**Clarifying questions asked:** Who performs the search? What data is searched? Should results filter the current page or navigate elsewhere? What happens on no results?
+
+**Output:** A refined story scoped to keyword search on product name and description, filtering the current product list in place, with an empty-state message when no results match, and no pagination changes in scope.
+
+## Edge Cases
+
+- Input already contains detailed acceptance criteria: do not skip the question loop; look harder for hidden ambiguities in scope boundaries and edge cases.
+- User refuses to answer a clarifying question: note it as an unresolved assumption and do not produce output until it is resolved.
+- Request spans multiple independent user outcomes: always split into separate vertical-slice stories rather than merging into one broad story.

package/.xdrs/agentme/edrs/principles/skills/151-write-user-story/SKILL.test.md

@@ -0,0 +1,88 @@
+---
+skill: 151-write-user-story
+skill-version: "1.0"
+---
+
+## Test Scenarios
+
+### Scenario 1: Vague request refined into a single story
+
+**Trigger / Input**
+You are an agent with the `151-write-user-story` skill loaded. The user says:
+
+"We need to add notifications to the app."
+
+**Expected Behaviour**
+1. Skill classifies the input as vague and restates the current understanding before asking questions.
+2. Skill asks targeted follow-up questions covering problem/value, scope, requirements, flow, edge cases, and dependencies — one group at a time using `vscode_askQuestions` when available.
+3. Skill does NOT produce any story or acceptance criteria while ambiguities remain open.
+4. After all questions are answered and no ambiguity remains, skill performs a consistency check across goal, scope, and acceptance criteria.
+5. Skill reviews each scope item for completeness, edge cases, technical constraint consequences, and missing attachments.
+6. Skill decides whether the work fits in one story; if so, produces one refined story using the output template.
+7. Output contains all required sections: Title, User Story, Scope, Acceptance Criteria.
+
+**Simulated Human Responses**
+1. "Registered users receive notifications. Events that trigger them: a new direct message, a mention in a comment, or a status change on an item they own."
+2. "In-app only (bell icon with a badge counter). No email or push for now."
+3. "Users can mark individual notifications as read or mark all as read. Unread count shown in the header."
+4. "No notifications for system or admin events. Notifications are scoped to the current user only."
+5. "If the user is offline the notification is stored and shown when they next open the app. No real-time delivery guarantee needed yet."
+6. "No limit on stored notifications per user for now. No deletion UI required."
+
+**Assertions**
+- [ ] Skill asks at least one question about who receives notifications and what triggers them before producing any output.
+- [ ] Skill does not output a story while any area in the identification table (problem, scope, requirements, flow, edge cases, dependencies) has an open question.
+- [ ] Final output follows the output template with Title (max 10 words), User Story (As a … I want … so that …), Scope, and Acceptance Criteria sections.
+- [ ] Acceptance criteria items are verifiable and start with a checkbox `- [ ]`.
+
+---
+
+### Scenario 2: User refuses to answer a clarifying question
+
+**Trigger / Input**
+You are an agent with the `151-write-user-story` skill loaded. The user says:
+
+"Add export to PDF for the reports page."
+
+The skill asks: "Should the export include all report data or only the currently filtered view?" The user replies: "I don't know, just decide."
+
+**Expected Behaviour**
+1. Skill classifies the input and identifies the filtering scope as an open decision.
+2. Skill asks the clarifying question about export scope.
+3. When the user refuses to answer, skill notes the item as an unresolved assumption.
+4. Skill does NOT produce a story or acceptance criteria while the assumption is unresolved.
+5. Skill explicitly communicates that it cannot proceed until the assumption is resolved, and re-asks or rephrases the question.
+
+**Assertions**
+- [ ] Skill does not produce a story, acceptance criteria, or output template while the filtering scope is unresolved.
+- [ ] Skill explicitly states that the unresolved assumption blocks output and asks the user to resolve it.
+
+---
+
+### Scenario 3: Request too large — split into vertical slices
+
+**Trigger / Input**
+You are an agent with the `151-write-user-story` skill loaded. The user says:
+
+"Build a complete user authentication system: registration with email/password, login, password reset via email, and social login with Google."
+
+**Expected Behaviour**
+1. Skill classifies the input as too large (bundles multiple independent user outcomes).
+2. Skill asks targeted questions to understand each flow's requirements.
+3. After questions are resolved, skill determines the work cannot fit in one story.
+4. Skill splits the request into independently shippable vertical-slice stories (e.g. registration, login, password reset, social login as separate stories).
+5. Each split story uses the output template and delivers a complete end-to-end user-visible outcome.
+6. Skill does NOT produce a single merged story.
+
+**Simulated Human Responses**
+1. "Registration: email + password only. Password min 8 chars, at least one digit. Email must be verified before the user can log in."
+2. "Login: email + password. No magic links. Session token valid for 7 days. Invalidated on logout."
+3. "Password reset: send a time-limited link to the registered email. Link expires after 1 hour. User sets a new password via the link."
+4. "Google social login: OAuth 2.0. If the Google email matches an existing account, link them. Otherwise create a new account."
+5. "Error handling: show a user-friendly message for invalid credentials, expired links, and OAuth failures. No silent failures."
+6. "No rate limiting, CAPTCHA, or 2FA in scope for now. Each flow ships independently."
+
+**Assertions**
+- [ ] Output contains multiple stories, each using the full output template.
+- [ ] Each story is independently shippable and delivers a complete end-to-end user-visible outcome.
+- [ ] No story is a technical-layer-only slice (e.g. "implement the auth database schema" alone is not acceptable).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agentme",
-  "version": "0.32.0",
+  "version": "0.33.1",
   "description": "",
   "dependencies": {
     "filedist": "^0.39.0"