agentme 0.24.2 → 0.25.2

package/.xdrs/agentme/edrs/application/030-ai-test-types-taxonomy.md

@@ -0,0 +1,98 @@
+---
+name: agentme-edr-policy-030-ai-test-types-taxonomy
+description: Names AI-application test types (safety, responsible-AI, quality-eval, prompt, code-level) with their group, objective, mocking constraint, and relevance, and defines the shared "golden dataset" entry envelope that agentme-edr-028's eval tooling filters by test_types. Use when deciding which AI test types to implement or when authoring a golden dataset entry.
+apply-to: AI projects (LLM, Agent, or Workflow tier) implementing AI-specific test types beyond generic code-level unit/integration tests
+valid-from: 2026-07-05
+---
+
+# agentme-edr-policy-030: AI test types taxonomy
+
+## Context and Problem Statement
+
+AI components need test types beyond generic unit/integration tests (safety, fairness, groundedness, functional accuracy, etc.). Which test types should be named, and how should their datasets and eval tooling work?
+
+## Decision Outcome
+
+**Adopt a named taxonomy of AI test types plus a shared "golden dataset" entry envelope that agentme-edr-028's eval tooling filters by `test_types`.**
+
+Each test type is named with its group, objective, mocking constraint, applicability, and relevance; every golden dataset entry is labeled with the test types it applies to.
+
+### Details
+
+#### 01-golden-dataset-concept
+
+Projects MUST use a golden dataset to test AI components. A **golden dataset** comprises all eval case entries used to test an AI component (LLM, Agent, or Workflow tier); each entry is labeled with the `test_types` (rule `04`) it applies to. It is the dataset consumed by [agentme-edr-028](028-ai-eval-standards.md) evals and stored as one JSON file per entry per [agentme-edr-024](024-ml-dataset-structure.md) rule `04`, at `evals/<component>/eval-<name>/golden_dataset/`.
+
+#### 02-golden-dataset-entry-envelope
+
+Every golden dataset entry (a JSON file in `golden_dataset/data/`) MUST have this shape, in addition to any project-specific fields:
+
+```json
+{
+  "$schema": "../dataset.schema.json",
+  "test_types": ["functional"],
+  "input": "...",
+  "expected_output": "...",
+  "mock_fixtures": {
+    "system_y": [{"123": {"name": "Flavio"}}, {"456": {"name": "Andrew"}}]
+  }
+}
+```
+
+- `test_types` — array, values MUST come from rule `04`'s enum, MUST contain at least one value. An entry MAY carry more than one value additively (e.g. `["functional", "smoke", "human"]`) — no test type excludes another.
+- `input` — for Prompt-tier components, a raw prompt string or the prompt template's input parameters object; for Agent/Workflow-tier components, the input attributes object passed to the component.
+- `expected_output` — the fields used to score the entry under each of its automated `test_types`: output attributes for an LLM-as-judge rubric, a target for vector-similarity scoring, or exact attribute values for strict comparison. When `human` is one of the entry's `test_types`, `expected_output` MUST additionally include a `human_test` string field with manual-verification instructions (e.g. `"check for ethical issues, verify record change in system X"`) — this supplements, and never replaces, the entry's automated scoring fields.
+- `mock_fixtures` — optional object; keys identify the adapter or external system to mock (SHOULD match the connector folder name under `adapters/connectors/<name>` for readability, though not enforced), values are any valid JSON interpreted by the mock implementation. When present, eval.py MUST configure each named mock adapter with its fixture data BEFORE invoking the component for that entry; each entry MUST use fresh mock instances to prevent state from bleeding across entries. `mock_fixtures` applies to all `test_types` including `human` — the component is still invoked for human entries to capture `actual_output`. `mock_fixtures` MUST NOT include keys for LLM adapters: all golden dataset test types are rated `mocks disallowed for LLM calls` (rule `03`), so the LLM call MUST always be real; LLM provider mocking belongs exclusively to unit tests via [agentme-edr-018](018-ai-llm-development-standards.md) rule `04`. See [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
+- The dataset's `dataset.schema.json` MUST require `test_types`, `input`, and `expected_output`, and SHOULD declare `mock_fixtures` as optional (`"type": "object", "additionalProperties": {}`), per [agentme-edr-024](024-ml-dataset-structure.md) rule `04`.
+
+#### 03-mocks-allowed-values
+
+The taxonomy in rule `05` rates each test type using one of three values:
+
+| Value | Meaning |
+|---|---|
+| `mocks allowed` | Fully offline; fakes may replace every dependency (e.g. `FakeListChatModel` per [agentme-edr-018](018-ai-llm-development-standards.md) rule `04`). |
+| `mocks disallowed` | No mocking of any dependency — real external systems required. |
+| `mocks disallowed for LLM calls` | Tools and other external/dependency calls MAY be mocked; only the LLM call itself MUST be real for the test to be meaningful. |
+
+#### 04-test-types-enum
+
+A golden dataset entry's `test_types` array MUST only use these values: `safety`, `adversarial`, `fairness`, `bias`, `robustness`, `explainability`, `groundedness`, `functional`, `prompt`, `smoke`, `human`. These correspond to the dataset-driven rows of rule `05`. **Unit test** and **Integration test** (the two Code-level rows) are NOT part of this enum — they have no golden dataset entries and remain governed entirely by [agentme-edr-004](../principles/004-unit-test-requirements.md) and [agentme-edr-007](../principles/007-project-quality-standards.md) rule `08`.
+
+#### 05-test-type-taxonomy
+
+Test types MUST be selected from this taxonomy. Each test type is named with its group, objective, mocking constraint, applicability, and relevance:
+
+| Test Type Name | Group | Test Objective | Mocks Allowed | When to Apply | Relevance – Business | Relevance – Development Team | Priority (1-5) |
+|---|---|---|---|---|---|---|---|
+| Safety/content eval | Safety & adversarial | Detect harmful, biased, or policy-violating output | mocks disallowed for LLM calls | Any user-facing release | Avoids reputational harm; acceptable-use compliance | Automated content gate before merge/release | 5 |
+| Adversarial/red-team test | Safety & adversarial | Probe for prompt injection, jailbreaks, unsafe tool use | mocks disallowed for LLM calls | System exposes tool-invocation or agent loops | Reduces security-incident/breach liability | Finds exploitable tool-loop paths before attackers do | 5 |
+| Fairness test | Responsible AI | Verify equitable outcomes across user groups | mocks disallowed for LLM calls | Output affects decisions about individuals/groups | Regulatory requirement; protects equitable access | Surfaces uneven outcomes before release | 4 |
+| Bias test | Responsible AI | Detect skewed or stereotyped associations | mocks disallowed for LLM calls | User-facing content generation | Lowers legal/reputational exposure | Catches bias introduced by data/prompts/fine-tuning | 3 |
+| Robustness test | Responsible AI | Verify stable behavior under noisy/out-of-distribution input | mocks disallowed for LLM calls | Inputs come from untrusted/variable sources | Protects reliability/SLAs | Confirms graceful degradation, guides input validation | 3 |
+| Explainability test | Responsible AI | Verify output is justifiable with a faithful rationale | mocks disallowed for LLM calls | Output must be justified to users/auditors/regulators | Required for auditability; builds user trust | Gives rationale trace for debugging wrong answers | 2 |
+| Groundedness (RAG) eval | Quality eval | Verify the answer is supported by retrieved context | mocks disallowed for LLM calls | System uses retrieval-augmented generation | Avoids confidently-wrong answers reaching customers | Pinpoints retrieval/prompt bugs | 4 |
+| Human evaluation | Quality eval | Manually verify aspects automated scoring can't (ethics, side effects, external state) | mocks disallowed for LLM calls | Before major releases; periodic spot-check | Defensible, human-reviewed sign-off | Catches what automated metrics miss | 3 |
+| Functional eval (golden-dataset accuracy / LLM-as-judge) | Quality eval | Measure output correctness against the golden dataset | mocks disallowed for LLM calls | Required before every Workflow release ([agentme-edr-007](../principles/007-project-quality-standards.md) rule `09`); advised elsewhere | Auditable evidence of business correctness before release | Detects regressions from model/provider/prompt changes | 5 |
+| Smoke test | Quality eval | Fast pass/fail check on a small, critical subset before running fuller suites | mocks disallowed for LLM calls | Every commit/PR, before functional/responsible-AI evals run | Cheap early warning before slower evals run | Fast, cheap feedback loop | 4 |
+| Prompt regression test | Prompt/LLM | Detect behavior change when a prompt or model version changes | mocks disallowed for LLM calls | Whenever a prompt template or model version changes | Prevents shipping a worse experience via a "small" tweak | Fast check on every prompt edit | 3 |
+| Integration test | Code-level | Verify real interaction with external systems | mocks disallowed | Component depends on external systems | Reduces production outages from integration mismatches | Catches wiring bugs unit tests can't see | 2 |
+| Unit test (offline, mocked) | Code-level | Verify deterministic logic in isolation, offline | mocks allowed | Required for Workflow tier every commit ([agentme-edr-007](../principles/007-project-quality-standards.md) rule `09`) | Lowest-cost point to catch defects | Fastest, fully offline feedback on every commit | 5 |
+
+#### 06-priority-and-relevance-are-descriptive-only
+
+Priority, Relevance, and When to Apply in rule `05` are guidance for prioritization conversations — they MUST NOT be treated as mandating which test types a project must implement, nor their thresholds. [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09` remains the only tier-level testing requirement in force (Workflow unit tests + functional evals). Once a project chooses to implement and threshold a test type, [agentme-edr-028](028-ai-eval-standards.md) rule `02`'s failing-threshold behavior applies uniformly, regardless of this table's priority rating — a project may enforce fairness at 70% and functional at 90%, or skip fairness entirely; that choice is a project/business decision, not one this Policy makes.
+
+#### 07-smoke-is-distinct-from-test-smoke
+
+The `smoke` test type (surfaced as the `eval-smoke` Makefile target, a fast subset of the golden-dataset functional eval) is a different concept from [agentme-edr-008](../devops/008-common-targets.md)'s existing `test-smoke` target (a fast subset of code-level tests). Both MAY exist in the same project; teams MUST NOT conflate them.
+
+## References
+
+- [agentme-edr-024](024-ml-dataset-structure.md) — Golden dataset file layout, per-entry JSON format, `$schema` pointer, and schema-lint validation
+- [agentme-edr-028](028-ai-eval-standards.md) — Eval folder structure, `--type` filtering, per-type Makefile targets, and per-type reports that consume this taxonomy
+- [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) — Rule `10`: `_mock` file naming and placement convention for mock adapters referenced by `mock_fixtures`
+- [agentme-edr-007](../principles/007-project-quality-standards.md) — Rule `09` tier-level testing requirements (the only mandated AI testing baseline)
+- [agentme-edr-008](../devops/008-common-targets.md) — Rule `03` `eval-<qualifier>` Makefile convention; rule `03`'s `test-smoke` (distinguished in rule `07`)
+- [agentme-edr-018](018-ai-llm-development-standards.md) — LLM tier definition and mocking utilities referenced by the `mocks allowed` value
+- [agentme-edr-004](../principles/004-unit-test-requirements.md) — Unit test requirements underlying the Code-level rows

package/.xdrs/agentme/edrs/devops/005-monorepo-structure.md

@@ -57,8 +57,6 @@ Module folder responsibilities, artifact locations, and test-folder conventions
 - **MUST** contain a `README.md` with: purpose, architecture overview, how to build, and how to run.
 - **MAY** contain `examples/`, `tests_integration/`, and `tests_benchmark/` when those artifacts apply to multiple modules inside the application.
 
-*Why:* Isolating applications prevents implicit coupling and makes the `shared/` boundary explicit and intentional.
-
 #### 03-module-folders
 
 - A module is a subfolder inside an application that is independently compilable and produces a build artifact.
@@ -87,8 +85,6 @@ Module Makefiles **SHOULD** also provide `lint-fix` and `install` when the under
 The root `Makefile` **MUST** also define a `setup` target that guides a new contributor to prepare their machine.
 The root `setup` target **MUST** run `mise install` and any small repository bootstrap required before routine targets work.
 
-*Why:* Makefiles provide a universal, stack-agnostic entry point regardless of programming language.
-
 #### 06-mise-for-tooling-management
 
 - [Mise](https://mise.jdx.dev/) **MUST** be used to pin all tool versions (compilers, runtimes, CLI tools).
@@ -100,8 +96,6 @@ The root `setup` target **MUST** run `mise install` and any small repository boo
 - If a required tool is missing, the first remediation step **MUST** be to update `.mise.toml` or run `mise install`, not to install ad-hoc global tools with language-specific installers such as `go install`, `npm install -g`, `pip install --user`, or `cargo install`.
 - Root and module `Makefile` targets **MUST** work when invoked as plain `make <target>` after `make setup`.
 
-*Why:* Eliminates "works on my machine" build failures by ensuring identical tool versions across all environments.
-
 #### 07-root-readme
 
 The root `README.md` **MUST** include: overview, machine setup, quickstart, and a repository map.
@@ -116,12 +110,12 @@ All releases **MUST** be tagged using the format `<module-name>/<semver>` (e.g.,
 
 `<module-name>` is preferably the path-like identifier of the module being released. A custom name is allowed but the folder name is strongly preferred.
 
-*Why:* Namespacing tags by module prevents collisions and makes it easy to filter release history when multiple modules release independently.
-
 ---
 
 #### 11-summary-of-requirements
 
+All requirements marked 'Yes' MUST be met. The table below summarizes the mandatory requirements:
+
 | Requirement | Scope | Mandatory |
 |---|---|---|
 | Lowercase folder/file names | All | Yes |

package/.xdrs/agentme/edrs/devops/006-github-pipelines.md

@@ -37,7 +37,7 @@ All workflows run on `ubuntu-latest`. Tool versions MUST be managed by Mise via
 
 File: `.github/workflows/ci.yml`
 
-Triggered on every PR targeting `main` and every push to `main`. Runs the standard `build`, `lint`, and `test` targets from the root Makefile and fails the workflow if any step exits non-zero.
+Projects MUST configure this workflow, triggered on every PR targeting `main` and every push to `main`. It runs the standard `build`, `lint`, and `test` targets from the root Makefile and fails the workflow if any step exits non-zero.
 
 ```yaml
 name: ci
@@ -67,7 +67,7 @@ jobs:
 
 File: `.github/workflows/release.yml`
 
-Manually dispatched (`workflow_dispatch`). Calculates the next semantic version tag using **monotag** and pushes that tag to the repository. Pushing the tag then automatically triggers the publish workflow.
+Projects MUST use this manually dispatched (`workflow_dispatch`) workflow. It calculates the next semantic version tag using **monotag** and pushes that tag to the repository. Pushing the tag then automatically triggers the publish workflow.
 
 The checkout step **must** use `fetch-depth: 0` so monotag can traverse the full commit history to determine the correct next version.
 
@@ -112,7 +112,7 @@ jobs:
 
 File: `.github/workflows/publish.yml`
 
-Triggered exclusively when a tag matching `v*.*.*` is pushed to the repository. This ensures only explicitly tagged commits produce published artifacts. Runs `make publish` against the tagged commit.
+Projects MUST configure this workflow, triggered exclusively when a tag matching `v*.*.*` is pushed to the repository. This ensures only explicitly tagged commits produce published artifacts. Runs `make publish` against the tagged commit.
 
 ```yaml
 name: publish

package/.xdrs/agentme/edrs/devops/008-common-targets.md

@@ -15,7 +15,7 @@ What standard set of Makefile target names and execution rules should projects a
 
 ## Decision Outcome
 
-**Every project must expose its development actions through a root `Makefile` using a defined set of standardized target names. Target implementation and tool-execution rules follow [agentme-edr-017](017-tool-execution-and-scripting.md), which requires `mise exec --` before routine tool commands.**
+**Every project MUST expose its development actions through a root `Makefile` using a defined set of standardized target names. Target implementation and tool-execution rules follow [agentme-edr-017](017-tool-execution-and-scripting.md), which requires `mise exec --` before routine tool commands.**
 
 Standardizing both the target names and the execution chain removes per-project guesswork, makes CI pipelines reusable, and keeps tooling behavior visible in one place.
 
@@ -23,18 +23,18 @@ Standardizing both the target names and the execution chain removes per-project
 
 #### 01-every-project-must-have-root-makefile
 
-The project root must contain a single authoritative `Makefile` that exposes the standard target names defined in rule 3. Developers and CI pipelines must invoke routine actions through this `Makefile`, never by calling underlying tools directly in documentation, CI, or daily workflow commands.
+The project root **MUST** contain a single authoritative `Makefile` that exposes the standard target names defined in rule 3. Developers and CI pipelines **MUST** invoke routine actions through this `Makefile`, **NEVER** by calling underlying tools directly in documentation, CI, or daily workflow commands.
 
 `make <target>` is the shared contract across projects and languages.
 
-- The root `Makefile` must be the entry point for both developers and pipelines.
-- The root `Makefile` must expose at minimum the common targets defined in this XDR.
-- Reverse-compatibility wrappers are allowed when an ecosystem expects them, but they must stay trivial.
+- The root `Makefile` **MUST** be the entry point for both developers and pipelines.
+- The root `Makefile` **MUST** expose at minimum the common targets defined in this XDR.
+- Reverse-compatibility wrappers are allowed when an ecosystem expects them, but they **MUST** stay trivial.
 	- Allowed: `package.json` script `"test": "make test"`
 	- Not allowed: `make test` -> `npm run test` -> tool command
-- Project logic must not live in npm scripts, Mise tasks, shell wrappers, or other secondary runners when the same logic belongs in the `Makefile`.
+- Project logic **MUST NOT** live in npm scripts, Mise tasks, shell wrappers, or other secondary runners when the same logic belongs in the `Makefile`.
 
-*Why:* The project entry point must stay language-agnostic and obvious. A developer should be able to inspect the `Makefile` and immediately see which real tool commands will run.
+*Why:* The project entry point **MUST** stay language-agnostic and obvious. A developer **SHOULD** be able to inspect the `Makefile` and immediately see which real tool commands will run.
 
 #### 02-makefile-recipes-must-use-mise
 
@@ -47,10 +47,10 @@ make <target>
 			-> explicit tool command
 ```
 
-- The `setup` target must run `mise install` and any small project-specific bootstrap needed before normal targets work.
-- Routine targets such as `build`, `lint`, `test`, `run`, and `publish` must be invoked as `make <target>` by both contributors and CI.
-- Each Makefile recipe must call the real underlying command through `mise exec --`, following [agentme-edr-017](017-tool-execution-and-scripting.md).
-- Makefile recipes must not add extra script layers such as `npm run`, `pnpm run`, `yarn run`, `mise run`, `mise tasks`, or shell aliases when those layers only forward to another command.
+- The `setup` target **MUST** run `mise install` and any small project-specific bootstrap needed before normal targets work.
+- Routine targets such as `build`, `lint`, `test`, `run`, and `publish` **MUST** be invoked as `make <target>` by both contributors and CI.
+- Each Makefile recipe **MUST** call the real underlying command through `mise exec --`, following [agentme-edr-017](017-tool-execution-and-scripting.md).
+- Makefile recipes **MUST NOT** add extra script layers such as `npm run`, `pnpm run`, `yarn run`, `mise run`, `mise tasks`, or shell aliases when those layers only forward to another command.
 - Calling the actual tool is allowed even when that tool itself launches another program as part of its normal interface.
 	- Allowed: `mise exec -- pnpm exec eslint ./src`
 	- Allowed: `mise exec -- go test -cover ./...`
@@ -66,17 +66,17 @@ make <target>
 
 #### 03-standard-target-groups-and-names
 
-Targets are organized into five lifecycle groups. Projects must use these names unchanged. Extensions are allowed (see rule 5) but the core names must not be repurposed.
+Targets are organized into five lifecycle groups. Projects **MUST** use these names unchanged. Extensions are allowed (see rule 5) but the core names **MUST NOT** be repurposed.
 
 ##### Developer group
 
 | Target | Purpose |
 |--------|---------|
 | `setup` | Run `mise install` and any small project bootstrap needed before normal targets work. This is the first command after checkout. |
-| `all` | Alias that runs `build`, `lint`, and `test` in sequence. Must be the default target (i.e., running `make` or the runner with no arguments invokes `all`). Used by developers as a fast pre-push check to verify the software meets minimum quality standards in one command. Must only invoke targets that run **offline** — no external credentials, running servers, paid APIs, or environment-specific configuration outside the repository. |
+| `all` | Alias that runs `build`, `lint`, and `test` in sequence. **MUST** be the default target (i.e., running `make` or the runner with no arguments invokes `all`). Used by developers as a fast pre-push check to verify the software meets minimum quality standards in one command. **MUST** only invoke targets that run **offline** — no external credentials, running servers, paid APIs, or environment-specific configuration outside the repository. |
 | `clean` | Remove all temporary or generated files created during build, lint, or test (e.g., `node_modules`, virtual environments, compiled binaries, generated files). Used both locally and in CI for a clean slate. |
 | `dev` | Run the software locally for development (e.g., start a Node.js API server, open a Jupyter notebook, launch a React dev server). May have debugging tools, verbose logging, or hot reloading features enabled. |
-| `run` | Run the software in production mode (e.g., start a compiled binary, launch a production server). No debugging or development-only features should be enabled. |
+| `run` | Run the software in production mode (e.g., start a compiled binary, launch a production server). Debugging or development-only features **SHOULD NOT** be enabled. |
 | `update-lockfile` | Update the dependency lockfile to reflect the latest resolved versions of all dependencies. |
 
 ##### Build group
@@ -93,17 +93,17 @@ Targets are organized into five lifecycle groups. Projects must use these names
 
 | Target | Purpose |
 |--------|---------|
-| `lint` | Run **all static quality checks** outside of tests. This MUST include: code formatting validation, code style enforcement, code smell detection, static analysis, dependency audits for known CVEs, security vulnerability scans (e.g., SAST), and project/configuration structure checks. All checks must be non-destructive (read-only); fixes are handled by `lint-fix`. Must only invoke subtargets that run **offline** (no external credentials or services). |
+| `lint` | Run **all static quality checks** outside of tests. This MUST include: code formatting validation, code style enforcement, code smell detection, static analysis, dependency audits for known CVEs, security vulnerability scans (e.g., SAST), and project/configuration structure checks. All checks **MUST** be non-destructive (read-only); fixes are handled by `lint-fix`. **MUST** only invoke subtargets that run **offline** (no external credentials or services). |
 | `lint-fix` | Automatically fix linting and formatting issues where possible. || `lint-format` | *(Optional)* Check code formatting only (e.g., Prettier, gofmt, Black). |
 ##### Test group
 
 | Target | Purpose |
 |--------|---------|
-| `test` | Run **all offline tests** required for the project. This MUST include unit tests (with coverage enforcement — the build MUST fail if coverage thresholds are not met) and any integration or end-to-end tests that run **offline** (no external servers, credentials, or paid APIs). Normally delegates to `test-unit` and, when offline, `test-integration` in sequence. Suffixed targets that require external dependencies must not be invoked automatically — see rule 08. |
+| `test` | Run **all offline tests** required for the project. This MUST include unit tests (with coverage enforcement — the build MUST fail if coverage thresholds are not met) and any integration or end-to-end tests that run **offline** (no external servers, credentials, or paid APIs). Normally delegates to `test-unit` and, when offline, `test-integration` in sequence. Suffixed targets that require external dependencies **MUST NOT** be invoked automatically — see rule 08. |
 | `test-unit` | Run unit tests only, including coverage report generation and coverage threshold enforcement. |
-| `test-integration` | *(Optional)* Run integration and end-to-end tests only. Projects without integration tests may omit this target. |
+| `test-integration` | *(Optional)* Run integration and end-to-end tests only. Projects without integration tests MAY omit this target. |
 | `test-smoke` | *(Optional)* Run a fast, minimal subset of tests to verify the software is basically functional. Useful as a post-deploy health check. |
-| `eval` | *(Optional)* Run **all evaluations** for the module. Used alongside `test` to measure the accuracy and performance of statistical systems such as ML models, AI agents, or noisy systems. Typically runs against a live or near-live system (similar to an integration test) and produces a performance analysis report (e.g., F1 score, Accuracy, Precision, Recall). Must not be included in `test` or `all` — evals are opt-in because they require live dependencies and may be slow or costly to run. Individual evaluations must follow the prefix convention: `eval-<qualifier>` (e.g., `eval-simple`, `eval-complex`). |
+| `eval` | *(Optional)* Run **all evaluations** for the module. Used alongside `test` to measure the accuracy and performance of statistical systems such as ML models, AI agents, or noisy systems. Typically runs against a live or near-live system (similar to an integration test) and produces a performance analysis report (e.g., F1 score, Accuracy, Precision, Recall). **MUST NOT** be included in `test` or `all` — evals are opt-in because they require live dependencies and MAY be slow or costly to run. Individual evaluations **MUST** follow the prefix convention: `eval-<qualifier>` (e.g., `eval-simple`, `eval-complex`). |
 
 ##### Release group
 
@@ -119,18 +119,18 @@ Targets are organized into five lifecycle groups. Projects must use these names
 
 #### 04-standard-environment-variables
 
-Two environment variables have defined semantics and must be used consistently.
+Two environment variables have defined semantics and **MUST** be used consistently.
 
 | Variable | Purpose |
 |----------|---------|
-| `STAGE` | Identifies the runtime environment. Format: `[prefix][-variant]`. Common prefixes: `dev`, `tst`, `acc`, `prd`. Examples: `dev`, `dev-pr123`, `tst`, `prd-blue`. May be required by any target that is environment-aware (build, lint, deploy, etc.). |
+| `STAGE` | Identifies the runtime environment. Format: `[prefix][-variant]`. Common prefixes: `dev`, `tst`, `acc`, `prd`. Examples: `dev`, `dev-pr123`, `tst`, `prd-blue`. **MAY** be required by any target that is environment-aware (build, lint, deploy, etc.). |
 | `VERSION` | Sets the explicit version used during packaging and deployment. Used when there is no automatic version-tagging utility, or to override it. |
 
 ---
 
 #### 05-extending-targets-with-prefixes
 
-Projects may add custom targets beyond the standard set. Custom targets must be named by prefixing a standard target name with a descriptive qualifier, keeping the naming intuitive and consistent with the group it belongs to.
+Projects **MAY** add custom targets beyond the standard set. Custom targets **MUST** be named by prefixing a standard target name with a descriptive qualifier, keeping the naming intuitive and consistent with the group it belongs to.
 
 **Examples:**
 
@@ -166,15 +166,15 @@ dev-mlflow:
 
 #### 08-default-targets-must-only-include-offline-subtargets
 
-`make all`, `make test`, and `make lint` must include every subtarget that runs **offline** — meaning it requires no external credentials, no running servers, no paid APIs, and no environment-specific configuration outside the repository.
+`make all`, `make test`, and `make lint` **MUST** include every subtarget that runs **offline** — meaning it requires no external credentials, no running servers, no paid APIs, and no environment-specific configuration outside the repository.
 
-Subtargets that require external dependencies (e.g., `test-integration` against a live database, `test-e2e` against a staging environment, `lint-api` against a remote schema registry) **must** exist as named targets so developers can invoke them explicitly, but **must not** be invoked from `all`, `test`, or `lint`.
+Subtargets that require external dependencies (e.g., `test-integration` against a live database, `test-e2e` against a staging environment, `lint-api` against a remote schema registry) **MUST** exist as named targets so developers can invoke them explicitly, but **MUST NOT** be invoked from `all`, `test`, or `lint`.
 
 ---
 
 #### 06-monorepo-usage
 
-In a monorepo, each module has its own `Makefile` with its own `build`, `lint`, `test`, and `deploy` targets scoped to that module. Parent-level Makefiles (at the application or repo root) delegate to child Makefiles in sequence. The parent Makefile should call `$(MAKE) -C <child> <target>` directly, while each child `Makefile` runs its actual tool commands through `mise exec --`.
+In a monorepo, each module has its own `Makefile` with its own `build`, `lint`, `test`, and `deploy` targets scoped to that module. Parent-level Makefiles (at the application or repo root) delegate to child Makefiles in sequence. The parent Makefile **SHOULD** call `$(MAKE) -C <child> <target>` directly, while each child `Makefile` runs its actual tool commands through `mise exec --`.
 
 ```makefile
 # root Makefile — delegates to all modules
@@ -187,7 +187,7 @@ test:
 	$(MAKE) -C module-b test
 ```
 
-A developer can run `make test` at the repo root to test everything, or `cd module-a && make test` to test a single module. Both must work.
+A developer can run `make test` at the repo root to test everything, or `cd module-a && make test` to test a single module. Both **MUST** work.
 
 **Reference:** See [agentme-edr-005](005-monorepo-structure.md) for the full monorepo layout convention.
 
@@ -195,7 +195,7 @@ A developer can run `make test` at the repo root to test everything, or `cd modu
 
 #### 07-quick-reference
 
-Any project following this EDR supports the following actions through the root `Makefile`.
+Any project following this EDR MUST support the following actions through the root `Makefile`.
 
 ```sh
 # install the pinned toolchain and project bootstrap

package/.xdrs/agentme/edrs/devops/027-environment-variable-configuration.md

@@ -25,7 +25,7 @@ Secrets (API keys, passwords, tokens) must never be placed in `.env` files. Thos
 
 #### 01-when-to-use-dotenv
 
-Use a `.env` file when either of the following is true:
+A `.env` file MUST be used when either of the following is true:
 
 1. **Spawned process needs env vars** — the project launches a process (a deployable service, background worker, or shell script) that reads configuration from OS environment variables such as port numbers or API endpoint URLs.
 2. **Value must not be committed** — a configuration value used in a YAML config file (see rule 07) is environment-specific or sensitive enough to exclude from version control. In that case, store the value in `.env` and reference it from the YAML file using env var substitution (see rule 08).
@@ -43,13 +43,13 @@ FEATURE_FLAG_NEW_UI=false
 
 #### 02-dotenv-not-committed
 
-`.env` must be listed in `.gitignore` and must never be committed to the repository. It is intended for local use in standalone projects and libraries that do not have a formal deployment pipeline.
+`.env` MUST be listed in `.gitignore` and must never be committed to the repository. It is intended for local use in standalone projects and libraries that do not have a formal deployment pipeline.
 
 ---
 
 #### 03-dotenv-example-committed
 
-A `.env.example` file must be committed alongside `.env`. It contains all the same variable names with placeholder or illustrative values — no real URLs, credentials, or server names. This file documents what configuration is expected without exposing real values.
+A `.env.example` file MUST be committed alongside `.env`. It contains all the same variable names with placeholder or illustrative values — no real URLs, credentials, or server names. This file documents what configuration is expected without exposing real values.
 
 Example `.env.example`:
 ```
@@ -62,7 +62,7 @@ FEATURE_FLAG_NEW_UI=false
 
 #### 04-stage-specific-dotenv-committed
 
-Stage-specific overrides must use the naming convention `.env.[stage]` (e.g., `.env.production`, `.env.staging`, `.env.test`). These files may be committed to the repository because they carry deployment-stage configuration rather than local developer configuration. They are used during deployment pipelines where the stage is known and explicit.
+Stage-specific overrides MUST use the naming convention `.env.[stage]` (e.g., `.env.production`, `.env.staging`, `.env.test`). These files may be committed to the repository because they carry deployment-stage configuration rather than local developer configuration. They are used during deployment pipelines where the stage is known and explicit.
 
 The generic `.env` must still not be committed. The distinction is: `.env` is for local, ad-hoc, standalone use; `.env.[stage]` is for deployment pipelines with a defined environment identity.
 
@@ -70,7 +70,7 @@ The generic `.env` must still not be committed. The distinction is: `.env` is fo
 
 #### 05-load-in-makefile-before-processes
 
-When `.env` defines variables consumed by shell scripts or spawned processes, the Makefile must load and export them before invoking those processes. Use the following pattern at the top of the relevant Makefile or in a shared include:
+When `.env` defines variables consumed by shell scripts or spawned processes, the Makefile MUST load and export them before invoking those processes. Use the following pattern at the top of the relevant Makefile or in a shared include:
 
 ```makefile
 ifneq (,$(wildcard .env))
@@ -85,7 +85,7 @@ This ensures all variables in `.env` are available as environment variables to e
 
 #### 06-no-application-level-dotenv-loading
 
-Applications must not load `.env` files directly inside their own code using dotenv libraries or equivalent mechanisms. Configuration must enter the process exclusively as OS-level environment variables, set before the process is launched (by the Makefile, a shell script, CI, or a container runtime).
+Applications MUST NOT load `.env` files directly inside their own code using dotenv libraries or equivalent mechanisms. Configuration must enter the process exclusively as OS-level environment variables, set before the process is launched (by the Makefile, a shell script, CI, or a container runtime).
 
 Prohibited patterns:
 
@@ -114,7 +114,7 @@ This rule prevents two parallel loading paths — OS env and file-based env —
 
 #### 07-cli-adapters-use-yaml-config
 
-CLI adapters with multiple configuration attributes must use a YAML config file rather than env vars or flags for those attributes. This applies whenever configuration is nested, repetitive, or too verbose for flags alone.
+CLI adapters with multiple configuration attributes MUST use a YAML config file rather than env vars or flags for those attributes. This applies whenever configuration is nested, repetitive, or too verbose for flags alone.
 
 The CLI layer is responsible for loading and parsing the YAML file and passing the resolved values to the application layer. The application layer must not read the config file directly.
 
@@ -131,7 +131,7 @@ max_retries: 3
 
 #### 08-env-var-substitution-in-config-files
 
-When a YAML config file contains a value that must not be committed (such as a real endpoint URL, a username, or any other environment-specific value), that value must be expressed as an environment variable reference using `${VAR_NAME}` syntax, and the actual value must be defined in `.env`.
+When a YAML config file contains a value that MUST NOT be committed (such as a real endpoint URL, a username, or any other environment-specific value), that value must be expressed as an environment variable reference using `${VAR_NAME}` syntax, and the actual value must be defined in `.env`.
 
 This keeps the YAML file committable while keeping the environment-specific value out of the repository.
 

package/.xdrs/agentme/edrs/governance/013-contributing-guide-requirements.md

@@ -15,9 +15,9 @@ What contributor workflow guidance must every project publish so contributors kn
 
 ## Decision Outcome
 
-**Every project must publish a root CONTRIBUTING.md with a small, explicit contribution workflow.**
+**Every project MUST publish a root CONTRIBUTING.md with a small, explicit contribution workflow.**
 
-Projects must keep a `CONTRIBUTING.md` file at the repository root. The file must explain where bugs, feature discussions, and code changes belong so contributors follow a predictable workflow before opening pull requests.
+Projects MUST keep a `CONTRIBUTING.md` file at the repository root. The file MUST explain where bugs, feature discussions, and code changes belong so contributors follow a predictable workflow before opening pull requests.
 
 ### Details
 

package/.xdrs/agentme/edrs/index.md

@@ -37,6 +37,7 @@ Language and framework-specific tooling and project structure.
 - [agentme-edr-021](application/021-ai-workflow-development-standards.md) - **AI workflow development standards** - Standard toolchain (LangGraph), evaluation, and testing patterns for workflow projects
 - [agentme-edr-029](application/029-ai-workflow-naming-conventions.md) - **AI workflow naming conventions** - Node suffix/prefix roles, state type and attribute naming, judge output schema, workflow class names, and cross-element coherence rules
 - [agentme-edr-028](application/028-ai-eval-standards.md) - **AI eval standards** - Folder structure, script requirements, and MLflow tracking for eval tests across LLM, Agent, and Workflow tiers
+- [agentme-edr-030](application/030-ai-test-types-taxonomy.md) - **AI test types taxonomy** - Names AI test types (safety, responsible-AI, quality-eval, prompt, code-level) with group, objective, mocking constraint, and relevance, and defines the shared golden dataset entry envelope
 - [agentme-edr-024](application/024-ml-dataset-structure.md) - **ML dataset structure** - Standard folder layout and file conventions for ML datasets
 - [agentme-edr-025](application/025-ai-agent-xdrs-knowledge-layer.md) - **AI agent XDRS knowledge layer** - How to integrate XDRS as the runtime source of truth for policies and skills in AI agents (apply only when the project explicitly uses XDRS)
 - [agentme-edr-026](application/026-pragmatic-hexagonal-architecture.md) - **Pragmatic hexagonal architecture** - Organize application layers as External/Adapters/Application with practical coupling rules

package/.xdrs/agentme/edrs/observability/011-service-health-check-endpoint.md

@@ -17,7 +17,7 @@ How should services expose their health status and validate operational readines
 
 **Standardized `/health` endpoint with dependency validation**
 
-All services must expose a `GET /health` endpoint that validates external dependencies using read-only operations and returns structured status with appropriate HTTP codes.
+All services **MUST** expose a `GET /health` endpoint that validates external dependencies using read-only operations and returns structured status with appropriate HTTP codes.
 
 ### Details
 
@@ -45,12 +45,12 @@ All services must expose a `GET /health` endpoint that validates external depend
 
 - `health` (required): overall state — `OK`, `WARNING`, or `ERROR`
 - `latencyMs` (required): total milliseconds to run all checks
-- `message` (required): human-readable summary; must never expose credentials, internal IPs, or stack traces
+- `message` (required): human-readable summary; **MUST NEVER** expose credentials, internal IPs, or stack traces
 
 **Dependency validation rules:**
 
 - Check all external dependencies (databases, downstream APIs, queues, caches) using read-only operations (e.g., `SELECT 1`, lightweight GET, connection ping)
-- Never execute write operations or create side effects
+- **MUST NOT** execute write operations or create side effects
 - `OK`: all dependencies healthy within expected thresholds
 - `WARNING`: non-critical dependency degraded, or elevated but acceptable response times
 - `ERROR`: critical dependency unavailable or service unable to process requests
@@ -70,6 +70,28 @@ All services must expose a `GET /health` endpoint that validates external depend
 - Monitoring: periodic polling with alerts on `210` and `503` responses
 - CI/CD: poll `/health` to confirm deployment success
 
+**Unit testing and mocking strategy:** Each dependency checker MUST be injectable so unit tests can simulate `OK`, `WARNING`, and `ERROR` states independently without a real database or API. Integration tests MUST NOT mock dependency checkers — they MUST run against real dependencies to verify the wiring.
+
+```typescript
+// Good — injectable checkers; unit test controls each state
+function buildHealthHandler(checkers: DependencyChecker[]) {
+  return async () => {
+    const results = await Promise.all(checkers.map(c => c.check()));
+    const status = results.some(r => r.status === "ERROR") ? "ERROR"
+      : results.some(r => r.status === "WARNING") ? "WARNING" : "OK";
+    return { health: status };
+  };
+}
+
+it("returns ERROR when the database checker fails", async () => {
+  const handler = buildHealthHandler([
+    { check: async () => ({ name: "db", status: "ERROR" }) },
+  ]);
+  const response = await handler();
+  expect(response.health).toBe("ERROR");
+});
+```
+
 ## Considered Options
 
 * (REJECTED) **No health checks** — detect failures through request errors

package/.xdrs/agentme/edrs/principles/002-coding-best-practices.md

@@ -21,12 +21,10 @@ What coding practices should be followed across all languages and projects to ke
 
 #### 01-keep-files-short
 
-A file must not exceed **400 lines**. When a file grows beyond this limit, split related functions or types into separate, focused modules.
+A file MUST NOT exceed **400 lines**. When a file grows beyond this limit, split related functions or types into separate, focused modules.
 
 One exception are test files, which normally are bigger than the tested resources.
 
-*Why:* Large files make navigation slow, increase merge conflicts, and obscure the single-responsibility principle.
-
 **Example (TypeScript):**
 
 ```
@@ -48,9 +46,7 @@ src/
 
 #### 02-apply-template-method-pattern
 
-When a function's main logic contains well-defined sections and **any individual section exceeds ~20 lines**, extract each section into its own named function. The outer function becomes an orchestrator that calls the extracted helpers in sequence.
-
-*Why:* Named sub-functions serve as inline documentation, are independently testable, and reduce cognitive load.
+When a function's main logic contains well-defined sections and **any individual section exceeds ~20 lines**, each section MUST be extracted into its own named function. The outer function becomes an orchestrator that calls the extracted helpers in sequence.
 
 **Example (Python):**
 
@@ -85,9 +81,7 @@ def _persist_order(order, total): ...
 
 #### 03-put-entry-point-function-first
 
-Place the **entry-point function** (the outermost caller) at the **top** of the file. All helper or sub-functions it calls internally must appear **below** it.
-
-*Why:* Readers can follow the overall logic top-down without jumping around the file. The most important function is immediately visible when the file is opened.
+Place the **entry-point function** (the outermost caller) at the **top** of the file. All helper or sub-functions it calls internally MUST appear **below** it.
 
 **Example (Python):**
 
@@ -106,27 +100,23 @@ def _persist_order(order, total): ...
 
 #### 04-keep-readme-tests-and-examples-in-sync
 
-Every change to a public interface, behavior, or configuration option must be reflected in:
+Every change to a public interface, behavior, or configuration option MUST be reflected in:
 
 - `README.md` — update usage examples, option tables, and feature descriptions.
 - Unit/integration tests — update or add tests that cover the changed behavior.
 - `examples/` resources — update runnable examples so they continue to work.
 
-*Why:* Stale documentation and broken examples erode trust and waste time for consumers of the code.
-
 ---
 
 #### 05-declare-types-in-file-where-used
 
-If a type (struct, interface, class, typedef, etc.) is used in only **one** file, declare it in that same file. Move a type to a shared module only when it is referenced in two or more files.
-
-*Why:* Co-locating a type with its sole consumer removes the need to navigate to a separate types file and makes the type's purpose immediately obvious from context.
+Types used in only **one** file MUST be declared in that same file. Move a type to a shared module only when it is referenced in two or more files.
 
 ---
 
 #### 06-keep-test-files-next-to-source
 
-Where the language ecosystem supports it (e.g. JavaScript/TypeScript, Go, Rust), place test files **beside** the source file they cover and use a consistent naming convention rather than mirroring the source tree in a separate `tests/` folder.
+Where the language ecosystem supports it (e.g. JavaScript/TypeScript, Go, Rust), test files MUST be placed **beside** the source file they cover and use a consistent naming convention rather than mirroring the source tree in a separate `tests/` folder.
 
 **Recommended naming conventions:**
 

package/.xdrs/agentme/edrs/principles/004-unit-test-requirements.md

@@ -21,7 +21,7 @@ What unit testing practices should be followed to ensure tests are meaningful, r
 
 #### 01-must-have-at-least-one-assertion-per-test
 
-```typescript
+Every test MUST have at least one assertion that validates the expected behavior.
 // bad — no assertion; passes even when code is broken
 it("processes the order", () => { processOrder(mockOrder); });
 
@@ -36,7 +36,7 @@ it("processes the order and returns a confirmation id", () => {
 
 #### 02-must-run-offline
 
-Unit tests must not depend on any external resources: no network calls, no running databases, no external APIs, no file system paths outside the repo. Tests must pass with only static code available.
+Unit tests MUST NOT depend on any external resources: no network calls, no running databases, no external APIs, no file system paths outside the repo. Tests must pass with only static code available.
 
 ```typescript
 // bad — hits a real HTTP endpoint
@@ -64,13 +64,13 @@ export default defineConfig({
 });
 ```
 
-Builds that miss the threshold must not be merged.
+Builds that miss the threshold MUST NOT be merged.
 
 ---
 
 #### 04-must-place-test-files-alongside-source
 
-Test files must live next to the source file they test, in the same directory, following the convention of the language/framework:
+Test files MUST live next to the source file they test, in the same directory, following the convention of the language/framework:
 
 | Language | Pattern | Example |
 |----------|---------|-------|
@@ -96,7 +96,7 @@ Do not flatten or reorganize paths when using a separate test folder.
 
 #### 05-should-extract-shared-setup
 
-When setup logic is repeated across two or more test files, centralize it (`src/test-utils/`, `internal/testutil/`, `tests/conftest.py`).
+When setup logic is repeated across two or more test files, it SHOULD be centralized (`src/test-utils/`, `internal/testutil/`, `tests/conftest.py`).
 
 ```typescript
 // src/test-utils/order-factory.ts
@@ -109,7 +109,7 @@ export function makeOrder(overrides: Partial<Order> = {}): Order {
 
 #### 06-should-avoid-mocks
 
-Use the lowest-cost alternative that exercises real behavior:
+Tests SHOULD use the lowest-cost alternative that exercises real behavior:
 
 1. **Real implementation** — always prefer this
 2. **In-memory / lightweight fake** — e.g. in-memory DB, stub HTTP server