agentme 0.25.2 → 0.27.0

package/.filedist-package.yml

@@ -1,5 +1,5 @@
 sets:
-  - package: xdrs-core@0.38.3
+  - package: xdrs-core@0.38.4
   # - package: git:https://github.com/flaviostutz/xdrs-core.git@main
     selector:
       files:

package/.xdrs/agentme/edrs/application/003-javascript-project-tooling.md

@@ -61,7 +61,7 @@ coverageProvider: 'v8',
 coverageDirectory: '.cache/coverage',
 ```
 
-Builds that miss the threshold must not be merged.
+Builds that miss the threshold MUST NOT be merged.
 
 #### Project structure
 
@@ -105,7 +105,7 @@ Internal source code MUST be organized following [agentme-edr-026](026-pragmatic
 
 When a repository contains multiple JavaScript/TypeScript packages, each package MUST live in its own module folder such as `lib/my-package/` or `services/my-service/`, each with its own `Makefile`, `README.md`, `dist/`, and `.cache/`.
 
-All tool caches, incremental state files, and workspace-local config outputs MUST be written under `.cache/`. This applies to every tool without exception. Cache and state paths MUST be declared in the tool's own configuration file — never on the command line — so that the location is enforced regardless of how the tool is invoked:
+All tool caches, incremental state files, and workspace-local config outputs MUST be written under `.cache/`. This applies to every tool without exception. Cache and state paths MUST be declared in the tool's own configuration file — MUST NOT be on the command line — so that the location is enforced regardless of how the tool is invoked:
 
 | Tool | Config file | Setting | Value |
 |------|------------|---------|-------|
@@ -114,7 +114,7 @@ All tool caches, incremental state files, and workspace-local config outputs MUS
 | **TypeScript** | `tsconfig.json` | `tsBuildInfoFile` | `.cache/tsbuildinfo` |
 | **Jest coverage** | `jest.config.js` | `coverageDirectory` | `.cache/coverage` |
 
-No tool MUST write cache or state files to the project root, `src/`, or any other directory outside `.cache/`. Passing cache paths as Makefile or CLI flags instead of config-file settings is not allowed.
+Tools MUST NOT write cache or state files to the project root, `src/`, or any other directory outside `.cache/`. Passing cache paths as Makefile or CLI flags instead of config-file settings is not allowed.
 
 Contributors and CI MUST invoke the commands below as `make <target>`. The Makefile recipes themselves MUST call the underlying tools through `mise exec -- <tool> ...`.
 
@@ -144,7 +144,7 @@ Contributors and CI MUST invoke the commands below as `make <target>`. The Makef
 
 Each sub-folder under `examples/` is an independent package. The Makefile installs the locally built `.tgz` pack from `lib/dist/` so examples simulate real external usage.
 
-Examples MUST remain outside the module root and MUST consume the package through the packed artifact in `dist/`, never through `../src` imports or other direct source links.
+Examples MUST remain outside the module root and MUST consume the package through the packed artifact in `dist/`. MUST NOT use `../src` imports or other direct source links.
 
 Module-specific integration tests that are not just runnable examples belong in `lib/tests_integration/` or a sibling `tests_integration/` when they cover multiple modules.
 

package/.xdrs/agentme/edrs/application/010-golang-project-tooling.md

@@ -30,7 +30,7 @@ A predictable layout and minimal external tooling keep Go projects approachable,
 | **golangci-lint** | Linting — aggregates many linters in one fast run; configured via `.golangci.yml` |
 | **monotag** | Version tagging from git history for the `publish` target |
 
-All commands are run exclusively through the Makefile, never ad-hoc. The project root **MUST** define a `.mise.toml` that pins `go`, `golangci-lint`, and any other Go-related CLIs used by the project. Contributors and CI **MUST** bootstrap with `make setup` or `mise install`, then invoke routine work with `make <target>`. Each Makefile recipe **MUST** execute the underlying tool through `mise exec -- <tool> ...`, following [agentme-edr-017](../devops/017-tool-execution-and-scripting.md).
+All commands MUST be run exclusively through the Makefile and MUST NOT be called ad-hoc. The project root **MUST** define a `.mise.toml` that pins `go`, `golangci-lint`, and any other Go-related CLIs used by the project. Contributors and CI **MUST** bootstrap with `make setup` or `mise install`, then invoke routine work with `make <target>`. Each Makefile recipe **MUST** execute the underlying tool through `mise exec -- <tool> ...`, following [agentme-edr-017](../devops/017-tool-execution-and-scripting.md).
 Direct installation of project-required Go CLIs with `go install ...@latest` as a repair step is **NOT** allowed unless an XDR for that repository explicitly permits it.
 
 #### Project structure
@@ -80,7 +80,7 @@ Direct installation of project-required Go CLIs with `go install ...@latest` as
 - Business logic lives in named feature packages under `app/` (e.g., `app/ownership/`, `app/changes/`). These packages are importable and testable without any CLI or adapter concerns.
 - `adapters/cli/` packages own flag parsing, output formatting, and the wiring between flags and `app/` functions. No business logic lives in adapter packages.
 - Outbound adapters live under `adapters/connectors/` with one subfolder per external resource, named descriptively (e.g., `postgres/`, `stripe-api/`, `redis-cache/`).
-- `shared/` must contain only infrastructure-agnostic utilities — not business rules or domain logic.
+- `shared/` MUST contain only infrastructure-agnostic utilities — not business rules or domain logic.
 - Packages are flat by default; sub-packages are only introduced when a feature package itself exceeds ~400 lines or has clearly separable sub-concerns.
 - Application MAY import from Adapters when it simplifies the design (pragmatic coupling per edr-022 rule 05).
 - Consumer examples for reusable libraries belong in a sibling `examples/` folder and MUST import the public module path rather than reaching into internal source paths. Because Go libraries are not typically consumed from a local packaged artifact, local example validation may use a temporary module replacement for resolution, but the import path MUST remain the public module path.
@@ -120,7 +120,7 @@ make test
 make lint
 ```
 
-The Makefile recipes themselves must use `mise exec --` for the underlying tool commands.
+The Makefile recipes themselves MUST use `mise exec --` for the underlying tool commands.
 
 #### Cross-platform binary distribution
 
@@ -148,7 +148,7 @@ All tool caches, incremental state files, and build outputs MUST be written unde
 | **golangci-lint cache** | `GOLANGCI_LINT_CACHE` env var | `export GOLANGCI_LINT_CACHE := $(CURDIR)/.cache/golangci-lint` |
 | **Test coverage output** | `-coverprofile` flag in `test` target | `.cache/coverage.out` |
 
-No tool MUST write cache or state files to the project root or any directory outside `.cache/`. Passing cache paths as per-recipe environment overrides instead of top-level Makefile exports is not allowed.
+Tools MUST NOT write cache or state files to the project root or any directory outside `.cache/`. Passing cache paths as per-recipe environment overrides instead of top-level Makefile exports is not allowed.
 
 #### Linting
 

package/.xdrs/agentme/edrs/application/014-python-project-tooling.md

@@ -34,13 +34,13 @@ A single dependency manager, isolated package internals under `lib/`, and a stan
 | **pytest-cov** | Coverage reporting and threshold enforcement |
 | **pip-audit** | Dependency CVE audit |
 
-All routine commands must run through the project `Makefile`, never by calling `uv`, `ruff`, `pytest`, or `ty` directly in docs, CI, or daily development workflows.
+All routine commands MUST run through the project `Makefile`. MUST NOT call `uv`, `ruff`, `pytest`, or `ty` directly in docs, CI, or daily development workflows.
 
 The repository root MUST define a `.mise.toml` that pins Python and uv. Contributors and CI MUST bootstrap with `make setup` or `mise install`, then invoke routine work with `make <target>`. Each Makefile recipe MUST execute the underlying tool through `mise exec -- <tool> ...`, following [agentme-edr-017](../devops/017-tool-execution-and-scripting.md). Using routine project CLI commands directly outside the Makefile contract is not allowed.
 
-The root `.venv/` is the canonical environment location for both the library and all examples. Subdirectory commands must set `UV_PROJECT_ENVIRONMENT` to the workspace root `.venv/` instead of creating nested virtual environments.
+The root `.venv/` is the canonical environment location for both the library and all examples. Subdirectory commands MUST set `UV_PROJECT_ENVIRONMENT` to the workspace root `.venv/` instead of creating nested virtual environments.
 
-All tool caches, incremental state files, and workspace-local outputs MUST be written under `.cache/`. Cache paths MUST be declared in the tool's own configuration file — never on the command line or as Makefile CLI flags — so the location is enforced regardless of how the tool is invoked. Configure the following in `lib/pyproject.toml`:
+All tool caches, incremental state files, and workspace-local outputs MUST be written under `.cache/`. Cache paths MUST be declared in the tool's own configuration file — MUST NOT be on the command line or as Makefile CLI flags — so the location is enforced regardless of how the tool is invoked. Configure the following in `lib/pyproject.toml`:
 
 | Tool | Config section | Setting | Value |
 |------|---------------|---------|-------|
@@ -50,7 +50,7 @@ All tool caches, incremental state files, and workspace-local outputs MUST be wr
 | **coverage HTML** | `[tool.coverage.html]` | `directory` | `".cache/coverage-html"` |
 | **uv** | `[tool.uv]` in `lib/pyproject.toml` | `cache-dir` | `".cache/uv"` |
 
-No tool MUST write cache or state files to the project root, `src/`, `tests/`, or any directory outside `.cache/`. Passing cache paths as CLI flags or Makefile recipe-level env overrides instead of `pyproject.toml` settings is not allowed.
+Tools MUST NOT write cache or state files to the project root, `src/`, `tests/`, or any directory outside `.cache/`. Passing cache paths as CLI flags or Makefile recipe-level env overrides instead of `pyproject.toml` settings is not allowed.
 
 #### Project structure
 
@@ -98,7 +98,7 @@ Use the `lib/src/` layout for import safety and packaging clarity. Keep tests un
 
 Internal source code MUST be organized following [agentme-edr-026](026-pragmatic-hexagonal-architecture.md): `adapters/` (inbound and outbound I/O boundaries), `app/` (business logic), and `shared/` (infrastructure-agnostic utilities).
 
-Libraries and shared utilities must include an `examples/` folder and wire example execution into the root `test` flow, following [agentme-edr-007](../principles/007-project-quality-standards.md). Each example directory is its own Python project with its own `pyproject.toml`, and examples must import the library as a consumer would rather than reaching back into `lib/src/` with relative imports. Local example verification must install the wheel built into `lib/dist/`; do not use editable or path-based dependencies back to `lib/`.
+Libraries and shared utilities MUST include an `examples/` folder and wire example execution into the root `test` flow, following [agentme-edr-007](../principles/007-project-quality-standards.md). Each example directory is its own Python project with its own `pyproject.toml`, and examples MUST import the library as a consumer would rather than reaching back into `lib/src/` with relative imports. Local example verification MUST install the wheel built into `lib/dist/`; do not use editable or path-based dependencies back to `lib/`.
 
 Python keeps unit tests under `lib/tests/` by default because that remains the more common and maintainable convention for typed/package-based projects than co-locating tests beside every source file. Integration tests belong in `lib/tests_integration/`, and benchmark harnesses belong in `lib/tests_benchmark/` when they are more than a single micro-benchmark helper.
 
@@ -114,7 +114,7 @@ When ty runs from `lib/`, it auto-discovers the virtual environment via the `VIR
 
 Ruff is the default formatter and linter. Do not add Black, isort, or Flake8 unless another XDR for that repository explicitly requires them.
 
-All Python projects must configure the following sections in `lib/pyproject.toml`. The cache-related settings are mandatory per the `.cache/` policy above:
+All Python projects MUST configure the following sections in `lib/pyproject.toml`. The cache-related settings are mandatory per the `.cache/` policy above:
 
 ```toml
 [tool.pytest.ini_options]
@@ -155,11 +155,11 @@ ignore = ["ANN002", "ANN003", "ANN401", "D100", "D101", "D102", "D103", "D104",
 ignore-overlong-task-comments = true
 ```
 
-Adjust `target-version` to match the project's minimum supported Python version. The `cache-dir` keeps Ruff's cache under `.cache/ruff` alongside other tool caches. The `src` list must include every directory that contains importable Python code. The `select` list enables a broad set of rules covering style, correctness, performance, security, and documentation. The `ignore` list suppresses rules that are either too noisy or conflict with the chosen docstring style.
+Adjust `target-version` to match the project's minimum supported Python version. The `cache-dir` keeps Ruff's cache under `.cache/ruff` alongside other tool caches. The `src` list MUST include every directory that contains importable Python code. The `select` list enables a broad set of rules covering style, correctness, performance, security, and documentation. The `ignore` list suppresses rules that are either too noisy or conflict with the chosen docstring style.
 
-ty must run on every lint pass. The default rule set is the minimum baseline; projects may enable stricter rules as the codebase matures.
+ty MUST run on every lint pass. The default rule set is the minimum baseline; projects may enable stricter rules as the codebase matures.
 
-Pytest coverage must fail below 80% line and branch coverage, following [agentme-edr-004](../principles/004-unit-test-requirements.md).
+Pytest coverage MUST fail below 80% line and branch coverage, following [agentme-edr-004](../principles/004-unit-test-requirements.md).
 
 #### Makefile targets
 
@@ -198,7 +198,7 @@ The root `Makefile` is the only contract for CI and contributors. It delegates l
 | `dev` | Same as `run`, optionally with repository-specific dev defaults |
 | `publish` | `mise exec -- uv publish --project .` after versioning and packaging are complete |
 
-The root `Makefile` must remain the only contract for CI and contributors, in line with [agentme-edr-008](../devops/008-common-targets.md).
+The root `Makefile` MUST remain the only contract for CI and contributors, in line with [agentme-edr-008](../devops/008-common-targets.md).
 
 ## Considered Options
 

package/.xdrs/agentme/edrs/application/018-ai-llm-development-standards.md

@@ -51,7 +51,7 @@ Every component that interacts with an LLM MUST be classified as exactly one of
 All direct LLM calls MUST use **LangChain** via the `langchain` packages.
 
 - Use `langchain-openai` as the provider integration layer. It supports both OpenAI and Azure OpenAI.
-- **Always configure LLM providers using explicit library attributes** such as `api_key`, `base_url`, `model`, `api_version`, etc. Never rely on environment variables for LLM configuration.
+- LLM providers MUST be configured using explicit library attributes such as `api_key`, `base_url`, `model`, `api_version`, etc. MUST NOT rely on environment variables for LLM configuration.
 - Configuration MUST be passed via constructor parameters or configuration objects, making dependencies explicit and testable.
 
 **Example of explicit configuration:**
@@ -218,4 +218,6 @@ Return a JSON object with:
 - [agentme-edr-004](../principles/004-unit-test-requirements.md) — Unit test requirements including external API mocking guidance
 - [agentme-edr-014](014-python-project-tooling.md) — Python project tooling and structure
 - [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards including AI-tier testing requirements (rule `09-ai-project-testing-requirements`)
-- [agentme-edr-028](028-ai-eval-standards.md) — AI eval standards: folder structure, script requirements, and MLflow tracking
+- [agentme-edr-028](028-ai-eval-core-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract (rule `02`)
+- [agentme-edr-031](031-ai-eval-script.md) — AI eval script: entry-first loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
+- [agentme-edr-032](032-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis

package/.xdrs/agentme/edrs/application/019-ai-agents-development-standards.md

@@ -46,11 +46,11 @@ When an agent requires a **local sandbox** — an isolated environment where the
 Use deepagents sandbox whenever ANY of the following is true:
 - The agent needs to execute shell commands or scripts in a controlled environment.
 - The agent needs to list, read, or search files across multiple directories at runtime.
-- The agent operates on user-supplied or generated file trees that must not escape a sandboxed boundary.
+- The agent operates on user-supplied or generated file trees that MUST NOT escape a sandboxed boundary.
 
 **Integration requirements:**
 
-- The sandbox MUST always be initialized with `virtual_mode=True` to prevent the agent from reading or writing files outside the mounted workspace. Omitting this flag allows the agent unrestricted host filesystem access, which is a security violation.
+- The sandbox MUST be initialized with `virtual_mode=True` to prevent the agent from reading or writing files outside the mounted workspace. Omitting this flag allows the agent unrestricted host filesystem access, which is a security violation.
 - Initialize the sandbox at the start of the agent run and shut it down in the same `try/finally` block.
 - Pass the sandbox handle into the agent's state so all tool calls share the same sandbox instance.
 - If the host-side code needs to pass files into the sandbox (e.g. generated config or input data), create a temporary directory with `tempfile.mkdtemp()`, write the files there, and mount it into the sandbox. Clean it up in the `finally` block.
@@ -120,11 +120,11 @@ When multiple agents are needed, one of these composition patterns MUST be chose
 |---|---|
 | Single agent + tools | All tools serve the same goal; agent completes in one session |
 | Multiple workflow-orchestrated agents | Each agent has a distinct goal; outputs flow between agents; deterministic sequencing needed |
-| Nested agents (FORBIDDEN) | Never — always use workflow orchestration instead |
+| Nested agents (FORBIDDEN) | MUST NOT use nested agents; MUST use workflow orchestration instead |
 
 #### 06-agent-system-prompt-structure
 
-Every agent system prompt MUST follow this XML-section template. Sections must appear in this order. Required sections must always be present; optional sections may be omitted when they genuinely do not apply; never reorder them.
+Every agent system prompt MUST follow this XML-section template. Sections MUST appear in this order. Required sections MUST be present; optional sections may be omitted when they genuinely do not apply; MUST NOT be reordered.
 
 ```xml
 [specific task description to the agent. if not defined use the default prompt "Execute your objective taking into consideration the inputs provided and all the sections described below"]
@@ -189,7 +189,7 @@ The current OS is: [operating system name].
 |---|---|---|
 | `<SYSTEM_CONTEXT>` | Optional | Runtime environment context injected at invocation time (e.g., current date in YYYY-MM-DD, OS). Include whenever the agent may need temporal or environment awareness. Time MUST NOT be included — it changes every second and breaks prompt caching. |
 | `<OBJECTIVE>` | Required | One or two sentences summarising the agent's main deliverable. |
-| `<ROLE>` | Required | Agent persona and expertise. When inside a workflow, MUST reference its node name from `<WORKFLOW_CONTEXT>`. |
+| `<AGENT_ROLE>` | Required | Agent persona and expertise. When inside a workflow, MUST reference its node name from `<WORKFLOW_CONTEXT>`. |
 | `<INPUT>` | Required | List ALL inputs. For workflow agents: workflow-level inputs first, then agent-specific inputs. |
 | `<STEPS>` | Optional | Include when the agent follows a non-trivial numbered sequence of steps. |
 | `<TOOL_GUIDANCE>` | Optional | Include when tool use order or conditions need explicit direction. |
@@ -200,7 +200,7 @@ The current OS is: [operating system name].
 **Formatting rules:**
 
 - MUST use XML tags to delimit every section.
-- The content of each section MUST start on the line immediately after the opening tag — never inline with it.
+- The content of each section MUST start on the line immediately after the opening tag — MUST NOT be inline with it.
 - Each closing tag MUST be followed by a blank line before the next opening tag, so sections are visually separated.
 
 ```xml
@@ -208,9 +208,9 @@ The current OS is: [operating system name].
 Produce a plan for the current batch of files.
 </OBJECTIVE>
 
-<ROLE>
+<AGENT_ROLE>
 You are the batch_plan_agent.
-</ROLE>
+</AGENT_ROLE>
 ```
 
 #### 07-agent-output-format

package/.xdrs/agentme/edrs/application/020-ai-agents-quality-standards.md

@@ -179,4 +179,6 @@ def test_workflow_calls_subagent(mocker):
 - [agentme-edr-018](018-ai-llm-development-standards.md) — LLM development standards (LangChain configuration, mocking patterns)
 - [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) — Hexagonal architecture (tool placement in adapters/connectors)
 - [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards including AI-tier testing requirements (rule `09-ai-project-testing-requirements`)
-- [agentme-edr-028](028-ai-eval-standards.md) — AI eval standards: folder structure, script requirements, and MLflow tracking
+- [agentme-edr-028](028-ai-eval-core-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract (rule `02`)
+- [agentme-edr-031](031-ai-eval-script.md) — AI eval script: entry-first loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
+- [agentme-edr-032](032-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis

package/.xdrs/agentme/edrs/application/021-ai-workflow-development-standards.md

@@ -37,11 +37,11 @@ Use **MLflow** for all workflow observability and evaluation:
 
 #### 04-dataset-driven-accuracy-measurement
 
-Projects MUST follow the eval dataset and implementation requirements defined in [agentme-edr-028](028-ai-eval-standards.md). Testing requirements (when evals are required, release gates) are defined in [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
+Projects MUST follow the eval dataset and implementation requirements defined in [agentme-edr-028](028-ai-eval-core-standards.md) and [agentme-edr-031](031-ai-eval-script.md). Testing requirements (when evals are required, release gates) are defined in [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
 
 #### 05-flow-documentation
 
-Each workflow MUST be documented as a **Mermaid graph** in a `README.md`. The diagram must match the LangGraph `StateGraph` definition:
+Each workflow MUST be documented as a **Mermaid graph** in a `README.md`. The diagram MUST match the LangGraph `StateGraph` definition:
 
 - Use `graph TD` or `graph LR` direction.
 - Label each node with its Python function name.
@@ -101,7 +101,7 @@ lib/src/<package_name>/
 
 #### 08-workflow-evals
 
-Projects MUST follow the eval folder structure and script requirements defined in [agentme-edr-028](028-ai-eval-standards.md).
+Projects MUST follow the eval folder structure defined in [agentme-edr-028](028-ai-eval-core-standards.md) rule `01` and the eval script requirements defined in [agentme-edr-031](031-ai-eval-script.md). For LLM-as-judge scoring used in workflow verification nodes and evals, see [agentme-edr-028](028-ai-eval-core-standards.md) rule `02`.
 
 #### 09-node-naming-conventions
 
@@ -109,12 +109,9 @@ Nodes MUST follow the naming conventions defined in [agentme-edr-029](029-ai-wor
 
 #### 10-workflow-unit-testing
 
-All LLM calls within workflow nodes are external API calls and MUST be mocked in unit tests per [agentme-edr-018](018-ai-llm-development-standards.md) rule `04-unit-test-mocking`. Workflow unit tests must run fully offline with no real LLM provider calls.
+All LLM calls within workflow nodes are external API calls and MUST be mocked in unit tests per [agentme-edr-018](018-ai-llm-development-standards.md) rule `04-unit-test-mocking`. Workflow unit tests MUST run fully offline with no real LLM provider calls.
 
-Choose the mock utility based on what the node under test expects from the model:
-
-- Use **`FakeListChatModel`** when nodes only read `AIMessage.content` (e.g. a routing node that checks a text label).
-- Use **`GenericFakeChatModel`** when any node in the workflow expects tool calls, structured outputs, or when the workflow contains `_agent` nodes that drive a tool-invocation loop.
+Choose the mock utility per [agentme-edr-018](018-ai-llm-development-standards.md) rule `04-unit-test-mocking`. For workflows containing `_agent` nodes that drive a tool-invocation loop, MUST use `GenericFakeChatModel`.
 
 **Example — workflow with plain-text LLM nodes:**
 
@@ -210,5 +207,8 @@ All workflow elements MUST maintain naming coherence as defined in [agentme-edr-
 - [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) — Adapter/application layer separation that defines the project layout
 - [agentme-edr-014](014-python-project-tooling.md) — Python project tooling and structure
 - [agentme-edr-024](024-ml-dataset-structure.md) — ML dataset structure for eval datasets
-- [agentme-edr-028](028-ai-eval-standards.md) — AI eval standards: folder structure, script requirements, and MLflow tracking
+- [agentme-edr-028](028-ai-eval-core-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract for all tiers (rule `02`)
+- [agentme-edr-031](031-ai-eval-script.md) — AI eval script: entry-first loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
+- [agentme-edr-032](032-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
+- [agentme-edr-033](033-ai-eval-repeatability.md) — AI eval repeatability: REPEAT_COUNT loop, scoring methods, and release cadence
 - [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards including AI-tier testing requirements (rule `09-ai-project-testing-requirements`)

package/.xdrs/agentme/edrs/application/025-ai-agent-xdrs-knowledge-layer.md

@@ -79,7 +79,7 @@ Follow [agentme-edr-019 rule 02-local-sandbox](019-ai-agents-development-standar
 | `lib/data/.xdrs/` | XDRS Policy and Skill documents | `/.xdrs/` (read-only) |
 | Generated at startup | `AGENTS.md` instructing the agent to consult XDRS | `/AGENTS.md` (read-only) |
 
-XDRS documents MUST always be mounted at `/.xdrs/`. `AGENTS.md` MUST always be placed at the sandbox root (`/AGENTS.md`).
+XDRS documents MUST be mounted at `/.xdrs/`. `AGENTS.md` MUST be placed at the sandbox root (`/AGENTS.md`).
 
 Example XDRS mount additions:
 

package/.xdrs/agentme/edrs/application/026-pragmatic-hexagonal-architecture.md

@@ -139,7 +139,7 @@ When a mock implementation needs to be **reused across multiple tests or importe
 - Single-test use → define the mock inline inside the test file (per rule `09` example; no file needed)
 - Reusable across multiple tests OR used from `eval.py` → define in a separate `_mock` file
 
-**Scope:** applies to any source file in `adapters/connectors/`, `app/`, or `shared/`. MUST NOT be used for inbound adapters (`cli/`, `http/`, `grpc/`) — those are entry points and are never mocked (rule `09`).
+**Scope:** applies to any source file in `adapters/connectors/`, `app/`, or `shared/`. MUST NOT be used for inbound adapters (`cli/`, `http/`, `grpc/`) — those are entry points and MUST NOT be mocked (rule `09`).
 
 **Naming:** insert `_mock` immediately before the file extension:
 
@@ -155,7 +155,7 @@ When a mock implementation needs to be **reused across multiple tests or importe
 
 **Mock contract:**
 - MUST accept a `fixtures` parameter (constructor argument or factory function argument); the value is whatever `mock_fixtures[key]` contains from the dataset entry — its internal structure is opaque and interpreted by the mock implementation
-- MUST NOT fall back to real external calls under any circumstance — if a call cannot be satisfied from the provided fixtures, MUST raise an explicit error (never silently return `null`, `undefined`, or an empty value)
+- MUST NOT fall back to real external calls under any circumstance — if a call cannot be satisfied from the provided fixtures, MUST raise an explicit error (MUST NOT silently return `null`, `undefined`, or an empty value)
 
 ## References
 

package/.xdrs/agentme/edrs/application/028-ai-eval-core-standards.md

@@ -0,0 +1,120 @@
+---
+name: agentme-edr-policy-028-ai-eval-core-standards
+description: Defines the eval folder structure and Makefile interface for AI projects, and the LLM-as-judge binary scoring contract applicable to all AI tiers and test types. Use when scaffolding evals or implementing an LLM judge. For eval script requirements see agentme-edr-031. For report format see agentme-edr-032. For repeatability see agentme-edr-033. For when evals are required see agentme-edr-007 rule 09-ai-project-testing-requirements. For the test type taxonomy see agentme-edr-030.
+apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
+valid-from: 2026-06-05
+---
+
+# agentme-edr-policy-028: AI eval core standards
+
+## Context and Problem Statement
+
+Eval tests measure AI component accuracy against expected outputs using real LLM providers. Without a shared folder layout and a common LLM judge contract, eval setups diverge across LLM, Agent, and Workflow projects, making them hard to scaffold, run, and compare.
+
+How should eval tests be structured, and how must LLM judges produce scores across all AI tiers?
+
+## Decision Outcome
+
+**Use a per-component folder structure under `evals/` with a standardized Makefile interface, and require all LLM judges to produce binary (`0`/`1`) output compatible with classification metrics.**
+
+For when evals are required per AI tier, see [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
+
+### Details
+
+#### 01-eval-folder-structure
+
+Evals are grouped first by the component being evaluated, then by the specific evaluation scenario. Create one directory per component under `evals/`, and one directory per eval scenario inside it. Place `evals/` at the same level as `lib/` and `examples/`:
+
+```text
+evals/
+  <component>/           # the component being evaluated (e.g., workflow-x, agent-y, model-z)
+    eval-<name>/
+      golden_dataset/    # EDR-024 + EDR-030 compliant golden dataset (README.md, dataset.schema.json, data/)
+      eval.py            # evaluation script
+      report-<type>.md   # generated report, one per evaluated test type (overwritten on each run — see agentme-edr-032 rule 01)
+      Makefile           # lint, eval, run, and eval-<type> targets
+    eval-<name2>/
+      ...
+  <component2>/
+    ...
+```
+
+`<component>` MUST match the name of the component under evaluation and use lowercase hyphen-separated words (e.g., `workflow-document-review`, `agent-support`, `model-classifier`).
+
+`<name>` identifies the specific evaluation scenario using lowercase hyphen-separated words (e.g., `eval-basic`, `eval-complex`, `eval-edge-cases`). A scenario's `golden_dataset` MAY mix multiple test types across its entries: label each entry with its applicable `test_types` ([agentme-edr-030](030-ai-test-types-taxonomy.md) rule `04`) and use the `eval-<type>` targets below to run one type at a time.
+
+The `golden_dataset/` subfolder MUST be a valid [agentme-edr-024](024-ml-dataset-structure.md) dataset (`README.md`, `dataset.schema.json`, one JSON file per entry under `data/` per rule `04-complex-structured-datasets-must-use-per-entry-json-files`, lint-validated per rule `06`) whose entries follow the golden dataset envelope defined in [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02`.
+
+Each `evals/<component>/eval-<name>/Makefile` MUST declare a `TEST_TYPES` variable listing the `test_types` values present in its golden dataset, and define:
+
+| Target | Behaviour |
+|---|---|
+| `lint` | Validates every `golden_dataset/data/*.json` file against `golden_dataset/dataset.schema.json` per [agentme-edr-024](024-ml-dataset-structure.md) rule `06` |
+| `eval` | Depends on `lint`; runs `eval.py --type=all` with threshold enforcement; exits non-zero on failure (CI-safe) |
+| `run` | Depends on `lint`; runs `eval.py --type=all` with threshold enforcement; same as `eval` but intended for local exploration |
+| `eval-<type>` | Depends on `lint`; runs `eval.py --type=<type>` for one declared test type, following [agentme-edr-008](../devops/008-common-targets.md) rule `03`'s `eval-<qualifier>` convention |
+
+```makefile
+TEST_TYPES := smoke functional safety
+
+lint:
+	mise exec -- uv run --project . python lint_dataset.py golden_dataset/
+
+eval: lint
+	mise exec -- uv run --project . python eval.py --type=all
+
+run: lint
+	mise exec -- uv run --project . python eval.py --type=all
+
+eval-%: lint
+	mise exec -- uv run --project . python eval.py --type=$*
+```
+
+The module root Makefile MUST expose `make eval` and `make lint` targets that delegate to `eval` and `lint` respectively in every `evals/<component>/eval-<name>/Makefile`:
+
+```makefile
+eval:
+	$(MAKE) -C evals/workflow-document-review/eval-basic eval
+	$(MAKE) -C evals/workflow-document-review/eval-complex eval
+
+lint:
+	$(MAKE) -C evals/workflow-document-review/eval-basic lint
+	$(MAKE) -C evals/workflow-document-review/eval-complex lint
+```
+
+#### 02-llm-as-judge-binary-output
+
+LLM judges scoring component outputs MUST produce binary output: `0` (fail) or `1` (success). This rule applies to all AI tiers (LLM, Agent, Workflow) and to all eval test types that use an LLM judge (functional, quality, safety, repeatability, or any other).
+
+**Requirements:**
+
+- Judge prompts MUST instruct the model to output exactly `0` or `1`
+- Scoring logic MUST parse the response and map to binary. Ambiguous/invalid responses: score as `0` or raise error
+- Reports using LLM judges MUST use classification metrics (Accuracy, F1, Precision, Recall), not regression metrics (RMSE, R2, MAE)
+- Multi-class classification not supported. For multiple quality levels, use multiple binary judges (e.g., one for "factually correct", another for "tone appropriate")
+
+**Rationale:** Binary output makes LLM judges compatible with classification metrics infrastructure (Accuracy, F1, Wilson CI, convergence analysis) defined in [agentme-edr-032](032-ai-eval-report-format.md) rule `01`.
+
+**Example LLM judge prompt:**
+
+```
+Evaluate whether the document review decision is correct.
+
+Input: {input_summary}
+Expected: {expected_decision}
+Actual: {actual_decision}
+
+Output exactly "1" if the actual decision matches the expected decision and reasoning, or "0" if it does not.
+
+Output:
+```
+
+## References
+
+- [agentme-edr-031](031-ai-eval-script.md) — AI eval script: entry-first eval loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
+- [agentme-edr-032](032-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
+- [agentme-edr-033](033-ai-eval-repeatability.md) — AI eval repeatability: loop exception, scoring methods (including LLM-as-judge per rule `02`), and cadence
+- [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09`) and statistical model eval targets (rule `07`)
+- [agentme-edr-030](030-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum and golden dataset entry envelope
+- [agentme-edr-024](024-ml-dataset-structure.md) — ML dataset structure, per-entry JSON format, and schema-lint validation for golden datasets
+- [agentme-edr-008](../devops/008-common-targets.md) — `eval-<qualifier>` Makefile convention (rule `03`)

package/.xdrs/agentme/edrs/application/029-ai-workflow-naming-conventions.md

@@ -46,7 +46,7 @@ def code_reviewer_agent(state): ...
 graph.add_node("code_reviewer_agent", code_reviewer_agent)
 ```
 
-Names MUST NOT use generic labels such as `node1`, `process`, or `run`. Each name must clearly express what action the node performs.
+Names MUST NOT use generic labels such as `node1`, `process`, or `run`. Each name MUST clearly express what action the node performs.
 
 Judge nodes use a **prefix** convention instead of a suffix: the name MUST start with `evaluate_` followed by the subject being judged (e.g. `evaluate_progress`, `evaluate_quality`, `evaluate_completeness`, `evaluate_relevance`). This makes judge nodes immediately distinguishable from all other node types at a glance.
 

package/.xdrs/agentme/edrs/application/030-ai-test-types-taxonomy.md

@@ -21,7 +21,7 @@ Each test type is named with its group, objective, mocking constraint, applicabi
 
 #### 01-golden-dataset-concept
 
-Projects MUST use a golden dataset to test AI components. A **golden dataset** comprises all eval case entries used to test an AI component (LLM, Agent, or Workflow tier); each entry is labeled with the `test_types` (rule `04`) it applies to. It is the dataset consumed by [agentme-edr-028](028-ai-eval-standards.md) evals and stored as one JSON file per entry per [agentme-edr-024](024-ml-dataset-structure.md) rule `04`, at `evals/<component>/eval-<name>/golden_dataset/`.
+Projects MUST use a golden dataset to test AI components. A **golden dataset** comprises all eval case entries used to test an AI component (LLM, Agent, or Workflow tier); each entry is labeled with the `test_types` (rule `04`) it applies to. It is the dataset consumed by [agentme-edr-031](031-ai-eval-script.md) evals and stored as one JSON file per entry per [agentme-edr-024](024-ml-dataset-structure.md) rule `04`, at `evals/<component>/eval-<name>/golden_dataset/`.
 
 #### 02-golden-dataset-entry-envelope
 
@@ -41,58 +41,76 @@ Every golden dataset entry (a JSON file in `golden_dataset/data/`) MUST have thi
 
 - `test_types` — array, values MUST come from rule `04`'s enum, MUST contain at least one value. An entry MAY carry more than one value additively (e.g. `["functional", "smoke", "human"]`) — no test type excludes another.
 - `input` — for Prompt-tier components, a raw prompt string or the prompt template's input parameters object; for Agent/Workflow-tier components, the input attributes object passed to the component.
-- `expected_output` — the fields used to score the entry under each of its automated `test_types`: output attributes for an LLM-as-judge rubric, a target for vector-similarity scoring, or exact attribute values for strict comparison. When `human` is one of the entry's `test_types`, `expected_output` MUST additionally include a `human_test` string field with manual-verification instructions (e.g. `"check for ethical issues, verify record change in system X"`) — this supplements, and never replaces, the entry's automated scoring fields.
-- `mock_fixtures` — optional object; keys identify the adapter or external system to mock (SHOULD match the connector folder name under `adapters/connectors/<name>` for readability, though not enforced), values are any valid JSON interpreted by the mock implementation. When present, eval.py MUST configure each named mock adapter with its fixture data BEFORE invoking the component for that entry; each entry MUST use fresh mock instances to prevent state from bleeding across entries. `mock_fixtures` applies to all `test_types` including `human` — the component is still invoked for human entries to capture `actual_output`. `mock_fixtures` MUST NOT include keys for LLM adapters: all golden dataset test types are rated `mocks disallowed for LLM calls` (rule `03`), so the LLM call MUST always be real; LLM provider mocking belongs exclusively to unit tests via [agentme-edr-018](018-ai-llm-development-standards.md) rule `04`. See [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
+- `expected_output` — the fields used to score the entry under each of its automated `test_types`: output attributes for an LLM-as-judge rubric, a target for vector-similarity scoring, or exact attribute values for strict comparison. When `human` is one of the entry's `test_types`, `expected_output` MUST additionally include a `human_test` string field with manual-verification instructions (e.g. `"check for ethical issues, verify record change in system X"`) — this supplements, and MUST NOT replace, the entry's automated scoring fields.
+- `mock_fixtures` — optional object; keys identify the adapter or external system to mock (SHOULD match the connector folder name under `adapters/connectors/<name>` for readability, though not enforced), values are any valid JSON interpreted by the mock implementation. When present, eval.py MUST configure each named mock adapter with its fixture data BEFORE invoking the component for that entry; each entry MUST use fresh mock instances to prevent state from bleeding across entries. `mock_fixtures` applies to all `test_types` including `human` — the component is still invoked for human entries to capture `actual_output`. `mock_fixtures` MUST NOT include keys for LLM adapters: all golden dataset test types are rated `mocks disallowed for LLM calls` (rule `03`), so the LLM call MUST be real; LLM provider mocking belongs exclusively to unit tests via [agentme-edr-018](018-ai-llm-development-standards.md) rule `04`. See [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
 - The dataset's `dataset.schema.json` MUST require `test_types`, `input`, and `expected_output`, and SHOULD declare `mock_fixtures` as optional (`"type": "object", "additionalProperties": {}`), per [agentme-edr-024](024-ml-dataset-structure.md) rule `04`.
 
 #### 03-mocks-allowed-values
 
-The taxonomy in rule `05` rates each test type using one of three values:
+The taxonomy in rule `05` rates each test type using one of three values under the `Mock Constraint` column:
 
 | Value | Meaning |
 |---|---|
-| `mocks allowed` | Fully offline; fakes may replace every dependency (e.g. `FakeListChatModel` per [agentme-edr-018](018-ai-llm-development-standards.md) rule `04`). |
-| `mocks disallowed` | No mocking of any dependency — real external systems required. |
-| `mocks disallowed for LLM calls` | Tools and other external/dependency calls MAY be mocked; only the LLM call itself MUST be real for the test to be meaningful. |
+| `mocks allowed` | Fully offline; fakes may replace every dependency including the LLM (e.g. `FakeListChatModel` per [agentme-edr-018](018-ai-llm-development-standards.md) rule `04`). Used only for code-level unit tests. |
+| `mocks disallowed` | No mocking of any dependency — all real external systems required. Used for integration tests. |
+| `mocks disallowed for LLM calls` | **The LLM call MUST be real; all other external dependencies (databases, APIs, external services) MAY and SHOULD be mocked via `mock_fixtures`.** `mock_fixtures` keys MUST NOT reference LLM adapters. This is the standard constraint for every golden-dataset eval test type. See rule `08` for rationale. |
 
 #### 04-test-types-enum
 
-A golden dataset entry's `test_types` array MUST only use these values: `safety`, `adversarial`, `fairness`, `bias`, `robustness`, `explainability`, `groundedness`, `functional`, `prompt`, `smoke`, `human`. These correspond to the dataset-driven rows of rule `05`. **Unit test** and **Integration test** (the two Code-level rows) are NOT part of this enum — they have no golden dataset entries and remain governed entirely by [agentme-edr-004](../principles/004-unit-test-requirements.md) and [agentme-edr-007](../principles/007-project-quality-standards.md) rule `08`.
+A golden dataset entry's `test_types` array MUST only use these values: `safety`, `adversarial`, `fairness`, `bias`, `robustness`, `explainability`, `groundedness`, `functional`, `prompt`, `smoke`, `human`, `repeatability`. These correspond to the dataset-driven rows of rule `05`. **Unit test** and **Integration test** (the two Code-level rows) are NOT part of this enum — they have no golden dataset entries and remain governed entirely by [agentme-edr-004](../principles/004-unit-test-requirements.md) and [agentme-edr-007](../principles/007-project-quality-standards.md) rule `08`.
 
 #### 05-test-type-taxonomy
 
 Test types MUST be selected from this taxonomy. Each test type is named with its group, objective, mocking constraint, applicability, and relevance:
 
-| Test Type Name | Group | Test Objective | Mocks Allowed | When to Apply | Relevance – Business | Relevance – Development Team | Priority (1-5) |
-|---|---|---|---|---|---|---|---|
-| Safety/content eval | Safety & adversarial | Detect harmful, biased, or policy-violating output | mocks disallowed for LLM calls | Any user-facing release | Avoids reputational harm; acceptable-use compliance | Automated content gate before merge/release | 5 |
-| Adversarial/red-team test | Safety & adversarial | Probe for prompt injection, jailbreaks, unsafe tool use | mocks disallowed for LLM calls | System exposes tool-invocation or agent loops | Reduces security-incident/breach liability | Finds exploitable tool-loop paths before attackers do | 5 |
-| Fairness test | Responsible AI | Verify equitable outcomes across user groups | mocks disallowed for LLM calls | Output affects decisions about individuals/groups | Regulatory requirement; protects equitable access | Surfaces uneven outcomes before release | 4 |
-| Bias test | Responsible AI | Detect skewed or stereotyped associations | mocks disallowed for LLM calls | User-facing content generation | Lowers legal/reputational exposure | Catches bias introduced by data/prompts/fine-tuning | 3 |
-| Robustness test | Responsible AI | Verify stable behavior under noisy/out-of-distribution input | mocks disallowed for LLM calls | Inputs come from untrusted/variable sources | Protects reliability/SLAs | Confirms graceful degradation, guides input validation | 3 |
-| Explainability test | Responsible AI | Verify output is justifiable with a faithful rationale | mocks disallowed for LLM calls | Output must be justified to users/auditors/regulators | Required for auditability; builds user trust | Gives rationale trace for debugging wrong answers | 2 |
-| Groundedness (RAG) eval | Quality eval | Verify the answer is supported by retrieved context | mocks disallowed for LLM calls | System uses retrieval-augmented generation | Avoids confidently-wrong answers reaching customers | Pinpoints retrieval/prompt bugs | 4 |
-| Human evaluation | Quality eval | Manually verify aspects automated scoring can't (ethics, side effects, external state) | mocks disallowed for LLM calls | Before major releases; periodic spot-check | Defensible, human-reviewed sign-off | Catches what automated metrics miss | 3 |
-| Functional eval (golden-dataset accuracy / LLM-as-judge) | Quality eval | Measure output correctness against the golden dataset | mocks disallowed for LLM calls | Required before every Workflow release ([agentme-edr-007](../principles/007-project-quality-standards.md) rule `09`); advised elsewhere | Auditable evidence of business correctness before release | Detects regressions from model/provider/prompt changes | 5 |
-| Smoke test | Quality eval | Fast pass/fail check on a small, critical subset before running fuller suites | mocks disallowed for LLM calls | Every commit/PR, before functional/responsible-AI evals run | Cheap early warning before slower evals run | Fast, cheap feedback loop | 4 |
-| Prompt regression test | Prompt/LLM | Detect behavior change when a prompt or model version changes | mocks disallowed for LLM calls | Whenever a prompt template or model version changes | Prevents shipping a worse experience via a "small" tweak | Fast check on every prompt edit | 3 |
-| Integration test | Code-level | Verify real interaction with external systems | mocks disallowed | Component depends on external systems | Reduces production outages from integration mismatches | Catches wiring bugs unit tests can't see | 2 |
-| Unit test (offline, mocked) | Code-level | Verify deterministic logic in isolation, offline | mocks allowed | Required for Workflow tier every commit ([agentme-edr-007](../principles/007-project-quality-standards.md) rule `09`) | Lowest-cost point to catch defects | Fastest, fully offline feedback on every commit | 5 |
+| Test Type Name | `test_types` value | Group | Test Objective | Mock Constraint | When to Apply | Relevance – Business | Relevance – Development Team | Priority (1-5) |
+|---|---|---|---|---|---|---|---|---|
+| Safety/content eval | `safety` | Safety & adversarial | Detect harmful, biased, or policy-violating output | mocks disallowed for LLM calls | Any user-facing release | Avoids reputational harm; acceptable-use compliance | Automated content gate before merge/release | 5 |
+| Adversarial/red-team test | `adversarial` | Safety & adversarial | Probe for prompt injection, jailbreaks, unsafe tool use | mocks disallowed for LLM calls | System exposes tool-invocation or agent loops | Reduces security-incident/breach liability | Finds exploitable tool-loop paths before attackers do | 5 |
+| Fairness test | `fairness` | Responsible AI | Verify equitable outcomes across user groups | mocks disallowed for LLM calls | Output affects decisions about individuals/groups | Regulatory requirement; protects equitable access | Surfaces uneven outcomes before release | 4 |
+| Bias test | `bias` | Responsible AI | Detect skewed or stereotyped associations | mocks disallowed for LLM calls | User-facing content generation | Lowers legal/reputational exposure | Catches bias introduced by data/prompts/fine-tuning | 3 |
+| Robustness test | `robustness` | Responsible AI | Verify stable behavior under noisy/out-of-distribution input | mocks disallowed for LLM calls | Inputs come from untrusted/variable sources | Protects reliability/SLAs | Confirms graceful degradation, guides input validation | 3 |
+| Explainability test | `explainability` | Responsible AI | Verify output is justifiable with a faithful rationale | mocks disallowed for LLM calls | Output must be justified to users/auditors/regulators | Required for auditability; builds user trust | Gives rationale trace for debugging wrong answers | 2 |
+| Groundedness (RAG) eval | `groundedness` | Quality eval | Verify the answer is supported by retrieved context | mocks disallowed for LLM calls | System uses retrieval-augmented generation | Avoids confidently-wrong answers reaching customers | Pinpoints retrieval/prompt bugs | 4 |
+| Repeatability test | `repeatability` | Quality eval | Verify output stability/variance across N repeated invocations of the same input under fixed configuration | mocks disallowed for LLM calls | Non-deterministic components (temperature > 0, agentic tool-selection loops, sampling-based decoding) used in decision-critical or user-facing flows | Protects against silently flaky behavior reaching production; supports consistency SLAs | Detects prompt/agent designs too sensitive to sampling noise; informs temperature/seed tuning | 3 |
+| Human evaluation | `human` | Quality eval | Manually verify aspects automated scoring can't (ethics, side effects, external state) | mocks disallowed for LLM calls | Before major releases; periodic spot-check | Defensible, human-reviewed sign-off | Catches what automated metrics miss | 3 |
+| Functional eval (golden-dataset accuracy / LLM-as-judge) | `functional` | Quality eval | Measure output correctness against the golden dataset | mocks disallowed for LLM calls | Required before every Workflow release ([agentme-edr-007](../principles/007-project-quality-standards.md) rule `09`); advised elsewhere | Auditable evidence of business correctness before release | Detects regressions from model/provider/prompt changes | 5 |
+| Smoke test | `smoke` | Quality eval | Fast pass/fail check on a small, critical subset before running fuller suites | mocks disallowed for LLM calls | Every commit/PR, before functional/responsible-AI evals run | Cheap early warning before slower evals run | Fast, cheap feedback loop | 4 |
+| Prompt regression test | `prompt` | Prompt/LLM | Detect behavior change when a prompt or model version changes | mocks disallowed for LLM calls | Whenever a prompt template or model version changes | Prevents shipping a worse experience via a "small" tweak | Fast check on every prompt edit | 3 |
+| Integration test | n/a — code-level only (see rule `04`) | Code-level | Verify real interaction with external systems | mocks disallowed | Component depends on external systems | Reduces production outages from integration mismatches | Catches wiring bugs unit tests can't see | 2 |
+| Unit test (offline, mocked) | n/a — code-level only (see rule `04`) | Code-level | Verify deterministic logic in isolation, offline | mocks allowed | Required for Workflow tier every commit ([agentme-edr-007](../principles/007-project-quality-standards.md) rule `09`) | Lowest-cost point to catch defects | Fastest, fully offline feedback on every commit | 5 |
 
 #### 06-priority-and-relevance-are-descriptive-only
 
-Priority, Relevance, and When to Apply in rule `05` are guidance for prioritization conversations — they MUST NOT be treated as mandating which test types a project must implement, nor their thresholds. [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09` remains the only tier-level testing requirement in force (Workflow unit tests + functional evals). Once a project chooses to implement and threshold a test type, [agentme-edr-028](028-ai-eval-standards.md) rule `02`'s failing-threshold behavior applies uniformly, regardless of this table's priority rating — a project may enforce fairness at 70% and functional at 90%, or skip fairness entirely; that choice is a project/business decision, not one this Policy makes.
+Priority, Relevance, and When to Apply in rule `05` are guidance for prioritization conversations — they MUST NOT be treated as mandating which test types a project must implement, nor their thresholds. [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09` remains the only tier-level testing requirement in force (Workflow unit tests + functional evals). Once a project chooses to implement and threshold a test type, [agentme-edr-031](031-ai-eval-script.md) rule `01`'s failing-threshold behavior applies uniformly, regardless of this table's priority rating — a project may enforce fairness at 70% and functional at 90%, or skip fairness entirely; that choice is a project/business decision, not one this Policy makes.
 
 #### 07-smoke-is-distinct-from-test-smoke
 
 The `smoke` test type (surfaced as the `eval-smoke` Makefile target, a fast subset of the golden-dataset functional eval) is a different concept from [agentme-edr-008](../devops/008-common-targets.md)'s existing `test-smoke` target (a fast subset of code-level tests). Both MAY exist in the same project; teams MUST NOT conflate them.
 
+#### 08-eval-mocking-constraint
+
+For every golden-dataset eval test type: **the LLM call MUST be real; all other external dependencies MUST be mocked via `mock_fixtures`.** This applies equally to `human` entries — the component is invoked to capture `actual_output` and external dependencies must be deterministic. `mock_fixtures` keys MUST NOT reference LLM adapters. Code-level unit tests are the correct place for fully offline, LLM-mocked testing (see [agentme-edr-018](018-ai-llm-development-standards.md) rule `04`).
+
+#### 09-repeatability-vs-reproducibility
+
+| Property | Definition | What varies | Measured by |
+|---|---|---|---|
+| **Repeatability** | Output stability across N invocations at non-zero temperature | Model sampling variance | `repeatability` test type per [agentme-edr-033](033-ai-eval-repeatability.md) |
+| **Reproducibility** | Deterministic output at temperature = 0 with fixed seed | Nothing — any variance is a config bug | Not a golden-dataset type; verified via config, documented in [agentme-edr-027](../devops/027-environment-variable-configuration.md) |
+
+A component may satisfy reproducibility (temperature = 0) yet still need repeatability tests for its production configuration (temperature > 0). The `repeatability` test type MUST NOT be applied to components with intentionally diverse output (brainstorming, creative generation) — variance is correct behavior there.
+
 ## References
 
 - [agentme-edr-024](024-ml-dataset-structure.md) — Golden dataset file layout, per-entry JSON format, `$schema` pointer, and schema-lint validation
-- [agentme-edr-028](028-ai-eval-standards.md) — Eval folder structure, `--type` filtering, per-type Makefile targets, and per-type reports that consume this taxonomy
+- [agentme-edr-028](028-ai-eval-core-standards.md) — AI eval core standards: eval folder structure and Makefile targets (rule `01`); LLM-as-judge binary scoring contract (rule `02`)
+- [agentme-edr-031](031-ai-eval-script.md) — AI eval script: `--type` filtering, entry-first loop, `mock_fixtures`, threshold enforcement, and MLflow conventions
+- [agentme-edr-032](032-ai-eval-report-format.md) — AI eval report format: per-type `report-<type>.md` that consumes this taxonomy's test types
+- [agentme-edr-033](033-ai-eval-repeatability.md) — AI eval repeatability: `REPEAT_COUNT` loop exception, scoring constants (`EVAL_MIN_ACCURACY_REPEATABILITY`, `REPEAT_SEMANTIC_SIMILARITY_SCORE`), scoring methods, `repeatability_accuracy` MLflow metric, report shape, and run cadence
 - [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) — Rule `10`: `_mock` file naming and placement convention for mock adapters referenced by `mock_fixtures`
 - [agentme-edr-007](../principles/007-project-quality-standards.md) — Rule `09` tier-level testing requirements (the only mandated AI testing baseline)
 - [agentme-edr-008](../devops/008-common-targets.md) — Rule `03` `eval-<qualifier>` Makefile convention; rule `03`'s `test-smoke` (distinguished in rule `07`)
+- [agentme-edr-027](../devops/027-environment-variable-configuration.md) — Environment-configuration conventions referenced in rule `09`'s reproducibility disambiguation
 - [agentme-edr-018](018-ai-llm-development-standards.md) — LLM tier definition and mocking utilities referenced by the `mocks allowed` value
 - [agentme-edr-004](../principles/004-unit-test-requirements.md) — Unit test requirements underlying the Code-level rows