agentme 0.24.2 → 0.25.2

package/.filedist-package.yml

@@ -1,5 +1,5 @@
 sets:
-  - package: xdrs-core@0.30.0
+  - package: xdrs-core@0.38.3
   # - package: git:https://github.com/flaviostutz/xdrs-core.git@main
     selector:
       files:

package/.xdrs/agentme/edrs/application/015-cli-tool-standards.md

@@ -23,21 +23,21 @@ This keeps the user-facing command predictable while preserving a clean library
 
 #### CLI command surface
 
-- CLI tools should default to the format `[tool] [command] [options] [arguments]`.
+- CLI tools SHOULD default to the format `[tool] [command] [options] [arguments]`.
 - Example: `filedist extract --packages=test mydir`
-- A single-action tool may omit `[command]` only when adding a subcommand would be artificial and there is no meaningful action split.
-- Every CLI tool must expose:
+- A single-action tool MAY omit `[command]` only when adding a subcommand would be artificial and there is no meaningful action split.
+- Every CLI tool MUST expose:
   - `--help` on the root command
   - `--version` on the root command
   - `--verbose` on the root command and on subcommands when flags are parsed per command
-- Root `--help` output must list all available commands, key options, and usage examples. Command-specific help must describe that command's arguments and options.
+- Root `--help` output MUST list all available commands, key options, and usage examples. Command-specific help MUST describe that command's arguments and options.
 
 #### CLI to application separation
 
 - Structure the software as `cli -> app` — the CLI adapter delegates to the application layer, following [agentme-edr-026](026-pragmatic-hexagonal-architecture.md).
-- The CLI layer must only parse arguments, load config, call the application layer, and format output.
-- Domain logic must live in the application layer and be usable without CLI globals such as `argv`, `stdout`, or process exit handlers.
-- Every feature available through the CLI must also be available through the application API.
+- The CLI layer MUST only parse arguments, load config, call the application layer, and format output.
+- Domain logic MUST live in the application layer and be usable without CLI globals such as `argv`, `stdout`, or process exit handlers.
+- Every feature available through the CLI MUST also be available through the application API.
 - Organize the application layer by action so the mapping stays direct and obvious.
   - `extract` command -> `app/extract(...)`
   - `validate` command -> `app/validate(...)`
@@ -45,49 +45,49 @@ This keeps the user-facing command predictable while preserving a clean library
 
 #### Application API shape
 
-- Each CLI action should map to a dedicated exported application function with typed inputs and outputs appropriate for the language.
-- Application APIs should accept in-memory options objects or typed parameters, not require config files or environment variables unless application-level config-file support is an explicit requirement.
+- Each CLI action SHOULD map to a dedicated exported application function with typed inputs and outputs appropriate for the language.
+- Application APIs SHOULD accept in-memory options objects or typed parameters, not require config files or environment variables unless application-level config-file support is an explicit requirement.
 - The CLI layer is responsible for translating flags, positional arguments, and config-file contents into application inputs.
-- The application layer should return explicit results and errors so the CLI can decide what to print and which exit code to use.
+- The application layer SHOULD return explicit results and errors so the CLI can decide what to print and which exit code to use.
 
 #### Configuration
 
 - Prefer flags and positional arguments for simple inputs.
 - When configuration becomes long, nested, or repetitive, use a YAML config file instead of pushing all values into flags. See [agentme-edr-027](../devops/027-environment-variable-configuration.md) for when `.env` values should be referenced from within that file.
-- By default, config-file discovery and loading must happen in the CLI layer, not in the application layer.
-- When a config file is supported, the CLI must try to load a YAML file from `[cwd]/[tool-name].yml` by default.
-- The CLI must also support an explicit config path flag such as `--config`.
-- The application layer must not depend on the presence of the config file; it should receive parsed configuration values from the CLI layer.
-- The application layer may load or parse config files only when that behavior is an explicit requirement of the application contract for non-CLI consumers as well.
+- By default, config-file discovery and loading MUST happen in the CLI layer, not in the application layer.
+- When a config file is supported, the CLI MUST try to load a YAML file from `[cwd]/[tool-name].yml` by default.
+- The CLI MUST also support an explicit config path flag such as `--config`.
+- The application layer MUST NOT depend on the presence of the config file; it SHOULD receive parsed configuration values from the CLI layer.
+- The application layer MAY load or parse config files only when that behavior is an explicit requirement of the application contract for non-CLI consumers as well.
 
 #### Output and progress
 
-- Standard output must show a start message when work begins and a result message when work completes successfully.
+- Standard output MUST show a start message when work begins and a result message when work completes successfully.
 - When processing is long-running or multi-stage, print concise intermediate progress messages.
-- `--verbose` must reveal more internal detail about what the tool is doing without changing the meaning of the command result.
-- Default output should stay concise and readable for humans.
-- Errors should be written to standard error with an actionable message. Stack traces or raw internal errors should stay hidden by default and may be shown in verbose mode.
+- `--verbose` MUST reveal more internal detail about what the tool is doing without changing the meaning of the command result.
+- Default output SHOULD stay concise and readable for humans.
+- Errors SHOULD be written to standard error with an actionable message. Stack traces or raw internal errors SHOULD stay hidden by default and MAY be shown in verbose mode.
 
 #### Exit behavior
 
 - Exit with `0` only when the requested action completed successfully.
 - Exit with `1` when the requested action could not be completed.
-- The application layer should surface failure as return values, result objects, or language-idiomatic errors; the CLI is responsible for converting that outcome into user-facing messages and process exit codes.
+- The application layer SHOULD surface failure as return values, result objects, or language-idiomatic errors; the CLI is responsible for converting that outcome into user-facing messages and process exit codes.
 
 #### Documentation
 
-- `README.md` must include at least 4 CLI usage examples.
-- `README.md` must include at least 2 application API examples for the same operation also available through the CLI.
-- If the tool supports config files, at least 1 README example should show config-file usage.
-- Examples must use the public command and public application API, not internal modules or private files.
+- `README.md` MUST include at least 4 CLI usage examples.
+- `README.md` MUST include at least 2 application API examples for the same operation also available through the CLI.
+- If the tool supports config files, at least 1 README example SHOULD show config-file usage.
+- Examples MUST use the public command and public application API, not internal modules or private files.
 
 #### Distribution and versioning
 
-- The implementation language is project-dependent, but the packaging and entry-point strategy must match how users are expected to run the tool.
+- The implementation language is project-dependent, but the packaging and entry-point strategy MUST match how users are expected to run the tool.
 - Choose language tooling that stays compatible with ecosystem launchers such as `npx`, `pnpm dlx`, `uvx`, or equivalent distribution commands for that ecosystem.
-- `--version` must print the same version declared in the published package or release artifact metadata.
+- `--version` MUST print the same version declared in the published package or release artifact metadata.
 - Do not hard-code a second version string that can drift from the published package version.
-- Language-specific project structure and packaging rules still apply and should be combined with this XDR, especially [agentme-edr-003](003-javascript-project-tooling.md), [agentme-edr-010](010-golang-project-tooling.md), and [agentme-edr-014](014-python-project-tooling.md).
+- Language-specific project structure and packaging rules still apply and SHOULD be combined with this XDR, especially [agentme-edr-003](003-javascript-project-tooling.md), [agentme-edr-010](010-golang-project-tooling.md), and [agentme-edr-014](014-python-project-tooling.md).
 
 ## Considered Options
 

package/.xdrs/agentme/edrs/application/019-ai-agents-development-standards.md

@@ -108,11 +108,11 @@ When agents are used as nodes in workflows, the node name MUST use the `_agent`
 
 #### 05-agent-composition
 
-When multiple agents are needed:
+When multiple agents are needed, one of these composition patterns MUST be chosen:
 
 - **Single agent with multiple tools:** Use when tools share a common goal and context (e.g., a code analysis agent with `read_file`, `search_code`, and `analyze_pattern` tools).
 - **Multiple agents as workflow nodes:** Use when agents have distinct responsibilities and outputs that feed into each other. Orchestrate them using LangGraph per [agentme-edr-021](021-ai-workflow-development-standards.md).
-- Do NOT create nested agent loops (agent calling agent autonomously). Use workflows for multi-agent orchestration.
+- Nested agent loops (agent calling agent autonomously) MUST NOT be created. Use workflows for multi-agent orchestration.
 
 **Decision criteria:**
 

package/.xdrs/agentme/edrs/application/021-ai-workflow-development-standards.md

@@ -37,7 +37,7 @@ Use **MLflow** for all workflow observability and evaluation:
 
 #### 04-dataset-driven-accuracy-measurement
 
-Eval dataset and implementation requirements are defined in [agentme-edr-028](028-ai-eval-standards.md). Testing requirements (when evals are required, release gates) are defined in [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
+Projects MUST follow the eval dataset and implementation requirements defined in [agentme-edr-028](028-ai-eval-standards.md). Testing requirements (when evals are required, release gates) are defined in [agentme-edr-007](../principles/007-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
 
 #### 05-flow-documentation
 
@@ -101,11 +101,11 @@ lib/src/<package_name>/
 
 #### 08-workflow-evals
 
-Eval folder structure and script requirements are defined in [agentme-edr-028](028-ai-eval-standards.md).
+Projects MUST follow the eval folder structure and script requirements defined in [agentme-edr-028](028-ai-eval-standards.md).
 
 #### 09-node-naming-conventions
 
-See [agentme-edr-029](029-ai-workflow-naming-conventions.md) rule `01-node-naming-conventions`.
+Nodes MUST follow the naming conventions defined in [agentme-edr-029](029-ai-workflow-naming-conventions.md) rule `01-node-naming-conventions`.
 
 #### 10-workflow-unit-testing
 
@@ -159,15 +159,15 @@ Workflows MUST accept the LLM instance as a constructor parameter so that unit t
 
 #### 11-state-type-conventions
 
-See [agentme-edr-029](029-ai-workflow-naming-conventions.md) rule `02-state-type-conventions`.
+State types MUST follow the conventions defined in [agentme-edr-029](029-ai-workflow-naming-conventions.md) rule `02-state-type-conventions`.
 
 #### 12-workflow-naming-conventions
 
-See [agentme-edr-029](029-ai-workflow-naming-conventions.md) rule `04-workflow-naming-conventions`.
+Workflows MUST be named following the conventions in [agentme-edr-029](029-ai-workflow-naming-conventions.md) rule `04-workflow-naming-conventions`.
 
 #### 13-judge-node-output-format
 
-See [agentme-edr-029](029-ai-workflow-naming-conventions.md) rule `03-judge-node-output-format`.
+Judge nodes MUST use the output format defined in [agentme-edr-029](029-ai-workflow-naming-conventions.md) rule `03-judge-node-output-format`.
 
 #### 15-workflow-state-persistence
 
@@ -200,7 +200,7 @@ result = graph.invoke(input_state, config={"thread_id": "session-123"})
 
 #### 16-cross-element-naming-coherence
 
-See [agentme-edr-029](029-ai-workflow-naming-conventions.md) rule `05-cross-element-naming-coherence`.
+All workflow elements MUST maintain naming coherence as defined in [agentme-edr-029](029-ai-workflow-naming-conventions.md) rule `05-cross-element-naming-coherence`.
 
 ## References
 

package/.xdrs/agentme/edrs/application/024-ml-dataset-structure.md

@@ -9,7 +9,7 @@ valid-from: 2026-05-27
 
 ## Context and Problem Statement
 
-ML projects accumulate datasets of different shapes: file-paired annotations, tabular CSVs, and structured JSONL records. Without a shared layout convention, tooling and agents cannot reliably discover schema files, consume data programmatically, or understand what a dataset contains.
+ML projects accumulate datasets of different shapes: file-paired annotations, tabular CSVs, and complex per-record structures. Without a shared layout convention, tooling and agents cannot reliably discover schema files, consume data programmatically, or understand what a dataset contains.
 
 How should ML datasets be organized on disk so they are self-describing, easy to consume, and consistent across dataset types?
 
@@ -58,6 +58,8 @@ Placing the annotation file next to its source file (same name + `.json`) keeps
 
 Subdirectories inside `data/` are allowed when the number of files warrants grouping, but the `.json` sibling convention MUST be preserved at each level.
 
+Each `.json` annotation file MUST include a top-level `$schema` property whose value is the correct relative path to the dataset's root `dataset.schema.json`, accounting for subdirectory depth (e.g. `"$schema": "../dataset.schema.json"` at one level, `"../../dataset.schema.json"` at two levels). This is the standard editor-tooling convention for associating a JSON instance with its schema, and is validated by rule `06`.
+
 #### 03-tabular-datasets-must-use-csv-files-at-root
 
 Datasets composed of column-oriented tabular data MUST place CSV files at the root of the dataset folder. All tabular files MUST conform to the schema defined in `dataset.schema.json`, which MUST describe columns as named attributes with their types.
@@ -70,27 +72,40 @@ Datasets composed of column-oriented tabular data MUST place CSV files at the ro
     README.md
 ```
 
-Multiple CSV files are allowed when they represent different slices or splits of the same schema (e.g. train/test splits, subsets by source). All files in the same dataset MUST share the same column schema.
+Multiple CSV files are allowed when they represent different slices or splits of the same schema (e.g. train/test splits, subsets by source). All files in the same dataset MUST share the same column schema. Each row MUST also be validated by `make lint` per rule `06` — CSV has no `$schema` field (not applicable to that format), so only the row content is checked, not a schema pointer.
 
-#### 04-complex-structured-datasets-must-use-jsonl
+#### 04-complex-structured-datasets-must-use-per-entry-json-files
 
-Datasets with complex or heterogeneous per-record structures (e.g. LLM workflow evaluation sets, Q&A pairs, input → expected_output pairs) MUST use JSONL files (one JSON object per line) placed at the root of the dataset folder. Each line MUST conform to the schema defined in `dataset.schema.json`.
+Datasets with complex or heterogeneous per-record structures (e.g. LLM workflow evaluation sets, Q&A pairs, input → expected_output pairs) MUST use one JSON file per entry, placed inside the `data/` subfolder. Each file MUST conform to the schema defined in `dataset.schema.json`.
 
 ```
 /[name-of-dataset]/
-    simple-cases-test.jsonl
-    edge-cases-test.jsonl
-    dataset.schema.json    (schema defining the structure of each line in the JSONL files)
+    data/
+        case-001.json
+        case-002.json
+    dataset.schema.json    (schema defining the structure of each entry file)
     README.md
 ```
 
-Multiple JSONL files are allowed when they represent different splits or categories (e.g. easy vs. edge cases). All files in the same dataset MUST conform to the same line schema.
+Each entry file MUST include a top-level `$schema` property whose value is the correct relative path to the dataset's root `dataset.schema.json`, accounting for subdirectory depth (e.g. `"$schema": "../dataset.schema.json"` at one level, `"../../dataset.schema.json"` at two levels), validated by rule `06`.
+
+Subdirectories inside `data/` are allowed for grouping entries by collection source, time period, actor, or similar dimensions when the number of files warrants it. ALL files in one dataset MUST conform to the SAME `dataset.schema.json` — if a project needs a different schema for a different set of entries, that MUST be a separate dataset (its own folder, README, and `dataset.schema.json`), not multiple schemas inside one dataset.
 
 #### 05-referenced-files-must-live-in-data-folder
 
-When any dataset type (tabular, JSONL, or annotation-pair) contains references to external files as part of the data (e.g. a JSONL record that includes a file path), those referenced files MUST be stored inside the `data/` subfolder of the dataset. Paths inside data records MUST be relative to the dataset root.
+When any dataset type (tabular, per-entry JSON, or annotation-pair) contains references to external files as part of the data (e.g. an entry record that includes a file path), those referenced files MUST be stored inside the `data/` subfolder of the dataset. Paths inside data records MUST be relative to the dataset root.
+
+#### 06-datasets-must-be-lint-validated-against-schema
+
+Every dataset MUST expose a `make lint` target (in the Makefile of the project/component that owns the dataset) that validates its data against `dataset.schema.json` using the Python [`jsonschema`](https://pypi.org/project/jsonschema/) library:
+
+- Per-entry JSON files (rule `04`) and annotation-pair `.json` siblings (rule `02`) MUST each be validated against `dataset.schema.json`, and their `$schema` property MUST be present and resolve to the dataset's actual schema file.
+- CSV rows (rule `03`) MUST each be converted to a JSON object (column header → value) and validated against the same `dataset.schema.json`.
+- `make lint` MUST list every violation found across all files/rows before exiting with a non-zero status (not fail-fast on the first violation).
+- `jsonschema` MUST be declared as a normal project dependency per [agentme-edr-014](014-python-project-tooling.md); no special-casing.
 
 ## References
 
 - [JSON Schema specification](https://json-schema.org/)
-- [JSONL format](https://jsonlines.org/)
+- [jsonschema (Python library)](https://pypi.org/project/jsonschema/)
+- [agentme-edr-014](014-python-project-tooling.md) — Python project tooling and dependency conventions

package/.xdrs/agentme/edrs/application/026-pragmatic-hexagonal-architecture.md

@@ -21,7 +21,7 @@ How should application source code be organized to separate business logic from
 
 #### 01-three-layer-separation
 
-Every application is conceptually divided into three layers:
+Every application MUST be organized into these three conceptual layers:
 
 | Layer | Description |
 |-------|-------------|
@@ -31,6 +31,8 @@ Every application is conceptually divided into three layers:
 
 #### 02-adapter-naming-conventions
 
+Adapters MUST follow these naming conventions:
+
 **Inbound adapters** receive external requests or events and trigger application logic. Each gets a flat folder under `adapters/`:
 
 - `cli/` — command-line interface entry point
@@ -50,14 +52,24 @@ Every application is conceptually divided into three layers:
 #### 03-application-layer-rules
 
 - Expose functionality as typed library interfaces
-- All inputs must be explicitly passed as typed parameters
+- All inputs MUST be explicitly passed as typed parameters
 - No global variables, no direct environment variable access in `app/` or `shared/`
 - Business logic with well-defined input/output behavior
 - Group related logic into subfolders (aggregation roots)
 - Environment variables must be read only in the bootstrap/entry-point layer of inbound adapters, converted into typed configuration objects, and passed explicitly to all other components
 
+- Data flow examples
+
+```text
+HTTP request  →  adapters/http/     →  app/create-user     →  adapters/connectors/postgres/
+CLI command   →  adapters/cli/      →  app/create-dir      →  adapters/connectors/local-fs/
+Kafka message →  adapters/kafka/    →  app/process-event   →  adapters/connectors/stripe-api/
+```
+
 #### 04-mandatory-folder-structure
 
+All projects MUST follow this folder structure:
+
 ```text
 mysystem/
   Makefile             # targets to run different inbound interfaces (e.g. run-http, run-cli)
@@ -89,7 +101,7 @@ mysystem/
 
 #### 06-bootstrap-and-entry-points
 
-- Each inbound adapter folder (`cli/`, `http/`, `grpc/`, etc.) contains the bootstrap and entry point for that interface
+- Each inbound adapter folder (`cli/`, `http/`, `grpc/`, etc.) MUST contain the bootstrap and entry point for that interface
 - The project root Makefile must have targets to run the different inbound interfaces following [agentme-edr-008](../devops/008-common-targets.md) extension conventions (e.g. `run-http`, `run-grpc`)
 - Bootstrap code lives in the adapter that receives inbound requests, not in a separate wiring layer
 
@@ -98,15 +110,56 @@ mysystem/
 - Trivial scripts and single-purpose tools (fewer than ~300 lines with a single I/O boundary) MAY skip this layering
 - All other projects MUST use this structure from the start
 
-#### 08-examples-of-data-flow
+#### 09-unit-testing-and-mocking-strategy
 
-```text
-HTTP request  →  adapters/http/     →  app/create-user     →  adapters/connectors/postgres/
-CLI command   →  adapters/cli/      →  app/create-dir      →  adapters/connectors/local-fs/
-Kafka message →  adapters/kafka/    →  app/process-event   →  adapters/connectors/stripe-api/
+Unit tests for the `app/` layer MUST mock outbound adapter/connector interfaces at the `app/` → `adapters/connectors/` boundary. Inject connectors as constructor parameters or function arguments so tests can substitute them without touching real databases, HTTP APIs, or external services.
+
+The connector implementations themselves SHOULD have their own unit tests that mock the underlying SDK or HTTP client.
+
+```python
+# Good — inject connector; unit test mocks it
+class OrderService:
+    def __init__(self, db: OrderRepository):
+        self.db = db
+
+def test_create_order_persists_record():
+    fake_db = FakeOrderRepository()
+    service = OrderService(db=fake_db)
+    order = service.create({"item": "widget", "qty": 2})
+    assert fake_db.find(order.id) is not None
 ```
 
+Inbound adapters (`cli/`, `http/`, `grpc/`) are entry points and do not need to be mocked — test the `app/` layer directly by injecting fakes for its outbound connectors. See rule `10` for the naming and placement convention for shared mock files.
+
+#### 10-mock-file-strategy
+
+When a mock implementation needs to be **reused across multiple tests or imported by an eval script** (e.g. `eval.py` using `mock_fixtures` from [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02`), define it in a dedicated `_mock` file rather than inline.
+
+**When to use a `_mock` file vs inline:**
+- Single-test use → define the mock inline inside the test file (per rule `09` example; no file needed)
+- Reusable across multiple tests OR used from `eval.py` → define in a separate `_mock` file
+
+**Scope:** applies to any source file in `adapters/connectors/`, `app/`, or `shared/`. MUST NOT be used for inbound adapters (`cli/`, `http/`, `grpc/`) — those are entry points and are never mocked (rule `09`).
+
+**Naming:** insert `_mock` immediately before the file extension:
+
+| Source file | Mock file |
+|---|---|
+| `client.py` | `client_mock.py` |
+| `order_service.ts` | `order_service_mock.ts` |
+| `user_store.go` | `user_store_mock_test.go` |
+
+**Placement:** follows the project's test file placement convention per [agentme-edr-004](../principles/004-unit-test-requirements.md) rule `04`:
+- Co-located test convention (TypeScript, Go) → mock file in the same directory as the source file
+- Separate test folder convention (Python) → mock file mirrors the source path under the test folder (e.g. `lib/src/<pkg>/adapters/connectors/user-db/client.py` → `lib/tests/<pkg>/adapters/connectors/user-db/client_mock.py`)
+
+**Mock contract:**
+- MUST accept a `fixtures` parameter (constructor argument or factory function argument); the value is whatever `mock_fixtures[key]` contains from the dataset entry — its internal structure is opaque and interpreted by the mock implementation
+- MUST NOT fall back to real external calls under any circumstance — if a call cannot be satisfied from the provided fixtures, MUST raise an explicit error (never silently return `null`, `undefined`, or an empty value)
+
 ## References
 
 - [agentme-edr-016](../principles/016-cross-language-module-structure.md) — Defines the module-root structure (Makefile, dist/, .cache/) that wraps this internal layout
 - [agentme-edr-002](../principles/002-coding-best-practices.md) — File size limits and code organization practices that complement this architecture
+- [agentme-edr-004](../principles/004-unit-test-requirements.md) — Rule `04`: test file placement convention per language (governs `_mock` file placement in rule `10`)
+- [agentme-edr-030](030-ai-test-types-taxonomy.md) — Rule `02`: `mock_fixtures` golden dataset envelope that drives `_mock` usage in eval scripts

package/.xdrs/agentme/edrs/application/028-ai-eval-standards.md

@@ -1,6 +1,6 @@
 ---
 name: agentme-edr-policy-028-ai-eval-standards
-description: Defines how to structure, write, and run eval tests for AI projects — folder layout, script requirements, and MLflow tracking. Use when implementing evals for LLM, Agent, or Workflow projects. For when evals are required see agentme-edr-007 rule 09-ai-project-testing-requirements.
+description: Defines how to structure, write, and run eval tests for AI projects — folder layout, golden dataset, --type test-type filtering, mock_fixtures wiring, entry-first eval loop, per-type Makefile targets and reports, and MLflow tracking. Use when implementing evals for LLM, Agent, or Workflow projects. For when evals are required see agentme-edr-007 rule 09-ai-project-testing-requirements. For the test type taxonomy and mock_fixtures envelope see agentme-edr-030. For mock file naming see agentme-edr-026 rule 10.
 apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
 valid-from: 2026-06-05
 ---
@@ -29,10 +29,10 @@ Evals are grouped first by the component being evaluated, then by the specific e
 evals/
   <component>/           # the component being evaluated (e.g., workflow-x, agent-y, model-z)
     eval-<name>/
-      dataset/           # EDR-024 compliant dataset (README.md, dataset.schema.json, data/)
+      golden_dataset/    # EDR-024 + EDR-030 compliant golden dataset (README.md, dataset.schema.json, data/)
       eval.py            # evaluation script
-      report.md          # generated report (overwritten on each run — see rule 03)
-      Makefile           # eval and run targets
+      report-<type>.md   # generated report, one per evaluated test type (overwritten on each run — see rule 03)
+      Makefile           # lint, eval, run, and eval-<type> targets
     eval-<name2>/
       ...
   <component2>/
@@ -41,73 +41,126 @@ evals/
 
 `<component>` MUST match the name of the component under evaluation and use lowercase hyphen-separated words (e.g., `workflow-document-review`, `agent-support`, `model-classifier`).
 
-`<name>` identifies the specific evaluation scenario using lowercase hyphen-separated words (e.g., `eval-basic`, `eval-complex`, `eval-edge-cases`, `eval-bias-test`).
+`<name>` identifies the specific evaluation scenario using lowercase hyphen-separated words (e.g., `eval-basic`, `eval-complex`, `eval-edge-cases`). A scenario's `golden_dataset` MAY mix multiple test types across its entries: label each entry with its applicable `test_types` ([agentme-edr-030](030-ai-test-types-taxonomy.md) rule `04`) and use the `eval-<type>` targets below to run one type at a time.
 
-The `dataset/` subfolder MUST be a valid [agentme-edr-024](024-ml-dataset-structure.md) dataset — it MUST include `README.md` and `dataset.schema.json` at its root. For input/output pairs, use JSONL files per `agentme-edr-024.04-complex-structured-datasets-must-use-jsonl`.
+The `golden_dataset/` subfolder MUST be a valid [agentme-edr-024](024-ml-dataset-structure.md) dataset (`README.md`, `dataset.schema.json`, one JSON file per entry under `data/` per rule `04-complex-structured-datasets-must-use-per-entry-json-files`, lint-validated per rule `06`) whose entries follow the golden dataset envelope defined in [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02`.
 
-Each `evals/<component>/eval-<name>/Makefile` MUST define:
+Each `evals/<component>/eval-<name>/Makefile` MUST declare a `TEST_TYPES` variable listing the `test_types` values present in its golden dataset, and define:
 
 | Target | Behaviour |
 |---|---|
-| `eval` | Runs the eval with threshold enforcement; exits non-zero on failure (CI-safe) |
-| `run` | Runs the eval without threshold enforcement (exploration / debugging) |
+| `lint` | Validates every `golden_dataset/data/*.json` file against `golden_dataset/dataset.schema.json` per [agentme-edr-024](024-ml-dataset-structure.md) rule `06` |
+| `eval` | Depends on `lint`; runs `eval.py --type=all` with threshold enforcement; exits non-zero on failure (CI-safe) |
+| `run` | Depends on `lint`; runs `eval.py --type=all` without threshold enforcement (exploration / debugging) |
+| `eval-<type>` | Depends on `lint`; runs `eval.py --type=<type>` for one declared test type, following [agentme-edr-008](../devops/008-common-targets.md) rule `03`'s `eval-<qualifier>` convention |
 
-The module root Makefile MUST expose a `make eval` target that delegates to `eval` in every `evals/<component>/eval-<name>/Makefile`:
+```makefile
+TEST_TYPES := smoke functional safety
+
+lint:
+	mise exec -- uv run --project . python lint_dataset.py golden_dataset/
+
+eval: lint
+	mise exec -- uv run --project . python eval.py --type=all
+
+run: lint
+	mise exec -- uv run --project . python eval.py --type=all --no-threshold
+
+eval-%: lint
+	mise exec -- uv run --project . python eval.py --type=$*
+```
+
+The module root Makefile MUST expose `make eval` and `make lint` targets that delegate to `eval` and `lint` respectively in every `evals/<component>/eval-<name>/Makefile`:
 
 ```makefile
 eval:
 	$(MAKE) -C evals/workflow-document-review/eval-basic eval
 	$(MAKE) -C evals/workflow-document-review/eval-complex eval
+
+lint:
+	$(MAKE) -C evals/workflow-document-review/eval-basic lint
+	$(MAKE) -C evals/workflow-document-review/eval-complex lint
 ```
 
 #### 02-eval-script-requirements
 
 Each `eval.py` script MUST:
 
-- Load the dataset from `dataset/` in the same eval folder, following [agentme-edr-024](024-ml-dataset-structure.md). For input/output pairs, use the JSONL format per `agentme-edr-024.04-complex-structured-datasets-must-use-jsonl`.
-- Run every input through the live component against **real LLM providers** (not mocked responses), to capture model drift.
-- Log per-sample and aggregate metrics to an MLflow experiment that runs **locally** — a remote MLflow server MUST NOT be required.
-- Compare outputs to expected values using project-defined quality thresholds. Thresholds MUST be declared explicitly (e.g., in a Makefile variable or README).
-- Write `report.md` in the same folder per rule `03-eval-report-file`.
-- Exit with a non-zero status when any metric falls below its defined threshold, consistent with [agentme-edr-007](../principles/007-project-quality-standards.md) rule `07-statistical-models-must-have-eval-targets`.
+- Load the golden dataset from `golden_dataset/` in the same eval folder, following [agentme-edr-024](024-ml-dataset-structure.md) and the entry envelope in [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02` (one JSON file per entry, `test_types` array, `input`, `expected_output`, optional `mock_fixtures`).
+- Accept a required `--type=<test_type>|all` CLI argument and filter entries whose `test_types` array contains the requested value; `--type=all` includes every entry.
+- Iterate **entry-first**: for each entry in the filtered set, invoke the real component exactly once; then score that single `actual_output` for every `test_types` value the entry carries that falls within the current `--type` scope — never invoke the component more than once per entry per run.
+- When an entry contains `mock_fixtures` ([agentme-edr-030](030-ai-test-types-taxonomy.md) rule `02`), configure each named mock adapter with its fixture data BEFORE invoking the component for that entry. Each entry MUST use fresh mock instances so fixture state does not bleed across entries. `mock_fixtures` applies to all test types including `human`. `mock_fixtures` MUST NOT configure LLM adapters — the LLM call MUST always be real (see [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `03`). How mock adapters are discovered and instantiated is left to the project; see [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
+- Run every component invocation against **real LLM providers** (not mocked responses), to capture model drift.
+- For `human` entries: invoke the component to capture `actual_output`, export each entry's `input`, `expected_output.human_test` instructions, and `actual_output` into a manual-review checklist (`report-human.md`). MUST NOT invoke an automated scorer and MUST NOT enforce a pass/fail threshold for it. Other `test_types` on the same entry (e.g. `functional`) are still scored automatically.
+- After all entries are processed, compute aggregate metrics per test type, log them to a local MLflow experiment (see rule `04`), write one `report-<type>.md` per evaluated test type (rule `03`), and exit with a non-zero status when any metric falls below its defined threshold per [agentme-edr-007](../principles/007-project-quality-standards.md) rule `07-statistical-models-must-have-eval-targets`. The `human` type has no threshold and does not trigger a non-zero exit.
+- Compare outputs to expected values using project-defined quality thresholds per test type. Thresholds MUST be declared explicitly (e.g., in a Makefile variable or README) — this Policy does not mandate which test types a project must threshold or what value to use (see [agentme-edr-030](030-ai-test-types-taxonomy.md) rule `06`).
 
 **Example:**
 
 ```python
+import argparse
+from collections import defaultdict
 import mlflow
 from my_package.app.workflows.document_review_workflow.graph import graph
 
-EVAL_MIN_ACCURACY = 0.85
+EVAL_MIN_ACCURACY = {"functional": 0.85, "smoke": 0.85}
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--type", required=True)
+args = parser.parse_args()
 
-with mlflow.start_run() as run:
-    results = []
-    for sample in load_dataset("dataset/"):
-        output = graph.invoke({"document": sample["input"]})
-        results.append(output["label"] == sample["expected_label"])
+entries = load_golden_dataset("golden_dataset/", test_type=args.type)  # "all" loads every entry
+resolved_types = resolve_types(args.type, entries)
 
-    accuracy = sum(results) / len(results)
-    mlflow.log_metric("accuracy", accuracy)
+mlflow.set_experiment("document-review/eval-basic")
 
-    write_eval_report(run, results, thresholds={"accuracy": EVAL_MIN_ACCURACY})
+with mlflow.start_run():
+    mlflow.set_tag("test_types", ",".join(sorted(resolved_types)))
 
-    if accuracy < EVAL_MIN_ACCURACY:
-        raise SystemExit(f"Eval failed: accuracy {accuracy:.2f} < {EVAL_MIN_ACCURACY}")
+    results = defaultdict(list)
+
+    # Entry-first loop: invoke each entry exactly once
+    for entry in entries:
+        # Configure mock adapters from mock_fixtures before invocation
+        # (implementation left to the project — see agentme-edr-026 rule 10)
+        if entry.get("mock_fixtures"):
+            configure_mocks(entry["mock_fixtures"])  # project-defined helper
+
+        actual_output = invoke_component(entry, graph)
+
+        for test_type in [t for t in entry["test_types"] if t in resolved_types]:
+            if test_type == "human":
+                export_human_review(entry, actual_output)
+                continue
+            results[test_type].append(score(test_type, actual_output, entry["expected_output"]))
+
+    # Aggregate, report, and enforce thresholds per test type
+    for test_type in resolved_types:
+        if test_type == "human":
+            continue
+
+        accuracy = sum(results[test_type]) / len(results[test_type])
+        mlflow.log_metric(f"{test_type}_accuracy", accuracy)
+        write_eval_report(test_type, results[test_type], thresholds={"accuracy": EVAL_MIN_ACCURACY[test_type]})
+
+        if accuracy < EVAL_MIN_ACCURACY[test_type]:
+            raise SystemExit(f"Eval failed: {test_type} accuracy {accuracy:.2f} < {EVAL_MIN_ACCURACY[test_type]}")
 ```
 
 #### 03-eval-report-file
 
-Each eval script MUST produce `report.md` in the same `evals/<component>/eval-<name>/` folder and overwrite it on every run.
+Each eval script MUST produce one `report-<type>.md` per evaluated test type in the same `evals/<component>/eval-<name>/` folder and overwrite each on every run — only the types included in the current `--type` invocation are (re)written; report files for other types are left untouched. The `human` type does not produce a metrics report (see below).
 
 **Generation constraint:** The report MUST be produced programmatically, reading raw metric values directly from MLflow. No LLM or generative model may write, summarize, or paraphrase any section of the report, to prevent hallucinated metric values.
 
 The report MUST follow this template:
 
 ```markdown
-# Eval Report: <name>
+# Eval Report: <name> — <type>
 
 **Date:** <ISO date>
-**Dataset:** dataset/
-**Script:** eval.py
+**Dataset:** golden_dataset/
+**Script:** eval.py --type=<type>
 **Thresholds:** accuracy ≥ <value>, F1 ≥ <value>
 
 ## Overall Results
@@ -142,14 +195,14 @@ $$\frac{\hat{p} + \frac{z^2}{2n} \pm z\sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac
 
 Where $\hat{p}$ is observed accuracy and $n$ is sample count. Accuracy and F1 are required; precision and recall are recommended.
 
-**Filled-in example** (`evals/workflow-document-review/eval-basic/report.md` for a document review workflow):
+**Filled-in example** (`evals/workflow-document-review/eval-basic/report-functional.md` for a document review workflow):
 
 ```markdown
-# Eval Report: eval-basic
+# Eval Report: eval-basic — functional
 
 **Date:** 2026-06-12
-**Dataset:** dataset/
-**Script:** eval.py
+**Dataset:** golden_dataset/
+**Script:** eval.py --type=functional
 **Thresholds:** accuracy ≥ 0.85, F1 ≥ 0.80
 
 ## Overall Results
@@ -169,7 +222,7 @@ Where $\hat{p}$ is observed accuracy and $n$ is sample count. Accuracy and F1 ar
 ## Per-item Results
 
 | ID  | Input Summary                       | Expected | Actual   | Correct |
-|-----|-------------------------------------|----------|----------|---------|
+|-----|--------------------------------------|----------|----------|---------|
 | 001 | Contract renewal, 3 pages, standard | approve  | approve  | ✓       |
 | 002 | NDA with unusual liability clause   | escalate | escalate | ✓       |
 | 003 | Vendor invoice, missing PO number   | reject   | reject   | ✓       |
@@ -179,20 +232,26 @@ Where $\hat{p}$ is observed accuracy and $n$ is sample count. Accuracy and F1 ar
 ## Notes
 
 - Sample 005 misclassified: redlined IP clause not flagged as escalation trigger. Possible model drift.
-- MLflow run: experiment `workflow-document-review/eval-basic` — view with `mlflow ui`
+- MLflow run: experiment `workflow-document-review/eval-basic`, tag `test_types=functional` — view with `mlflow ui`
 ```
 
+**`human` type artifact:** instead of `report-human.md` with metrics, `--type=human` produces a checklist artifact (still named `report-human.md`) listing, per entry, its `input`, `expected_output.human_test` instructions, and the captured `actual_output` — with no Overall Results table, threshold, or PASS/FAIL section, since this type is never auto-scored.
+
 #### 04-eval-mlflow-unique-port
 
 Each `evals/<component>/eval-<name>/Makefile` MUST start its MLflow tracking server on a **unique port** to prevent conflicts when multiple eval Makefiles are run concurrently or in parallel (e.g., in CI or across multiple terminal sessions).
 
-Ports MUST be statically assigned per eval scenario and MUST NOT reuse the default `5000` port (reserved for `dev-mlflow` per [agentme-edr-008](../devops/008-common-targets.md) rule `09-ai-project-dev-targets`). Assign ports starting at `5100` and incrementing by 1 for each additional eval scenario across the entire project.
+Ports MUST be statically assigned per eval scenario (not per test type) and MUST NOT reuse the default `5000` port (reserved for `dev-mlflow` per [agentme-edr-008](../devops/008-common-targets.md) rule `09-ai-project-dev-targets`). Assign ports starting at `5100` and incrementing by 1 for each additional eval scenario across the entire project.
+
+The MLflow **experiment** is scoped to the eval scenario: `<component>/<eval-name>` (e.g. `document-review/eval-basic`). Each `mlflow.start_run()` call MUST set a `test_types` tag listing the test types evaluated in that invocation (comma-separated, e.g. `"functional,smoke"` for `--type=all`, `"smoke"` for `--type=smoke`). A remote MLflow server MUST NOT be required — all tracking is local.
 
 ## References
 
 - [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09-ai-project-testing-requirements`) and statistical model eval targets (rule `07-statistical-models-must-have-eval-targets`)
+- [agentme-edr-030](030-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum, golden dataset entry envelope (including `mock_fixtures`), and mocking constraints per type
+- [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) — Rule `10`: `_mock` file naming and placement convention for mock adapters used in `mock_fixtures`
 - [agentme-edr-018](018-ai-llm-development-standards.md) — LLM development standards: LangChain framework and observability
 - [agentme-edr-019](019-ai-agents-development-standards.md) — Agent development standards
 - [agentme-edr-021](021-ai-workflow-development-standards.md) — Workflow development standards
-
-- [agentme-edr-024](024-ml-dataset-structure.md) — ML dataset structure for eval datasets
+- [agentme-edr-024](024-ml-dataset-structure.md) — ML dataset structure, per-entry JSON format, and schema-lint validation for golden datasets
+- [agentme-edr-008](../devops/008-common-targets.md) — `eval-<qualifier>` Makefile convention (rule `03`) and Mise tool-execution flow (rule `02`)