agentme 0.28.1 → 0.29.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (40) hide show
  1. package/.filedist-package.yml +1 -1
  2. package/.xdrs/agentme/edrs/application/{003-javascript-project-tooling.md → 101-javascript-project-tooling.md} +8 -8
  3. package/.xdrs/agentme/edrs/application/{010-golang-project-tooling.md → 102-golang-project-tooling.md} +6 -6
  4. package/.xdrs/agentme/edrs/application/{014-python-project-tooling.md → 103-python-project-tooling.md} +11 -11
  5. package/.xdrs/agentme/edrs/application/{015-cli-tool-standards.md → 104-cli-tool-standards.md} +13 -13
  6. package/.xdrs/agentme/edrs/application/{002-coding-best-practices.md → 121-coding-best-practices.md} +3 -3
  7. package/.xdrs/agentme/edrs/application/{004-unit-test-requirements.md → 122-unit-test-requirements.md} +2 -2
  8. package/.xdrs/agentme/edrs/application/{009-error-handling.md → 123-error-handling.md} +2 -2
  9. package/.xdrs/agentme/edrs/application/{022-secrets-management.md → 124-secrets-management.md} +5 -5
  10. package/.xdrs/agentme/edrs/application/{023-coding-abstraction-practices.md → 125-coding-abstraction-practices.md} +3 -3
  11. package/.xdrs/agentme/edrs/application/{026-pragmatic-hexagonal-architecture.md → 126-pragmatic-hexagonal-architecture.md} +9 -9
  12. package/.xdrs/agentme/edrs/application/{040-ai-llm-development-standards.md → 141-ai-llm-development-standards.md} +14 -14
  13. package/.xdrs/agentme/edrs/application/{041-ai-agents-development-standards.md → 142-ai-agents-development-standards.md} +11 -11
  14. package/.xdrs/agentme/edrs/application/{042-ai-agents-quality-standards.md → 143-ai-agents-quality-standards.md} +16 -16
  15. package/.xdrs/agentme/edrs/application/{043-ai-workflow-development-standards.md → 144-ai-workflow-development-standards.md} +28 -28
  16. package/.xdrs/agentme/edrs/application/{044-ai-workflow-naming-conventions.md → 145-ai-workflow-naming-conventions.md} +9 -9
  17. package/.xdrs/agentme/edrs/application/{045-ai-agent-xdrs-knowledge-layer.md → 146-ai-agent-xdrs-knowledge-layer.md} +7 -7
  18. package/.xdrs/agentme/edrs/application/{051-ai-eval-core-standards.md → 151-ai-eval-core-standards.md} +17 -17
  19. package/.xdrs/agentme/edrs/application/{052-ai-test-types-taxonomy.md → 152-ai-test-types-taxonomy.md} +27 -27
  20. package/.xdrs/agentme/edrs/application/{053-ai-eval-script.md → 153-ai-eval-script.md} +18 -18
  21. package/.xdrs/agentme/edrs/application/{054-ai-eval-report-format.md → 154-ai-eval-report-format.md} +6 -6
  22. package/.xdrs/agentme/edrs/application/{055-ai-eval-repeatability.md → 155-ai-eval-repeatability.md} +14 -14
  23. package/.xdrs/agentme/edrs/application/skills/001-create-javascript-project/SKILL.md +4 -4
  24. package/.xdrs/agentme/edrs/application/skills/003-create-golang-project/SKILL.md +3 -3
  25. package/.xdrs/agentme/edrs/application/skills/004-select-relevant-xdrs/SKILL.md +9 -9
  26. package/.xdrs/agentme/edrs/application/skills/005-create-python-project/SKILL.md +5 -5
  27. package/.xdrs/agentme/edrs/data/{050-ml-dataset-structure.md → 201-ml-dataset-structure.md} +4 -4
  28. package/.xdrs/agentme/edrs/governance/{007-project-quality-standards.md → 501-project-quality-standards.md} +16 -16
  29. package/.xdrs/agentme/edrs/governance/{013-contributing-guide-requirements.md → 502-contributing-guide-requirements.md} +3 -3
  30. package/.xdrs/agentme/edrs/index.md +30 -30
  31. package/.xdrs/agentme/edrs/operations/{011-service-health-check-endpoint.md → 401-service-health-check-endpoint.md} +2 -2
  32. package/.xdrs/agentme/edrs/platform/{005-monorepo-structure.md → 301-monorepo-structure.md} +4 -4
  33. package/.xdrs/agentme/edrs/platform/{006-github-pipelines.md → 302-github-pipelines.md} +3 -3
  34. package/.xdrs/agentme/edrs/platform/{008-common-targets.md → 303-common-targets.md} +8 -8
  35. package/.xdrs/agentme/edrs/platform/{017-tool-execution-and-scripting.md → 304-tool-execution-and-scripting.md} +10 -10
  36. package/.xdrs/agentme/edrs/platform/{027-environment-variable-configuration.md → 305-environment-variable-configuration.md} +14 -14
  37. package/.xdrs/agentme/edrs/platform/skills/002-monorepo-setup/SKILL.md +3 -3
  38. package/.xdrs/agentme/edrs/principles/012-continuous-xdr-enrichment.md +1 -1
  39. package/.xdrs/agentme/edrs/principles/016-cross-language-module-structure.md +4 -4
  40. package/package.json +2 -2
@@ -1,11 +1,11 @@
1
1
  ---
2
- name: agentme-edr-policy-042-ai-agents-quality-standards
3
- description: Defines implementation quality standards for AI agents: tool definition patterns, error handling and recovery, observability, and unit testing. Apply alongside agentme-edr-041 when implementing or reviewing agent code. For agent architecture and structural decisions (framework, sandbox, naming, composition, system prompts) see agentme-edr-041.
2
+ name: agentme-edr-policy-143-ai-agents-quality-standards
3
+ description: Defines implementation quality standards for AI agents: tool definition patterns, error handling and recovery, observability, and unit testing. Apply alongside agentme-edr-142 when implementing or reviewing agent code. For agent architecture and structural decisions (framework, sandbox, naming, composition, system prompts) see agentme-edr-142.
4
4
  apply-to: AI agent implementation code — apply when writing tools, error handlers, logging, and unit tests for agents
5
5
  valid-from: 2026-06-09
6
6
  ---
7
7
 
8
- # agentme-edr-policy-042: AI agents quality standards
8
+ # agentme-edr-policy-143: AI agents quality standards
9
9
 
10
10
  ## Context and Problem Statement
11
11
 
@@ -15,7 +15,7 @@ How should agent tools be defined, what error handling must agents implement, ho
15
15
 
16
16
  ## Decision Outcome
17
17
 
18
- **Agent implementations MUST follow the tool definition, error handling, observability, and unit testing standards defined here, alongside the structural decisions in [agentme-edr-041](041-ai-agents-development-standards.md).**
18
+ **Agent implementations MUST follow the tool definition, error handling, observability, and unit testing standards defined here, alongside the structural decisions in [agentme-edr-142](142-ai-agents-development-standards.md).**
19
19
 
20
20
  ### Details
21
21
 
@@ -49,7 +49,7 @@ def tool_name(arg1: str, arg2: int) -> Dict[str, Any]:
49
49
  - Tool docstrings MUST clearly describe the tool's purpose, arguments, and return value (the LLM reads these)
50
50
  - Tools MUST return structured data (dictionaries or dataclasses), not bare strings or untyped values
51
51
  - Tools MUST handle errors gracefully and return error information in the result structure, not raise exceptions
52
- - Tools that interact with external systems MUST be placed in `adapters/connectors/` per [agentme-edr-026](026-pragmatic-hexagonal-architecture.md)
52
+ - Tools that interact with external systems MUST be placed in `adapters/connectors/` per [agentme-edr-126](126-pragmatic-hexagonal-architecture.md)
53
53
 
54
54
  **Error handling in tools:**
55
55
 
@@ -100,9 +100,9 @@ Agent execution MUST be observable through logging and tracing:
100
100
 
101
101
  - Log each iteration of the perceive → plan → act → observe cycle with iteration number and tool selection.
102
102
  - Use structured logging (JSON) with fields: `iteration`, `tool_selected`, `tool_result_status`, `decision`.
103
- - For LLM calls within agents, follow [agentme-edr-040](040-ai-llm-development-standards.md) rule `03-llm-observability`.
103
+ - For LLM calls within agents, follow [agentme-edr-141](141-ai-llm-development-standards.md) rule `03-llm-observability`.
104
104
  - When agents run as workflow nodes, MLflow tracking from the parent workflow automatically captures agent-level traces.
105
- - The project Makefile MUST expose a `dev-mlflow` target to start a local MLflow tracking server for development inspection, per [agentme-edr-008](../platform/008-common-targets.md) rule `09-ai-project-dev-targets`.
105
+ - The project Makefile MUST expose a `dev-mlflow` target to start a local MLflow tracking server for development inspection, per [agentme-edr-303](../platform/303-common-targets.md) rule `09-ai-project-dev-targets`.
106
106
 
107
107
  **Example structured log entry:**
108
108
 
@@ -120,7 +120,7 @@ Agent execution MUST be observable through logging and tracing:
120
120
 
121
121
  #### 04-agent-unit-testing
122
122
 
123
- Agent LLM calls are external API calls and MUST be mocked in unit tests per [agentme-edr-040](040-ai-llm-development-standards.md) rule `04-unit-test-mocking`.
123
+ Agent LLM calls are external API calls and MUST be mocked in unit tests per [agentme-edr-141](141-ai-llm-development-standards.md) rule `04-unit-test-mocking`.
124
124
 
125
125
  Because agents drive a tool-invocation loop — where the LLM decides which tools to call — the fake model must return **tool-call messages** followed by a final answer. Use **`GenericFakeChatModel`** for this:
126
126
 
@@ -150,7 +150,7 @@ def test_file_analyzer_agent_calls_search_then_stops():
150
150
  assert "3 Python files" in result.summary
151
151
  ```
152
152
 
153
- Agents MUST be designed so that the LLM instance is injectable (constructor parameter) to allow test doubles. See [agentme-edr-040](040-ai-llm-development-standards.md) rule `04-unit-test-mocking` for the injectable LLM pattern.
153
+ Agents MUST be designed so that the LLM instance is injectable (constructor parameter) to allow test doubles. See [agentme-edr-141](141-ai-llm-development-standards.md) rule `04-unit-test-mocking` for the injectable LLM pattern.
154
154
 
155
155
  **`mock_deep_agent`**
156
156
 
@@ -175,10 +175,10 @@ def test_workflow_calls_subagent(mocker):
175
175
 
176
176
  ## References
177
177
 
178
- - [agentme-edr-041](041-ai-agents-development-standards.md) — Agent development standards (framework, sandbox, naming, composition, system prompts)
179
- - [agentme-edr-040](040-ai-llm-development-standards.md) — LLM development standards (LangChain configuration, mocking patterns)
180
- - [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) — Hexagonal architecture (tool placement in adapters/connectors)
181
- - [agentme-edr-007](../governance/007-project-quality-standards.md) — Project quality standards including AI-tier testing requirements (rule `09-ai-project-testing-requirements`)
182
- - [agentme-edr-051](051-ai-eval-core-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract (rule `02`)
183
- - [agentme-edr-053](053-ai-eval-script.md) — AI eval script: entry-first loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
184
- - [agentme-edr-054](054-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
178
+ - [agentme-edr-142](142-ai-agents-development-standards.md) — Agent development standards (framework, sandbox, naming, composition, system prompts)
179
+ - [agentme-edr-141](141-ai-llm-development-standards.md) — LLM development standards (LangChain configuration, mocking patterns)
180
+ - [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) — Hexagonal architecture (tool placement in adapters/connectors)
181
+ - [agentme-edr-501](../governance/501-project-quality-standards.md) — Project quality standards including AI-tier testing requirements (rule `09-ai-project-testing-requirements`)
182
+ - [agentme-edr-151](151-ai-eval-core-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract (rule `02`)
183
+ - [agentme-edr-153](153-ai-eval-script.md) — AI eval script: entry-first loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
184
+ - [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
@@ -1,11 +1,11 @@
1
1
  ---
2
- name: agentme-edr-policy-043-ai-workflow-development-standards
3
- description: Defines the standard toolchain, framework, observability, and workflow patterns for building LangGraph workflows in Python. Use when scaffolding, reviewing, or extending AI workflow projects that orchestrate LLM calls, agents, and algorithmic nodes. For simple LLM calls see agentme-edr-040, for agentic patterns see agentme-edr-041. For naming conventions (nodes, states, routes, judge output schema) see agentme-edr-044.
2
+ name: agentme-edr-policy-144-ai-workflow-development-standards
3
+ description: Defines the standard toolchain, framework, observability, and workflow patterns for building LangGraph workflows in Python. Use when scaffolding, reviewing, or extending AI workflow projects that orchestrate LLM calls, agents, and algorithmic nodes. For simple LLM calls see agentme-edr-141, for agentic patterns see agentme-edr-142. For naming conventions (nodes, states, routes, judge output schema) see agentme-edr-145.
4
4
  apply-to: AI workflow projects using LangGraph StateGraph built with Python
5
5
  valid-from: 2026-06-05
6
6
  ---
7
7
 
8
- # agentme-edr-policy-043: AI workflow development standards
8
+ # agentme-edr-policy-144: AI workflow development standards
9
9
 
10
10
  ## Context and Problem Statement
11
11
 
@@ -23,21 +23,21 @@ Which tools, frameworks, and design patterns should AI workflow projects follow
23
23
 
24
24
  Workflows MUST be built with **LangGraph**. Use LangGraph `StateGraph` to model each distinct workflow as an explicit directed graph with typed state.
25
25
 
26
- For all direct LLM calls within workflow nodes, use LangChain per [agentme-edr-040](040-ai-llm-development-standards.md). For agent nodes with tool-invocation loops, use deepagents per [agentme-edr-041](041-ai-agents-development-standards.md).
26
+ For all direct LLM calls within workflow nodes, use LangChain per [agentme-edr-141](141-ai-llm-development-standards.md). For agent nodes with tool-invocation loops, use deepagents per [agentme-edr-142](142-ai-agents-development-standards.md).
27
27
 
28
28
  #### 03-observability-and-experiment-tracking
29
29
 
30
30
  Use **MLflow** for all workflow observability and evaluation:
31
31
 
32
32
  - **Workflow-level tracking:** Wrap each workflow run with `mlflow.start_run()` to capture traces, parameters, and metrics locally.
33
- - **LLM-level auto-tracing:** Enable LangChain auto-tracing per [agentme-edr-040](040-ai-llm-development-standards.md) rule `03-llm-observability` by calling `mlflow.langchain.autolog()` during application startup. This captures inputs, outputs, token counts, and latency for every LangChain call within workflow nodes.
33
+ - **LLM-level auto-tracing:** Enable LangChain auto-tracing per [agentme-edr-141](141-ai-llm-development-standards.md) rule `03-llm-observability` by calling `mlflow.langchain.autolog()` during application startup. This captures inputs, outputs, token counts, and latency for every LangChain call within workflow nodes.
34
34
  - Log run parameters (model name, temperature, prompt version) and output metrics (accuracy, latency, token counts) using `mlflow.log_param` / `mlflow.log_metric`.
35
35
  - Run a local MLflow tracking server with `mlflow ui` to inspect runs during development. Do not require a remote MLflow server for local development.
36
- - The project Makefile MUST expose a `dev-mlflow` target to start the local MLflow tracking server, per [agentme-edr-008](../platform/008-common-targets.md) rule `09-ai-project-dev-targets`.
36
+ - The project Makefile MUST expose a `dev-mlflow` target to start the local MLflow tracking server, per [agentme-edr-303](../platform/303-common-targets.md) rule `09-ai-project-dev-targets`.
37
37
 
38
38
  #### 04-dataset-driven-accuracy-measurement
39
39
 
40
- Projects MUST follow the eval dataset and implementation requirements defined in [agentme-edr-051](051-ai-eval-core-standards.md) and [agentme-edr-053](053-ai-eval-script.md). Testing requirements (when evals are required, release gates) are defined in [agentme-edr-007](../governance/007-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
40
+ Projects MUST follow the eval dataset and implementation requirements defined in [agentme-edr-151](151-ai-eval-core-standards.md) and [agentme-edr-153](153-ai-eval-script.md). Testing requirements (when evals are required, release gates) are defined in [agentme-edr-501](../governance/501-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
41
41
 
42
42
  #### 05-flow-documentation
43
43
 
@@ -69,7 +69,7 @@ Workflows MUST include at least one explicit verification node before producing
69
69
 
70
70
  #### 07-workflow-structure
71
71
 
72
- Workflow logic MUST be organized as named workflows following [agentme-edr-026](026-pragmatic-hexagonal-architecture.md). Each workflow is an independent LangGraph `StateGraph` with a defined start node and end node, connecting LLM nodes, agent nodes, algorithmic nodes, states, routes, and decision nodes.
72
+ Workflow logic MUST be organized as named workflows following [agentme-edr-126](126-pragmatic-hexagonal-architecture.md). Each workflow is an independent LangGraph `StateGraph` with a defined start node and end node, connecting LLM nodes, agent nodes, algorithmic nodes, states, routes, and decision nodes.
73
73
 
74
74
  Workflows live inside `app/workflows/` (the application layer), while external integrations such as LLM providers, vector stores, and third-party APIs live under `adapters/connectors/` (the outbound adapter layer). Inbound interfaces (HTTP API, CLI) live under `adapters/` as inbound adapters.
75
75
 
@@ -101,17 +101,17 @@ lib/src/<package_name>/
101
101
 
102
102
  #### 08-workflow-evals
103
103
 
104
- Projects MUST follow the eval folder structure defined in [agentme-edr-051](051-ai-eval-core-standards.md) rule `01` and the eval script requirements defined in [agentme-edr-053](053-ai-eval-script.md). For LLM-as-judge scoring used in workflow verification nodes and evals, see [agentme-edr-051](051-ai-eval-core-standards.md) rule `02`.
104
+ Projects MUST follow the eval folder structure defined in [agentme-edr-151](151-ai-eval-core-standards.md) rule `01` and the eval script requirements defined in [agentme-edr-153](153-ai-eval-script.md). For LLM-as-judge scoring used in workflow verification nodes and evals, see [agentme-edr-151](151-ai-eval-core-standards.md) rule `02`.
105
105
 
106
106
  #### 09-node-naming-conventions
107
107
 
108
- Nodes MUST follow the naming conventions defined in [agentme-edr-044](044-ai-workflow-naming-conventions.md) rule `01-node-naming-conventions`.
108
+ Nodes MUST follow the naming conventions defined in [agentme-edr-145](145-ai-workflow-naming-conventions.md) rule `01-node-naming-conventions`.
109
109
 
110
110
  #### 10-workflow-unit-testing
111
111
 
112
- All LLM calls within workflow nodes are external API calls and MUST be mocked in unit tests per [agentme-edr-040](040-ai-llm-development-standards.md) rule `04-unit-test-mocking`. Workflow unit tests MUST run fully offline with no real LLM provider calls.
112
+ All LLM calls within workflow nodes are external API calls and MUST be mocked in unit tests per [agentme-edr-141](141-ai-llm-development-standards.md) rule `04-unit-test-mocking`. Workflow unit tests MUST run fully offline with no real LLM provider calls.
113
113
 
114
- Choose the mock utility per [agentme-edr-040](040-ai-llm-development-standards.md) rule `04-unit-test-mocking`. For workflows containing `_agent` nodes that drive a tool-invocation loop, MUST use `GenericFakeChatModel`.
114
+ Choose the mock utility per [agentme-edr-141](141-ai-llm-development-standards.md) rule `04-unit-test-mocking`. For workflows containing `_agent` nodes that drive a tool-invocation loop, MUST use `GenericFakeChatModel`.
115
115
 
116
116
  **Example — workflow with plain-text LLM nodes:**
117
117
 
@@ -152,19 +152,19 @@ def test_document_workflow_with_agent_node():
152
152
  assert result.status == "approved"
153
153
  ```
154
154
 
155
- Workflows MUST accept the LLM instance as a constructor parameter so that unit tests can inject a fake. See the injectable LLM pattern in [agentme-edr-040](040-ai-llm-development-standards.md) rule `04-unit-test-mocking`.
155
+ Workflows MUST accept the LLM instance as a constructor parameter so that unit tests can inject a fake. See the injectable LLM pattern in [agentme-edr-141](141-ai-llm-development-standards.md) rule `04-unit-test-mocking`.
156
156
 
157
157
  #### 11-state-type-conventions
158
158
 
159
- State types MUST follow the conventions defined in [agentme-edr-044](044-ai-workflow-naming-conventions.md) rule `02-state-type-conventions`.
159
+ State types MUST follow the conventions defined in [agentme-edr-145](145-ai-workflow-naming-conventions.md) rule `02-state-type-conventions`.
160
160
 
161
161
  #### 12-workflow-naming-conventions
162
162
 
163
- Workflows MUST be named following the conventions in [agentme-edr-044](044-ai-workflow-naming-conventions.md) rule `04-workflow-naming-conventions`.
163
+ Workflows MUST be named following the conventions in [agentme-edr-145](145-ai-workflow-naming-conventions.md) rule `04-workflow-naming-conventions`.
164
164
 
165
165
  #### 13-judge-node-output-format
166
166
 
167
- Judge nodes MUST use the output format defined in [agentme-edr-044](044-ai-workflow-naming-conventions.md) rule `03-judge-node-output-format`.
167
+ Judge nodes MUST use the output format defined in [agentme-edr-145](145-ai-workflow-naming-conventions.md) rule `03-judge-node-output-format`.
168
168
 
169
169
  #### 15-workflow-state-persistence
170
170
 
@@ -197,18 +197,18 @@ result = graph.invoke(input_state, config={"thread_id": "session-123"})
197
197
 
198
198
  #### 16-cross-element-naming-coherence
199
199
 
200
- All workflow elements MUST maintain naming coherence as defined in [agentme-edr-044](044-ai-workflow-naming-conventions.md) rule `05-cross-element-naming-coherence`.
200
+ All workflow elements MUST maintain naming coherence as defined in [agentme-edr-145](145-ai-workflow-naming-conventions.md) rule `05-cross-element-naming-coherence`.
201
201
 
202
202
  ## References
203
203
 
204
- - [agentme-edr-044](044-ai-workflow-naming-conventions.md) — AI workflow naming conventions: node suffixes/prefixes, state types, judge output schema, workflow names, and cross-element coherence
205
- - [agentme-edr-040](040-ai-llm-development-standards.md) — LLM development standards: LangChain framework, provider configuration, LLM observability, and unit test mocking
206
- - [agentme-edr-041](041-ai-agents-development-standards.md) — Agent development standards: deepagents framework, tool-invocation loops, and agent patterns
207
- - [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) — Adapter/application layer separation that defines the project layout
208
- - [agentme-edr-014](014-python-project-tooling.md) — Python project tooling and structure
209
- - [agentme-edr-050](../data/050-ml-dataset-structure.md) — ML dataset structure for eval datasets
210
- - [agentme-edr-051](051-ai-eval-core-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract for all tiers (rule `02`)
211
- - [agentme-edr-053](053-ai-eval-script.md) — AI eval script: entry-first loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
212
- - [agentme-edr-054](054-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
213
- - [agentme-edr-055](055-ai-eval-repeatability.md) — AI eval repeatability: REPEAT_COUNT loop, scoring methods, and release cadence
214
- - [agentme-edr-007](../governance/007-project-quality-standards.md) — Project quality standards including AI-tier testing requirements (rule `09-ai-project-testing-requirements`)
204
+ - [agentme-edr-145](145-ai-workflow-naming-conventions.md) — AI workflow naming conventions: node suffixes/prefixes, state types, judge output schema, workflow names, and cross-element coherence
205
+ - [agentme-edr-141](141-ai-llm-development-standards.md) — LLM development standards: LangChain framework, provider configuration, LLM observability, and unit test mocking
206
+ - [agentme-edr-142](142-ai-agents-development-standards.md) — Agent development standards: deepagents framework, tool-invocation loops, and agent patterns
207
+ - [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) — Adapter/application layer separation that defines the project layout
208
+ - [agentme-edr-103](103-python-project-tooling.md) — Python project tooling and structure
209
+ - [agentme-edr-201](../data/201-ml-dataset-structure.md) — ML dataset structure for eval datasets
210
+ - [agentme-edr-151](151-ai-eval-core-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract for all tiers (rule `02`)
211
+ - [agentme-edr-153](153-ai-eval-script.md) — AI eval script: entry-first loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
212
+ - [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
213
+ - [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: REPEAT_COUNT loop, scoring methods, and release cadence
214
+ - [agentme-edr-501](../governance/501-project-quality-standards.md) — Project quality standards including AI-tier testing requirements (rule `09-ai-project-testing-requirements`)
@@ -1,11 +1,11 @@
1
1
  ---
2
- name: agentme-edr-policy-044-ai-workflow-naming-conventions
3
- description: Defines the naming vocabulary for LangGraph workflow elements: node suffixes/prefixes, state type names, state attribute grouping, workflow class/variable names, judge node output schema, route function names, and cross-element coherence rules. Use when naming any part of a LangGraph workflow — nodes, states, routes, or the workflow itself. For workflow structure and toolchain see agentme-edr-043.
2
+ name: agentme-edr-policy-145-ai-workflow-naming-conventions
3
+ description: Defines the naming vocabulary for LangGraph workflow elements: node suffixes/prefixes, state type names, state attribute grouping, workflow class/variable names, judge node output schema, route function names, and cross-element coherence rules. Use when naming any part of a LangGraph workflow — nodes, states, routes, or the workflow itself. For workflow structure and toolchain see agentme-edr-144.
4
4
  apply-to: AI workflow projects using LangGraph StateGraph built with Python
5
5
  valid-from: 2026-06-21
6
6
  ---
7
7
 
8
- # agentme-edr-policy-044: AI workflow naming conventions
8
+ # agentme-edr-policy-145: AI workflow naming conventions
9
9
 
10
10
  ## Context and Problem Statement
11
11
 
@@ -25,10 +25,10 @@ LangGraph node names MUST follow a suffix convention that communicates the node'
25
25
 
26
26
  | Convention | Node type | When to use |
27
27
  |---|---|---|
28
- | suffix `_llm` | LLM call | Any node whose primary action is a direct LLM inference call (see [agentme-edr-040](040-ai-llm-development-standards.md)) |
28
+ | suffix `_llm` | LLM call | Any node whose primary action is a direct LLM inference call (see [agentme-edr-141](141-ai-llm-development-standards.md)) |
29
29
  | suffix `_step` | Algorithmic step | Deterministic logic with no LLM involvement (transformation, validation, routing) |
30
30
  | suffix `_tool` | Tool/API call | A node that wraps a single external tool or API (e.g. a REST endpoint, DB query) |
31
- | suffix `_agent` | Subgraph agent | A node that invokes a nested subgraph containing its own tool-invocation cycle and LLM calls; use the **deepagents** library for these nodes (see [agentme-edr-041](041-ai-agents-development-standards.md)) |
31
+ | suffix `_agent` | Subgraph agent | A node that invokes a nested subgraph containing its own tool-invocation cycle and LLM calls; use the **deepagents** library for these nodes (see [agentme-edr-142](142-ai-agents-development-standards.md)) |
32
32
  | prefix `evaluate_` | Judge node | A node that evaluates the quality, correctness, completeness, or progress of prior outputs and returns a structured verdict; MUST follow rule `03-judge-node-output-format` |
33
33
 
34
34
  The Python function implementing the node SHOULD share the same name as the node alias passed to `add_node`, so that graph definitions and stack traces remain unambiguous:
@@ -193,7 +193,7 @@ def route_after_evaluate_quality(state) -> str:
193
193
  return "publish_step"
194
194
  ```
195
195
 
196
- **Logging:** Log `verdict` and the count of each level as MLflow metrics on the current run per [agentme-edr-043](043-ai-workflow-development-standards.md) rule `03-observability-and-experiment-tracking`.
196
+ **Logging:** Log `verdict` and the count of each level as MLflow metrics on the current run per [agentme-edr-144](144-ai-workflow-development-standards.md) rule `03-observability-and-experiment-tracking`.
197
197
 
198
198
  #### 04-workflow-naming-conventions
199
199
 
@@ -276,6 +276,6 @@ def after_quality_check(state): ... # FORBIDDEN: not named "route_after_<node>"
276
276
 
277
277
  ## References
278
278
 
279
- - [agentme-edr-043](043-ai-workflow-development-standards.md) — Workflow structure, LangGraph toolchain, observability, and testing patterns
280
- - [agentme-edr-040](040-ai-llm-development-standards.md) — LLM development standards (drives `_llm` node suffix and mocking patterns)
281
- - [agentme-edr-041](041-ai-agents-development-standards.md) — Agent development standards (drives `_agent` node suffix and state conventions)
279
+ - [agentme-edr-144](144-ai-workflow-development-standards.md) — Workflow structure, LangGraph toolchain, observability, and testing patterns
280
+ - [agentme-edr-141](141-ai-llm-development-standards.md) — LLM development standards (drives `_llm` node suffix and mocking patterns)
281
+ - [agentme-edr-142](142-ai-agents-development-standards.md) — Agent development standards (drives `_agent` node suffix and state conventions)
@@ -1,11 +1,11 @@
1
1
  ---
2
- name: agentme-edr-policy-045-ai-agent-xdrs-knowledge-layer
2
+ name: agentme-edr-policy-146-ai-agent-xdrs-knowledge-layer
3
3
  description: Defines how to integrate XDRS as the runtime knowledge source of truth for AI agents — covering document placement, AGENTS.md setup, file tools, and local sandbox configuration. Apply only when the project explicitly uses XDRS to govern agent behavior.
4
4
  apply-to: AI agent projects that use XDRS as the source of truth for policies and skills
5
5
  valid-from: 2026-05-27
6
6
  ---
7
7
 
8
- # agentme-edr-policy-045: AI agent XDRS knowledge layer
8
+ # agentme-edr-policy-146: AI agent XDRS knowledge layer
9
9
 
10
10
  ## Context and Problem Statement
11
11
 
@@ -17,7 +17,7 @@ How should an AI agent project integrate XDRS as its runtime source of truth for
17
17
 
18
18
  **Embed XDRS documents in `lib/data/.xdrs/`, instruct the agent to consult them via `AGENTS.md`, equip the agent with sandboxed file tools, and use the deepagents framework when a local sandbox is required.**
19
19
 
20
- This policy MUST only be applied when the project explicitly chooses XDRS as its knowledge governance layer. It is not required by [agentme-edr-041](041-ai-agents-development-standards.md) or [agentme-edr-043](043-ai-workflow-development-standards.md) in general.
20
+ This policy MUST only be applied when the project explicitly chooses XDRS as its knowledge governance layer. It is not required by [agentme-edr-142](142-ai-agents-development-standards.md) or [agentme-edr-144](144-ai-workflow-development-standards.md) in general.
21
21
 
22
22
  ### Details
23
23
 
@@ -59,7 +59,7 @@ Read /AGENTS.md and follow all instructions in it before proceeding.
59
59
 
60
60
  #### 02-agent-file-tools
61
61
 
62
- Every agent that uses the XDRS knowledge layer MUST use the file tools provided by the deepagents framework. Do not implement hand-rolled alternatives — see [agentme-edr-041 rule 02-local-sandbox](041-ai-agents-development-standards.md) for the full sandbox and tool requirements.
62
+ Every agent that uses the XDRS knowledge layer MUST use the file tools provided by the deepagents framework. Do not implement hand-rolled alternatives — see [agentme-edr-142 rule 02-local-sandbox](142-ai-agents-development-standards.md) for the full sandbox and tool requirements.
63
63
 
64
64
  These tools operate over two sandboxed roots (configured in rule `03-local-sandbox`):
65
65
 
@@ -72,7 +72,7 @@ These tools operate over two sandboxed roots (configured in rule `03-local-sandb
72
72
 
73
73
  #### 03-local-sandbox
74
74
 
75
- Follow [agentme-edr-041 rule 02-local-sandbox](041-ai-agents-development-standards.md) for the general deepagents sandbox setup. When XDRS is in use, add the following mounts to the sandbox configuration:
75
+ Follow [agentme-edr-142 rule 02-local-sandbox](142-ai-agents-development-standards.md) for the general deepagents sandbox setup. When XDRS is in use, add the following mounts to the sandbox configuration:
76
76
 
77
77
  | Source | Content | Deepagents sandbox path |
78
78
  |---|---|---|
@@ -91,8 +91,8 @@ data_root = str(files("myagent").joinpath("data"))
91
91
  agents_md = Path(temp_root) / "AGENTS.md"
92
92
  agents_md.write_text(_AGENTS_MD) # content from xdrs-core AGENTS.md template; see rule 01-xdrs-knowledge-layer
93
93
 
94
- # Add these mounts alongside the base mounts from agentme-edr-041 rule 02-local-sandbox:
95
- # (mount_paths uses {src: dst} dict format per agentme-edr-041)
94
+ # Add these mounts alongside the base mounts from agentme-edr-142 rule 02-local-sandbox:
95
+ # (mount_paths uses {src: dst} dict format per agentme-edr-142)
96
96
  sandbox = Sandbox(
97
97
  mount_paths={
98
98
  tmp_dir: "/workspace",
@@ -1,11 +1,11 @@
1
1
  ---
2
- name: agentme-edr-policy-051-ai-eval-core-standards
3
- description: Defines the eval folder structure and Makefile interface for AI projects, and the LLM-as-judge binary scoring contract applicable to all AI tiers and test types. Use when scaffolding evals or implementing an LLM judge. For eval script requirements see agentme-edr-053. For report format see agentme-edr-054. For repeatability see agentme-edr-055. For when evals are required see agentme-edr-007 rule 09-ai-project-testing-requirements. For the test type taxonomy see agentme-edr-052.
2
+ name: agentme-edr-policy-151-ai-eval-core-standards
3
+ description: Defines the eval folder structure and Makefile interface for AI projects, and the LLM-as-judge binary scoring contract applicable to all AI tiers and test types. Use when scaffolding evals or implementing an LLM judge. For eval script requirements see agentme-edr-153. For report format see agentme-edr-154. For repeatability see agentme-edr-155. For when evals are required see agentme-edr-501 rule 09-ai-project-testing-requirements. For the test type taxonomy see agentme-edr-152.
4
4
  apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
5
5
  valid-from: 2026-06-05
6
6
  ---
7
7
 
8
- # agentme-edr-policy-051: AI eval core standards
8
+ # agentme-edr-policy-151: AI eval core standards
9
9
 
10
10
  ## Context and Problem Statement
11
11
 
@@ -17,7 +17,7 @@ How should eval tests be structured, and how must LLM judges produce scores acro
17
17
 
18
18
  **Use a per-component folder structure under `evals/` with a standardized Makefile interface, and require all LLM judges to produce binary (`0`/`1`) output compatible with classification metrics.**
19
19
 
20
- For when evals are required per AI tier, see [agentme-edr-007](../governance/007-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
20
+ For when evals are required per AI tier, see [agentme-edr-501](../governance/501-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
21
21
 
22
22
  ### Details
23
23
 
@@ -31,7 +31,7 @@ evals/
31
31
  eval-<name>/
32
32
  golden_dataset/ # EDR-024 + EDR-030 compliant golden dataset (README.md, dataset.schema.json, data/)
33
33
  eval.py # evaluation script
34
- report-<type>.md # generated report, one per evaluated test type (overwritten on each run — see agentme-edr-054 rule 01)
34
+ report-<type>.md # generated report, one per evaluated test type (overwritten on each run — see agentme-edr-154 rule 01)
35
35
  Makefile # lint, eval, run, and eval-<type> targets
36
36
  eval-<name2>/
37
37
  ...
@@ -41,18 +41,18 @@ evals/
41
41
 
42
42
  `<component>` MUST match the name of the component under evaluation and use lowercase hyphen-separated words (e.g., `workflow-document-review`, `agent-support`, `model-classifier`).
43
43
 
44
- `<name>` identifies the specific evaluation scenario using lowercase hyphen-separated words (e.g., `eval-basic`, `eval-complex`, `eval-edge-cases`). A scenario's `golden_dataset` MAY mix multiple test types across its entries: label each entry with its applicable `test_types` ([agentme-edr-052](052-ai-test-types-taxonomy.md) rule `04`) and use the `eval-<type>` targets below to run one type at a time.
44
+ `<name>` identifies the specific evaluation scenario using lowercase hyphen-separated words (e.g., `eval-basic`, `eval-complex`, `eval-edge-cases`). A scenario's `golden_dataset` MAY mix multiple test types across its entries: label each entry with its applicable `test_types` ([agentme-edr-152](152-ai-test-types-taxonomy.md) rule `04`) and use the `eval-<type>` targets below to run one type at a time.
45
45
 
46
- The `golden_dataset/` subfolder MUST be a valid [agentme-edr-050](../data/050-ml-dataset-structure.md) dataset (`README.md`, `dataset.schema.json`, one JSON file per entry under `data/` per rule `04-complex-structured-datasets-must-use-per-entry-json-files`, lint-validated per rule `06`) whose entries follow the golden dataset envelope defined in [agentme-edr-052](052-ai-test-types-taxonomy.md) rule `02`.
46
+ The `golden_dataset/` subfolder MUST be a valid [agentme-edr-201](../data/201-ml-dataset-structure.md) dataset (`README.md`, `dataset.schema.json`, one JSON file per entry under `data/` per rule `04-complex-structured-datasets-must-use-per-entry-json-files`, lint-validated per rule `06`) whose entries follow the golden dataset envelope defined in [agentme-edr-152](152-ai-test-types-taxonomy.md) rule `02`.
47
47
 
48
48
  Each `evals/<component>/eval-<name>/Makefile` MUST declare a `TEST_TYPES` variable listing the `test_types` values present in its golden dataset, and define:
49
49
 
50
50
  | Target | Behaviour |
51
51
  |---|---|
52
- | `lint` | Validates every `golden_dataset/data/*.json` file against `golden_dataset/dataset.schema.json` per [agentme-edr-050](../data/050-ml-dataset-structure.md) rule `06` |
52
+ | `lint` | Validates every `golden_dataset/data/*.json` file against `golden_dataset/dataset.schema.json` per [agentme-edr-201](../data/201-ml-dataset-structure.md) rule `06` |
53
53
  | `eval` | Depends on `lint`; runs `eval.py --type=all` with threshold enforcement; exits non-zero on failure (CI-safe) |
54
54
  | `run` | Depends on `lint`; runs `eval.py --type=all` with threshold enforcement; same as `eval` but intended for local exploration |
55
- | `eval-<type>` | Depends on `lint`; runs `eval.py --type=<type>` for one declared test type, following [agentme-edr-008](../platform/008-common-targets.md) rule `03`'s `eval-<qualifier>` convention |
55
+ | `eval-<type>` | Depends on `lint`; runs `eval.py --type=<type>` for one declared test type, following [agentme-edr-303](../platform/303-common-targets.md) rule `03`'s `eval-<qualifier>` convention |
56
56
 
57
57
  ```makefile
58
58
  TEST_TYPES := smoke functional safety
@@ -94,7 +94,7 @@ LLM judges scoring component outputs MUST produce binary output: `0` (fail) or `
94
94
  - **F1 computation:** F1 MUST be computed from a binary confusion matrix treating score `1` as the positive class. In a standard golden dataset where every entry is expected to pass, all entries are true positives or false negatives (no true negatives exist), so Recall = TP / (TP + FN) and Precision = TP / (TP + FP) where FP = entries the judge scores `1` that the entry-level expected outcome marks as expected-fail. When all entries are expected-pass, Recall = Accuracy and Precision = 1, making F1 a conservative lower-bound on accuracy. Projects with adversarial or expected-fail entries MUST annotate each golden dataset entry with its expected binary outcome (`1` = should pass, `0` = should fail) in `expected_output` to enable a correct confusion matrix.
95
95
  - Multi-class classification not supported. For multiple quality levels, use multiple binary judges (e.g., one for "factually correct", another for "tone appropriate")
96
96
 
97
- **Rationale:** Binary output makes LLM judges compatible with classification metrics infrastructure (Accuracy, F1, Wilson CI, convergence analysis) defined in [agentme-edr-054](054-ai-eval-report-format.md) rule `01`.
97
+ **Rationale:** Binary output makes LLM judges compatible with classification metrics infrastructure (Accuracy, F1, Wilson CI, convergence analysis) defined in [agentme-edr-154](154-ai-eval-report-format.md) rule `01`.
98
98
 
99
99
  **Example LLM judge prompt:**
100
100
 
@@ -112,10 +112,10 @@ Output:
112
112
 
113
113
  ## References
114
114
 
115
- - [agentme-edr-053](053-ai-eval-script.md) — AI eval script: entry-first eval loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
116
- - [agentme-edr-054](054-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
117
- - [agentme-edr-055](055-ai-eval-repeatability.md) — AI eval repeatability: loop exception, scoring methods (including LLM-as-judge per rule `02`), and cadence
118
- - [agentme-edr-007](../governance/007-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09`) and statistical model eval targets (rule `07`)
119
- - [agentme-edr-052](052-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum and golden dataset entry envelope
120
- - [agentme-edr-050](../data/050-ml-dataset-structure.md) — ML dataset structure, per-entry JSON format, and schema-lint validation for golden datasets
121
- - [agentme-edr-008](../platform/008-common-targets.md) — `eval-<qualifier>` Makefile convention (rule `03`)
115
+ - [agentme-edr-153](153-ai-eval-script.md) — AI eval script: entry-first eval loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
116
+ - [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
117
+ - [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: loop exception, scoring methods (including LLM-as-judge per rule `02`), and cadence
118
+ - [agentme-edr-501](../governance/501-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09`) and statistical model eval targets (rule `07`)
119
+ - [agentme-edr-152](152-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum and golden dataset entry envelope
120
+ - [agentme-edr-201](../data/201-ml-dataset-structure.md) — ML dataset structure, per-entry JSON format, and schema-lint validation for golden datasets
121
+ - [agentme-edr-303](../platform/303-common-targets.md) — `eval-<qualifier>` Makefile convention (rule `03`)
@@ -1,11 +1,11 @@
1
1
  ---
2
- name: agentme-edr-policy-052-ai-test-types-taxonomy
3
- description: Names AI-application test types grouped as "Safety & adversarial", "Responsible AI", "Quality eval", "Prompt/LLM", and "Code-level" with their objective, mocking constraint, and relevance, and defines the shared "golden dataset" entry envelope that agentme-edr-051's eval tooling filters by test_types. Use when deciding which AI test types to implement or when authoring a golden dataset entry.
2
+ name: agentme-edr-policy-152-ai-test-types-taxonomy
3
+ description: Names AI-application test types grouped as "Safety & adversarial", "Responsible AI", "Quality eval", "Prompt/LLM", and "Code-level" with their objective, mocking constraint, and relevance, and defines the shared "golden dataset" entry envelope that agentme-edr-151's eval tooling filters by test_types. Use when deciding which AI test types to implement or when authoring a golden dataset entry.
4
4
  apply-to: AI projects (LLM, Agent, or Workflow tier) implementing AI-specific test types beyond generic code-level unit/integration tests
5
5
  valid-from: 2026-07-05
6
6
  ---
7
7
 
8
- # agentme-edr-policy-052: AI test types taxonomy
8
+ # agentme-edr-policy-152: AI test types taxonomy
9
9
 
10
10
  ## Context and Problem Statement
11
11
 
@@ -13,7 +13,7 @@ AI components need test types beyond generic unit/integration tests (safety, fai
13
13
 
14
14
  ## Decision Outcome
15
15
 
16
- **Adopt a named taxonomy of AI test types plus a shared "golden dataset" entry envelope that agentme-edr-051's eval tooling filters by `test_types`.**
16
+ **Adopt a named taxonomy of AI test types plus a shared "golden dataset" entry envelope that agentme-edr-151's eval tooling filters by `test_types`.**
17
17
 
18
18
  Each test type is named with its group, objective, mocking constraint, applicability, and relevance; every golden dataset entry is labeled with the test types it applies to.
19
19
 
@@ -21,7 +21,7 @@ Each test type is named with its group, objective, mocking constraint, applicabi
21
21
 
22
22
  #### 01-golden-dataset-concept
23
23
 
24
- Projects MUST use a golden dataset to test AI components. A **golden dataset** comprises all eval case entries used to test an AI component (LLM, Agent, or Workflow tier); each entry is labeled with the `test_types` (rule `04`) it applies to. It is the dataset consumed by [agentme-edr-053](053-ai-eval-script.md) evals and stored as one JSON file per entry per [agentme-edr-050](../data/050-ml-dataset-structure.md) rule `04`, at `evals/<component>/eval-<name>/golden_dataset/`.
24
+ Projects MUST use a golden dataset to test AI components. A **golden dataset** comprises all eval case entries used to test an AI component (LLM, Agent, or Workflow tier); each entry is labeled with the `test_types` (rule `04`) it applies to. It is the dataset consumed by [agentme-edr-153](153-ai-eval-script.md) evals and stored as one JSON file per entry per [agentme-edr-201](../data/201-ml-dataset-structure.md) rule `04`, at `evals/<component>/eval-<name>/golden_dataset/`.
25
25
 
26
26
  #### 02-golden-dataset-entry-envelope
27
27
 
@@ -42,8 +42,8 @@ Every golden dataset entry (a JSON file in `golden_dataset/data/`) MUST have thi
42
42
  - `test_types` — array, values MUST come from rule `04`'s enum, MUST contain at least one value. An entry MAY carry more than one value additively (e.g. `["functional", "smoke", "human"]`) — no test type excludes another.
43
43
  - `input` — for Prompt-tier components, a raw prompt string or the prompt template's input parameters object; for Agent/Workflow-tier components, the input attributes object passed to the component.
44
44
  - `expected_output` — the fields used to score the entry under each of its automated `test_types`: output attributes for an LLM-as-judge rubric, a target for vector-similarity scoring, or exact attribute values for strict comparison. When `human` is one of the entry's `test_types`, `expected_output` MUST additionally include a `human_test` string field with manual-verification instructions (e.g. `"check for ethical issues, verify record change in system X"`) — this supplements, and MUST NOT replace, the entry's automated scoring fields.
45
- - `mock_fixtures` — optional object; keys identify the adapter or external system to mock (SHOULD match the connector folder name under `adapters/connectors/<name>` for readability, though not enforced), values are any valid JSON interpreted by the mock implementation. When present, eval.py MUST configure each named mock adapter with its fixture data BEFORE invoking the component for that entry; each entry MUST use fresh mock instances to prevent state from bleeding across entries. `mock_fixtures` applies to all `test_types` including `human` — the component is still invoked for human entries to capture `actual_output`. `mock_fixtures` MUST NOT include keys for LLM adapters: all golden dataset test types are rated `mocks disallowed for LLM calls` (rule `03`), so the LLM call MUST be real; LLM provider mocking belongs exclusively to unit tests via [agentme-edr-040](040-ai-llm-development-standards.md) rule `04`. See [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
46
- - The dataset's `dataset.schema.json` MUST require `test_types`, `input`, and `expected_output`, and SHOULD declare `mock_fixtures` as optional (`"type": "object", "additionalProperties": {}`), per [agentme-edr-050](../data/050-ml-dataset-structure.md) rule `04`.
45
+ - `mock_fixtures` — optional object; keys identify the adapter or external system to mock (SHOULD match the connector folder name under `adapters/connectors/<name>` for readability, though not enforced), values are any valid JSON interpreted by the mock implementation. When present, eval.py MUST configure each named mock adapter with its fixture data BEFORE invoking the component for that entry; each entry MUST use fresh mock instances to prevent state from bleeding across entries. `mock_fixtures` applies to all `test_types` including `human` — the component is still invoked for human entries to capture `actual_output`. `mock_fixtures` MUST NOT include keys for LLM adapters: all golden dataset test types are rated `mocks disallowed for LLM calls` (rule `03`), so the LLM call MUST be real; LLM provider mocking belongs exclusively to unit tests via [agentme-edr-141](141-ai-llm-development-standards.md) rule `04`. See [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
46
+ - The dataset's `dataset.schema.json` MUST require `test_types`, `input`, and `expected_output`, and SHOULD declare `mock_fixtures` as optional (`"type": "object", "additionalProperties": {}`), per [agentme-edr-201](../data/201-ml-dataset-structure.md) rule `04`.
47
47
 
48
48
  #### 03-mocks-allowed-values
49
49
 
@@ -51,13 +51,13 @@ The taxonomy in rule `05` rates each test type using one of three values under t
51
51
 
52
52
  | Value | Meaning |
53
53
  |---|---|
54
- | `mocks allowed` | Fully offline; fakes may replace every dependency including the LLM (e.g. `FakeListChatModel` per [agentme-edr-040](040-ai-llm-development-standards.md) rule `04`). Used only for code-level unit tests. |
54
+ | `mocks allowed` | Fully offline; fakes may replace every dependency including the LLM (e.g. `FakeListChatModel` per [agentme-edr-141](141-ai-llm-development-standards.md) rule `04`). Used only for code-level unit tests. |
55
55
  | `mocks disallowed` | No mocking of any dependency — all real external systems required. Used for integration tests. |
56
56
  | `mocks disallowed for LLM calls` | **The LLM call MUST be real; all other external dependencies (databases, APIs, external services) MAY and SHOULD be mocked via `mock_fixtures`.** `mock_fixtures` keys MUST NOT reference LLM adapters. This is the standard constraint for every golden-dataset eval test type. See rule `08` for rationale. |
57
57
 
58
58
  #### 04-test-types-enum
59
59
 
60
- A golden dataset entry's `test_types` array MUST only use these values: `safety`, `adversarial`, `fairness`, `bias`, `robustness`, `explainability`, `groundedness`, `functional`, `prompt`, `smoke`, `human`, `repeatability`. These correspond to the dataset-driven rows of rule `05`. **Unit test** and **Integration test** (the two Code-level rows) are NOT part of this enum — they have no golden dataset entries and remain governed entirely by [agentme-edr-004](004-unit-test-requirements.md) and [agentme-edr-007](../governance/007-project-quality-standards.md) rule `08`.
60
+ A golden dataset entry's `test_types` array MUST only use these values: `safety`, `adversarial`, `fairness`, `bias`, `robustness`, `explainability`, `groundedness`, `functional`, `prompt`, `smoke`, `human`, `repeatability`. These correspond to the dataset-driven rows of rule `05`. **Unit test** and **Integration test** (the two Code-level rows) are NOT part of this enum — they have no golden dataset entries and remain governed entirely by [agentme-edr-122](122-unit-test-requirements.md) and [agentme-edr-501](../governance/501-project-quality-standards.md) rule `08`.
61
61
 
62
62
  #### 05-test-type-taxonomy
63
63
 
@@ -74,43 +74,43 @@ Test types MUST be selected from this taxonomy. Each test type is named with its
74
74
  | Groundedness (RAG) eval | `groundedness` | Quality eval | Verify the answer is supported by retrieved context | mocks disallowed for LLM calls | System uses retrieval-augmented generation | Avoids confidently-wrong answers reaching customers | Pinpoints retrieval/prompt bugs | 4 |
75
75
  | Repeatability test | `repeatability` | Quality eval | Verify output stability/variance across N repeated invocations of the same input under fixed configuration | mocks disallowed for LLM calls | Non-deterministic components (temperature > 0, agentic tool-selection loops, sampling-based decoding) used in decision-critical or user-facing flows | Protects against silently flaky behavior reaching production; supports consistency SLAs | Detects prompt/agent designs too sensitive to sampling noise; informs temperature/seed tuning | 3 |
76
76
  | Human evaluation | `human` | Quality eval | Manually verify aspects automated scoring can't (ethics, side effects, external state) | mocks disallowed for LLM calls | Before major releases; periodic spot-check | Defensible, human-reviewed sign-off | Catches what automated metrics miss | 3 |
77
- | Functional eval (golden-dataset accuracy / LLM-as-judge) | `functional` | Quality eval | Measure output correctness against the golden dataset | mocks disallowed for LLM calls | Required before every Workflow release ([agentme-edr-007](../governance/007-project-quality-standards.md) rule `09`); advised elsewhere | Auditable evidence of business correctness before release | Detects regressions from model/provider/prompt changes | 5 |
77
+ | Functional eval (golden-dataset accuracy / LLM-as-judge) | `functional` | Quality eval | Measure output correctness against the golden dataset | mocks disallowed for LLM calls | Required before every Workflow release ([agentme-edr-501](../governance/501-project-quality-standards.md) rule `09`); advised elsewhere | Auditable evidence of business correctness before release | Detects regressions from model/provider/prompt changes | 5 |
78
78
  | Smoke test | `smoke` | Quality eval | Fast pass/fail check on a small, critical subset before running fuller suites | mocks disallowed for LLM calls | Every commit/PR, before functional/responsible-AI evals run | Cheap early warning before slower evals run | Fast, cheap feedback loop | 4 |
79
79
  | Prompt regression test | `prompt` | Prompt/LLM | Detect behavior change when a prompt or model version changes | mocks disallowed for LLM calls | Whenever a prompt template or model version changes | Prevents shipping a worse experience via a "small" tweak | Fast check on every prompt edit | 3 |
80
80
  | Integration test | n/a — code-level only (see rule `04`) | Code-level | Verify real interaction with external systems | mocks disallowed | Component depends on external systems | Reduces production outages from integration mismatches | Catches wiring bugs unit tests can't see | 2 |
81
- | Unit test (offline, mocked) | n/a — code-level only (see rule `04`) | Code-level | Verify deterministic logic in isolation, offline | mocks allowed | Required for Workflow tier every commit ([agentme-edr-007](../governance/007-project-quality-standards.md) rule `09`) | Lowest-cost point to catch defects | Fastest, fully offline feedback on every commit | 5 |
81
+ | Unit test (offline, mocked) | n/a — code-level only (see rule `04`) | Code-level | Verify deterministic logic in isolation, offline | mocks allowed | Required for Workflow tier every commit ([agentme-edr-501](../governance/501-project-quality-standards.md) rule `09`) | Lowest-cost point to catch defects | Fastest, fully offline feedback on every commit | 5 |
82
82
 
83
83
  #### 06-priority-and-relevance-are-descriptive-only
84
84
 
85
- Priority, Relevance, and When to Apply in rule `05` are guidance for prioritization conversations — they MUST NOT be treated as mandating which test types a project must implement, nor their thresholds. [agentme-edr-007](../governance/007-project-quality-standards.md) rule `09` remains the only tier-level testing requirement in force (Workflow unit tests + functional evals). Once a project chooses to implement and threshold a test type, [agentme-edr-053](053-ai-eval-script.md) rule `01`'s failing-threshold behavior applies uniformly, regardless of this table's priority rating — a project may enforce fairness at 70% and functional at 90%, or skip fairness entirely; that choice is a project/business decision, not one this Policy makes.
85
+ Priority, Relevance, and When to Apply in rule `05` are guidance for prioritization conversations — they MUST NOT be treated as mandating which test types a project must implement, nor their thresholds. [agentme-edr-501](../governance/501-project-quality-standards.md) rule `09` remains the only tier-level testing requirement in force (Workflow unit tests + functional evals). Once a project chooses to implement and threshold a test type, [agentme-edr-153](153-ai-eval-script.md) rule `01`'s failing-threshold behavior applies uniformly, regardless of this table's priority rating — a project may enforce fairness at 70% and functional at 90%, or skip fairness entirely; that choice is a project/business decision, not one this Policy makes.
86
86
 
87
87
  #### 07-smoke-is-distinct-from-test-smoke
88
88
 
89
- The `smoke` test type (surfaced as the `eval-smoke` Makefile target, a fast subset of the golden-dataset functional eval) is a different concept from [agentme-edr-008](../platform/008-common-targets.md)'s existing `test-smoke` target (a fast subset of code-level tests). Both MAY exist in the same project; teams MUST NOT conflate them.
89
+ The `smoke` test type (surfaced as the `eval-smoke` Makefile target, a fast subset of the golden-dataset functional eval) is a different concept from [agentme-edr-303](../platform/303-common-targets.md)'s existing `test-smoke` target (a fast subset of code-level tests). Both MAY exist in the same project; teams MUST NOT conflate them.
90
90
 
91
91
  #### 08-eval-mocking-constraint
92
92
 
93
- For every golden-dataset eval test type: **the LLM call MUST be real; all other external dependencies MUST be mocked via `mock_fixtures`.** This applies equally to `human` entries — the component is invoked to capture `actual_output` and external dependencies must be deterministic. `mock_fixtures` keys MUST NOT reference LLM adapters. Code-level unit tests are the correct place for fully offline, LLM-mocked testing (see [agentme-edr-040](040-ai-llm-development-standards.md) rule `04`).
93
+ For every golden-dataset eval test type: **the LLM call MUST be real; all other external dependencies MUST be mocked via `mock_fixtures`.** This applies equally to `human` entries — the component is invoked to capture `actual_output` and external dependencies must be deterministic. `mock_fixtures` keys MUST NOT reference LLM adapters. Code-level unit tests are the correct place for fully offline, LLM-mocked testing (see [agentme-edr-141](141-ai-llm-development-standards.md) rule `04`).
94
94
 
95
95
  #### 09-repeatability-vs-reproducibility
96
96
 
97
97
  | Property | Definition | What varies | Measured by |
98
98
  |---|---|---|---|
99
- | **Repeatability** | Output stability across N invocations at non-zero temperature | Model sampling variance | `repeatability` test type per [agentme-edr-055](055-ai-eval-repeatability.md) |
100
- | **Reproducibility** | Deterministic output at temperature = 0 with fixed seed | Nothing — any variance is a config bug | Not a golden-dataset type; verified via config, documented in [agentme-edr-027](../platform/027-environment-variable-configuration.md) |
99
+ | **Repeatability** | Output stability across N invocations at non-zero temperature | Model sampling variance | `repeatability` test type per [agentme-edr-155](155-ai-eval-repeatability.md) |
100
+ | **Reproducibility** | Deterministic output at temperature = 0 with fixed seed | Nothing — any variance is a config bug | Not a golden-dataset type; verified via config, documented in [agentme-edr-305](../platform/305-environment-variable-configuration.md) |
101
101
 
102
102
  A component may satisfy reproducibility (temperature = 0) yet still need repeatability tests for its production configuration (temperature > 0). The `repeatability` test type MUST NOT be applied to components with intentionally diverse output (brainstorming, creative generation) — variance is correct behavior there.
103
103
 
104
104
  ## References
105
105
 
106
- - [agentme-edr-050](../data/050-ml-dataset-structure.md) — Golden dataset file layout, per-entry JSON format, `$schema` pointer, and schema-lint validation
107
- - [agentme-edr-051](051-ai-eval-core-standards.md) — AI eval core standards: eval folder structure and Makefile targets (rule `01`); LLM-as-judge binary scoring contract (rule `02`)
108
- - [agentme-edr-053](053-ai-eval-script.md) — AI eval script: `--type` filtering, entry-first loop, `mock_fixtures`, threshold enforcement, and MLflow conventions
109
- - [agentme-edr-054](054-ai-eval-report-format.md) — AI eval report format: per-type `report-<type>.md` that consumes this taxonomy's test types
110
- - [agentme-edr-055](055-ai-eval-repeatability.md) — AI eval repeatability: `REPEAT_COUNT` loop exception, scoring constants (`EVAL_MIN_ACCURACY_REPEATABILITY`, `REPEAT_SEMANTIC_SIMILARITY_SCORE`), scoring methods, `repeatability_accuracy` MLflow metric, report shape, and run cadence
111
- - [agentme-edr-026](026-pragmatic-hexagonal-architecture.md) — Rule `10`: `_mock` file naming and placement convention for mock adapters referenced by `mock_fixtures`
112
- - [agentme-edr-007](../governance/007-project-quality-standards.md) — Rule `09` tier-level testing requirements (the only mandated AI testing baseline)
113
- - [agentme-edr-008](../platform/008-common-targets.md) — Rule `03` `eval-<qualifier>` Makefile convention; rule `03`'s `test-smoke` (distinguished in rule `07`)
114
- - [agentme-edr-027](../platform/027-environment-variable-configuration.md) — Environment-configuration conventions referenced in rule `09`'s reproducibility disambiguation
115
- - [agentme-edr-040](040-ai-llm-development-standards.md) — LLM tier definition and mocking utilities referenced by the `mocks allowed` value
116
- - [agentme-edr-004](004-unit-test-requirements.md) — Unit test requirements underlying the Code-level rows
106
+ - [agentme-edr-201](../data/201-ml-dataset-structure.md) — Golden dataset file layout, per-entry JSON format, `$schema` pointer, and schema-lint validation
107
+ - [agentme-edr-151](151-ai-eval-core-standards.md) — AI eval core standards: eval folder structure and Makefile targets (rule `01`); LLM-as-judge binary scoring contract (rule `02`)
108
+ - [agentme-edr-153](153-ai-eval-script.md) — AI eval script: `--type` filtering, entry-first loop, `mock_fixtures`, threshold enforcement, and MLflow conventions
109
+ - [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: per-type `report-<type>.md` that consumes this taxonomy's test types
110
+ - [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: `REPEAT_COUNT` loop exception, scoring constants (`EVAL_MIN_ACCURACY_REPEATABILITY`, `REPEAT_SEMANTIC_SIMILARITY_SCORE`), scoring methods, `repeatability_accuracy` MLflow metric, report shape, and run cadence
111
+ - [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) — Rule `10`: `_mock` file naming and placement convention for mock adapters referenced by `mock_fixtures`
112
+ - [agentme-edr-501](../governance/501-project-quality-standards.md) — Rule `09` tier-level testing requirements (the only mandated AI testing baseline)
113
+ - [agentme-edr-303](../platform/303-common-targets.md) — Rule `03` `eval-<qualifier>` Makefile convention; rule `03`'s `test-smoke` (distinguished in rule `07`)
114
+ - [agentme-edr-305](../platform/305-environment-variable-configuration.md) — Environment-configuration conventions referenced in rule `09`'s reproducibility disambiguation
115
+ - [agentme-edr-141](141-ai-llm-development-standards.md) — LLM tier definition and mocking utilities referenced by the `mocks allowed` value
116
+ - [agentme-edr-122](122-unit-test-requirements.md) — Unit test requirements underlying the Code-level rows