agentme 0.28.1 → 0.29.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.filedist-package.yml +1 -1
- package/.xdrs/agentme/edrs/application/{003-javascript-project-tooling.md → 101-javascript-project-tooling.md} +8 -8
- package/.xdrs/agentme/edrs/application/{010-golang-project-tooling.md → 102-golang-project-tooling.md} +6 -6
- package/.xdrs/agentme/edrs/application/{014-python-project-tooling.md → 103-python-project-tooling.md} +11 -11
- package/.xdrs/agentme/edrs/application/{015-cli-tool-standards.md → 104-cli-tool-standards.md} +13 -13
- package/.xdrs/agentme/edrs/application/{002-coding-best-practices.md → 121-coding-best-practices.md} +3 -3
- package/.xdrs/agentme/edrs/application/{004-unit-test-requirements.md → 122-unit-test-requirements.md} +2 -2
- package/.xdrs/agentme/edrs/application/{009-error-handling.md → 123-error-handling.md} +2 -2
- package/.xdrs/agentme/edrs/application/{022-secrets-management.md → 124-secrets-management.md} +5 -5
- package/.xdrs/agentme/edrs/application/{023-coding-abstraction-practices.md → 125-coding-abstraction-practices.md} +3 -3
- package/.xdrs/agentme/edrs/application/{026-pragmatic-hexagonal-architecture.md → 126-pragmatic-hexagonal-architecture.md} +9 -9
- package/.xdrs/agentme/edrs/application/{040-ai-llm-development-standards.md → 141-ai-llm-development-standards.md} +14 -14
- package/.xdrs/agentme/edrs/application/{041-ai-agents-development-standards.md → 142-ai-agents-development-standards.md} +11 -11
- package/.xdrs/agentme/edrs/application/{042-ai-agents-quality-standards.md → 143-ai-agents-quality-standards.md} +16 -16
- package/.xdrs/agentme/edrs/application/{043-ai-workflow-development-standards.md → 144-ai-workflow-development-standards.md} +28 -28
- package/.xdrs/agentme/edrs/application/{044-ai-workflow-naming-conventions.md → 145-ai-workflow-naming-conventions.md} +9 -9
- package/.xdrs/agentme/edrs/application/{045-ai-agent-xdrs-knowledge-layer.md → 146-ai-agent-xdrs-knowledge-layer.md} +7 -7
- package/.xdrs/agentme/edrs/application/{051-ai-eval-core-standards.md → 151-ai-eval-core-standards.md} +17 -17
- package/.xdrs/agentme/edrs/application/{052-ai-test-types-taxonomy.md → 152-ai-test-types-taxonomy.md} +27 -27
- package/.xdrs/agentme/edrs/application/{053-ai-eval-script.md → 153-ai-eval-script.md} +18 -18
- package/.xdrs/agentme/edrs/application/{054-ai-eval-report-format.md → 154-ai-eval-report-format.md} +6 -6
- package/.xdrs/agentme/edrs/application/{055-ai-eval-repeatability.md → 155-ai-eval-repeatability.md} +14 -14
- package/.xdrs/agentme/edrs/application/skills/001-create-javascript-project/SKILL.md +4 -4
- package/.xdrs/agentme/edrs/application/skills/003-create-golang-project/SKILL.md +3 -3
- package/.xdrs/agentme/edrs/application/skills/004-select-relevant-xdrs/SKILL.md +9 -9
- package/.xdrs/agentme/edrs/application/skills/005-create-python-project/SKILL.md +5 -5
- package/.xdrs/agentme/edrs/data/{050-ml-dataset-structure.md → 201-ml-dataset-structure.md} +4 -4
- package/.xdrs/agentme/edrs/governance/{007-project-quality-standards.md → 501-project-quality-standards.md} +16 -16
- package/.xdrs/agentme/edrs/governance/{013-contributing-guide-requirements.md → 502-contributing-guide-requirements.md} +3 -3
- package/.xdrs/agentme/edrs/index.md +30 -30
- package/.xdrs/agentme/edrs/operations/{011-service-health-check-endpoint.md → 401-service-health-check-endpoint.md} +2 -2
- package/.xdrs/agentme/edrs/platform/{005-monorepo-structure.md → 301-monorepo-structure.md} +4 -4
- package/.xdrs/agentme/edrs/platform/{006-github-pipelines.md → 302-github-pipelines.md} +3 -3
- package/.xdrs/agentme/edrs/platform/{008-common-targets.md → 303-common-targets.md} +8 -8
- package/.xdrs/agentme/edrs/platform/{017-tool-execution-and-scripting.md → 304-tool-execution-and-scripting.md} +10 -10
- package/.xdrs/agentme/edrs/platform/{027-environment-variable-configuration.md → 305-environment-variable-configuration.md} +14 -14
- package/.xdrs/agentme/edrs/platform/skills/002-monorepo-setup/SKILL.md +3 -3
- package/.xdrs/agentme/edrs/principles/012-continuous-xdr-enrichment.md +1 -1
- package/.xdrs/agentme/edrs/principles/016-cross-language-module-structure.md +4 -4
- package/package.json +2 -2
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
3
|
-
description: Defines implementation quality standards for AI agents: tool definition patterns, error handling and recovery, observability, and unit testing. Apply alongside agentme-edr-
|
|
2
|
+
name: agentme-edr-policy-143-ai-agents-quality-standards
|
|
3
|
+
description: Defines implementation quality standards for AI agents: tool definition patterns, error handling and recovery, observability, and unit testing. Apply alongside agentme-edr-142 when implementing or reviewing agent code. For agent architecture and structural decisions (framework, sandbox, naming, composition, system prompts) see agentme-edr-142.
|
|
4
4
|
apply-to: AI agent implementation code — apply when writing tools, error handlers, logging, and unit tests for agents
|
|
5
5
|
valid-from: 2026-06-09
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-143: AI agents quality standards
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -15,7 +15,7 @@ How should agent tools be defined, what error handling must agents implement, ho
|
|
|
15
15
|
|
|
16
16
|
## Decision Outcome
|
|
17
17
|
|
|
18
|
-
**Agent implementations MUST follow the tool definition, error handling, observability, and unit testing standards defined here, alongside the structural decisions in [agentme-edr-
|
|
18
|
+
**Agent implementations MUST follow the tool definition, error handling, observability, and unit testing standards defined here, alongside the structural decisions in [agentme-edr-142](142-ai-agents-development-standards.md).**
|
|
19
19
|
|
|
20
20
|
### Details
|
|
21
21
|
|
|
@@ -49,7 +49,7 @@ def tool_name(arg1: str, arg2: int) -> Dict[str, Any]:
|
|
|
49
49
|
- Tool docstrings MUST clearly describe the tool's purpose, arguments, and return value (the LLM reads these)
|
|
50
50
|
- Tools MUST return structured data (dictionaries or dataclasses), not bare strings or untyped values
|
|
51
51
|
- Tools MUST handle errors gracefully and return error information in the result structure, not raise exceptions
|
|
52
|
-
- Tools that interact with external systems MUST be placed in `adapters/connectors/` per [agentme-edr-
|
|
52
|
+
- Tools that interact with external systems MUST be placed in `adapters/connectors/` per [agentme-edr-126](126-pragmatic-hexagonal-architecture.md)
|
|
53
53
|
|
|
54
54
|
**Error handling in tools:**
|
|
55
55
|
|
|
@@ -100,9 +100,9 @@ Agent execution MUST be observable through logging and tracing:
|
|
|
100
100
|
|
|
101
101
|
- Log each iteration of the perceive → plan → act → observe cycle with iteration number and tool selection.
|
|
102
102
|
- Use structured logging (JSON) with fields: `iteration`, `tool_selected`, `tool_result_status`, `decision`.
|
|
103
|
-
- For LLM calls within agents, follow [agentme-edr-
|
|
103
|
+
- For LLM calls within agents, follow [agentme-edr-141](141-ai-llm-development-standards.md) rule `03-llm-observability`.
|
|
104
104
|
- When agents run as workflow nodes, MLflow tracking from the parent workflow automatically captures agent-level traces.
|
|
105
|
-
- The project Makefile MUST expose a `dev-mlflow` target to start a local MLflow tracking server for development inspection, per [agentme-edr-
|
|
105
|
+
- The project Makefile MUST expose a `dev-mlflow` target to start a local MLflow tracking server for development inspection, per [agentme-edr-303](../platform/303-common-targets.md) rule `09-ai-project-dev-targets`.
|
|
106
106
|
|
|
107
107
|
**Example structured log entry:**
|
|
108
108
|
|
|
@@ -120,7 +120,7 @@ Agent execution MUST be observable through logging and tracing:
|
|
|
120
120
|
|
|
121
121
|
#### 04-agent-unit-testing
|
|
122
122
|
|
|
123
|
-
Agent LLM calls are external API calls and MUST be mocked in unit tests per [agentme-edr-
|
|
123
|
+
Agent LLM calls are external API calls and MUST be mocked in unit tests per [agentme-edr-141](141-ai-llm-development-standards.md) rule `04-unit-test-mocking`.
|
|
124
124
|
|
|
125
125
|
Because agents drive a tool-invocation loop — where the LLM decides which tools to call — the fake model must return **tool-call messages** followed by a final answer. Use **`GenericFakeChatModel`** for this:
|
|
126
126
|
|
|
@@ -150,7 +150,7 @@ def test_file_analyzer_agent_calls_search_then_stops():
|
|
|
150
150
|
assert "3 Python files" in result.summary
|
|
151
151
|
```
|
|
152
152
|
|
|
153
|
-
Agents MUST be designed so that the LLM instance is injectable (constructor parameter) to allow test doubles. See [agentme-edr-
|
|
153
|
+
Agents MUST be designed so that the LLM instance is injectable (constructor parameter) to allow test doubles. See [agentme-edr-141](141-ai-llm-development-standards.md) rule `04-unit-test-mocking` for the injectable LLM pattern.
|
|
154
154
|
|
|
155
155
|
**`mock_deep_agent`**
|
|
156
156
|
|
|
@@ -175,10 +175,10 @@ def test_workflow_calls_subagent(mocker):
|
|
|
175
175
|
|
|
176
176
|
## References
|
|
177
177
|
|
|
178
|
-
- [agentme-edr-
|
|
179
|
-
- [agentme-edr-
|
|
180
|
-
- [agentme-edr-
|
|
181
|
-
- [agentme-edr-
|
|
182
|
-
- [agentme-edr-
|
|
183
|
-
- [agentme-edr-
|
|
184
|
-
- [agentme-edr-
|
|
178
|
+
- [agentme-edr-142](142-ai-agents-development-standards.md) — Agent development standards (framework, sandbox, naming, composition, system prompts)
|
|
179
|
+
- [agentme-edr-141](141-ai-llm-development-standards.md) — LLM development standards (LangChain configuration, mocking patterns)
|
|
180
|
+
- [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) — Hexagonal architecture (tool placement in adapters/connectors)
|
|
181
|
+
- [agentme-edr-501](../governance/501-project-quality-standards.md) — Project quality standards including AI-tier testing requirements (rule `09-ai-project-testing-requirements`)
|
|
182
|
+
- [agentme-edr-151](151-ai-eval-core-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract (rule `02`)
|
|
183
|
+
- [agentme-edr-153](153-ai-eval-script.md) — AI eval script: entry-first loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
|
|
184
|
+
- [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
3
|
-
description: Defines the standard toolchain, framework, observability, and workflow patterns for building LangGraph workflows in Python. Use when scaffolding, reviewing, or extending AI workflow projects that orchestrate LLM calls, agents, and algorithmic nodes. For simple LLM calls see agentme-edr-
|
|
2
|
+
name: agentme-edr-policy-144-ai-workflow-development-standards
|
|
3
|
+
description: Defines the standard toolchain, framework, observability, and workflow patterns for building LangGraph workflows in Python. Use when scaffolding, reviewing, or extending AI workflow projects that orchestrate LLM calls, agents, and algorithmic nodes. For simple LLM calls see agentme-edr-141, for agentic patterns see agentme-edr-142. For naming conventions (nodes, states, routes, judge output schema) see agentme-edr-145.
|
|
4
4
|
apply-to: AI workflow projects using LangGraph StateGraph built with Python
|
|
5
5
|
valid-from: 2026-06-05
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-144: AI workflow development standards
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -23,21 +23,21 @@ Which tools, frameworks, and design patterns should AI workflow projects follow
|
|
|
23
23
|
|
|
24
24
|
Workflows MUST be built with **LangGraph**. Use LangGraph `StateGraph` to model each distinct workflow as an explicit directed graph with typed state.
|
|
25
25
|
|
|
26
|
-
For all direct LLM calls within workflow nodes, use LangChain per [agentme-edr-
|
|
26
|
+
For all direct LLM calls within workflow nodes, use LangChain per [agentme-edr-141](141-ai-llm-development-standards.md). For agent nodes with tool-invocation loops, use deepagents per [agentme-edr-142](142-ai-agents-development-standards.md).
|
|
27
27
|
|
|
28
28
|
#### 03-observability-and-experiment-tracking
|
|
29
29
|
|
|
30
30
|
Use **MLflow** for all workflow observability and evaluation:
|
|
31
31
|
|
|
32
32
|
- **Workflow-level tracking:** Wrap each workflow run with `mlflow.start_run()` to capture traces, parameters, and metrics locally.
|
|
33
|
-
- **LLM-level auto-tracing:** Enable LangChain auto-tracing per [agentme-edr-
|
|
33
|
+
- **LLM-level auto-tracing:** Enable LangChain auto-tracing per [agentme-edr-141](141-ai-llm-development-standards.md) rule `03-llm-observability` by calling `mlflow.langchain.autolog()` during application startup. This captures inputs, outputs, token counts, and latency for every LangChain call within workflow nodes.
|
|
34
34
|
- Log run parameters (model name, temperature, prompt version) and output metrics (accuracy, latency, token counts) using `mlflow.log_param` / `mlflow.log_metric`.
|
|
35
35
|
- Run a local MLflow tracking server with `mlflow ui` to inspect runs during development. Do not require a remote MLflow server for local development.
|
|
36
|
-
- The project Makefile MUST expose a `dev-mlflow` target to start the local MLflow tracking server, per [agentme-edr-
|
|
36
|
+
- The project Makefile MUST expose a `dev-mlflow` target to start the local MLflow tracking server, per [agentme-edr-303](../platform/303-common-targets.md) rule `09-ai-project-dev-targets`.
|
|
37
37
|
|
|
38
38
|
#### 04-dataset-driven-accuracy-measurement
|
|
39
39
|
|
|
40
|
-
Projects MUST follow the eval dataset and implementation requirements defined in [agentme-edr-
|
|
40
|
+
Projects MUST follow the eval dataset and implementation requirements defined in [agentme-edr-151](151-ai-eval-core-standards.md) and [agentme-edr-153](153-ai-eval-script.md). Testing requirements (when evals are required, release gates) are defined in [agentme-edr-501](../governance/501-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
|
|
41
41
|
|
|
42
42
|
#### 05-flow-documentation
|
|
43
43
|
|
|
@@ -69,7 +69,7 @@ Workflows MUST include at least one explicit verification node before producing
|
|
|
69
69
|
|
|
70
70
|
#### 07-workflow-structure
|
|
71
71
|
|
|
72
|
-
Workflow logic MUST be organized as named workflows following [agentme-edr-
|
|
72
|
+
Workflow logic MUST be organized as named workflows following [agentme-edr-126](126-pragmatic-hexagonal-architecture.md). Each workflow is an independent LangGraph `StateGraph` with a defined start node and end node, connecting LLM nodes, agent nodes, algorithmic nodes, states, routes, and decision nodes.
|
|
73
73
|
|
|
74
74
|
Workflows live inside `app/workflows/` (the application layer), while external integrations such as LLM providers, vector stores, and third-party APIs live under `adapters/connectors/` (the outbound adapter layer). Inbound interfaces (HTTP API, CLI) live under `adapters/` as inbound adapters.
|
|
75
75
|
|
|
@@ -101,17 +101,17 @@ lib/src/<package_name>/
|
|
|
101
101
|
|
|
102
102
|
#### 08-workflow-evals
|
|
103
103
|
|
|
104
|
-
Projects MUST follow the eval folder structure defined in [agentme-edr-
|
|
104
|
+
Projects MUST follow the eval folder structure defined in [agentme-edr-151](151-ai-eval-core-standards.md) rule `01` and the eval script requirements defined in [agentme-edr-153](153-ai-eval-script.md). For LLM-as-judge scoring used in workflow verification nodes and evals, see [agentme-edr-151](151-ai-eval-core-standards.md) rule `02`.
|
|
105
105
|
|
|
106
106
|
#### 09-node-naming-conventions
|
|
107
107
|
|
|
108
|
-
Nodes MUST follow the naming conventions defined in [agentme-edr-
|
|
108
|
+
Nodes MUST follow the naming conventions defined in [agentme-edr-145](145-ai-workflow-naming-conventions.md) rule `01-node-naming-conventions`.
|
|
109
109
|
|
|
110
110
|
#### 10-workflow-unit-testing
|
|
111
111
|
|
|
112
|
-
All LLM calls within workflow nodes are external API calls and MUST be mocked in unit tests per [agentme-edr-
|
|
112
|
+
All LLM calls within workflow nodes are external API calls and MUST be mocked in unit tests per [agentme-edr-141](141-ai-llm-development-standards.md) rule `04-unit-test-mocking`. Workflow unit tests MUST run fully offline with no real LLM provider calls.
|
|
113
113
|
|
|
114
|
-
Choose the mock utility per [agentme-edr-
|
|
114
|
+
Choose the mock utility per [agentme-edr-141](141-ai-llm-development-standards.md) rule `04-unit-test-mocking`. For workflows containing `_agent` nodes that drive a tool-invocation loop, MUST use `GenericFakeChatModel`.
|
|
115
115
|
|
|
116
116
|
**Example — workflow with plain-text LLM nodes:**
|
|
117
117
|
|
|
@@ -152,19 +152,19 @@ def test_document_workflow_with_agent_node():
|
|
|
152
152
|
assert result.status == "approved"
|
|
153
153
|
```
|
|
154
154
|
|
|
155
|
-
Workflows MUST accept the LLM instance as a constructor parameter so that unit tests can inject a fake. See the injectable LLM pattern in [agentme-edr-
|
|
155
|
+
Workflows MUST accept the LLM instance as a constructor parameter so that unit tests can inject a fake. See the injectable LLM pattern in [agentme-edr-141](141-ai-llm-development-standards.md) rule `04-unit-test-mocking`.
|
|
156
156
|
|
|
157
157
|
#### 11-state-type-conventions
|
|
158
158
|
|
|
159
|
-
State types MUST follow the conventions defined in [agentme-edr-
|
|
159
|
+
State types MUST follow the conventions defined in [agentme-edr-145](145-ai-workflow-naming-conventions.md) rule `02-state-type-conventions`.
|
|
160
160
|
|
|
161
161
|
#### 12-workflow-naming-conventions
|
|
162
162
|
|
|
163
|
-
Workflows MUST be named following the conventions in [agentme-edr-
|
|
163
|
+
Workflows MUST be named following the conventions in [agentme-edr-145](145-ai-workflow-naming-conventions.md) rule `04-workflow-naming-conventions`.
|
|
164
164
|
|
|
165
165
|
#### 13-judge-node-output-format
|
|
166
166
|
|
|
167
|
-
Judge nodes MUST use the output format defined in [agentme-edr-
|
|
167
|
+
Judge nodes MUST use the output format defined in [agentme-edr-145](145-ai-workflow-naming-conventions.md) rule `03-judge-node-output-format`.
|
|
168
168
|
|
|
169
169
|
#### 15-workflow-state-persistence
|
|
170
170
|
|
|
@@ -197,18 +197,18 @@ result = graph.invoke(input_state, config={"thread_id": "session-123"})
|
|
|
197
197
|
|
|
198
198
|
#### 16-cross-element-naming-coherence
|
|
199
199
|
|
|
200
|
-
All workflow elements MUST maintain naming coherence as defined in [agentme-edr-
|
|
200
|
+
All workflow elements MUST maintain naming coherence as defined in [agentme-edr-145](145-ai-workflow-naming-conventions.md) rule `05-cross-element-naming-coherence`.
|
|
201
201
|
|
|
202
202
|
## References
|
|
203
203
|
|
|
204
|
-
- [agentme-edr-
|
|
205
|
-
- [agentme-edr-
|
|
206
|
-
- [agentme-edr-
|
|
207
|
-
- [agentme-edr-
|
|
208
|
-
- [agentme-edr-
|
|
209
|
-
- [agentme-edr-
|
|
210
|
-
- [agentme-edr-
|
|
211
|
-
- [agentme-edr-
|
|
212
|
-
- [agentme-edr-
|
|
213
|
-
- [agentme-edr-
|
|
214
|
-
- [agentme-edr-
|
|
204
|
+
- [agentme-edr-145](145-ai-workflow-naming-conventions.md) — AI workflow naming conventions: node suffixes/prefixes, state types, judge output schema, workflow names, and cross-element coherence
|
|
205
|
+
- [agentme-edr-141](141-ai-llm-development-standards.md) — LLM development standards: LangChain framework, provider configuration, LLM observability, and unit test mocking
|
|
206
|
+
- [agentme-edr-142](142-ai-agents-development-standards.md) — Agent development standards: deepagents framework, tool-invocation loops, and agent patterns
|
|
207
|
+
- [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) — Adapter/application layer separation that defines the project layout
|
|
208
|
+
- [agentme-edr-103](103-python-project-tooling.md) — Python project tooling and structure
|
|
209
|
+
- [agentme-edr-201](../data/201-ml-dataset-structure.md) — ML dataset structure for eval datasets
|
|
210
|
+
- [agentme-edr-151](151-ai-eval-core-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract for all tiers (rule `02`)
|
|
211
|
+
- [agentme-edr-153](153-ai-eval-script.md) — AI eval script: entry-first loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
|
|
212
|
+
- [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
|
|
213
|
+
- [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: REPEAT_COUNT loop, scoring methods, and release cadence
|
|
214
|
+
- [agentme-edr-501](../governance/501-project-quality-standards.md) — Project quality standards including AI-tier testing requirements (rule `09-ai-project-testing-requirements`)
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
3
|
-
description: Defines the naming vocabulary for LangGraph workflow elements: node suffixes/prefixes, state type names, state attribute grouping, workflow class/variable names, judge node output schema, route function names, and cross-element coherence rules. Use when naming any part of a LangGraph workflow — nodes, states, routes, or the workflow itself. For workflow structure and toolchain see agentme-edr-
|
|
2
|
+
name: agentme-edr-policy-145-ai-workflow-naming-conventions
|
|
3
|
+
description: Defines the naming vocabulary for LangGraph workflow elements: node suffixes/prefixes, state type names, state attribute grouping, workflow class/variable names, judge node output schema, route function names, and cross-element coherence rules. Use when naming any part of a LangGraph workflow — nodes, states, routes, or the workflow itself. For workflow structure and toolchain see agentme-edr-144.
|
|
4
4
|
apply-to: AI workflow projects using LangGraph StateGraph built with Python
|
|
5
5
|
valid-from: 2026-06-21
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-145: AI workflow naming conventions
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -25,10 +25,10 @@ LangGraph node names MUST follow a suffix convention that communicates the node'
|
|
|
25
25
|
|
|
26
26
|
| Convention | Node type | When to use |
|
|
27
27
|
|---|---|---|
|
|
28
|
-
| suffix `_llm` | LLM call | Any node whose primary action is a direct LLM inference call (see [agentme-edr-
|
|
28
|
+
| suffix `_llm` | LLM call | Any node whose primary action is a direct LLM inference call (see [agentme-edr-141](141-ai-llm-development-standards.md)) |
|
|
29
29
|
| suffix `_step` | Algorithmic step | Deterministic logic with no LLM involvement (transformation, validation, routing) |
|
|
30
30
|
| suffix `_tool` | Tool/API call | A node that wraps a single external tool or API (e.g. a REST endpoint, DB query) |
|
|
31
|
-
| suffix `_agent` | Subgraph agent | A node that invokes a nested subgraph containing its own tool-invocation cycle and LLM calls; use the **deepagents** library for these nodes (see [agentme-edr-
|
|
31
|
+
| suffix `_agent` | Subgraph agent | A node that invokes a nested subgraph containing its own tool-invocation cycle and LLM calls; use the **deepagents** library for these nodes (see [agentme-edr-142](142-ai-agents-development-standards.md)) |
|
|
32
32
|
| prefix `evaluate_` | Judge node | A node that evaluates the quality, correctness, completeness, or progress of prior outputs and returns a structured verdict; MUST follow rule `03-judge-node-output-format` |
|
|
33
33
|
|
|
34
34
|
The Python function implementing the node SHOULD share the same name as the node alias passed to `add_node`, so that graph definitions and stack traces remain unambiguous:
|
|
@@ -193,7 +193,7 @@ def route_after_evaluate_quality(state) -> str:
|
|
|
193
193
|
return "publish_step"
|
|
194
194
|
```
|
|
195
195
|
|
|
196
|
-
**Logging:** Log `verdict` and the count of each level as MLflow metrics on the current run per [agentme-edr-
|
|
196
|
+
**Logging:** Log `verdict` and the count of each level as MLflow metrics on the current run per [agentme-edr-144](144-ai-workflow-development-standards.md) rule `03-observability-and-experiment-tracking`.
|
|
197
197
|
|
|
198
198
|
#### 04-workflow-naming-conventions
|
|
199
199
|
|
|
@@ -276,6 +276,6 @@ def after_quality_check(state): ... # FORBIDDEN: not named "route_after_<node>"
|
|
|
276
276
|
|
|
277
277
|
## References
|
|
278
278
|
|
|
279
|
-
- [agentme-edr-
|
|
280
|
-
- [agentme-edr-
|
|
281
|
-
- [agentme-edr-
|
|
279
|
+
- [agentme-edr-144](144-ai-workflow-development-standards.md) — Workflow structure, LangGraph toolchain, observability, and testing patterns
|
|
280
|
+
- [agentme-edr-141](141-ai-llm-development-standards.md) — LLM development standards (drives `_llm` node suffix and mocking patterns)
|
|
281
|
+
- [agentme-edr-142](142-ai-agents-development-standards.md) — Agent development standards (drives `_agent` node suffix and state conventions)
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
2
|
+
name: agentme-edr-policy-146-ai-agent-xdrs-knowledge-layer
|
|
3
3
|
description: Defines how to integrate XDRS as the runtime knowledge source of truth for AI agents — covering document placement, AGENTS.md setup, file tools, and local sandbox configuration. Apply only when the project explicitly uses XDRS to govern agent behavior.
|
|
4
4
|
apply-to: AI agent projects that use XDRS as the source of truth for policies and skills
|
|
5
5
|
valid-from: 2026-05-27
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-146: AI agent XDRS knowledge layer
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -17,7 +17,7 @@ How should an AI agent project integrate XDRS as its runtime source of truth for
|
|
|
17
17
|
|
|
18
18
|
**Embed XDRS documents in `lib/data/.xdrs/`, instruct the agent to consult them via `AGENTS.md`, equip the agent with sandboxed file tools, and use the deepagents framework when a local sandbox is required.**
|
|
19
19
|
|
|
20
|
-
This policy MUST only be applied when the project explicitly chooses XDRS as its knowledge governance layer. It is not required by [agentme-edr-
|
|
20
|
+
This policy MUST only be applied when the project explicitly chooses XDRS as its knowledge governance layer. It is not required by [agentme-edr-142](142-ai-agents-development-standards.md) or [agentme-edr-144](144-ai-workflow-development-standards.md) in general.
|
|
21
21
|
|
|
22
22
|
### Details
|
|
23
23
|
|
|
@@ -59,7 +59,7 @@ Read /AGENTS.md and follow all instructions in it before proceeding.
|
|
|
59
59
|
|
|
60
60
|
#### 02-agent-file-tools
|
|
61
61
|
|
|
62
|
-
Every agent that uses the XDRS knowledge layer MUST use the file tools provided by the deepagents framework. Do not implement hand-rolled alternatives — see [agentme-edr-
|
|
62
|
+
Every agent that uses the XDRS knowledge layer MUST use the file tools provided by the deepagents framework. Do not implement hand-rolled alternatives — see [agentme-edr-142 rule 02-local-sandbox](142-ai-agents-development-standards.md) for the full sandbox and tool requirements.
|
|
63
63
|
|
|
64
64
|
These tools operate over two sandboxed roots (configured in rule `03-local-sandbox`):
|
|
65
65
|
|
|
@@ -72,7 +72,7 @@ These tools operate over two sandboxed roots (configured in rule `03-local-sandb
|
|
|
72
72
|
|
|
73
73
|
#### 03-local-sandbox
|
|
74
74
|
|
|
75
|
-
Follow [agentme-edr-
|
|
75
|
+
Follow [agentme-edr-142 rule 02-local-sandbox](142-ai-agents-development-standards.md) for the general deepagents sandbox setup. When XDRS is in use, add the following mounts to the sandbox configuration:
|
|
76
76
|
|
|
77
77
|
| Source | Content | Deepagents sandbox path |
|
|
78
78
|
|---|---|---|
|
|
@@ -91,8 +91,8 @@ data_root = str(files("myagent").joinpath("data"))
|
|
|
91
91
|
agents_md = Path(temp_root) / "AGENTS.md"
|
|
92
92
|
agents_md.write_text(_AGENTS_MD) # content from xdrs-core AGENTS.md template; see rule 01-xdrs-knowledge-layer
|
|
93
93
|
|
|
94
|
-
# Add these mounts alongside the base mounts from agentme-edr-
|
|
95
|
-
# (mount_paths uses {src: dst} dict format per agentme-edr-
|
|
94
|
+
# Add these mounts alongside the base mounts from agentme-edr-142 rule 02-local-sandbox:
|
|
95
|
+
# (mount_paths uses {src: dst} dict format per agentme-edr-142)
|
|
96
96
|
sandbox = Sandbox(
|
|
97
97
|
mount_paths={
|
|
98
98
|
tmp_dir: "/workspace",
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
3
|
-
description: Defines the eval folder structure and Makefile interface for AI projects, and the LLM-as-judge binary scoring contract applicable to all AI tiers and test types. Use when scaffolding evals or implementing an LLM judge. For eval script requirements see agentme-edr-
|
|
2
|
+
name: agentme-edr-policy-151-ai-eval-core-standards
|
|
3
|
+
description: Defines the eval folder structure and Makefile interface for AI projects, and the LLM-as-judge binary scoring contract applicable to all AI tiers and test types. Use when scaffolding evals or implementing an LLM judge. For eval script requirements see agentme-edr-153. For report format see agentme-edr-154. For repeatability see agentme-edr-155. For when evals are required see agentme-edr-501 rule 09-ai-project-testing-requirements. For the test type taxonomy see agentme-edr-152.
|
|
4
4
|
apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
|
|
5
5
|
valid-from: 2026-06-05
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-151: AI eval core standards
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -17,7 +17,7 @@ How should eval tests be structured, and how must LLM judges produce scores acro
|
|
|
17
17
|
|
|
18
18
|
**Use a per-component folder structure under `evals/` with a standardized Makefile interface, and require all LLM judges to produce binary (`0`/`1`) output compatible with classification metrics.**
|
|
19
19
|
|
|
20
|
-
For when evals are required per AI tier, see [agentme-edr-
|
|
20
|
+
For when evals are required per AI tier, see [agentme-edr-501](../governance/501-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
|
|
21
21
|
|
|
22
22
|
### Details
|
|
23
23
|
|
|
@@ -31,7 +31,7 @@ evals/
|
|
|
31
31
|
eval-<name>/
|
|
32
32
|
golden_dataset/ # EDR-024 + EDR-030 compliant golden dataset (README.md, dataset.schema.json, data/)
|
|
33
33
|
eval.py # evaluation script
|
|
34
|
-
report-<type>.md # generated report, one per evaluated test type (overwritten on each run — see agentme-edr-
|
|
34
|
+
report-<type>.md # generated report, one per evaluated test type (overwritten on each run — see agentme-edr-154 rule 01)
|
|
35
35
|
Makefile # lint, eval, run, and eval-<type> targets
|
|
36
36
|
eval-<name2>/
|
|
37
37
|
...
|
|
@@ -41,18 +41,18 @@ evals/
|
|
|
41
41
|
|
|
42
42
|
`<component>` MUST match the name of the component under evaluation and use lowercase hyphen-separated words (e.g., `workflow-document-review`, `agent-support`, `model-classifier`).
|
|
43
43
|
|
|
44
|
-
`<name>` identifies the specific evaluation scenario using lowercase hyphen-separated words (e.g., `eval-basic`, `eval-complex`, `eval-edge-cases`). A scenario's `golden_dataset` MAY mix multiple test types across its entries: label each entry with its applicable `test_types` ([agentme-edr-
|
|
44
|
+
`<name>` identifies the specific evaluation scenario using lowercase hyphen-separated words (e.g., `eval-basic`, `eval-complex`, `eval-edge-cases`). A scenario's `golden_dataset` MAY mix multiple test types across its entries: label each entry with its applicable `test_types` ([agentme-edr-152](152-ai-test-types-taxonomy.md) rule `04`) and use the `eval-<type>` targets below to run one type at a time.
|
|
45
45
|
|
|
46
|
-
The `golden_dataset/` subfolder MUST be a valid [agentme-edr-
|
|
46
|
+
The `golden_dataset/` subfolder MUST be a valid [agentme-edr-201](../data/201-ml-dataset-structure.md) dataset (`README.md`, `dataset.schema.json`, one JSON file per entry under `data/` per rule `04-complex-structured-datasets-must-use-per-entry-json-files`, lint-validated per rule `06`) whose entries follow the golden dataset envelope defined in [agentme-edr-152](152-ai-test-types-taxonomy.md) rule `02`.
|
|
47
47
|
|
|
48
48
|
Each `evals/<component>/eval-<name>/Makefile` MUST declare a `TEST_TYPES` variable listing the `test_types` values present in its golden dataset, and define:
|
|
49
49
|
|
|
50
50
|
| Target | Behaviour |
|
|
51
51
|
|---|---|
|
|
52
|
-
| `lint` | Validates every `golden_dataset/data/*.json` file against `golden_dataset/dataset.schema.json` per [agentme-edr-
|
|
52
|
+
| `lint` | Validates every `golden_dataset/data/*.json` file against `golden_dataset/dataset.schema.json` per [agentme-edr-201](../data/201-ml-dataset-structure.md) rule `06` |
|
|
53
53
|
| `eval` | Depends on `lint`; runs `eval.py --type=all` with threshold enforcement; exits non-zero on failure (CI-safe) |
|
|
54
54
|
| `run` | Depends on `lint`; runs `eval.py --type=all` with threshold enforcement; same as `eval` but intended for local exploration |
|
|
55
|
-
| `eval-<type>` | Depends on `lint`; runs `eval.py --type=<type>` for one declared test type, following [agentme-edr-
|
|
55
|
+
| `eval-<type>` | Depends on `lint`; runs `eval.py --type=<type>` for one declared test type, following [agentme-edr-303](../platform/303-common-targets.md) rule `03`'s `eval-<qualifier>` convention |
|
|
56
56
|
|
|
57
57
|
```makefile
|
|
58
58
|
TEST_TYPES := smoke functional safety
|
|
@@ -94,7 +94,7 @@ LLM judges scoring component outputs MUST produce binary output: `0` (fail) or `
|
|
|
94
94
|
- **F1 computation:** F1 MUST be computed from a binary confusion matrix treating score `1` as the positive class. In a standard golden dataset where every entry is expected to pass, all entries are true positives or false negatives (no true negatives exist), so Recall = TP / (TP + FN) and Precision = TP / (TP + FP) where FP = entries the judge scores `1` that the entry-level expected outcome marks as expected-fail. When all entries are expected-pass, Recall = Accuracy and Precision = 1, making F1 a conservative lower-bound on accuracy. Projects with adversarial or expected-fail entries MUST annotate each golden dataset entry with its expected binary outcome (`1` = should pass, `0` = should fail) in `expected_output` to enable a correct confusion matrix.
|
|
95
95
|
- Multi-class classification not supported. For multiple quality levels, use multiple binary judges (e.g., one for "factually correct", another for "tone appropriate")
|
|
96
96
|
|
|
97
|
-
**Rationale:** Binary output makes LLM judges compatible with classification metrics infrastructure (Accuracy, F1, Wilson CI, convergence analysis) defined in [agentme-edr-
|
|
97
|
+
**Rationale:** Binary output makes LLM judges compatible with classification metrics infrastructure (Accuracy, F1, Wilson CI, convergence analysis) defined in [agentme-edr-154](154-ai-eval-report-format.md) rule `01`.
|
|
98
98
|
|
|
99
99
|
**Example LLM judge prompt:**
|
|
100
100
|
|
|
@@ -112,10 +112,10 @@ Output:
|
|
|
112
112
|
|
|
113
113
|
## References
|
|
114
114
|
|
|
115
|
-
- [agentme-edr-
|
|
116
|
-
- [agentme-edr-
|
|
117
|
-
- [agentme-edr-
|
|
118
|
-
- [agentme-edr-
|
|
119
|
-
- [agentme-edr-
|
|
120
|
-
- [agentme-edr-
|
|
121
|
-
- [agentme-edr-
|
|
115
|
+
- [agentme-edr-153](153-ai-eval-script.md) — AI eval script: entry-first eval loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
|
|
116
|
+
- [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
|
|
117
|
+
- [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: loop exception, scoring methods (including LLM-as-judge per rule `02`), and cadence
|
|
118
|
+
- [agentme-edr-501](../governance/501-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09`) and statistical model eval targets (rule `07`)
|
|
119
|
+
- [agentme-edr-152](152-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum and golden dataset entry envelope
|
|
120
|
+
- [agentme-edr-201](../data/201-ml-dataset-structure.md) — ML dataset structure, per-entry JSON format, and schema-lint validation for golden datasets
|
|
121
|
+
- [agentme-edr-303](../platform/303-common-targets.md) — `eval-<qualifier>` Makefile convention (rule `03`)
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
3
|
-
description: Names AI-application test types grouped as "Safety & adversarial", "Responsible AI", "Quality eval", "Prompt/LLM", and "Code-level" with their objective, mocking constraint, and relevance, and defines the shared "golden dataset" entry envelope that agentme-edr-
|
|
2
|
+
name: agentme-edr-policy-152-ai-test-types-taxonomy
|
|
3
|
+
description: Names AI-application test types grouped as "Safety & adversarial", "Responsible AI", "Quality eval", "Prompt/LLM", and "Code-level" with their objective, mocking constraint, and relevance, and defines the shared "golden dataset" entry envelope that agentme-edr-151's eval tooling filters by test_types. Use when deciding which AI test types to implement or when authoring a golden dataset entry.
|
|
4
4
|
apply-to: AI projects (LLM, Agent, or Workflow tier) implementing AI-specific test types beyond generic code-level unit/integration tests
|
|
5
5
|
valid-from: 2026-07-05
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-152: AI test types taxonomy
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -13,7 +13,7 @@ AI components need test types beyond generic unit/integration tests (safety, fai
|
|
|
13
13
|
|
|
14
14
|
## Decision Outcome
|
|
15
15
|
|
|
16
|
-
**Adopt a named taxonomy of AI test types plus a shared "golden dataset" entry envelope that agentme-edr-
|
|
16
|
+
**Adopt a named taxonomy of AI test types plus a shared "golden dataset" entry envelope that agentme-edr-151's eval tooling filters by `test_types`.**
|
|
17
17
|
|
|
18
18
|
Each test type is named with its group, objective, mocking constraint, applicability, and relevance; every golden dataset entry is labeled with the test types it applies to.
|
|
19
19
|
|
|
@@ -21,7 +21,7 @@ Each test type is named with its group, objective, mocking constraint, applicabi
|
|
|
21
21
|
|
|
22
22
|
#### 01-golden-dataset-concept
|
|
23
23
|
|
|
24
|
-
Projects MUST use a golden dataset to test AI components. A **golden dataset** comprises all eval case entries used to test an AI component (LLM, Agent, or Workflow tier); each entry is labeled with the `test_types` (rule `04`) it applies to. It is the dataset consumed by [agentme-edr-
|
|
24
|
+
Projects MUST use a golden dataset to test AI components. A **golden dataset** comprises all eval case entries used to test an AI component (LLM, Agent, or Workflow tier); each entry is labeled with the `test_types` (rule `04`) it applies to. It is the dataset consumed by [agentme-edr-153](153-ai-eval-script.md) evals and stored as one JSON file per entry per [agentme-edr-201](../data/201-ml-dataset-structure.md) rule `04`, at `evals/<component>/eval-<name>/golden_dataset/`.
|
|
25
25
|
|
|
26
26
|
#### 02-golden-dataset-entry-envelope
|
|
27
27
|
|
|
@@ -42,8 +42,8 @@ Every golden dataset entry (a JSON file in `golden_dataset/data/`) MUST have thi
|
|
|
42
42
|
- `test_types` — array, values MUST come from rule `04`'s enum, MUST contain at least one value. An entry MAY carry more than one value additively (e.g. `["functional", "smoke", "human"]`) — no test type excludes another.
|
|
43
43
|
- `input` — for Prompt-tier components, a raw prompt string or the prompt template's input parameters object; for Agent/Workflow-tier components, the input attributes object passed to the component.
|
|
44
44
|
- `expected_output` — the fields used to score the entry under each of its automated `test_types`: output attributes for an LLM-as-judge rubric, a target for vector-similarity scoring, or exact attribute values for strict comparison. When `human` is one of the entry's `test_types`, `expected_output` MUST additionally include a `human_test` string field with manual-verification instructions (e.g. `"check for ethical issues, verify record change in system X"`) — this supplements, and MUST NOT replace, the entry's automated scoring fields.
|
|
45
|
-
- `mock_fixtures` — optional object; keys identify the adapter or external system to mock (SHOULD match the connector folder name under `adapters/connectors/<name>` for readability, though not enforced), values are any valid JSON interpreted by the mock implementation. When present, eval.py MUST configure each named mock adapter with its fixture data BEFORE invoking the component for that entry; each entry MUST use fresh mock instances to prevent state from bleeding across entries. `mock_fixtures` applies to all `test_types` including `human` — the component is still invoked for human entries to capture `actual_output`. `mock_fixtures` MUST NOT include keys for LLM adapters: all golden dataset test types are rated `mocks disallowed for LLM calls` (rule `03`), so the LLM call MUST be real; LLM provider mocking belongs exclusively to unit tests via [agentme-edr-
|
|
46
|
-
- The dataset's `dataset.schema.json` MUST require `test_types`, `input`, and `expected_output`, and SHOULD declare `mock_fixtures` as optional (`"type": "object", "additionalProperties": {}`), per [agentme-edr-
|
|
45
|
+
- `mock_fixtures` — optional object; keys identify the adapter or external system to mock (SHOULD match the connector folder name under `adapters/connectors/<name>` for readability, though not enforced), values are any valid JSON interpreted by the mock implementation. When present, eval.py MUST configure each named mock adapter with its fixture data BEFORE invoking the component for that entry; each entry MUST use fresh mock instances to prevent state from bleeding across entries. `mock_fixtures` applies to all `test_types` including `human` — the component is still invoked for human entries to capture `actual_output`. `mock_fixtures` MUST NOT include keys for LLM adapters: all golden dataset test types are rated `mocks disallowed for LLM calls` (rule `03`), so the LLM call MUST be real; LLM provider mocking belongs exclusively to unit tests via [agentme-edr-141](141-ai-llm-development-standards.md) rule `04`. See [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
|
|
46
|
+
- The dataset's `dataset.schema.json` MUST require `test_types`, `input`, and `expected_output`, and SHOULD declare `mock_fixtures` as optional (`"type": "object", "additionalProperties": {}`), per [agentme-edr-201](../data/201-ml-dataset-structure.md) rule `04`.
|
|
47
47
|
|
|
48
48
|
#### 03-mocks-allowed-values
|
|
49
49
|
|
|
@@ -51,13 +51,13 @@ The taxonomy in rule `05` rates each test type using one of three values under t
|
|
|
51
51
|
|
|
52
52
|
| Value | Meaning |
|
|
53
53
|
|---|---|
|
|
54
|
-
| `mocks allowed` | Fully offline; fakes may replace every dependency including the LLM (e.g. `FakeListChatModel` per [agentme-edr-
|
|
54
|
+
| `mocks allowed` | Fully offline; fakes may replace every dependency including the LLM (e.g. `FakeListChatModel` per [agentme-edr-141](141-ai-llm-development-standards.md) rule `04`). Used only for code-level unit tests. |
|
|
55
55
|
| `mocks disallowed` | No mocking of any dependency — all real external systems required. Used for integration tests. |
|
|
56
56
|
| `mocks disallowed for LLM calls` | **The LLM call MUST be real; all other external dependencies (databases, APIs, external services) MAY and SHOULD be mocked via `mock_fixtures`.** `mock_fixtures` keys MUST NOT reference LLM adapters. This is the standard constraint for every golden-dataset eval test type. See rule `08` for rationale. |
|
|
57
57
|
|
|
58
58
|
#### 04-test-types-enum
|
|
59
59
|
|
|
60
|
-
A golden dataset entry's `test_types` array MUST only use these values: `safety`, `adversarial`, `fairness`, `bias`, `robustness`, `explainability`, `groundedness`, `functional`, `prompt`, `smoke`, `human`, `repeatability`. These correspond to the dataset-driven rows of rule `05`. **Unit test** and **Integration test** (the two Code-level rows) are NOT part of this enum — they have no golden dataset entries and remain governed entirely by [agentme-edr-
|
|
60
|
+
A golden dataset entry's `test_types` array MUST only use these values: `safety`, `adversarial`, `fairness`, `bias`, `robustness`, `explainability`, `groundedness`, `functional`, `prompt`, `smoke`, `human`, `repeatability`. These correspond to the dataset-driven rows of rule `05`. **Unit test** and **Integration test** (the two Code-level rows) are NOT part of this enum — they have no golden dataset entries and remain governed entirely by [agentme-edr-122](122-unit-test-requirements.md) and [agentme-edr-501](../governance/501-project-quality-standards.md) rule `08`.
|
|
61
61
|
|
|
62
62
|
#### 05-test-type-taxonomy
|
|
63
63
|
|
|
@@ -74,43 +74,43 @@ Test types MUST be selected from this taxonomy. Each test type is named with its
|
|
|
74
74
|
| Groundedness (RAG) eval | `groundedness` | Quality eval | Verify the answer is supported by retrieved context | mocks disallowed for LLM calls | System uses retrieval-augmented generation | Avoids confidently-wrong answers reaching customers | Pinpoints retrieval/prompt bugs | 4 |
|
|
75
75
|
| Repeatability test | `repeatability` | Quality eval | Verify output stability/variance across N repeated invocations of the same input under fixed configuration | mocks disallowed for LLM calls | Non-deterministic components (temperature > 0, agentic tool-selection loops, sampling-based decoding) used in decision-critical or user-facing flows | Protects against silently flaky behavior reaching production; supports consistency SLAs | Detects prompt/agent designs too sensitive to sampling noise; informs temperature/seed tuning | 3 |
|
|
76
76
|
| Human evaluation | `human` | Quality eval | Manually verify aspects automated scoring can't (ethics, side effects, external state) | mocks disallowed for LLM calls | Before major releases; periodic spot-check | Defensible, human-reviewed sign-off | Catches what automated metrics miss | 3 |
|
|
77
|
-
| Functional eval (golden-dataset accuracy / LLM-as-judge) | `functional` | Quality eval | Measure output correctness against the golden dataset | mocks disallowed for LLM calls | Required before every Workflow release ([agentme-edr-
|
|
77
|
+
| Functional eval (golden-dataset accuracy / LLM-as-judge) | `functional` | Quality eval | Measure output correctness against the golden dataset | mocks disallowed for LLM calls | Required before every Workflow release ([agentme-edr-501](../governance/501-project-quality-standards.md) rule `09`); advised elsewhere | Auditable evidence of business correctness before release | Detects regressions from model/provider/prompt changes | 5 |
|
|
78
78
|
| Smoke test | `smoke` | Quality eval | Fast pass/fail check on a small, critical subset before running fuller suites | mocks disallowed for LLM calls | Every commit/PR, before functional/responsible-AI evals run | Cheap early warning before slower evals run | Fast, cheap feedback loop | 4 |
|
|
79
79
|
| Prompt regression test | `prompt` | Prompt/LLM | Detect behavior change when a prompt or model version changes | mocks disallowed for LLM calls | Whenever a prompt template or model version changes | Prevents shipping a worse experience via a "small" tweak | Fast check on every prompt edit | 3 |
|
|
80
80
|
| Integration test | n/a — code-level only (see rule `04`) | Code-level | Verify real interaction with external systems | mocks disallowed | Component depends on external systems | Reduces production outages from integration mismatches | Catches wiring bugs unit tests can't see | 2 |
|
|
81
|
-
| Unit test (offline, mocked) | n/a — code-level only (see rule `04`) | Code-level | Verify deterministic logic in isolation, offline | mocks allowed | Required for Workflow tier every commit ([agentme-edr-
|
|
81
|
+
| Unit test (offline, mocked) | n/a — code-level only (see rule `04`) | Code-level | Verify deterministic logic in isolation, offline | mocks allowed | Required for Workflow tier every commit ([agentme-edr-501](../governance/501-project-quality-standards.md) rule `09`) | Lowest-cost point to catch defects | Fastest, fully offline feedback on every commit | 5 |
|
|
82
82
|
|
|
83
83
|
#### 06-priority-and-relevance-are-descriptive-only
|
|
84
84
|
|
|
85
|
-
Priority, Relevance, and When to Apply in rule `05` are guidance for prioritization conversations — they MUST NOT be treated as mandating which test types a project must implement, nor their thresholds. [agentme-edr-
|
|
85
|
+
Priority, Relevance, and When to Apply in rule `05` are guidance for prioritization conversations — they MUST NOT be treated as mandating which test types a project must implement, nor their thresholds. [agentme-edr-501](../governance/501-project-quality-standards.md) rule `09` remains the only tier-level testing requirement in force (Workflow unit tests + functional evals). Once a project chooses to implement and threshold a test type, [agentme-edr-153](153-ai-eval-script.md) rule `01`'s failing-threshold behavior applies uniformly, regardless of this table's priority rating — a project may enforce fairness at 70% and functional at 90%, or skip fairness entirely; that choice is a project/business decision, not one this Policy makes.
|
|
86
86
|
|
|
87
87
|
#### 07-smoke-is-distinct-from-test-smoke
|
|
88
88
|
|
|
89
|
-
The `smoke` test type (surfaced as the `eval-smoke` Makefile target, a fast subset of the golden-dataset functional eval) is a different concept from [agentme-edr-
|
|
89
|
+
The `smoke` test type (surfaced as the `eval-smoke` Makefile target, a fast subset of the golden-dataset functional eval) is a different concept from [agentme-edr-303](../platform/303-common-targets.md)'s existing `test-smoke` target (a fast subset of code-level tests). Both MAY exist in the same project; teams MUST NOT conflate them.
|
|
90
90
|
|
|
91
91
|
#### 08-eval-mocking-constraint
|
|
92
92
|
|
|
93
|
-
For every golden-dataset eval test type: **the LLM call MUST be real; all other external dependencies MUST be mocked via `mock_fixtures`.** This applies equally to `human` entries — the component is invoked to capture `actual_output` and external dependencies must be deterministic. `mock_fixtures` keys MUST NOT reference LLM adapters. Code-level unit tests are the correct place for fully offline, LLM-mocked testing (see [agentme-edr-
|
|
93
|
+
For every golden-dataset eval test type: **the LLM call MUST be real; all other external dependencies MUST be mocked via `mock_fixtures`.** This applies equally to `human` entries — the component is invoked to capture `actual_output` and external dependencies must be deterministic. `mock_fixtures` keys MUST NOT reference LLM adapters. Code-level unit tests are the correct place for fully offline, LLM-mocked testing (see [agentme-edr-141](141-ai-llm-development-standards.md) rule `04`).
|
|
94
94
|
|
|
95
95
|
#### 09-repeatability-vs-reproducibility
|
|
96
96
|
|
|
97
97
|
| Property | Definition | What varies | Measured by |
|
|
98
98
|
|---|---|---|---|
|
|
99
|
-
| **Repeatability** | Output stability across N invocations at non-zero temperature | Model sampling variance | `repeatability` test type per [agentme-edr-
|
|
100
|
-
| **Reproducibility** | Deterministic output at temperature = 0 with fixed seed | Nothing — any variance is a config bug | Not a golden-dataset type; verified via config, documented in [agentme-edr-
|
|
99
|
+
| **Repeatability** | Output stability across N invocations at non-zero temperature | Model sampling variance | `repeatability` test type per [agentme-edr-155](155-ai-eval-repeatability.md) |
|
|
100
|
+
| **Reproducibility** | Deterministic output at temperature = 0 with fixed seed | Nothing — any variance is a config bug | Not a golden-dataset type; verified via config, documented in [agentme-edr-305](../platform/305-environment-variable-configuration.md) |
|
|
101
101
|
|
|
102
102
|
A component may satisfy reproducibility (temperature = 0) yet still need repeatability tests for its production configuration (temperature > 0). The `repeatability` test type MUST NOT be applied to components with intentionally diverse output (brainstorming, creative generation) — variance is correct behavior there.
|
|
103
103
|
|
|
104
104
|
## References
|
|
105
105
|
|
|
106
|
-
- [agentme-edr-
|
|
107
|
-
- [agentme-edr-
|
|
108
|
-
- [agentme-edr-
|
|
109
|
-
- [agentme-edr-
|
|
110
|
-
- [agentme-edr-
|
|
111
|
-
- [agentme-edr-
|
|
112
|
-
- [agentme-edr-
|
|
113
|
-
- [agentme-edr-
|
|
114
|
-
- [agentme-edr-
|
|
115
|
-
- [agentme-edr-
|
|
116
|
-
- [agentme-edr-
|
|
106
|
+
- [agentme-edr-201](../data/201-ml-dataset-structure.md) — Golden dataset file layout, per-entry JSON format, `$schema` pointer, and schema-lint validation
|
|
107
|
+
- [agentme-edr-151](151-ai-eval-core-standards.md) — AI eval core standards: eval folder structure and Makefile targets (rule `01`); LLM-as-judge binary scoring contract (rule `02`)
|
|
108
|
+
- [agentme-edr-153](153-ai-eval-script.md) — AI eval script: `--type` filtering, entry-first loop, `mock_fixtures`, threshold enforcement, and MLflow conventions
|
|
109
|
+
- [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: per-type `report-<type>.md` that consumes this taxonomy's test types
|
|
110
|
+
- [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: `REPEAT_COUNT` loop exception, scoring constants (`EVAL_MIN_ACCURACY_REPEATABILITY`, `REPEAT_SEMANTIC_SIMILARITY_SCORE`), scoring methods, `repeatability_accuracy` MLflow metric, report shape, and run cadence
|
|
111
|
+
- [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) — Rule `10`: `_mock` file naming and placement convention for mock adapters referenced by `mock_fixtures`
|
|
112
|
+
- [agentme-edr-501](../governance/501-project-quality-standards.md) — Rule `09` tier-level testing requirements (the only mandated AI testing baseline)
|
|
113
|
+
- [agentme-edr-303](../platform/303-common-targets.md) — Rule `03` `eval-<qualifier>` Makefile convention; rule `03`'s `test-smoke` (distinguished in rule `07`)
|
|
114
|
+
- [agentme-edr-305](../platform/305-environment-variable-configuration.md) — Environment-configuration conventions referenced in rule `09`'s reproducibility disambiguation
|
|
115
|
+
- [agentme-edr-141](141-ai-llm-development-standards.md) — LLM tier definition and mocking utilities referenced by the `mocks allowed` value
|
|
116
|
+
- [agentme-edr-122](122-unit-test-requirements.md) — Unit test requirements underlying the Code-level rows
|