agentme 0.28.1 → 0.30.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.filedist-package.yml +2 -2
- package/.xdrs/agentme/edrs/application/{003-javascript-project-tooling.md → 101-javascript-project-tooling.md} +9 -9
- package/.xdrs/agentme/edrs/application/{010-golang-project-tooling.md → 102-golang-project-tooling.md} +7 -7
- package/.xdrs/agentme/edrs/application/{014-python-project-tooling.md → 103-python-project-tooling.md} +12 -12
- package/.xdrs/agentme/edrs/application/{015-cli-tool-standards.md → 104-cli-tool-standards.md} +13 -13
- package/.xdrs/agentme/edrs/application/{002-coding-best-practices.md → 121-coding-best-practices.md} +3 -3
- package/.xdrs/agentme/edrs/application/{004-unit-test-requirements.md → 122-unit-test-requirements.md} +2 -2
- package/.xdrs/agentme/edrs/application/{009-error-handling.md → 123-error-handling.md} +2 -2
- package/.xdrs/agentme/edrs/application/{022-secrets-management.md → 124-secrets-management.md} +5 -5
- package/.xdrs/agentme/edrs/application/{023-coding-abstraction-practices.md → 125-coding-abstraction-practices.md} +3 -3
- package/.xdrs/agentme/edrs/application/{026-pragmatic-hexagonal-architecture.md → 126-pragmatic-hexagonal-architecture.md} +9 -9
- package/.xdrs/agentme/edrs/application/127-external-system-adapter-skills.md +93 -0
- package/.xdrs/agentme/edrs/application/{040-ai-llm-development-standards.md → 141-ai-llm-development-standards.md} +14 -14
- package/.xdrs/agentme/edrs/application/{041-ai-agents-development-standards.md → 142-ai-agents-development-standards.md} +11 -11
- package/.xdrs/agentme/edrs/application/{042-ai-agents-quality-standards.md → 143-ai-agents-quality-standards.md} +16 -16
- package/.xdrs/agentme/edrs/application/{043-ai-workflow-development-standards.md → 144-ai-workflow-development-standards.md} +28 -28
- package/.xdrs/agentme/edrs/application/{044-ai-workflow-naming-conventions.md → 145-ai-workflow-naming-conventions.md} +9 -9
- package/.xdrs/agentme/edrs/application/{045-ai-agent-xdrs-knowledge-layer.md → 146-ai-agent-xdrs-knowledge-layer.md} +7 -7
- package/.xdrs/agentme/edrs/application/{051-ai-eval-core-standards.md → 151-ai-eval-standards.md} +17 -17
- package/.xdrs/agentme/edrs/application/{052-ai-test-types-taxonomy.md → 152-ai-test-types-taxonomy.md} +27 -27
- package/.xdrs/agentme/edrs/application/{053-ai-eval-script.md → 153-ai-eval-script.md} +18 -18
- package/.xdrs/agentme/edrs/application/{054-ai-eval-report-format.md → 154-ai-eval-report-format.md} +6 -6
- package/.xdrs/agentme/edrs/application/{055-ai-eval-repeatability.md → 155-ai-eval-repeatability.md} +14 -14
- package/.xdrs/agentme/edrs/application/skills/{004-select-relevant-xdrs → 010-select-relevant-xdrs}/SKILL.md +10 -10
- package/.xdrs/agentme/edrs/application/skills/{001-create-javascript-project → 050-create-javascript-project}/SKILL.md +5 -5
- package/.xdrs/agentme/edrs/application/skills/{003-create-golang-project → 051-create-golang-project}/SKILL.md +4 -4
- package/.xdrs/agentme/edrs/application/skills/{005-create-python-project → 052-create-python-project}/SKILL.md +6 -6
- package/.xdrs/agentme/edrs/application/skills/200-run-skill-tests/SKILL.md +113 -0
- package/.xdrs/agentme/edrs/application/skills/200-run-skill-tests/SKILL.test.md +47 -0
- package/.xdrs/agentme/edrs/data/{050-ml-dataset-structure.md → 201-ml-dataset-structure.md} +4 -4
- package/.xdrs/agentme/edrs/governance/{007-project-quality-standards.md → 501-project-quality-standards.md} +16 -16
- package/.xdrs/agentme/edrs/governance/{013-contributing-guide-requirements.md → 502-contributing-guide-requirements.md} +4 -4
- package/.xdrs/agentme/edrs/index.md +33 -31
- package/.xdrs/agentme/edrs/operations/{011-service-health-check-endpoint.md → 401-service-health-check-endpoint.md} +2 -2
- package/.xdrs/agentme/edrs/platform/{005-monorepo-structure.md → 301-monorepo-structure.md} +5 -5
- package/.xdrs/agentme/edrs/platform/{006-github-pipelines.md → 302-github-pipelines.md} +3 -3
- package/.xdrs/agentme/edrs/platform/{008-common-targets.md → 303-common-targets.md} +8 -8
- package/.xdrs/agentme/edrs/platform/{017-tool-execution-and-scripting.md → 304-tool-execution-and-scripting.md} +10 -10
- package/.xdrs/agentme/edrs/platform/{027-environment-variable-configuration.md → 305-environment-variable-configuration.md} +14 -14
- package/.xdrs/agentme/edrs/platform/skills/{002-monorepo-setup → 053-monorepo-setup}/SKILL.md +4 -4
- package/.xdrs/agentme/edrs/principles/012-continuous-xdr-enrichment.md +2 -2
- package/.xdrs/agentme/edrs/principles/016-cross-language-module-structure.md +4 -4
- package/.xdrs/agentme/edrs/principles/017-skill-testing.md +77 -0
- package/.xdrs/agentme/edrs/principles/articles/001-continuous-xdr-improvement.md +1 -1
- package/package.json +2 -2
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
3
|
-
description: Defines the structural patterns and design decisions for building AI agents with tool-invocation loops using the deepagents framework: framework selection, sandbox setup, state naming, agent naming, composition patterns, and system prompt structure. Use when designing or scaffolding a new agent. For tool definitions, error handling, observability, and testing see agentme-edr-
|
|
2
|
+
name: agentme-edr-policy-142-ai-agents-development-standards
|
|
3
|
+
description: Defines the structural patterns and design decisions for building AI agents with tool-invocation loops using the deepagents framework: framework selection, sandbox setup, state naming, agent naming, composition patterns, and system prompt structure. Use when designing or scaffolding a new agent. For tool definitions, error handling, observability, and testing see agentme-edr-143. For simple LLM calls see agentme-edr-141, for workflow orchestration see agentme-edr-144.
|
|
4
4
|
apply-to: AI agent projects — consult when designing agent structure, choosing sandbox approach, defining naming conventions, and composing multi-agent systems
|
|
5
5
|
valid-from: 2026-06-05
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-142: AI agents development standards
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -29,7 +29,7 @@ All agent implementations MUST use the **deepagents** framework.
|
|
|
29
29
|
|
|
30
30
|
- Use deepagents whenever the LLM needs to autonomously select and invoke tools to accomplish a task.
|
|
31
31
|
- The agent MUST follow the perceive → plan → act → observe cycle where the LLM observes tool outputs and decides the next action.
|
|
32
|
-
- All LLM calls within agents MUST follow [agentme-edr-
|
|
32
|
+
- All LLM calls within agents MUST follow [agentme-edr-141](141-ai-llm-development-standards.md) for LangChain configuration and observability.
|
|
33
33
|
|
|
34
34
|
**When to use agents vs workflows:**
|
|
35
35
|
|
|
@@ -87,7 +87,7 @@ def run_file_analysis_agent(input_files: List[Path]) -> AnalysisResult:
|
|
|
87
87
|
**State type naming:**
|
|
88
88
|
|
|
89
89
|
- Agent state types MUST end with `_agent_state` suffix (e.g., `file_analyzer_agent_state`)
|
|
90
|
-
- Follow [agentme-edr-
|
|
90
|
+
- Follow [agentme-edr-144](144-ai-workflow-development-standards.md) rule `11-state-type-conventions` when agents are used as workflow nodes
|
|
91
91
|
|
|
92
92
|
#### 04-agent-naming-conventions
|
|
93
93
|
|
|
@@ -104,14 +104,14 @@ Agent class names MUST follow the pattern `<Purpose>Agent` where `<Purpose>` des
|
|
|
104
104
|
- `MyAgent` (not descriptive)
|
|
105
105
|
- `Agent1` (numbered, not semantic)
|
|
106
106
|
|
|
107
|
-
When agents are used as nodes in workflows, the node name MUST use the `_agent` suffix per [agentme-edr-
|
|
107
|
+
When agents are used as nodes in workflows, the node name MUST use the `_agent` suffix per [agentme-edr-144](144-ai-workflow-development-standards.md) rule `09-node-naming-conventions`.
|
|
108
108
|
|
|
109
109
|
#### 05-agent-composition
|
|
110
110
|
|
|
111
111
|
When multiple agents are needed, one of these composition patterns MUST be chosen:
|
|
112
112
|
|
|
113
113
|
- **Single agent with multiple tools:** Use when tools share a common goal and context (e.g., a code analysis agent with `read_file`, `search_code`, and `analyze_pattern` tools).
|
|
114
|
-
- **Multiple agents as workflow nodes:** Use when agents have distinct responsibilities and outputs that feed into each other. Orchestrate them using LangGraph per [agentme-edr-
|
|
114
|
+
- **Multiple agents as workflow nodes:** Use when agents have distinct responsibilities and outputs that feed into each other. Orchestrate them using LangGraph per [agentme-edr-144](144-ai-workflow-development-standards.md).
|
|
115
115
|
- Nested agent loops (agent calling agent autonomously) MUST NOT be created. Use workflows for multi-agent orchestration.
|
|
116
116
|
|
|
117
117
|
**Decision criteria:**
|
|
@@ -267,7 +267,7 @@ def parse_agent_output(raw: str) -> FileAnalysisResult:
|
|
|
267
267
|
|
|
268
268
|
## References
|
|
269
269
|
|
|
270
|
-
- [agentme-edr-
|
|
271
|
-
- [agentme-edr-
|
|
272
|
-
- [agentme-edr-
|
|
273
|
-
- [agentme-edr-
|
|
270
|
+
- [agentme-edr-141](141-ai-llm-development-standards.md) — LLM development standards (LangChain configuration, mocking patterns)
|
|
271
|
+
- [agentme-edr-144](144-ai-workflow-development-standards.md) — Workflow development standards (using agents as workflow nodes)
|
|
272
|
+
- [agentme-edr-143](143-ai-agents-quality-standards.md) — Agent implementation quality standards (tool definitions, error handling, observability, unit testing)
|
|
273
|
+
- [agentme-edr-103](103-python-project-tooling.md) — Python project tooling and structure
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
3
|
-
description: Defines implementation quality standards for AI agents: tool definition patterns, error handling and recovery, observability, and unit testing. Apply alongside agentme-edr-
|
|
2
|
+
name: agentme-edr-policy-143-ai-agents-quality-standards
|
|
3
|
+
description: Defines implementation quality standards for AI agents: tool definition patterns, error handling and recovery, observability, and unit testing. Apply alongside agentme-edr-142 when implementing or reviewing agent code. For agent architecture and structural decisions (framework, sandbox, naming, composition, system prompts) see agentme-edr-142.
|
|
4
4
|
apply-to: AI agent implementation code — apply when writing tools, error handlers, logging, and unit tests for agents
|
|
5
5
|
valid-from: 2026-06-09
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-143: AI agents quality standards
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -15,7 +15,7 @@ How should agent tools be defined, what error handling must agents implement, ho
|
|
|
15
15
|
|
|
16
16
|
## Decision Outcome
|
|
17
17
|
|
|
18
|
-
**Agent implementations MUST follow the tool definition, error handling, observability, and unit testing standards defined here, alongside the structural decisions in [agentme-edr-
|
|
18
|
+
**Agent implementations MUST follow the tool definition, error handling, observability, and unit testing standards defined here, alongside the structural decisions in [agentme-edr-142](142-ai-agents-development-standards.md).**
|
|
19
19
|
|
|
20
20
|
### Details
|
|
21
21
|
|
|
@@ -49,7 +49,7 @@ def tool_name(arg1: str, arg2: int) -> Dict[str, Any]:
|
|
|
49
49
|
- Tool docstrings MUST clearly describe the tool's purpose, arguments, and return value (the LLM reads these)
|
|
50
50
|
- Tools MUST return structured data (dictionaries or dataclasses), not bare strings or untyped values
|
|
51
51
|
- Tools MUST handle errors gracefully and return error information in the result structure, not raise exceptions
|
|
52
|
-
- Tools that interact with external systems MUST be placed in `adapters/connectors/` per [agentme-edr-
|
|
52
|
+
- Tools that interact with external systems MUST be placed in `adapters/connectors/` per [agentme-edr-126](126-pragmatic-hexagonal-architecture.md)
|
|
53
53
|
|
|
54
54
|
**Error handling in tools:**
|
|
55
55
|
|
|
@@ -100,9 +100,9 @@ Agent execution MUST be observable through logging and tracing:
|
|
|
100
100
|
|
|
101
101
|
- Log each iteration of the perceive → plan → act → observe cycle with iteration number and tool selection.
|
|
102
102
|
- Use structured logging (JSON) with fields: `iteration`, `tool_selected`, `tool_result_status`, `decision`.
|
|
103
|
-
- For LLM calls within agents, follow [agentme-edr-
|
|
103
|
+
- For LLM calls within agents, follow [agentme-edr-141](141-ai-llm-development-standards.md) rule `03-llm-observability`.
|
|
104
104
|
- When agents run as workflow nodes, MLflow tracking from the parent workflow automatically captures agent-level traces.
|
|
105
|
-
- The project Makefile MUST expose a `dev-mlflow` target to start a local MLflow tracking server for development inspection, per [agentme-edr-
|
|
105
|
+
- The project Makefile MUST expose a `dev-mlflow` target to start a local MLflow tracking server for development inspection, per [agentme-edr-303](../platform/303-common-targets.md) rule `09-ai-project-dev-targets`.
|
|
106
106
|
|
|
107
107
|
**Example structured log entry:**
|
|
108
108
|
|
|
@@ -120,7 +120,7 @@ Agent execution MUST be observable through logging and tracing:
|
|
|
120
120
|
|
|
121
121
|
#### 04-agent-unit-testing
|
|
122
122
|
|
|
123
|
-
Agent LLM calls are external API calls and MUST be mocked in unit tests per [agentme-edr-
|
|
123
|
+
Agent LLM calls are external API calls and MUST be mocked in unit tests per [agentme-edr-141](141-ai-llm-development-standards.md) rule `04-unit-test-mocking`.
|
|
124
124
|
|
|
125
125
|
Because agents drive a tool-invocation loop — where the LLM decides which tools to call — the fake model must return **tool-call messages** followed by a final answer. Use **`GenericFakeChatModel`** for this:
|
|
126
126
|
|
|
@@ -150,7 +150,7 @@ def test_file_analyzer_agent_calls_search_then_stops():
|
|
|
150
150
|
assert "3 Python files" in result.summary
|
|
151
151
|
```
|
|
152
152
|
|
|
153
|
-
Agents MUST be designed so that the LLM instance is injectable (constructor parameter) to allow test doubles. See [agentme-edr-
|
|
153
|
+
Agents MUST be designed so that the LLM instance is injectable (constructor parameter) to allow test doubles. See [agentme-edr-141](141-ai-llm-development-standards.md) rule `04-unit-test-mocking` for the injectable LLM pattern.
|
|
154
154
|
|
|
155
155
|
**`mock_deep_agent`**
|
|
156
156
|
|
|
@@ -175,10 +175,10 @@ def test_workflow_calls_subagent(mocker):
|
|
|
175
175
|
|
|
176
176
|
## References
|
|
177
177
|
|
|
178
|
-
- [agentme-edr-
|
|
179
|
-
- [agentme-edr-
|
|
180
|
-
- [agentme-edr-
|
|
181
|
-
- [agentme-edr-
|
|
182
|
-
- [agentme-edr-
|
|
183
|
-
- [agentme-edr-
|
|
184
|
-
- [agentme-edr-
|
|
178
|
+
- [agentme-edr-142](142-ai-agents-development-standards.md) — Agent development standards (framework, sandbox, naming, composition, system prompts)
|
|
179
|
+
- [agentme-edr-141](141-ai-llm-development-standards.md) — LLM development standards (LangChain configuration, mocking patterns)
|
|
180
|
+
- [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) — Hexagonal architecture (tool placement in adapters/connectors)
|
|
181
|
+
- [agentme-edr-501](../governance/501-project-quality-standards.md) — Project quality standards including AI-tier testing requirements (rule `09-ai-project-testing-requirements`)
|
|
182
|
+
- [agentme-edr-151](151-ai-eval-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract (rule `02`)
|
|
183
|
+
- [agentme-edr-153](153-ai-eval-script.md) — AI eval script: entry-first loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
|
|
184
|
+
- [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
3
|
-
description: Defines the standard toolchain, framework, observability, and workflow patterns for building LangGraph workflows in Python. Use when scaffolding, reviewing, or extending AI workflow projects that orchestrate LLM calls, agents, and algorithmic nodes. For simple LLM calls see agentme-edr-
|
|
2
|
+
name: agentme-edr-policy-144-ai-workflow-development-standards
|
|
3
|
+
description: Defines the standard toolchain, framework, observability, and workflow patterns for building LangGraph workflows in Python. Use when scaffolding, reviewing, or extending AI workflow projects that orchestrate LLM calls, agents, and algorithmic nodes. For simple LLM calls see agentme-edr-141, for agentic patterns see agentme-edr-142. For naming conventions (nodes, states, routes, judge output schema) see agentme-edr-145.
|
|
4
4
|
apply-to: AI workflow projects using LangGraph StateGraph built with Python
|
|
5
5
|
valid-from: 2026-06-05
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-144: AI workflow development standards
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -23,21 +23,21 @@ Which tools, frameworks, and design patterns should AI workflow projects follow
|
|
|
23
23
|
|
|
24
24
|
Workflows MUST be built with **LangGraph**. Use LangGraph `StateGraph` to model each distinct workflow as an explicit directed graph with typed state.
|
|
25
25
|
|
|
26
|
-
For all direct LLM calls within workflow nodes, use LangChain per [agentme-edr-
|
|
26
|
+
For all direct LLM calls within workflow nodes, use LangChain per [agentme-edr-141](141-ai-llm-development-standards.md). For agent nodes with tool-invocation loops, use deepagents per [agentme-edr-142](142-ai-agents-development-standards.md).
|
|
27
27
|
|
|
28
28
|
#### 03-observability-and-experiment-tracking
|
|
29
29
|
|
|
30
30
|
Use **MLflow** for all workflow observability and evaluation:
|
|
31
31
|
|
|
32
32
|
- **Workflow-level tracking:** Wrap each workflow run with `mlflow.start_run()` to capture traces, parameters, and metrics locally.
|
|
33
|
-
- **LLM-level auto-tracing:** Enable LangChain auto-tracing per [agentme-edr-
|
|
33
|
+
- **LLM-level auto-tracing:** Enable LangChain auto-tracing per [agentme-edr-141](141-ai-llm-development-standards.md) rule `03-llm-observability` by calling `mlflow.langchain.autolog()` during application startup. This captures inputs, outputs, token counts, and latency for every LangChain call within workflow nodes.
|
|
34
34
|
- Log run parameters (model name, temperature, prompt version) and output metrics (accuracy, latency, token counts) using `mlflow.log_param` / `mlflow.log_metric`.
|
|
35
35
|
- Run a local MLflow tracking server with `mlflow ui` to inspect runs during development. Do not require a remote MLflow server for local development.
|
|
36
|
-
- The project Makefile MUST expose a `dev-mlflow` target to start the local MLflow tracking server, per [agentme-edr-
|
|
36
|
+
- The project Makefile MUST expose a `dev-mlflow` target to start the local MLflow tracking server, per [agentme-edr-303](../platform/303-common-targets.md) rule `09-ai-project-dev-targets`.
|
|
37
37
|
|
|
38
38
|
#### 04-dataset-driven-accuracy-measurement
|
|
39
39
|
|
|
40
|
-
Projects MUST follow the eval dataset and implementation requirements defined in [agentme-edr-
|
|
40
|
+
Projects MUST follow the eval dataset and implementation requirements defined in [agentme-edr-151](151-ai-eval-standards.md) and [agentme-edr-153](153-ai-eval-script.md). Testing requirements (when evals are required, release gates) are defined in [agentme-edr-501](../governance/501-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
|
|
41
41
|
|
|
42
42
|
#### 05-flow-documentation
|
|
43
43
|
|
|
@@ -69,7 +69,7 @@ Workflows MUST include at least one explicit verification node before producing
|
|
|
69
69
|
|
|
70
70
|
#### 07-workflow-structure
|
|
71
71
|
|
|
72
|
-
Workflow logic MUST be organized as named workflows following [agentme-edr-
|
|
72
|
+
Workflow logic MUST be organized as named workflows following [agentme-edr-126](126-pragmatic-hexagonal-architecture.md). Each workflow is an independent LangGraph `StateGraph` with a defined start node and end node, connecting LLM nodes, agent nodes, algorithmic nodes, states, routes, and decision nodes.
|
|
73
73
|
|
|
74
74
|
Workflows live inside `app/workflows/` (the application layer), while external integrations such as LLM providers, vector stores, and third-party APIs live under `adapters/connectors/` (the outbound adapter layer). Inbound interfaces (HTTP API, CLI) live under `adapters/` as inbound adapters.
|
|
75
75
|
|
|
@@ -101,17 +101,17 @@ lib/src/<package_name>/
|
|
|
101
101
|
|
|
102
102
|
#### 08-workflow-evals
|
|
103
103
|
|
|
104
|
-
Projects MUST follow the eval folder structure defined in [agentme-edr-
|
|
104
|
+
Projects MUST follow the eval folder structure defined in [agentme-edr-151](151-ai-eval-standards.md) rule `01` and the eval script requirements defined in [agentme-edr-153](153-ai-eval-script.md). For LLM-as-judge scoring used in workflow verification nodes and evals, see [agentme-edr-151](151-ai-eval-standards.md) rule `02`.
|
|
105
105
|
|
|
106
106
|
#### 09-node-naming-conventions
|
|
107
107
|
|
|
108
|
-
Nodes MUST follow the naming conventions defined in [agentme-edr-
|
|
108
|
+
Nodes MUST follow the naming conventions defined in [agentme-edr-145](145-ai-workflow-naming-conventions.md) rule `01-node-naming-conventions`.
|
|
109
109
|
|
|
110
110
|
#### 10-workflow-unit-testing
|
|
111
111
|
|
|
112
|
-
All LLM calls within workflow nodes are external API calls and MUST be mocked in unit tests per [agentme-edr-
|
|
112
|
+
All LLM calls within workflow nodes are external API calls and MUST be mocked in unit tests per [agentme-edr-141](141-ai-llm-development-standards.md) rule `04-unit-test-mocking`. Workflow unit tests MUST run fully offline with no real LLM provider calls.
|
|
113
113
|
|
|
114
|
-
Choose the mock utility per [agentme-edr-
|
|
114
|
+
Choose the mock utility per [agentme-edr-141](141-ai-llm-development-standards.md) rule `04-unit-test-mocking`. For workflows containing `_agent` nodes that drive a tool-invocation loop, MUST use `GenericFakeChatModel`.
|
|
115
115
|
|
|
116
116
|
**Example — workflow with plain-text LLM nodes:**
|
|
117
117
|
|
|
@@ -152,19 +152,19 @@ def test_document_workflow_with_agent_node():
|
|
|
152
152
|
assert result.status == "approved"
|
|
153
153
|
```
|
|
154
154
|
|
|
155
|
-
Workflows MUST accept the LLM instance as a constructor parameter so that unit tests can inject a fake. See the injectable LLM pattern in [agentme-edr-
|
|
155
|
+
Workflows MUST accept the LLM instance as a constructor parameter so that unit tests can inject a fake. See the injectable LLM pattern in [agentme-edr-141](141-ai-llm-development-standards.md) rule `04-unit-test-mocking`.
|
|
156
156
|
|
|
157
157
|
#### 11-state-type-conventions
|
|
158
158
|
|
|
159
|
-
State types MUST follow the conventions defined in [agentme-edr-
|
|
159
|
+
State types MUST follow the conventions defined in [agentme-edr-145](145-ai-workflow-naming-conventions.md) rule `02-state-type-conventions`.
|
|
160
160
|
|
|
161
161
|
#### 12-workflow-naming-conventions
|
|
162
162
|
|
|
163
|
-
Workflows MUST be named following the conventions in [agentme-edr-
|
|
163
|
+
Workflows MUST be named following the conventions in [agentme-edr-145](145-ai-workflow-naming-conventions.md) rule `04-workflow-naming-conventions`.
|
|
164
164
|
|
|
165
165
|
#### 13-judge-node-output-format
|
|
166
166
|
|
|
167
|
-
Judge nodes MUST use the output format defined in [agentme-edr-
|
|
167
|
+
Judge nodes MUST use the output format defined in [agentme-edr-145](145-ai-workflow-naming-conventions.md) rule `03-judge-node-output-format`.
|
|
168
168
|
|
|
169
169
|
#### 15-workflow-state-persistence
|
|
170
170
|
|
|
@@ -197,18 +197,18 @@ result = graph.invoke(input_state, config={"thread_id": "session-123"})
|
|
|
197
197
|
|
|
198
198
|
#### 16-cross-element-naming-coherence
|
|
199
199
|
|
|
200
|
-
All workflow elements MUST maintain naming coherence as defined in [agentme-edr-
|
|
200
|
+
All workflow elements MUST maintain naming coherence as defined in [agentme-edr-145](145-ai-workflow-naming-conventions.md) rule `05-cross-element-naming-coherence`.
|
|
201
201
|
|
|
202
202
|
## References
|
|
203
203
|
|
|
204
|
-
- [agentme-edr-
|
|
205
|
-
- [agentme-edr-
|
|
206
|
-
- [agentme-edr-
|
|
207
|
-
- [agentme-edr-
|
|
208
|
-
- [agentme-edr-
|
|
209
|
-
- [agentme-edr-
|
|
210
|
-
- [agentme-edr-
|
|
211
|
-
- [agentme-edr-
|
|
212
|
-
- [agentme-edr-
|
|
213
|
-
- [agentme-edr-
|
|
214
|
-
- [agentme-edr-
|
|
204
|
+
- [agentme-edr-145](145-ai-workflow-naming-conventions.md) — AI workflow naming conventions: node suffixes/prefixes, state types, judge output schema, workflow names, and cross-element coherence
|
|
205
|
+
- [agentme-edr-141](141-ai-llm-development-standards.md) — LLM development standards: LangChain framework, provider configuration, LLM observability, and unit test mocking
|
|
206
|
+
- [agentme-edr-142](142-ai-agents-development-standards.md) — Agent development standards: deepagents framework, tool-invocation loops, and agent patterns
|
|
207
|
+
- [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) — Adapter/application layer separation that defines the project layout
|
|
208
|
+
- [agentme-edr-103](103-python-project-tooling.md) — Python project tooling and structure
|
|
209
|
+
- [agentme-edr-201](../data/201-ml-dataset-structure.md) — ML dataset structure for eval datasets
|
|
210
|
+
- [agentme-edr-151](151-ai-eval-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract for all tiers (rule `02`)
|
|
211
|
+
- [agentme-edr-153](153-ai-eval-script.md) — AI eval script: entry-first loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
|
|
212
|
+
- [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
|
|
213
|
+
- [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: REPEAT_COUNT loop, scoring methods, and release cadence
|
|
214
|
+
- [agentme-edr-501](../governance/501-project-quality-standards.md) — Project quality standards including AI-tier testing requirements (rule `09-ai-project-testing-requirements`)
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
3
|
-
description: Defines the naming vocabulary for LangGraph workflow elements: node suffixes/prefixes, state type names, state attribute grouping, workflow class/variable names, judge node output schema, route function names, and cross-element coherence rules. Use when naming any part of a LangGraph workflow — nodes, states, routes, or the workflow itself. For workflow structure and toolchain see agentme-edr-
|
|
2
|
+
name: agentme-edr-policy-145-ai-workflow-naming-conventions
|
|
3
|
+
description: Defines the naming vocabulary for LangGraph workflow elements: node suffixes/prefixes, state type names, state attribute grouping, workflow class/variable names, judge node output schema, route function names, and cross-element coherence rules. Use when naming any part of a LangGraph workflow — nodes, states, routes, or the workflow itself. For workflow structure and toolchain see agentme-edr-144.
|
|
4
4
|
apply-to: AI workflow projects using LangGraph StateGraph built with Python
|
|
5
5
|
valid-from: 2026-06-21
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-145: AI workflow naming conventions
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -25,10 +25,10 @@ LangGraph node names MUST follow a suffix convention that communicates the node'
|
|
|
25
25
|
|
|
26
26
|
| Convention | Node type | When to use |
|
|
27
27
|
|---|---|---|
|
|
28
|
-
| suffix `_llm` | LLM call | Any node whose primary action is a direct LLM inference call (see [agentme-edr-
|
|
28
|
+
| suffix `_llm` | LLM call | Any node whose primary action is a direct LLM inference call (see [agentme-edr-141](141-ai-llm-development-standards.md)) |
|
|
29
29
|
| suffix `_step` | Algorithmic step | Deterministic logic with no LLM involvement (transformation, validation, routing) |
|
|
30
30
|
| suffix `_tool` | Tool/API call | A node that wraps a single external tool or API (e.g. a REST endpoint, DB query) |
|
|
31
|
-
| suffix `_agent` | Subgraph agent | A node that invokes a nested subgraph containing its own tool-invocation cycle and LLM calls; use the **deepagents** library for these nodes (see [agentme-edr-
|
|
31
|
+
| suffix `_agent` | Subgraph agent | A node that invokes a nested subgraph containing its own tool-invocation cycle and LLM calls; use the **deepagents** library for these nodes (see [agentme-edr-142](142-ai-agents-development-standards.md)) |
|
|
32
32
|
| prefix `evaluate_` | Judge node | A node that evaluates the quality, correctness, completeness, or progress of prior outputs and returns a structured verdict; MUST follow rule `03-judge-node-output-format` |
|
|
33
33
|
|
|
34
34
|
The Python function implementing the node SHOULD share the same name as the node alias passed to `add_node`, so that graph definitions and stack traces remain unambiguous:
|
|
@@ -193,7 +193,7 @@ def route_after_evaluate_quality(state) -> str:
|
|
|
193
193
|
return "publish_step"
|
|
194
194
|
```
|
|
195
195
|
|
|
196
|
-
**Logging:** Log `verdict` and the count of each level as MLflow metrics on the current run per [agentme-edr-
|
|
196
|
+
**Logging:** Log `verdict` and the count of each level as MLflow metrics on the current run per [agentme-edr-144](144-ai-workflow-development-standards.md) rule `03-observability-and-experiment-tracking`.
|
|
197
197
|
|
|
198
198
|
#### 04-workflow-naming-conventions
|
|
199
199
|
|
|
@@ -276,6 +276,6 @@ def after_quality_check(state): ... # FORBIDDEN: not named "route_after_<node>"
|
|
|
276
276
|
|
|
277
277
|
## References
|
|
278
278
|
|
|
279
|
-
- [agentme-edr-
|
|
280
|
-
- [agentme-edr-
|
|
281
|
-
- [agentme-edr-
|
|
279
|
+
- [agentme-edr-144](144-ai-workflow-development-standards.md) — Workflow structure, LangGraph toolchain, observability, and testing patterns
|
|
280
|
+
- [agentme-edr-141](141-ai-llm-development-standards.md) — LLM development standards (drives `_llm` node suffix and mocking patterns)
|
|
281
|
+
- [agentme-edr-142](142-ai-agents-development-standards.md) — Agent development standards (drives `_agent` node suffix and state conventions)
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
2
|
+
name: agentme-edr-policy-146-ai-agent-xdrs-knowledge-layer
|
|
3
3
|
description: Defines how to integrate XDRS as the runtime knowledge source of truth for AI agents — covering document placement, AGENTS.md setup, file tools, and local sandbox configuration. Apply only when the project explicitly uses XDRS to govern agent behavior.
|
|
4
4
|
apply-to: AI agent projects that use XDRS as the source of truth for policies and skills
|
|
5
5
|
valid-from: 2026-05-27
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-146: AI agent XDRS knowledge layer
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -17,7 +17,7 @@ How should an AI agent project integrate XDRS as its runtime source of truth for
|
|
|
17
17
|
|
|
18
18
|
**Embed XDRS documents in `lib/data/.xdrs/`, instruct the agent to consult them via `AGENTS.md`, equip the agent with sandboxed file tools, and use the deepagents framework when a local sandbox is required.**
|
|
19
19
|
|
|
20
|
-
This policy MUST only be applied when the project explicitly chooses XDRS as its knowledge governance layer. It is not required by [agentme-edr-
|
|
20
|
+
This policy MUST only be applied when the project explicitly chooses XDRS as its knowledge governance layer. It is not required by [agentme-edr-142](142-ai-agents-development-standards.md) or [agentme-edr-144](144-ai-workflow-development-standards.md) in general.
|
|
21
21
|
|
|
22
22
|
### Details
|
|
23
23
|
|
|
@@ -59,7 +59,7 @@ Read /AGENTS.md and follow all instructions in it before proceeding.
|
|
|
59
59
|
|
|
60
60
|
#### 02-agent-file-tools
|
|
61
61
|
|
|
62
|
-
Every agent that uses the XDRS knowledge layer MUST use the file tools provided by the deepagents framework. Do not implement hand-rolled alternatives — see [agentme-edr-
|
|
62
|
+
Every agent that uses the XDRS knowledge layer MUST use the file tools provided by the deepagents framework. Do not implement hand-rolled alternatives — see [agentme-edr-142 rule 02-local-sandbox](142-ai-agents-development-standards.md) for the full sandbox and tool requirements.
|
|
63
63
|
|
|
64
64
|
These tools operate over two sandboxed roots (configured in rule `03-local-sandbox`):
|
|
65
65
|
|
|
@@ -72,7 +72,7 @@ These tools operate over two sandboxed roots (configured in rule `03-local-sandb
|
|
|
72
72
|
|
|
73
73
|
#### 03-local-sandbox
|
|
74
74
|
|
|
75
|
-
Follow [agentme-edr-
|
|
75
|
+
Follow [agentme-edr-142 rule 02-local-sandbox](142-ai-agents-development-standards.md) for the general deepagents sandbox setup. When XDRS is in use, add the following mounts to the sandbox configuration:
|
|
76
76
|
|
|
77
77
|
| Source | Content | Deepagents sandbox path |
|
|
78
78
|
|---|---|---|
|
|
@@ -91,8 +91,8 @@ data_root = str(files("myagent").joinpath("data"))
|
|
|
91
91
|
agents_md = Path(temp_root) / "AGENTS.md"
|
|
92
92
|
agents_md.write_text(_AGENTS_MD) # content from xdrs-core AGENTS.md template; see rule 01-xdrs-knowledge-layer
|
|
93
93
|
|
|
94
|
-
# Add these mounts alongside the base mounts from agentme-edr-
|
|
95
|
-
# (mount_paths uses {src: dst} dict format per agentme-edr-
|
|
94
|
+
# Add these mounts alongside the base mounts from agentme-edr-142 rule 02-local-sandbox:
|
|
95
|
+
# (mount_paths uses {src: dst} dict format per agentme-edr-142)
|
|
96
96
|
sandbox = Sandbox(
|
|
97
97
|
mount_paths={
|
|
98
98
|
tmp_dir: "/workspace",
|
package/.xdrs/agentme/edrs/application/{051-ai-eval-core-standards.md → 151-ai-eval-standards.md}
RENAMED
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
3
|
-
description: Defines the eval folder structure and Makefile interface for AI projects, and the LLM-as-judge binary scoring contract applicable to all AI tiers and test types. Use when scaffolding evals or implementing an LLM judge. For eval script requirements see agentme-edr-
|
|
2
|
+
name: agentme-edr-policy-151-ai-eval-standards
|
|
3
|
+
description: Defines the eval folder structure and Makefile interface for AI projects, and the LLM-as-judge binary scoring contract applicable to all AI tiers and test types. Use when scaffolding evals or implementing an LLM judge. For eval script requirements see agentme-edr-153. For report format see agentme-edr-154. For repeatability see agentme-edr-155. For when evals are required see agentme-edr-501 rule 09-ai-project-testing-requirements. For the test type taxonomy see agentme-edr-152.
|
|
4
4
|
apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
|
|
5
5
|
valid-from: 2026-06-05
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-151: AI eval standards
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -17,7 +17,7 @@ How should eval tests be structured, and how must LLM judges produce scores acro
|
|
|
17
17
|
|
|
18
18
|
**Use a per-component folder structure under `evals/` with a standardized Makefile interface, and require all LLM judges to produce binary (`0`/`1`) output compatible with classification metrics.**
|
|
19
19
|
|
|
20
|
-
For when evals are required per AI tier, see [agentme-edr-
|
|
20
|
+
For when evals are required per AI tier, see [agentme-edr-501](../governance/501-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
|
|
21
21
|
|
|
22
22
|
### Details
|
|
23
23
|
|
|
@@ -31,7 +31,7 @@ evals/
|
|
|
31
31
|
eval-<name>/
|
|
32
32
|
golden_dataset/ # EDR-024 + EDR-030 compliant golden dataset (README.md, dataset.schema.json, data/)
|
|
33
33
|
eval.py # evaluation script
|
|
34
|
-
report-<type>.md # generated report, one per evaluated test type (overwritten on each run — see agentme-edr-
|
|
34
|
+
report-<type>.md # generated report, one per evaluated test type (overwritten on each run — see agentme-edr-154 rule 01)
|
|
35
35
|
Makefile # lint, eval, run, and eval-<type> targets
|
|
36
36
|
eval-<name2>/
|
|
37
37
|
...
|
|
@@ -41,18 +41,18 @@ evals/
|
|
|
41
41
|
|
|
42
42
|
`<component>` MUST match the name of the component under evaluation and use lowercase hyphen-separated words (e.g., `workflow-document-review`, `agent-support`, `model-classifier`).
|
|
43
43
|
|
|
44
|
-
`<name>` identifies the specific evaluation scenario using lowercase hyphen-separated words (e.g., `eval-basic`, `eval-complex`, `eval-edge-cases`). A scenario's `golden_dataset` MAY mix multiple test types across its entries: label each entry with its applicable `test_types` ([agentme-edr-
|
|
44
|
+
`<name>` identifies the specific evaluation scenario using lowercase hyphen-separated words (e.g., `eval-basic`, `eval-complex`, `eval-edge-cases`). A scenario's `golden_dataset` MAY mix multiple test types across its entries: label each entry with its applicable `test_types` ([agentme-edr-152](152-ai-test-types-taxonomy.md) rule `04`) and use the `eval-<type>` targets below to run one type at a time.
|
|
45
45
|
|
|
46
|
-
The `golden_dataset/` subfolder MUST be a valid [agentme-edr-
|
|
46
|
+
The `golden_dataset/` subfolder MUST be a valid [agentme-edr-201](../data/201-ml-dataset-structure.md) dataset (`README.md`, `dataset.schema.json`, one JSON file per entry under `data/` per rule `04-complex-structured-datasets-must-use-per-entry-json-files`, lint-validated per rule `06`) whose entries follow the golden dataset envelope defined in [agentme-edr-152](152-ai-test-types-taxonomy.md) rule `02`.
|
|
47
47
|
|
|
48
48
|
Each `evals/<component>/eval-<name>/Makefile` MUST declare a `TEST_TYPES` variable listing the `test_types` values present in its golden dataset, and define:
|
|
49
49
|
|
|
50
50
|
| Target | Behaviour |
|
|
51
51
|
|---|---|
|
|
52
|
-
| `lint` | Validates every `golden_dataset/data/*.json` file against `golden_dataset/dataset.schema.json` per [agentme-edr-
|
|
52
|
+
| `lint` | Validates every `golden_dataset/data/*.json` file against `golden_dataset/dataset.schema.json` per [agentme-edr-201](../data/201-ml-dataset-structure.md) rule `06` |
|
|
53
53
|
| `eval` | Depends on `lint`; runs `eval.py --type=all` with threshold enforcement; exits non-zero on failure (CI-safe) |
|
|
54
54
|
| `run` | Depends on `lint`; runs `eval.py --type=all` with threshold enforcement; same as `eval` but intended for local exploration |
|
|
55
|
-
| `eval-<type>` | Depends on `lint`; runs `eval.py --type=<type>` for one declared test type, following [agentme-edr-
|
|
55
|
+
| `eval-<type>` | Depends on `lint`; runs `eval.py --type=<type>` for one declared test type, following [agentme-edr-303](../platform/303-common-targets.md) rule `03`'s `eval-<qualifier>` convention |
|
|
56
56
|
|
|
57
57
|
```makefile
|
|
58
58
|
TEST_TYPES := smoke functional safety
|
|
@@ -94,7 +94,7 @@ LLM judges scoring component outputs MUST produce binary output: `0` (fail) or `
|
|
|
94
94
|
- **F1 computation:** F1 MUST be computed from a binary confusion matrix treating score `1` as the positive class. In a standard golden dataset where every entry is expected to pass, all entries are true positives or false negatives (no true negatives exist), so Recall = TP / (TP + FN) and Precision = TP / (TP + FP) where FP = entries the judge scores `1` that the entry-level expected outcome marks as expected-fail. When all entries are expected-pass, Recall = Accuracy and Precision = 1, making F1 a conservative lower-bound on accuracy. Projects with adversarial or expected-fail entries MUST annotate each golden dataset entry with its expected binary outcome (`1` = should pass, `0` = should fail) in `expected_output` to enable a correct confusion matrix.
|
|
95
95
|
- Multi-class classification not supported. For multiple quality levels, use multiple binary judges (e.g., one for "factually correct", another for "tone appropriate")
|
|
96
96
|
|
|
97
|
-
**Rationale:** Binary output makes LLM judges compatible with classification metrics infrastructure (Accuracy, F1, Wilson CI, convergence analysis) defined in [agentme-edr-
|
|
97
|
+
**Rationale:** Binary output makes LLM judges compatible with classification metrics infrastructure (Accuracy, F1, Wilson CI, convergence analysis) defined in [agentme-edr-154](154-ai-eval-report-format.md) rule `01`.
|
|
98
98
|
|
|
99
99
|
**Example LLM judge prompt:**
|
|
100
100
|
|
|
@@ -112,10 +112,10 @@ Output:
|
|
|
112
112
|
|
|
113
113
|
## References
|
|
114
114
|
|
|
115
|
-
- [agentme-edr-
|
|
116
|
-
- [agentme-edr-
|
|
117
|
-
- [agentme-edr-
|
|
118
|
-
- [agentme-edr-
|
|
119
|
-
- [agentme-edr-
|
|
120
|
-
- [agentme-edr-
|
|
121
|
-
- [agentme-edr-
|
|
115
|
+
- [agentme-edr-153](153-ai-eval-script.md) — AI eval script: entry-first eval loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
|
|
116
|
+
- [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
|
|
117
|
+
- [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: loop exception, scoring methods (including LLM-as-judge per rule `02`), and cadence
|
|
118
|
+
- [agentme-edr-501](../governance/501-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09`) and statistical model eval targets (rule `07`)
|
|
119
|
+
- [agentme-edr-152](152-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum and golden dataset entry envelope
|
|
120
|
+
- [agentme-edr-201](../data/201-ml-dataset-structure.md) — ML dataset structure, per-entry JSON format, and schema-lint validation for golden datasets
|
|
121
|
+
- [agentme-edr-303](../platform/303-common-targets.md) — `eval-<qualifier>` Makefile convention (rule `03`)
|