agentme 0.26.0 → 0.28.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.filedist-package.yml +1 -1
- package/.xdrs/agentme/edrs/{principles → application}/002-coding-best-practices.md +1 -10
- package/.xdrs/agentme/edrs/application/003-javascript-project-tooling.md +3 -3
- package/.xdrs/agentme/edrs/application/010-golang-project-tooling.md +1 -1
- package/.xdrs/agentme/edrs/application/014-python-project-tooling.md +7 -7
- package/.xdrs/agentme/edrs/application/015-cli-tool-standards.md +5 -5
- package/.xdrs/agentme/edrs/{principles → application}/022-secrets-management.md +1 -1
- package/.xdrs/agentme/edrs/application/026-pragmatic-hexagonal-architecture.md +6 -6
- package/.xdrs/agentme/edrs/application/{018-ai-llm-development-standards.md → 040-ai-llm-development-standards.md} +13 -11
- package/.xdrs/agentme/edrs/application/{019-ai-agents-development-standards.md → 041-ai-agents-development-standards.md} +10 -10
- package/.xdrs/agentme/edrs/application/{020-ai-agents-quality-standards.md → 042-ai-agents-quality-standards.md} +14 -12
- package/.xdrs/agentme/edrs/application/{021-ai-workflow-development-standards.md → 043-ai-workflow-development-standards.md} +25 -22
- package/.xdrs/agentme/edrs/application/{029-ai-workflow-naming-conventions.md → 044-ai-workflow-naming-conventions.md} +9 -9
- package/.xdrs/agentme/edrs/application/{025-ai-agent-xdrs-knowledge-layer.md → 045-ai-agent-xdrs-knowledge-layer.md} +7 -7
- package/.xdrs/agentme/edrs/application/051-ai-eval-core-standards.md +121 -0
- package/.xdrs/agentme/edrs/application/052-ai-test-types-taxonomy.md +116 -0
- package/.xdrs/agentme/edrs/application/053-ai-eval-script.md +136 -0
- package/.xdrs/agentme/edrs/application/054-ai-eval-report-format.md +171 -0
- package/.xdrs/agentme/edrs/application/055-ai-eval-repeatability.md +75 -0
- package/.xdrs/agentme/edrs/application/skills/004-select-relevant-xdrs/SKILL.md +7 -7
- package/.xdrs/agentme/edrs/{application/024-ml-dataset-structure.md → data/050-ml-dataset-structure.md} +4 -4
- package/.xdrs/agentme/edrs/{principles → governance}/007-project-quality-standards.md +26 -21
- package/.xdrs/agentme/edrs/governance/013-contributing-guide-requirements.md +2 -2
- package/.xdrs/agentme/edrs/index.md +46 -25
- package/.xdrs/agentme/edrs/{devops → platform}/005-monorepo-structure.md +2 -0
- package/.xdrs/agentme/edrs/{devops → platform}/008-common-targets.md +13 -13
- package/.xdrs/agentme/edrs/{devops → platform}/027-environment-variable-configuration.md +2 -2
- package/.xdrs/agentme/edrs/principles/016-cross-language-module-structure.md +4 -4
- package/package.json +2 -2
- package/.xdrs/agentme/edrs/application/028-ai-eval-standards.md +0 -257
- package/.xdrs/agentme/edrs/application/030-ai-test-types-taxonomy.md +0 -98
- /package/.xdrs/agentme/edrs/{principles → application}/004-unit-test-requirements.md +0 -0
- /package/.xdrs/agentme/edrs/{principles → application}/009-error-handling.md +0 -0
- /package/.xdrs/agentme/edrs/{principles → application}/023-coding-abstraction-practices.md +0 -0
- /package/.xdrs/agentme/edrs/{observability → operations}/011-service-health-check-endpoint.md +0 -0
- /package/.xdrs/agentme/edrs/{devops → platform}/006-github-pipelines.md +0 -0
- /package/.xdrs/agentme/edrs/{devops → platform}/017-tool-execution-and-scripting.md +0 -0
- /package/.xdrs/agentme/edrs/{devops → platform}/skills/002-monorepo-setup/SKILL.md +0 -0
|
@@ -61,8 +61,8 @@ extraction.
|
|
|
61
61
|
2. Reduce that set only by excluding the XDRs that clearly do not fit the repository. Use
|
|
62
62
|
path-stable identifiers so the extraction command is auditable, for example:
|
|
63
63
|
- `.xdrs/agentme/edrs/application/010-golang-project-tooling.md` for non-Go projects
|
|
64
|
-
- `.xdrs/agentme/edrs/
|
|
65
|
-
- `.xdrs/agentme/edrs/
|
|
64
|
+
- `.xdrs/agentme/edrs/platform/005-monorepo-structure.md` for non-monorepos
|
|
65
|
+
- `.xdrs/agentme/edrs/operations/011-service-health-check-endpoint.md` for projects without
|
|
66
66
|
a long-running service surface
|
|
67
67
|
3. If the repository does not want agentme XDRs, stop and explain why instead of forcing an
|
|
68
68
|
installation.
|
|
@@ -109,8 +109,8 @@ extraction.
|
|
|
109
109
|
Input: "Install the right agentme XDR presets for this Node.js library"
|
|
110
110
|
- Inventory the shipped agentme XDR files
|
|
111
111
|
- Analyze the repository and detect a JavaScript library with Makefiles
|
|
112
|
-
- Exclude `.xdrs/agentme/edrs/application/010-golang-project-tooling.md` and `.xdrs/agentme/edrs/
|
|
113
|
-
- Run `npx -y agentme extract --output . --all --exclude .xdrs/agentme/edrs/application/010-golang-project-tooling.md --exclude .xdrs/agentme/edrs/
|
|
112
|
+
- Exclude `.xdrs/agentme/edrs/application/010-golang-project-tooling.md` and `.xdrs/agentme/edrs/operations/011-service-health-check-endpoint.md`
|
|
113
|
+
- Run `npx -y agentme extract --output . --all --exclude .xdrs/agentme/edrs/application/010-golang-project-tooling.md --exclude .xdrs/agentme/edrs/operations/011-service-health-check-endpoint.md`
|
|
114
114
|
|
|
115
115
|
Input: "Set up agentme for this repo"
|
|
116
116
|
- Inventory the shipped agentme XDR files
|
|
@@ -135,7 +135,7 @@ Input: "Set up agentme for this repo"
|
|
|
135
135
|
|
|
136
136
|
- [agentme README](../../../../../../README.md)
|
|
137
137
|
- [agentme-edr-003 - JavaScript project tooling and structure](../../003-javascript-project-tooling.md)
|
|
138
|
-
- [agentme-edr-005 - Monorepo structure](../../../
|
|
139
|
-
- [agentme-edr-007 - Project quality standards](../../../
|
|
140
|
-
- [agentme-edr-008 - Common development script names](../../../
|
|
138
|
+
- [agentme-edr-005 - Monorepo structure](../../../platform/005-monorepo-structure.md)
|
|
139
|
+
- [agentme-edr-007 - Project quality standards](../../../governance/007-project-quality-standards.md)
|
|
140
|
+
- [agentme-edr-008 - Common development script names](../../../platform/008-common-targets.md)
|
|
141
141
|
- [_core-adr-003 - Skill standards](../../../../../_core/adrs/principles/003-skill-standards.md)
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
2
|
+
name: agentme-edr-policy-050-ml-dataset-structure
|
|
3
3
|
description: Defines the standard folder layout and file conventions for ML datasets used in AI/ML projects. Use when creating, organizing, or consuming datasets for machine learning tasks such as image labeling, document extraction, tabular data, LLM evaluation, and Q&A sets.
|
|
4
4
|
apply-to: ML and AI projects that produce or consume datasets
|
|
5
5
|
valid-from: 2026-05-27
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-050: ML dataset structure
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -102,10 +102,10 @@ Every dataset MUST expose a `make lint` target (in the Makefile of the project/c
|
|
|
102
102
|
- Per-entry JSON files (rule `04`) and annotation-pair `.json` siblings (rule `02`) MUST each be validated against `dataset.schema.json`, and their `$schema` property MUST be present and resolve to the dataset's actual schema file.
|
|
103
103
|
- CSV rows (rule `03`) MUST each be converted to a JSON object (column header → value) and validated against the same `dataset.schema.json`.
|
|
104
104
|
- `make lint` MUST list every violation found across all files/rows before exiting with a non-zero status (not fail-fast on the first violation).
|
|
105
|
-
- `jsonschema` MUST be declared as a normal project dependency per [agentme-edr-014](014-python-project-tooling.md); no special-casing.
|
|
105
|
+
- `jsonschema` MUST be declared as a normal project dependency per [agentme-edr-014](../application/014-python-project-tooling.md); no special-casing.
|
|
106
106
|
|
|
107
107
|
## References
|
|
108
108
|
|
|
109
109
|
- [JSON Schema specification](https://json-schema.org/)
|
|
110
110
|
- [jsonschema (Python library)](https://pypi.org/project/jsonschema/)
|
|
111
|
-
- [agentme-edr-014](014-python-project-tooling.md) — Python project tooling and dependency conventions
|
|
111
|
+
- [agentme-edr-014](../application/014-python-project-tooling.md) — Python project tooling and dependency conventions
|
|
@@ -61,7 +61,7 @@ A unit test suite MUST run automatically before every release. Failing tests MUS
|
|
|
61
61
|
|
|
62
62
|
**Exception:** Projects with fewer than 100 lines of code, or whose `README.md` prominently marks them as a **Spike** or **Experiment**, are exempt from this requirement. Such projects MUST NOT be deployed to production.
|
|
63
63
|
|
|
64
|
-
**Reference:** [agentme-edr-004](004-unit-test-requirements.md) for detailed unit test requirements.
|
|
64
|
+
**Reference:** [agentme-edr-004](../application/004-unit-test-requirements.md) for detailed unit test requirements.
|
|
65
65
|
|
|
66
66
|
---
|
|
67
67
|
|
|
@@ -156,7 +156,7 @@ test-examples:
|
|
|
156
156
|
$(MAKE) -C examples
|
|
157
157
|
```
|
|
158
158
|
|
|
159
|
-
If examples require live services or credentials, remove `test-examples` from the `test` dependency list and keep it as a standalone named target only. See [agentme-edr-008](../
|
|
159
|
+
If examples require live services or credentials, remove `test-examples` from the `test` dependency list and keep it as a standalone named target only. See [agentme-edr-008](../platform/008-common-targets.md) rule 08 for the full offline/online decision table.
|
|
160
160
|
|
|
161
161
|
**Examples Makefile:**
|
|
162
162
|
|
|
@@ -175,23 +175,28 @@ Projects that contain statistical models (e.g., ML models, LLM-based evaluators,
|
|
|
175
175
|
**Requirements:**
|
|
176
176
|
- A `make eval` target MUST exist and execute all performance evaluations
|
|
177
177
|
- Each evaluation MUST have a **documented minimum performance threshold** (e.g., accuracy ≥ 0.85, F1 ≥ 0.80, BLEU ≥ 0.70)
|
|
178
|
-
- Thresholds MUST be declared
|
|
178
|
+
- Thresholds and all scoring parameters MUST be declared as constants in `eval.py` — they are design decisions about acceptable quality for the component under test, not runtime configuration, and MUST NOT be passed as Makefile variables or CLI flags. See [agentme-edr-053](../application/053-ai-eval-script.md) rule `01`.
|
|
179
179
|
- `make eval` MUST **exit with a non-zero status** (fail) if:
|
|
180
180
|
- The evaluation cannot be executed (missing data, environment errors, model load failures)
|
|
181
181
|
- Any metric falls below its defined minimum threshold
|
|
182
182
|
- CI/CD MUST invoke `make eval` before releasing any version that changes model weights, prompts, or evaluation logic
|
|
183
183
|
|
|
184
|
-
**Threshold declaration example (
|
|
184
|
+
**Threshold declaration example (eval.py):**
|
|
185
185
|
|
|
186
|
-
```
|
|
187
|
-
EVAL_MIN_ACCURACY
|
|
188
|
-
EVAL_MIN_F1
|
|
186
|
+
```python
|
|
187
|
+
EVAL_MIN_ACCURACY = {"functional": 0.85, "smoke": 0.90}
|
|
188
|
+
EVAL_MIN_F1 = {"functional": 0.80}
|
|
189
|
+
|
|
190
|
+
# Thresholds are declared here and enforced by the script:
|
|
191
|
+
if accuracy < EVAL_MIN_ACCURACY.get(test_type, 0):
|
|
192
|
+
raise SystemExit(f"Eval failed: {test_type} accuracy {accuracy:.2f} < {EVAL_MIN_ACCURACY[test_type]}")
|
|
193
|
+
```
|
|
189
194
|
|
|
195
|
+
**Makefile target (delegates to eval.py — no threshold variables here):**
|
|
196
|
+
|
|
197
|
+
```makefile
|
|
190
198
|
eval:
|
|
191
|
-
python eval.py
|
|
192
|
-
--min-accuracy $(EVAL_MIN_ACCURACY) \
|
|
193
|
-
--min-f1 $(EVAL_MIN_F1) \
|
|
194
|
-
|| (echo "Evaluation failed: metrics below threshold"; exit 1)
|
|
199
|
+
mise exec -- uv run --project . python eval.py --type=all
|
|
195
200
|
```
|
|
196
201
|
|
|
197
202
|
---
|
|
@@ -239,27 +244,27 @@ Projects are not required to implement integration tests, but when present, they
|
|
|
239
244
|
|
|
240
245
|
#### 09-ai-project-testing-requirements
|
|
241
246
|
|
|
242
|
-
AI projects are classified into three tiers — LLM, Agent, and Workflow — defined in [agentme-edr-
|
|
247
|
+
AI projects are classified into three tiers — LLM, Agent, and Workflow — defined in [agentme-edr-040](../application/040-ai-llm-development-standards.md). Testing requirements differ per tier:
|
|
243
248
|
|
|
244
249
|
| Tier | Unit tests | Evals | Integration tests |
|
|
245
250
|
|---|---|---|---|
|
|
246
|
-
| **LLM** ([agentme-edr-
|
|
247
|
-
| **Agent** ([agentme-edr-
|
|
248
|
-
| **Workflow** ([agentme-edr-
|
|
251
|
+
| **LLM** ([agentme-edr-040](../application/040-ai-llm-development-standards.md)) | Not required | Not required; SHOULD be used when critical prompts are in use to measure accuracy and detect model drift | Not required |
|
|
252
|
+
| **Agent** ([agentme-edr-041](../application/041-ai-agents-development-standards.md)) | Not required | Not required; MAY be used | Not required |
|
|
253
|
+
| **Workflow** ([agentme-edr-043](../application/043-ai-workflow-development-standards.md)) | Required — see below | Required before every release; failed evals block release | Advised |
|
|
249
254
|
|
|
250
255
|
**Workflow unit test requirements:**
|
|
251
256
|
|
|
252
|
-
- MUST use mocked LLM providers. See [agentme-edr-
|
|
253
|
-
- MUST run offline with no external dependencies per [agentme-edr-004](004-unit-test-requirements.md) rule `02-must-run-offline`.
|
|
254
|
-
- MUST achieve 80% code coverage per [agentme-edr-004](004-unit-test-requirements.md) rule `03-must-maintain-80-percent-coverage`.
|
|
257
|
+
- MUST use mocked LLM providers. See [agentme-edr-040](../application/040-ai-llm-development-standards.md) rule `04-unit-test-mocking` for the mocking pattern.
|
|
258
|
+
- MUST run offline with no external dependencies per [agentme-edr-004](../application/004-unit-test-requirements.md) rule `02-must-run-offline`.
|
|
259
|
+
- MUST achieve 80% code coverage per [agentme-edr-004](../application/004-unit-test-requirements.md) rule `03-must-maintain-80-percent-coverage`.
|
|
255
260
|
- MUST test workflow routing logic, conditional edges, state transformations, and error handling.
|
|
256
261
|
- MUST achieve **80% coverage of LangGraph graph edges and branches**: every conditional edge MUST have test cases covering each possible branch, and every node→node transition MUST be exercised by at least one test.
|
|
257
|
-
- Files MUST be named `<name>_test.py` and placed alongside the source file per [agentme-edr-004](004-unit-test-requirements.md) rule `04-must-place-test-files-alongside-source`.
|
|
262
|
+
- Files MUST be named `<name>_test.py` and placed alongside the source file per [agentme-edr-004](../application/004-unit-test-requirements.md) rule `04-must-place-test-files-alongside-source`.
|
|
258
263
|
|
|
259
264
|
**Workflow eval requirements:**
|
|
260
265
|
|
|
261
266
|
- Evals MUST be executed before every release.
|
|
262
267
|
- Accuracy below project-defined thresholds MUST block the release. Thresholds MUST be documented in the eval Makefile or README.
|
|
263
268
|
- Evals MUST run against real LLM providers (not mocks) to capture model drift.
|
|
264
|
-
- For eval folder structure and script requirements, see [agentme-edr-
|
|
265
|
-
- For the taxonomy of AI test types (safety, responsible-AI, quality-eval, prompt, code-level) and the golden dataset entry format, see [agentme-edr-
|
|
269
|
+
- For eval folder structure and LLM-as-judge scoring, see [agentme-edr-051](../application/051-ai-eval-core-standards.md). For eval script requirements, see [agentme-edr-053](../application/053-ai-eval-script.md).
|
|
270
|
+
- For the taxonomy of AI test types (safety, responsible-AI, quality-eval, prompt, code-level) and the golden dataset entry format, see [agentme-edr-052](../application/052-ai-test-types-taxonomy.md).
|
|
@@ -66,5 +66,5 @@ The content SHOULD stay concise and practical; do not turn `CONTRIBUTING.md` int
|
|
|
66
66
|
|
|
67
67
|
## References
|
|
68
68
|
|
|
69
|
-
- [agentme-edr-005 - Monorepo structure](../
|
|
70
|
-
- [002-monorepo-setup skill](../
|
|
69
|
+
- [agentme-edr-005 - Monorepo structure](../platform/005-monorepo-structure.md)
|
|
70
|
+
- [002-monorepo-setup skill](../platform/skills/002-monorepo-setup/SKILL.md)
|
|
@@ -8,14 +8,8 @@ Propose changes via pull request. All changes must be verified for clarity and n
|
|
|
8
8
|
|
|
9
9
|
Foundational standards, principles, and guidelines.
|
|
10
10
|
|
|
11
|
-
- [agentme-edr-002](principles/002-coding-best-practices.md) - **Coding best practices** - Keep files small, tests nearby, and docs synchronized
|
|
12
|
-
- [agentme-edr-023](principles/023-coding-abstraction-practices.md) - **Coding abstraction practices** - Define when abstractions are justified and when they must be inlined
|
|
13
|
-
- [agentme-edr-004](principles/004-unit-test-requirements.md) - **Unit test requirements** - Define minimum unit-test coverage and naming expectations
|
|
14
|
-
- [agentme-edr-007](principles/007-project-quality-standards.md) - **Project quality standards** - Require build, lint, and test verification before completion
|
|
15
|
-
- [agentme-edr-009](principles/009-error-handling.md) - **Error handling** - Standardize explicit errors, logging, and propagation rules
|
|
16
11
|
- [agentme-edr-012](principles/012-continuous-xdr-enrichment.md) - **Continuous xdr improvement policy** - Promote recurring delivery lessons into reusable XDRs
|
|
17
12
|
- [agentme-edr-016](principles/016-cross-language-module-structure.md) - **Cross-language module structure** - Organize modules consistently across supported languages
|
|
18
|
-
- [agentme-edr-022](principles/022-secrets-management.md) - **Secrets management** - Handle secrets securely using native keychains and cloud secret managers
|
|
19
13
|
|
|
20
14
|
## Articles
|
|
21
15
|
|
|
@@ -27,40 +21,67 @@ Synthetic views combining agentme XDRs and skills around a specific topic.
|
|
|
27
21
|
|
|
28
22
|
Language and framework-specific tooling and project structure.
|
|
29
23
|
|
|
24
|
+
- [agentme-edr-002](application/002-coding-best-practices.md) - **Coding best practices** - Keep files small, tests nearby, and docs synchronized
|
|
25
|
+
- [agentme-edr-004](application/004-unit-test-requirements.md) - **Unit test requirements** - Define minimum unit-test coverage and naming expectations
|
|
26
|
+
- [agentme-edr-009](application/009-error-handling.md) - **Error handling** - Standardize explicit errors, logging, and propagation rules
|
|
27
|
+
- [agentme-edr-022](application/022-secrets-management.md) - **Secrets management** - Handle secrets securely using native keychains and cloud secret managers
|
|
28
|
+
- [agentme-edr-023](application/023-coding-abstraction-practices.md) - **Coding abstraction practices** - Define when abstractions are justified and when they must be inlined
|
|
29
|
+
|
|
30
|
+
### Language and framework tooling
|
|
31
|
+
|
|
30
32
|
- [agentme-edr-003](application/003-javascript-project-tooling.md) - **JavaScript project tooling and structure** - Scaffold JavaScript libraries with the standard toolchain *(includes skill: [001-create-javascript-project](application/skills/001-create-javascript-project/SKILL.md))*
|
|
31
33
|
- [agentme-edr-010](application/010-golang-project-tooling.md) - **Go project tooling and structure** - Scaffold Go CLIs and libraries with the standard layout *(includes skill: [003-create-golang-project](application/skills/003-create-golang-project/SKILL.md))*
|
|
32
34
|
- [agentme-edr-014](application/014-python-project-tooling.md) - **Python project tooling and structure** - Scaffold Python packages and CLIs with the standard layout *(includes skill: [005-create-python-project](application/skills/005-create-python-project/SKILL.md))*
|
|
33
35
|
- [agentme-edr-015](application/015-cli-tool-standards.md) - **CLI tool standards** - Define command UX and behavior for CLI tools
|
|
34
|
-
- [agentme-edr-018](application/018-ai-llm-development-standards.md) - **AI LLM development standards** - Standard framework (LangChain) and patterns for simple LLM calls with explicit configuration (no environment variables)
|
|
35
|
-
- [agentme-edr-019](application/019-ai-agents-development-standards.md) - **AI agents development standards** - Structural patterns for agents: framework selection, sandbox setup, naming conventions, composition, and system prompt structure
|
|
36
|
-
- [agentme-edr-020](application/020-ai-agents-quality-standards.md) - **AI agents implementation quality standards** - Tool definition patterns, error handling, observability, and unit testing for agents
|
|
37
|
-
- [agentme-edr-021](application/021-ai-workflow-development-standards.md) - **AI workflow development standards** - Standard toolchain (LangGraph), evaluation, and testing patterns for workflow projects
|
|
38
|
-
- [agentme-edr-029](application/029-ai-workflow-naming-conventions.md) - **AI workflow naming conventions** - Node suffix/prefix roles, state type and attribute naming, judge output schema, workflow class names, and cross-element coherence rules
|
|
39
|
-
- [agentme-edr-028](application/028-ai-eval-standards.md) - **AI eval standards** - Folder structure, script requirements, and MLflow tracking for eval tests across LLM, Agent, and Workflow tiers
|
|
40
|
-
- [agentme-edr-030](application/030-ai-test-types-taxonomy.md) - **AI test types taxonomy** - Names AI test types (safety, responsible-AI, quality-eval, prompt, code-level) with group, objective, mocking constraint, and relevance, and defines the shared golden dataset entry envelope
|
|
41
|
-
- [agentme-edr-024](application/024-ml-dataset-structure.md) - **ML dataset structure** - Standard folder layout and file conventions for ML datasets
|
|
42
|
-
- [agentme-edr-025](application/025-ai-agent-xdrs-knowledge-layer.md) - **AI agent XDRS knowledge layer** - How to integrate XDRS as the runtime source of truth for policies and skills in AI agents (apply only when the project explicitly uses XDRS)
|
|
43
36
|
- [agentme-edr-026](application/026-pragmatic-hexagonal-architecture.md) - **Pragmatic hexagonal architecture** - Organize application layers as External/Adapters/Application with practical coupling rules
|
|
44
37
|
- [004-select-relevant-xdrs](application/skills/004-select-relevant-xdrs/SKILL.md) - **Select relevant XDRs**
|
|
45
38
|
|
|
46
|
-
|
|
39
|
+
### AI development
|
|
40
|
+
|
|
41
|
+
Standards for building LLM, Agent, and Workflow components.
|
|
42
|
+
|
|
43
|
+
- [agentme-edr-040](application/040-ai-llm-development-standards.md) - **AI LLM development standards** - Standard framework (LangChain) and patterns for simple LLM calls with explicit configuration (no environment variables)
|
|
44
|
+
- [agentme-edr-041](application/041-ai-agents-development-standards.md) - **AI agents development standards** - Structural patterns for agents: framework selection, sandbox setup, naming conventions, composition, and system prompt structure
|
|
45
|
+
- [agentme-edr-042](application/042-ai-agents-quality-standards.md) - **AI agents implementation quality standards** - Tool definition patterns, error handling, observability, and unit testing for agents
|
|
46
|
+
- [agentme-edr-043](application/043-ai-workflow-development-standards.md) - **AI workflow development standards** - Standard toolchain (LangGraph), evaluation, and testing patterns for workflow projects
|
|
47
|
+
- [agentme-edr-044](application/044-ai-workflow-naming-conventions.md) - **AI workflow naming conventions** - Node suffix/prefix roles, state type and attribute naming, judge output schema, workflow class names, and cross-element coherence rules
|
|
48
|
+
- [agentme-edr-045](application/045-ai-agent-xdrs-knowledge-layer.md) - **AI agent XDRS knowledge layer** - How to integrate XDRS as the runtime source of truth for policies and skills in AI agents (apply only when the project explicitly uses XDRS)
|
|
49
|
+
|
|
50
|
+
### AI evaluation and testing
|
|
51
|
+
|
|
52
|
+
Standards for eval datasets, scripts, reports, and test type taxonomy.
|
|
53
|
+
|
|
54
|
+
- [agentme-edr-051](application/051-ai-eval-core-standards.md) - **AI eval core standards** - Eval folder structure and Makefile interface; LLM-as-judge binary scoring contract applicable to all AI tiers and test types
|
|
55
|
+
- [agentme-edr-052](application/052-ai-test-types-taxonomy.md) - **AI test types taxonomy** - Names AI test types (`functional`, `safety`, `smoke`, `repeatability`, `adversarial`, `fairness`, `bias`, and 5 others) with group, objective, mocking constraint, and relevance, and defines the shared golden dataset entry envelope
|
|
56
|
+
- [agentme-edr-053](application/053-ai-eval-script.md) - **AI eval script** - eval.py requirements: entry-first loop, --type filtering, mock_fixtures wiring, human entries, threshold enforcement, and MLflow experiment conventions
|
|
57
|
+
- [agentme-edr-054](application/054-ai-eval-report-format.md) - **AI eval report format** - report-<type>.md template, Wilson score confidence interval, convergence analysis, and human-type checklist artifact
|
|
58
|
+
- [agentme-edr-055](application/055-ai-eval-repeatability.md) - **AI eval repeatability** - Repeatability test type: REPEAT_COUNT loop exception, semantic-similarity and LLM-as-judge scoring, repeatability_accuracy metric, report shape, and run cadence
|
|
59
|
+
|
|
60
|
+
## Data
|
|
61
|
+
|
|
62
|
+
Data layer implementation and data management decisions.
|
|
63
|
+
|
|
64
|
+
- [agentme-edr-050](data/050-ml-dataset-structure.md) - **ML dataset structure** - Standard folder layout and file conventions for ML datasets
|
|
65
|
+
|
|
66
|
+
## Platform
|
|
47
67
|
|
|
48
|
-
|
|
68
|
+
Infrastructure implementation, delivery pipeline, and developer environment decisions.
|
|
49
69
|
|
|
50
|
-
- [agentme-edr-005](
|
|
51
|
-
- [agentme-edr-006](
|
|
52
|
-
- [agentme-edr-008](
|
|
53
|
-
- [agentme-edr-017](
|
|
54
|
-
- [agentme-edr-027](
|
|
70
|
+
- [agentme-edr-005](platform/005-monorepo-structure.md) - **Monorepo structure** - Standardize monorepo layout, tooling, and package boundaries *(includes skill: [002-monorepo-setup](platform/skills/002-monorepo-setup/SKILL.md))*
|
|
71
|
+
- [agentme-edr-006](platform/006-github-pipelines.md) - **GitHub CI/CD pipelines** - Define required CI stages and workflow structure
|
|
72
|
+
- [agentme-edr-008](platform/008-common-targets.md) - **Common development script names** - Reuse standard build, lint, and test target names
|
|
73
|
+
- [agentme-edr-017](platform/017-tool-execution-and-scripting.md) - **Tool execution and scripting** - Run tools consistently across shells, Makefiles, and CI
|
|
74
|
+
- [agentme-edr-027](platform/027-environment-variable-configuration.md) - **Environment variable configuration files** - Manage non-secret configuration with `.env` files, `.gitignore` rules, stage variants, and Makefile loading
|
|
55
75
|
|
|
56
76
|
## Governance
|
|
57
77
|
|
|
58
78
|
Contribution and collaboration standards shared across projects.
|
|
59
79
|
|
|
80
|
+
- [agentme-edr-007](governance/007-project-quality-standards.md) - **Project quality standards** - Require build, lint, and test verification before completion
|
|
60
81
|
- [agentme-edr-013](governance/013-contributing-guide-requirements.md) - **Contributing guide requirements** - Define the minimum structure for CONTRIBUTING guides
|
|
61
82
|
|
|
62
|
-
##
|
|
83
|
+
## Operations
|
|
63
84
|
|
|
64
|
-
|
|
85
|
+
Production behavior and operational response decisions.
|
|
65
86
|
|
|
66
|
-
- [agentme-edr-011](
|
|
87
|
+
- [agentme-edr-011](operations/011-service-health-check-endpoint.md) - **Service health check endpoint** - Expose a standard runtime health-check endpoint for services
|
|
@@ -24,6 +24,8 @@ Module folder responsibilities, artifact locations, and test-folder conventions
|
|
|
24
24
|
|
|
25
25
|
#### 01-top-level-directory-layout
|
|
26
26
|
|
|
27
|
+
Every monorepo MUST follow this top-level directory layout:
|
|
28
|
+
|
|
27
29
|
```
|
|
28
30
|
/
|
|
29
31
|
├── .cache/ # Optional shared cache for repo-level tooling
|
|
@@ -73,10 +73,10 @@ Targets are organized into five lifecycle groups. Projects MUST use these names
|
|
|
73
73
|
| Target | Purpose |
|
|
74
74
|
|--------|---------|
|
|
75
75
|
| `setup` | Run `mise install` and any small project bootstrap needed before normal targets work. This is the first command after checkout. |
|
|
76
|
-
| `all` | Alias that runs `build`, `lint`, and `test` in sequence.
|
|
76
|
+
| `all` | Alias that runs `build`, `lint`, and `test` in sequence. MUST be the default target (i.e., running `make` or the runner with no arguments invokes `all`). Used by developers as a fast pre-push check to verify the software meets minimum quality standards in one command. MUST only invoke targets that run **offline** — no external credentials, running servers, paid APIs, or environment-specific configuration outside the repository. |
|
|
77
77
|
| `clean` | Remove all temporary or generated files created during build, lint, or test (e.g., `node_modules`, virtual environments, compiled binaries, generated files). Used both locally and in CI for a clean slate. |
|
|
78
78
|
| `dev` | Run the software locally for development (e.g., start a Node.js API server, open a Jupyter notebook, launch a React dev server). May have debugging tools, verbose logging, or hot reloading features enabled. |
|
|
79
|
-
| `run` | Run the software in production mode (e.g., start a compiled binary, launch a production server). Debugging or development-only features
|
|
79
|
+
| `run` | Run the software in production mode (e.g., start a compiled binary, launch a production server). Debugging or development-only features SHOULD NOT be enabled. |
|
|
80
80
|
| `update-lockfile` | Update the dependency lockfile to reflect the latest resolved versions of all dependencies. |
|
|
81
81
|
|
|
82
82
|
##### Build group
|
|
@@ -93,17 +93,17 @@ Targets are organized into five lifecycle groups. Projects MUST use these names
|
|
|
93
93
|
|
|
94
94
|
| Target | Purpose |
|
|
95
95
|
|--------|---------|
|
|
96
|
-
| `lint` | Run **all static quality checks** outside of tests. This MUST include: code formatting validation, code style enforcement, code smell detection, static analysis, dependency audits for known CVEs, security vulnerability scans (e.g., SAST), and project/configuration structure checks. All checks
|
|
96
|
+
| `lint` | Run **all static quality checks** outside of tests. This MUST include: code formatting validation, code style enforcement, code smell detection, static analysis, dependency audits for known CVEs, security vulnerability scans (e.g., SAST), and project/configuration structure checks. All checks MUST be non-destructive (read-only); fixes are handled by `lint-fix`. MUST only invoke subtargets that run **offline** (no external credentials or services). |
|
|
97
97
|
| `lint-fix` | Automatically fix linting and formatting issues where possible. || `lint-format` | *(Optional)* Check code formatting only (e.g., Prettier, gofmt, Black). |
|
|
98
98
|
##### Test group
|
|
99
99
|
|
|
100
100
|
| Target | Purpose |
|
|
101
101
|
|--------|---------|
|
|
102
|
-
| `test` | Run **all offline tests** required for the project. This MUST include unit tests (with coverage enforcement — the build MUST fail if coverage thresholds are not met) and any integration or end-to-end tests that run **offline** (no external servers, credentials, or paid APIs). Normally delegates to `test-unit` and, when offline, `test-integration` in sequence. Suffixed targets that require external dependencies
|
|
102
|
+
| `test` | Run **all offline tests** required for the project. This MUST include unit tests (with coverage enforcement — the build MUST fail if coverage thresholds are not met) and any integration or end-to-end tests that run **offline** (no external servers, credentials, or paid APIs). Normally delegates to `test-unit` and, when offline, `test-integration` in sequence. Suffixed targets that require external dependencies MUST NOT be invoked automatically — see rule 08. |
|
|
103
103
|
| `test-unit` | Run unit tests only, including coverage report generation and coverage threshold enforcement. |
|
|
104
104
|
| `test-integration` | *(Optional)* Run integration and end-to-end tests only. Projects without integration tests MAY omit this target. |
|
|
105
105
|
| `test-smoke` | *(Optional)* Run a fast, minimal subset of tests to verify the software is basically functional. Useful as a post-deploy health check. |
|
|
106
|
-
| `eval` | *(Optional)* Run **all evaluations** for the module. Used alongside `test` to measure the accuracy and performance of statistical systems such as ML models, AI agents, or noisy systems. Typically runs against a live or near-live system (similar to an integration test) and produces a performance analysis report (e.g., F1 score, Accuracy, Precision, Recall).
|
|
106
|
+
| `eval` | *(Optional)* Run **all evaluations** for the module. Used alongside `test` to measure the accuracy and performance of statistical systems such as ML models, AI agents, or noisy systems. Typically runs against a live or near-live system (similar to an integration test) and produces a performance analysis report (e.g., F1 score, Accuracy, Precision, Recall). MUST NOT be included in `test` or `all` — evals are opt-in because they require live dependencies and MAY be slow or costly to run. Individual evaluations MUST follow the prefix convention: `eval-<qualifier>` (e.g., `eval-simple`, `eval-complex`). |
|
|
107
107
|
|
|
108
108
|
##### Release group
|
|
109
109
|
|
|
@@ -119,18 +119,18 @@ Targets are organized into five lifecycle groups. Projects MUST use these names
|
|
|
119
119
|
|
|
120
120
|
#### 04-standard-environment-variables
|
|
121
121
|
|
|
122
|
-
Two environment variables have defined semantics and
|
|
122
|
+
Two environment variables have defined semantics and MUST be used consistently.
|
|
123
123
|
|
|
124
124
|
| Variable | Purpose |
|
|
125
125
|
|----------|---------|
|
|
126
|
-
| `STAGE` | Identifies the runtime environment. Format: `[prefix][-variant]`. Common prefixes: `dev`, `tst`, `acc`, `prd`. Examples: `dev`, `dev-pr123`, `tst`, `prd-blue`.
|
|
126
|
+
| `STAGE` | Identifies the runtime environment. Format: `[prefix][-variant]`. Common prefixes: `dev`, `tst`, `acc`, `prd`. Examples: `dev`, `dev-pr123`, `tst`, `prd-blue`. MAY be required by any target that is environment-aware (build, lint, deploy, etc.). |
|
|
127
127
|
| `VERSION` | Sets the explicit version used during packaging and deployment. Used when there is no automatic version-tagging utility, or to override it. |
|
|
128
128
|
|
|
129
129
|
---
|
|
130
130
|
|
|
131
131
|
#### 05-extending-targets-with-prefixes
|
|
132
132
|
|
|
133
|
-
Projects
|
|
133
|
+
Projects MAY add custom targets beyond the standard set. Custom targets MUST be named by prefixing a standard target name with a descriptive qualifier, keeping the naming intuitive and consistent with the group it belongs to.
|
|
134
134
|
|
|
135
135
|
**Examples:**
|
|
136
136
|
|
|
@@ -152,7 +152,7 @@ The prefix convention ensures developers can infer the purpose of any target wit
|
|
|
152
152
|
|
|
153
153
|
#### 06-monorepo-usage
|
|
154
154
|
|
|
155
|
-
In a monorepo, each module has its own `Makefile` with its own `build`, `lint`, `test`, and `deploy` targets scoped to that module. Parent-level Makefiles (at the application or repo root) delegate to child Makefiles in sequence. The parent Makefile
|
|
155
|
+
In a monorepo, each module has its own `Makefile` with its own `build`, `lint`, `test`, and `deploy` targets scoped to that module. Parent-level Makefiles (at the application or repo root) delegate to child Makefiles in sequence. The parent Makefile SHOULD call `$(MAKE) -C <child> <target>` directly, while each child `Makefile` runs its actual tool commands through `mise exec --`.
|
|
156
156
|
|
|
157
157
|
```makefile
|
|
158
158
|
# root Makefile — delegates to all modules
|
|
@@ -165,7 +165,7 @@ test:
|
|
|
165
165
|
$(MAKE) -C module-b test
|
|
166
166
|
```
|
|
167
167
|
|
|
168
|
-
A developer can run `make test` at the repo root to test everything, or `cd module-a && make test` to test a single module. Both
|
|
168
|
+
A developer can run `make test` at the repo root to test everything, or `cd module-a && make test` to test a single module. Both MUST work.
|
|
169
169
|
|
|
170
170
|
**Reference:** See [agentme-edr-005](005-monorepo-structure.md) for the full monorepo layout convention.
|
|
171
171
|
|
|
@@ -220,15 +220,15 @@ make all
|
|
|
220
220
|
|
|
221
221
|
#### 08-default-targets-must-only-include-offline-subtargets
|
|
222
222
|
|
|
223
|
-
`make all`, `make test`, and `make lint`
|
|
223
|
+
`make all`, `make test`, and `make lint` MUST include every subtarget that runs **offline** — meaning it requires no external credentials, no running servers, no paid APIs, and no environment-specific configuration outside the repository.
|
|
224
224
|
|
|
225
|
-
Subtargets that require external dependencies (e.g., `test-integration` against a live database, `test-e2e` against a staging environment, `lint-api` against a remote schema registry)
|
|
225
|
+
Subtargets that require external dependencies (e.g., `test-integration` against a live database, `test-e2e` against a staging environment, `lint-api` against a remote schema registry) MUST exist as named targets so developers can invoke them explicitly, but MUST NOT be invoked from `all`, `test`, or `lint`.
|
|
226
226
|
|
|
227
227
|
---
|
|
228
228
|
|
|
229
229
|
#### 09-ai-project-dev-targets
|
|
230
230
|
|
|
231
|
-
AI-based projects (LLM, Agent, and Workflow tiers as defined in [agentme-edr-
|
|
231
|
+
AI-based projects (LLM, Agent, and Workflow tiers as defined in [agentme-edr-040](../application/040-ai-llm-development-standards.md)) MUST expose a `dev-mlflow` target that starts a local MLflow tracking server for development inspection.
|
|
232
232
|
|
|
233
233
|
**Example implementation:**
|
|
234
234
|
|
|
@@ -19,7 +19,7 @@ How should projects manage environment variable configuration and CLI invocation
|
|
|
19
19
|
|
|
20
20
|
**Use YAML config files for CLI invocation configuration with multiple attributes; use `.env` files to supply environment variables to spawned processes and to hold uncommitted values referenced by config files. Load `.env` exclusively at process launch time — MUST NOT be loaded inside application code.**
|
|
21
21
|
|
|
22
|
-
Secrets (API keys, passwords, tokens) MUST NOT be placed in `.env` files. Those are handled by [agentme-edr-022](../
|
|
22
|
+
Secrets (API keys, passwords, tokens) MUST NOT be placed in `.env` files. Those are handled by [agentme-edr-022](../application/022-secrets-management.md).
|
|
23
23
|
|
|
24
24
|
### Details
|
|
25
25
|
|
|
@@ -152,7 +152,7 @@ The `.env` file must be loaded in the Makefile before launching the process (see
|
|
|
152
152
|
|
|
153
153
|
## References
|
|
154
154
|
|
|
155
|
-
- [agentme-edr-022](../
|
|
155
|
+
- [agentme-edr-022](../application/022-secrets-management.md) - Secrets must use OS keychains or cloud secret managers, not `.env` files
|
|
156
156
|
- [agentme-edr-017](017-tool-execution-and-scripting.md) - Makefiles are the authoritative command entry point; rule 05 above integrates with that standard
|
|
157
157
|
- [agentme-edr-008](008-common-targets.md) - Standard Makefile target names
|
|
158
158
|
- [agentme-edr-015](../application/015-cli-tool-standards.md) - CLI config file discovery and CLI-to-application separation
|
|
@@ -25,7 +25,7 @@ Language-specific EDRs MAY add ecosystem details, but they MUST NOT redefine the
|
|
|
25
25
|
|
|
26
26
|
A module is the smallest independently buildable, testable, or publishable unit. It MUST live in its own folder and that folder MUST contain:
|
|
27
27
|
|
|
28
|
-
- a `Makefile` following [agentme-edr-008](../
|
|
28
|
+
- a `Makefile` following [agentme-edr-008](../platform/008-common-targets.md)
|
|
29
29
|
- a `README.md` for the module itself
|
|
30
30
|
- all configuration files needed to build, lint, test, package, or publish that module
|
|
31
31
|
- its generated `dist/` directory when the module produces distributable artifacts
|
|
@@ -127,9 +127,9 @@ Benchmark tests MUST live in one of these locations:
|
|
|
127
127
|
|
|
128
128
|
#### 08-module-makefiles-must-expose-shared-targets
|
|
129
129
|
|
|
130
|
-
Every module `Makefile` MUST expose the common target names from [agentme-edr-008](../
|
|
130
|
+
Every module `Makefile` MUST expose the common target names from [agentme-edr-008](../platform/008-common-targets.md). At minimum, modules MUST provide `build`, `lint`, and `test`, and SHOULD also provide `all`, `clean`, and `lint-fix` when meaningful.
|
|
131
131
|
|
|
132
132
|
## References
|
|
133
133
|
|
|
134
|
-
- [agentme-edr-005](../
|
|
135
|
-
- [agentme-edr-008](../
|
|
134
|
+
- [agentme-edr-005](../platform/005-monorepo-structure.md) - Monorepo aggregation and delegation rules
|
|
135
|
+
- [agentme-edr-008](../platform/008-common-targets.md) - Shared Makefile target names
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agentme",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.28.1",
|
|
4
4
|
"description": "",
|
|
5
5
|
"dependencies": {
|
|
6
6
|
"filedist": "^0.39.0"
|
|
@@ -18,6 +18,6 @@
|
|
|
18
18
|
"url": "https://github.com/flaviostutz/agentme.git"
|
|
19
19
|
},
|
|
20
20
|
"devDependencies": {
|
|
21
|
-
"xdrs-core": "^0.
|
|
21
|
+
"xdrs-core": "^0.39.0"
|
|
22
22
|
}
|
|
23
23
|
}
|