agentme 0.28.1 → 0.29.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.filedist-package.yml +1 -1
- package/.xdrs/agentme/edrs/application/{003-javascript-project-tooling.md → 101-javascript-project-tooling.md} +8 -8
- package/.xdrs/agentme/edrs/application/{010-golang-project-tooling.md → 102-golang-project-tooling.md} +6 -6
- package/.xdrs/agentme/edrs/application/{014-python-project-tooling.md → 103-python-project-tooling.md} +11 -11
- package/.xdrs/agentme/edrs/application/{015-cli-tool-standards.md → 104-cli-tool-standards.md} +13 -13
- package/.xdrs/agentme/edrs/application/{002-coding-best-practices.md → 121-coding-best-practices.md} +3 -3
- package/.xdrs/agentme/edrs/application/{004-unit-test-requirements.md → 122-unit-test-requirements.md} +2 -2
- package/.xdrs/agentme/edrs/application/{009-error-handling.md → 123-error-handling.md} +2 -2
- package/.xdrs/agentme/edrs/application/{022-secrets-management.md → 124-secrets-management.md} +5 -5
- package/.xdrs/agentme/edrs/application/{023-coding-abstraction-practices.md → 125-coding-abstraction-practices.md} +3 -3
- package/.xdrs/agentme/edrs/application/{026-pragmatic-hexagonal-architecture.md → 126-pragmatic-hexagonal-architecture.md} +9 -9
- package/.xdrs/agentme/edrs/application/{040-ai-llm-development-standards.md → 141-ai-llm-development-standards.md} +14 -14
- package/.xdrs/agentme/edrs/application/{041-ai-agents-development-standards.md → 142-ai-agents-development-standards.md} +11 -11
- package/.xdrs/agentme/edrs/application/{042-ai-agents-quality-standards.md → 143-ai-agents-quality-standards.md} +16 -16
- package/.xdrs/agentme/edrs/application/{043-ai-workflow-development-standards.md → 144-ai-workflow-development-standards.md} +28 -28
- package/.xdrs/agentme/edrs/application/{044-ai-workflow-naming-conventions.md → 145-ai-workflow-naming-conventions.md} +9 -9
- package/.xdrs/agentme/edrs/application/{045-ai-agent-xdrs-knowledge-layer.md → 146-ai-agent-xdrs-knowledge-layer.md} +7 -7
- package/.xdrs/agentme/edrs/application/{051-ai-eval-core-standards.md → 151-ai-eval-core-standards.md} +17 -17
- package/.xdrs/agentme/edrs/application/{052-ai-test-types-taxonomy.md → 152-ai-test-types-taxonomy.md} +27 -27
- package/.xdrs/agentme/edrs/application/{053-ai-eval-script.md → 153-ai-eval-script.md} +18 -18
- package/.xdrs/agentme/edrs/application/{054-ai-eval-report-format.md → 154-ai-eval-report-format.md} +6 -6
- package/.xdrs/agentme/edrs/application/{055-ai-eval-repeatability.md → 155-ai-eval-repeatability.md} +14 -14
- package/.xdrs/agentme/edrs/application/skills/001-create-javascript-project/SKILL.md +4 -4
- package/.xdrs/agentme/edrs/application/skills/003-create-golang-project/SKILL.md +3 -3
- package/.xdrs/agentme/edrs/application/skills/004-select-relevant-xdrs/SKILL.md +9 -9
- package/.xdrs/agentme/edrs/application/skills/005-create-python-project/SKILL.md +5 -5
- package/.xdrs/agentme/edrs/data/{050-ml-dataset-structure.md → 201-ml-dataset-structure.md} +4 -4
- package/.xdrs/agentme/edrs/governance/{007-project-quality-standards.md → 501-project-quality-standards.md} +16 -16
- package/.xdrs/agentme/edrs/governance/{013-contributing-guide-requirements.md → 502-contributing-guide-requirements.md} +3 -3
- package/.xdrs/agentme/edrs/index.md +30 -30
- package/.xdrs/agentme/edrs/operations/{011-service-health-check-endpoint.md → 401-service-health-check-endpoint.md} +2 -2
- package/.xdrs/agentme/edrs/platform/{005-monorepo-structure.md → 301-monorepo-structure.md} +4 -4
- package/.xdrs/agentme/edrs/platform/{006-github-pipelines.md → 302-github-pipelines.md} +3 -3
- package/.xdrs/agentme/edrs/platform/{008-common-targets.md → 303-common-targets.md} +8 -8
- package/.xdrs/agentme/edrs/platform/{017-tool-execution-and-scripting.md → 304-tool-execution-and-scripting.md} +10 -10
- package/.xdrs/agentme/edrs/platform/{027-environment-variable-configuration.md → 305-environment-variable-configuration.md} +14 -14
- package/.xdrs/agentme/edrs/platform/skills/002-monorepo-setup/SKILL.md +3 -3
- package/.xdrs/agentme/edrs/principles/012-continuous-xdr-enrichment.md +1 -1
- package/.xdrs/agentme/edrs/principles/016-cross-language-module-structure.md +4 -4
- package/package.json +2 -2
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
3
|
-
description: Defines eval.py script requirements for AI projects — entry-first eval loop, --type test-type filtering, mock_fixtures wiring, human entries, threshold enforcement, and MLflow experiment naming and port assignment. Use when implementing eval scripts. For eval folder structure see agentme-edr-
|
|
2
|
+
name: agentme-edr-policy-153-ai-eval-script
|
|
3
|
+
description: Defines eval.py script requirements for AI projects — entry-first eval loop, --type test-type filtering, mock_fixtures wiring, human entries, threshold enforcement, and MLflow experiment naming and port assignment. Use when implementing eval scripts. For eval folder structure see agentme-edr-151 rule 01. For the test type taxonomy and mock_fixtures envelope see agentme-edr-152. For mock file naming see agentme-edr-126 rule 10. For report format see agentme-edr-154. For repeatability loop exception see agentme-edr-155.
|
|
4
4
|
apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
|
|
5
5
|
valid-from: 2026-07-07
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-153: AI eval script
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -17,7 +17,7 @@ How should eval scripts load datasets, iterate entries, handle mocking, and prod
|
|
|
17
17
|
|
|
18
18
|
**Use an entry-first eval loop with `--type` filtering, fresh mock isolation per entry, real LLM providers, and MLflow-backed metrics with explicit per-type thresholds.**
|
|
19
19
|
|
|
20
|
-
For when evals are required per AI tier, see [agentme-edr-
|
|
20
|
+
For when evals are required per AI tier, see [agentme-edr-501](../governance/501-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
|
|
21
21
|
|
|
22
22
|
### Details
|
|
23
23
|
|
|
@@ -25,14 +25,14 @@ For when evals are required per AI tier, see [agentme-edr-007](../governance/007
|
|
|
25
25
|
|
|
26
26
|
Each `eval.py` script MUST:
|
|
27
27
|
|
|
28
|
-
- Load the golden dataset from `golden_dataset/` in the same eval folder, following [agentme-edr-
|
|
28
|
+
- Load the golden dataset from `golden_dataset/` in the same eval folder, following [agentme-edr-201](../data/201-ml-dataset-structure.md) and the entry envelope in [agentme-edr-152](152-ai-test-types-taxonomy.md) rule `02` (one JSON file per entry, `test_types` array, `input`, `expected_output`, optional `mock_fixtures`).
|
|
29
29
|
- Accept a required `--type=<test_type>|all` CLI argument and filter entries whose `test_types` array contains the requested value; `--type=all` includes every entry.
|
|
30
30
|
- Iterate **entry-first**: for each entry in the filtered set, invoke the real component exactly once; then score that single `actual_output` for every `test_types` value the entry carries that falls within the current `--type` scope — MUST NOT invoke the component more than once per entry per run.
|
|
31
|
-
- When an entry contains `mock_fixtures` ([agentme-edr-
|
|
31
|
+
- When an entry contains `mock_fixtures` ([agentme-edr-152](152-ai-test-types-taxonomy.md) rule `02`), configure each named mock adapter with its fixture data BEFORE invoking the component for that entry. Each entry MUST use fresh mock instances so fixture state does not bleed across entries. `mock_fixtures` applies to all test types including `human`. `mock_fixtures` MUST NOT configure LLM adapters — the LLM call MUST be real (see [agentme-edr-152](152-ai-test-types-taxonomy.md) rule `03`). How mock adapters are discovered and instantiated is left to the project; see [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) rule `10` for the `_mock` file naming and placement convention.
|
|
32
32
|
- Run every component invocation against **real LLM providers** (not mocked responses), to capture model drift.
|
|
33
33
|
- For `human` entries: invoke the component to capture `actual_output`, export each entry's `input`, `expected_output.human_test` instructions, and `actual_output` into a manual-review checklist (`report-human.md`). MUST NOT invoke an automated scorer and MUST NOT enforce a pass/fail threshold for it. Other `test_types` on the same entry (e.g. `functional`) are still scored automatically.
|
|
34
|
-
- After all entries are processed, compute aggregate metrics per test type, log them to a local MLflow experiment (see rule `02`), write one `report-<type>.md` per evaluated test type ([agentme-edr-
|
|
35
|
-
- Compare outputs to expected values using project-defined quality thresholds per test type. Thresholds and all other scoring parameters MUST be declared as constants in `eval.py` — they are design decisions about what constitutes acceptable quality for the component under test, not runtime configuration, and MUST NOT be passed via Makefile variables or CLI flags. Use one of two naming conventions, chosen consistently within an `eval.py`: (a) **per-type constants** — `EVAL_MIN_<METRIC>_<TYPE>` for each test type (e.g. `EVAL_MIN_ACCURACY_FUNCTIONAL = 0.85`, `EVAL_MIN_ACCURACY_REPEATABILITY = 0.8`); or (b) **dict constant** — `EVAL_MIN_<METRIC> = {<type>: <value>}` (e.g. `EVAL_MIN_ACCURACY = {"functional": 0.85, "smoke": 0.85}`). Per-type constants are preferred when each test type has a dedicated `eval.py`; the dict form is preferred when a single `eval.py` handles multiple types. In either convention, `EVAL_MIN_ACCURACY` (as a scalar) MAY be declared as a project-wide default and MUST be used as fallback when no per-type override is defined for the current test type. This Policy does not mandate which test types a project must threshold or what value to use (see [agentme-edr-
|
|
34
|
+
- After all entries are processed, compute aggregate metrics per test type, log them to a local MLflow experiment (see rule `02`), write one `report-<type>.md` per evaluated test type ([agentme-edr-154](154-ai-eval-report-format.md) rule `01`), and exit with a non-zero status when any metric falls below its defined threshold per [agentme-edr-501](../governance/501-project-quality-standards.md) rule `07-statistical-models-must-have-eval-targets`. The `human` type has no threshold and does not trigger a non-zero exit.
|
|
35
|
+
- Compare outputs to expected values using project-defined quality thresholds per test type. Thresholds and all other scoring parameters MUST be declared as constants in `eval.py` — they are design decisions about what constitutes acceptable quality for the component under test, not runtime configuration, and MUST NOT be passed via Makefile variables or CLI flags. Use one of two naming conventions, chosen consistently within an `eval.py`: (a) **per-type constants** — `EVAL_MIN_<METRIC>_<TYPE>` for each test type (e.g. `EVAL_MIN_ACCURACY_FUNCTIONAL = 0.85`, `EVAL_MIN_ACCURACY_REPEATABILITY = 0.8`); or (b) **dict constant** — `EVAL_MIN_<METRIC> = {<type>: <value>}` (e.g. `EVAL_MIN_ACCURACY = {"functional": 0.85, "smoke": 0.85}`). Per-type constants are preferred when each test type has a dedicated `eval.py`; the dict form is preferred when a single `eval.py` handles multiple types. In either convention, `EVAL_MIN_ACCURACY` (as a scalar) MAY be declared as a project-wide default and MUST be used as fallback when no per-type override is defined for the current test type. This Policy does not mandate which test types a project must threshold or what value to use (see [agentme-edr-152](152-ai-test-types-taxonomy.md) rule `06`).
|
|
36
36
|
|
|
37
37
|
**Example:**
|
|
38
38
|
|
|
@@ -69,7 +69,7 @@ with mlflow.start_run():
|
|
|
69
69
|
# Entry-first loop: invoke each entry exactly once
|
|
70
70
|
for idx, entry in enumerate(entries, start=1):
|
|
71
71
|
# Configure mock adapters from mock_fixtures before invocation
|
|
72
|
-
# (implementation left to the project — see agentme-edr-
|
|
72
|
+
# (implementation left to the project — see agentme-edr-126 rule 10)
|
|
73
73
|
if entry.get("mock_fixtures"):
|
|
74
74
|
configure_mocks(entry["mock_fixtures"]) # project-defined helper
|
|
75
75
|
|
|
@@ -120,17 +120,17 @@ with mlflow.start_run():
|
|
|
120
120
|
|
|
121
121
|
Each `evals/<component>/eval-<name>/Makefile` MUST start its MLflow tracking server on a **unique port** to prevent conflicts when multiple eval Makefiles are run concurrently or in parallel (e.g., in CI or across multiple terminal sessions).
|
|
122
122
|
|
|
123
|
-
Ports MUST be statically assigned per eval scenario (not per test type) and MUST NOT reuse the default `5000` port (reserved for `dev-mlflow` per [agentme-edr-
|
|
123
|
+
Ports MUST be statically assigned per eval scenario (not per test type) and MUST NOT reuse the default `5000` port (reserved for `dev-mlflow` per [agentme-edr-303](../platform/303-common-targets.md) rule `09-ai-project-dev-targets`). Assign ports starting at `5100` and incrementing by 1 for each additional eval scenario across the entire project.
|
|
124
124
|
|
|
125
125
|
The MLflow **experiment** is scoped to the eval scenario: `<component>/<eval-name>` (e.g. `document-review/eval-basic`). Each `mlflow.start_run()` call MUST set a `test_types` tag listing the test types evaluated in that invocation (comma-separated, e.g. `"functional,smoke"` for `--type=all`, `"smoke"` for `--type=smoke`). A remote MLflow server MUST NOT be required — all tracking is local.
|
|
126
126
|
|
|
127
127
|
## References
|
|
128
128
|
|
|
129
|
-
- [agentme-edr-
|
|
130
|
-
- [agentme-edr-
|
|
131
|
-
- [agentme-edr-
|
|
132
|
-
- [agentme-edr-
|
|
133
|
-
- [agentme-edr-
|
|
134
|
-
- [agentme-edr-
|
|
135
|
-
- [agentme-edr-
|
|
136
|
-
- [agentme-edr-
|
|
129
|
+
- [agentme-edr-151](151-ai-eval-core-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract (rule `02`)
|
|
130
|
+
- [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
|
|
131
|
+
- [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: loop exception to rule `01`'s entry-first constraint, scoring methods, and cadence
|
|
132
|
+
- [agentme-edr-152](152-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum, golden dataset entry envelope (including `mock_fixtures`), and mocking constraints per type
|
|
133
|
+
- [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) — Rule `10`: `_mock` file naming and placement convention for mock adapters used in `mock_fixtures`
|
|
134
|
+
- [agentme-edr-201](../data/201-ml-dataset-structure.md) — ML dataset structure, per-entry JSON format, and schema-lint validation for golden datasets
|
|
135
|
+
- [agentme-edr-501](../governance/501-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09`) and statistical model eval targets (rule `07`)
|
|
136
|
+
- [agentme-edr-303](../platform/303-common-targets.md) — `eval-<qualifier>` Makefile convention (rule `03`) and reserved MLflow port `5000` (rule `09`)
|
package/.xdrs/agentme/edrs/application/{054-ai-eval-report-format.md → 154-ai-eval-report-format.md}
RENAMED
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
3
|
-
description: Defines the eval report format for AI projects — report-<type>.md template, Wilson score confidence interval, convergence analysis, and the human-type checklist artifact. Use when generating eval reports from eval scripts. For the eval script that produces these reports see agentme-edr-
|
|
2
|
+
name: agentme-edr-policy-154-ai-eval-report-format
|
|
3
|
+
description: Defines the eval report format for AI projects — report-<type>.md template, Wilson score confidence interval, convergence analysis, and the human-type checklist artifact. Use when generating eval reports from eval scripts. For the eval script that produces these reports see agentme-edr-153. For repeatability report shape see agentme-edr-155 rule 02.
|
|
4
4
|
apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
|
|
5
5
|
valid-from: 2026-07-07
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-154: AI eval report format
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -166,6 +166,6 @@ xychart-beta
|
|
|
166
166
|
|
|
167
167
|
## References
|
|
168
168
|
|
|
169
|
-
- [agentme-edr-
|
|
170
|
-
- [agentme-edr-
|
|
171
|
-
- [agentme-edr-
|
|
169
|
+
- [agentme-edr-153](153-ai-eval-script.md) — AI eval script: the script that produces these reports (rule `01`)
|
|
170
|
+
- [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: rule `02` defines the adapted report shape for `report-repeatability.md`
|
|
171
|
+
- [agentme-edr-151](151-ai-eval-core-standards.md) — AI eval core standards: folder structure (rule `01`) and LLM-as-judge binary scoring (rule `02`)
|
package/.xdrs/agentme/edrs/application/{055-ai-eval-repeatability.md → 155-ai-eval-repeatability.md}
RENAMED
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
3
|
-
description: Defines the repeatability test type for AI evals — REPEAT_COUNT loop exception to the entry-first constraint, semantic-similarity and LLM-as-judge scoring methods, repeatability_accuracy metric, and the repeatability report format and run cadence. Use when implementing repeatability evals. For the entry-first constraint see agentme-edr-
|
|
2
|
+
name: agentme-edr-policy-155-ai-eval-repeatability
|
|
3
|
+
description: Defines the repeatability test type for AI evals — REPEAT_COUNT loop exception to the entry-first constraint, semantic-similarity and LLM-as-judge scoring methods, repeatability_accuracy metric, and the repeatability report format and run cadence. Use when implementing repeatability evals. For the entry-first constraint see agentme-edr-153 rule 01. For LLM-as-judge binary output see agentme-edr-151 rule 02. For the base report template see agentme-edr-154 rule 01. For the repeatability test type definition see agentme-edr-152.
|
|
4
4
|
apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement repeatability eval testing
|
|
5
5
|
valid-from: 2026-07-07
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-155: AI eval repeatability
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -17,13 +17,13 @@ How should repeatability be measured in evals, and how should its results be rep
|
|
|
17
17
|
|
|
18
18
|
**Exempt `repeatability` entries from the entry-first constraint, invoking each `REPEAT_COUNT` times and scoring via semantic-similarity or LLM-as-judge; schedule at release cadence rather than per-commit.**
|
|
19
19
|
|
|
20
|
-
For the `repeatability` test type definition and its disambiguation from `reproducibility`, see [agentme-edr-
|
|
20
|
+
For the `repeatability` test type definition and its disambiguation from `reproducibility`, see [agentme-edr-152](152-ai-test-types-taxonomy.md) rule `09`.
|
|
21
21
|
|
|
22
22
|
### Details
|
|
23
23
|
|
|
24
24
|
#### 01-repeatability-eval-loop-exception
|
|
25
25
|
|
|
26
|
-
Entries whose `test_types` includes `repeatability` are exempt from [agentme-edr-
|
|
26
|
+
Entries whose `test_types` includes `repeatability` are exempt from [agentme-edr-153](153-ai-eval-script.md) rule `01`'s "invoke exactly once per entry" constraint. The following constants MUST be declared in `eval.py` and MUST NOT be exposed as Makefile variables, CLI flags, or stored as per-entry dataset fields:
|
|
27
27
|
|
|
28
28
|
- `REPEAT_COUNT` — number of times each repeatability entry is invoked. SHOULD default to 3-5 for routine CI runs and 10-20 for focused passes on decision-critical or previously-flagged components. Projects SHOULD calibrate the value once per component by plotting cumulative pass rate against repeat count for a few representative entries and picking the point where it plateaus, rather than guessing.
|
|
29
29
|
- `EVAL_MIN_ACCURACY_REPEATABILITY` — minimum fraction of repeatability entries that must PASS for the eval to exit 0.
|
|
@@ -34,7 +34,7 @@ Entries whose `test_types` includes `repeatability` are exempt from [agentme-edr
|
|
|
34
34
|
**Choosing the scoring method:** Two approaches are supported, declared as a constant in `eval.py`:
|
|
35
35
|
|
|
36
36
|
- **Semantic-similarity:** Embed all `REPEAT_COUNT` outputs for an entry into vectors and compute the average pairwise cosine similarity. The entry passes (score = 1) if the average similarity ≥ `REPEAT_SEMANTIC_SIMILARITY_SCORE`; otherwise it fails (score = 0). Use for classification or short structured outputs.
|
|
37
|
-
- **LLM-as-judge:** Provide all `REPEAT_COUNT` outputs for an entry to an LLM judge (kept at low/zero temperature) that returns 0 (fail) or 1 (pass) directly, based on whether the outputs are sufficiently consistent. No `REPEAT_SEMANTIC_SIMILARITY_SCORE` constant is needed. Use for free-text or complex structured outputs where vector distance is an unreliable proxy for agreement. The judge MUST follow [agentme-edr-
|
|
37
|
+
- **LLM-as-judge:** Provide all `REPEAT_COUNT` outputs for an entry to an LLM judge (kept at low/zero temperature) that returns 0 (fail) or 1 (pass) directly, based on whether the outputs are sufficiently consistent. No `REPEAT_SEMANTIC_SIMILARITY_SCORE` constant is needed. Use for free-text or complex structured outputs where vector distance is an unreliable proxy for agreement. The judge MUST follow [agentme-edr-151](151-ai-eval-core-standards.md) rule `02`'s binary output contract.
|
|
38
38
|
|
|
39
39
|
**`repeatability_accuracy`:** the fraction of repeatability entries that received PASS (score = 1), logged to MLflow as `repeatability_accuracy`. The eval exits non-zero if `repeatability_accuracy` < `EVAL_MIN_ACCURACY_REPEATABILITY`. Both `repeatability_accuracy` and `repeat_count` MUST be logged to MLflow and included in `report-repeatability.md` (rule `02`).
|
|
40
40
|
|
|
@@ -56,20 +56,20 @@ if repeatability_accuracy < EVAL_MIN_ACCURACY_REPEATABILITY:
|
|
|
56
56
|
raise SystemExit(f"Eval failed: repeatability_accuracy {repeatability_accuracy:.2f} < {EVAL_MIN_ACCURACY_REPEATABILITY}")
|
|
57
57
|
```
|
|
58
58
|
|
|
59
|
-
`mock_fixtures` configuration per [agentme-edr-
|
|
59
|
+
`mock_fixtures` configuration per [agentme-edr-153](153-ai-eval-script.md) rule `01` applies to each of the `REPEAT_COUNT` invocations. Any prompt or response caching (provider-side or gateway-side) MUST be bypassed for these invocations — a cache hit would return an identical cached response and falsely report perfect stability instead of measuring the model's actual variance.
|
|
60
60
|
|
|
61
61
|
**Scoping:** this test type MUST NOT be applied to components whose intended behavior is diverse or creative output (e.g. brainstorming, creative writing) — low agreement there is correct behavior, not a defect.
|
|
62
62
|
|
|
63
63
|
#### 02-repeatability-report-and-cadence
|
|
64
64
|
|
|
65
|
-
`--type=repeatability` MUST produce `report-repeatability.md` with a shape adapted from [agentme-edr-
|
|
65
|
+
`--type=repeatability` MUST produce `report-repeatability.md` with a shape adapted from [agentme-edr-154](154-ai-eval-report-format.md) rule `01`'s template: the header MUST include a **Repeat count:** line stating the `REPEAT_COUNT` value used for the run, alongside the usual Date/Dataset/Script/Thresholds lines. The body MUST have an aggregate row reporting `repeatability_accuracy` (the fraction of entries that PASS — see rule `01`) with a Wilson score interval computed over the number of `repeatability` entries, plus a per-item table listing each entry's individual pass/fail result and, when using semantic-similarity, its computed average pairwise cosine similarity — instead of the `Expected | Actual | Correct` columns used by other types.
|
|
66
66
|
|
|
67
|
-
Because `repeatability` entries multiply real LLM-provider calls by `REPEAT_COUNT`, projects SHOULD schedule `make eval-repeatability` at release cadence rather than on every commit, aligned with the Workflow eval cadence in [agentme-edr-
|
|
67
|
+
Because `repeatability` entries multiply real LLM-provider calls by `REPEAT_COUNT`, projects SHOULD schedule `make eval-repeatability` at release cadence rather than on every commit, aligned with the Workflow eval cadence in [agentme-edr-501](../governance/501-project-quality-standards.md) rule `09`, rather than treating it as a mandatory per-commit gate.
|
|
68
68
|
|
|
69
69
|
## References
|
|
70
70
|
|
|
71
|
-
- [agentme-edr-
|
|
72
|
-
- [agentme-edr-
|
|
73
|
-
- [agentme-edr-
|
|
74
|
-
- [agentme-edr-
|
|
75
|
-
- [agentme-edr-
|
|
71
|
+
- [agentme-edr-153](153-ai-eval-script.md) — AI eval script: rule `01` defines the entry-first constraint this policy exempts for repeatability entries
|
|
72
|
+
- [agentme-edr-151](151-ai-eval-core-standards.md) — AI eval core standards: rule `02` defines the LLM-as-judge binary output contract used by the LLM-as-judge scoring method in rule `01`
|
|
73
|
+
- [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: rule `01` defines the base report template that `report-repeatability.md` adapts
|
|
74
|
+
- [agentme-edr-152](152-ai-test-types-taxonomy.md) — AI test types taxonomy: `repeatability` test type definition and disambiguation from `reproducibility` (rule `09`)
|
|
75
|
+
- [agentme-edr-501](../governance/501-project-quality-standards.md) — Project quality standards: Workflow eval cadence (rule `09`) that repeatability runs align with
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
name: 001-create-javascript-project
|
|
3
3
|
description: >
|
|
4
4
|
Scaffolds the initial boilerplate structure for a JavaScript/TypeScript project following
|
|
5
|
-
the standard tooling and layout defined in agentme-edr-
|
|
5
|
+
the standard tooling and layout defined in agentme-edr-101. Activate this skill when the user
|
|
6
6
|
asks to create, scaffold, or initialize a new JavaScript or TypeScript project, npm
|
|
7
7
|
package, or similar project structure.
|
|
8
8
|
metadata:
|
|
@@ -15,12 +15,12 @@ compatibility: JavaScript/TypeScript, Node.js 18+
|
|
|
15
15
|
|
|
16
16
|
Creates a complete JavaScript/TypeScript project from scratch. The layout keeps the
|
|
17
17
|
package self-contained in its module root (`lib/`), organizes internal code following
|
|
18
|
-
[agentme-edr-
|
|
18
|
+
[agentme-edr-126](../../126-pragmatic-hexagonal-architecture.md) (`adapters/`, `app/`, `shared/`),
|
|
19
19
|
places runnable consumer examples in the sibling `examples/` folder, redirects persistent caches
|
|
20
20
|
into `.cache/`, and uses Makefiles as the only entry points. Boilerplate is derived from the
|
|
21
21
|
[filedist](https://github.com/flaviostutz/filedist) project.
|
|
22
22
|
|
|
23
|
-
Related EDRs: [agentme-edr-
|
|
23
|
+
Related EDRs: [agentme-edr-101](../../101-javascript-project-tooling.md), [agentme-edr-016](../../../principles/016-cross-language-module-structure.md), [agentme-edr-126](../../126-pragmatic-hexagonal-architecture.md)
|
|
24
24
|
|
|
25
25
|
## Instructions
|
|
26
26
|
|
|
@@ -433,7 +433,7 @@ Review all created files and confirm:
|
|
|
433
433
|
- [ ] `lib/eslint.config.mjs` points `parserOptions.project` to `tsconfig.json`
|
|
434
434
|
- [ ] `lib/README.md` starts with Quick Start and ends with module development commands
|
|
435
435
|
- [ ] All `[package-name]` placeholders are replaced with the actual name
|
|
436
|
-
- [ ] Structure matches the layout in [agentme-edr-
|
|
436
|
+
- [ ] Structure matches the layout in [agentme-edr-101](../../101-javascript-project-tooling.md)
|
|
437
437
|
|
|
438
438
|
## Examples
|
|
439
439
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
name: 003-create-golang-project
|
|
3
3
|
description: >
|
|
4
4
|
Scaffolds the initial boilerplate structure for a Go (Golang) CLI or library project following
|
|
5
|
-
the standard tooling and layout defined in agentme-edr-
|
|
5
|
+
the standard tooling and layout defined in agentme-edr-102. Activate this skill when the user
|
|
6
6
|
asks to create, scaffold, or initialize a new Go project, CLI binary, or Go module.
|
|
7
7
|
metadata:
|
|
8
8
|
author: flaviostutz
|
|
@@ -12,9 +12,9 @@ compatibility: Go 1.21+
|
|
|
12
12
|
|
|
13
13
|
## Overview
|
|
14
14
|
|
|
15
|
-
Creates a complete Go project from scratch, following the layout from [agentme-edr-
|
|
15
|
+
Creates a complete Go project from scratch, following the layout from [agentme-edr-102](../../102-golang-project-tooling.md) and [agentme-edr-126](../../126-pragmatic-hexagonal-architecture.md). Business logic lives in `app/<feature>/` packages; CLI wiring lives in `adapters/cli/`; outbound integrations live in `adapters/connectors/`; `main.go` is a thin dispatcher. The module root owns its `Makefile`, `README.md`, `dist/`, and `.cache/` folders.
|
|
16
16
|
|
|
17
|
-
Related EDRs: [agentme-edr-
|
|
17
|
+
Related EDRs: [agentme-edr-102](../../102-golang-project-tooling.md), [agentme-edr-016](../../../principles/016-cross-language-module-structure.md), [agentme-edr-126](../../126-pragmatic-hexagonal-architecture.md)
|
|
18
18
|
|
|
19
19
|
## Instructions
|
|
20
20
|
|
|
@@ -60,9 +60,9 @@ extraction.
|
|
|
60
60
|
1. Start from the full shipped agentme XDR set as the default installation target.
|
|
61
61
|
2. Reduce that set only by excluding the XDRs that clearly do not fit the repository. Use
|
|
62
62
|
path-stable identifiers so the extraction command is auditable, for example:
|
|
63
|
-
- `.xdrs/agentme/edrs/application/
|
|
64
|
-
- `.xdrs/agentme/edrs/platform/
|
|
65
|
-
- `.xdrs/agentme/edrs/operations/
|
|
63
|
+
- `.xdrs/agentme/edrs/application/102-golang-project-tooling.md` for non-Go projects
|
|
64
|
+
- `.xdrs/agentme/edrs/platform/301-monorepo-structure.md` for non-monorepos
|
|
65
|
+
- `.xdrs/agentme/edrs/operations/401-service-health-check-endpoint.md` for projects without
|
|
66
66
|
a long-running service surface
|
|
67
67
|
3. If the repository does not want agentme XDRs, stop and explain why instead of forcing an
|
|
68
68
|
installation.
|
|
@@ -109,8 +109,8 @@ extraction.
|
|
|
109
109
|
Input: "Install the right agentme XDR presets for this Node.js library"
|
|
110
110
|
- Inventory the shipped agentme XDR files
|
|
111
111
|
- Analyze the repository and detect a JavaScript library with Makefiles
|
|
112
|
-
- Exclude `.xdrs/agentme/edrs/application/
|
|
113
|
-
- Run `npx -y agentme extract --output . --all --exclude .xdrs/agentme/edrs/application/
|
|
112
|
+
- Exclude `.xdrs/agentme/edrs/application/102-golang-project-tooling.md` and `.xdrs/agentme/edrs/operations/401-service-health-check-endpoint.md`
|
|
113
|
+
- Run `npx -y agentme extract --output . --all --exclude .xdrs/agentme/edrs/application/102-golang-project-tooling.md --exclude .xdrs/agentme/edrs/operations/401-service-health-check-endpoint.md`
|
|
114
114
|
|
|
115
115
|
Input: "Set up agentme for this repo"
|
|
116
116
|
- Inventory the shipped agentme XDR files
|
|
@@ -134,8 +134,8 @@ Input: "Set up agentme for this repo"
|
|
|
134
134
|
## References
|
|
135
135
|
|
|
136
136
|
- [agentme README](../../../../../../README.md)
|
|
137
|
-
- [agentme-edr-
|
|
138
|
-
- [agentme-edr-
|
|
139
|
-
- [agentme-edr-
|
|
140
|
-
- [agentme-edr-
|
|
137
|
+
- [agentme-edr-101 - JavaScript project tooling and structure](../../101-javascript-project-tooling.md)
|
|
138
|
+
- [agentme-edr-301 - Monorepo structure](../../../platform/301-monorepo-structure.md)
|
|
139
|
+
- [agentme-edr-501 - Project quality standards](../../../governance/501-project-quality-standards.md)
|
|
140
|
+
- [agentme-edr-303 - Common development script names](../../../platform/303-common-targets.md)
|
|
141
141
|
- [_core-adr-003 - Skill standards](../../../../../_core/adrs/principles/003-skill-standards.md)
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
name: 005-create-python-project
|
|
3
3
|
description: >
|
|
4
4
|
Scaffolds the initial boilerplate structure for a Python project following the standard tooling
|
|
5
|
-
and layout defined in agentme-edr-
|
|
5
|
+
and layout defined in agentme-edr-103. Activate this skill when the user asks to create,
|
|
6
6
|
scaffold, or initialize a new Python package, CLI, or similar project structure.
|
|
7
7
|
metadata:
|
|
8
8
|
author: flaviostutz
|
|
@@ -14,11 +14,11 @@ compatibility: Python 3.12+
|
|
|
14
14
|
|
|
15
15
|
Creates a complete Python project from scratch using Mise, `uv`, `pyproject.toml`, Ruff,
|
|
16
16
|
ty, Pytest, and Makefiles. The layout keeps the package self-contained under `lib/`,
|
|
17
|
-
organizes internal code following [agentme-edr-
|
|
17
|
+
organizes internal code following [agentme-edr-126](../../126-pragmatic-hexagonal-architecture.md)
|
|
18
18
|
(`adapters/`, `app/`, `shared/`), uses a shared root `.venv/`, redirects persistent caches into
|
|
19
19
|
`.cache/`, and places runnable consumer projects under the sibling `examples/` folder.
|
|
20
20
|
|
|
21
|
-
Related EDRs: [agentme-edr-
|
|
21
|
+
Related EDRs: [agentme-edr-103](../../103-python-project-tooling.md), [agentme-edr-016](../../../principles/016-cross-language-module-structure.md), [agentme-edr-126](../../126-pragmatic-hexagonal-architecture.md)
|
|
22
22
|
|
|
23
23
|
## Instructions
|
|
24
24
|
|
|
@@ -282,7 +282,7 @@ make test
|
|
|
282
282
|
|
|
283
283
|
### Phase 4: Create the package and tests inside `lib/`
|
|
284
284
|
|
|
285
|
-
Create this baseline structure following [agentme-edr-
|
|
285
|
+
Create this baseline structure following [agentme-edr-126](../../126-pragmatic-hexagonal-architecture.md).
|
|
286
286
|
|
|
287
287
|
**`lib/src/[package_name]/__init__.py`**
|
|
288
288
|
|
|
@@ -406,5 +406,5 @@ After creating the files:
|
|
|
406
406
|
|
|
407
407
|
## References
|
|
408
408
|
|
|
409
|
-
- [agentme-edr-
|
|
409
|
+
- [agentme-edr-103](../../103-python-project-tooling.md)
|
|
410
410
|
- [_core-adr-003 - Skill standards](../../../../../_core/adrs/principles/003-skill-standards.md)
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
2
|
+
name: agentme-edr-policy-201-ml-dataset-structure
|
|
3
3
|
description: Defines the standard folder layout and file conventions for ML datasets used in AI/ML projects. Use when creating, organizing, or consuming datasets for machine learning tasks such as image labeling, document extraction, tabular data, LLM evaluation, and Q&A sets.
|
|
4
4
|
apply-to: ML and AI projects that produce or consume datasets
|
|
5
5
|
valid-from: 2026-05-27
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-201: ML dataset structure
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -102,10 +102,10 @@ Every dataset MUST expose a `make lint` target (in the Makefile of the project/c
|
|
|
102
102
|
- Per-entry JSON files (rule `04`) and annotation-pair `.json` siblings (rule `02`) MUST each be validated against `dataset.schema.json`, and their `$schema` property MUST be present and resolve to the dataset's actual schema file.
|
|
103
103
|
- CSV rows (rule `03`) MUST each be converted to a JSON object (column header → value) and validated against the same `dataset.schema.json`.
|
|
104
104
|
- `make lint` MUST list every violation found across all files/rows before exiting with a non-zero status (not fail-fast on the first violation).
|
|
105
|
-
- `jsonschema` MUST be declared as a normal project dependency per [agentme-edr-
|
|
105
|
+
- `jsonschema` MUST be declared as a normal project dependency per [agentme-edr-103](../application/103-python-project-tooling.md); no special-casing.
|
|
106
106
|
|
|
107
107
|
## References
|
|
108
108
|
|
|
109
109
|
- [JSON Schema specification](https://json-schema.org/)
|
|
110
110
|
- [jsonschema (Python library)](https://pypi.org/project/jsonschema/)
|
|
111
|
-
- [agentme-edr-
|
|
111
|
+
- [agentme-edr-103](../application/103-python-project-tooling.md) — Python project tooling and dependency conventions
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
2
|
+
name: agentme-edr-policy-501-project-quality-standards
|
|
3
3
|
description: Defines minimum project quality standards for README onboarding, testing (unit, integration, and AI-tier evals), linting, XDR compliance, and runnable examples. Use when scaffolding or reviewing projects.
|
|
4
4
|
apply-to: All projects
|
|
5
5
|
valid-from: 2026-05-25
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-501: Project quality standards
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -61,7 +61,7 @@ A unit test suite MUST run automatically before every release. Failing tests MUS
|
|
|
61
61
|
|
|
62
62
|
**Exception:** Projects with fewer than 100 lines of code, or whose `README.md` prominently marks them as a **Spike** or **Experiment**, are exempt from this requirement. Such projects MUST NOT be deployed to production.
|
|
63
63
|
|
|
64
|
-
**Reference:** [agentme-edr-
|
|
64
|
+
**Reference:** [agentme-edr-122](../application/122-unit-test-requirements.md) for detailed unit test requirements.
|
|
65
65
|
|
|
66
66
|
---
|
|
67
67
|
|
|
@@ -87,7 +87,7 @@ Projects larger than 10 files or 200 lines of code MUST have a linter configured
|
|
|
87
87
|
|
|
88
88
|
**Exception:** Projects with fewer than 100 lines of code, or whose `README.md` prominently marks them as a **Spike** or **Experiment**, are exempt from this requirement. Such projects MUST NOT be deployed to production.
|
|
89
89
|
|
|
90
|
-
**Reference:** [agentme-edr-
|
|
90
|
+
**Reference:** [agentme-edr-101](../application/101-javascript-project-tooling.md) for JavaScript-specific tooling.
|
|
91
91
|
|
|
92
92
|
---
|
|
93
93
|
|
|
@@ -156,7 +156,7 @@ test-examples:
|
|
|
156
156
|
$(MAKE) -C examples
|
|
157
157
|
```
|
|
158
158
|
|
|
159
|
-
If examples require live services or credentials, remove `test-examples` from the `test` dependency list and keep it as a standalone named target only. See [agentme-edr-
|
|
159
|
+
If examples require live services or credentials, remove `test-examples` from the `test` dependency list and keep it as a standalone named target only. See [agentme-edr-303](../platform/303-common-targets.md) rule 08 for the full offline/online decision table.
|
|
160
160
|
|
|
161
161
|
**Examples Makefile:**
|
|
162
162
|
|
|
@@ -175,7 +175,7 @@ Projects that contain statistical models (e.g., ML models, LLM-based evaluators,
|
|
|
175
175
|
**Requirements:**
|
|
176
176
|
- A `make eval` target MUST exist and execute all performance evaluations
|
|
177
177
|
- Each evaluation MUST have a **documented minimum performance threshold** (e.g., accuracy ≥ 0.85, F1 ≥ 0.80, BLEU ≥ 0.70)
|
|
178
|
-
- Thresholds and all scoring parameters MUST be declared as constants in `eval.py` — they are design decisions about acceptable quality for the component under test, not runtime configuration, and MUST NOT be passed as Makefile variables or CLI flags. See [agentme-edr-
|
|
178
|
+
- Thresholds and all scoring parameters MUST be declared as constants in `eval.py` — they are design decisions about acceptable quality for the component under test, not runtime configuration, and MUST NOT be passed as Makefile variables or CLI flags. See [agentme-edr-153](../application/153-ai-eval-script.md) rule `01`.
|
|
179
179
|
- `make eval` MUST **exit with a non-zero status** (fail) if:
|
|
180
180
|
- The evaluation cannot be executed (missing data, environment errors, model load failures)
|
|
181
181
|
- Any metric falls below its defined minimum threshold
|
|
@@ -244,27 +244,27 @@ Projects are not required to implement integration tests, but when present, they
|
|
|
244
244
|
|
|
245
245
|
#### 09-ai-project-testing-requirements
|
|
246
246
|
|
|
247
|
-
AI projects are classified into three tiers — LLM, Agent, and Workflow — defined in [agentme-edr-
|
|
247
|
+
AI projects are classified into three tiers — LLM, Agent, and Workflow — defined in [agentme-edr-141](../application/141-ai-llm-development-standards.md). Testing requirements differ per tier:
|
|
248
248
|
|
|
249
249
|
| Tier | Unit tests | Evals | Integration tests |
|
|
250
250
|
|---|---|---|---|
|
|
251
|
-
| **LLM** ([agentme-edr-
|
|
252
|
-
| **Agent** ([agentme-edr-
|
|
253
|
-
| **Workflow** ([agentme-edr-
|
|
251
|
+
| **LLM** ([agentme-edr-141](../application/141-ai-llm-development-standards.md)) | Not required | Not required; SHOULD be used when critical prompts are in use to measure accuracy and detect model drift | Not required |
|
|
252
|
+
| **Agent** ([agentme-edr-142](../application/142-ai-agents-development-standards.md)) | Not required | Not required; MAY be used | Not required |
|
|
253
|
+
| **Workflow** ([agentme-edr-144](../application/144-ai-workflow-development-standards.md)) | Required — see below | Required before every release; failed evals block release | Advised |
|
|
254
254
|
|
|
255
255
|
**Workflow unit test requirements:**
|
|
256
256
|
|
|
257
|
-
- MUST use mocked LLM providers. See [agentme-edr-
|
|
258
|
-
- MUST run offline with no external dependencies per [agentme-edr-
|
|
259
|
-
- MUST achieve 80% code coverage per [agentme-edr-
|
|
257
|
+
- MUST use mocked LLM providers. See [agentme-edr-141](../application/141-ai-llm-development-standards.md) rule `04-unit-test-mocking` for the mocking pattern.
|
|
258
|
+
- MUST run offline with no external dependencies per [agentme-edr-122](../application/122-unit-test-requirements.md) rule `02-must-run-offline`.
|
|
259
|
+
- MUST achieve 80% code coverage per [agentme-edr-122](../application/122-unit-test-requirements.md) rule `03-must-maintain-80-percent-coverage`.
|
|
260
260
|
- MUST test workflow routing logic, conditional edges, state transformations, and error handling.
|
|
261
261
|
- MUST achieve **80% coverage of LangGraph graph edges and branches**: every conditional edge MUST have test cases covering each possible branch, and every node→node transition MUST be exercised by at least one test.
|
|
262
|
-
- Files MUST be named `<name>_test.py` and placed alongside the source file per [agentme-edr-
|
|
262
|
+
- Files MUST be named `<name>_test.py` and placed alongside the source file per [agentme-edr-122](../application/122-unit-test-requirements.md) rule `04-must-place-test-files-alongside-source`.
|
|
263
263
|
|
|
264
264
|
**Workflow eval requirements:**
|
|
265
265
|
|
|
266
266
|
- Evals MUST be executed before every release.
|
|
267
267
|
- Accuracy below project-defined thresholds MUST block the release. Thresholds MUST be documented in the eval Makefile or README.
|
|
268
268
|
- Evals MUST run against real LLM providers (not mocks) to capture model drift.
|
|
269
|
-
- For eval folder structure and LLM-as-judge scoring, see [agentme-edr-
|
|
270
|
-
- For the taxonomy of AI test types (safety, responsible-AI, quality-eval, prompt, code-level) and the golden dataset entry format, see [agentme-edr-
|
|
269
|
+
- For eval folder structure and LLM-as-judge scoring, see [agentme-edr-151](../application/151-ai-eval-core-standards.md). For eval script requirements, see [agentme-edr-153](../application/153-ai-eval-script.md).
|
|
270
|
+
- For the taxonomy of AI test types (safety, responsible-AI, quality-eval, prompt, code-level) and the golden dataset entry format, see [agentme-edr-152](../application/152-ai-test-types-taxonomy.md).
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
2
|
+
name: agentme-edr-policy-502-contributing-guide-requirements
|
|
3
3
|
description: Defines the minimum contributor workflow guidance required in root CONTRIBUTING.md files. Use when scaffolding or reviewing contribution processes.
|
|
4
4
|
apply-to: Projects with open-source contributions
|
|
5
5
|
valid-from: 2026-05-25
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-502: Contributing guide requirements
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -66,5 +66,5 @@ The content SHOULD stay concise and practical; do not turn `CONTRIBUTING.md` int
|
|
|
66
66
|
|
|
67
67
|
## References
|
|
68
68
|
|
|
69
|
-
- [agentme-edr-
|
|
69
|
+
- [agentme-edr-301 - Monorepo structure](../platform/301-monorepo-structure.md)
|
|
70
70
|
- [002-monorepo-setup skill](../platform/skills/002-monorepo-setup/SKILL.md)
|
|
@@ -21,67 +21,67 @@ Synthetic views combining agentme XDRs and skills around a specific topic.
|
|
|
21
21
|
|
|
22
22
|
Language and framework-specific tooling and project structure.
|
|
23
23
|
|
|
24
|
-
- [agentme-edr-
|
|
25
|
-
- [agentme-edr-
|
|
26
|
-
- [agentme-edr-
|
|
27
|
-
- [agentme-edr-
|
|
28
|
-
- [agentme-edr-
|
|
24
|
+
- [agentme-edr-121](application/121-coding-best-practices.md) - **Coding best practices** - Keep files small, tests nearby, and docs synchronized
|
|
25
|
+
- [agentme-edr-122](application/122-unit-test-requirements.md) - **Unit test requirements** - Define minimum unit-test coverage and naming expectations
|
|
26
|
+
- [agentme-edr-123](application/123-error-handling.md) - **Error handling** - Standardize explicit errors, logging, and propagation rules
|
|
27
|
+
- [agentme-edr-124](application/124-secrets-management.md) - **Secrets management** - Handle secrets securely using native keychains and cloud secret managers
|
|
28
|
+
- [agentme-edr-125](application/125-coding-abstraction-practices.md) - **Coding abstraction practices** - Define when abstractions are justified and when they must be inlined
|
|
29
29
|
|
|
30
30
|
### Language and framework tooling
|
|
31
31
|
|
|
32
|
-
- [agentme-edr-
|
|
33
|
-
- [agentme-edr-
|
|
34
|
-
- [agentme-edr-
|
|
35
|
-
- [agentme-edr-
|
|
36
|
-
- [agentme-edr-
|
|
32
|
+
- [agentme-edr-101](application/101-javascript-project-tooling.md) - **JavaScript project tooling and structure** - Scaffold JavaScript libraries with the standard toolchain *(includes skill: [001-create-javascript-project](application/skills/001-create-javascript-project/SKILL.md))*
|
|
33
|
+
- [agentme-edr-102](application/102-golang-project-tooling.md) - **Go project tooling and structure** - Scaffold Go CLIs and libraries with the standard layout *(includes skill: [003-create-golang-project](application/skills/003-create-golang-project/SKILL.md))*
|
|
34
|
+
- [agentme-edr-103](application/103-python-project-tooling.md) - **Python project tooling and structure** - Scaffold Python packages and CLIs with the standard layout *(includes skill: [005-create-python-project](application/skills/005-create-python-project/SKILL.md))*
|
|
35
|
+
- [agentme-edr-104](application/104-cli-tool-standards.md) - **CLI tool standards** - Define command UX and behavior for CLI tools
|
|
36
|
+
- [agentme-edr-126](application/126-pragmatic-hexagonal-architecture.md) - **Pragmatic hexagonal architecture** - Organize application layers as External/Adapters/Application with practical coupling rules
|
|
37
37
|
- [004-select-relevant-xdrs](application/skills/004-select-relevant-xdrs/SKILL.md) - **Select relevant XDRs**
|
|
38
38
|
|
|
39
39
|
### AI development
|
|
40
40
|
|
|
41
41
|
Standards for building LLM, Agent, and Workflow components.
|
|
42
42
|
|
|
43
|
-
- [agentme-edr-
|
|
44
|
-
- [agentme-edr-
|
|
45
|
-
- [agentme-edr-
|
|
46
|
-
- [agentme-edr-
|
|
47
|
-
- [agentme-edr-
|
|
48
|
-
- [agentme-edr-
|
|
43
|
+
- [agentme-edr-141](application/141-ai-llm-development-standards.md) - **AI LLM development standards** - Standard framework (LangChain) and patterns for simple LLM calls with explicit configuration (no environment variables)
|
|
44
|
+
- [agentme-edr-142](application/142-ai-agents-development-standards.md) - **AI agents development standards** - Structural patterns for agents: framework selection, sandbox setup, naming conventions, composition, and system prompt structure
|
|
45
|
+
- [agentme-edr-143](application/143-ai-agents-quality-standards.md) - **AI agents implementation quality standards** - Tool definition patterns, error handling, observability, and unit testing for agents
|
|
46
|
+
- [agentme-edr-144](application/144-ai-workflow-development-standards.md) - **AI workflow development standards** - Standard toolchain (LangGraph), evaluation, and testing patterns for workflow projects
|
|
47
|
+
- [agentme-edr-145](application/145-ai-workflow-naming-conventions.md) - **AI workflow naming conventions** - Node suffix/prefix roles, state type and attribute naming, judge output schema, workflow class names, and cross-element coherence rules
|
|
48
|
+
- [agentme-edr-146](application/146-ai-agent-xdrs-knowledge-layer.md) - **AI agent XDRS knowledge layer** - How to integrate XDRS as the runtime source of truth for policies and skills in AI agents (apply only when the project explicitly uses XDRS)
|
|
49
49
|
|
|
50
50
|
### AI evaluation and testing
|
|
51
51
|
|
|
52
52
|
Standards for eval datasets, scripts, reports, and test type taxonomy.
|
|
53
53
|
|
|
54
|
-
- [agentme-edr-
|
|
55
|
-
- [agentme-edr-
|
|
56
|
-
- [agentme-edr-
|
|
57
|
-
- [agentme-edr-
|
|
58
|
-
- [agentme-edr-
|
|
54
|
+
- [agentme-edr-151](application/151-ai-eval-core-standards.md) - **AI eval core standards** - Eval folder structure and Makefile interface; LLM-as-judge binary scoring contract applicable to all AI tiers and test types
|
|
55
|
+
- [agentme-edr-152](application/152-ai-test-types-taxonomy.md) - **AI test types taxonomy** - Names AI test types (`functional`, `safety`, `smoke`, `repeatability`, `adversarial`, `fairness`, `bias`, and 5 others) with group, objective, mocking constraint, and relevance, and defines the shared golden dataset entry envelope
|
|
56
|
+
- [agentme-edr-153](application/153-ai-eval-script.md) - **AI eval script** - eval.py requirements: entry-first loop, --type filtering, mock_fixtures wiring, human entries, threshold enforcement, and MLflow experiment conventions
|
|
57
|
+
- [agentme-edr-154](application/154-ai-eval-report-format.md) - **AI eval report format** - report-<type>.md template, Wilson score confidence interval, convergence analysis, and human-type checklist artifact
|
|
58
|
+
- [agentme-edr-155](application/155-ai-eval-repeatability.md) - **AI eval repeatability** - Repeatability test type: REPEAT_COUNT loop exception, semantic-similarity and LLM-as-judge scoring, repeatability_accuracy metric, report shape, and run cadence
|
|
59
59
|
|
|
60
60
|
## Data
|
|
61
61
|
|
|
62
62
|
Data layer implementation and data management decisions.
|
|
63
63
|
|
|
64
|
-
- [agentme-edr-
|
|
64
|
+
- [agentme-edr-201](data/201-ml-dataset-structure.md) - **ML dataset structure** - Standard folder layout and file conventions for ML datasets
|
|
65
65
|
|
|
66
66
|
## Platform
|
|
67
67
|
|
|
68
68
|
Infrastructure implementation, delivery pipeline, and developer environment decisions.
|
|
69
69
|
|
|
70
|
-
- [agentme-edr-
|
|
71
|
-
- [agentme-edr-
|
|
72
|
-
- [agentme-edr-
|
|
73
|
-
- [agentme-edr-
|
|
74
|
-
- [agentme-edr-
|
|
70
|
+
- [agentme-edr-301](platform/301-monorepo-structure.md) - **Monorepo structure** - Standardize monorepo layout, tooling, and package boundaries *(includes skill: [002-monorepo-setup](platform/skills/002-monorepo-setup/SKILL.md))*
|
|
71
|
+
- [agentme-edr-302](platform/302-github-pipelines.md) - **GitHub CI/CD pipelines** - Define required CI stages and workflow structure
|
|
72
|
+
- [agentme-edr-303](platform/303-common-targets.md) - **Common development script names** - Reuse standard build, lint, and test target names
|
|
73
|
+
- [agentme-edr-304](platform/304-tool-execution-and-scripting.md) - **Tool execution and scripting** - Run tools consistently across shells, Makefiles, and CI
|
|
74
|
+
- [agentme-edr-305](platform/305-environment-variable-configuration.md) - **Environment variable configuration files** - Manage non-secret configuration with `.env` files, `.gitignore` rules, stage variants, and Makefile loading
|
|
75
75
|
|
|
76
76
|
## Governance
|
|
77
77
|
|
|
78
78
|
Contribution and collaboration standards shared across projects.
|
|
79
79
|
|
|
80
|
-
- [agentme-edr-
|
|
81
|
-
- [agentme-edr-
|
|
80
|
+
- [agentme-edr-501](governance/501-project-quality-standards.md) - **Project quality standards** - Require build, lint, and test verification before completion
|
|
81
|
+
- [agentme-edr-502](governance/502-contributing-guide-requirements.md) - **Contributing guide requirements** - Define the minimum structure for CONTRIBUTING guides
|
|
82
82
|
|
|
83
83
|
## Operations
|
|
84
84
|
|
|
85
85
|
Production behavior and operational response decisions.
|
|
86
86
|
|
|
87
|
-
- [agentme-edr-
|
|
87
|
+
- [agentme-edr-401](operations/401-service-health-check-endpoint.md) - **Service health check endpoint** - Expose a standard runtime health-check endpoint for services
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-
|
|
2
|
+
name: agentme-edr-policy-401-service-health-check-endpoint
|
|
3
3
|
description: Defines the required health endpoint contract for service availability and dependency readiness checks. Use when implementing or reviewing service health endpoints.
|
|
4
4
|
apply-to: Service projects with health endpoints
|
|
5
5
|
valid-from: 2026-05-25
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-
|
|
8
|
+
# agentme-edr-policy-401: Service health check endpoint
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|