agentme 0.29.0 → 0.30.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.filedist-package.yml +1 -1
- package/.xdrs/agentme/edrs/application/101-javascript-project-tooling.md +1 -1
- package/.xdrs/agentme/edrs/application/102-golang-project-tooling.md +1 -1
- package/.xdrs/agentme/edrs/application/103-python-project-tooling.md +1 -1
- package/.xdrs/agentme/edrs/application/127-external-system-adapter-skills.md +93 -0
- package/.xdrs/agentme/edrs/application/141-ai-llm-development-standards.md +1 -1
- package/.xdrs/agentme/edrs/application/143-ai-agents-quality-standards.md +1 -1
- package/.xdrs/agentme/edrs/application/144-ai-workflow-development-standards.md +3 -3
- package/.xdrs/agentme/edrs/application/{151-ai-eval-core-standards.md → 151-ai-eval-standards.md} +2 -2
- package/.xdrs/agentme/edrs/application/152-ai-test-types-taxonomy.md +1 -1
- package/.xdrs/agentme/edrs/application/153-ai-eval-script.md +1 -1
- package/.xdrs/agentme/edrs/application/154-ai-eval-report-format.md +1 -1
- package/.xdrs/agentme/edrs/application/155-ai-eval-repeatability.md +2 -2
- package/.xdrs/agentme/edrs/application/skills/{004-select-relevant-xdrs → 010-select-relevant-xdrs}/SKILL.md +1 -1
- package/.xdrs/agentme/edrs/application/skills/{001-create-javascript-project → 050-create-javascript-project}/SKILL.md +1 -1
- package/.xdrs/agentme/edrs/application/skills/{003-create-golang-project → 051-create-golang-project}/SKILL.md +1 -1
- package/.xdrs/agentme/edrs/application/skills/{005-create-python-project → 052-create-python-project}/SKILL.md +1 -1
- package/.xdrs/agentme/edrs/application/skills/200-run-skill-tests/SKILL.md +113 -0
- package/.xdrs/agentme/edrs/application/skills/200-run-skill-tests/SKILL.test.md +47 -0
- package/.xdrs/agentme/edrs/governance/501-project-quality-standards.md +1 -1
- package/.xdrs/agentme/edrs/governance/502-contributing-guide-requirements.md +1 -1
- package/.xdrs/agentme/edrs/index.md +8 -6
- package/.xdrs/agentme/edrs/platform/301-monorepo-structure.md +1 -1
- package/.xdrs/agentme/edrs/platform/skills/{002-monorepo-setup → 053-monorepo-setup}/SKILL.md +1 -1
- package/.xdrs/agentme/edrs/principles/012-continuous-xdr-enrichment.md +1 -1
- package/.xdrs/agentme/edrs/principles/017-skill-testing.md +77 -0
- package/.xdrs/agentme/edrs/principles/articles/001-continuous-xdr-improvement.md +1 -1
- package/package.json +2 -2
package/.filedist-package.yml
CHANGED
|
@@ -156,5 +156,5 @@ The examples folder MUST exist for any libraries and utilities that are publishe
|
|
|
156
156
|
|
|
157
157
|
- [agentme-edr-122](122-unit-test-requirements.md) — Coverage and unit-test baseline
|
|
158
158
|
- [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) — Internal adapter/application layer separation for applications
|
|
159
|
-
- [
|
|
159
|
+
- [050-create-javascript-project](skills/050-create-javascript-project/SKILL.md) — scaffolds a new project following this structure
|
|
160
160
|
|
|
@@ -179,4 +179,4 @@ Use the standard library `flag` package for CLI flags. Each `adapters/cli/<featu
|
|
|
179
179
|
## References
|
|
180
180
|
|
|
181
181
|
- [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) — Defines the adapter/application separation that this layout follows
|
|
182
|
-
- [
|
|
182
|
+
- [051-create-golang-project](skills/051-create-golang-project/SKILL.md) — scaffolds a new Go project following this structure
|
|
@@ -212,4 +212,4 @@ The root `Makefile` MUST remain the only contract for CI and contributors, in li
|
|
|
212
212
|
- [agentme-edr-122](122-unit-test-requirements.md) - Coverage and unit-test baseline
|
|
213
213
|
- [agentme-edr-501](../governance/501-project-quality-standards.md) - Examples and quality requirements
|
|
214
214
|
- [agentme-edr-303](../platform/303-common-targets.md) - Standard Makefile target names
|
|
215
|
-
- [
|
|
215
|
+
- [052-create-python-project](skills/052-create-python-project/SKILL.md) - Scaffold a project following this EDR
|
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: agentme-edr-policy-127-external-system-adapter-skills
|
|
3
|
+
description: Defines how agents must approach reading or mutating data in external systems (priority order, credential handling, browser configuration, mutation safety) and what constraints apply when encoding that knowledge as reusable adapter skills. Use when automating any interaction with an external system or when authoring adapter skills for a system.
|
|
4
|
+
apply-to: All automation tasks that read from or write to external systems; all adapter skill authoring
|
|
5
|
+
valid-from: 2026-08-04
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# agentme-edr-policy-127: External system adapter skills
|
|
9
|
+
|
|
10
|
+
## Context and Problem Statement
|
|
11
|
+
|
|
12
|
+
When automating tasks that interact with external systems (SaaS tools, REST APIs, internal platforms), agents face recurring decisions: which integration channel to use, how to handle credentials, and how to capture accumulated system knowledge so it can be reused.
|
|
13
|
+
|
|
14
|
+
How should agents approach external system interaction, and how should that knowledge be encoded as reusable skills?
|
|
15
|
+
|
|
16
|
+
## Decision Outcome
|
|
17
|
+
|
|
18
|
+
Use a priority-ordered integration approach — always leveraging the user's existing session context — and encode system-specific interaction knowledge as business-logic-free adapter skills.
|
|
19
|
+
|
|
20
|
+
Rules 01–04 govern **runtime behavior** (an agent executing a task against an external system). Rule 05 governs **authoring** (a skill author writing an adapter skill).
|
|
21
|
+
|
|
22
|
+
### Details
|
|
23
|
+
|
|
24
|
+
#### 01-approach-priority-order
|
|
25
|
+
|
|
26
|
+
Agents MUST attempt the following approaches in order, stopping at the first that is feasible:
|
|
27
|
+
|
|
28
|
+
1. **API via curl** — If the system exposes an API, interact with it directly using the `curl` CLI. Another HTTP client MAY be used only when `curl` is insufficient for the specific operation (e.g. streaming, binary upload).
|
|
29
|
+
2. **Playwright UI scraping** — If no API is available or the user cannot provide an API credential, use the `playwright` CLI (via `npx -y --package=@playwright/cli@latest playwright-cli`) to interact with the system's UI using the user's existing browser profile (see rule 03). Another browser automation tool MUST NOT be used.
|
|
30
|
+
3. **Git clone** — If the target data is read-only and lives in a git repository, clone the repository locally and read from the local path. For private repositories, follow [agentme-edr-124](124-secrets-management.md) to retrieve the PAT or SSH key from the native keychain.
|
|
31
|
+
4. **Local folder** — As a last resort, ask the user to provide a path to a local folder containing the relevant data.
|
|
32
|
+
|
|
33
|
+
When a higher-priority approach is attempted and fails, the failure reason MUST be stated before trying the next approach.
|
|
34
|
+
|
|
35
|
+
#### 02-api-credential-handling
|
|
36
|
+
|
|
37
|
+
API credentials (keys, tokens, passwords) MUST be stored and retrieved using the native OS keychain following [agentme-edr-124](124-secrets-management.md). Agents MUST NOT hardcode, log, or persist credentials to disk. When a required credential is absent, agents MUST prompt the user to store it via the `setup-secrets` Makefile target before proceeding.
|
|
38
|
+
|
|
39
|
+
#### 03-playwright-browser-config
|
|
40
|
+
|
|
41
|
+
When using Playwright, agents MUST run the `playwright` CLI via `npx -y --package=@playwright/cli@latest playwright-cli` and MUST use the user's existing browser profile to preserve SSO sessions, CA certificates, cookies, and extensions that the target system depends on.
|
|
42
|
+
|
|
43
|
+
- MUST use `--user-data-dir` pointing to the user's active browser profile directory, or attach to a running browser instance via CDP.
|
|
44
|
+
- MUST NOT launch a blank, incognito, or freshly provisioned profile.
|
|
45
|
+
- MUST keep the browser window visible throughout the interaction so the user can follow and intervene.
|
|
46
|
+
- SHOULD prefer CDP attachment to an already-running browser over launching a new instance, when the browser is already open.
|
|
47
|
+
|
|
48
|
+
When the `playwright` CLI is technically insufficient for a required integration capability (e.g., network response interception, which the CLI does not expose), the Playwright Node.js API MAY be used instead. The justification MUST be documented in a `## Conflicts` section within the adapter skill, following the same format used in Policy conflict declarations (citing the policy rule being overridden, the reason, and the mitigations applied).
|
|
49
|
+
|
|
50
|
+
#### 04-human-in-the-loop-before-mutations
|
|
51
|
+
|
|
52
|
+
Before executing any write, mutate, or delete operation on an external system, agents MUST present a plain-language summary containing at minimum:
|
|
53
|
+
|
|
54
|
+
- **System:** name and environment (e.g. "ServiceNow production")
|
|
55
|
+
- **Operation:** what action will be taken
|
|
56
|
+
- **Fields/values:** which fields will be changed and to what values
|
|
57
|
+
- **Estimated impact:** a brief statement of what the change will affect
|
|
58
|
+
|
|
59
|
+
Agents MUST wait for explicit user confirmation before proceeding. Read-only and query operations do not require confirmation.
|
|
60
|
+
|
|
61
|
+
#### 05-adapter-skill-no-business-logic
|
|
62
|
+
|
|
63
|
+
Adapter skills MUST be pure I/O bridges between the agent and the external system. They MUST NOT contain business rules, domain decisions, validation logic, or application-layer concerns, which belong in the application or workflow layer following [agentme-edr-126](126-pragmatic-hexagonal-architecture.md).
|
|
64
|
+
|
|
65
|
+
An adapter skill SHOULD provide only: session setup, navigation, field interaction, and response parsing specific to the target system.
|
|
66
|
+
|
|
67
|
+
#### 06-connector-skill-naming
|
|
68
|
+
|
|
69
|
+
A skill that serves as a base to connect to an external system via API, UI scraping, or file access MUST have a name ending with `-connector` (e.g. `servicenow-connector`, `sap-api-connector`).
|
|
70
|
+
|
|
71
|
+
#### 07-connector-known-issues-section
|
|
72
|
+
|
|
73
|
+
Every connector skill MUST contain a `## Known Issues` section documenting previous problems encountered when using that connector and how to overcome them. This section is read by running agents at execution time to self-correct without human intervention. Each entry SHOULD follow this structure:
|
|
74
|
+
|
|
75
|
+
- **Symptom:** observable sign that the problem has occurred
|
|
76
|
+
- **Cause:** brief explanation of the root cause
|
|
77
|
+
- **Fix:** concrete steps the agent MUST take to resolve the issue
|
|
78
|
+
|
|
79
|
+
---
|
|
80
|
+
|
|
81
|
+
#### guidance
|
|
82
|
+
|
|
83
|
+
> Non-normative. The following is illustrative guidance, not a requirement.
|
|
84
|
+
|
|
85
|
+
Adapter skills for a given system benefit from layering so that knowledge at each level can be activated and reused independently:
|
|
86
|
+
|
|
87
|
+
- **Base skill (connector)** — Session setup, authentication, system overview, and top-level screen or API structure. MUST follow the `-connector` naming rule (rule 06) and MUST include a `## Known Issues` section (rule 07). This skill gives the agent enough context to orient itself in the system. Examples: `servicenow-connector` (login, workspace layout, navigation patterns), `sap-api-connector` (base URL, API key retrieval, OpenAPI spec location).
|
|
88
|
+
- **Domain skill** — A specific feature area, building on the base skill. Examples: `servicenow-incidents` (incident concept, workspace, field structure), `sap-api-mutations` (confirmation procedures, human-in-the-loop steps per rule 04).
|
|
89
|
+
- **Operation skill** — A specific action within a domain, building on the domain skill. Examples: `servicenow-incidents-change` (open an incident, edit fields, click action buttons, handle known errors), `servicenow-vulnerabilities` (filter, triage, export).
|
|
90
|
+
|
|
91
|
+
The depth of the hierarchy is discretionary. A simple system may need only a base skill. Complex systems benefit from the full three-layer structure. Higher-level skills SHOULD activate their lower-level dependencies explicitly at the start of their instructions.
|
|
92
|
+
|
|
93
|
+
When creating adapter skills for a new system, document the chosen integration approach (rule 01) in the skill's frontmatter description, follow the `-connector` naming convention (rule 06) for the base skill, and populate the `## Known Issues` section (rule 07) as experience accumulates. All system-specific adapter skills SHOULD be placed in the `_local` scope of the consuming project.
|
|
@@ -218,6 +218,6 @@ Return a JSON object with:
|
|
|
218
218
|
- [agentme-edr-122](122-unit-test-requirements.md) — Unit test requirements including external API mocking guidance
|
|
219
219
|
- [agentme-edr-103](103-python-project-tooling.md) — Python project tooling and structure
|
|
220
220
|
- [agentme-edr-501](../governance/501-project-quality-standards.md) — Project quality standards including AI-tier testing requirements (rule `09-ai-project-testing-requirements`)
|
|
221
|
-
- [agentme-edr-151](151-ai-eval-
|
|
221
|
+
- [agentme-edr-151](151-ai-eval-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract (rule `02`)
|
|
222
222
|
- [agentme-edr-153](153-ai-eval-script.md) — AI eval script: entry-first loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
|
|
223
223
|
- [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
|
|
@@ -179,6 +179,6 @@ def test_workflow_calls_subagent(mocker):
|
|
|
179
179
|
- [agentme-edr-141](141-ai-llm-development-standards.md) — LLM development standards (LangChain configuration, mocking patterns)
|
|
180
180
|
- [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) — Hexagonal architecture (tool placement in adapters/connectors)
|
|
181
181
|
- [agentme-edr-501](../governance/501-project-quality-standards.md) — Project quality standards including AI-tier testing requirements (rule `09-ai-project-testing-requirements`)
|
|
182
|
-
- [agentme-edr-151](151-ai-eval-
|
|
182
|
+
- [agentme-edr-151](151-ai-eval-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract (rule `02`)
|
|
183
183
|
- [agentme-edr-153](153-ai-eval-script.md) — AI eval script: entry-first loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
|
|
184
184
|
- [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
|
|
@@ -37,7 +37,7 @@ Use **MLflow** for all workflow observability and evaluation:
|
|
|
37
37
|
|
|
38
38
|
#### 04-dataset-driven-accuracy-measurement
|
|
39
39
|
|
|
40
|
-
Projects MUST follow the eval dataset and implementation requirements defined in [agentme-edr-151](151-ai-eval-
|
|
40
|
+
Projects MUST follow the eval dataset and implementation requirements defined in [agentme-edr-151](151-ai-eval-standards.md) and [agentme-edr-153](153-ai-eval-script.md). Testing requirements (when evals are required, release gates) are defined in [agentme-edr-501](../governance/501-project-quality-standards.md) rule `09-ai-project-testing-requirements`.
|
|
41
41
|
|
|
42
42
|
#### 05-flow-documentation
|
|
43
43
|
|
|
@@ -101,7 +101,7 @@ lib/src/<package_name>/
|
|
|
101
101
|
|
|
102
102
|
#### 08-workflow-evals
|
|
103
103
|
|
|
104
|
-
Projects MUST follow the eval folder structure defined in [agentme-edr-151](151-ai-eval-
|
|
104
|
+
Projects MUST follow the eval folder structure defined in [agentme-edr-151](151-ai-eval-standards.md) rule `01` and the eval script requirements defined in [agentme-edr-153](153-ai-eval-script.md). For LLM-as-judge scoring used in workflow verification nodes and evals, see [agentme-edr-151](151-ai-eval-standards.md) rule `02`.
|
|
105
105
|
|
|
106
106
|
#### 09-node-naming-conventions
|
|
107
107
|
|
|
@@ -207,7 +207,7 @@ All workflow elements MUST maintain naming coherence as defined in [agentme-edr-
|
|
|
207
207
|
- [agentme-edr-126](126-pragmatic-hexagonal-architecture.md) — Adapter/application layer separation that defines the project layout
|
|
208
208
|
- [agentme-edr-103](103-python-project-tooling.md) — Python project tooling and structure
|
|
209
209
|
- [agentme-edr-201](../data/201-ml-dataset-structure.md) — ML dataset structure for eval datasets
|
|
210
|
-
- [agentme-edr-151](151-ai-eval-
|
|
210
|
+
- [agentme-edr-151](151-ai-eval-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract for all tiers (rule `02`)
|
|
211
211
|
- [agentme-edr-153](153-ai-eval-script.md) — AI eval script: entry-first loop, `--type` filtering, `mock_fixtures`, and MLflow conventions
|
|
212
212
|
- [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
|
|
213
213
|
- [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: REPEAT_COUNT loop, scoring methods, and release cadence
|
package/.xdrs/agentme/edrs/application/{151-ai-eval-core-standards.md → 151-ai-eval-standards.md}
RENAMED
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: agentme-edr-policy-151-ai-eval-
|
|
2
|
+
name: agentme-edr-policy-151-ai-eval-standards
|
|
3
3
|
description: Defines the eval folder structure and Makefile interface for AI projects, and the LLM-as-judge binary scoring contract applicable to all AI tiers and test types. Use when scaffolding evals or implementing an LLM judge. For eval script requirements see agentme-edr-153. For report format see agentme-edr-154. For repeatability see agentme-edr-155. For when evals are required see agentme-edr-501 rule 09-ai-project-testing-requirements. For the test type taxonomy see agentme-edr-152.
|
|
4
4
|
apply-to: Python AI projects (LLM, Agent, or Workflow tier) that implement eval testing
|
|
5
5
|
valid-from: 2026-06-05
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# agentme-edr-policy-151: AI eval
|
|
8
|
+
# agentme-edr-policy-151: AI eval standards
|
|
9
9
|
|
|
10
10
|
## Context and Problem Statement
|
|
11
11
|
|
|
@@ -104,7 +104,7 @@ A component may satisfy reproducibility (temperature = 0) yet still need repeata
|
|
|
104
104
|
## References
|
|
105
105
|
|
|
106
106
|
- [agentme-edr-201](../data/201-ml-dataset-structure.md) — Golden dataset file layout, per-entry JSON format, `$schema` pointer, and schema-lint validation
|
|
107
|
-
- [agentme-edr-151](151-ai-eval-
|
|
107
|
+
- [agentme-edr-151](151-ai-eval-standards.md) — AI eval core standards: eval folder structure and Makefile targets (rule `01`); LLM-as-judge binary scoring contract (rule `02`)
|
|
108
108
|
- [agentme-edr-153](153-ai-eval-script.md) — AI eval script: `--type` filtering, entry-first loop, `mock_fixtures`, threshold enforcement, and MLflow conventions
|
|
109
109
|
- [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: per-type `report-<type>.md` that consumes this taxonomy's test types
|
|
110
110
|
- [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: `REPEAT_COUNT` loop exception, scoring constants (`EVAL_MIN_ACCURACY_REPEATABILITY`, `REPEAT_SEMANTIC_SIMILARITY_SCORE`), scoring methods, `repeatability_accuracy` MLflow metric, report shape, and run cadence
|
|
@@ -126,7 +126,7 @@ The MLflow **experiment** is scoped to the eval scenario: `<component>/<eval-nam
|
|
|
126
126
|
|
|
127
127
|
## References
|
|
128
128
|
|
|
129
|
-
- [agentme-edr-151](151-ai-eval-
|
|
129
|
+
- [agentme-edr-151](151-ai-eval-standards.md) — AI eval core standards: eval folder structure (rule `01`) and LLM-as-judge binary scoring contract (rule `02`)
|
|
130
130
|
- [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: `report-<type>.md` template, Wilson CI, and convergence analysis
|
|
131
131
|
- [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: loop exception to rule `01`'s entry-first constraint, scoring methods, and cadence
|
|
132
132
|
- [agentme-edr-152](152-ai-test-types-taxonomy.md) — AI test types taxonomy: `test_types` enum, golden dataset entry envelope (including `mock_fixtures`), and mocking constraints per type
|
|
@@ -168,4 +168,4 @@ xychart-beta
|
|
|
168
168
|
|
|
169
169
|
- [agentme-edr-153](153-ai-eval-script.md) — AI eval script: the script that produces these reports (rule `01`)
|
|
170
170
|
- [agentme-edr-155](155-ai-eval-repeatability.md) — AI eval repeatability: rule `02` defines the adapted report shape for `report-repeatability.md`
|
|
171
|
-
- [agentme-edr-151](151-ai-eval-
|
|
171
|
+
- [agentme-edr-151](151-ai-eval-standards.md) — AI eval core standards: folder structure (rule `01`) and LLM-as-judge binary scoring (rule `02`)
|
|
@@ -34,7 +34,7 @@ Entries whose `test_types` includes `repeatability` are exempt from [agentme-edr
|
|
|
34
34
|
**Choosing the scoring method:** Two approaches are supported, declared as a constant in `eval.py`:
|
|
35
35
|
|
|
36
36
|
- **Semantic-similarity:** Embed all `REPEAT_COUNT` outputs for an entry into vectors and compute the average pairwise cosine similarity. The entry passes (score = 1) if the average similarity ≥ `REPEAT_SEMANTIC_SIMILARITY_SCORE`; otherwise it fails (score = 0). Use for classification or short structured outputs.
|
|
37
|
-
- **LLM-as-judge:** Provide all `REPEAT_COUNT` outputs for an entry to an LLM judge (kept at low/zero temperature) that returns 0 (fail) or 1 (pass) directly, based on whether the outputs are sufficiently consistent. No `REPEAT_SEMANTIC_SIMILARITY_SCORE` constant is needed. Use for free-text or complex structured outputs where vector distance is an unreliable proxy for agreement. The judge MUST follow [agentme-edr-151](151-ai-eval-
|
|
37
|
+
- **LLM-as-judge:** Provide all `REPEAT_COUNT` outputs for an entry to an LLM judge (kept at low/zero temperature) that returns 0 (fail) or 1 (pass) directly, based on whether the outputs are sufficiently consistent. No `REPEAT_SEMANTIC_SIMILARITY_SCORE` constant is needed. Use for free-text or complex structured outputs where vector distance is an unreliable proxy for agreement. The judge MUST follow [agentme-edr-151](151-ai-eval-standards.md) rule `02`'s binary output contract.
|
|
38
38
|
|
|
39
39
|
**`repeatability_accuracy`:** the fraction of repeatability entries that received PASS (score = 1), logged to MLflow as `repeatability_accuracy`. The eval exits non-zero if `repeatability_accuracy` < `EVAL_MIN_ACCURACY_REPEATABILITY`. Both `repeatability_accuracy` and `repeat_count` MUST be logged to MLflow and included in `report-repeatability.md` (rule `02`).
|
|
40
40
|
|
|
@@ -69,7 +69,7 @@ Because `repeatability` entries multiply real LLM-provider calls by `REPEAT_COUN
|
|
|
69
69
|
## References
|
|
70
70
|
|
|
71
71
|
- [agentme-edr-153](153-ai-eval-script.md) — AI eval script: rule `01` defines the entry-first constraint this policy exempts for repeatability entries
|
|
72
|
-
- [agentme-edr-151](151-ai-eval-
|
|
72
|
+
- [agentme-edr-151](151-ai-eval-standards.md) — AI eval core standards: rule `02` defines the LLM-as-judge binary output contract used by the LLM-as-judge scoring method in rule `01`
|
|
73
73
|
- [agentme-edr-154](154-ai-eval-report-format.md) — AI eval report format: rule `01` defines the base report template that `report-repeatability.md` adapts
|
|
74
74
|
- [agentme-edr-152](152-ai-test-types-taxonomy.md) — AI test types taxonomy: `repeatability` test type definition and disambiguation from `reproducibility` (rule `09`)
|
|
75
75
|
- [agentme-edr-501](../governance/501-project-quality-standards.md) — Project quality standards: Workflow eval cadence (rule `09`) that repeatability runs align with
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
---
|
|
2
|
-
name:
|
|
2
|
+
name: 010-select-relevant-xdrs
|
|
3
3
|
description: >
|
|
4
4
|
Analyzes a client repository, extracts the full agentme XDR set, and excludes the records that do
|
|
5
5
|
not fit the project's structure and workflow needs. Activate this skill when the user asks to
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
---
|
|
2
|
-
name:
|
|
2
|
+
name: 050-create-javascript-project
|
|
3
3
|
description: >
|
|
4
4
|
Scaffolds the initial boilerplate structure for a JavaScript/TypeScript project following
|
|
5
5
|
the standard tooling and layout defined in agentme-edr-101. Activate this skill when the user
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
---
|
|
2
|
-
name:
|
|
2
|
+
name: 051-create-golang-project
|
|
3
3
|
description: >
|
|
4
4
|
Scaffolds the initial boilerplate structure for a Go (Golang) CLI or library project following
|
|
5
5
|
the standard tooling and layout defined in agentme-edr-102. Activate this skill when the user
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
---
|
|
2
|
-
name:
|
|
2
|
+
name: 052-create-python-project
|
|
3
3
|
description: >
|
|
4
4
|
Scaffolds the initial boilerplate structure for a Python project following the standard tooling
|
|
5
5
|
and layout defined in agentme-edr-103. Activate this skill when the user asks to create,
|
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: 200-run-skill-tests
|
|
3
|
+
description: >
|
|
4
|
+
Executes the SKILL.test.md test scenarios for a given skill, evaluates each assertion against the skill's
|
|
5
|
+
actual output, and reports a pass/fail result per scenario. Activate when the user asks to test, verify,
|
|
6
|
+
or validate a skill, or before merging a PR that modifies a skill or its SKILL.test.md.
|
|
7
|
+
metadata:
|
|
8
|
+
author: flaviostutz
|
|
9
|
+
version: "1.0"
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
## Overview
|
|
13
|
+
|
|
14
|
+
Loads `SKILL.test.md` from a skill directory, runs each scenario by invoking the target skill with the specified trigger, evaluates every assertion against the output, and produces a structured test report.
|
|
15
|
+
|
|
16
|
+
## Instructions
|
|
17
|
+
|
|
18
|
+
### Phase 1: Locate and Validate SKILL.test.md
|
|
19
|
+
|
|
20
|
+
1. Accept the skill path from the user (e.g., `.agents/skills/001-review` or `.xdrs/_local/adrs/principles/skills/005-my-skill`). If no path is given, ask: *"Which skill do you want to test? Provide the path to the skill directory."*
|
|
21
|
+
2. Resolve the path to the directory containing `SKILL.md`. If the path is a symlink, follow it to the real directory.
|
|
22
|
+
3. Check that `SKILL.test.md` exists in the same directory as `SKILL.md`. If it is missing, output:
|
|
23
|
+
|
|
24
|
+
```
|
|
25
|
+
ERROR: SKILL.test.md not found in [resolved path].
|
|
26
|
+
Create it following agentme-edr-policy-017 before running tests.
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Then halt — do not proceed.
|
|
30
|
+
4. Read `SKILL.test.md` in full. Parse the frontmatter (`skill`, `skill-version`) and all `### Scenario N:` sections. Each scenario has three sub-sections: `**Trigger / Input**`, `**Expected Behaviour**`, and `**Assertions**`.
|
|
31
|
+
5. Read `SKILL.md` in full to understand what the skill does. Verify that the `skill-version` in `SKILL.test.md` matches the `version` field in `SKILL.md` metadata. If they differ, output a WARNING and continue.
|
|
32
|
+
|
|
33
|
+
### Phase 2: Execute Scenarios
|
|
34
|
+
|
|
35
|
+
For each scenario in order:
|
|
36
|
+
|
|
37
|
+
1. Record the scenario title and number.
|
|
38
|
+
2. Present the trigger/input to a fresh agent context with the target skill loaded, exactly as written in `**Trigger / Input**`. Do not add context beyond what the trigger specifies.
|
|
39
|
+
3. Capture the full output from the skill execution.
|
|
40
|
+
4. Move to Phase 3 for this scenario before executing the next.
|
|
41
|
+
|
|
42
|
+
### Phase 3: Evaluate Assertions
|
|
43
|
+
|
|
44
|
+
For each assertion in the current scenario:
|
|
45
|
+
|
|
46
|
+
1. Read the assertion text.
|
|
47
|
+
2. Determine whether the captured output satisfies the assertion. Apply the following rules:
|
|
48
|
+
- An assertion is PASS if the output unambiguously satisfies the stated condition.
|
|
49
|
+
- An assertion is FAIL if the output clearly does not satisfy the condition.
|
|
50
|
+
- An assertion is INCONCLUSIVE if the output is ambiguous with respect to the condition; treat INCONCLUSIVE as FAIL and note the reason.
|
|
51
|
+
3. Record the result (PASS / FAIL) and, for FAIL/INCONCLUSIVE, a one-sentence explanation referencing the specific output evidence.
|
|
52
|
+
|
|
53
|
+
### Phase 4: Report Results
|
|
54
|
+
|
|
55
|
+
After all scenarios are executed, produce the report using this template exactly:
|
|
56
|
+
|
|
57
|
+
```
|
|
58
|
+
## Skill Test Report: [skill-name] v[skill-version]
|
|
59
|
+
Tested: [ISO date]
|
|
60
|
+
|
|
61
|
+
### Scenario 1: [Title]
|
|
62
|
+
- Assertion 1: [PASS|FAIL] — [one-line reason if FAIL]
|
|
63
|
+
- Assertion 2: [PASS|FAIL] — [one-line reason if FAIL]
|
|
64
|
+
...
|
|
65
|
+
Overall: [PASS|FAIL]
|
|
66
|
+
|
|
67
|
+
### Scenario 2: [Title]
|
|
68
|
+
...
|
|
69
|
+
|
|
70
|
+
---
|
|
71
|
+
## Summary
|
|
72
|
+
- Scenarios: [total]
|
|
73
|
+
- Passed: [count]
|
|
74
|
+
- Failed: [count]
|
|
75
|
+
- Outcome: [PASS — all scenarios passed | FAIL — [N] scenario(s) failed]
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
A scenario passes only when every one of its assertions passes. The overall outcome is PASS only when every scenario passes.
|
|
79
|
+
|
|
80
|
+
### Constraints
|
|
81
|
+
|
|
82
|
+
- MUST NOT modify the skill or its test file.
|
|
83
|
+
- MUST execute scenarios in the order they appear in `SKILL.test.md`.
|
|
84
|
+
- MUST treat INCONCLUSIVE assertions as FAIL.
|
|
85
|
+
- MUST halt and report ERROR if `SKILL.test.md` is absent (Phase 1).
|
|
86
|
+
- MUST follow the report template exactly; do not add commentary outside the template.
|
|
87
|
+
|
|
88
|
+
## Examples
|
|
89
|
+
|
|
90
|
+
**Input**: "Test the skill at `.agents/skills/001-review`"
|
|
91
|
+
|
|
92
|
+
- Phase 1 resolves the symlink → `.xdrs/_core/adrs/principles/skills/001-review/`
|
|
93
|
+
- Reads `SKILL.test.md`, parses 3 scenarios
|
|
94
|
+
- Executes each scenario with the target skill loaded
|
|
95
|
+
- Reports per-assertion PASS/FAIL and an overall outcome
|
|
96
|
+
|
|
97
|
+
**Input**: "Verify `.xdrs/agentme/edrs/application/skills/050-setup-project` before merging"
|
|
98
|
+
|
|
99
|
+
- Phase 1 reads the real directory (no symlink)
|
|
100
|
+
- If `SKILL.test.md` is missing → ERROR and halt
|
|
101
|
+
- Otherwise proceeds through all phases
|
|
102
|
+
|
|
103
|
+
## Edge Cases
|
|
104
|
+
|
|
105
|
+
- If the skill path does not exist, output `ERROR: skill directory not found at [path].` and halt.
|
|
106
|
+
- If `SKILL.test.md` contains no scenarios, output `ERROR: SKILL.test.md has no scenarios.` and halt.
|
|
107
|
+
- If the target skill fails to activate (e.g., not registered in VS Code), note this in the report as FAIL with reason "skill could not be activated" and continue to remaining scenarios.
|
|
108
|
+
- If `skill-version` in `SKILL.test.md` does not match `SKILL.md`, emit a WARNING at the top of the report but do not halt.
|
|
109
|
+
|
|
110
|
+
## References
|
|
111
|
+
|
|
112
|
+
- [`agentme-edr-policy-017`](../../../principles/017-skill-testing.md) — Skill testing mandate and SKILL.test.md format specification
|
|
113
|
+
- [`_core-adr-policy-003`](../../../../../_core/adrs/principles/003-skill-standards.md) — Skill package standards and folder layout
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
---
|
|
2
|
+
skill: 200-run-skill-tests
|
|
3
|
+
skill-version: "1.0"
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
## Test Scenarios
|
|
7
|
+
|
|
8
|
+
### Scenario 1: Successfully executes scenarios and reports results
|
|
9
|
+
|
|
10
|
+
**Trigger / Input**
|
|
11
|
+
You are an agent with the `200-run-skill-tests` skill loaded. The workspace has a skill at `.agents/skills/001-review` with both `SKILL.md` and `SKILL.test.md` present. The user says:
|
|
12
|
+
|
|
13
|
+
"Test the skill at `.agents/skills/001-review`"
|
|
14
|
+
|
|
15
|
+
**Expected Behaviour**
|
|
16
|
+
1. Skill resolves the path to the real directory (following the symlink).
|
|
17
|
+
2. Skill confirms `SKILL.test.md` is present.
|
|
18
|
+
3. Skill reads `SKILL.test.md` in full and parses all scenarios.
|
|
19
|
+
4. Skill reads `SKILL.md` and verifies `skill-version` matches.
|
|
20
|
+
5. Skill executes each scenario by presenting the trigger to the target skill.
|
|
21
|
+
6. Skill evaluates every assertion for each scenario and records PASS or FAIL.
|
|
22
|
+
7. Skill outputs a structured report using the mandated template with a final Outcome line.
|
|
23
|
+
|
|
24
|
+
**Assertions**
|
|
25
|
+
- [ ] Output contains `## Skill Test Report: 001-review`.
|
|
26
|
+
- [ ] Output contains a `### Scenario` section for each scenario in `SKILL.test.md`.
|
|
27
|
+
- [ ] Each scenario section lists individual assertion results as `PASS` or `FAIL`.
|
|
28
|
+
- [ ] Output ends with a `## Summary` block containing `Scenarios:`, `Passed:`, `Failed:`, and `Outcome:`.
|
|
29
|
+
|
|
30
|
+
### Scenario 2: Halts with ERROR when SKILL.test.md is missing
|
|
31
|
+
|
|
32
|
+
**Trigger / Input**
|
|
33
|
+
You are an agent with the `200-run-skill-tests` skill loaded. The workspace has a skill directory `.xdrs/agentme/edrs/application/skills/050-create-javascript-project` that contains `SKILL.md` but no `SKILL.test.md`. The user says:
|
|
34
|
+
|
|
35
|
+
"Run tests for `.xdrs/agentme/edrs/application/skills/050-create-javascript-project`"
|
|
36
|
+
|
|
37
|
+
**Expected Behaviour**
|
|
38
|
+
1. Skill resolves the path to the directory.
|
|
39
|
+
2. Skill checks for `SKILL.test.md` and finds it absent.
|
|
40
|
+
3. Skill immediately outputs an ERROR message referencing the missing file and the policy.
|
|
41
|
+
4. Skill does NOT proceed to execute any scenarios.
|
|
42
|
+
|
|
43
|
+
**Assertions**
|
|
44
|
+
- [ ] Output contains `ERROR` indicating `SKILL.test.md` was not found.
|
|
45
|
+
- [ ] Output references the resolved path of the skill directory.
|
|
46
|
+
- [ ] Output does NOT contain any `### Scenario` execution section.
|
|
47
|
+
- [ ] Output does NOT contain a `## Summary` report block.
|
|
@@ -266,5 +266,5 @@ AI projects are classified into three tiers — LLM, Agent, and Workflow — def
|
|
|
266
266
|
- Evals MUST be executed before every release.
|
|
267
267
|
- Accuracy below project-defined thresholds MUST block the release. Thresholds MUST be documented in the eval Makefile or README.
|
|
268
268
|
- Evals MUST run against real LLM providers (not mocks) to capture model drift.
|
|
269
|
-
- For eval folder structure and LLM-as-judge scoring, see [agentme-edr-151](../application/151-ai-eval-
|
|
269
|
+
- For eval folder structure and LLM-as-judge scoring, see [agentme-edr-151](../application/151-ai-eval-standards.md). For eval script requirements, see [agentme-edr-153](../application/153-ai-eval-script.md).
|
|
270
270
|
- For the taxonomy of AI test types (safety, responsible-AI, quality-eval, prompt, code-level) and the golden dataset entry format, see [agentme-edr-152](../application/152-ai-test-types-taxonomy.md).
|
|
@@ -67,4 +67,4 @@ The content SHOULD stay concise and practical; do not turn `CONTRIBUTING.md` int
|
|
|
67
67
|
## References
|
|
68
68
|
|
|
69
69
|
- [agentme-edr-301 - Monorepo structure](../platform/301-monorepo-structure.md)
|
|
70
|
-
- [
|
|
70
|
+
- [053-monorepo-setup skill](../platform/skills/053-monorepo-setup/SKILL.md)
|
|
@@ -10,6 +10,7 @@ Foundational standards, principles, and guidelines.
|
|
|
10
10
|
|
|
11
11
|
- [agentme-edr-012](principles/012-continuous-xdr-enrichment.md) - **Continuous xdr improvement policy** - Promote recurring delivery lessons into reusable XDRs
|
|
12
12
|
- [agentme-edr-016](principles/016-cross-language-module-structure.md) - **Cross-language module structure** - Organize modules consistently across supported languages
|
|
13
|
+
- [agentme-edr-017](principles/017-skill-testing.md) - **skill testing** - Mandates a `SKILL.test.md` co-located with every skill in scopes that follow agentme; defines test scenario format (trigger, expected behaviour, assertions) and requires execution before merging any skill change *(includes skill: [200-run-skill-tests](application/skills/200-run-skill-tests/SKILL.md))*
|
|
13
14
|
|
|
14
15
|
## Articles
|
|
15
16
|
|
|
@@ -26,15 +27,16 @@ Language and framework-specific tooling and project structure.
|
|
|
26
27
|
- [agentme-edr-123](application/123-error-handling.md) - **Error handling** - Standardize explicit errors, logging, and propagation rules
|
|
27
28
|
- [agentme-edr-124](application/124-secrets-management.md) - **Secrets management** - Handle secrets securely using native keychains and cloud secret managers
|
|
28
29
|
- [agentme-edr-125](application/125-coding-abstraction-practices.md) - **Coding abstraction practices** - Define when abstractions are justified and when they must be inlined
|
|
30
|
+
- [agentme-edr-127](application/127-external-system-adapter-skills.md) - **External system adapter skills** - Priority-ordered approach and adapter skill authoring standards for automating interactions with external systems
|
|
29
31
|
|
|
30
32
|
### Language and framework tooling
|
|
31
33
|
|
|
32
|
-
- [agentme-edr-101](application/101-javascript-project-tooling.md) - **JavaScript project tooling and structure** - Scaffold JavaScript libraries with the standard toolchain *(includes skill: [
|
|
33
|
-
- [agentme-edr-102](application/102-golang-project-tooling.md) - **Go project tooling and structure** - Scaffold Go CLIs and libraries with the standard layout *(includes skill: [
|
|
34
|
-
- [agentme-edr-103](application/103-python-project-tooling.md) - **Python project tooling and structure** - Scaffold Python packages and CLIs with the standard layout *(includes skill: [
|
|
34
|
+
- [agentme-edr-101](application/101-javascript-project-tooling.md) - **JavaScript project tooling and structure** - Scaffold JavaScript libraries with the standard toolchain *(includes skill: [050-create-javascript-project](application/skills/050-create-javascript-project/SKILL.md))*
|
|
35
|
+
- [agentme-edr-102](application/102-golang-project-tooling.md) - **Go project tooling and structure** - Scaffold Go CLIs and libraries with the standard layout *(includes skill: [051-create-golang-project](application/skills/051-create-golang-project/SKILL.md))*
|
|
36
|
+
- [agentme-edr-103](application/103-python-project-tooling.md) - **Python project tooling and structure** - Scaffold Python packages and CLIs with the standard layout *(includes skill: [052-create-python-project](application/skills/052-create-python-project/SKILL.md))*
|
|
35
37
|
- [agentme-edr-104](application/104-cli-tool-standards.md) - **CLI tool standards** - Define command UX and behavior for CLI tools
|
|
36
38
|
- [agentme-edr-126](application/126-pragmatic-hexagonal-architecture.md) - **Pragmatic hexagonal architecture** - Organize application layers as External/Adapters/Application with practical coupling rules
|
|
37
|
-
- [
|
|
39
|
+
- [010-select-relevant-xdrs](application/skills/010-select-relevant-xdrs/SKILL.md) - **Select relevant XDRs**
|
|
38
40
|
|
|
39
41
|
### AI development
|
|
40
42
|
|
|
@@ -51,7 +53,7 @@ Standards for building LLM, Agent, and Workflow components.
|
|
|
51
53
|
|
|
52
54
|
Standards for eval datasets, scripts, reports, and test type taxonomy.
|
|
53
55
|
|
|
54
|
-
- [agentme-edr-151](application/151-ai-eval-
|
|
56
|
+
- [agentme-edr-151](application/151-ai-eval-standards.md) - **AI eval core standards** - Eval folder structure and Makefile interface; LLM-as-judge binary scoring contract applicable to all AI tiers and test types
|
|
55
57
|
- [agentme-edr-152](application/152-ai-test-types-taxonomy.md) - **AI test types taxonomy** - Names AI test types (`functional`, `safety`, `smoke`, `repeatability`, `adversarial`, `fairness`, `bias`, and 5 others) with group, objective, mocking constraint, and relevance, and defines the shared golden dataset entry envelope
|
|
56
58
|
- [agentme-edr-153](application/153-ai-eval-script.md) - **AI eval script** - eval.py requirements: entry-first loop, --type filtering, mock_fixtures wiring, human entries, threshold enforcement, and MLflow experiment conventions
|
|
57
59
|
- [agentme-edr-154](application/154-ai-eval-report-format.md) - **AI eval report format** - report-<type>.md template, Wilson score confidence interval, convergence analysis, and human-type checklist artifact
|
|
@@ -67,7 +69,7 @@ Data layer implementation and data management decisions.
|
|
|
67
69
|
|
|
68
70
|
Infrastructure implementation, delivery pipeline, and developer environment decisions.
|
|
69
71
|
|
|
70
|
-
- [agentme-edr-301](platform/301-monorepo-structure.md) - **Monorepo structure** - Standardize monorepo layout, tooling, and package boundaries *(includes skill: [
|
|
72
|
+
- [agentme-edr-301](platform/301-monorepo-structure.md) - **Monorepo structure** - Standardize monorepo layout, tooling, and package boundaries *(includes skill: [053-monorepo-setup](platform/skills/053-monorepo-setup/SKILL.md))*
|
|
71
73
|
- [agentme-edr-302](platform/302-github-pipelines.md) - **GitHub CI/CD pipelines** - Define required CI stages and workflow structure
|
|
72
74
|
- [agentme-edr-303](platform/303-common-targets.md) - **Common development script names** - Reuse standard build, lint, and test target names
|
|
73
75
|
- [agentme-edr-304](platform/304-tool-execution-and-scripting.md) - **Tool execution and scripting** - Run tools consistently across shells, Makefiles, and CI
|
|
@@ -17,7 +17,7 @@ What monorepo structure, naming conventions, tooling, and build standards should
|
|
|
17
17
|
|
|
18
18
|
**Adopt a standardized monorepo layout with top-level application folders that aggregate independent module roots, shared parent-level example and test areas, Mise-managed tooling, and Makefiles at every level.**
|
|
19
19
|
|
|
20
|
-
For step-by-step scaffolding instructions see [skill
|
|
20
|
+
For step-by-step scaffolding instructions see [skill 053-monorepo-setup](skills/053-monorepo-setup/SKILL.md).
|
|
21
21
|
Module folder responsibilities, artifact locations, and test-folder conventions follow [agentme-edr-016](../principles/016-cross-language-module-structure.md).
|
|
22
22
|
|
|
23
23
|
### Details
|
|
@@ -58,7 +58,7 @@ In SDD, specifications describe the feature being built; XDRs describe reusable
|
|
|
58
58
|
|
|
59
59
|
## References
|
|
60
60
|
|
|
61
|
-
- [_core-adr-001](../../../_core/adrs/principles/001-xdrs-
|
|
61
|
+
- [_core-adr-001](../../../_core/adrs/principles/001-xdrs-standards.md)
|
|
62
62
|
- [_core-article-001](../../../_core/adrs/principles/articles/001-xdrs-overview.md)
|
|
63
63
|
- [agentme-article-001](articles/001-continuous-xdr-improvement.md)
|
|
64
64
|
- [002-write-policy skill](../../../_core/adrs/principles/skills/002-write-policy/SKILL.md)
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: agentme-edr-policy-017-skill-testing
|
|
3
|
+
description: >
|
|
4
|
+
Mandates a SKILL.test.md test file co-located with every skill, defines the test file format (scenarios with trigger, expected behaviour, and assertions), and requires execution before merging any change to the skill. Use when creating, updating, or reviewing skills.
|
|
5
|
+
apply-to: contributors working in any scope that directly or transitively follows or extends agentme
|
|
6
|
+
valid-from: 2026-08-06
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# agentme-edr-policy-017: skill testing
|
|
10
|
+
|
|
11
|
+
## Context and Problem Statement
|
|
12
|
+
|
|
13
|
+
Skills evolve over time: phases are reworded, routing tables change, constraints are added. Without an executable specification of what the skill must do, regressions are silent — a changed skill may produce incorrect outputs or miss required steps and nothing catches it until a user files a complaint.
|
|
14
|
+
|
|
15
|
+
How should skill correctness be verified after every change, in a way that is consistent, co-located with the skill, and executable by either a human or an agent?
|
|
16
|
+
|
|
17
|
+
## Decision Outcome
|
|
18
|
+
|
|
19
|
+
**Every skill MUST have a `SKILL.test.md` file in the same directory as its `SKILL.md`. The file defines test scenarios with explicit input triggers and falsifiable assertions. It MUST be executed — using the `200-run-skill-tests` skill or manually — before merging any PR that modifies the skill or its test file.**
|
|
20
|
+
|
|
21
|
+
### Details
|
|
22
|
+
|
|
23
|
+
#### 01-mandatory-presence
|
|
24
|
+
|
|
25
|
+
A skill MUST have a `SKILL.test.md` file in the same directory as `SKILL.md`. The absence of `SKILL.test.md` is treated as an ERROR during any review of the skill directory.
|
|
26
|
+
|
|
27
|
+
#### 02-file-format
|
|
28
|
+
|
|
29
|
+
`SKILL.test.md` MUST follow this structure exactly:
|
|
30
|
+
|
|
31
|
+
```markdown
|
|
32
|
+
---
|
|
33
|
+
skill: [skill-name matching the name: field in SKILL.md frontmatter]
|
|
34
|
+
skill-version: "[x.y matching the version in SKILL.md metadata]"
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## Test Scenarios
|
|
38
|
+
|
|
39
|
+
### Scenario N: [Short Title]
|
|
40
|
+
|
|
41
|
+
**Trigger / Input**
|
|
42
|
+
[Exact prompt or agent context given to activate and exercise the skill. Must be
|
|
43
|
+
specific enough that two independent agents produce comparable outputs.]
|
|
44
|
+
|
|
45
|
+
**Expected Behaviour**
|
|
46
|
+
[Numbered list of steps the skill must perform, derived from its Instructions section.]
|
|
47
|
+
|
|
48
|
+
**Assertions**
|
|
49
|
+
- [ ] [Specific, falsifiable check on the output or behaviour. Start with a verb.]
|
|
50
|
+
- [ ] ...
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Rules:
|
|
54
|
+
- MUST contain at least two scenarios: one happy path and one edge or failure case.
|
|
55
|
+
- Each scenario MUST have at least two assertions.
|
|
56
|
+
- Assertions MUST be falsifiable (a pass/fail determination must be possible without ambiguity).
|
|
57
|
+
- Assertion text MUST start with a verb ("Output contains …", "Skill asks …", "Review reports …").
|
|
58
|
+
- MUST NOT duplicate SKILL.md content; reference phases by name only when needed.
|
|
59
|
+
- `skill-version` in frontmatter MUST be updated whenever `version` in SKILL.md changes.
|
|
60
|
+
|
|
61
|
+
#### 03-execution-requirement
|
|
62
|
+
|
|
63
|
+
`SKILL.test.md` MUST be executed before merging any PR that modifies the skill or its test file. Execution means running each scenario and verifying all assertions pass.
|
|
64
|
+
|
|
65
|
+
Use the `200-run-skill-tests` skill to execute the file, or run each scenario manually when automation is not available.
|
|
66
|
+
|
|
67
|
+
|
|
68
|
+
## Considered Options
|
|
69
|
+
|
|
70
|
+
- **Inline test section in SKILL.md** — rejected because it mixes specification and verification, inflating file size past the 6500-word limit and making test-only changes noisy in diffs.
|
|
71
|
+
- **External test registry** — rejected because co-location is the simplest discoverability model and matches the skill folder convention already established by `_core-adr-policy-003`.
|
|
72
|
+
- **Required only for new skills** — rejected because existing skills carry the same regression risk after every edit.
|
|
73
|
+
|
|
74
|
+
## References
|
|
75
|
+
|
|
76
|
+
- [`_core-adr-policy-003`](../../../_core/adrs/principles/003-skill-standards.md) — Skill package standards and folder layout
|
|
77
|
+
- [`200-run-skill-tests`](../application/skills/200-run-skill-tests/SKILL.md) — Runner skill that executes `SKILL.test.md` scenarios
|
|
@@ -87,7 +87,7 @@ If the same clarification would likely be needed in another feature, by another
|
|
|
87
87
|
|
|
88
88
|
## References
|
|
89
89
|
|
|
90
|
-
- [_core-adr-001](../../../../_core/adrs/principles/001-xdrs-
|
|
90
|
+
- [_core-adr-001](../../../../_core/adrs/principles/001-xdrs-standards.md) - XDR structure, numbering, and mandatory template
|
|
91
91
|
- [_core-article-001](../../../../_core/adrs/principles/articles/001-xdrs-overview.md) - XDR introduction and general adoption guidance
|
|
92
92
|
- [agentme-edr-012](../012-continuous-xdr-enrichment.md) - Shared-first XDR enrichment policy and 80% coverage target
|
|
93
93
|
- [002-write-policy skill](../../../../_core/adrs/principles/skills/002-write-policy/SKILL.md) - Step-by-step procedure for drafting new XDRs
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agentme",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.30.0",
|
|
4
4
|
"description": "",
|
|
5
5
|
"dependencies": {
|
|
6
6
|
"filedist": "^0.39.0"
|
|
@@ -18,6 +18,6 @@
|
|
|
18
18
|
"url": "https://github.com/flaviostutz/agentme.git"
|
|
19
19
|
},
|
|
20
20
|
"devDependencies": {
|
|
21
|
-
"xdrs-core": "^0.
|
|
21
|
+
"xdrs-core": "^0.45.0"
|
|
22
22
|
}
|
|
23
23
|
}
|