agentme 0.20.1 → 0.22.0

package/.xdrs/agentme/edrs/application/018-ai-llm-development-standards.md

@@ -171,6 +171,46 @@ prompt = PromptTemplate.from_file(
 )
 ```
 
+#### 06-output-length-constraints
+
+Every free-text field generated by an LLM MUST have an explicit word or token limit defined wherever the content is specified — in prompt text, output schema definitions, or both. This prevents runaway verbosity, reduces token costs, and makes quality evaluation deterministic.
+
+**Rules:**
+
+- Append `[max N words]` (or `[max N tokens]` when appropriate) directly inside the instruction or field description that requests the generated content.
+- Apply the constraint to every level: the top-level prompt instruction AND any nested schema field that contains free text.
+- When a field is an enumeration or a short code (e.g., `"APPROVE"` / `"REJECT"`), no word limit is needed.
+- Inline examples are encouraged whenever the expected output style is non-obvious.
+
+**Prompt example:**
+
+```text
+Generate a summary of this text. [max 40 words]
+Identify the three main topics covered. [max 10 words each]
+```
+
+**Output schema example:**
+
+```python
+class EvaluationResult(BaseModel):
+    evaluation: str = Field(description="Most important aspects of the text [max 100 words]")
+    verdict: Literal["PASS", "FAIL"]
+    improvement_suggestion: str = Field(description="Concrete next step for the author [max 30 words]")
+```
+
+**Combined prompt + schema example:**
+
+```python
+prompt = """
+Evaluate the following document against our quality criteria. [max 200 words total]
+
+Return a JSON object with:
+- "evaluation": overall assessment [max 100 words]
+- "verdict": "PASS" or "FAIL"
+- "improvement_suggestion": one concrete improvement [max 30 words]
+"""
+```
+
 ## References
 
 - [agentme-edr-019](019-ai-agents-development-standards.md) — Agent implementation standards (deepagents, tool-invocation loops)

package/.xdrs/agentme/edrs/application/019-ai-agents-development-standards.md

@@ -197,6 +197,58 @@ The current OS is: [operating system name].
 | `<OUTPUT_FORMAT>` | Required | MUST include a concrete schema or templated example; do not leave it vague. When multiple output formats are possible, MUST use mandatory language to specify exactly which one to use and explicitly exclude the others. |
 | `<WORKFLOW_CONTEXT>` | Conditional | MUST be omitted for standalone agents. MUST be present when the agent runs as a node inside a LangGraph workflow. |
 
+#### 07-agent-output-format
+
+The format of an agent's final output MUST be chosen based on who or what consumes it:
+
+| Consumer | Required format |
+|---|---|
+| Code (workflow node, downstream function, API caller) | JSON object matching a declared schema |
+| Human (user-facing summary, review comment, prose report) | Natural language; JSON is NOT required |
+
+**When output is consumed by code:**
+
+- The agent's `<OUTPUT_FORMAT>` system prompt section MUST specify a JSON schema or a concrete typed example.
+- The `<OUTPUT_FORMAT>` section MUST include the instruction: `ALWAYS return valid JSON. NEVER include prose outside the JSON block.`
+- The corresponding Python return type MUST be a typed dataclass or Pydantic model that mirrors the schema, not a plain `dict` or `str`.
+- The caller MUST parse and validate the JSON against the declared type before passing it downstream.
+
+**Example — code-consumed agent output:**
+
+```python
+from dataclasses import dataclass
+from typing import List
+import json
+
+@dataclass
+class FileAnalysisResult:
+    status: str          # "success" | "failure"
+    files_found: int
+    issues: List[str]
+
+# System prompt OUTPUT_FORMAT section:
+# <OUTPUT_FORMAT>
+# Respond with a JSON object matching this schema:
+# {
+#   "status": "success" | "failure",
+#   "files_found": <integer>,
+#   "issues": ["<string>", ...],
+#   "summary": "summary of the work performed. max 30 words",
+# }
+# ALWAYS return valid JSON. NEVER include prose outside the JSON block.
+# </OUTPUT_FORMAT>
+
+def parse_agent_output(raw: str) -> FileAnalysisResult:
+    data = json.loads(raw)
+    return FileAnalysisResult(**data)
+```
+
+**When output is consumed by a human:**
+
+- Natural language is correct and preferred.
+- Do NOT wrap prose in JSON — it adds noise without value.
+- The `<OUTPUT_FORMAT>` section MUST still describe the expected structure (e.g., a list of findings, a summary paragraph) using a prose template.
+
 ## References
 
 - [agentme-edr-018](018-ai-llm-development-standards.md) — LLM development standards (LangChain configuration, mocking patterns)

package/.xdrs/agentme/edrs/application/028-ai-eval-standards.md

@@ -23,47 +23,52 @@ For when evals are required per AI tier, see [agentme-edr-007](../principles/007
 
 #### 01-eval-folder-structure
 
-Each named eval is a self-contained unit. Create one directory per eval under `evals/` at the same level as `lib/` and `examples/`:
+Evals are grouped first by the component being evaluated, then by the specific evaluation scenario. Create one directory per component under `evals/`, and one directory per eval scenario inside it. Place `evals/` at the same level as `lib/` and `examples/`:
 
 ```text
 evals/
-  eval-<name>/
-    dataset/              # EDR-024 compliant dataset (README.md, dataset.schema.json, data/)
-    eval-<name>.py        # evaluation script
-    eval-report.md        # generated report (overwritten on each run — see rule 03)
-    Makefile              # eval and run targets
-  eval-<name2>/
+  <component>/           # the component being evaluated (e.g., workflow-x, agent-y, model-z)
+    eval-<name>/
+      dataset/           # EDR-024 compliant dataset (README.md, dataset.schema.json, data/)
+      eval.py            # evaluation script
+      report.md          # generated report (overwritten on each run — see rule 03)
+      Makefile           # eval and run targets
+    eval-<name2>/
+      ...
+  <component2>/
     ...
 ```
 
-Where `<name>` identifies the specific evaluation scenario (e.g., `eval-basic`, `eval-complex`, `eval-edge-cases`).
+`<component>` MUST match the name of the component under evaluation and use lowercase hyphen-separated words (e.g., `workflow-document-review`, `agent-support`, `model-classifier`).
+
+`<name>` identifies the specific evaluation scenario using lowercase hyphen-separated words (e.g., `eval-basic`, `eval-complex`, `eval-edge-cases`, `eval-bias-test`).
 
 The `dataset/` subfolder MUST be a valid [agentme-edr-024](024-ml-dataset-structure.md) dataset — it MUST include `README.md` and `dataset.schema.json` at its root. For input/output pairs, use JSONL files per `agentme-edr-024.04-complex-structured-datasets-must-use-jsonl`.
 
-Each `evals/eval-<name>/Makefile` MUST define:
+Each `evals/<component>/eval-<name>/Makefile` MUST define:
 
 | Target | Behaviour |
 |---|---|
 | `eval` | Runs the eval with threshold enforcement; exits non-zero on failure (CI-safe) |
 | `run` | Runs the eval without threshold enforcement (exploration / debugging) |
 
-The module root Makefile MUST expose a `make eval` target that delegates to `eval` in every `evals/eval-<name>/Makefile`:
+The module root Makefile MUST expose a `make eval` target that delegates to `eval` in every `evals/<component>/eval-<name>/Makefile`:
 
 ```makefile
 eval:
-	$(MAKE) -C evals/eval-basic eval
-	$(MAKE) -C evals/eval-complex eval
+	$(MAKE) -C evals/workflow-document-review/eval-basic eval
+	$(MAKE) -C evals/workflow-document-review/eval-complex eval
 ```
 
 #### 02-eval-script-requirements
 
-Each `eval-<name>.py` script MUST:
+Each `eval.py` script MUST:
 
 - Load the dataset from `dataset/` in the same eval folder, following [agentme-edr-024](024-ml-dataset-structure.md). For input/output pairs, use the JSONL format per `agentme-edr-024.04-complex-structured-datasets-must-use-jsonl`.
 - Run every input through the live component against **real LLM providers** (not mocked responses), to capture model drift.
 - Log per-sample and aggregate metrics to an MLflow experiment that runs **locally** — a remote MLflow server MUST NOT be required.
 - Compare outputs to expected values using project-defined quality thresholds. Thresholds MUST be declared explicitly (e.g., in a Makefile variable or README).
-- Write `eval-report.md` in the same folder per rule `03-eval-report-file`.
+- Write `report.md` in the same folder per rule `03-eval-report-file`.
 - Exit with a non-zero status when any metric falls below its defined threshold, consistent with [agentme-edr-007](../principles/007-project-quality-standards.md) rule `07-statistical-models-must-have-eval-targets`.
 
 **Example:**
@@ -91,7 +96,7 @@ with mlflow.start_run() as run:
 
 #### 03-eval-report-file
 
-Each eval script MUST produce `eval-report.md` in the same `evals/eval-<name>/` folder and overwrite it on every run.
+Each eval script MUST produce `report.md` in the same `evals/<component>/eval-<name>/` folder and overwrite it on every run.
 
 **Generation constraint:** The report MUST be produced programmatically, reading raw metric values directly from MLflow. No LLM or generative model may write, summarize, or paraphrase any section of the report, to prevent hallucinated metric values.
 
@@ -102,7 +107,7 @@ The report MUST follow this template:
 
 **Date:** <ISO date>
 **Dataset:** dataset/
-**Script:** eval-<name>.py
+**Script:** eval.py
 **Thresholds:** accuracy ≥ <value>, F1 ≥ <value>
 
 ## Overall Results
@@ -137,14 +142,14 @@ $$\frac{\hat{p} + \frac{z^2}{2n} \pm z\sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac
 
 Where $\hat{p}$ is observed accuracy and $n$ is sample count. Accuracy and F1 are required; precision and recall are recommended.
 
-**Filled-in example** (`evals/eval-basic/eval-report.md` for a document review workflow):
+**Filled-in example** (`evals/workflow-document-review/eval-basic/report.md` for a document review workflow):
 
 ```markdown
 # Eval Report: eval-basic
 
 **Date:** 2026-06-12
 **Dataset:** dataset/
-**Script:** eval-basic.py
+**Script:** eval.py
 **Thresholds:** accuracy ≥ 0.85, F1 ≥ 0.80
 
 ## Overall Results
@@ -174,7 +179,7 @@ Where $\hat{p}$ is observed accuracy and $n$ is sample count. Accuracy and F1 ar
 ## Notes
 
 - Sample 005 misclassified: redlined IP clause not flagged as escalation trigger. Possible model drift.
-- MLflow run: experiment `eval_basic` — view with `mlflow ui`
+- MLflow run: experiment `workflow-document-review/eval-basic` — view with `mlflow ui`
 ```
 
 ## References

package/.xdrs/agentme/edrs/principles/007-project-quality-standards.md

@@ -118,13 +118,14 @@ Directory and file layout must be self-explanatory: source code, tests, configur
 
 #### 06-libraries-must-have-runnable-examples
 
-Projects that are libraries or shared utilities must include an `examples/` directory. Each subdirectory represents a usage scenario and must be independently runnable. Examples are executed as part of `make test`.
+Projects that are libraries or shared utilities must include an `examples/` directory. Each subdirectory represents a usage scenario and must be independently runnable. Examples that are "offline" (require no external credentials, no running servers, no paid APIs, and no environment-specific configuration outside the repository) must be executed as part of `make test`. Examples that depend on external entities may be left out of `make test`.
 
 **Requirements:**
 - `examples/` must contain at least one subdirectory per major usage scenario
 - Each scenario subdirectory must have a `Makefile` with a `run` target
 - Examples must import the library as an external consumer (not via relative `../src` imports)
-- `make test` in the root must run all examples; failures block CI and releases
+- `make test` in the root must run all offline examples; failures block CI and releases
+- Examples that depend on external entities must not be included in `make test`
 
 **Directory layout:**
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agentme",
-  "version": "0.20.1",
+  "version": "0.22.0",
   "description": "",
   "dependencies": {
     "filedist": "^0.36.0"