agentme 0.21.0 → 0.22.1

package/.xdrs/agentme/edrs/application/018-ai-llm-development-standards.md

@@ -171,6 +171,46 @@ prompt = PromptTemplate.from_file(
 )
 ```
 
+#### 06-output-length-constraints
+
+Every free-text field generated by an LLM MUST have an explicit word or token limit defined wherever the content is specified — in prompt text, output schema definitions, or both. This prevents runaway verbosity, reduces token costs, and makes quality evaluation deterministic.
+
+**Rules:**
+
+- Append `[max N words]` (or `[max N tokens]` when appropriate) directly inside the instruction or field description that requests the generated content.
+- Apply the constraint to every level: the top-level prompt instruction AND any nested schema field that contains free text.
+- When a field is an enumeration or a short code (e.g., `"APPROVE"` / `"REJECT"`), no word limit is needed.
+- Inline examples are encouraged whenever the expected output style is non-obvious.
+
+**Prompt example:**
+
+```text
+Generate a summary of this text. [max 40 words]
+Identify the three main topics covered. [max 10 words each]
+```
+
+**Output schema example:**
+
+```python
+class EvaluationResult(BaseModel):
+    evaluation: str = Field(description="Most important aspects of the text [max 100 words]")
+    verdict: Literal["PASS", "FAIL"]
+    improvement_suggestion: str = Field(description="Concrete next step for the author [max 30 words]")
+```
+
+**Combined prompt + schema example:**
+
+```python
+prompt = """
+Evaluate the following document against our quality criteria. [max 200 words total]
+
+Return a JSON object with:
+- "evaluation": overall assessment [max 100 words]
+- "verdict": "PASS" or "FAIL"
+- "improvement_suggestion": one concrete improvement [max 30 words]
+"""
+```
+
 ## References
 
 - [agentme-edr-019](019-ai-agents-development-standards.md) — Agent implementation standards (deepagents, tool-invocation loops)

package/.xdrs/agentme/edrs/application/019-ai-agents-development-standards.md

@@ -197,6 +197,58 @@ The current OS is: [operating system name].
 | `<OUTPUT_FORMAT>` | Required | MUST include a concrete schema or templated example; do not leave it vague. When multiple output formats are possible, MUST use mandatory language to specify exactly which one to use and explicitly exclude the others. |
 | `<WORKFLOW_CONTEXT>` | Conditional | MUST be omitted for standalone agents. MUST be present when the agent runs as a node inside a LangGraph workflow. |
 
+#### 07-agent-output-format
+
+The format of an agent's final output MUST be chosen based on who or what consumes it:
+
+| Consumer | Required format |
+|---|---|
+| Code (workflow node, downstream function, API caller) | JSON object matching a declared schema |
+| Human (user-facing summary, review comment, prose report) | Natural language; JSON is NOT required |
+
+**When output is consumed by code:**
+
+- The agent's `<OUTPUT_FORMAT>` system prompt section MUST specify a JSON schema or a concrete typed example.
+- The `<OUTPUT_FORMAT>` section MUST include the instruction: `ALWAYS return valid JSON. NEVER include prose outside the JSON block.`
+- The corresponding Python return type MUST be a typed dataclass or Pydantic model that mirrors the schema, not a plain `dict` or `str`.
+- The caller MUST parse and validate the JSON against the declared type before passing it downstream.
+
+**Example — code-consumed agent output:**
+
+```python
+from dataclasses import dataclass
+from typing import List
+import json
+
+@dataclass
+class FileAnalysisResult:
+    status: str          # "success" | "failure"
+    files_found: int
+    issues: List[str]
+
+# System prompt OUTPUT_FORMAT section:
+# <OUTPUT_FORMAT>
+# Respond with a JSON object matching this schema:
+# {
+#   "status": "success" | "failure",
+#   "files_found": <integer>,
+#   "issues": ["<string>", ...],
+#   "summary": "summary of the work performed. max 30 words",
+# }
+# ALWAYS return valid JSON. NEVER include prose outside the JSON block.
+# </OUTPUT_FORMAT>
+
+def parse_agent_output(raw: str) -> FileAnalysisResult:
+    data = json.loads(raw)
+    return FileAnalysisResult(**data)
+```
+
+**When output is consumed by a human:**
+
+- Natural language is correct and preferred.
+- Do NOT wrap prose in JSON — it adds noise without value.
+- The `<OUTPUT_FORMAT>` section MUST still describe the expected structure (e.g., a list of findings, a summary paragraph) using a prose template.
+
 ## References
 
 - [agentme-edr-018](018-ai-llm-development-standards.md) — LLM development standards (LangChain configuration, mocking patterns)

package/.xdrs/agentme/edrs/application/028-ai-eval-standards.md

@@ -30,8 +30,8 @@ evals/
   <component>/           # the component being evaluated (e.g., workflow-x, agent-y, model-z)
     eval-<name>/
       dataset/           # EDR-024 compliant dataset (README.md, dataset.schema.json, data/)
-      eval-<name>.py     # evaluation script
-      eval-<name>-report.md     # generated report (overwritten on each run — see rule 03)
+      eval.py            # evaluation script
+      report.md          # generated report (overwritten on each run — see rule 03)
       Makefile           # eval and run targets
     eval-<name2>/
       ...
@@ -62,13 +62,13 @@ eval:
 
 #### 02-eval-script-requirements
 
-Each `eval-<name>.py` script MUST:
+Each `eval.py` script MUST:
 
 - Load the dataset from `dataset/` in the same eval folder, following [agentme-edr-024](024-ml-dataset-structure.md). For input/output pairs, use the JSONL format per `agentme-edr-024.04-complex-structured-datasets-must-use-jsonl`.
 - Run every input through the live component against **real LLM providers** (not mocked responses), to capture model drift.
 - Log per-sample and aggregate metrics to an MLflow experiment that runs **locally** — a remote MLflow server MUST NOT be required.
 - Compare outputs to expected values using project-defined quality thresholds. Thresholds MUST be declared explicitly (e.g., in a Makefile variable or README).
-- Write `eval-<name>-report.md` in the same folder per rule `03-eval-report-file`.
+- Write `report.md` in the same folder per rule `03-eval-report-file`.
 - Exit with a non-zero status when any metric falls below its defined threshold, consistent with [agentme-edr-007](../principles/007-project-quality-standards.md) rule `07-statistical-models-must-have-eval-targets`.
 
 **Example:**
@@ -96,7 +96,7 @@ with mlflow.start_run() as run:
 
 #### 03-eval-report-file
 
-Each eval script MUST produce `eval-<name>-report.md` in the same `evals/<component>/eval-<name>/` folder and overwrite it on every run.
+Each eval script MUST produce `report.md` in the same `evals/<component>/eval-<name>/` folder and overwrite it on every run.
 
 **Generation constraint:** The report MUST be produced programmatically, reading raw metric values directly from MLflow. No LLM or generative model may write, summarize, or paraphrase any section of the report, to prevent hallucinated metric values.
 
@@ -107,7 +107,7 @@ The report MUST follow this template:
 
 **Date:** <ISO date>
 **Dataset:** dataset/
-**Script:** eval-<name>.py
+**Script:** eval.py
 **Thresholds:** accuracy ≥ <value>, F1 ≥ <value>
 
 ## Overall Results
@@ -142,14 +142,14 @@ $$\frac{\hat{p} + \frac{z^2}{2n} \pm z\sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac
 
 Where $\hat{p}$ is observed accuracy and $n$ is sample count. Accuracy and F1 are required; precision and recall are recommended.
 
-**Filled-in example** (`evals/workflow-document-review/eval-basic/eval-basic-report.md` for a document review workflow):
+**Filled-in example** (`evals/workflow-document-review/eval-basic/report.md` for a document review workflow):
 
 ```markdown
 # Eval Report: eval-basic
 
 **Date:** 2026-06-12
 **Dataset:** dataset/
-**Script:** eval-basic.py
+**Script:** eval.py
 **Thresholds:** accuracy ≥ 0.85, F1 ≥ 0.80
 
 ## Overall Results
@@ -182,6 +182,12 @@ Where $\hat{p}$ is observed accuracy and $n$ is sample count. Accuracy and F1 ar
 - MLflow run: experiment `workflow-document-review/eval-basic` — view with `mlflow ui`
 ```
 
+#### 04-eval-mlflow-unique-port
+
+Each `evals/<component>/eval-<name>/Makefile` MUST start its MLflow tracking server on a **unique port** to prevent conflicts when multiple eval Makefiles are run concurrently or in parallel (e.g., in CI or across multiple terminal sessions).
+
+Ports MUST be statically assigned per eval scenario and MUST NOT reuse the default `5000` port (reserved for `dev-mlflow` per [agentme-edr-008](../devops/008-common-targets.md) rule `09-ai-project-dev-targets`). Assign ports starting at `5100` and incrementing by 1 for each additional eval scenario across the entire project.
+
 ## References
 
 - [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09-ai-project-testing-requirements`) and statistical model eval targets (rule `07-statistical-models-must-have-eval-targets`)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agentme",
-  "version": "0.21.0",
+  "version": "0.22.1",
   "description": "",
   "dependencies": {
     "filedist": "^0.36.0"