agentme 0.17.0 → 0.19.0

package/.xdrs/agentme/edrs/application/019-ai-agents-development-standards.md

@@ -127,16 +127,18 @@ When multiple agents are needed:
 Every agent system prompt MUST follow this XML-section template. Sections must appear in this order. Required sections must always be present; optional sections may be omitted when they genuinely do not apply; never reorder them.
 
 ```xml
+[specific task description to the agent. if not defined use the default prompt "Execute your objective taking into consideration the inputs provided and all the sections described below"]
+
 <OBJECTIVE>
 [A one or two-sentence summary of the agent's main deliverable.
 e.g.: Split the incoming file list into logical batches for parallel processing.]
 </OBJECTIVE>
 
-<ROLE>
+<AGENT_ROLE>
 [Defines who the agent is, its area of expertise, and its core persona.
 If running inside a workflow, define exactly which node in WORKFLOW_CONTEXT this agent corresponds to.
 e.g.: You are the batch_planning_agent (see WORKFLOW_CONTEXT). You are an expert at partitioning large file sets into balanced, directory-aware batches.]
-</ROLE>
+</AGENT_ROLE>
 
 <INPUT>
 [All inputs for this agent invocation. For standalone agents: list only the agent-specific inputs. For workflow agents: list workflow-level inputs shared across all agents first, then agent-specific inputs such as judge outcomes, counters, or intermediate results from upstream nodes.]
@@ -160,31 +162,39 @@ e.g.: Do not call any tools. All reasoning is done in-context using the INPUT fi
 <!-- Optional: include when hard constraints need to be stated explicitly -->
 <GUARDRAILS>
 [Hard, non-negotiable constraints the agent must never violate.
+All constraints MUST use mandatory language: MUST, MUST NOT, NEVER, ALWAYS, SHALL, SHALL NOT.
 e.g.: NEVER create batches with fewer than 5 or more than 20 files. NEVER split files from the same directory across different batches unless unavoidable.]
 </GUARDRAILS>
 
 <OUTPUT_FORMAT>
 [A templated example or JSON schema specifying exactly how the final response should look.
-e.g.: Respond with a JSON object matching this schema: ...]
+When multiple output formats are possible, use mandatory language (MUST, ALWAYS, NEVER) to specify exactly which format to use and exclude the others.
+e.g.: Respond with a JSON object matching this schema: ... ALWAYS return valid JSON. NEVER include prose outside the JSON block.]
 </OUTPUT_FORMAT>
 
 <!-- Workflow-only: omit this section for standalone (non-workflow) agents -->
 <WORKFLOW_CONTEXT>
 [A detailed prose or diagram description of the overall workflow graph so the agent understands its role within the larger flow. Reference the specific node name that maps to this agent. Include enough detail about upstream and downstream nodes so the agent can reason about its context.]
 </WORKFLOW_CONTEXT>
+
+<SYSTEM_CONTEXT>
+The current date/time is [now in ISO 8601 format].
+The current OS is: [operating system name].
+</SYSTEM_CONTEXT>
 ```
 
 **Rules:**
 
 | Section | Required? | Notes |
 |---|---|---|
+| `<SYSTEM_CONTEXT>` | Optional | Runtime environment context injected at invocation time (e.g., current date/time in ISO 8601, OS). Include whenever the agent may need temporal or environment awareness. |
 | `<OBJECTIVE>` | Required | One or two sentences summarising the agent's main deliverable. |
 | `<ROLE>` | Required | Agent persona and expertise. When inside a workflow, MUST reference its node name from `<WORKFLOW_CONTEXT>`. |
 | `<INPUT>` | Required | List ALL inputs. For workflow agents: workflow-level inputs first, then agent-specific inputs. |
 | `<STEPS>` | Optional | Include when the agent follows a non-trivial numbered sequence of steps. |
 | `<TOOL_GUIDANCE>` | Optional | Include when tool use order or conditions need explicit direction. |
-| `<GUARDRAILS>` | Optional | Hard constraints that must never be violated. |
-| `<OUTPUT_FORMAT>` | Required | MUST include a concrete schema or templated example; do not leave it vague. |
+| `<GUARDRAILS>` | Optional | Hard constraints that must never be violated. All constraint statements MUST use mandatory language: MUST, MUST NOT, NEVER, ALWAYS, SHALL, SHALL NOT. |
+| `<OUTPUT_FORMAT>` | Required | MUST include a concrete schema or templated example; do not leave it vague. When multiple output formats are possible, MUST use mandatory language to specify exactly which one to use and explicitly exclude the others. |
 | `<WORKFLOW_CONTEXT>` | Conditional | MUST be omitted for standalone agents. MUST be present when the agent runs as a node inside a LangGraph workflow. |
 
 ## References

package/.xdrs/agentme/edrs/application/021-ai-workflow-development-standards.md

@@ -107,12 +107,13 @@ Eval folder structure and script requirements are defined in [agentme-edr-028](0
 
 LangGraph node names MUST follow a suffix convention that communicates the node's role at a glance. Names MUST be action-oriented and descriptive.
 
-| Suffix | Node type | When to use |
+| Convention | Node type | When to use |
 |---|---|---|
-| `_llm` | LLM call | Any node whose primary action is a direct LLM inference call (see [agentme-edr-018](018-ai-llm-development-standards.md)) |
-| `_step` | Algorithmic step | Deterministic logic with no LLM involvement (transformation, validation, routing) |
-| `_tool` | Tool/API call | A node that wraps a single external tool or API (e.g. a REST endpoint, DB query) |
-| `_agent` | Subgraph agent | A node that invokes a nested subgraph containing its own tool-invocation cycle and LLM calls; use the **deepagents** library for these nodes (see [agentme-edr-019](019-ai-agents-development-standards.md)) |
+| suffix `_llm` | LLM call | Any node whose primary action is a direct LLM inference call (see [agentme-edr-018](018-ai-llm-development-standards.md)) |
+| suffix `_step` | Algorithmic step | Deterministic logic with no LLM involvement (transformation, validation, routing) |
+| suffix `_tool` | Tool/API call | A node that wraps a single external tool or API (e.g. a REST endpoint, DB query) |
+| suffix `_agent` | Subgraph agent | A node that invokes a nested subgraph containing its own tool-invocation cycle and LLM calls; use the **deepagents** library for these nodes (see [agentme-edr-019](019-ai-agents-development-standards.md)) |
+| prefix `evaluate_` | Judge node | A node that evaluates the quality, correctness, completeness, or progress of prior outputs and returns a structured verdict; MUST follow rule `13-judge-node-output-format` |
 
 The Python function implementing the node SHOULD share the same name as the node alias passed to `add_node`, so that graph definitions and stack traces remain unambiguous:
 
@@ -131,6 +132,8 @@ graph.add_node("code_reviewer_agent", code_reviewer_agent)
 
 Names MUST NOT use generic labels such as `node1`, `process`, or `run`. Each name must clearly express what action the node performs.
 
+Judge nodes use a **prefix** convention instead of a suffix: the name MUST start with `evaluate_` followed by the subject being judged (e.g. `evaluate_progress`, `evaluate_quality`, `evaluate_completeness`, `evaluate_relevance`). This makes judge nodes immediately distinguishable from all other node types at a glance.
+
 #### 10-workflow-unit-testing
 
 All LLM calls within workflow nodes are external API calls and MUST be mocked in unit tests per [agentme-edr-018](018-ai-llm-development-standards.md) rule `04-unit-test-mocking`. Workflow unit tests must run fully offline with no real LLM provider calls.
@@ -222,6 +225,72 @@ Choose a name that summarises what the workflow consumes, processes, and produce
 
 **Bad names** (FORBIDDEN): `MainWorkflow`, `AgentGraph`, `ProcessFlow`, `Workflow1`, `RunGraph`.
 
+#### 13-judge-node-output-format
+
+Every node whose name starts with `evaluate_` (a judge node) MUST return a structured verdict object as its output. This ensures all judge nodes are interchangeable and their results can be uniformly consumed by downstream routing logic, logged, and compared across runs.
+
+**Required output schema:**
+
+```python
+from typing import Literal, Optional
+from dataclasses import dataclass, field
+
+FindingLevel = Literal["OK", "INFO", "WARNING", "ERROR"]
+
+@dataclass
+class JudgeFinding:
+    level: FindingLevel
+    # MUST: short action-oriented label; < 10 words
+    title: str
+    # MUST when level != "OK": why this is an issue; < 30 words
+    reason: Optional[str] = None
+    # MUST when level != "OK": notes/findings using mandatory (MUST) or advisory (SHOULD) language; < 400 words
+    details: Optional[str] = None
+    # OPTIONAL: possible fixes, only when directly inferrable from the finding without further analysis; < 200 words
+    fix: Optional[str] = None
+
+@dataclass
+class JudgeVerdict:
+    # MUST: highest severity level across all findings; "OK" only when every finding is "OK"
+    verdict: FindingLevel
+    # MUST: at least one finding present
+    findings: list[JudgeFinding] = field(default_factory=list)
+```
+
+Example (for logging, state storage, and inter-node communication):
+
+```json
+{
+  "verdict": "WARNING",
+  "findings": [
+    {
+      "level": "OK",
+      "title": "All required sections present"
+    },
+    {
+      "level": "WARNING",
+      "title": "Code coverage below threshold",
+      "reason": "Current coverage is 62%, minimum required is 80%.",
+      "details": "The following modules have no test coverage: auth.py, payments.py. SHOULD add unit tests for all public methods in these modules.",
+      "fix": "Add unit tests for auth.py and payments.py. Run `make test-coverage` to verify the threshold is met."
+    }
+  ]
+}
+```
+
+**Routing from judge nodes:**
+
+Downstream conditional edges MUST route on `verdict` only:
+
+```python
+def route_after_evaluate_quality(state) -> str:
+    if state["evaluate_quality_result"].verdict in ("ERROR", "WARNING"):
+        return "revise_draft_llm"
+    return "publish_step"
+```
+
+**Logging:** Log `verdict` and the count of each level as MLflow metrics on the current run per rule `03-observability-and-experiment-tracking`.
+
 #### 15-workflow-state-persistence
 
 For long-running workflows that may need to be paused and resumed:

package/.xdrs/agentme/edrs/application/028-ai-eval-standards.md

@@ -23,41 +23,47 @@ For when evals are required per AI tier, see [agentme-edr-007](../principles/007
 
 #### 01-eval-folder-structure
 
-For each AI component being evaluated (an LLM chain, agent, or workflow), create a corresponding directory under `evals/` at the same level as `lib/` and `examples/`:
+Each named eval is a self-contained unit. Create one directory per eval under `evals/` at the same level as `lib/` and `examples/`:
 
 ```text
 evals/
-  <component>/
-    Makefile                  # eval targets for this component
-    dataset_<group>/          # one folder per eval group (see agentme-edr-024)
-    eval_<group>.py           # evaluation script for each group
+  eval-<name>/
+    dataset/              # EDR-024 compliant dataset (README.md, dataset.schema.json, data/)
+    eval-<name>.py        # evaluation script
+    eval-report.md        # generated report (overwritten on each run — see rule 03)
+    Makefile              # eval and run targets
+  eval-<name2>/
+    ...
 ```
 
-Where `<component>` is the name of the LLM chain, agent, or workflow being evaluated (e.g., `summarizer`, `file_analyzer_agent`, `document_review_workflow`).
+Where `<name>` identifies the specific evaluation scenario (e.g., `eval-basic`, `eval-complex`, `eval-edge-cases`).
 
-The per-component `evals/<component>/Makefile` MUST define:
+The `dataset/` subfolder MUST be a valid [agentme-edr-024](024-ml-dataset-structure.md) dataset — it MUST include `README.md` and `dataset.schema.json` at its root. For input/output pairs, use JSONL files per `agentme-edr-024.04-complex-structured-datasets-must-use-jsonl`.
+
+Each `evals/eval-<name>/Makefile` MUST define:
 
 | Target | Behaviour |
 |---|---|
-| `eval` | Runs all eval groups for the component |
-| `eval-<group>` | Runs one named group (e.g. `eval-simple`, `eval-complex`) |
+| `eval` | Runs the eval with threshold enforcement; exits non-zero on failure (CI-safe) |
+| `run` | Runs the eval without threshold enforcement (exploration / debugging) |
 
-The module root Makefile MUST expose a `make eval` target that delegates to `eval` in every `evals/<component>/Makefile`:
+The module root Makefile MUST expose a `make eval` target that delegates to `eval` in every `evals/eval-<name>/Makefile`:
 
 ```makefile
 eval:
-	$(MAKE) -C evals/summarizer eval
-	$(MAKE) -C evals/document_review_workflow eval
+	$(MAKE) -C evals/eval-basic eval
+	$(MAKE) -C evals/eval-complex eval
 ```
 
 #### 02-eval-script-requirements
 
-Each `eval_<group>.py` script MUST:
+Each `eval-<name>.py` script MUST:
 
-- Load the dataset from `evals/<component>/dataset_<group>/` following [agentme-edr-024](024-ml-dataset-structure.md). For input/output pairs, use the JSONL format per `agentme-edr-024.04-complex-structured-datasets-must-use-jsonl`.
+- Load the dataset from `dataset/` in the same eval folder, following [agentme-edr-024](024-ml-dataset-structure.md). For input/output pairs, use the JSONL format per `agentme-edr-024.04-complex-structured-datasets-must-use-jsonl`.
 - Run every input through the live component against **real LLM providers** (not mocked responses), to capture model drift.
 - Log per-sample and aggregate metrics to an MLflow experiment that runs **locally** — a remote MLflow server MUST NOT be required.
 - Compare outputs to expected values using project-defined quality thresholds. Thresholds MUST be declared explicitly (e.g., in a Makefile variable or README).
+- Write `eval-report.md` in the same folder per rule `03-eval-report-file`.
 - Exit with a non-zero status when any metric falls below its defined threshold, consistent with [agentme-edr-007](../principles/007-project-quality-standards.md) rule `07-statistical-models-must-have-eval-targets`.
 
 **Example:**
@@ -68,19 +74,109 @@ from my_package.app.workflows.document_review_workflow.graph import graph
 
 EVAL_MIN_ACCURACY = 0.85
 
-with mlflow.start_run():
+with mlflow.start_run() as run:
     results = []
-    for sample in load_dataset("evals/document_review_workflow/dataset_basic/"):
+    for sample in load_dataset("dataset/"):
         output = graph.invoke({"document": sample["input"]})
         results.append(output["label"] == sample["expected_label"])
 
     accuracy = sum(results) / len(results)
     mlflow.log_metric("accuracy", accuracy)
 
+    write_eval_report(run, results, thresholds={"accuracy": EVAL_MIN_ACCURACY})
+
     if accuracy < EVAL_MIN_ACCURACY:
         raise SystemExit(f"Eval failed: accuracy {accuracy:.2f} < {EVAL_MIN_ACCURACY}")
 ```
 
+#### 03-eval-report-file
+
+Each eval script MUST produce `eval-report.md` in the same `evals/eval-<name>/` folder and overwrite it on every run.
+
+**Generation constraint:** The report MUST be produced programmatically, reading raw metric values directly from MLflow. No LLM or generative model may write, summarize, or paraphrase any section of the report, to prevent hallucinated metric values.
+
+The report MUST follow this template:
+
+```markdown
+# Eval Report: <name>
+
+**Date:** <ISO date>
+**Dataset:** dataset/
+**Script:** eval-<name>.py
+**Thresholds:** accuracy ≥ <value>, F1 ≥ <value>
+
+## Overall Results
+
+| Metric    | Value  | 95% CI         | Threshold | Status  |
+|-----------|--------|----------------|-----------|---------|
+| Accuracy  | <val>  | [<low>, <high>]| ≥ <thr>   | ✓/✗ PASS/FAIL |
+| F1 Score  | <val>  | —              | ≥ <thr>   | ✓/✗ PASS/FAIL |
+| Precision | <val>  | —              | —         | —       |
+| Recall    | <val>  | —              | —         | —       |
+| Samples   | <n>    | —              | —         | —       |
+
+**Overall: PASS / FAIL**
+
+## Per-item Results
+
+| ID  | Input Summary | Expected | Actual | Correct |
+|-----|---------------|----------|--------|---------|
+| 001 | <summary>     | <label>  | <label>| ✓       |
+| 002 | <summary>     | <label>  | <label>| ✗       |
+
+## Notes
+
+- <observations, failure patterns, MLflow run link>
+```
+
+**Confidence interval:** The 95% CI for accuracy MUST be computed using the **Wilson score interval** (preferred over the normal approximation for small $n$). A wide interval signals that the dataset is too small to support confident conclusions and the sample count should be increased.
+
+The Wilson score bounds at 95% confidence ($z = 1.96$) are:
+
+$$\frac{\hat{p} + \frac{z^2}{2n} \pm z\sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}$$
+
+Where $\hat{p}$ is observed accuracy and $n$ is sample count. Accuracy and F1 are required; precision and recall are recommended.
+
+**Filled-in example** (`evals/eval-basic/eval-report.md` for a document review workflow):
+
+```markdown
+# Eval Report: eval-basic
+
+**Date:** 2026-06-12
+**Dataset:** dataset/
+**Script:** eval-basic.py
+**Thresholds:** accuracy ≥ 0.85, F1 ≥ 0.80
+
+## Overall Results
+
+| Metric    | Value | 95% CI       | Threshold | Status      |
+|-----------|-------|--------------|-----------|-------------|
+| Accuracy  | 0.88  | [0.69, 0.97] | ≥ 0.85    | ✓ PASS      |
+| F1 Score  | 0.86  | —            | ≥ 0.80    | ✓ PASS      |
+| Precision | 0.89  | —            | —         | —           |
+| Recall    | 0.84  | —            | —         | —           |
+| Samples   | 25    | —            | —         | —           |
+
+**Overall: PASS**
+
+> Note: CI [0.69, 0.97] is wide — 25 samples may be insufficient for high confidence. Consider expanding the dataset.
+
+## Per-item Results
+
+| ID  | Input Summary                       | Expected | Actual   | Correct |
+|-----|-------------------------------------|----------|----------|---------|
+| 001 | Contract renewal, 3 pages, standard | approve  | approve  | ✓       |
+| 002 | NDA with unusual liability clause   | escalate | escalate | ✓       |
+| 003 | Vendor invoice, missing PO number   | reject   | reject   | ✓       |
+| 004 | Employment agreement, standard terms| approve  | approve  | ✓       |
+| 005 | Amendment with redlined IP clause   | escalate | approve  | ✗       |
+
+## Notes
+
+- Sample 005 misclassified: redlined IP clause not flagged as escalation trigger. Possible model drift.
+- MLflow run: experiment `eval_basic` — view with `mlflow ui`
+```
+
 ## References
 
 - [agentme-edr-007](../principles/007-project-quality-standards.md) — Project quality standards: when evals are required per AI tier (rule `09-ai-project-testing-requirements`) and statistical model eval targets (rule `07-statistical-models-must-have-eval-targets`)

package/.xdrs/agentme/edrs/principles/002-coding-best-practices.md

@@ -83,7 +83,28 @@ def _persist_order(order, total): ...
 
 ---
 
-#### 03-keep-readme-tests-and-examples-in-sync
+#### 03-put-entry-point-function-first
+
+Place the **entry-point function** (the outermost caller) at the **top** of the file. All helper or sub-functions it calls internally must appear **below** it.
+
+*Why:* Readers can follow the overall logic top-down without jumping around the file. The most important function is immediately visible when the file is opened.
+
+**Example (Python):**
+
+```python
+def process_order(order):          # entry point at the top
+    _validate_order(order)
+    total = _calculate_price(order)
+    _persist_order(order, total)
+
+def _validate_order(order): ...
+def _calculate_price(order) -> Decimal: ...
+def _persist_order(order, total): ...
+```
+
+---
+
+#### 04-keep-readme-tests-and-examples-in-sync
 
 Every change to a public interface, behavior, or configuration option must be reflected in:
 
@@ -95,7 +116,7 @@ Every change to a public interface, behavior, or configuration option must be re
 
 ---
 
-#### 04-declare-types-in-file-where-used
+#### 05-declare-types-in-file-where-used
 
 If a type (struct, interface, class, typedef, etc.) is used in only **one** file, declare it in that same file. Move a type to a shared module only when it is referenced in two or more files.
 
@@ -103,7 +124,7 @@ If a type (struct, interface, class, typedef, etc.) is used in only **one** file
 
 ---
 
-#### 05-keep-test-files-next-to-source
+#### 06-keep-test-files-next-to-source
 
 Where the language ecosystem supports it (e.g. JavaScript/TypeScript, Go, Rust), place test files **beside** the source file they cover and use a consistent naming convention rather than mirroring the source tree in a separate `tests/` folder.
 

package/.xdrs/index.md

@@ -25,4 +25,6 @@ Opiniated set of decisions and skills for common development tasks
 
 ### _local (reserved)
 
-Project-local XDRs that must not be shared with other contexts. Always keep this scope last so its decisions override or extend all scopes listed above. Keep `_local` canonical indexes in the workspace tree only; do not link them from this shared index. Readers and tools should still try to discover existing `_local` indexes in the current workspace by default.
+_local scope is the default scope for new xdrs and might override other scope decisions. These decisions are local and are not supposed to be shared in other contexts.
+
+Read _local scope index at `_local/index.md` when it exists.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agentme",
-  "version": "0.17.0",
+  "version": "0.19.0",
   "description": "",
   "dependencies": {
     "filedist": "^0.35.0"