@tea-agent/loop-agent 0.13.0-beta.0 → 0.13.0

package/docs/templates/backend-test-dag.retrospect.prompt.md

@@ -1,125 +1,139 @@
-# Backend Test DAG Retrospect Prompt Template
-
-## Purpose
-
-Use this prompt for a **test retrospective** node: `executor: "pi"`, `role: "closeout"`, `toolProfile: "write"`, `writePolicy: "exclusive"`. The closeout agent reads upstream review reports and pytest execution results, then generates a retrospective report with an objective maturity rating.
-
-Do **not** create a new executor type. This is a standard `executor: pi` writer node.
-
-## Recommended DAG Node Shape
-
-```json
-{
-  "id": "test-retrospect-pi",
-  "depends_on": ["execute-backend-pytest-shell"],
-  "complexity": "MED",
-  "executor": "pi",
-  "role": "closeout",
-  "toolProfile": "write",
-  "writePolicy": "exclusive",
-  "writeSet": ["docs/test-reports/**"],
-  "allowedPaths": ["docs/test-reports/**"],
-  "forbiddenPaths": [".harness/**", "artifacts/**"],
-  "outputContract": "Markdown retrospective report under docs/test-reports/ with coverage summary, review findings, pytest results, and maturity rating (A/B/C/D).",
-  "subtask_prompt_markdown": "./backend-test-dag.retrospect.prompt.md"
-}
-```
-
-## Prompt Body
-
-You are the Backend Test DAG **test retrospective** agent.
-
-Your job is to read upstream outputs (review report + pytest results) and generate a retrospective report with a maturity rating. Write the report under `docs/test-reports/` only. Stay within `writeSet`. Do not write root `artifacts/**`.
-
-### Output Steps (do in order)
-
-1. First, output the maturity rating on the first line: `Rating: A/B/C/D`
-2. Then write the full report under `docs/test-reports/`
-
-### Inputs
-
-1. **Review report** — `review-backend-cases-pi` output (VERDICT, findings, coverage assessment).
-2. **Pytest output** — `execute-backend-pytest-shell` stdout/stderr and exit code.
-3. **Machine-readable report** — `$HARNESS_DAG_RUN_DIR/reports/backend-test-junit.xml` (runner-owned evidence; use the path reported by `execute-backend-pytest-shell`).
-
-Do NOT re-read source documents. Use upstream outputs only.
-
-### Maturity Rating Criteria
-
-| Rating | Coverage | Pass Rate | Review Findings |
-|--------|----------|-----------|-----------------|
-| **A** | 100% acceptance criteria covered | 100% pytest pass | No Critical or Important findings |
-| **B** | ≥80% acceptance criteria covered | ≥90% pytest pass | Only Informational findings |
-| **C** | ≥60% acceptance criteria covered | ≥70% pytest pass | No Critical findings (Important allowed) |
-| **D** | Below C thresholds | Below C thresholds | Or any Critical finding unresolved |
-
-#### Rating Rules
-
-- **Coverage** = (acceptance criteria with ≥1 covering test case) / (total acceptance criteria) × 100%
-- **Pass rate** = (passed pytest functions) / (total non-skipped pytest functions) × 100%
-- If `review-backend-cases-pi` returned `VERDICT: request-revision` and revision was not completed, cap at **D**.
-- If pytest exit code is non-zero and >30% tests failed, cap at **D** regardless of coverage.
-- Skipped tests (`@pytest.mark.skip`) count as "not covered" for pass rate but not as failures.
-
-### Report Structure
-
-Write the report as a Markdown file named `backend-test-retrospect-<date>.md` under `docs/test-reports/`.
-
-```markdown
-# Backend Test Retrospective Report
-
-**Date:** <YYYY-MM-DD>
-**Task:** <task-id>
-**Maturity Rating:** <A|B|C|D>
-
-## 1. Test Coverage Summary
-
-| Metric | Value |
-|--------|-------|
-| Total acceptance criteria | N |
-| Covered by test cases | N (X%) |
-| Total functional test cases | N |
-| Positive path cases | N |
-| Negative path cases | N |
-| Boundary cases | N |
-
-## 2. Automation Results
-
-| Metric | Value |
-|--------|-------|
-| Total pytest functions | N |
-| Passed | N |
-| Failed | N |
-| Skipped | N |
-| Pass rate | X% |
-| Pytest exit code | N |
-
-### Failed Test Analysis
-
-| Test Case ID | Function | Failure Reason | Root Cause |
-|--------------|----------|----------------|------------|
-| ... | ... | ... | ... |
-
-## 3. Review Findings
-
-| Severity | Finding | Status |
-|----------|---------|--------|
-| Critical | ... | Resolved / Unresolved |
-| Important | ... | Resolved / Unresolved |
-| Informational | ... | Resolved / Unresolved |
-
-## 4. Maturity Rating Rationale
-
-Explain which threshold was met or missed, and why the specific rating was assigned.
-
-## 5. Recommendations
-
-- Actionable items for improving the rating in the next iteration.
-- Specific gaps to close (uncovered criteria, flaky tests, missing negative paths).
-```
-
-### Output Shape (after rating line)
-
-After the mandatory maturity rating line, provide a brief summary paragraph before writing the full report file.
-
-Do not include chain-of-thought. Do not write root `artifacts/**`.
+# Backend Test DAG Retrospect Prompt Template
+
+## Purpose
+
+Use this prompt for a **test retrospective** node: `executor: "pi"`, `role: "closeout"`, `toolProfile: "write"`, `writePolicy: "exclusive"`. The closeout agent reads **Backend Test Result v1** (and classification), then generates a retrospective report with an objective maturity rating.
+
+Do **not** create a new executor type. This is a standard `executor: pi` writer node.
+
+## Recommended DAG Node Shape
+
+```json
+{
+  "id": "test-retrospect-pi",
+  "depends_on": ["classify-backend-test-result-pi"],
+  "complexity": "MED",
+  "executor": "pi",
+  "role": "closeout",
+  "toolProfile": "write",
+  "writePolicy": "exclusive",
+  "writeSet": ["docs/test-reports/**"],
+  "allowedPaths": ["docs/test-reports/**"],
+  "forbiddenPaths": [".harness/**", "artifacts/**"],
+  "outputContract": "Markdown retrospective report under docs/test-reports/ with coverage summary, review findings, Result v1 stats, classification, and maturity rating (A/B/C/D).",
+  "subtask_prompt_markdown": "./backend-test-dag.retrospect.prompt.md"
+}
+```
+
+## Prompt Body
+
+You are the Backend Test DAG **test retrospective** agent.
+
+Your job is to read upstream Result v1 + classification (+ review report) and generate a retrospective report with a maturity rating. Write the report under `docs/test-reports/` only. Stay within `writeSet`. Do not write root `artifacts/**`.
+
+This node runs on **both pass and assertion-fail** paths (after parse + classify). Final task success is decided later by `backend-test-outcome-gate-shell` using Result v1 shell facts only — **never** rewrite a failed result as passed in this report.
+
+### Output Steps (do in order)
+
+1. First, output the maturity rating on the first line: `Rating: A/B/C/D`
+2. Then write the full report under `docs/test-reports/`
+
+### Inputs
+
+1. **Result v1 (authoritative stats)** — `$HARNESS_DAG_RUN_DIR/contracts/backend-test-result.json`
+   Use `passed` / `failed` / `error` / `skipped` / `outcome` / `failures[]` / `pytestExitCode` only from this artifact.
+2. **Case Manifest v1 (authoritative AC coverage)** — `$HARNESS_DAG_RUN_DIR/contracts/backend-test-case-manifest.json`
+   Use `coverageSummary.acCoverageRatio`, `coveredAcCount`, `explicitAcCount`, case counts only from this artifact.
+3. **Classification** — `classify-backend-test-result-pi` JSON (`category`, `confidence`, `evidence`). Interpretive only; does not override outcome.
+4. **Review report** — `review-backend-cases-pi` output (VERDICT, findings, coverage assessment).
+5. Optional secondary: execute stdout markers / JUnit path (do not re-parse logs for counts when Result v1 exists).
+
+Do NOT re-read source documents. Use upstream outputs only.
+
+### Stats authority
+
+- Pass rate = `passed / (passed + failed + error)` when denominator > 0 (skipped excluded from denominator unless Result documents otherwise) — **Result v1 only**.
+- AC coverage = `coverageSummary.acCoverageRatio` from Case Manifest v1 only (do **not** recompute or invent percentages).
+- Failed case table rows must match `failures[]` from Result v1.
+- If Result v1 `outcome` is not `passed`, the retrospective **must not** claim overall success.
+
+### Maturity Rating Criteria
+
+| Rating | Coverage (manifest) | Pass Rate (Result v1) | Review Findings |
+|--------|---------------------|------------------------|-----------------|
+| **A** | `acCoverageRatio` = 1 | 100% pytest pass | No Critical or Important findings |
+| **B** | `acCoverageRatio` ≥ 0.8 | ≥90% pytest pass | Only Informational findings |
+| **C** | `acCoverageRatio` ≥ 0.6 | ≥70% pytest pass | No Critical findings (Important allowed) |
+| **D** | Below C thresholds | Below C thresholds | Or any Critical finding unresolved |
+
+#### Rating Rules
+
+- **Coverage** from Case Manifest `coverageSummary` only (deterministic gate product).
+- **Pass rate** from Result v1 only (not guessed from logs).
+- If `review-backend-cases-pi` returned `VERDICT: request-revision` and revision was not completed, cap at **D**.
+- If Result v1 shows >30% failed+error among executed tests, cap at **D** regardless of coverage.
+- Skipped tests count as "not covered" for pass rate but not as failures.
+- Collection/command/report errors → cap at **D** and record classification (not ProductBug by default).
+
+### Report Structure
+
+Write the report as a Markdown file named `backend-test-retrospect-<date>.md` under `docs/test-reports/`.
+
+```markdown
+# Backend Test Retrospective Report
+
+**Date:** <YYYY-MM-DD>
+**Task:** <task-id>
+**Maturity Rating:** <A|B|C|D>
+**Result outcome:** <from Result v1>
+**Classification:** <from classify JSON>
+
+## 1. Test Coverage Summary
+
+| Metric | Value |
+|--------|-------|
+| Total acceptance criteria | N |
+| Covered by test cases | N (X%) |
+| Total functional test cases | N |
+
+## 2. Automation Results (from Result v1)
+
+| Metric | Value |
+|--------|-------|
+| Total tests | N |
+| Passed | N |
+| Failed | N |
+| Error | N |
+| Skipped | N |
+| Pass rate | X% |
+| Pytest exit code | N |
+| Outcome | … |
+| Execution status | … |
+
+### Failed Test Analysis
+
+| Test Case / Function | Message (truncated) | Classification |
+|----------------------|-------------------|----------------|
+| ... | ... | ... |
+
+## 3. Review Findings
+
+| Severity | Finding | Status |
+|----------|---------|--------|
+| … | … | … |
+
+## 4. Maturity Rating Rationale
+
+Explain which threshold was met or missed.
+
+## 5. Recommendations
+
+- Actionable items for the next iteration.
+- Do not propose changing production code solely to greenwash tests.
+```
+
+### Output Shape (after rating line)
+
+After the mandatory maturity rating line, provide a brief summary paragraph before writing the full report file.
+
+Do not include chain-of-thought. Do not write root `artifacts/**`.

package/docs/templates/backend-test-dag.review-cases.prompt.md

@@ -1,81 +1,83 @@
-# Backend Test DAG Review Cases Prompt Template
-
-## Purpose
-
-Use this prompt for a read-only **backend test case review** node: `executor: "pi"`, `role: "reviewer"`, `writePolicy: "read-only"`. The reviewer audits generated backend functional test cases for completeness, format compliance, and traceability to source requirements. Downstream `generate-backend-pytest-pi` depends on a `VERDICT: pass` to proceed.
-
-Do **not** create `executor: reviewer`. Reviewer is a **role** on `executor: pi`.
-
-## Recommended DAG Node Shape
-
-```json
-{
-  "id": "review-backend-cases-pi",
-  "depends_on": ["generate-backend-functional-cases-pi", "backend-test-analysis-contract-shell"],
-  "complexity": "HIGH",
-  "executor": "pi",
-  "role": "reviewer",
-  "writePolicy": "read-only",
-  "allowedPaths": ["**"],
-  "forbiddenPaths": [".harness/**", "artifacts/**"],
-  "outputContract": "Plain Markdown whose first non-empty line is VERDICT: pass or VERDICT: request-revision; followed by Findings and Coverage Assessment. No file writes.",
-  "subtask_prompt_markdown": "./backend-test-dag.review-cases.prompt.md"
-}
-```
-
-## Prompt Body
-
-You are the Backend Test DAG **test case reviewer** (read-only).
-
-Your job is to audit the generated backend functional test cases for completeness, format compliance, requirement coverage, and traceability. You are **not** an implementer or test generator. Do not edit repository files, including root `artifacts/**`.
-
-### Mandatory First Line
-
-The **first non-empty line** of your response must be exactly one of:
-
-- `VERDICT: pass`
-- `VERDICT: request-revision`
-
-No preamble, heading, or blank lines before the verdict line.
-
-### Inputs to Review
-
-1. **Acceptance criteria / analysis** — from the validated Backend Test Analysis v1 artifact materialized by `backend-test-analysis-contract-shell` (`contracts/backend-test-analysis.json` under the current DAG run). Do not treat free-form Markdown from `analyze-inputs-pi` as the contract.
-2. **Generated test cases** — files under `testcase/md/`.
-
-Do NOT re-read source documents. Use the validated analysis artifact and generated cases only.
-
-### Review Checklist
-
-| Area | Check | Severity if Missing |
-|------|-------|---------------------|
-| **ID format** | Every test case ID matches `BE-<MODULE>-<NNN>` (e.g. `BE-ORDER-001`) | Critical |
-| **Positive path coverage** | Happy-path scenarios for each acceptance criterion | Critical |
-| **Negative path coverage** | Error/exception scenarios (invalid input, not found, state violations) | Important |
-| **Boundary conditions** | Edge cases (empty input, max length, edge values) | Important |
-| **State transitions** | Illegal state changes covered | Important |
-| **Requirement traceability** | Each acceptance criterion (AC-xxx) maps to at least one test case ID | Critical |
-| **Case structure** | Each case has: ID, Title, Precondition, Steps, Expected Result | Important |
-| **No duplicate IDs** | All test case IDs are unique across files | Critical |
-
-### Conditional Coverage (check ONLY if mentioned in upstream analysis)
-
-- **Authentication coverage**: check ONLY if the validated analysis artifact mentions auth mechanism (JWT, OAuth2, API Key, etc.)
-- **Timeout coverage**: check ONLY if the validated analysis artifact mentions timeout handling or degradation strategy
-- If not mentioned in the validated analysis artifact, do NOT flag as missing
-
-### Verdict Rules
-
-| Condition | Verdict |
-|-----------|---------|
-| All Critical checks pass, Important checks have no more than 2 findings | `VERDICT: pass` |
-| Any Critical check fails | `VERDICT: request-revision` |
-| More than 2 Important findings | `VERDICT: request-revision` |
-| Only Informational findings | `VERDICT: pass` (with findings listed) |
-
-### Output Shape (after verdict line)
-
-1. **Coverage Assessment** — table mapping each AC to covering test case IDs (or "uncovered").
-2. **Findings** — bullet list tagged `Critical`, `Important`, or `Informational`.
-3. **Statistics** — total case count, positive/negative/boundary breakdown, module distribution.
-4. **Required revisions** (only when `request-revision`) — numbered items for the upstream generator to fix.
+# Backend Test DAG Review Cases Prompt Template
+
+## Purpose
+
+Use this prompt for a read-only **backend test case review** node: `executor: "pi"`, `role: "reviewer"`, `writePolicy: "read-only"`. The reviewer audits generated backend functional test cases for completeness, format compliance, and traceability to source requirements. Downstream `generate-backend-pytest-pi` depends on a `VERDICT: pass` to proceed.
+
+Do **not** create `executor: reviewer`. Reviewer is a **role** on `executor: pi`.
+
+## Recommended DAG Node Shape
+
+```json
+{
+  "id": "review-backend-cases-pi",
+  "depends_on": ["backend-test-case-manifest-shell", "backend-test-analysis-contract-shell"],
+  "complexity": "HIGH",
+  "executor": "pi",
+  "role": "reviewer",
+  "writePolicy": "read-only",
+  "allowedPaths": ["**"],
+  "forbiddenPaths": [".harness/**", "artifacts/**"],
+  "outputContract": "Plain Markdown whose first non-empty line is VERDICT: pass or VERDICT: request-revision; followed by Findings and Coverage Assessment. No file writes.",
+  "subtask_prompt_markdown": "./backend-test-dag.review-cases.prompt.md"
+}
+```
+
+## Prompt Body
+
+You are the Backend Test DAG **test case reviewer** (read-only).
+
+Your job is to audit the generated backend functional test cases for completeness, format compliance, requirement coverage, and traceability. You are **not** an implementer or test generator. Do not edit repository files, including root `artifacts/**`.
+
+### Mandatory First Line
+
+The **first non-empty line** of your response must be exactly one of:
+
+- `VERDICT: pass`
+- `VERDICT: request-revision`
+
+No preamble, heading, or blank lines before the verdict line.
+
+### Inputs to Review
+
+1. **Acceptance criteria / analysis** — from the validated Backend Test Analysis v1 artifact materialized by `backend-test-analysis-contract-shell` (`contracts/backend-test-analysis.json` under the current DAG run). Do not treat free-form Markdown from `analyze-inputs-pi` as the contract.
+2. **Case Manifest v1** — `contracts/backend-test-case-manifest.json` (schemaId `backend-test-case-manifest-v1`). Prefer `coverageSummary` and caseId↔acIds from this artifact; do not invent coverage percentages.
+3. **Generated test cases** — files under `testcase/md/`.
+
+Do NOT re-read source documents. Use the validated analysis artifact, case manifest, and generated cases only.
+
+### Review Checklist
+
+| Area | Check | Severity if Missing |
+|------|-------|---------------------|
+| **ID format** | Every test case ID matches `BE-<MODULE>-<NNN>` (e.g. `BE-ORDER-001`) | Critical |
+| **Positive path coverage** | Happy-path scenarios for each acceptance criterion | Critical |
+| **Negative path coverage** | Error/exception scenarios (invalid input, not found, state violations) | Important |
+| **Boundary conditions** | Edge cases (empty input, max length, edge values) | Important |
+| **State transitions** | Illegal state changes covered | Important |
+| **Requirement traceability** | Each acceptance criterion (AC-xxx) maps to at least one test case ID (manifest coverageSummary or evidenceGaps) | Critical |
+| **Manifest consistency** | Markdown cases align with Case Manifest v1 caseId/acIds | Critical |
+| **Case structure** | Each case has: ID, Title, Precondition, Steps, Expected Result | Important |
+| **No duplicate IDs** | All test case IDs are unique across files | Critical |
+
+### Conditional Coverage (check ONLY if mentioned in upstream analysis)
+
+- **Authentication coverage**: check ONLY if the validated analysis artifact mentions auth mechanism (JWT, OAuth2, API Key, etc.)
+- **Timeout coverage**: check ONLY if the validated analysis artifact mentions timeout handling or degradation strategy
+- If not mentioned in the validated analysis artifact, do NOT flag as missing
+
+### Verdict Rules
+
+| Condition | Verdict |
+|-----------|---------|
+| All Critical checks pass, Important checks have no more than 2 findings | `VERDICT: pass` |
+| Any Critical check fails | `VERDICT: request-revision` |
+| More than 2 Important findings | `VERDICT: request-revision` |
+| Only Informational findings | `VERDICT: pass` (with findings listed) |
+
+### Output Shape (after verdict line)
+
+1. **Coverage Assessment** — table mapping each AC to covering test case IDs (or "uncovered").
+2. **Findings** — bullet list tagged `Critical`, `Important`, or `Informational`.
+3. **Statistics** — total case count, positive/negative/boundary breakdown, module distribution.
+4. **Required revisions** (only when `request-revision`) — numbered items for the upstream generator to fix.

package/docs/templates/backend-test-execution.schema.json

@@ -0,0 +1,133 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://tea-agent.dev/schemas/backend-test-execution-v1.json",
+  "title": "Backend Test Execution v1",
+  "type": "object",
+  "additionalProperties": false,
+  "required": [
+    "schemaVersion",
+    "framework",
+    "runner",
+    "testRoot",
+    "workingDirectory",
+    "report",
+    "targetMode",
+    "existingFixtures",
+    "authenticationMode",
+    "requiredEnvNames",
+    "dataIsolation",
+    "evidenceGaps",
+    "evidenceRefs"
+  ],
+  "properties": {
+    "schemaVersion": { "const": 1 },
+    "framework": { "const": "pytest" },
+    "runner": {
+      "type": "object",
+      "additionalProperties": false,
+      "properties": {
+        "commandParts": {
+          "type": "array",
+          "items": { "type": "string", "minLength": 1 }
+        },
+        "frozenCommandHints": {
+          "type": "array",
+          "items": { "type": "string", "minLength": 1 }
+        }
+      }
+    },
+    "testRoot": {
+      "type": "string",
+      "minLength": 1,
+      "description": "Repo-relative posix path; no absolute form or .. segments"
+    },
+    "workingDirectory": {
+      "type": "string",
+      "minLength": 1,
+      "default": "."
+    },
+    "report": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["format", "relativeHint"],
+      "properties": {
+        "format": { "const": "junit" },
+        "relativeHint": { "type": "string", "minLength": 1 }
+      }
+    },
+    "targetMode": {
+      "enum": ["in-process", "external-running-service", "managed-command"]
+    },
+    "baseUrlEnvName": {
+      "type": "string",
+      "pattern": "^[A-Z_][A-Z0-9_]*$"
+    },
+    "readiness": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["path", "description"],
+        "properties": {
+          "path": { "type": "string", "minLength": 1 },
+          "description": { "type": "string", "minLength": 1 }
+        }
+      }
+    },
+    "existingFixtures": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["name", "sourcePath", "kind"],
+        "properties": {
+          "name": { "type": "string", "minLength": 1 },
+          "sourcePath": { "type": "string", "minLength": 1 },
+          "kind": { "type": "string", "minLength": 1 }
+        }
+      }
+    },
+    "authenticationMode": { "type": "string", "minLength": 1 },
+    "requiredEnvNames": {
+      "type": "array",
+      "items": {
+        "type": "string",
+        "pattern": "^[A-Z_][A-Z0-9_]*$"
+      }
+    },
+    "dataIsolation": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["mode"],
+      "properties": {
+        "mode": { "type": "string", "minLength": 1 },
+        "evidence": { "type": "string", "minLength": 1 }
+      }
+    },
+    "managedCommand": {
+      "type": "object",
+      "additionalProperties": false,
+      "properties": {
+        "start": { "type": "string", "minLength": 1 },
+        "stop": { "type": "string", "minLength": 1 },
+        "sourceRef": { "type": "string", "minLength": 1 }
+      }
+    },
+    "evidenceGaps": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["description"],
+        "properties": {
+          "description": { "type": "string", "minLength": 1 },
+          "sourceRef": { "type": "string", "minLength": 1 }
+        }
+      }
+    },
+    "evidenceRefs": {
+      "type": "array",
+      "items": { "type": "string", "minLength": 1 }
+    }
+  }
+}