@mmerterden/multi-agent-pipeline 20.27.2 → 20.28.0

package/pipeline/multi-agent-refs/test-gen/stages.md

@@ -0,0 +1,227 @@
+# test-gen stages
+
+<!-- toc -->
+- [Artefacts and gates](#artefacts-and-gates)
+- [Read-only stages](#read-only-stages)
+- [1. Inventory](#1-inventory)
+- [Priority](#priority)
+- [2. Discovery](#2-discovery)
+- [3. Gaps](#3-gaps)
+- [4. Handles](#4-handles)
+- [5. Write](#5-write)
+- [6. Run](#6-run)
+- [7. Classify](#7-classify)
+- [8. Triage](#8-triage)
+- [End: the change base](#end-the-change-base)
+<!-- /toc -->
+
+The contract behind `/multi-agent:test-gen`. Each stage is a subagent with one
+artefact under `$RUN` (`$HOME/.claude/logs/test-gen/<repo>/<timestamp>/`), and
+no stage starts until the previous artefact has passed its gate. A stage that
+says it is done without its artefact on disk is not done.
+
+## Artefacts and gates
+
+| # | Stage | Runs as | Artefact | Gate |
+|---|---|---|---|---|
+| 1 | inventory | orchestrator, with the toolkit inventory tools | `inventory.json` | `test-gen-gate.mjs check --stage inventory`, `evidence-gate.mjs --claim artefact` |
+| - | priority | AskUserQuestion; autopilot takes the top three | `inventory.json` `priority` | `check --stage inventory` again |
+| 2 | discovery | `test-gen-analyst` (read-only) | `discovery.json` | `check --stage discovery`, `evidence-gate.mjs --claim artefact` |
+| 3 | gaps | `coverage_gap`, then `test-gen-analyst` (read-only) | `gaps.json` | `check --stage gaps`, `evidence-gate.mjs --claim artefact` |
+| 4 | handles | general-purpose subagent, product code, handles only | `handles.md` and the stage's delta | `test-gen-gate.mjs handles`, `handle-diff-audit.mjs --baseline` exit 0 |
+| 5 | write | general-purpose subagent, test files only | the test files and `written.json` | `test-gen-gate.mjs written --baseline` |
+| 6 | run | orchestrator | `run-<k>.log` and per-test reports | `evidence-gate.mjs --claim test` |
+| 7 | classify | orchestrator | `results.json` | `check --stage results` |
+| 8 | triage | orchestrator, result by result | `results.json` `triage` | `test-gen-state.mjs revise` |
+
+Before each stage take `STAGE_START=$(date -u +%Y-%m-%dT%H:%M:%SZ)`. The
+artefact gate (`--claim artefact --stage-start "$STAGE_START" --evidence <its own
+file>`) rejects a file written before the stage began and a file that is not
+whole (JSON that does not parse). It is applied to the JSON a model writes
+(inventory, discovery, gaps); the scripts write `results.json` and the
+baselines whole.
+
+The two stages that may edit the repository (handles, write) start with a
+snapshot: `test-gen-gate.mjs baseline` records the working tree, untracked files
+included, as a git tree without touching it or the index. Their gates read only
+what changed after it, so uncommitted work the person had before the stage is
+never taken for the stage's change, and never reverted to satisfy a gate.
+
+## Read-only stages
+
+Discovery and gaps are dispatched with `subagent_type: "test-gen-analyst"`
+(`agents/test-gen-analyst.md`). The persona's grant is an allowlist: Read, Grep,
+Glob and the read-only toolkit tools (`coverage_gap`, the `repo_*` readers, the
+`code_*` language-server queries). No Write, Edit, Bash, Agent or other MCP tool
+is granted, so an analysis stage cannot change the repository however it is
+prompted. It returns JSON; the orchestrator writes it with the Write tool.
+Copilot CLI installs the same persona with the same grant. Codex CLI has no
+per-agent tool grant: there, run the two stages in the main session and change
+nothing during them.
+
+## 1. Inventory
+
+Prefer a tool over reading: `design_scenario_inventory` (a screen's states from
+the design), `web_map` (a web app's routes), the UI tree (`ios_get_ui_tree`,
+`android_get_ui_tree`) of each reachable screen, or the OpenAPI spec. Without a
+tool, read the navigation and routes and set `source: "agent"`. Every item has a
+stable `id`: route, screen id, design node id, endpoint path.
+
+When a change base exists (`test-gen-state.mjs base-scan` measured), keep the
+items whose `path` is among the scan's `files` or `graph.affectedFiles`, and say
+how many fell away. A scan with `measured: false` narrows nothing: the inventory
+stays whole and the reason is reported.
+
+## Priority
+
+Show the items in one table (id, kind, title, existing tests from a quick grep)
+and ask which to cover (AskUserQuestion, multi-select, following
+`multi-agent-refs/picker-contract.md`). Autopilot asks nothing:
+it takes the three items most likely to hide a failure, in this order: no test
+reaches them, they changed since the base, they are flows rather than single
+screens. Record `priority: {mode, selected}`.
+
+## 2. Discovery
+
+`test-gen-analyst`, stage `discovery`, with the selected items. It returns test
+roots and runners, the tests that reach each item (joined by identifier), the
+handles already present, and the conventions for new tests. Save as
+`discovery.json` and gate it with `check --stage discovery`.
+
+## 3. Gaps
+
+With `coverage_gap` (toolkit >= v3.25.0): call it with the inventory and the
+test root from discovery; its `covered`, `uncovered` and `unmatched` buckets go
+into the analyst's prompt, and `coverage.method` is `coverage_gap`. Without it:
+`coverage.method` is `agent` and every match the analyst makes is
+`method: "inferred"`. Then `test-gen-analyst`, stage `gaps`. Save its answer as
+`gaps.json`. The scenario-gap-analysis skill (ai-common-toolkit) defines the
+buckets and the scenario fields when it is enabled.
+
+Before writing any test, show the gaps and the `dataNeeds[]`. Each need is
+named, never filled: the person supplies the value, or names the credential the
+test resolves by `secretRef`. Autopilot writes only the scenarios whose needs
+are already met and lists the rest as blocked by data.
+
+## 4. Handles
+
+Only gaps with `handlesNeeded`. Before dispatch:
+
+```bash
+node "$HOME/.claude/scripts/test-gen-gate.mjs" baseline --repo "$REPO" --out "$RUN/handles-baseline.json"
+```
+
+The subagent adds the missing handles and nothing else: iOS
+`accessibilityIdentifier`, Compose `testTag` (or `testTagsAsResourceId` in a
+semantics block), web `data-testid`. Never a list index, never a renamed or
+moved element, never a handle the accessibility tree cannot see. It writes
+`handles.md`, one line per request: `ADDED file:line`, `ALREADY file:line`,
+`COULD NOT ADD <reason>` (the handle-request skill, when enabled). When nothing
+was missing, the first line is `HANDLES: none-needed` followed by the
+`ALREADY file:line` lines that prove it. Then:
+
+```bash
+node "$HOME/.claude/scripts/test-gen-gate.mjs" handles --repo "$REPO" \
+  --baseline "$RUN/handles-baseline.json" --handles "$RUN/handles.md"
+node "$HOME/.claude/scripts/handle-diff-audit.mjs" --repo "$REPO" \
+  --baseline "$RUN/handles-baseline.json" > "$RUN/handle-audit.json"
+```
+
+`handles` passes when the stage added a handle line, or on
+`HANDLES: none-needed` with at least one cited line, inside the repository, that
+carries a handle. `handle-diff-audit.mjs` must exit 0: 2 means the stage changed
+more than handles (`violations[]` name the file and the lost, extra or
+reordered tokens), so revert those hunks of the stage's own change and rerun; 3
+means it could not read the diff, which is not a pass.
+
+## 5. Write
+
+Take a new snapshot first (`test-gen-gate.mjs baseline --out
+"$RUN/write-baseline.json"`), then one subagent per platform, with the gaps,
+discovery's conventions and the handles. With `ui_locator_codegen` (toolkit >= v3.25.0), locators come from the
+UI tree node: `{platform, node}` returns the XCUITest or Compose code, or
+`NO_HANDLE` with the handle to add, which goes back to stage 4. Without it,
+write locators from the handles by hand. A locator is a handle, never a list
+index or a visible string that is translated.
+
+Each test asserts the gap's `expectation`. Test data comes from `secretRef` or
+the fixtures the person supplied, never a literal secret. The subagent writes
+`written.json`: `{files: [...], command: "<runs exactly these tests>",
+tests: [{id, file}]}`. Then run the command once with a per-test report and:
+
+```bash
+node "$HOME/.claude/scripts/test-gen-gate.mjs" written --repo "$REPO" \
+  --baseline "$RUN/write-baseline.json" --written "$RUN/written.json" \
+  --discovery "$RUN/discovery.json" --junit "$RUN/run-1.junit.xml"
+```
+
+(`--xcresult-json` or `--log` where the runner writes no JUnit.) The gate fails
+when the stage changed anything but test files, when a listed file is outside
+the repository, missing, empty or not changed by the stage, when the stage's
+diff carries test gaming (an environment branch, a swallowed error, an inflated
+wait), when the command is none of the runners discovery found, or when the run
+executed zero tests.
+
+## 6. Run
+
+Run `written.json`'s command `testStability.repeatCount` times (default 3),
+each run's log and per-test report kept as `run-<k>.log` and
+`run-<k>.junit.xml` (or `.xcresult`). A UI suite goes through
+`run-ui-tests.sh run`, whose `UI_TEST_RESULT` names the report. Each log must
+clear `evidence-gate.mjs --claim test --status passed` before a pass is
+recorded; a failing run is evidence too and is kept.
+
+## 7. Classify
+
+`features/test-classification.md`: `test_results_classify` with run 1 as
+`head`, the others as `confirm`, and the project's flake ledger, then
+`test-classify.mjs merge`; without the tool, `test-classify.mjs fallback` over
+the runs. Then:
+
+```bash
+node "$HOME/.claude/scripts/test-gen-gate.mjs" results --classification "$RUN/test-classification.json" \
+  --written "$RUN/written.json" --out "$RUN/results.json"
+node "$HOME/.claude/scripts/test-gen-gate.mjs" check --stage results --file "$RUN/results.json"
+```
+
+`results.json` is three-state: `pass`, `fail` (a regression, an unstable test, a
+preexisting failure), `not_measured` (no base, a setup failure); every non-pass
+carries its reason. `preexisting` is a fail here although Gate 3 inherits it: the
+test is new, so red on the base too means the behaviour it asserts is missing,
+which is a finding for triage, not someone else's failure.
+
+## 8. Triage
+
+Each `fail` and `not_measured` gets one verdict (the test-result-triage skill,
+when enabled): `product-bug`, `stale-test`, `new-scenario`, `valid` or
+`ambiguous`, and `ambiguous` is the default. `stale-test` needs the `file:line`
+in the diff that changed the behaviour on purpose; a locator that found nothing
+goes back to stage 4 before it may be called stale. A green test whose subject
+changed behaviour is questioned too. Record each verdict, then:
+
+```bash
+node "$HOME/.claude/scripts/test-gen-state.mjs" revise --state "$TG_STATE" --run "$RUN_ID" \
+  --test "<id>" --verdict "<verdict>" ${AUTOPILOT:+--autopilot}
+```
+
+| Action | Then |
+|---|---|
+| `apply` | revise the test, rerun from stage 6 for that test |
+| `park` | autopilot stops, because the verdict is `product-bug` (never applied without a person) or the stale-test cap (`prefs.global.testGen.revisionCap`, per run and per day) is reached. Report and end |
+| `ask` | attended `product-bug`: show the evidence and ask whether to open it through `/multi-agent` |
+| `none` | keep the verdict in `results.json` |
+
+The count is read and written under a lock, so parallel triage cannot pass the
+cap. A state or prefs file that does not parse is never counted as empty:
+autopilot parks (exit 3), an attended run stops with exit 4 and records nothing.
+
+## End: the change base
+
+After the report, record where this run ended so the next one diffs from here:
+
+```bash
+node "$HOME/.claude/scripts/test-gen-state.mjs" base-record --state "$TG_STATE" --repo "$REPO" --run "$RUN_ID"
+```
+
+The reason names the run (`test-gen run <id> finished`). It records which code
+the tests were written against, not that the app is understood.

package/pipeline/preferences-template.json

@@ -106,6 +106,9 @@
     },
     "resume": {
       "autoFix": false
+    },
+    "testGen": {
+      "revisionCap": 20
     }
   },
   "projects": {},

package/pipeline/schemas/prefs.schema.json

@@ -1123,6 +1123,20 @@
             }
           }
         },
+        "testGen": {
+          "type": "object",
+          "additionalProperties": false,
+          "description": "/multi-agent:test-gen knobs. Read by test-gen-state.mjs revise.",
+          "properties": {
+            "revisionCap": {
+              "type": "integer",
+              "minimum": 0,
+              "maximum": 500,
+              "default": 20,
+              "description": "Stale-test revisions autopilot may apply, counted per run and per calendar day (UTC), whichever is reached first. At the cap the run parks instead of editing more tests. 0 parks on the first one."
+            }
+          }
+        },
         "testStability": {
           "type": "object",
           "additionalProperties": false,

package/pipeline/schemas/test-gen-discovery.schema.json

@@ -0,0 +1,73 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://github.com/mmerterden/multi-agent-pipeline/pipeline/schemas/test-gen-discovery.schema.json",
+  "title": "Multi-Agent Pipeline  -  test-gen discovery",
+  "description": "What the repository already has for testing the selected inventory items, $RUN/discovery.json of /multi-agent:test-gen (multi-agent-refs/test-gen/stages.md). Returned by the read-only test-gen-analyst persona and written by the orchestrator. testRoots[].command is what the write stage's run command is checked against (test-gen-gate.mjs written --discovery). Existing tests are joined to items by the identifier the test uses, never by a name. test-gen-gate.mjs check --stage discovery checks this shape without a dependency.",
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["stage", "testRoots", "existing", "handles"],
+  "$defs": {
+    "fileLine": { "type": "string", "pattern": "^[^\\s:]+:[1-9][0-9]*$" }
+  },
+  "properties": {
+    "stage": { "const": "discovery" },
+    "testRoots": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["path", "runner", "command"],
+        "properties": {
+          "path": { "type": "string", "minLength": 1 },
+          "runner": {
+            "enum": [
+              "xcuitest",
+              "xctest",
+              "swift-testing",
+              "espresso",
+              "compose",
+              "junit",
+              "playwright",
+              "cypress",
+              "jest",
+              "vitest",
+              "pytest",
+              "other"
+            ]
+          },
+          "command": { "type": "string", "minLength": 1 },
+          "report": { "enum": ["junit", "xcresult", "none"] }
+        }
+      }
+    },
+    "existing": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["inventoryId", "tests"],
+        "properties": {
+          "inventoryId": { "type": "string", "minLength": 1 },
+          "tests": { "type": "array", "items": { "$ref": "#/$defs/fileLine" } },
+          "joinedBy": { "type": "string", "minLength": 1 }
+        }
+      }
+    },
+    "handles": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["inventoryId", "file", "line", "handle"],
+        "properties": {
+          "inventoryId": { "type": "string", "minLength": 1 },
+          "file": { "type": "string", "minLength": 1 },
+          "line": { "type": "integer", "minimum": 1 },
+          "handle": { "type": "string", "minLength": 1 }
+        }
+      }
+    },
+    "conventions": { "type": "array", "items": { "type": "string", "minLength": 1 } },
+    "notes": { "type": "array", "items": { "type": "string" } }
+  }
+}

package/pipeline/schemas/test-gen-gaps.schema.json

@@ -0,0 +1,122 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://github.com/mmerterden/multi-agent-pipeline/pipeline/schemas/test-gen-gaps.schema.json",
+  "title": "Multi-Agent Pipeline  -  test-gen gaps",
+  "description": "The gap analysis of /multi-agent:test-gen, $RUN/gaps.json (multi-agent-refs/test-gen/stages.md). Written from the read-only analyst's answer: the selected inventory items set against the existing tests, each gap a scenario to write. `wrongly-covered` (a test exists but cannot see the failure) is kept apart from `missing`. dataNeeds[] says what test data a scenario needs and who provides it; it never holds a value, because values are entered by a person or resolved by secret reference at run time. unmeasured[] lists what the analysis could not measure, which is not the same as covered. test-gen-gate.mjs check --stage gaps checks this shape without a dependency.",
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["schemaVersion", "coverage", "gaps", "dataNeeds", "unmeasured"],
+  "$defs": {
+    "fileLine": { "type": "string", "pattern": "^[^\\s:]+:[1-9][0-9]*$" }
+  },
+  "properties": {
+    "schemaVersion": { "const": "1.0.0" },
+    "coverage": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["method", "measured"],
+      "properties": {
+        "method": {
+          "enum": ["coverage_gap", "agent"],
+          "description": "coverage_gap: the toolkit tool joined inventory and tests by identifier; agent: the analyst's own reading, on a toolkit without the tool."
+        },
+        "measured": { "type": "boolean" },
+        "reason": { "type": "string", "minLength": 1 },
+        "covered": { "type": "integer", "minimum": 0 },
+        "uncovered": { "type": "integer", "minimum": 0 },
+        "unmatched": { "type": "integer", "minimum": 0 }
+      }
+    },
+    "gaps": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": [
+          "id",
+          "inventoryId",
+          "bucket",
+          "title",
+          "priority",
+          "rationale",
+          "expectation",
+          "automation",
+          "method"
+        ],
+        "properties": {
+          "id": { "type": "string", "minLength": 1 },
+          "inventoryId": { "type": "string", "minLength": 1 },
+          "bucket": { "enum": ["missing", "wrongly-covered"] },
+          "title": { "type": "string", "minLength": 1 },
+          "priority": {
+            "enum": ["bug-catching", "edge", "basic"],
+            "description": "bug-catching: would catch a failure users hit; edge: a boundary or error path; basic: the happy path."
+          },
+          "rationale": {
+            "type": "string",
+            "minLength": 1,
+            "description": "Why the business cares."
+          },
+          "expectation": {
+            "type": "string",
+            "minLength": 1,
+            "description": "What the test asserts, measurable."
+          },
+          "automation": { "enum": ["yes", "no", "needs-code"] },
+          "method": {
+            "enum": ["exact", "inferred"],
+            "description": "exact: matched by identifier; inferred: the analyst's judgement, kept apart."
+          },
+          "evidence": { "type": "array", "items": { "$ref": "#/$defs/fileLine" } },
+          "handlesNeeded": {
+            "type": "array",
+            "items": {
+              "type": "object",
+              "additionalProperties": false,
+              "required": ["target", "suggestion"],
+              "properties": {
+                "target": { "type": "string", "minLength": 1 },
+                "suggestion": { "type": "string", "minLength": 1 }
+              }
+            }
+          }
+        }
+      }
+    },
+    "dataNeeds": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["id", "gapIds", "what", "kind"],
+        "properties": {
+          "id": { "type": "string", "minLength": 1 },
+          "gapIds": {
+            "type": "array",
+            "minItems": 1,
+            "items": { "type": "string", "minLength": 1 }
+          },
+          "what": { "type": "string", "minLength": 1 },
+          "kind": { "enum": ["account", "seed", "fixture", "secret", "other"] },
+          "secretRef": {
+            "type": "string",
+            "minLength": 1,
+            "description": "Credential name a test resolves at run time; never the value."
+          }
+        }
+      }
+    },
+    "unmeasured": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["reason"],
+        "properties": {
+          "inventoryId": { "type": "string", "minLength": 1 },
+          "reason": { "type": "string", "minLength": 1 }
+        }
+      }
+    }
+  }
+}

package/pipeline/schemas/test-gen-inventory.schema.json

@@ -0,0 +1,59 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://github.com/mmerterden/multi-agent-pipeline/pipeline/schemas/test-gen-inventory.schema.json",
+  "title": "Multi-Agent Pipeline  -  test-gen inventory",
+  "description": "What the app has that a test could cover, written by the inventory stage of /multi-agent:test-gen as $RUN/inventory.json (multi-agent-refs/test-gen/stages.md). Items are joined to tests by their stable id (route, screen id, Figma node id, endpoint path), never by title. `priority` is added after the inventory, from the user's pick or, in autopilot, the top three. test-gen-gate.mjs check --stage inventory checks this shape without a dependency.",
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["schemaVersion", "platform", "source", "items"],
+  "properties": {
+    "schemaVersion": { "const": "1.0.0" },
+    "platform": { "enum": ["ios", "android", "web", "api"] },
+    "repo": { "type": "string", "minLength": 1 },
+    "source": {
+      "enum": ["design_scenario_inventory", "web_map", "ui_tree", "openapi", "agent"],
+      "description": "Where the items came from. `agent` is the stage's own reading of the code, used when no tool answered."
+    },
+    "generatedAt": { "type": "string", "format": "date-time" },
+    "items": {
+      "type": "array",
+      "minItems": 1,
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["id", "kind", "title"],
+        "properties": {
+          "id": { "type": "string", "minLength": 1, "description": "Stable identifier." },
+          "kind": { "enum": ["screen", "flow", "endpoint", "component"] },
+          "title": { "type": "string", "minLength": 1 },
+          "path": {
+            "type": "string",
+            "minLength": 1,
+            "description": "Code path that implements it, file or directory."
+          },
+          "handles": {
+            "type": "array",
+            "items": { "type": "string", "minLength": 1 },
+            "description": "Automation handles already on it (accessibilityIdentifier, testTag, data-testid)."
+          }
+        }
+      }
+    },
+    "priority": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["mode", "selected"],
+      "properties": {
+        "mode": {
+          "enum": ["user", "autopilot-top3"],
+          "description": "user: picked by the person; autopilot-top3: the three the run proceeded with unattended."
+        },
+        "selected": {
+          "type": "array",
+          "minItems": 1,
+          "items": { "type": "string", "minLength": 1 }
+        }
+      }
+    }
+  }
+}

package/pipeline/schemas/test-gen-results.schema.json

@@ -0,0 +1,44 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://github.com/mmerterden/multi-agent-pipeline/pipeline/schemas/test-gen-results.schema.json",
+  "title": "Multi-Agent Pipeline  -  test-gen results",
+  "description": "The outcome of the tests /multi-agent:test-gen wrote, $RUN/results.json, built by test-gen-gate.mjs results from the classify stage (multi-agent-refs/features/test-classification.md). Three states: pass, fail, not_measured; anything but pass carries its reason. `classification` keeps the measured class behind the state (an unstable test is a fail, a test with no base is not_measured), and `triage` the result-triage verdict once one is made. test-gen-gate.mjs check --stage results checks this shape without a dependency.",
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["schemaVersion", "source", "green", "tests", "summary"],
+  "properties": {
+    "schemaVersion": { "const": "1.0.0" },
+    "source": { "enum": ["toolkit", "fallback"] },
+    "green": { "type": "boolean" },
+    "tests": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["id", "status"],
+        "properties": {
+          "id": { "type": "string", "minLength": 1 },
+          "file": { "type": "string", "minLength": 1 },
+          "status": { "enum": ["pass", "fail", "not_measured"] },
+          "classification": {
+            "enum": ["passed", "regression", "preexisting", "unstable", "not_measured"]
+          },
+          "reason": { "type": "string", "minLength": 1 },
+          "triage": { "enum": ["product-bug", "stale-test", "new-scenario", "valid", "ambiguous"] }
+        },
+        "if": { "properties": { "status": { "enum": ["fail", "not_measured"] } } },
+        "then": { "required": ["reason"] }
+      }
+    },
+    "summary": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["pass", "fail", "not_measured"],
+      "properties": {
+        "pass": { "type": "integer", "minimum": 0 },
+        "fail": { "type": "integer", "minimum": 0 },
+        "not_measured": { "type": "integer", "minimum": 0 }
+      }
+    }
+  }
+}