@harness-engineering/cli 2.8.0 → 3.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (120) hide show
  1. package/dist/agents/skills/claude-code/harness-audit-harness-strength/SKILL.md +188 -0
  2. package/dist/agents/skills/claude-code/harness-audit-harness-strength/skill.yaml +54 -0
  3. package/dist/agents/skills/claude-code/harness-autopilot/SKILL.md +2 -0
  4. package/dist/agents/skills/claude-code/harness-brainstorming/SKILL.md +28 -14
  5. package/dist/agents/skills/claude-code/harness-execution/SKILL.md +1 -1
  6. package/dist/agents/skills/claude-code/harness-git-workflow/SKILL.md +2 -0
  7. package/dist/agents/skills/claude-code/harness-ideate/SKILL.md +2 -0
  8. package/dist/agents/skills/claude-code/harness-knowledge-pipeline/SKILL.md +2 -0
  9. package/dist/agents/skills/claude-code/harness-roadmap/SKILL.md +62 -4
  10. package/dist/agents/skills/claude-code/harness-roadmap-pilot/SKILL.md +2 -0
  11. package/dist/agents/skills/claude-code/harness-test-advisor/SKILL.md +80 -2
  12. package/dist/agents/skills/claude-code/harness-test-advisor/skill.yaml +4 -1
  13. package/dist/agents/skills/claude-code/initialize-harness-project/SKILL.md +27 -25
  14. package/dist/agents/skills/claude-code/initialize-harness-project/skill.yaml +1 -0
  15. package/dist/agents/skills/claude-code/outcome-eval/SKILL.md +97 -0
  16. package/dist/agents/skills/claude-code/outcome-eval/skill.yaml +51 -0
  17. package/dist/agents/skills/codex/harness-audit-harness-strength/SKILL.md +188 -0
  18. package/dist/agents/skills/codex/harness-audit-harness-strength/skill.yaml +54 -0
  19. package/dist/agents/skills/codex/harness-autopilot/SKILL.md +2 -0
  20. package/dist/agents/skills/codex/harness-brainstorming/SKILL.md +28 -14
  21. package/dist/agents/skills/codex/harness-execution/SKILL.md +1 -1
  22. package/dist/agents/skills/codex/harness-git-workflow/SKILL.md +2 -0
  23. package/dist/agents/skills/codex/harness-ideate/SKILL.md +2 -0
  24. package/dist/agents/skills/codex/harness-knowledge-pipeline/SKILL.md +2 -0
  25. package/dist/agents/skills/codex/harness-roadmap/SKILL.md +62 -4
  26. package/dist/agents/skills/codex/harness-roadmap-pilot/SKILL.md +2 -0
  27. package/dist/agents/skills/codex/harness-test-advisor/SKILL.md +80 -2
  28. package/dist/agents/skills/codex/harness-test-advisor/skill.yaml +4 -1
  29. package/dist/agents/skills/codex/initialize-harness-project/SKILL.md +27 -25
  30. package/dist/agents/skills/codex/initialize-harness-project/skill.yaml +1 -0
  31. package/dist/agents/skills/codex/outcome-eval/SKILL.md +97 -0
  32. package/dist/agents/skills/codex/outcome-eval/skill.yaml +51 -0
  33. package/dist/agents/skills/cursor/harness-audit-harness-strength/SKILL.md +188 -0
  34. package/dist/agents/skills/cursor/harness-audit-harness-strength/skill.yaml +54 -0
  35. package/dist/agents/skills/cursor/harness-autopilot/SKILL.md +2 -0
  36. package/dist/agents/skills/cursor/harness-brainstorming/SKILL.md +28 -14
  37. package/dist/agents/skills/cursor/harness-execution/SKILL.md +1 -1
  38. package/dist/agents/skills/cursor/harness-git-workflow/SKILL.md +2 -0
  39. package/dist/agents/skills/cursor/harness-ideate/SKILL.md +2 -0
  40. package/dist/agents/skills/cursor/harness-knowledge-pipeline/SKILL.md +2 -0
  41. package/dist/agents/skills/cursor/harness-roadmap/SKILL.md +62 -4
  42. package/dist/agents/skills/cursor/harness-roadmap-pilot/SKILL.md +2 -0
  43. package/dist/agents/skills/cursor/harness-test-advisor/SKILL.md +80 -2
  44. package/dist/agents/skills/cursor/harness-test-advisor/skill.yaml +4 -1
  45. package/dist/agents/skills/cursor/initialize-harness-project/SKILL.md +27 -25
  46. package/dist/agents/skills/cursor/initialize-harness-project/skill.yaml +1 -0
  47. package/dist/agents/skills/cursor/outcome-eval/SKILL.md +97 -0
  48. package/dist/agents/skills/cursor/outcome-eval/skill.yaml +51 -0
  49. package/dist/agents/skills/gemini-cli/harness-audit-harness-strength/SKILL.md +188 -0
  50. package/dist/agents/skills/gemini-cli/harness-audit-harness-strength/skill.yaml +54 -0
  51. package/dist/agents/skills/gemini-cli/harness-autopilot/SKILL.md +2 -0
  52. package/dist/agents/skills/gemini-cli/harness-brainstorming/SKILL.md +28 -14
  53. package/dist/agents/skills/gemini-cli/harness-execution/SKILL.md +1 -1
  54. package/dist/agents/skills/gemini-cli/harness-git-workflow/SKILL.md +2 -0
  55. package/dist/agents/skills/gemini-cli/harness-ideate/SKILL.md +2 -0
  56. package/dist/agents/skills/gemini-cli/harness-knowledge-pipeline/SKILL.md +2 -0
  57. package/dist/agents/skills/gemini-cli/harness-roadmap/SKILL.md +62 -4
  58. package/dist/agents/skills/gemini-cli/harness-roadmap-pilot/SKILL.md +2 -0
  59. package/dist/agents/skills/gemini-cli/harness-test-advisor/SKILL.md +80 -2
  60. package/dist/agents/skills/gemini-cli/harness-test-advisor/skill.yaml +4 -1
  61. package/dist/agents/skills/gemini-cli/initialize-harness-project/SKILL.md +27 -25
  62. package/dist/agents/skills/gemini-cli/initialize-harness-project/skill.yaml +1 -0
  63. package/dist/agents/skills/gemini-cli/outcome-eval/SKILL.md +97 -0
  64. package/dist/agents/skills/gemini-cli/outcome-eval/skill.yaml +51 -0
  65. package/dist/agents/skills/tests/harness-test-advisor.test.ts +94 -0
  66. package/dist/{agents-md-WMJO5HSM.js → agents-md-XOJE5ETA.js} +2 -2
  67. package/dist/{architecture-EVYVGPRZ.js → architecture-JEZPJ725.js} +3 -3
  68. package/dist/{assess-project-KZG563SO.js → assess-project-RHXOQNMM.js} +1 -1
  69. package/dist/bin/harness-mcp.js +13 -13
  70. package/dist/bin/harness.js +22 -22
  71. package/dist/{check-phase-gate-MK364VXS.js → check-phase-gate-GNDOMS35.js} +3 -3
  72. package/dist/{chunk-VBNJAKUH.js → chunk-3ISHDWO7.js} +1 -1
  73. package/dist/{chunk-UJFNIWC7.js → chunk-527G27VI.js} +2 -2
  74. package/dist/{chunk-5XL5QJ5B.js → chunk-6WQCLCBO.js} +1 -1
  75. package/dist/{chunk-UXPWSV3B.js → chunk-BYVT5LVO.js} +3 -3
  76. package/dist/{chunk-HRSE6IQD.js → chunk-DE5U6KOL.js} +1 -1
  77. package/dist/{chunk-B7KMTHN3.js → chunk-GGKRA7A7.js} +1 -1
  78. package/dist/{chunk-STESM73J.js → chunk-GSP2XIVS.js} +1 -1
  79. package/dist/{chunk-KBCPELBC.js → chunk-HEIMJJI4.js} +2 -2
  80. package/dist/{chunk-KI6PHH6Q.js → chunk-HWTUUQOW.js} +3 -3
  81. package/dist/{chunk-2SNRCAC3.js → chunk-KAJOS3BA.js} +2 -2
  82. package/dist/{chunk-C7J55C6E.js → chunk-MGUPSE6D.js} +5 -5
  83. package/dist/{chunk-GOHWCHCS.js → chunk-N24HCQA4.js} +2243 -776
  84. package/dist/{chunk-EEFRPEWT.js → chunk-NWLGL3IR.js} +1 -1
  85. package/dist/{chunk-UUO4K7NX.js → chunk-ONCPJGHY.js} +6 -6
  86. package/dist/{chunk-X4EFYVCZ.js → chunk-PC2R5RUJ.js} +1 -1
  87. package/dist/{chunk-7EQAZKJ5.js → chunk-QBC7DI4Q.js} +1 -1
  88. package/dist/{chunk-UBZQE3JN.js → chunk-UX3PQKVZ.js} +7 -2
  89. package/dist/{chunk-CAQPSFPT.js → chunk-WWEVS7RO.js} +453 -91
  90. package/dist/{chunk-4BDOSCNY.js → chunk-XFU6SB2Q.js} +1015 -648
  91. package/dist/{chunk-LADPYELQ.js → chunk-XX4OLNWQ.js} +2 -2
  92. package/dist/{chunk-FZ5MBXEG.js → chunk-YQSYCBAH.js} +6 -6
  93. package/dist/{chunk-ACOOG3UW.js → chunk-Z4AWEIBY.js} +3 -3
  94. package/dist/{ci-workflow-IW2H5DC4.js → ci-workflow-ORZ4F45F.js} +2 -2
  95. package/dist/{dist-JCFK263L.js → dist-LHINSVK4.js} +85 -1
  96. package/dist/{docs-7CP6VV42.js → docs-AU3DXFFO.js} +3 -3
  97. package/dist/{engine-MKJEBWU7.js → engine-6GPWOYQH.js} +2 -2
  98. package/dist/{entropy-6JOZJYG7.js → entropy-VLKDRMRT.js} +2 -2
  99. package/dist/{feedback-T65AEJEG.js → feedback-YXYY44ZC.js} +1 -1
  100. package/dist/{generate-agent-definitions-4VI4YTO2.js → generate-agent-definitions-XIHQPKG3.js} +3 -3
  101. package/dist/hooks/cost-tracker.js +29 -6
  102. package/dist/index.d.ts +8 -8
  103. package/dist/index.js +22 -22
  104. package/dist/{loader-AWTHHE3A.js → loader-YAN56MT3.js} +2 -2
  105. package/dist/{mcp-JBIBINO2.js → mcp-RK3SFVYB.js} +13 -13
  106. package/dist/{performance-NCOUDOMW.js → performance-64DPTLLQ.js} +3 -3
  107. package/dist/{review-pipeline-YIC6JPMY.js → review-pipeline-NENG6LW7.js} +2 -2
  108. package/dist/{runtime-5MHV4UEE.js → runtime-FFIIRT2T.js} +2 -2
  109. package/dist/{security-VZPJNIDP.js → security-3PIBN35B.js} +1 -1
  110. package/dist/templates/basic/harness.config.json.hbs +34 -0
  111. package/dist/templates/ci/README.md +86 -0
  112. package/dist/templates/ci/required-review.ruleset.json +20 -0
  113. package/dist/templates/ci/required-review.yml.hbs +57 -0
  114. package/dist/templates/ci/template.json +5 -0
  115. package/dist/templates/intermediate/harness.config.json.hbs +36 -0
  116. package/dist/templates/orchestrator/harness.orchestrator.md +9 -0
  117. package/dist/{tool-tiers-B7JC2XC3.js → tool-tiers-OKK5FE57.js} +2 -0
  118. package/dist/{validate-7BE4VTFI.js → validate-7IAF2ES3.js} +3 -3
  119. package/dist/{validate-cross-check-F7GEBRDH.js → validate-cross-check-QPSAOZKD.js} +2 -2
  120. package/package.json +7 -7
@@ -136,7 +136,7 @@ These flow into `handoff.json` `contextKeywords` field. Select keywords that hel
136
136
  - **Entry Points** -- Which system entry points does this feature touch or create? (e.g., new CLI command, new MCP tool, new skill, new API route, new barrel export)
137
137
  - **Registrations Required** -- What registrations are needed for the feature to be discoverable? (e.g., barrel export regeneration, skill tier assignment, route registration)
138
138
  - **Documentation Updates** -- What docs need updating to reflect the new capability? (e.g., AGENTS.md section, API docs, README, guides)
139
- - **Architectural Decisions** -- What decisions warrant ADRs? List the decision and a one-line rationale. Only for medium/large tier changes -- omit for small changes.
139
+ - **Architectural Decisions** -- Which decisions from the **Decisions made** section rise to a standalone ADR? Reference each by name with a one-line note on _why_ it warrants an ADR. Do **not** restate the decisions here -- this subsection points back to the canonical **Decisions made** section; duplicating them creates two sources that drift (spec-craft flags this as SPEC-R004). Only for medium/large tier changes -- "None" for small changes.
140
140
  - **Knowledge Impact** -- What domain concepts, patterns, or relationships should enter the knowledge graph?
141
141
 
142
142
  If the feature is a small change (bug fix, config tweak, < 3 files), the section may contain only Entry Points and Registrations Required with "None" for the others. The section must still be present.
@@ -186,21 +186,35 @@ These flow into `handoff.json` `contextKeywords` field. Select keywords that hel
186
186
 
187
187
  The human must explicitly approve before this skill is complete.
188
188
 
189
- 7. **Commit spec artifacts.** After approval, commit the spec and skill recommendations so the paper trail enters git history at sign-off rather than retroactively:
189
+ 7. **Promote the roadmap row.** If `docs/roadmap.md` exists, transition the named row to `planned` and link the spec in a single structured call (steps 7 and 8 are intentionally ordered so the commit captures the roadmap mutation). Skip silently only when no roadmap exists.
190
+ - Derive the lookup key from the slash-command `ARGUMENTS` string (D1). Derive the summary from the spec title (the H1 heading).
191
+ - Call `manage_roadmap` with action `promote`, `feature: "<ARGUMENTS>"`, `spec: "docs/changes/<feature>/proposal.md"`, and `summary: "<H1>"`.
192
+ - Branch on the returned `PromoteResult` envelope:
193
+
194
+ | Envelope | Skill behavior |
195
+ | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
196
+ | `ok: true, transitioned: 'backlog→planned'` | Log "Promoted `<feature>`: backlog → planned". Continue to step 8. |
197
+ | `ok: true, transitioned: 'spec-updated'` | Log "Updated spec link for `<feature>` (status preserved)". Continue to step 8. |
198
+ | `ok: true, transitioned: 'noop'` | Log "No change — `<feature>` already promoted with this spec". Continue to step 8. |
199
+ | `ok: false, reason: 'in-progress'` | **STOP.** Surface: "Refused to promote `<feature>`: an agent is currently dispatched against this row. Stop the agent or use a different feature name." |
200
+ | `ok: false, reason: 'done'` | **STOP.** Surface: "Refused to promote `<feature>`: row is already 'done'. To revise a shipped feature, use a new name." |
201
+ | `ok: false, reason: 'not-found'` | If no row was expected to exist, fall through to a `created` outcome. Otherwise surface `closestMatches` as a typo hint and **STOP**. |
202
+ | `ok: false, reason: 'ambiguous'` | **STOP.** Surface: "Refused to promote `<feature>`: matches multiple rows across milestones. Re-invoke with one of: `<matches>`." Matches are milestone-qualified. |
203
+ | `ok: false, reason: 'write-failed'` | **STOP.** Surface `detail` verbatim. The brainstorm is not considered complete. |
204
+
205
+ - The `not-found` create path: when the row genuinely does not exist yet, call `manage_roadmap` action `add` with `status: "planned"`, `milestone: "Current Work"`, the spec path, and the H1 summary (unchanged from the legacy behavior), then continue to step 8.
206
+ - "STOP" cases skip step 8 (no commit) and step 9 (no handoff/transition). The spec file written in step 3 stays on disk; the user re-runs the skill after resolving the conflict.
207
+ - If `manage_roadmap` is unavailable, fall back to `parseRoadmap`/`promoteFeature`/`serializeRoadmap` from core. Warn: "External sync skipped (MCP unavailable). Run `manage_roadmap sync` when MCP is restored."
208
+ - If no roadmap exists, skip silently (no envelope) and continue to step 8; the commit then contains only the spec files.
209
+
210
+ 8. **Commit spec artifacts and roadmap promotion.** After a successful promote (or a silent skip when no roadmap exists), commit the spec, skill recommendations, and the roadmap mutation together so the promotion ships atomically with the spec:
190
211
 
191
212
  ```bash
192
- git add docs/changes/<feature>/proposal.md docs/changes/<feature>/SKILLS.md
193
- git commit -m "docs(<feature>): add spec"
213
+ git add docs/changes/<feature>/proposal.md docs/changes/<feature>/SKILLS.md docs/roadmap.md
214
+ git commit -m "docs(<feature>): add spec and promote to planned"
194
215
  ```
195
216
 
196
- Do not skip this step. `harness-execution` commits only implementation files, so a spec uncommitted here stays untracked until someone notices. If a pre-commit hook reformats either file, re-add and re-commit.
197
-
198
- 8. **Add feature to roadmap.** If `docs/roadmap.md` exists:
199
- - Derive feature name from the spec title (the H1 heading).
200
- - Call `manage_roadmap` with action `add`, `status: "planned"`, `milestone: "Current Work"`, and the spec path.
201
- - If the feature already exists, skip silently.
202
- - If `manage_roadmap` is unavailable, fall back to `parseRoadmap`/`serializeRoadmap` from core. Warn: "External sync skipped (MCP unavailable). Run `manage_roadmap sync` when MCP is restored."
203
- - If no roadmap exists, skip silently.
217
+ Include `docs/roadmap.md` in `git add` only when it exists; omit it when the project has no roadmap. Do not skip this step `harness-execution` commits only implementation files, so a spec uncommitted here stays untracked until someone notices. If a pre-commit hook reformats a file, re-add and re-commit.
204
218
 
205
219
  9. **Write handoff and suggest transition.** After approval:
206
220
 
@@ -336,8 +350,8 @@ Technical claims about existing code, architecture, or tradeoffs MUST cite evide
336
350
  - **Spec location** -- `docs/changes/<feature>/proposal.md`.
337
351
  - **Handoff** -- Once approved, invoke harness-planning to create the implementation plan.
338
352
  - **Session directory** — When session slug is known, handoff goes to `.harness/sessions/<slug>/handoff.json`. The session directory structure is: `handoff.json`, `state.json`, `artifacts.json` (registry of spec/plan paths and file lists). Do not write to `.harness/handoff.json` in session context.
339
- - **Spec commit** -- After sign-off (Phase 4 Step 7), commit `docs/changes/<feature>/proposal.md` and `docs/changes/<feature>/SKILLS.md` so the spec enters git history at approval, not retroactively. `harness-execution` does not backfill these — see issue #487.
340
- - **Roadmap sync** -- After approval, call `manage_roadmap` action `add` to register as `planned`. Skip silently if no roadmap. Duplicates ignored.
353
+ - **Spec commit** -- After sign-off, the Phase 4 step 8 commit captures `docs/changes/<feature>/proposal.md`, `docs/changes/<feature>/SKILLS.md`, and the promoted `docs/roadmap.md` together so the spec enters git history at approval, not retroactively. `harness-execution` does not backfill these — see issue #487.
354
+ - **Roadmap promotion** -- After approval (Phase 4 step 7), call `manage_roadmap` action `promote` to transition the named row to `planned` and link the spec atomically with the spec commit. Falls back to `add` (create new row) when the row does not exist; refuses on `in-progress`/`done`/`ambiguous`. Skip silently if no roadmap.
341
355
  - **`emit_interaction`** -- End of Phase 4 to suggest transition to harness-planning (confirmed transition).
342
356
 
343
357
  #### Requirement Phrasing
@@ -56,7 +56,7 @@ When no arguments are provided (standalone invocation), discover plan from `docs
56
56
  4. **Load session summary for cold start.** If resuming (session slug known):
57
57
  - Call `listActiveSessions()` to read the session index.
58
58
  - Call `loadSessionSummary()` for the target session.
59
- - If ambiguous, present the index and ask which session to resume.
59
+ - If ambiguous, present the index and ask which session to resume. Ask in plain text — do not elevate this to `AskUserQuestion` (the session index can exceed its 4-option cap and a natural header like "Pick session" can exceed its 12-char cap, rendering the call as ERR).
60
60
 
61
61
  5. **Check for known dead ends.** Review `learnings` tagged `[outcome:failure]`. Warn if any match current plan approaches.
62
62
 
@@ -13,6 +13,8 @@
13
13
 
14
14
  ## Process
15
15
 
16
+ **Prompt the human in plain text** — every choice and destructive confirmation in this skill (worktree location, discard-experiment, etc.) is plain text only. Do not elevate to `AskUserQuestion`: option labels like the A/B/C worktree-location choices and natural headers like "Discard commits" routinely exceed its 4-option / 12-char caps and render as ERR.
17
+
16
18
  ### Part A: Worktree Creation
17
19
 
18
20
  #### Step 1: Choose Worktree Location
@@ -15,6 +15,8 @@
15
15
 
16
16
  ## Process
17
17
 
18
+ **Prompt the human in plain text** — every confirmation and selection in this skill (`Proceed? (y/n)`, post-critique selection, etc.) is plain text only. Do not elevate to `AskUserQuestion`: natural headers like "Confirm inputs" exceed its 12-char cap and option lists routinely exceed its 4-option cap, rendering the call as ERR.
19
+
18
20
  ### Iron Law
19
21
 
20
22
  **The skill produces exactly ONE ranked artifact per run, at `docs/ideation/<slug>-YYYY-MM-DD.md`, and NEVER produces a spec, plan, ADR, or code.** If the user wants a spec from a ranked idea, they invoke `harness-brainstorming` next — that is the contract.
@@ -14,6 +14,8 @@
14
14
 
15
15
  ## Process
16
16
 
17
+ **Prompt the human in plain text** — every confirmation, drift acknowledgement, and contradiction resolution in this skill is plain text only. Do not elevate to `AskUserQuestion`: source labels and finding descriptions routinely exceed its 4-option / 12-char caps, rendering the call as ERR.
18
+
17
19
  1. **EXTRACT:** Run code signal extractors, diagram parsers, BusinessKnowledgeIngestor, and KnowledgeLinker against the project
18
20
  2. **RECONCILE:** Build pre-extraction snapshot (current graph state) and post-extraction snapshot, then run StructuralDriftDetector
19
21
  3. **DETECT:** Partition findings by classification (new/stale/drifted/contradicting), generate gap report
@@ -9,6 +9,7 @@
9
9
  - When adding a new feature to an existing roadmap (`--add <feature-name>`)
10
10
  - When roadmap statuses may be stale and need updating from plan execution state (`--sync`)
11
11
  - When features need reordering, moving between milestones, or blocker updates (`--edit`)
12
+ - When the roadmap needs tidying -- completed work archived, dead `planned` rows demoted (`--groom`)
12
13
  - When user asks about project status and no roadmap exists -- suggest `--create`
13
14
  - NOT for programmatic CRUD (use `manage_roadmap` MCP tool directly)
14
15
 
@@ -20,6 +21,8 @@
20
21
 
21
22
  If the human has not seen and approved the milestone groupings and feature list, do not write the file. Present. Wait. Confirm. Then write.
22
23
 
24
+ **Prompt the human in plain text — every `(y/n)`, "which feature?", and similar prompt in this skill is plain text only.** Do not elevate them to `AskUserQuestion`: feature lists routinely exceed its 4-option cap, and natural header choices ("Pick feature", "Remove feature") exceed its 12-char cap, causing the call to render as ERR.
25
+
23
26
  ---
24
27
 
25
28
  ### Command: `--create` -- Bootstrap Roadmap
@@ -428,12 +431,67 @@ Choice?
428
431
 
429
432
  ---
430
433
 
434
+ ### Command: `--groom` -- Tidy the Roadmap
435
+
436
+ Keeps the roadmap manageable over time. **Milestones are themes; statuses are lifecycle stages** -- grooming enforces that separation so the backlog never decays back into an undifferentiated dump.
437
+
438
+ #### Phase 1: SCAN -- Detect Untidiness
439
+
440
+ 1. Check if `docs/roadmap.md` exists. If missing: error and direct the user to `--create`.
441
+ 2. Run `manage_roadmap` (`action: "groom"`) in a dry-run frame, or call `checkRoadmapHealth` from `@harness-engineering/core`, to surface the four health signals:
442
+ - **RMH001** -- completed (`done`) features still sitting in an active milestone.
443
+ - **RMH002** -- `planned` rows with neither a spec nor a plan (the orchestrator cannot auto-execute these; it escalates them to a human).
444
+ - **RMH003** -- lifecycle catch-all milestones (`Backlog`, `Current Work`) that should not exist.
445
+ - **RMH004** -- active milestones that have grown past the size cap (a mini-dump).
446
+
447
+ #### Phase 2: PROPOSE -- Present the Plan
448
+
449
+ Show the human exactly what grooming will do, in plain text:
450
+
451
+ ```
452
+ GROOM PLAN
453
+
454
+ Demote to backlog (planned with no spec/plan):
455
+ - Feature A (Theme X)
456
+ - Feature B (Theme Y)
457
+
458
+ Archive to docs/roadmap-archive.md (completed):
459
+ - Feature C (Theme X)
460
+
461
+ Flagged for manual routing (not auto-changed):
462
+ - Intake lane has 3 items awaiting a theme
463
+ - "Theme Z" has 28 features (cap 25) -- consider splitting
464
+
465
+ Apply? (y/n)
466
+ ```
467
+
468
+ Wait for confirmation. The mechanical changes (demote, archive) are safe and automated; **draining the Intake lane into themed milestones and splitting oversized milestones are human decisions** -- propose, do not auto-apply.
469
+
470
+ #### Phase 3: WRITE -- Apply
471
+
472
+ 1. Run `manage_roadmap` (`action: "groom"`). It demotes unactionable `planned` rows to `backlog` and moves `done` features into `docs/roadmap-archive.md` under a `Shipped` milestone, returning the list of changes.
473
+ 2. For Intake-draining or milestone-splitting the human approved, follow up with `--edit` (move features between milestones).
474
+
475
+ #### Phase 4: VALIDATE -- Verify
476
+
477
+ 1. Run `harness validate` and confirm the `roadmapHealth` check passes (no RMH003 errors; RMH001/002/004 warnings cleared or acknowledged).
478
+ 2. Summarize:
479
+
480
+ ```
481
+ Groom complete.
482
+ Demoted: N | Archived: N -> docs/roadmap-archive.md | Flagged for manual routing: N
483
+ harness validate (roadmapHealth): passed
484
+ ```
485
+
486
+ ---
487
+
431
488
  ## Harness Integration
432
489
 
433
- - **`manage_roadmap` MCP tool** -- Primary read/write interface for roadmap operations. Supports `show`, `add`, `update`, `remove`, and `query` actions. Use this when MCP is available for structured CRUD.
434
- - **`harness validate`** -- Run after any roadmap modification to verify project health. Mandatory in the VALIDATE phase of both `--create` and `--add`.
435
- - **Core `parseRoadmap`/`serializeRoadmap`** -- Fallback when MCP is unavailable. These functions in `packages/core/src/roadmap/` handle parsing and serializing the roadmap markdown format directly.
436
- - **Roadmap file** -- Always at `docs/roadmap.md`. This is the single source of truth for the project roadmap.
490
+ - **`manage_roadmap` MCP tool** -- Primary read/write interface for roadmap operations. Supports `show`, `add`, `update`, `remove`, `query`, `sync`, `promote`, and `groom` actions. Use this when MCP is available for structured CRUD.
491
+ - **`harness validate`** -- Run after any roadmap modification to verify project health. Mandatory in the VALIDATE phase of `--create`, `--add`, and `--groom`. The `roadmapHealth` check enforces the maintenance rules (RMH001-RMH004) as a regression guard.
492
+ - **Core `checkRoadmapHealth`/`groomRoadmap`** -- Maintenance engine in `packages/core/src/roadmap/health.ts`. `checkRoadmapHealth` is read-only diagnostics; `groomRoadmap` is the pure transform (demote unactionable planned, archive done). Both are surfaced via `manage_roadmap` and `harness validate`.
493
+ - **Core `parseRoadmap`/`serializeRoadmap`** -- Fallback when MCP is unavailable. These functions in `packages/core/src/roadmap/` handle parsing and serializing the roadmap markdown format directly. Note: the serializer only preserves frontmatter, milestones, features, and the Assignment History table -- never add convention prose or comments to `docs/roadmap.md`, they are dropped on the next write.
494
+ - **Roadmap files** -- Live work in `docs/roadmap.md` (the orchestrator's source of truth); completed work archived to `docs/roadmap-archive.md` by `--groom`. Milestones are themes, not lifecycle stages -- promoted items land in the `Intake` lane and are groomed into themes.
437
495
 
438
496
  ## Success Criteria
439
497
 
@@ -126,6 +126,8 @@ Proceed with Feature A? (y/n/pick another)
126
126
 
127
127
  ### Phase 3: CONFIRM -- Human Decision
128
128
 
129
+ Ask the human in plain text (matching the `y/n/pick another` example above). Do not elevate this confirmation to an `AskUserQuestion` tool call — candidate labels and natural header choices ("Pick candidate", etc.) exceed its 12-char `header` cap and the prompt is rejected as ERR.
130
+
129
131
  1. Wait for human confirmation.
130
132
  - If **yes**: proceed to Phase 4.
131
133
  - If **pick another**: ask which candidate number, then proceed with that pick.
@@ -8,8 +8,9 @@
8
8
  - In CI — optimize test suite execution order
9
9
  - When a test fails — understand which changes could have caused it
10
10
  - When `on_pr` triggers fire
11
- - NOT for writing tests (use harness-tdd)
12
- - NOT for test quality analysis (out of scope)
11
+ - **Coverage Audit mode** — when no diff is available and the user asks for "coverage gaps", "deep dive", "coverage plan", or "what's untested", run project-wide gap analysis instead of test selection
12
+ - NOT for writing tests (use harness-tdd, or `canary:canary-write-test` for uncovered files surfaced by Coverage Audit)
13
+ - NOT for test quality analysis at the test-selection level (Coverage Audit mode does include a capped quality review via `canary:canary-review-test`)
13
14
 
14
15
  ## Prerequisites
15
16
 
@@ -112,6 +113,83 @@ npx vitest run tests/services/auth.test.ts tests/types/user.test.ts tests/routes
112
113
  npx vitest run tests/services/auth.test.ts tests/types/user.test.ts tests/routes/login.test.ts tests/middleware/verify.test.ts tests/integration/auth-flow.test.ts
113
114
  ```
114
115
 
116
+ ## Coverage Audit Mode
117
+
118
+ Activates when `--audit` is passed, OR when no diff is available AND the user's
119
+ language matches audit intent ("coverage gaps", "deep dive", "coverage plan",
120
+ "what's untested"). When this mode is selected, skip Phases 1–3 above and run
121
+ the audit phases below instead.
122
+
123
+ ### Audit Phase 0: PROBE — Detect canary CLI availability
124
+
125
+ Call the `canary_probe` MCP tool once at the start of the audit. It returns
126
+ `{ status: "available" | "degraded", version?, reason? }` and never errors.
127
+
128
+ - **`available`** — the deterministic canary CLI is usable; use `canary_recommend_framework`
129
+ for framework selection in Phase 3 (GAP REPORT).
130
+ - **`degraded`** — the CLI is not usable (reason: `not-installed`, `binary-missing`,
131
+ `exec-failed`, or `bad-output`). Print one line:
132
+
133
+ > canary CLI unavailable (`<reason>`) — install `canary-test-cli` for deterministic
134
+ > framework recommendations. Proceeding; framework picks fall back to the plugin.
135
+
136
+ Then skip `canary_recommend_framework` calls for the rest of the run. This does **not**
137
+ affect the plugin-based Quality Review in Phase 2 (`canary:canary-review-test` is a
138
+ separate Claude Code plugin, installed independently of the CLI).
139
+
140
+ ### Audit Phase 1: INVENTORY — Build the Source-to-Test Map
141
+
142
+ 1. **Enumerate source files**: glob for `.ts`, `.tsx`, `.js`, `.jsx` under the
143
+ project's source roots (e.g., `src/`, `packages/*/src/`). Skip `node_modules`,
144
+ `dist`, build outputs, and fixtures.
145
+ 2. **Enumerate test files**: glob for `*.test.*`, `*.spec.*`, and files under
146
+ `__tests__/` and parallel `tests/` directories.
147
+ 3. **Map source → test**: for each source file, find its test file using the
148
+ Tier 1 / Tier 2 / Tier 3 strategies described in `Phase 2: DISCOVER`. With a
149
+ graph, prefer `get_impact`; without one, use naming conventions and import
150
+ parsing.
151
+ 4. **Split**: produce two lists — `covered` (source files with at least one
152
+ matching test) and `uncovered` (source files with no matching test).
153
+
154
+ ### Audit Phase 2: QUALITY REVIEW — Critique Covered Test Files
155
+
156
+ 1. **Cap at 10 files per run**: a deep quality review is expensive. Pick the
157
+ top 10 covered files prioritized by size (lines of code) and criticality
158
+ (depth in the graph / co-change frequency). The remaining covered files are
159
+ accepted as-is for this run.
160
+ 2. **For each of the 10**: dispatch `canary:canary-review-test` against the
161
+ test file. Collect its findings (missing edge cases, anti-patterns,
162
+ brittleness, oracle gaps).
163
+ 3. **Aggregate**: bin findings by severity (high / medium / low) and by file.
164
+
165
+ ### Audit Phase 3: GAP REPORT — Synthesize a Unified Coverage Plan
166
+
167
+ Emit a single report with three sections:
168
+
169
+ ```
170
+ ## Coverage Audit Report
171
+
172
+ ### Uncovered Files (no test found)
173
+ | File | Lines | Priority | Suggested Action |
174
+ |------|-------|----------|------------------|
175
+ | src/services/billing.ts | 412 | high | canary:canary-write-test |
176
+ | src/utils/format.ts | 38 | low | canary:canary-write-test |
177
+
178
+ ### Quality Gaps (from Quality Review, capped at 10 reviewed)
179
+ | Test File | Severity | Top Gap |
180
+ |-----------|----------|---------|
181
+ | tests/services/auth.test.ts | high | no failure-path assertions |
182
+ | tests/utils/date.test.ts | med | mock leaks across cases |
183
+
184
+ ### Recommended Next Steps
185
+ - Generate tests for high-priority uncovered files via `canary:canary-write-test`.
186
+ - Resolve high-severity quality gaps via `canary:canary-review-test` follow-ups.
187
+ - For uncovered files, pick a framework first: when Phase 0 reported `available`, call
188
+ the `canary_recommend_framework` MCP tool with a prompt describing the file's purpose
189
+ (deterministic, no LLM round-trip) and put its `framework` in the Suggested Action.
190
+ When `degraded`, fall back to the `canary:canary-pick-framework` plugin.
191
+ ```
192
+
115
193
  ## Harness Integration
116
194
 
117
195
  - **`harness scan`** — Recommended before this skill for full graph-enhanced analysis. If graph is missing, skill uses naming convention and import parsing fallbacks.
@@ -1,6 +1,6 @@
1
1
  name: harness-test-advisor
2
2
  version: "1.0.0"
3
- description: Graph-based test selection — answers "what tests should I run?"
3
+ description: Graph-based test selection and project-wide coverage audit — answers "what tests should I run?" or "what's untested?"
4
4
  stability: static
5
5
  cognitive_mode: advisory-guide
6
6
  triggers:
@@ -25,6 +25,9 @@ cli:
25
25
  - name: files
26
26
  description: Comma-separated list of changed files
27
27
  required: false
28
+ - name: audit
29
+ description: Run Coverage Audit mode (project-wide gap analysis) instead of test selection
30
+ required: false
28
31
  mcp:
29
32
  tool: run_skill
30
33
  input:
@@ -29,6 +29,8 @@ Detect plugin-only state by checking whether `harness` resolves on PATH (`comman
29
29
 
30
30
  ## Process
31
31
 
32
+ **Prompt the human in plain text** — every framework confirmation, migration check, and telemetry-identity question in this skill is plain text only. Do not elevate to `AskUserQuestion`: the framework list (~10 options) exceeds its 4-option cap and natural headers like "Confirm framework" exceed its 12-char cap, rendering the call as ERR.
33
+
32
34
  ### Phase 1: ASSESS — Determine Current State
33
35
 
34
36
  1. **Check for existing harness configuration.** Look for `.harness/` directory, `AGENTS.md`, `harness.config.json`, and any skill definitions. Their presence determines whether this is a new project or a migration.
@@ -96,7 +98,7 @@ Detect plugin-only state by checking whether `harness` resolves on PATH (`comman
96
98
  5. **Configure i18n (all levels).** Ask: "Will this project support multiple languages?" Based on the answer:
97
99
  - **Yes:** Invoke `harness-i18n-workflow` configure phase to set up i18n config in `harness.config.json` (source locale, target locales, framework, strictness). Then invoke `harness-i18n-workflow` scaffold phase to create translation file structure and extraction config. Set `i18n.enabled: true`.
98
100
  - **No:** Set `i18n.enabled: false` in `harness.config.json`. The `harness-i18n-process` skill will still fire gentle prompts for unconfigured projects when features touch user-facing strings.
99
- - **Not sure yet:** Skip i18n configuration entirely. Do not set `i18n.enabled`. The project can enable i18n later by running `harness-i18n-workflow` directly.
101
+ - **Not sure:** Skip i18n configuration entirely. Do not set `i18n.enabled`. The project can enable i18n later by running `harness-i18n-workflow` directly.
100
102
 
101
103
  5b. **Configure design system (non-test-suite projects).** Ask: "Will this project have a UI requiring a design system?" Mirror the i18n step's three-way response shape. Use `emit_interaction`:
102
104
 
@@ -136,7 +138,7 @@ Detect plugin-only state by checking whether `harness` resolves on PATH (`comman
136
138
  Based on the answer:
137
139
  - **Yes:** Ask a follow-up: "Which platforms? `web`, `mobile`, or both?" Write `design.enabled: true` and `design.platforms: [...]` (a non-empty array of `web` and/or `mobile`) to `harness.config.json`. Inform the user: "Design tokens will be generated when you start your first design-touching feature — `harness-design-system` fires automatically via `on_new_feature`."
138
140
  - **No:** Write `design.enabled: false` to `harness.config.json`. Do not write `design.platforms`. The `on_new_feature` trigger respects this flag and will not fire `harness-design-system`.
139
- - **Not sure yet:** Do not write `design.enabled` or `design.platforms`. The project can enable design later by running `harness-design-system` directly; `on_new_feature` will prompt gently when a feature touches user-facing UI.
141
+ - **Not sure:** Do not write `design.enabled` or `design.platforms`. The project can enable design later by running `harness-design-system` directly; `on_new_feature` will prompt gently when a feature touches user-facing UI.
140
142
 
141
143
  **Skip this step entirely if Phase 1 step 5 classified the project as a test suite.** Test-suite projects will be dispatched at step 6 below to `initialize-test-suite-project` and have no UI to govern.
142
144
 
@@ -183,7 +185,7 @@ Detect plugin-only state by checking whether `harness` resolves on PATH (`comman
183
185
  - **Absent (most common on init).** Present the prompt above. Apply the answer:
184
186
  - **Yes:** delegate to `harness-strategy` (which routes via its own Phase 0 to the first-run interview). When `harness-strategy` completes, continue with step 6.
185
187
  - **No:** write `init.strategy.declined: true` to `.harness/state.json` (merge into existing JSON; do not clobber). This is the explicit decline location; re-running init detects the flag and skips the prompt.
186
- - **Not sure yet:** no state write. `/harness:strategy` remains available standalone, and a future re-run of init will re-offer.
188
+ - **Not sure:** no state write. `/harness:strategy` remains available standalone, and a future re-run of init will re-offer.
187
189
 
188
190
  - **Present and valid.** Skip the prompt silently. Surface a one-line note: `STRATEGY.md detected — downstream skills will pick it up as grounding`. No `init.strategy.declined` write.
189
191
 
@@ -303,7 +305,7 @@ This phase closes the parity gap that the marketplace plugin install does not co
303
305
 
304
306
  Based on the answer:
305
307
  - **Yes:** Invoke `harness-roadmap` (skill) or run `/harness:roadmap --create` to create `docs/roadmap.md`. Verify the file exists. The `manage_roadmap` MCP tool is for managing entries in an existing roadmap, not for creating one.
306
- - **If `design.enabled === true` in `harness.config.json`** (set by Phase 3 step 5b), call `manage_roadmap` with `action: add`, `feature: "Set up design system"`, `status: "planned"`, `milestone: "Current Work"`, `summary: "Run harness-design-system to define palette, typography, and generate W3C DTCG tokens. Deferred from project init — fires on first design-touching feature via on_new_feature."`. Skip silently if `manage_roadmap show` reports a duplicate `(feature, milestone)` pair. This closes the loop between deferred design intent and visible planning work.
308
+ - **If `design.enabled === true` in `harness.config.json`** (set by Phase 3 step 5b), call `manage_roadmap` with `action: add`, `feature: "Set up design system"`, `status: "planned"`, `milestone: "Intake"`, `summary: "Run harness-design-system to define palette, typography, and generate W3C DTCG tokens. Deferred from project init — fires on first design-touching feature via on_new_feature."`. Skip silently if `manage_roadmap show` reports a duplicate `(feature, milestone)` pair. This closes the loop between deferred design intent and visible planning work.
307
309
  - **No:** Skip silently. The user can still run `/harness:roadmap --create` later — that informational fallback remains valid.
308
310
 
309
311
  2. **Commit the initialization.** All generated, configured, and instrumentation files in a single commit. Include `harness.config.json`, `AGENTS.md`, `.harness/arch/baselines.json`, `.harness/telemetry.json` (if created), updates to project `.mcp.json` (if Tier-0 integrations were wired), and any roadmap files.
@@ -323,11 +325,11 @@ This phase closes the parity gap that the marketplace plugin install does not co
323
325
  - **`harness integrations list` / `harness integrations add <name>`** — Phase 5 step 6. Lists Tier-0 (zero-config) and Tier-1 (API-key) MCP integrations and adds them to project `.mcp.json`.
324
326
  - **`harness-i18n-workflow configure` + `harness-i18n-workflow scaffold`** — Invoked during Phase 3 if the project will support multiple languages. Sets up i18n configuration and translation file structure.
325
327
  - **`harness-design-system` (deferred via `on_new_feature`)** — Phase 3 step 5b records `design.enabled` + `design.platforms` in `harness.config.json` but does NOT run the full design-system skill. Token generation defers to the first design-touching feature, where `harness-design-system` fires via `on_new_feature` and reads `design.enabled` to decide whether to proceed.
326
- - **`harness-strategy`** — Phase 3 step 5c delegates to this skill on "Yes". The skill conducts a first-run interview (pushback rules with a 2-round cap per section) and writes a valid `STRATEGY.md` at repo root via `writeStrategyDoc`. On "No" init writes `init.strategy.declined: true` to `.harness/state.json`. On "Not sure yet" no state is written; the user can run `/harness:strategy` standalone. When `STRATEGY.md` already exists and is valid the prompt is skipped; when present-but-invalid the user gets the three-path repair offer.
328
+ - **`harness-strategy`** — Phase 3 step 5c delegates to this skill on "Yes". The skill conducts a first-run interview (pushback rules with a 2-round cap per section) and writes a valid `STRATEGY.md` at repo root via `writeStrategyDoc`. On "No" init writes `init.strategy.declined: true` to `.harness/state.json`. On "not sure" no state is written; the user can run `/harness:strategy` standalone. When `STRATEGY.md` already exists and is valid the prompt is skipped; when present-but-invalid the user gets the three-path repair offer.
327
329
  - **`validateStrategy`** — `@harness-engineering/core` helper used by Phase 3 step 5c to detect present-but-invalid `STRATEGY.md`. Invoked through a Node one-liner so init does not require importing core directly.
328
330
  - **`initialize-test-suite-project`** — Sub-skill. Invoked during Phase 3 step 6 when Phase 1 step 5 classified the project as a test suite. Owns archetype selection, shared-library vs in-repo decision, layer variants, tag taxonomy, reporter stack, custom report, and "prove the guards fire" verification.
329
331
  - **`harness-roadmap` skill** — Phase 6 step 1 invokes this skill (or `/harness:roadmap --create`) when the user opts in to creating `docs/roadmap.md`. The `manage_roadmap` MCP tool does not create roadmaps; it manages entries in an existing one.
330
- - **`manage_roadmap` MCP tool** — Phase 6 step 1, when `design.enabled === true`, calls `manage_roadmap` with `action: add` to insert a `planned` "Set up design system" item under milestone `Current Work` with a summary describing the deferred work.
332
+ - **`manage_roadmap` MCP tool** — Phase 6 step 1, when `design.enabled === true`, calls `manage_roadmap` with `action: add` to insert a `planned` "Set up design system" item under milestone `Intake` with a summary describing the deferred work.
331
333
 
332
334
  ## Success Criteria
333
335
 
@@ -340,29 +342,29 @@ This phase closes the parity gap that the marketplace plugin install does not co
340
342
  - The adoption level matches what was agreed upon with the human
341
343
  - All generated files are committed in a single atomic commit
342
344
  - i18n configuration is set if the human chose to enable it during init
343
- - For non-test-suite projects, the design-system question was asked and `harness.config.json` reflects the answer: `design.enabled: true` (with `design.platforms` populated) for yes, `design.enabled: false` for no, or absent for not-sure.
344
- - The strategy question (Phase 3 step 5c) was asked unless `STRATEGY.md` already existed and was valid (in which case the prompt was skipped silently with a one-line detection note). The answer was recorded according to the documented semantics: Yes → `STRATEGY.md` exists and `harness validate` passes against `StrategyDocSchema`; No → `.harness/state.json` contains `init.strategy.declined: true`; Not sure yet → no `STRATEGY.md` and no `init.strategy.declined` flag.
345
+ - For non-test-suite projects, the design-system question was asked and `harness.config.json` reflects the answer: `design.enabled: true` (with `design.platforms` populated) for yes, `design.enabled: false` for no, or absent for not sure.
346
+ - The strategy question (Phase 3 step 5c) was asked unless `STRATEGY.md` already existed and was valid (in which case the prompt was skipped silently with a one-line detection note). The answer was recorded according to the documented semantics: Yes → `STRATEGY.md` exists and `harness validate` passes against `StrategyDocSchema`; No → `.harness/state.json` contains `init.strategy.declined: true`; Not sure → no `STRATEGY.md` and no `init.strategy.declined` flag.
345
347
  - **Phase 5 (INSTRUMENT) outputs:** `.harness/graph/` is populated; `.harness/arch/baselines.json` exists; `harness check-perf` ran without errors (intermediate and above); `.harness/telemetry.json` exists if the human opted into identity tagging; legacy layout warnings were surfaced via `harness migrate --dry-run` and either resolved or explicitly deferred; Tier-0 MCP integrations (context7, sequential-thinking, playwright) are present in the project's `.mcp.json` (or the human declined and that decision is recorded).
346
348
  - The roadmap question was asked. If the user answered yes, `docs/roadmap.md` exists and was created via `harness-roadmap` (or the documented `/harness:roadmap --create` fallback).
347
- - When `design.enabled === true` AND the user answered yes to the roadmap question, `docs/roadmap.md` contains a `planned` entry titled "Set up design system" under milestone `Current Work` with a summary describing the deferred work. The entry is absent in all other answer combinations.
349
+ - When `design.enabled === true` AND the user answered yes to the roadmap question, `docs/roadmap.md` contains a `planned` entry titled "Set up design system" under milestone `Intake` with a summary describing the deferred work. The entry is absent in all other answer combinations.
348
350
  - For test suites: `initialize-test-suite-project` ran to completion and its Success Criteria are also met
349
351
  - For plugin-only invocations: every CLI invocation in this skill ran successfully via `npx @harness-engineering/cli <cmd>` without requiring a global install. If any invocation failed because the npx download failed, the human was prompted to retry or `npm install -g @harness-engineering/cli` instead.
350
352
 
351
353
  ## Rationalizations to Reject
352
354
 
353
- | Rationalization | Why It Is Wrong |
354
- | ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
355
- | "The generated AGENTS.md template looks fine -- no need to customize it" | Phase 3 says do not blindly accept generated content. Without project-specific descriptions, agents receive generic instructions. |
356
- | "We should start at the advanced level since we want full coverage" | The skill recommends basic for new projects. Each level builds on the previous. Jumping to advanced creates misconfigured rules. |
357
- | "I will skip the i18n question to keep setup fast" | Phase 3 requires asking about i18n and recording the decision. Skipping creates ambiguity about whether the omission was intentional. |
358
- | "I will skip the design-system question to keep setup fast" | Phase 3 step 5b requires asking about design and recording the answer in `design.enabled`. Skipping creates ambiguity about whether the omission was intentional and bypasses the linkage between init and the deferred `harness-design-system` invocation on `on_new_feature`. |
359
- | "I will skip the strategy question — it's just paperwork" | Phase 3 step 5c is the only point in the workflow where init asks the user to capture the strategic anchor. Skipping bypasses the grounding signal that brainstorming, ideate, and roadmap-pilot read; downstream skill output degrades silently. Even a "No" or "Not sure yet" answer is better than no answer because it locks in the decision (or absence of one). |
360
- | "STRATEGY.md exists already, so I should re-run the interview to refresh it" | When `STRATEGY.md` is present and valid, Phase 3 step 5c skips the prompt silently. Refreshing strategy mid-init is out of scope — that is what the standalone `/harness:strategy` update flow is for. Surface a one-line detection note and continue. |
361
- | "STRATEGY.md is present but invalid, so I should block init" | Phase 3 step 5c explicitly does NOT block. It surfaces the validation error, offers three repair paths (fix now / move-to-bak / ignore), and continues based on the user's choice. Init is the wrong place to gate on strategy correctness. |
362
- | "Validation passed, so the project is ready" | Phase 5 captures baselines, configures telemetry identity, surfaces legacy warnings, and wires Tier-0 integrations. Validation alone is not sufficient. |
363
- | "Plugin install means setup is done" | The marketplace plugin ships skills, slash commands, subagents, hooks, and MCP — but it cannot mutate the user's project state. `harness.config.json`, baselines, telemetry identity, and Tier-0 integrations require running this skill once per project. |
364
- | "Skip Phase 5 if we already ran `harness setup`" | Phase 5 is idempotent. It safely no-ops where setup already wired things and fills gaps where it didn't (common case: setup ran once, then a new Tier-0 integration was added or a layout was migrated). Re-running is the right behavior. |
365
- | "This is a test suite, we'll configure layers in this skill" | Phase 3 step 6 dispatches to `initialize-test-suite-project` for archetype selection, layer variants, and the rest. Do not inline test-suite-specific configuration here — the sub-skill owns it and carries the gotchas. |
355
+ | Rationalization | Why It Is Wrong |
356
+ | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
357
+ | "The generated AGENTS.md template looks fine -- no need to customize it" | Phase 3 says do not blindly accept generated content. Without project-specific descriptions, agents receive generic instructions. |
358
+ | "We should start at the advanced level since we want full coverage" | The skill recommends basic for new projects. Each level builds on the previous. Jumping to advanced creates misconfigured rules. |
359
+ | "I will skip the i18n question to keep setup fast" | Phase 3 requires asking about i18n and recording the decision. Skipping creates ambiguity about whether the omission was intentional. |
360
+ | "I will skip the design-system question to keep setup fast" | Phase 3 step 5b requires asking about design and recording the answer in `design.enabled`. Skipping creates ambiguity about whether the omission was intentional and bypasses the linkage between init and the deferred `harness-design-system` invocation on `on_new_feature`. |
361
+ | "I will skip the strategy question — it's just paperwork" | Phase 3 step 5c is the only point in the workflow where init asks the user to capture the strategic anchor. Skipping bypasses the grounding signal that brainstorming, ideate, and roadmap-pilot read; downstream skill output degrades silently. Even a "no" or "not sure" answer is better than no answer because it locks in the decision (or absence of one). |
362
+ | "STRATEGY.md exists already, so I should re-run the interview to refresh it" | When `STRATEGY.md` is present and valid, Phase 3 step 5c skips the prompt silently. Refreshing strategy mid-init is out of scope — that is what the standalone `/harness:strategy` update flow is for. Surface a one-line detection note and continue. |
363
+ | "STRATEGY.md is present but invalid, so I should block init" | Phase 3 step 5c explicitly does NOT block. It surfaces the validation error, offers three repair paths (fix now / move-to-bak / ignore), and continues based on the user's choice. Init is the wrong place to gate on strategy correctness. |
364
+ | "Validation passed, so the project is ready" | Phase 5 captures baselines, configures telemetry identity, surfaces legacy warnings, and wires Tier-0 integrations. Validation alone is not sufficient. |
365
+ | "Plugin install means setup is done" | The marketplace plugin ships skills, slash commands, subagents, hooks, and MCP — but it cannot mutate the user's project state. `harness.config.json`, baselines, telemetry identity, and Tier-0 integrations require running this skill once per project. |
366
+ | "Skip Phase 5 if we already ran `harness setup`" | Phase 5 is idempotent. It safely no-ops where setup already wired things and fills gaps where it didn't (common case: setup ran once, then a new Tier-0 integration was added or a layout was migrated). Re-running is the right behavior. |
367
+ | "This is a test suite, we'll configure layers in this skill" | Phase 3 step 6 dispatches to `initialize-test-suite-project` for archetype selection, layer variants, and the rest. Do not inline test-suite-specific configuration here — the sub-skill owns it and carries the gotchas. |
366
368
 
367
369
  ## Examples
368
370
 
@@ -458,7 +460,7 @@ Step 4 (roadmap): "Set up a project roadmap now?"
458
460
  design.enabled === true detected → manage_roadmap action: add
459
461
  feature: "Set up design system"
460
462
  status: planned
461
- milestone: Current Work
463
+ milestone: Intake
462
464
  summary: "Run harness-design-system to define palette, typography, and generate W3C DTCG tokens..."
463
465
  Result: docs/roadmap.md contains the planned design item.
464
466
 
@@ -470,7 +472,7 @@ git add harness.config.json .harness/ AGENTS.md docs/roadmap.md
470
472
  git commit -m "feat: initialize harness project with design and roadmap"
471
473
  ```
472
474
 
473
- **Final state:** `harness.config.json` has `design.enabled: true` + `design.platforms: ["web"]`; `docs/roadmap.md` lists "Set up design system" as a `planned` item under `Current Work`; on the first feature touching UI, `on_new_feature` fires `harness-design-system` which reads `design.enabled` and runs the full discover/define/generate/validate flow.
475
+ **Final state:** `harness.config.json` has `design.enabled: true` + `design.platforms: ["web"]`; `docs/roadmap.md` lists "Set up design system" as a `planned` item under `Intake`; on the first feature touching UI, `on_new_feature` fires `harness-design-system` which reads `design.enabled` and runs the full discover/define/generate/validate flow.
474
476
 
475
477
  ### Example: Migrating Existing Project from Basic to Intermediate
476
478
 
@@ -547,7 +549,7 @@ npx @harness-engineering/cli init # auto-detects framework
547
549
  Customize AGENTS.md with project-specific content.
548
550
  Phase 3 step 5 (i18n): "No, English only." → i18n.enabled = false.
549
551
  Phase 3 step 5b (design): "No, this is a backend service." → design.enabled = false.
550
- Phase 3 step 5c (strategy): "Not sure yet." → no STRATEGY.md, no init.strategy.declined flag; user can run /harness:strategy later.
552
+ Phase 3 step 5c (strategy): "not sure." → no STRATEGY.md, no init.strategy.declined flag; user can run /harness:strategy later.
551
553
  Phase 3 step 6 (test-suite dispatch): not a test suite, skipped.
552
554
  ```
553
555
 
@@ -35,3 +35,4 @@ state:
35
35
  depends_on:
36
36
  - initialize-test-suite-project
37
37
  - harness-design-system
38
+ - harness-roadmap
@@ -0,0 +1,97 @@
1
+ # Outcome Eval
2
+
3
+ > Post-execution LLM-judgment: did the implementation actually satisfy its spec? Reads the spec's acceptance section, the change diff, and test output, and emits a confidence-rated `OutcomeVerdict` (`SATISFIED | NOT_SATISFIED | INCONCLUSIVE`) with a rationale and unmet criteria. Ship authority is derived in TypeScript, never trusted from the LLM: a high-confidence `NOT_SATISFIED` blocks ship; every other verdict is advisory. The harness's first blocking post-execution spec-satisfaction gate (the roadmap's named #1 gap). Each verdict persists as an `execution_outcome` node, compounding into skill-effectiveness baselines.
4
+
5
+ ## When to Use
6
+
7
+ - At orchestrator step 6.5 — after Code Review, before Ship — on every change with a spec.
8
+ - When you need a durable, structured answer to "did this code do what the spec said?"
9
+ - NOT for pre-execution risk simulation (use PESL).
10
+ - NOT for rule-based floors (lint/architecture/entropy) or craft ceilings (naming/spec/security) — those run elsewhere.
11
+ - NOT for auto-remediation. outcome-eval judges; it does not fix.
12
+ - NOT when no judgable spec section exists — the verdict degrades to INCONCLUSIVE/advisory and never blocks.
13
+
14
+ ## Process
15
+
16
+ ### Phase 1: GATHER — Collect inputs
17
+
18
+ 1. Capture the change under judgment as a unified diff: `git diff` (or `git diff <base>...HEAD` for a branch). Record it as `diff`.
19
+ 2. Capture test-runner output. If a test command is known, run it and capture stdout+stderr as `testOutput`; otherwise pass the most recent captured output. Empty/unparseable test output is tolerated (degrades to advisory).
20
+ 3. Resolve the spec path. Prefer the spec under `docs/changes/<feature>/proposal.md` for the current change. Record as `specPath`.
21
+
22
+ ### Phase 2: RESOLVE — Find the judgment section
23
+
24
+ The evaluator resolves the section internally via the fallback chain `## Success Criteria` -> `## User-Visible Behavior` -> `## Overview`, recording the match in `judgedAgainst`. No manual action — pass `specPath` and let `OutcomeEvaluator` resolve. If no section is judgable, the verdict is INCONCLUSIVE/advisory.
25
+
26
+ ### Phase 3: JUDGE — Invoke the evaluator
27
+
28
+ 1. Invoke the MCP tool `mcp__harness__outcome_eval` with `{ specPath, diff, testOutput }` (optional `model`). The tool constructs `OutcomeEvaluator` cli-side and calls `evaluate({ specPath, diff, testOutput })`; the supported v1 provider is the anthropic analysis provider (`ANTHROPIC_API_KEY`).
29
+ 2. **`diff` and `testOutput` are required inputs and the agent MUST supply them** from the session (`git diff` + captured test-runner output). They are the evidence the judge reasons over — passing an empty `diff` or empty `testOutput` is the degradation path, not the normal path: the verdict degrades to INCONCLUSIVE/advisory (never blocking), which defeats the gate. Do not invoke the tool without real diff/test content.
30
+ 3. The LLM returns ONLY `verdict / confidence / rationale / unmetCriteria`. `authority` is computed in TypeScript from `(verdict, confidence)` and is never read from the LLM — do not attempt to override it. The tool returns the verdict exactly as the evaluator derives it.
31
+ 4. The call is degrade-safe: provider failure (incl. no `ANTHROPIC_API_KEY`), empty diff, empty test output, or missing judgable section yields INCONCLUSIVE/low/advisory. It never throws and never blocks.
32
+
33
+ ### Phase 4: GATE — Render and (conditionally) halt
34
+
35
+ 1. Render the verdict: `verdict`, `confidence`, `judgedAgainst`, `rationale`, and `unmetCriteria`.
36
+ 2. Authority rule (must match `deriveAuthority`): authority is `blocking` **iff** `verdict === 'NOT_SATISFIED' && confidence === 'high'`; every other combination — including all `INCONCLUSIVE` and `SATISFIED` cases, and all `medium`/`low` `NOT_SATISFIED` — is `advisory`.
37
+ 3. **On a blocking verdict: HALT before the Ship step.** Report the unmet criteria and stop; do not proceed to step 7. Resolution requires fixing the implementation (or the spec) and re-running outcome-eval.
38
+ 4. On an advisory verdict: report it and proceed. Advisory `NOT_SATISFIED` is surfaced for human attention but does not stop the workflow.
39
+
40
+ ## Harness Integration
41
+
42
+ - **`mcp__harness__outcome_eval`** — MCP tool (the invocation surface). Inputs: `specPath` (required), `diff` (required), `testOutput` (required), `model` (optional), `path` (optional project root for graph persistence). The agent supplies `diff` and `testOutput` from the session; omitting them degrades the verdict to INCONCLUSIVE/advisory (never blocking). The handler builds the cli `AnalysisProvider` + a `GraphStore`, constructs `OutcomeEvaluator`, and returns the `OutcomeVerdict` with authority exactly as derived in TypeScript.
43
+ - **Evaluator surface:** `OutcomeEvaluator`, `deriveAuthority`, `verdictSchema`, `OutcomeVerdict` are exported from `@harness-engineering/intelligence`.
44
+ - **Provider path (v1 supported):** the anthropic analysis provider (`ANTHROPIC_API_KEY`). When no provider is configured the call degrades to INCONCLUSIVE/advisory. The openai-compatible _strict_ structured-output path is a known follow-up (see Known Limitations).
45
+ - **Orchestrator:** runs as step 6.5 between Code Review and Ship in `harness.orchestrator.md`.
46
+ - **Persistence:** each `evaluate()` writes one `execution_outcome` node via `ExecutionOutcomeConnector`, consumable by `effectiveness/scorer.ts`.
47
+
48
+ ## Known Limitations
49
+
50
+ - **INCONCLUSIVE persistence:** the persisted node maps `INCONCLUSIVE -> result: 'failure'` for type-validity, but it OMITS `agentPersona` and writes `affectedSystemNodeIds: []`. The effectiveness scorer (`gatherOutcomes`) ignores any node missing `agentPersona` or `outcome_of` edges, so outcome-eval nodes are **scorer-non-counting** in v1 — the INCONCLUSIVE-as-failure mapping is therefore harmless and does not punish any persona. If a future change attaches persona/affected-system attribution, it MUST first change INCONCLUSIVE modeling (do not persist INCONCLUSIVE, or use a distinct result value the scorer excludes) before the node becomes scorer-counted.
51
+ - **openai-compatible strict mode:** `zodToJsonSchema` does not emit `additionalProperties: false`, which OpenAI strict structured output requires. The v1 supported path is claude-cli / anthropic. Follow-up tracked.
52
+ - **CI required-check wiring:** deferred to roadmap #540 (unbuilt CI workflow template).
53
+
54
+ ## Success Criteria
55
+
56
+ See `docs/changes/outcome-eval/proposal.md` for the full 9 criteria. This skill satisfies SC8 (orchestrator step 6.5 + blocking halt) and SC9 (introduces no new `harness validate` findings; layer rules respected).
57
+
58
+ ## Examples
59
+
60
+ ### Example: NOT_SATISFIED with high confidence (blocks)
61
+
62
+ **Input:** spec Success Criteria require `GET /api/users/:id` to return 404 with `{ error: 'User not found' }`; the diff implements the happy path only, no 404 branch; test output shows the 404 test failing.
63
+
64
+ **Verdict:**
65
+
66
+ ```
67
+ verdict: NOT_SATISFIED
68
+ confidence: high
69
+ judgedAgainst: success-criteria
70
+ authority: blocking
71
+ unmetCriteria:
72
+ - "404 path for nonexistent user is unimplemented; the failing test asserts { error: 'User not found' }."
73
+ rationale: "The diff adds the lookup but returns 200 with an empty body when the user is missing."
74
+ ```
75
+
76
+ **Action:** HALT before Ship. Report unmet criteria; do not open the PR.
77
+
78
+ ### Example: partial implementation (advisory)
79
+
80
+ **Input:** the diff meets most criteria; one acceptance item is ambiguous in the diff.
81
+
82
+ **Verdict:** `NOT_SATISFIED confidence: medium authority: advisory` — surfaced for review, workflow proceeds.
83
+
84
+ ## Gates
85
+
86
+ - **Authority is never read from the LLM.** The verdict's `authority` is always `deriveAuthority(verdict, confidence)` computed in TypeScript. If you find yourself letting the model assert blocking/advisory, STOP — that defeats the entire purpose of this gate.
87
+ - **Block only on high-confidence NOT_SATISFIED.** `authority === 'blocking'` iff `verdict === 'NOT_SATISFIED' && confidence === 'high'`. Every other combination — all `SATISFIED`, all `INCONCLUSIVE`, and every `medium`/`low` `NOT_SATISFIED` — is advisory. Do not halt the workflow on an advisory verdict.
88
+ - **Always supply `diff` and `testOutput`.** Omitting them degrades the verdict to INCONCLUSIVE/advisory (a silent false-negative at the ship gate). Gather them from the session before invoking the tool.
89
+ - **Never block on infrastructure noise.** A provider failure, an unparseable response, or a missing spec must resolve to INCONCLUSIVE/advisory, never a thrown error or a block. The evaluator enforces this; do not reintroduce a hard failure in the wrapper.
90
+ - **Do not skip step 6.5.** The gate runs after Code Review and before Ship. A blocking verdict halts before step 7; it is not optional.
91
+
92
+ ## Escalation
93
+
94
+ - **Blocking verdict the author disputes:** the resolution is to fix the implementation (or amend the spec's Success Criteria if they were wrong) and re-run outcome-eval — not to override the gate. If the spec itself is wrong, that is a spec change, reviewed on its own merits.
95
+ - **Repeated INCONCLUSIVE on a real change:** usually means `diff`/`testOutput` were not supplied, or no judgable section exists in the spec. Confirm inputs and that the spec has a Success Criteria / User-Visible Behavior / Overview section.
96
+ - **No `ANTHROPIC_API_KEY` configured:** every verdict degrades to INCONCLUSIVE/advisory and nothing blocks. Surface this to the human — the gate is effectively disabled until a provider is configured.
97
+ - **Verdict seems wrong (false positive/negative):** capture the spec section, diff, and verdict, and route to the maintainers; do not loosen the conservative-confidence prompt ad hoc.