@harness-engineering/cli 2.8.0 → 3.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/agents/skills/claude-code/harness-audit-harness-strength/SKILL.md +188 -0
- package/dist/agents/skills/claude-code/harness-audit-harness-strength/skill.yaml +54 -0
- package/dist/agents/skills/claude-code/harness-autopilot/SKILL.md +2 -0
- package/dist/agents/skills/claude-code/harness-brainstorming/SKILL.md +28 -14
- package/dist/agents/skills/claude-code/harness-execution/SKILL.md +1 -1
- package/dist/agents/skills/claude-code/harness-git-workflow/SKILL.md +2 -0
- package/dist/agents/skills/claude-code/harness-ideate/SKILL.md +2 -0
- package/dist/agents/skills/claude-code/harness-knowledge-pipeline/SKILL.md +2 -0
- package/dist/agents/skills/claude-code/harness-roadmap/SKILL.md +62 -4
- package/dist/agents/skills/claude-code/harness-roadmap-pilot/SKILL.md +2 -0
- package/dist/agents/skills/claude-code/harness-test-advisor/SKILL.md +80 -2
- package/dist/agents/skills/claude-code/harness-test-advisor/skill.yaml +4 -1
- package/dist/agents/skills/claude-code/initialize-harness-project/SKILL.md +27 -25
- package/dist/agents/skills/claude-code/initialize-harness-project/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/outcome-eval/SKILL.md +97 -0
- package/dist/agents/skills/claude-code/outcome-eval/skill.yaml +51 -0
- package/dist/agents/skills/codex/harness-audit-harness-strength/SKILL.md +188 -0
- package/dist/agents/skills/codex/harness-audit-harness-strength/skill.yaml +54 -0
- package/dist/agents/skills/codex/harness-autopilot/SKILL.md +2 -0
- package/dist/agents/skills/codex/harness-brainstorming/SKILL.md +28 -14
- package/dist/agents/skills/codex/harness-execution/SKILL.md +1 -1
- package/dist/agents/skills/codex/harness-git-workflow/SKILL.md +2 -0
- package/dist/agents/skills/codex/harness-ideate/SKILL.md +2 -0
- package/dist/agents/skills/codex/harness-knowledge-pipeline/SKILL.md +2 -0
- package/dist/agents/skills/codex/harness-roadmap/SKILL.md +62 -4
- package/dist/agents/skills/codex/harness-roadmap-pilot/SKILL.md +2 -0
- package/dist/agents/skills/codex/harness-test-advisor/SKILL.md +80 -2
- package/dist/agents/skills/codex/harness-test-advisor/skill.yaml +4 -1
- package/dist/agents/skills/codex/initialize-harness-project/SKILL.md +27 -25
- package/dist/agents/skills/codex/initialize-harness-project/skill.yaml +1 -0
- package/dist/agents/skills/codex/outcome-eval/SKILL.md +97 -0
- package/dist/agents/skills/codex/outcome-eval/skill.yaml +51 -0
- package/dist/agents/skills/cursor/harness-audit-harness-strength/SKILL.md +188 -0
- package/dist/agents/skills/cursor/harness-audit-harness-strength/skill.yaml +54 -0
- package/dist/agents/skills/cursor/harness-autopilot/SKILL.md +2 -0
- package/dist/agents/skills/cursor/harness-brainstorming/SKILL.md +28 -14
- package/dist/agents/skills/cursor/harness-execution/SKILL.md +1 -1
- package/dist/agents/skills/cursor/harness-git-workflow/SKILL.md +2 -0
- package/dist/agents/skills/cursor/harness-ideate/SKILL.md +2 -0
- package/dist/agents/skills/cursor/harness-knowledge-pipeline/SKILL.md +2 -0
- package/dist/agents/skills/cursor/harness-roadmap/SKILL.md +62 -4
- package/dist/agents/skills/cursor/harness-roadmap-pilot/SKILL.md +2 -0
- package/dist/agents/skills/cursor/harness-test-advisor/SKILL.md +80 -2
- package/dist/agents/skills/cursor/harness-test-advisor/skill.yaml +4 -1
- package/dist/agents/skills/cursor/initialize-harness-project/SKILL.md +27 -25
- package/dist/agents/skills/cursor/initialize-harness-project/skill.yaml +1 -0
- package/dist/agents/skills/cursor/outcome-eval/SKILL.md +97 -0
- package/dist/agents/skills/cursor/outcome-eval/skill.yaml +51 -0
- package/dist/agents/skills/gemini-cli/harness-audit-harness-strength/SKILL.md +188 -0
- package/dist/agents/skills/gemini-cli/harness-audit-harness-strength/skill.yaml +54 -0
- package/dist/agents/skills/gemini-cli/harness-autopilot/SKILL.md +2 -0
- package/dist/agents/skills/gemini-cli/harness-brainstorming/SKILL.md +28 -14
- package/dist/agents/skills/gemini-cli/harness-execution/SKILL.md +1 -1
- package/dist/agents/skills/gemini-cli/harness-git-workflow/SKILL.md +2 -0
- package/dist/agents/skills/gemini-cli/harness-ideate/SKILL.md +2 -0
- package/dist/agents/skills/gemini-cli/harness-knowledge-pipeline/SKILL.md +2 -0
- package/dist/agents/skills/gemini-cli/harness-roadmap/SKILL.md +62 -4
- package/dist/agents/skills/gemini-cli/harness-roadmap-pilot/SKILL.md +2 -0
- package/dist/agents/skills/gemini-cli/harness-test-advisor/SKILL.md +80 -2
- package/dist/agents/skills/gemini-cli/harness-test-advisor/skill.yaml +4 -1
- package/dist/agents/skills/gemini-cli/initialize-harness-project/SKILL.md +27 -25
- package/dist/agents/skills/gemini-cli/initialize-harness-project/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/outcome-eval/SKILL.md +97 -0
- package/dist/agents/skills/gemini-cli/outcome-eval/skill.yaml +51 -0
- package/dist/agents/skills/tests/harness-test-advisor.test.ts +94 -0
- package/dist/{agents-md-WMJO5HSM.js → agents-md-XOJE5ETA.js} +2 -2
- package/dist/{architecture-EVYVGPRZ.js → architecture-JEZPJ725.js} +3 -3
- package/dist/{assess-project-KZG563SO.js → assess-project-RHXOQNMM.js} +1 -1
- package/dist/bin/harness-mcp.js +13 -13
- package/dist/bin/harness.js +22 -22
- package/dist/{check-phase-gate-MK364VXS.js → check-phase-gate-GNDOMS35.js} +3 -3
- package/dist/{chunk-VBNJAKUH.js → chunk-3ISHDWO7.js} +1 -1
- package/dist/{chunk-UJFNIWC7.js → chunk-527G27VI.js} +2 -2
- package/dist/{chunk-5XL5QJ5B.js → chunk-6WQCLCBO.js} +1 -1
- package/dist/{chunk-UXPWSV3B.js → chunk-BYVT5LVO.js} +3 -3
- package/dist/{chunk-HRSE6IQD.js → chunk-DE5U6KOL.js} +1 -1
- package/dist/{chunk-B7KMTHN3.js → chunk-GGKRA7A7.js} +1 -1
- package/dist/{chunk-STESM73J.js → chunk-GSP2XIVS.js} +1 -1
- package/dist/{chunk-KBCPELBC.js → chunk-HEIMJJI4.js} +2 -2
- package/dist/{chunk-KI6PHH6Q.js → chunk-HWTUUQOW.js} +3 -3
- package/dist/{chunk-2SNRCAC3.js → chunk-KAJOS3BA.js} +2 -2
- package/dist/{chunk-C7J55C6E.js → chunk-MGUPSE6D.js} +5 -5
- package/dist/{chunk-GOHWCHCS.js → chunk-N24HCQA4.js} +2243 -776
- package/dist/{chunk-EEFRPEWT.js → chunk-NWLGL3IR.js} +1 -1
- package/dist/{chunk-UUO4K7NX.js → chunk-ONCPJGHY.js} +6 -6
- package/dist/{chunk-X4EFYVCZ.js → chunk-PC2R5RUJ.js} +1 -1
- package/dist/{chunk-7EQAZKJ5.js → chunk-QBC7DI4Q.js} +1 -1
- package/dist/{chunk-UBZQE3JN.js → chunk-UX3PQKVZ.js} +7 -2
- package/dist/{chunk-CAQPSFPT.js → chunk-WWEVS7RO.js} +453 -91
- package/dist/{chunk-4BDOSCNY.js → chunk-XFU6SB2Q.js} +1015 -648
- package/dist/{chunk-LADPYELQ.js → chunk-XX4OLNWQ.js} +2 -2
- package/dist/{chunk-FZ5MBXEG.js → chunk-YQSYCBAH.js} +6 -6
- package/dist/{chunk-ACOOG3UW.js → chunk-Z4AWEIBY.js} +3 -3
- package/dist/{ci-workflow-IW2H5DC4.js → ci-workflow-ORZ4F45F.js} +2 -2
- package/dist/{dist-JCFK263L.js → dist-LHINSVK4.js} +85 -1
- package/dist/{docs-7CP6VV42.js → docs-AU3DXFFO.js} +3 -3
- package/dist/{engine-MKJEBWU7.js → engine-6GPWOYQH.js} +2 -2
- package/dist/{entropy-6JOZJYG7.js → entropy-VLKDRMRT.js} +2 -2
- package/dist/{feedback-T65AEJEG.js → feedback-YXYY44ZC.js} +1 -1
- package/dist/{generate-agent-definitions-4VI4YTO2.js → generate-agent-definitions-XIHQPKG3.js} +3 -3
- package/dist/hooks/cost-tracker.js +29 -6
- package/dist/index.d.ts +8 -8
- package/dist/index.js +22 -22
- package/dist/{loader-AWTHHE3A.js → loader-YAN56MT3.js} +2 -2
- package/dist/{mcp-JBIBINO2.js → mcp-RK3SFVYB.js} +13 -13
- package/dist/{performance-NCOUDOMW.js → performance-64DPTLLQ.js} +3 -3
- package/dist/{review-pipeline-YIC6JPMY.js → review-pipeline-NENG6LW7.js} +2 -2
- package/dist/{runtime-5MHV4UEE.js → runtime-FFIIRT2T.js} +2 -2
- package/dist/{security-VZPJNIDP.js → security-3PIBN35B.js} +1 -1
- package/dist/templates/basic/harness.config.json.hbs +34 -0
- package/dist/templates/ci/README.md +86 -0
- package/dist/templates/ci/required-review.ruleset.json +20 -0
- package/dist/templates/ci/required-review.yml.hbs +57 -0
- package/dist/templates/ci/template.json +5 -0
- package/dist/templates/intermediate/harness.config.json.hbs +36 -0
- package/dist/templates/orchestrator/harness.orchestrator.md +9 -0
- package/dist/{tool-tiers-B7JC2XC3.js → tool-tiers-OKK5FE57.js} +2 -0
- package/dist/{validate-7BE4VTFI.js → validate-7IAF2ES3.js} +3 -3
- package/dist/{validate-cross-check-F7GEBRDH.js → validate-cross-check-QPSAOZKD.js} +2 -2
- package/package.json +7 -7
|
@@ -136,7 +136,7 @@ These flow into `handoff.json` `contextKeywords` field. Select keywords that hel
|
|
|
136
136
|
- **Entry Points** -- Which system entry points does this feature touch or create? (e.g., new CLI command, new MCP tool, new skill, new API route, new barrel export)
|
|
137
137
|
- **Registrations Required** -- What registrations are needed for the feature to be discoverable? (e.g., barrel export regeneration, skill tier assignment, route registration)
|
|
138
138
|
- **Documentation Updates** -- What docs need updating to reflect the new capability? (e.g., AGENTS.md section, API docs, README, guides)
|
|
139
|
-
- **Architectural Decisions** --
|
|
139
|
+
- **Architectural Decisions** -- Which decisions from the **Decisions made** section rise to a standalone ADR? Reference each by name with a one-line note on _why_ it warrants an ADR. Do **not** restate the decisions here -- this subsection points back to the canonical **Decisions made** section; duplicating them creates two sources that drift (spec-craft flags this as SPEC-R004). Only for medium/large tier changes -- "None" for small changes.
|
|
140
140
|
- **Knowledge Impact** -- What domain concepts, patterns, or relationships should enter the knowledge graph?
|
|
141
141
|
|
|
142
142
|
If the feature is a small change (bug fix, config tweak, < 3 files), the section may contain only Entry Points and Registrations Required with "None" for the others. The section must still be present.
|
|
@@ -186,21 +186,35 @@ These flow into `handoff.json` `contextKeywords` field. Select keywords that hel
|
|
|
186
186
|
|
|
187
187
|
The human must explicitly approve before this skill is complete.
|
|
188
188
|
|
|
189
|
-
7. **
|
|
189
|
+
7. **Promote the roadmap row.** If `docs/roadmap.md` exists, transition the named row to `planned` and link the spec in a single structured call (steps 7 and 8 are intentionally ordered so the commit captures the roadmap mutation). Skip silently only when no roadmap exists.
|
|
190
|
+
- Derive the lookup key from the slash-command `ARGUMENTS` string (D1). Derive the summary from the spec title (the H1 heading).
|
|
191
|
+
- Call `manage_roadmap` with action `promote`, `feature: "<ARGUMENTS>"`, `spec: "docs/changes/<feature>/proposal.md"`, and `summary: "<H1>"`.
|
|
192
|
+
- Branch on the returned `PromoteResult` envelope:
|
|
193
|
+
|
|
194
|
+
| Envelope | Skill behavior |
|
|
195
|
+
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
196
|
+
| `ok: true, transitioned: 'backlog→planned'` | Log "Promoted `<feature>`: backlog → planned". Continue to step 8. |
|
|
197
|
+
| `ok: true, transitioned: 'spec-updated'` | Log "Updated spec link for `<feature>` (status preserved)". Continue to step 8. |
|
|
198
|
+
| `ok: true, transitioned: 'noop'` | Log "No change — `<feature>` already promoted with this spec". Continue to step 8. |
|
|
199
|
+
| `ok: false, reason: 'in-progress'` | **STOP.** Surface: "Refused to promote `<feature>`: an agent is currently dispatched against this row. Stop the agent or use a different feature name." |
|
|
200
|
+
| `ok: false, reason: 'done'` | **STOP.** Surface: "Refused to promote `<feature>`: row is already 'done'. To revise a shipped feature, use a new name." |
|
|
201
|
+
| `ok: false, reason: 'not-found'` | If no row was expected to exist, fall through to a `created` outcome. Otherwise surface `closestMatches` as a typo hint and **STOP**. |
|
|
202
|
+
| `ok: false, reason: 'ambiguous'` | **STOP.** Surface: "Refused to promote `<feature>`: matches multiple rows across milestones. Re-invoke with one of: `<matches>`." Matches are milestone-qualified. |
|
|
203
|
+
| `ok: false, reason: 'write-failed'` | **STOP.** Surface `detail` verbatim. The brainstorm is not considered complete. |
|
|
204
|
+
|
|
205
|
+
- The `not-found` create path: when the row genuinely does not exist yet, call `manage_roadmap` action `add` with `status: "planned"`, `milestone: "Current Work"`, the spec path, and the H1 summary (unchanged from the legacy behavior), then continue to step 8.
|
|
206
|
+
- "STOP" cases skip step 8 (no commit) and step 9 (no handoff/transition). The spec file written in step 3 stays on disk; the user re-runs the skill after resolving the conflict.
|
|
207
|
+
- If `manage_roadmap` is unavailable, fall back to `parseRoadmap`/`promoteFeature`/`serializeRoadmap` from core. Warn: "External sync skipped (MCP unavailable). Run `manage_roadmap sync` when MCP is restored."
|
|
208
|
+
- If no roadmap exists, skip silently (no envelope) and continue to step 8; the commit then contains only the spec files.
|
|
209
|
+
|
|
210
|
+
8. **Commit spec artifacts and roadmap promotion.** After a successful promote (or a silent skip when no roadmap exists), commit the spec, skill recommendations, and the roadmap mutation together so the promotion ships atomically with the spec:
|
|
190
211
|
|
|
191
212
|
```bash
|
|
192
|
-
git add docs/changes/<feature>/proposal.md docs/changes/<feature>/SKILLS.md
|
|
193
|
-
git commit -m "docs(<feature>): add spec"
|
|
213
|
+
git add docs/changes/<feature>/proposal.md docs/changes/<feature>/SKILLS.md docs/roadmap.md
|
|
214
|
+
git commit -m "docs(<feature>): add spec and promote to planned"
|
|
194
215
|
```
|
|
195
216
|
|
|
196
|
-
Do not skip this step
|
|
197
|
-
|
|
198
|
-
8. **Add feature to roadmap.** If `docs/roadmap.md` exists:
|
|
199
|
-
- Derive feature name from the spec title (the H1 heading).
|
|
200
|
-
- Call `manage_roadmap` with action `add`, `status: "planned"`, `milestone: "Current Work"`, and the spec path.
|
|
201
|
-
- If the feature already exists, skip silently.
|
|
202
|
-
- If `manage_roadmap` is unavailable, fall back to `parseRoadmap`/`serializeRoadmap` from core. Warn: "External sync skipped (MCP unavailable). Run `manage_roadmap sync` when MCP is restored."
|
|
203
|
-
- If no roadmap exists, skip silently.
|
|
217
|
+
Include `docs/roadmap.md` in `git add` only when it exists; omit it when the project has no roadmap. Do not skip this step — `harness-execution` commits only implementation files, so a spec uncommitted here stays untracked until someone notices. If a pre-commit hook reformats a file, re-add and re-commit.
|
|
204
218
|
|
|
205
219
|
9. **Write handoff and suggest transition.** After approval:
|
|
206
220
|
|
|
@@ -336,8 +350,8 @@ Technical claims about existing code, architecture, or tradeoffs MUST cite evide
|
|
|
336
350
|
- **Spec location** -- `docs/changes/<feature>/proposal.md`.
|
|
337
351
|
- **Handoff** -- Once approved, invoke harness-planning to create the implementation plan.
|
|
338
352
|
- **Session directory** — When session slug is known, handoff goes to `.harness/sessions/<slug>/handoff.json`. The session directory structure is: `handoff.json`, `state.json`, `artifacts.json` (registry of spec/plan paths and file lists). Do not write to `.harness/handoff.json` in session context.
|
|
339
|
-
- **Spec commit** -- After sign-off
|
|
340
|
-
- **Roadmap
|
|
353
|
+
- **Spec commit** -- After sign-off, the Phase 4 step 8 commit captures `docs/changes/<feature>/proposal.md`, `docs/changes/<feature>/SKILLS.md`, and the promoted `docs/roadmap.md` together so the spec enters git history at approval, not retroactively. `harness-execution` does not backfill these — see issue #487.
|
|
354
|
+
- **Roadmap promotion** -- After approval (Phase 4 step 7), call `manage_roadmap` action `promote` to transition the named row to `planned` and link the spec atomically with the spec commit. Falls back to `add` (create new row) when the row does not exist; refuses on `in-progress`/`done`/`ambiguous`. Skip silently if no roadmap.
|
|
341
355
|
- **`emit_interaction`** -- End of Phase 4 to suggest transition to harness-planning (confirmed transition).
|
|
342
356
|
|
|
343
357
|
#### Requirement Phrasing
|
|
@@ -56,7 +56,7 @@ When no arguments are provided (standalone invocation), discover plan from `docs
|
|
|
56
56
|
4. **Load session summary for cold start.** If resuming (session slug known):
|
|
57
57
|
- Call `listActiveSessions()` to read the session index.
|
|
58
58
|
- Call `loadSessionSummary()` for the target session.
|
|
59
|
-
- If ambiguous, present the index and ask which session to resume.
|
|
59
|
+
- If ambiguous, present the index and ask which session to resume. Ask in plain text — do not elevate this to `AskUserQuestion` (the session index can exceed its 4-option cap and a natural header like "Pick session" can exceed its 12-char cap, rendering the call as ERR).
|
|
60
60
|
|
|
61
61
|
5. **Check for known dead ends.** Review `learnings` tagged `[outcome:failure]`. Warn if any match current plan approaches.
|
|
62
62
|
|
|
@@ -13,6 +13,8 @@
|
|
|
13
13
|
|
|
14
14
|
## Process
|
|
15
15
|
|
|
16
|
+
**Prompt the human in plain text** — every choice and destructive confirmation in this skill (worktree location, discard-experiment, etc.) is plain text only. Do not elevate to `AskUserQuestion`: option labels like the A/B/C worktree-location choices and natural headers like "Discard commits" routinely exceed its 4-option / 12-char caps and render as ERR.
|
|
17
|
+
|
|
16
18
|
### Part A: Worktree Creation
|
|
17
19
|
|
|
18
20
|
#### Step 1: Choose Worktree Location
|
|
@@ -15,6 +15,8 @@
|
|
|
15
15
|
|
|
16
16
|
## Process
|
|
17
17
|
|
|
18
|
+
**Prompt the human in plain text** — every confirmation and selection in this skill (`Proceed? (y/n)`, post-critique selection, etc.) is plain text only. Do not elevate to `AskUserQuestion`: natural headers like "Confirm inputs" exceed its 12-char cap and option lists routinely exceed its 4-option cap, rendering the call as ERR.
|
|
19
|
+
|
|
18
20
|
### Iron Law
|
|
19
21
|
|
|
20
22
|
**The skill produces exactly ONE ranked artifact per run, at `docs/ideation/<slug>-YYYY-MM-DD.md`, and NEVER produces a spec, plan, ADR, or code.** If the user wants a spec from a ranked idea, they invoke `harness-brainstorming` next — that is the contract.
|
|
@@ -14,6 +14,8 @@
|
|
|
14
14
|
|
|
15
15
|
## Process
|
|
16
16
|
|
|
17
|
+
**Prompt the human in plain text** — every confirmation, drift acknowledgement, and contradiction resolution in this skill is plain text only. Do not elevate to `AskUserQuestion`: source labels and finding descriptions routinely exceed its 4-option / 12-char caps, rendering the call as ERR.
|
|
18
|
+
|
|
17
19
|
1. **EXTRACT:** Run code signal extractors, diagram parsers, BusinessKnowledgeIngestor, and KnowledgeLinker against the project
|
|
18
20
|
2. **RECONCILE:** Build pre-extraction snapshot (current graph state) and post-extraction snapshot, then run StructuralDriftDetector
|
|
19
21
|
3. **DETECT:** Partition findings by classification (new/stale/drifted/contradicting), generate gap report
|
|
@@ -9,6 +9,7 @@
|
|
|
9
9
|
- When adding a new feature to an existing roadmap (`--add <feature-name>`)
|
|
10
10
|
- When roadmap statuses may be stale and need updating from plan execution state (`--sync`)
|
|
11
11
|
- When features need reordering, moving between milestones, or blocker updates (`--edit`)
|
|
12
|
+
- When the roadmap needs tidying -- completed work archived, dead `planned` rows demoted (`--groom`)
|
|
12
13
|
- When user asks about project status and no roadmap exists -- suggest `--create`
|
|
13
14
|
- NOT for programmatic CRUD (use `manage_roadmap` MCP tool directly)
|
|
14
15
|
|
|
@@ -20,6 +21,8 @@
|
|
|
20
21
|
|
|
21
22
|
If the human has not seen and approved the milestone groupings and feature list, do not write the file. Present. Wait. Confirm. Then write.
|
|
22
23
|
|
|
24
|
+
**Prompt the human in plain text — every `(y/n)`, "which feature?", and similar prompt in this skill is plain text only.** Do not elevate them to `AskUserQuestion`: feature lists routinely exceed its 4-option cap, and natural header choices ("Pick feature", "Remove feature") exceed its 12-char cap, causing the call to render as ERR.
|
|
25
|
+
|
|
23
26
|
---
|
|
24
27
|
|
|
25
28
|
### Command: `--create` -- Bootstrap Roadmap
|
|
@@ -428,12 +431,67 @@ Choice?
|
|
|
428
431
|
|
|
429
432
|
---
|
|
430
433
|
|
|
434
|
+
### Command: `--groom` -- Tidy the Roadmap
|
|
435
|
+
|
|
436
|
+
Keeps the roadmap manageable over time. **Milestones are themes; statuses are lifecycle stages** -- grooming enforces that separation so the backlog never decays back into an undifferentiated dump.
|
|
437
|
+
|
|
438
|
+
#### Phase 1: SCAN -- Detect Untidiness
|
|
439
|
+
|
|
440
|
+
1. Check if `docs/roadmap.md` exists. If missing: error and direct the user to `--create`.
|
|
441
|
+
2. Run `manage_roadmap` (`action: "groom"`) in a dry-run frame, or call `checkRoadmapHealth` from `@harness-engineering/core`, to surface the four health signals:
|
|
442
|
+
- **RMH001** -- completed (`done`) features still sitting in an active milestone.
|
|
443
|
+
- **RMH002** -- `planned` rows with neither a spec nor a plan (the orchestrator cannot auto-execute these; it escalates them to a human).
|
|
444
|
+
- **RMH003** -- lifecycle catch-all milestones (`Backlog`, `Current Work`) that should not exist.
|
|
445
|
+
- **RMH004** -- active milestones that have grown past the size cap (a mini-dump).
|
|
446
|
+
|
|
447
|
+
#### Phase 2: PROPOSE -- Present the Plan
|
|
448
|
+
|
|
449
|
+
Show the human exactly what grooming will do, in plain text:
|
|
450
|
+
|
|
451
|
+
```
|
|
452
|
+
GROOM PLAN
|
|
453
|
+
|
|
454
|
+
Demote to backlog (planned with no spec/plan):
|
|
455
|
+
- Feature A (Theme X)
|
|
456
|
+
- Feature B (Theme Y)
|
|
457
|
+
|
|
458
|
+
Archive to docs/roadmap-archive.md (completed):
|
|
459
|
+
- Feature C (Theme X)
|
|
460
|
+
|
|
461
|
+
Flagged for manual routing (not auto-changed):
|
|
462
|
+
- Intake lane has 3 items awaiting a theme
|
|
463
|
+
- "Theme Z" has 28 features (cap 25) -- consider splitting
|
|
464
|
+
|
|
465
|
+
Apply? (y/n)
|
|
466
|
+
```
|
|
467
|
+
|
|
468
|
+
Wait for confirmation. The mechanical changes (demote, archive) are safe and automated; **draining the Intake lane into themed milestones and splitting oversized milestones are human decisions** -- propose, do not auto-apply.
|
|
469
|
+
|
|
470
|
+
#### Phase 3: WRITE -- Apply
|
|
471
|
+
|
|
472
|
+
1. Run `manage_roadmap` (`action: "groom"`). It demotes unactionable `planned` rows to `backlog` and moves `done` features into `docs/roadmap-archive.md` under a `Shipped` milestone, returning the list of changes.
|
|
473
|
+
2. For Intake-draining or milestone-splitting the human approved, follow up with `--edit` (move features between milestones).
|
|
474
|
+
|
|
475
|
+
#### Phase 4: VALIDATE -- Verify
|
|
476
|
+
|
|
477
|
+
1. Run `harness validate` and confirm the `roadmapHealth` check passes (no RMH003 errors; RMH001/002/004 warnings cleared or acknowledged).
|
|
478
|
+
2. Summarize:
|
|
479
|
+
|
|
480
|
+
```
|
|
481
|
+
Groom complete.
|
|
482
|
+
Demoted: N | Archived: N -> docs/roadmap-archive.md | Flagged for manual routing: N
|
|
483
|
+
harness validate (roadmapHealth): passed
|
|
484
|
+
```
|
|
485
|
+
|
|
486
|
+
---
|
|
487
|
+
|
|
431
488
|
## Harness Integration
|
|
432
489
|
|
|
433
|
-
- **`manage_roadmap` MCP tool** -- Primary read/write interface for roadmap operations. Supports `show`, `add`, `update`, `remove`, and `
|
|
434
|
-
- **`harness validate`** -- Run after any roadmap modification to verify project health. Mandatory in the VALIDATE phase of
|
|
435
|
-
- **Core `
|
|
436
|
-
- **
|
|
490
|
+
- **`manage_roadmap` MCP tool** -- Primary read/write interface for roadmap operations. Supports `show`, `add`, `update`, `remove`, `query`, `sync`, `promote`, and `groom` actions. Use this when MCP is available for structured CRUD.
|
|
491
|
+
- **`harness validate`** -- Run after any roadmap modification to verify project health. Mandatory in the VALIDATE phase of `--create`, `--add`, and `--groom`. The `roadmapHealth` check enforces the maintenance rules (RMH001-RMH004) as a regression guard.
|
|
492
|
+
- **Core `checkRoadmapHealth`/`groomRoadmap`** -- Maintenance engine in `packages/core/src/roadmap/health.ts`. `checkRoadmapHealth` is read-only diagnostics; `groomRoadmap` is the pure transform (demote unactionable planned, archive done). Both are surfaced via `manage_roadmap` and `harness validate`.
|
|
493
|
+
- **Core `parseRoadmap`/`serializeRoadmap`** -- Fallback when MCP is unavailable. These functions in `packages/core/src/roadmap/` handle parsing and serializing the roadmap markdown format directly. Note: the serializer only preserves frontmatter, milestones, features, and the Assignment History table -- never add convention prose or comments to `docs/roadmap.md`, they are dropped on the next write.
|
|
494
|
+
- **Roadmap files** -- Live work in `docs/roadmap.md` (the orchestrator's source of truth); completed work archived to `docs/roadmap-archive.md` by `--groom`. Milestones are themes, not lifecycle stages -- promoted items land in the `Intake` lane and are groomed into themes.
|
|
437
495
|
|
|
438
496
|
## Success Criteria
|
|
439
497
|
|
|
@@ -126,6 +126,8 @@ Proceed with Feature A? (y/n/pick another)
|
|
|
126
126
|
|
|
127
127
|
### Phase 3: CONFIRM -- Human Decision
|
|
128
128
|
|
|
129
|
+
Ask the human in plain text (matching the `y/n/pick another` example above). Do not elevate this confirmation to an `AskUserQuestion` tool call — candidate labels and natural header choices ("Pick candidate", etc.) exceed its 12-char `header` cap and the prompt is rejected as ERR.
|
|
130
|
+
|
|
129
131
|
1. Wait for human confirmation.
|
|
130
132
|
- If **yes**: proceed to Phase 4.
|
|
131
133
|
- If **pick another**: ask which candidate number, then proceed with that pick.
|
|
@@ -8,8 +8,9 @@
|
|
|
8
8
|
- In CI — optimize test suite execution order
|
|
9
9
|
- When a test fails — understand which changes could have caused it
|
|
10
10
|
- When `on_pr` triggers fire
|
|
11
|
-
-
|
|
12
|
-
- NOT for test
|
|
11
|
+
- **Coverage Audit mode** — when no diff is available and the user asks for "coverage gaps", "deep dive", "coverage plan", or "what's untested", run project-wide gap analysis instead of test selection
|
|
12
|
+
- NOT for writing tests (use harness-tdd, or `canary:canary-write-test` for uncovered files surfaced by Coverage Audit)
|
|
13
|
+
- NOT for test quality analysis at the test-selection level (Coverage Audit mode does include a capped quality review via `canary:canary-review-test`)
|
|
13
14
|
|
|
14
15
|
## Prerequisites
|
|
15
16
|
|
|
@@ -112,6 +113,83 @@ npx vitest run tests/services/auth.test.ts tests/types/user.test.ts tests/routes
|
|
|
112
113
|
npx vitest run tests/services/auth.test.ts tests/types/user.test.ts tests/routes/login.test.ts tests/middleware/verify.test.ts tests/integration/auth-flow.test.ts
|
|
113
114
|
```
|
|
114
115
|
|
|
116
|
+
## Coverage Audit Mode
|
|
117
|
+
|
|
118
|
+
Activates when `--audit` is passed, OR when no diff is available AND the user's
|
|
119
|
+
language matches audit intent ("coverage gaps", "deep dive", "coverage plan",
|
|
120
|
+
"what's untested"). When this mode is selected, skip Phases 1–3 above and run
|
|
121
|
+
the audit phases below instead.
|
|
122
|
+
|
|
123
|
+
### Audit Phase 0: PROBE — Detect canary CLI availability
|
|
124
|
+
|
|
125
|
+
Call the `canary_probe` MCP tool once at the start of the audit. It returns
|
|
126
|
+
`{ status: "available" | "degraded", version?, reason? }` and never errors.
|
|
127
|
+
|
|
128
|
+
- **`available`** — the deterministic canary CLI is usable; use `canary_recommend_framework`
|
|
129
|
+
for framework selection in Phase 3 (GAP REPORT).
|
|
130
|
+
- **`degraded`** — the CLI is not usable (reason: `not-installed`, `binary-missing`,
|
|
131
|
+
`exec-failed`, or `bad-output`). Print one line:
|
|
132
|
+
|
|
133
|
+
> canary CLI unavailable (`<reason>`) — install `canary-test-cli` for deterministic
|
|
134
|
+
> framework recommendations. Proceeding; framework picks fall back to the plugin.
|
|
135
|
+
|
|
136
|
+
Then skip `canary_recommend_framework` calls for the rest of the run. This does **not**
|
|
137
|
+
affect the plugin-based Quality Review in Phase 2 (`canary:canary-review-test` is a
|
|
138
|
+
separate Claude Code plugin, installed independently of the CLI).
|
|
139
|
+
|
|
140
|
+
### Audit Phase 1: INVENTORY — Build the Source-to-Test Map
|
|
141
|
+
|
|
142
|
+
1. **Enumerate source files**: glob for `.ts`, `.tsx`, `.js`, `.jsx` under the
|
|
143
|
+
project's source roots (e.g., `src/`, `packages/*/src/`). Skip `node_modules`,
|
|
144
|
+
`dist`, build outputs, and fixtures.
|
|
145
|
+
2. **Enumerate test files**: glob for `*.test.*`, `*.spec.*`, and files under
|
|
146
|
+
`__tests__/` and parallel `tests/` directories.
|
|
147
|
+
3. **Map source → test**: for each source file, find its test file using the
|
|
148
|
+
Tier 1 / Tier 2 / Tier 3 strategies described in `Phase 2: DISCOVER`. With a
|
|
149
|
+
graph, prefer `get_impact`; without one, use naming conventions and import
|
|
150
|
+
parsing.
|
|
151
|
+
4. **Split**: produce two lists — `covered` (source files with at least one
|
|
152
|
+
matching test) and `uncovered` (source files with no matching test).
|
|
153
|
+
|
|
154
|
+
### Audit Phase 2: QUALITY REVIEW — Critique Covered Test Files
|
|
155
|
+
|
|
156
|
+
1. **Cap at 10 files per run**: a deep quality review is expensive. Pick the
|
|
157
|
+
top 10 covered files prioritized by size (lines of code) and criticality
|
|
158
|
+
(depth in the graph / co-change frequency). The remaining covered files are
|
|
159
|
+
accepted as-is for this run.
|
|
160
|
+
2. **For each of the 10**: dispatch `canary:canary-review-test` against the
|
|
161
|
+
test file. Collect its findings (missing edge cases, anti-patterns,
|
|
162
|
+
brittleness, oracle gaps).
|
|
163
|
+
3. **Aggregate**: bin findings by severity (high / medium / low) and by file.
|
|
164
|
+
|
|
165
|
+
### Audit Phase 3: GAP REPORT — Synthesize a Unified Coverage Plan
|
|
166
|
+
|
|
167
|
+
Emit a single report with three sections:
|
|
168
|
+
|
|
169
|
+
```
|
|
170
|
+
## Coverage Audit Report
|
|
171
|
+
|
|
172
|
+
### Uncovered Files (no test found)
|
|
173
|
+
| File | Lines | Priority | Suggested Action |
|
|
174
|
+
|------|-------|----------|------------------|
|
|
175
|
+
| src/services/billing.ts | 412 | high | canary:canary-write-test |
|
|
176
|
+
| src/utils/format.ts | 38 | low | canary:canary-write-test |
|
|
177
|
+
|
|
178
|
+
### Quality Gaps (from Quality Review, capped at 10 reviewed)
|
|
179
|
+
| Test File | Severity | Top Gap |
|
|
180
|
+
|-----------|----------|---------|
|
|
181
|
+
| tests/services/auth.test.ts | high | no failure-path assertions |
|
|
182
|
+
| tests/utils/date.test.ts | med | mock leaks across cases |
|
|
183
|
+
|
|
184
|
+
### Recommended Next Steps
|
|
185
|
+
- Generate tests for high-priority uncovered files via `canary:canary-write-test`.
|
|
186
|
+
- Resolve high-severity quality gaps via `canary:canary-review-test` follow-ups.
|
|
187
|
+
- For uncovered files, pick a framework first: when Phase 0 reported `available`, call
|
|
188
|
+
the `canary_recommend_framework` MCP tool with a prompt describing the file's purpose
|
|
189
|
+
(deterministic, no LLM round-trip) and put its `framework` in the Suggested Action.
|
|
190
|
+
When `degraded`, fall back to the `canary:canary-pick-framework` plugin.
|
|
191
|
+
```
|
|
192
|
+
|
|
115
193
|
## Harness Integration
|
|
116
194
|
|
|
117
195
|
- **`harness scan`** — Recommended before this skill for full graph-enhanced analysis. If graph is missing, skill uses naming convention and import parsing fallbacks.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
name: harness-test-advisor
|
|
2
2
|
version: "1.0.0"
|
|
3
|
-
description: Graph-based test selection — answers "what tests should I run?"
|
|
3
|
+
description: Graph-based test selection and project-wide coverage audit — answers "what tests should I run?" or "what's untested?"
|
|
4
4
|
stability: static
|
|
5
5
|
cognitive_mode: advisory-guide
|
|
6
6
|
triggers:
|
|
@@ -25,6 +25,9 @@ cli:
|
|
|
25
25
|
- name: files
|
|
26
26
|
description: Comma-separated list of changed files
|
|
27
27
|
required: false
|
|
28
|
+
- name: audit
|
|
29
|
+
description: Run Coverage Audit mode (project-wide gap analysis) instead of test selection
|
|
30
|
+
required: false
|
|
28
31
|
mcp:
|
|
29
32
|
tool: run_skill
|
|
30
33
|
input:
|
|
@@ -29,6 +29,8 @@ Detect plugin-only state by checking whether `harness` resolves on PATH (`comman
|
|
|
29
29
|
|
|
30
30
|
## Process
|
|
31
31
|
|
|
32
|
+
**Prompt the human in plain text** — every framework confirmation, migration check, and telemetry-identity question in this skill is plain text only. Do not elevate to `AskUserQuestion`: the framework list (~10 options) exceeds its 4-option cap and natural headers like "Confirm framework" exceed its 12-char cap, rendering the call as ERR.
|
|
33
|
+
|
|
32
34
|
### Phase 1: ASSESS — Determine Current State
|
|
33
35
|
|
|
34
36
|
1. **Check for existing harness configuration.** Look for `.harness/` directory, `AGENTS.md`, `harness.config.json`, and any skill definitions. Their presence determines whether this is a new project or a migration.
|
|
@@ -96,7 +98,7 @@ Detect plugin-only state by checking whether `harness` resolves on PATH (`comman
|
|
|
96
98
|
5. **Configure i18n (all levels).** Ask: "Will this project support multiple languages?" Based on the answer:
|
|
97
99
|
- **Yes:** Invoke `harness-i18n-workflow` configure phase to set up i18n config in `harness.config.json` (source locale, target locales, framework, strictness). Then invoke `harness-i18n-workflow` scaffold phase to create translation file structure and extraction config. Set `i18n.enabled: true`.
|
|
98
100
|
- **No:** Set `i18n.enabled: false` in `harness.config.json`. The `harness-i18n-process` skill will still fire gentle prompts for unconfigured projects when features touch user-facing strings.
|
|
99
|
-
- **Not sure
|
|
101
|
+
- **Not sure:** Skip i18n configuration entirely. Do not set `i18n.enabled`. The project can enable i18n later by running `harness-i18n-workflow` directly.
|
|
100
102
|
|
|
101
103
|
5b. **Configure design system (non-test-suite projects).** Ask: "Will this project have a UI requiring a design system?" Mirror the i18n step's three-way response shape. Use `emit_interaction`:
|
|
102
104
|
|
|
@@ -136,7 +138,7 @@ Detect plugin-only state by checking whether `harness` resolves on PATH (`comman
|
|
|
136
138
|
Based on the answer:
|
|
137
139
|
- **Yes:** Ask a follow-up: "Which platforms? `web`, `mobile`, or both?" Write `design.enabled: true` and `design.platforms: [...]` (a non-empty array of `web` and/or `mobile`) to `harness.config.json`. Inform the user: "Design tokens will be generated when you start your first design-touching feature — `harness-design-system` fires automatically via `on_new_feature`."
|
|
138
140
|
- **No:** Write `design.enabled: false` to `harness.config.json`. Do not write `design.platforms`. The `on_new_feature` trigger respects this flag and will not fire `harness-design-system`.
|
|
139
|
-
- **Not sure
|
|
141
|
+
- **Not sure:** Do not write `design.enabled` or `design.platforms`. The project can enable design later by running `harness-design-system` directly; `on_new_feature` will prompt gently when a feature touches user-facing UI.
|
|
140
142
|
|
|
141
143
|
**Skip this step entirely if Phase 1 step 5 classified the project as a test suite.** Test-suite projects will be dispatched at step 6 below to `initialize-test-suite-project` and have no UI to govern.
|
|
142
144
|
|
|
@@ -183,7 +185,7 @@ Detect plugin-only state by checking whether `harness` resolves on PATH (`comman
|
|
|
183
185
|
- **Absent (most common on init).** Present the prompt above. Apply the answer:
|
|
184
186
|
- **Yes:** delegate to `harness-strategy` (which routes via its own Phase 0 to the first-run interview). When `harness-strategy` completes, continue with step 6.
|
|
185
187
|
- **No:** write `init.strategy.declined: true` to `.harness/state.json` (merge into existing JSON; do not clobber). This is the explicit decline location; re-running init detects the flag and skips the prompt.
|
|
186
|
-
- **Not sure
|
|
188
|
+
- **Not sure:** no state write. `/harness:strategy` remains available standalone, and a future re-run of init will re-offer.
|
|
187
189
|
|
|
188
190
|
- **Present and valid.** Skip the prompt silently. Surface a one-line note: `STRATEGY.md detected — downstream skills will pick it up as grounding`. No `init.strategy.declined` write.
|
|
189
191
|
|
|
@@ -303,7 +305,7 @@ This phase closes the parity gap that the marketplace plugin install does not co
|
|
|
303
305
|
|
|
304
306
|
Based on the answer:
|
|
305
307
|
- **Yes:** Invoke `harness-roadmap` (skill) or run `/harness:roadmap --create` to create `docs/roadmap.md`. Verify the file exists. The `manage_roadmap` MCP tool is for managing entries in an existing roadmap, not for creating one.
|
|
306
|
-
- **If `design.enabled === true` in `harness.config.json`** (set by Phase 3 step 5b), call `manage_roadmap` with `action: add`, `feature: "Set up design system"`, `status: "planned"`, `milestone: "
|
|
308
|
+
- **If `design.enabled === true` in `harness.config.json`** (set by Phase 3 step 5b), call `manage_roadmap` with `action: add`, `feature: "Set up design system"`, `status: "planned"`, `milestone: "Intake"`, `summary: "Run harness-design-system to define palette, typography, and generate W3C DTCG tokens. Deferred from project init — fires on first design-touching feature via on_new_feature."`. Skip silently if `manage_roadmap show` reports a duplicate `(feature, milestone)` pair. This closes the loop between deferred design intent and visible planning work.
|
|
307
309
|
- **No:** Skip silently. The user can still run `/harness:roadmap --create` later — that informational fallback remains valid.
|
|
308
310
|
|
|
309
311
|
2. **Commit the initialization.** All generated, configured, and instrumentation files in a single commit. Include `harness.config.json`, `AGENTS.md`, `.harness/arch/baselines.json`, `.harness/telemetry.json` (if created), updates to project `.mcp.json` (if Tier-0 integrations were wired), and any roadmap files.
|
|
@@ -323,11 +325,11 @@ This phase closes the parity gap that the marketplace plugin install does not co
|
|
|
323
325
|
- **`harness integrations list` / `harness integrations add <name>`** — Phase 5 step 6. Lists Tier-0 (zero-config) and Tier-1 (API-key) MCP integrations and adds them to project `.mcp.json`.
|
|
324
326
|
- **`harness-i18n-workflow configure` + `harness-i18n-workflow scaffold`** — Invoked during Phase 3 if the project will support multiple languages. Sets up i18n configuration and translation file structure.
|
|
325
327
|
- **`harness-design-system` (deferred via `on_new_feature`)** — Phase 3 step 5b records `design.enabled` + `design.platforms` in `harness.config.json` but does NOT run the full design-system skill. Token generation defers to the first design-touching feature, where `harness-design-system` fires via `on_new_feature` and reads `design.enabled` to decide whether to proceed.
|
|
326
|
-
- **`harness-strategy`** — Phase 3 step 5c delegates to this skill on "Yes". The skill conducts a first-run interview (pushback rules with a 2-round cap per section) and writes a valid `STRATEGY.md` at repo root via `writeStrategyDoc`. On "No" init writes `init.strategy.declined: true` to `.harness/state.json`. On "
|
|
328
|
+
- **`harness-strategy`** — Phase 3 step 5c delegates to this skill on "Yes". The skill conducts a first-run interview (pushback rules with a 2-round cap per section) and writes a valid `STRATEGY.md` at repo root via `writeStrategyDoc`. On "No" init writes `init.strategy.declined: true` to `.harness/state.json`. On "not sure" no state is written; the user can run `/harness:strategy` standalone. When `STRATEGY.md` already exists and is valid the prompt is skipped; when present-but-invalid the user gets the three-path repair offer.
|
|
327
329
|
- **`validateStrategy`** — `@harness-engineering/core` helper used by Phase 3 step 5c to detect present-but-invalid `STRATEGY.md`. Invoked through a Node one-liner so init does not require importing core directly.
|
|
328
330
|
- **`initialize-test-suite-project`** — Sub-skill. Invoked during Phase 3 step 6 when Phase 1 step 5 classified the project as a test suite. Owns archetype selection, shared-library vs in-repo decision, layer variants, tag taxonomy, reporter stack, custom report, and "prove the guards fire" verification.
|
|
329
331
|
- **`harness-roadmap` skill** — Phase 6 step 1 invokes this skill (or `/harness:roadmap --create`) when the user opts in to creating `docs/roadmap.md`. The `manage_roadmap` MCP tool does not create roadmaps; it manages entries in an existing one.
|
|
330
|
-
- **`manage_roadmap` MCP tool** — Phase 6 step 1, when `design.enabled === true`, calls `manage_roadmap` with `action: add` to insert a `planned` "Set up design system" item under milestone `
|
|
332
|
+
- **`manage_roadmap` MCP tool** — Phase 6 step 1, when `design.enabled === true`, calls `manage_roadmap` with `action: add` to insert a `planned` "Set up design system" item under milestone `Intake` with a summary describing the deferred work.
|
|
331
333
|
|
|
332
334
|
## Success Criteria
|
|
333
335
|
|
|
@@ -340,29 +342,29 @@ This phase closes the parity gap that the marketplace plugin install does not co
|
|
|
340
342
|
- The adoption level matches what was agreed upon with the human
|
|
341
343
|
- All generated files are committed in a single atomic commit
|
|
342
344
|
- i18n configuration is set if the human chose to enable it during init
|
|
343
|
-
- For non-test-suite projects, the design-system question was asked and `harness.config.json` reflects the answer: `design.enabled: true` (with `design.platforms` populated) for yes, `design.enabled: false` for no, or absent for not
|
|
344
|
-
- The strategy question (Phase 3 step 5c) was asked unless `STRATEGY.md` already existed and was valid (in which case the prompt was skipped silently with a one-line detection note). The answer was recorded according to the documented semantics: Yes → `STRATEGY.md` exists and `harness validate` passes against `StrategyDocSchema`; No → `.harness/state.json` contains `init.strategy.declined: true`; Not sure
|
|
345
|
+
- For non-test-suite projects, the design-system question was asked and `harness.config.json` reflects the answer: `design.enabled: true` (with `design.platforms` populated) for yes, `design.enabled: false` for no, or absent for not sure.
|
|
346
|
+
- The strategy question (Phase 3 step 5c) was asked unless `STRATEGY.md` already existed and was valid (in which case the prompt was skipped silently with a one-line detection note). The answer was recorded according to the documented semantics: Yes → `STRATEGY.md` exists and `harness validate` passes against `StrategyDocSchema`; No → `.harness/state.json` contains `init.strategy.declined: true`; Not sure → no `STRATEGY.md` and no `init.strategy.declined` flag.
|
|
345
347
|
- **Phase 5 (INSTRUMENT) outputs:** `.harness/graph/` is populated; `.harness/arch/baselines.json` exists; `harness check-perf` ran without errors (intermediate and above); `.harness/telemetry.json` exists if the human opted into identity tagging; legacy layout warnings were surfaced via `harness migrate --dry-run` and either resolved or explicitly deferred; Tier-0 MCP integrations (context7, sequential-thinking, playwright) are present in the project's `.mcp.json` (or the human declined and that decision is recorded).
|
|
346
348
|
- The roadmap question was asked. If the user answered yes, `docs/roadmap.md` exists and was created via `harness-roadmap` (or the documented `/harness:roadmap --create` fallback).
|
|
347
|
-
- When `design.enabled === true` AND the user answered yes to the roadmap question, `docs/roadmap.md` contains a `planned` entry titled "Set up design system" under milestone `
|
|
349
|
+
- When `design.enabled === true` AND the user answered yes to the roadmap question, `docs/roadmap.md` contains a `planned` entry titled "Set up design system" under milestone `Intake` with a summary describing the deferred work. The entry is absent in all other answer combinations.
|
|
348
350
|
- For test suites: `initialize-test-suite-project` ran to completion and its Success Criteria are also met
|
|
349
351
|
- For plugin-only invocations: every CLI invocation in this skill ran successfully via `npx @harness-engineering/cli <cmd>` without requiring a global install. If any invocation failed because the npx download failed, the human was prompted to retry or `npm install -g @harness-engineering/cli` instead.
|
|
350
352
|
|
|
351
353
|
## Rationalizations to Reject
|
|
352
354
|
|
|
353
|
-
| Rationalization | Why It Is Wrong
|
|
354
|
-
| ---------------------------------------------------------------------------- |
|
|
355
|
-
| "The generated AGENTS.md template looks fine -- no need to customize it" | Phase 3 says do not blindly accept generated content. Without project-specific descriptions, agents receive generic instructions.
|
|
356
|
-
| "We should start at the advanced level since we want full coverage" | The skill recommends basic for new projects. Each level builds on the previous. Jumping to advanced creates misconfigured rules.
|
|
357
|
-
| "I will skip the i18n question to keep setup fast" | Phase 3 requires asking about i18n and recording the decision. Skipping creates ambiguity about whether the omission was intentional.
|
|
358
|
-
| "I will skip the design-system question to keep setup fast" | Phase 3 step 5b requires asking about design and recording the answer in `design.enabled`. Skipping creates ambiguity about whether the omission was intentional and bypasses the linkage between init and the deferred `harness-design-system` invocation on `on_new_feature`.
|
|
359
|
-
| "I will skip the strategy question — it's just paperwork" | Phase 3 step 5c is the only point in the workflow where init asks the user to capture the strategic anchor. Skipping bypasses the grounding signal that brainstorming, ideate, and roadmap-pilot read; downstream skill output degrades silently. Even a "
|
|
360
|
-
| "STRATEGY.md exists already, so I should re-run the interview to refresh it" | When `STRATEGY.md` is present and valid, Phase 3 step 5c skips the prompt silently. Refreshing strategy mid-init is out of scope — that is what the standalone `/harness:strategy` update flow is for. Surface a one-line detection note and continue.
|
|
361
|
-
| "STRATEGY.md is present but invalid, so I should block init" | Phase 3 step 5c explicitly does NOT block. It surfaces the validation error, offers three repair paths (fix now / move-to-bak / ignore), and continues based on the user's choice. Init is the wrong place to gate on strategy correctness.
|
|
362
|
-
| "Validation passed, so the project is ready" | Phase 5 captures baselines, configures telemetry identity, surfaces legacy warnings, and wires Tier-0 integrations. Validation alone is not sufficient.
|
|
363
|
-
| "Plugin install means setup is done" | The marketplace plugin ships skills, slash commands, subagents, hooks, and MCP — but it cannot mutate the user's project state. `harness.config.json`, baselines, telemetry identity, and Tier-0 integrations require running this skill once per project.
|
|
364
|
-
| "Skip Phase 5 if we already ran `harness setup`" | Phase 5 is idempotent. It safely no-ops where setup already wired things and fills gaps where it didn't (common case: setup ran once, then a new Tier-0 integration was added or a layout was migrated). Re-running is the right behavior.
|
|
365
|
-
| "This is a test suite, we'll configure layers in this skill" | Phase 3 step 6 dispatches to `initialize-test-suite-project` for archetype selection, layer variants, and the rest. Do not inline test-suite-specific configuration here — the sub-skill owns it and carries the gotchas.
|
|
355
|
+
| Rationalization | Why It Is Wrong |
|
|
356
|
+
| ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
357
|
+
| "The generated AGENTS.md template looks fine -- no need to customize it" | Phase 3 says do not blindly accept generated content. Without project-specific descriptions, agents receive generic instructions. |
|
|
358
|
+
| "We should start at the advanced level since we want full coverage" | The skill recommends basic for new projects. Each level builds on the previous. Jumping to advanced creates misconfigured rules. |
|
|
359
|
+
| "I will skip the i18n question to keep setup fast" | Phase 3 requires asking about i18n and recording the decision. Skipping creates ambiguity about whether the omission was intentional. |
|
|
360
|
+
| "I will skip the design-system question to keep setup fast" | Phase 3 step 5b requires asking about design and recording the answer in `design.enabled`. Skipping creates ambiguity about whether the omission was intentional and bypasses the linkage between init and the deferred `harness-design-system` invocation on `on_new_feature`. |
|
|
361
|
+
| "I will skip the strategy question — it's just paperwork" | Phase 3 step 5c is the only point in the workflow where init asks the user to capture the strategic anchor. Skipping bypasses the grounding signal that brainstorming, ideate, and roadmap-pilot read; downstream skill output degrades silently. Even a "no" or "not sure" answer is better than no answer because it locks in the decision (or absence of one). |
|
|
362
|
+
| "STRATEGY.md exists already, so I should re-run the interview to refresh it" | When `STRATEGY.md` is present and valid, Phase 3 step 5c skips the prompt silently. Refreshing strategy mid-init is out of scope — that is what the standalone `/harness:strategy` update flow is for. Surface a one-line detection note and continue. |
|
|
363
|
+
| "STRATEGY.md is present but invalid, so I should block init" | Phase 3 step 5c explicitly does NOT block. It surfaces the validation error, offers three repair paths (fix now / move-to-bak / ignore), and continues based on the user's choice. Init is the wrong place to gate on strategy correctness. |
|
|
364
|
+
| "Validation passed, so the project is ready" | Phase 5 captures baselines, configures telemetry identity, surfaces legacy warnings, and wires Tier-0 integrations. Validation alone is not sufficient. |
|
|
365
|
+
| "Plugin install means setup is done" | The marketplace plugin ships skills, slash commands, subagents, hooks, and MCP — but it cannot mutate the user's project state. `harness.config.json`, baselines, telemetry identity, and Tier-0 integrations require running this skill once per project. |
|
|
366
|
+
| "Skip Phase 5 if we already ran `harness setup`" | Phase 5 is idempotent. It safely no-ops where setup already wired things and fills gaps where it didn't (common case: setup ran once, then a new Tier-0 integration was added or a layout was migrated). Re-running is the right behavior. |
|
|
367
|
+
| "This is a test suite, we'll configure layers in this skill" | Phase 3 step 6 dispatches to `initialize-test-suite-project` for archetype selection, layer variants, and the rest. Do not inline test-suite-specific configuration here — the sub-skill owns it and carries the gotchas. |
|
|
366
368
|
|
|
367
369
|
## Examples
|
|
368
370
|
|
|
@@ -458,7 +460,7 @@ Step 4 (roadmap): "Set up a project roadmap now?"
|
|
|
458
460
|
design.enabled === true detected → manage_roadmap action: add
|
|
459
461
|
feature: "Set up design system"
|
|
460
462
|
status: planned
|
|
461
|
-
milestone:
|
|
463
|
+
milestone: Intake
|
|
462
464
|
summary: "Run harness-design-system to define palette, typography, and generate W3C DTCG tokens..."
|
|
463
465
|
Result: docs/roadmap.md contains the planned design item.
|
|
464
466
|
|
|
@@ -470,7 +472,7 @@ git add harness.config.json .harness/ AGENTS.md docs/roadmap.md
|
|
|
470
472
|
git commit -m "feat: initialize harness project with design and roadmap"
|
|
471
473
|
```
|
|
472
474
|
|
|
473
|
-
**Final state:** `harness.config.json` has `design.enabled: true` + `design.platforms: ["web"]`; `docs/roadmap.md` lists "Set up design system" as a `planned` item under `
|
|
475
|
+
**Final state:** `harness.config.json` has `design.enabled: true` + `design.platforms: ["web"]`; `docs/roadmap.md` lists "Set up design system" as a `planned` item under `Intake`; on the first feature touching UI, `on_new_feature` fires `harness-design-system` which reads `design.enabled` and runs the full discover/define/generate/validate flow.
|
|
474
476
|
|
|
475
477
|
### Example: Migrating Existing Project from Basic to Intermediate
|
|
476
478
|
|
|
@@ -547,7 +549,7 @@ npx @harness-engineering/cli init # auto-detects framework
|
|
|
547
549
|
Customize AGENTS.md with project-specific content.
|
|
548
550
|
Phase 3 step 5 (i18n): "No, English only." → i18n.enabled = false.
|
|
549
551
|
Phase 3 step 5b (design): "No, this is a backend service." → design.enabled = false.
|
|
550
|
-
Phase 3 step 5c (strategy): "
|
|
552
|
+
Phase 3 step 5c (strategy): "not sure." → no STRATEGY.md, no init.strategy.declined flag; user can run /harness:strategy later.
|
|
551
553
|
Phase 3 step 6 (test-suite dispatch): not a test suite, skipped.
|
|
552
554
|
```
|
|
553
555
|
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
# Outcome Eval
|
|
2
|
+
|
|
3
|
+
> Post-execution LLM-judgment: did the implementation actually satisfy its spec? Reads the spec's acceptance section, the change diff, and test output, and emits a confidence-rated `OutcomeVerdict` (`SATISFIED | NOT_SATISFIED | INCONCLUSIVE`) with a rationale and unmet criteria. Ship authority is derived in TypeScript, never trusted from the LLM: a high-confidence `NOT_SATISFIED` blocks ship; every other verdict is advisory. The harness's first blocking post-execution spec-satisfaction gate (the roadmap's named #1 gap). Each verdict persists as an `execution_outcome` node, compounding into skill-effectiveness baselines.
|
|
4
|
+
|
|
5
|
+
## When to Use
|
|
6
|
+
|
|
7
|
+
- At orchestrator step 6.5 — after Code Review, before Ship — on every change with a spec.
|
|
8
|
+
- When you need a durable, structured answer to "did this code do what the spec said?"
|
|
9
|
+
- NOT for pre-execution risk simulation (use PESL).
|
|
10
|
+
- NOT for rule-based floors (lint/architecture/entropy) or craft ceilings (naming/spec/security) — those run elsewhere.
|
|
11
|
+
- NOT for auto-remediation. outcome-eval judges; it does not fix.
|
|
12
|
+
- NOT when no judgable spec section exists — the verdict degrades to INCONCLUSIVE/advisory and never blocks.
|
|
13
|
+
|
|
14
|
+
## Process
|
|
15
|
+
|
|
16
|
+
### Phase 1: GATHER — Collect inputs
|
|
17
|
+
|
|
18
|
+
1. Capture the change under judgment as a unified diff: `git diff` (or `git diff <base>...HEAD` for a branch). Record it as `diff`.
|
|
19
|
+
2. Capture test-runner output. If a test command is known, run it and capture stdout+stderr as `testOutput`; otherwise pass the most recent captured output. Empty/unparseable test output is tolerated (degrades to advisory).
|
|
20
|
+
3. Resolve the spec path. Prefer the spec under `docs/changes/<feature>/proposal.md` for the current change. Record as `specPath`.
|
|
21
|
+
|
|
22
|
+
### Phase 2: RESOLVE — Find the judgment section
|
|
23
|
+
|
|
24
|
+
The evaluator resolves the section internally via the fallback chain `## Success Criteria` -> `## User-Visible Behavior` -> `## Overview`, recording the match in `judgedAgainst`. No manual action — pass `specPath` and let `OutcomeEvaluator` resolve. If no section is judgable, the verdict is INCONCLUSIVE/advisory.
|
|
25
|
+
|
|
26
|
+
### Phase 3: JUDGE — Invoke the evaluator
|
|
27
|
+
|
|
28
|
+
1. Invoke the MCP tool `mcp__harness__outcome_eval` with `{ specPath, diff, testOutput }` (optional `model`). The tool constructs `OutcomeEvaluator` cli-side and calls `evaluate({ specPath, diff, testOutput })`; the supported v1 provider is the anthropic analysis provider (`ANTHROPIC_API_KEY`).
|
|
29
|
+
2. **`diff` and `testOutput` are required inputs and the agent MUST supply them** from the session (`git diff` + captured test-runner output). They are the evidence the judge reasons over — passing an empty `diff` or empty `testOutput` is the degradation path, not the normal path: the verdict degrades to INCONCLUSIVE/advisory (never blocking), which defeats the gate. Do not invoke the tool without real diff/test content.
|
|
30
|
+
3. The LLM returns ONLY `verdict / confidence / rationale / unmetCriteria`. `authority` is computed in TypeScript from `(verdict, confidence)` and is never read from the LLM — do not attempt to override it. The tool returns the verdict exactly as the evaluator derives it.
|
|
31
|
+
4. The call is degrade-safe: provider failure (incl. no `ANTHROPIC_API_KEY`), empty diff, empty test output, or missing judgable section yields INCONCLUSIVE/low/advisory. It never throws and never blocks.
|
|
32
|
+
|
|
33
|
+
### Phase 4: GATE — Render and (conditionally) halt
|
|
34
|
+
|
|
35
|
+
1. Render the verdict: `verdict`, `confidence`, `judgedAgainst`, `rationale`, and `unmetCriteria`.
|
|
36
|
+
2. Authority rule (must match `deriveAuthority`): authority is `blocking` **iff** `verdict === 'NOT_SATISFIED' && confidence === 'high'`; every other combination — including all `INCONCLUSIVE` and `SATISFIED` cases, and all `medium`/`low` `NOT_SATISFIED` — is `advisory`.
|
|
37
|
+
3. **On a blocking verdict: HALT before the Ship step.** Report the unmet criteria and stop; do not proceed to step 7. Resolution requires fixing the implementation (or the spec) and re-running outcome-eval.
|
|
38
|
+
4. On an advisory verdict: report it and proceed. Advisory `NOT_SATISFIED` is surfaced for human attention but does not stop the workflow.
|
|
39
|
+
|
|
40
|
+
## Harness Integration
|
|
41
|
+
|
|
42
|
+
- **`mcp__harness__outcome_eval`** — MCP tool (the invocation surface). Inputs: `specPath` (required), `diff` (required), `testOutput` (required), `model` (optional), `path` (optional project root for graph persistence). The agent supplies `diff` and `testOutput` from the session; omitting them degrades the verdict to INCONCLUSIVE/advisory (never blocking). The handler builds the cli `AnalysisProvider` + a `GraphStore`, constructs `OutcomeEvaluator`, and returns the `OutcomeVerdict` with authority exactly as derived in TypeScript.
|
|
43
|
+
- **Evaluator surface:** `OutcomeEvaluator`, `deriveAuthority`, `verdictSchema`, `OutcomeVerdict` are exported from `@harness-engineering/intelligence`.
|
|
44
|
+
- **Provider path (v1 supported):** the anthropic analysis provider (`ANTHROPIC_API_KEY`). When no provider is configured the call degrades to INCONCLUSIVE/advisory. The openai-compatible _strict_ structured-output path is a known follow-up (see Known Limitations).
|
|
45
|
+
- **Orchestrator:** runs as step 6.5 between Code Review and Ship in `harness.orchestrator.md`.
|
|
46
|
+
- **Persistence:** each `evaluate()` writes one `execution_outcome` node via `ExecutionOutcomeConnector`, consumable by `effectiveness/scorer.ts`.
|
|
47
|
+
|
|
48
|
+
## Known Limitations
|
|
49
|
+
|
|
50
|
+
- **INCONCLUSIVE persistence:** the persisted node maps `INCONCLUSIVE -> result: 'failure'` for type-validity, but it OMITS `agentPersona` and writes `affectedSystemNodeIds: []`. The effectiveness scorer (`gatherOutcomes`) ignores any node missing `agentPersona` or `outcome_of` edges, so outcome-eval nodes are **scorer-non-counting** in v1 — the INCONCLUSIVE-as-failure mapping is therefore harmless and does not punish any persona. If a future change attaches persona/affected-system attribution, it MUST first change INCONCLUSIVE modeling (do not persist INCONCLUSIVE, or use a distinct result value the scorer excludes) before the node becomes scorer-counted.
|
|
51
|
+
- **openai-compatible strict mode:** `zodToJsonSchema` does not emit `additionalProperties: false`, which OpenAI strict structured output requires. The v1 supported path is claude-cli / anthropic. Follow-up tracked.
|
|
52
|
+
- **CI required-check wiring:** deferred to roadmap #540 (unbuilt CI workflow template).
|
|
53
|
+
|
|
54
|
+
## Success Criteria
|
|
55
|
+
|
|
56
|
+
See `docs/changes/outcome-eval/proposal.md` for the full 9 criteria. This skill satisfies SC8 (orchestrator step 6.5 + blocking halt) and SC9 (introduces no new `harness validate` findings; layer rules respected).
|
|
57
|
+
|
|
58
|
+
## Examples
|
|
59
|
+
|
|
60
|
+
### Example: NOT_SATISFIED with high confidence (blocks)
|
|
61
|
+
|
|
62
|
+
**Input:** spec Success Criteria require `GET /api/users/:id` to return 404 with `{ error: 'User not found' }`; the diff implements the happy path only, no 404 branch; test output shows the 404 test failing.
|
|
63
|
+
|
|
64
|
+
**Verdict:**
|
|
65
|
+
|
|
66
|
+
```
|
|
67
|
+
verdict: NOT_SATISFIED
|
|
68
|
+
confidence: high
|
|
69
|
+
judgedAgainst: success-criteria
|
|
70
|
+
authority: blocking
|
|
71
|
+
unmetCriteria:
|
|
72
|
+
- "404 path for nonexistent user is unimplemented; the failing test asserts { error: 'User not found' }."
|
|
73
|
+
rationale: "The diff adds the lookup but returns 200 with an empty body when the user is missing."
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
**Action:** HALT before Ship. Report unmet criteria; do not open the PR.
|
|
77
|
+
|
|
78
|
+
### Example: partial implementation (advisory)
|
|
79
|
+
|
|
80
|
+
**Input:** the diff meets most criteria; one acceptance item is ambiguous in the diff.
|
|
81
|
+
|
|
82
|
+
**Verdict:** `NOT_SATISFIED confidence: medium authority: advisory` — surfaced for review, workflow proceeds.
|
|
83
|
+
|
|
84
|
+
## Gates
|
|
85
|
+
|
|
86
|
+
- **Authority is never read from the LLM.** The verdict's `authority` is always `deriveAuthority(verdict, confidence)` computed in TypeScript. If you find yourself letting the model assert blocking/advisory, STOP — that defeats the entire purpose of this gate.
|
|
87
|
+
- **Block only on high-confidence NOT_SATISFIED.** `authority === 'blocking'` iff `verdict === 'NOT_SATISFIED' && confidence === 'high'`. Every other combination — all `SATISFIED`, all `INCONCLUSIVE`, and every `medium`/`low` `NOT_SATISFIED` — is advisory. Do not halt the workflow on an advisory verdict.
|
|
88
|
+
- **Always supply `diff` and `testOutput`.** Omitting them degrades the verdict to INCONCLUSIVE/advisory (a silent false-negative at the ship gate). Gather them from the session before invoking the tool.
|
|
89
|
+
- **Never block on infrastructure noise.** A provider failure, an unparseable response, or a missing spec must resolve to INCONCLUSIVE/advisory, never a thrown error or a block. The evaluator enforces this; do not reintroduce a hard failure in the wrapper.
|
|
90
|
+
- **Do not skip step 6.5.** The gate runs after Code Review and before Ship. A blocking verdict halts before step 7; it is not optional.
|
|
91
|
+
|
|
92
|
+
## Escalation
|
|
93
|
+
|
|
94
|
+
- **Blocking verdict the author disputes:** the resolution is to fix the implementation (or amend the spec's Success Criteria if they were wrong) and re-run outcome-eval — not to override the gate. If the spec itself is wrong, that is a spec change, reviewed on its own merits.
|
|
95
|
+
- **Repeated INCONCLUSIVE on a real change:** usually means `diff`/`testOutput` were not supplied, or no judgable section exists in the spec. Confirm inputs and that the spec has a Success Criteria / User-Visible Behavior / Overview section.
|
|
96
|
+
- **No `ANTHROPIC_API_KEY` configured:** every verdict degrades to INCONCLUSIVE/advisory and nothing blocks. Surface this to the human — the gate is effectively disabled until a provider is configured.
|
|
97
|
+
- **Verdict seems wrong (false positive/negative):** capture the spec section, diff, and verdict, and route to the maintainers; do not loosen the conservative-confidence prompt ad hoc.
|