@harness-engineering/cli 2.6.1 → 2.7.1

package/dist/agents/skills/claude-code/test-craft/SKILL.md

@@ -0,0 +1,212 @@
+# Test Craft
+
+> LLM-judgment critique of test quality across vitest / jest / mocha / playwright. Fourth member of the craft-pipeline initiative. Per-`it`/`test` block critique with best-effort source pairing for contract-vs-implementation rubrics. Tests are often the worst-written code in a codebase precisely because the rule-based floor (coverage threshold) is so easy to clear. Emits 3-axis findings (tier × impact × confidence per ADR 0019).
+
+## When to Use
+
+- During PR review on code that adds or changes tests
+- After ramping up coverage to audit whether new tests actually add signal
+- When onboarding a contributor — audit tests they introduced
+- Periodically to catch accumulated low-signal tests + redundant fixtures
+- NOT for coverage analysis (use `vitest --coverage` or harness-tdd — that's the floor)
+- NOT for autofix / rewriting (v2's `align-test` may add safe renames)
+- NOT for `.test-d.ts` type tests in v1 (v1.x — different rubric vocabulary)
+- NOT for fixture / helper / mock files in v1 (v1.x)
+- NOT for snapshot tests in v1 (different correctness criteria; v1.x)
+- NOT for non-TS/JS languages in v1 (v1.x)
+
+## Process
+
+### Phase 1: DISCOVER — Test files + framework
+
+1. **Read project configuration.** Check `harness.config.json` for:
+   - `craft.test.enabled` — gate (default `true`)
+   - `craft.test.maxFiles` (default 100), `craft.test.maxTestsPerFile` (default 20)
+   - `craft.test.frameworks` — restrict to subset (default: all four)
+   - `craft.test.sourcePair` — toggle source-pairing (default true)
+
+2. **Glob test files** under project root: `**/*.{test,spec}.{ts,tsx,js,jsx}`. Skip `node_modules`, `dist`, `build`, `coverage`, dotdirs.
+
+3. **Detect framework per file** via import signatures (order matters; most-specific first):
+   - `@playwright/test` → playwright
+   - `@jest/globals` → jest
+   - `vitest` → vitest
+   - `import 'mocha'` → mocha
+   - Fallback: vitest (most common; jest-with-globals projects still extract correctly because the AST shape matches)
+
+### Phase 2: EXTRACT — Per-test AST walk
+
+Single TS Compiler API walk per file. For each `CallExpression`:
+
+- **describe** — push name onto nesting stack, recurse, pop
+- **it / test** — extract `testName` (first string-literal arg), capture current nesting, capture callback body text (truncated to 1500 chars in prompt)
+- Modifiers handled: `.skip`, `.only`, `.todo` (todo excluded from critique — no body to critique)
+- Non-string-literal test names (computed / templates) skip silently
+
+### Phase 3: PAIR — Best-effort source resolution
+
+For each test file, try in order:
+
+1. **Sibling** — `foo.test.ts` → `foo.ts` (same dir)
+2. **Co-located in src** — `tests/foo.test.ts` → `../src/foo.ts`
+3. **Monorepo-style** — `tests/foo.test.ts` → `../../src/foo.ts`
+
+When source resolves, content (truncated to 2000 chars) is added to the LLM prompt under `Source under test:`. This enables `TEST-R007` (contract-not-implementation) to actually compare assertions against the function's public surface. When no source resolves, test-file-only rubrics still fire.
+
+### Phase 4: CRITIQUE — Per (test, rubric) loop
+
+8 seed rubrics:
+
+| Rubric                                  | Description                                                                 |
+| --------------------------------------- | --------------------------------------------------------------------------- |
+| `TEST-R001` contract-not-narrative-name | Test name describes the contract ("returns null when empty"), not narrative |
+| `TEST-R002` meaningful-assertion        | Assertion proves something the implementation could plausibly violate       |
+| `TEST-R003` arrange-act-assert          | Three visually distinct phases, not interleaved                             |
+| `TEST-R004` fixture-earns-setup-cost    | Heavy beforeEach must be justified by what's asserted                       |
+| `TEST-R005` single-responsibility       | One assertion target per `it`; multiple unrelated assertions split poorly   |
+| `TEST-R006` deleting-loses-something    | Would removing this test lose specific coverage, or is it redundant?        |
+| `TEST-R007` contract-not-implementation | Tests the documented behaviour, not the internal structure                  |
+| `TEST-R008` explicit-failure-mode       | Failure message narrates what went wrong without reading the test           |
+
+For each (test, rubric) pair:
+
+1. Build prompt with rubric + test (name + nesting + body) + optional source + framework label.
+2. LLM returns fenced JSON: `null` (rubric doesn't apply / test is fine) OR `{ tier, impact, confidence, message }`.
+3. On non-null: emit `TestFinding` with `cite.rubricId` for ADR 0020 traceability.
+
+### Phase 5: REPORT — Aggregate + cost telemetry
+
+Emit `TestCraftOutput`:
+
+```ts
+{
+  findings: TestFinding[];
+  summary: {
+    phaseRun: ['critique'];
+    durationMs: number;
+    llmCalls: { provider, model, count, costUsd };
+    catalog: { rubricsApplied: string[] };
+    counts: { filesScanned, testsExtracted, testsSkippedOrTodo, sourcePaired };
+    frameworksDetected: Record<TestFramework, number>;
+    runId: string;
+  }
+}
+```
+
+## Harness Integration
+
+- **`harness test-craft`** — CLI entry. `--files` / `--frameworks` / `--max-files` / `--max-tests-per-file` / `--no-source-pair` / `--json` / `--verbose`.
+- **`mcp__harness__test_craft`** — MCP tool. Same input/output. Consumed by agents.
+- **Cross-cutting API:** `critiqueTestsInFile(file, opts)` exported. Works on a single test file without project walk; honours framework filter and source-pairing toggle.
+- **Shared craft infrastructure:** imports `LlmProvider` + 3-axis types + `derivePriority` from `packages/cli/src/shared/craft/`.
+
+## Success Criteria
+
+See `docs/changes/craft-pipeline/test-craft/proposal.md` for the full 36 success criteria. Highlights:
+
+- 8 seed rubrics ship in `catalog/rubrics/<id>.ts` (file-per-rubric, matches naming/spec/copy-craft)
+- 3-axis output preserved (tier × impact × confidence)
+- `cite.rubricId` populated on every finding (ADR 0020)
+- Framework detection: `@playwright/test` / `@jest/globals` / `vitest` / `mocha` import signatures; vitest fallback
+- Per-test extraction handles `.skip` / `.only` / `.todo` with metadata; describe-chain nesting captured
+- Source pairing best-effort with silent skip when no match
+- Plugin slash-commands pre-generated (avoids CI drift failure pattern)
+
+## Examples
+
+### Example: Narrative test name
+
+**Input:** `src/parse.test.ts`:
+
+```ts
+describe('parseTokens', () => {
+  it('works correctly', () => {
+    expect(parseTokens('a,b,c')).toEqual(['a', 'b', 'c']);
+  });
+});
+```
+
+**Output (mock LLM):**
+
+```
+src/parse.test.ts
+  TEST-R001 [polish/medium/high] vitest:2
+    parseTokens > works correctly
+    "works correctly" narrates the test without naming the contract. Try
+    "splits comma-separated input into trimmed segments" or "preserves
+    order across the split".
+```
+
+### Example: Tautological assertion
+
+**Input:**
+
+```ts
+it('returns a user', () => {
+  const user = createUser({ name: 'a' });
+  expect(user).toBeDefined();
+});
+```
+
+**Output:**
+
+```
+TEST-R002 [foundational/medium/high] vitest:8
+  createUser > returns a user
+  `toBeDefined()` after `createUser(...)` cannot fail — the return is
+  always defined by the function's signature. Assert the contract:
+  `expect(user.name).toBe('a')` or `expect(user).toMatchObject({...})`.
+```
+
+### Example: Heavy fixture, trivial assertion
+
+**Input:**
+
+```ts
+it('handles input', () => {
+  // 47 lines of beforeEach setup that mocks 6 collaborators
+  const result = handler.handle({});
+  expect(result).toBeTruthy();
+});
+```
+
+**Output:**
+
+```
+TEST-R004 [polish/large/medium] vitest:142
+  handler > handles input
+  47-line beforeEach + 6 mocks produce a result that's asserted with
+  `toBeTruthy`. The fixture's complexity exceeds the test's signal —
+  either drop the fixture and use a simpler stub, or assert the
+  specific outcome that justified the setup (e.g., which collaborator
+  was called with what arguments).
+```
+
+## Gates
+
+- **No autofix.** v2's `align-test`.
+- **No coverage analysis.** That's the floor (`vitest --coverage`).
+- **No `.test-d.ts` type tests** (v1.x).
+- **No fixture / helper / mock file critique** (v1.x).
+- **No snapshot rubrics** (v1.x).
+- **No non-TS/JS language support** (v1.x).
+- **No B' bootstrap.** Same posture as naming/spec/copy-craft.
+- **No graph persistence.** Phase 1 MVP.
+
+## Escalation
+
+- **When LLM cost is too high:** drop `maxTestsPerFile` (default 20) or scope to specific frameworks with `--frameworks vitest`. Disable source-pairing with `--no-source-pair` to halve prompt size on every test.
+- **When intentionally narrative test names get flagged (e.g., learning tests, examples):** scope via `--files` to exclude. v1.x adds `// test-craft:skip` annotation.
+- **When source-pairing finds the wrong source for ambiguous names:** the LLM gets misleading context for `TEST-R007`. Use `--no-source-pair` to fall back to test-file-only rubrics, or v1.x's `harness.config.json` test→source mapping.
+- **When jest-with-globals tests aren't detected:** the framework defaults to vitest (AST is compatible); critique still runs. If you need jest tagging specifically, add `import { describe, it } from '@jest/globals'` to make the signature explicit.
+- **When `it.each` / `test.each` blocks get critiqued as a single test:** v1 captures the `each` invocation as one item with a generic name pattern. v1.x adds per-iteration critique.
+
+## Status
+
+**v1 — in implementation.** See:
+
+- Spec: `docs/changes/craft-pipeline/test-craft/proposal.md`
+- Roadmap entry: `craft-pipeline sub-project #3`
+- Sibling craft skills: `naming-craft` (#1), `spec-craft` (#6), `copy-craft` (#5)
+- Shared infrastructure: `packages/cli/src/shared/craft/`
+- Future: `align-test` (FIX side, v2), docs-craft (#2), code-craft (#4)

package/dist/agents/skills/claude-code/test-craft/skill.yaml

@@ -0,0 +1,58 @@
+name: test-craft
+version: "0.1.0"
+description: LLM-judgment critique of test quality across vitest / jest / mocha / playwright. Fourth craft-pipeline ceiling skill. Per-test critique with best-effort source pairing for contract-vs-implementation rubrics. Tests are often the worst-written code in a codebase precisely because the rule-based floor is so easy to clear.
+stability: draft
+cognitive_mode: constructive-architect
+triggers:
+  - manual
+  - on_pr
+  - on_new_feature
+platforms:
+  - claude-code
+tools:
+  - Bash
+  - Read
+  - Glob
+  - Grep
+cli:
+  command: harness test-craft
+  args:
+    - name: path
+      description: Project root path
+      required: false
+    - name: files
+      description: Optional test file/glob scope
+      required: false
+    - name: frameworks
+      description: Restrict to vitest/jest/mocha/playwright
+      required: false
+    - name: no-source-pair
+      description: Skip source-pairing resolution
+      required: false
+mcp:
+  tool: test_craft
+  input:
+    path: string
+type: rigid
+tier: 2
+phases:
+  - name: discover
+    description: Glob test files; detect framework per-file via import signatures
+    required: true
+  - name: extract
+    description: Walk TS AST for per-it/test block with nesting, skip/todo/only flags
+    required: true
+  - name: pair
+    description: Best-effort resolve source file under test (sibling / ../src / monorepo)
+    required: false
+  - name: critique
+    description: LLM-rubric loop per (test, rubric); 8 seed rubrics
+    required: true
+  - name: report
+    description: Aggregate findings + frameworks-detected counts + cost telemetry
+    required: true
+state:
+  persistent: false
+depends_on:
+  - harness-tdd
+  - harness-design-craft

package/dist/agents/skills/codex/align-design-system/SKILL.md

@@ -0,0 +1,174 @@
+# Align Design System
+
+> Apply codemods for safe DRIFT-T001/T002/T003 token-bypass findings (replace hex / font-family / px-spacing literals with token references) and emit precise suggestions for DRIFT-T004 (deprecated tokens) and all DRIFT-P\* (primitive adoption). The FIX half of design-pipeline sub-project #1, paired with detect-design-drift.
+
+## When to Use
+
+- After detect-design-drift reports DRIFT-T001/T002/T003 findings — align replaces literals with token references where the fix is unambiguous
+- Before a PR that touches UI code lands — pair with detect-design-drift to surface AND fix drift in one shot
+- Inside a (future) #5 design-pipeline orchestrator's convergence loop — align is the FIX step the loop runs between DETECT and VERIFY
+- When you want to dry-run the fixes first (`--dry-run`) and review the diff before writing
+- NOT for primitive-adoption codemods — v1 emits suggestions only for DRIFT-P\*; the prop-translation work lives in v1.x
+- NOT for adding new tokens to tokens.json (no auto-add of palette entries; that's a separate intentional act)
+- NOT for non-design-system fixes (use cleanup-dead-code, align-documentation, etc. for their respective domains)
+
+## Process
+
+### Phase 1: GATHER — Load drift findings
+
+1. **Read project configuration.** Check `harness.config.json` for:
+   - `design.strictness` — passes through to detect-design-drift
+   - `design.audit.driftDetection.*` — passes through to detect
+
+2. **In standalone mode (default):** invoke `detect-design-drift` internally with the same project root + strictness; receive the full `DriftFinding[]`.
+
+3. **In pipeline mode:** read `.harness/handoff.json` and pull `pipeline.driftFindings` (pre-classified by the orchestrator). Honor `pipeline.fixBatch` if present to limit application to a specific subset of findings (the orchestrator may apply fixes in batches across iterations).
+
+### Phase 2: CLASSIFY — Pre-flight safe-codemod vs suggestion
+
+For each finding, the pre-flight classifier inspects file context and chooses:
+
+- **DRIFT-T001 (hex)** — safe-codemod iff token import already present in the file AND hex is in single string-literal context (not a template literal or concatenation) AND exactly one palette token matches by value. Otherwise → suggestion.
+- **DRIFT-T002 (font-family)** — same shape as T001 against the typography palette.
+- **DRIFT-T003 (px spacing)** — safe-codemod iff token import present AND px matches a spacing token's `$value` EXACTLY (no rounding) AND not in an arithmetic expression. Otherwise → suggestion.
+- **DRIFT-T004 (deprecated)** — always suggestion in v1. Migration target may not be in the token's `$description`.
+- **DRIFT-P\* (primitive adoption)** — always suggestion in v1. Prop translation across `<button>` ⇄ `<Button>` is genuinely ambiguous (event handlers, ref forwarding, class merging) — codemods deferred to v1.x.
+
+Token import discovery recognizes three forms:
+
+- ES named: `import { tokens } from '...'`
+- ES default: `import tokens from '...'`
+- CJS: `const tokens = require('...')`
+
+### Phase 3: APPLY — Codemod or emit suggestion
+
+For each finding classified as `safe-codemod`:
+
+1. Read the source file (cached per-run to avoid re-reads when multiple findings hit the same file).
+2. Locate the exact match position from the finding's `line` + `evidence.snippet`.
+3. Compute the replacement using file-extension-aware syntax:
+   - `.ts` / `.tsx` / `.js` / `.jsx` → `tokens.<dotted.path>`
+   - `.css` / `.scss` → `var(--<dotted-path-as-kebab>)`
+4. Replace in-place. Emit `FixOutcome.applied` with a structured diff (file / line / before / after).
+5. If `--dry-run`: compute the diff but DO NOT write to disk.
+6. If the file changed between detect-time and apply-time: skip with `kind: 'skipped-unsafe', reason: 'file changed since finding'`.
+
+For each finding classified as `suggestion`: emit a human-readable description plus a preview of the suggested change. No file mutation.
+
+### Phase 4: REPORT — Aggregate + (pipeline-mode) handoff writeback
+
+1. Aggregate `FixOutcome[]` into a summary: counts by kind (applied / suggestion / skipped / failed) plus files modified and duration.
+2. Build a `catalog` of finding codes that produced codemods and codes that produced suggestions.
+3. In pipeline mode: write `pipeline.fixesApplied: FixOutcome[]` back to `.harness/handoff.json` so the orchestrator can re-verify only the affected findings on the next loop iteration.
+
+## Harness Integration
+
+- **`harness align-design-system`** — the CLI entry point. `--dry-run` for preview; `--write` is the default. Standard `--json` / `--verbose` / `--quiet` flags.
+- **`harness align-design-system --mode pipeline`** — orchestrator-driven mode. Reads pre-classified findings from handoff.json; writes outcomes back.
+- **`mcp__harness__align_design_system`** — MCP tool for agent consumption. Same input/output shape as the function call.
+- **`detect-design-drift`** — soft dependency. Standalone mode invokes detect internally; pipeline mode trusts the orchestrator to have done it.
+- **`harness check-design`** — composes detect (as 3rd verifier) into a single-pass design check. align is the matching FIX step; together they form the DETECT → FIX cycle that the (future) #5 orchestrator will loop.
+- **`DesignConstraintAdapter`** — align does NOT write to the graph. The graph already tracks `VIOLATES_design` edges (the findings). Re-running detect after align shows the delta — no separate fix-edge needed.
+
+## Success Criteria
+
+See `docs/changes/design-pipeline/align-design-system/proposal.md` for the full 34 success criteria. Highlights:
+
+- T001/T002/T003 codemods write to disk only when pre-flight classifier returns `safe-codemod`
+- Codemods are idempotent — running twice produces zero additional changes on the second run
+- Classifier downgrades to suggestion when token import is missing, when value appears in template/concatenation, or when multiple tokens share the value
+- Classifier ALWAYS returns suggestion for T004 + all P\* findings (no source inspection)
+- Pipeline mode reads `pipeline.driftFindings` from handoff.json; standalone mode runs detect internally
+- Pipeline mode writes `pipeline.fixesApplied` back to handoff.json
+- `--dry-run` produces identical `FixOutcome` shapes but never writes files
+- Re-running detect after align produces strictly fewer T001/T002/T003 findings
+
+## Examples
+
+### Example: Apply a T001 codemod (hex → token reference)
+
+**Input:**
+
+`design-system/tokens.json` has:
+
+```json
+{ "color": { "brand": { "primary": { "$type": "color", "$value": "#0066cc" } } } }
+```
+
+`src/Card.tsx` has:
+
+```ts
+import { tokens } from '@/design-system/tokens';
+const styles = { color: '#0066cc' }; // raw literal where token exists
+```
+
+**Output:**
+
+```
+src/Card.tsx
+  ✓ DRIFT-T001:2 — Hex color "#0066cc" should use a token reference instead of a raw literal
+     before: const styles = { color: "#0066cc" };
+     after:  const styles = { color: tokens.color.brand.primary };
+
+Summary: 1 applied, 0 suggestions, 0 skipped, 0 failed (1 files modified, 5ms)
+```
+
+### Example: Downgrade to suggestion (no token import)
+
+Same finding as above, but `src/Card.tsx` does NOT import tokens. align emits a suggestion instead:
+
+```
+src/Card.tsx
+  ? DRIFT-T001:2 — Hex color "#0066cc" should use a token reference instead of a raw literal
+
+Summary: 0 applied, 1 suggestions, 0 skipped, 0 failed (0 files modified, 3ms)
+```
+
+(Run with `--verbose` to see the suggestion text and the classifier's reason for downgrading.)
+
+### Example: Suggestion-only for DRIFT-P001
+
+**Input:**
+
+`design-system/DESIGN.md` registers `Button` in `## Component Registry`. `src/SaveButton.tsx` has:
+
+```tsx
+export const S = () => <button onClick={() => save()}>Save</button>;
+```
+
+**Output:**
+
+```
+src/SaveButton.tsx
+  ? DRIFT-P001:1 — Raw <button> element where the registered component "Button" should be used
+
+(use --verbose to see prop-translation suggestion)
+```
+
+v1 never auto-applies primitive adoption — prop translation across `<button>` ⇄ `<Button>` is the kind of judgment-call that benefits from a human or LLM review.
+
+## Gates
+
+- **No autofix without classifier approval.** Every codemod application goes through `classifyFinding`. If the classifier returns `suggestion`, NO file write occurs — even for the same finding code in the same file.
+- **No autofix when token import is missing.** Adding the import line is its own ambiguity surface (alias? barrel? relative?). v1 skips with a suggestion.
+- **No autofix for non-exact px matches.** "Round 13px to the nearest 16px" is a design decision the tool shouldn't make.
+- **No autofix for primitive adoption.** v1 has no codemod for DRIFT-P\* — always suggestions.
+- **No autofix in pipeline mode unless a `pipeline.driftFindings` field is present.** Empty handoff = empty run.
+- **No graph writes.** align modifies source files; the graph is read-only from align's perspective.
+
+## Escalation
+
+- **When a T001 finding has multiple matching tokens (ambiguous):** the classifier downgrades to suggestion. To resolve, declare a primary token in DESIGN.md `## Token Primary Resolution Overrides` (v1.x), OR pick one in the source manually and re-run.
+- **When the codemod corrupts a file (rare):** every application includes a structured diff. Recover via `git checkout <file>`. If the same input repeatedly corrupts, report the case — pre-flight classifier rules are conservative by design.
+- **When pipeline-mode run finds no `pipeline.driftFindings` field:** align exits cleanly with empty outcomes. The orchestrator's contract is to write the field BEFORE invoking align.
+- **When `--dry-run` shows fixes you don't want applied:** scope with `--files <glob>` to apply only specific files, or invoke align in pipeline mode with a curated `pipeline.fixBatch` list.
+- **When you want primitive-adoption fixes today:** apply the suggestion manually. The v1.x sub-project will add prop-translation tables + import resolution + revert-on-test-fail.
+- **When align is invoked without detect having run first (standalone mode):** standalone mode runs detect internally — no manual ordering needed. Pipeline mode trusts the orchestrator to populate findings.
+
+## Status
+
+**v1 — in implementation.** See:
+
+- Spec: `docs/changes/design-pipeline/align-design-system/proposal.md`
+- Roadmap entry: `design-pipeline sub-project #1` (align half) in `docs/roadmap.md`
+- Sibling: `detect-design-drift` (detect half — shipped PR #396)

package/dist/agents/skills/codex/align-design-system/skill.yaml

@@ -0,0 +1,57 @@
+name: align-design-system
+version: "0.1.0"
+description: Apply codemods for safe DRIFT-T001/T002/T003 token-bypass findings; emit precise suggestions for DRIFT-T004 (deprecated tokens) and all DRIFT-P* (primitive adoption). FIX half of design-pipeline sub-project #1; pairs with detect-design-drift.
+stability: draft
+cognitive_mode: constructive-architect
+triggers:
+  - manual
+  - on_pr
+  - on_new_feature
+platforms:
+  - claude-code
+tools:
+  - Bash
+  - Read
+  - Write
+  - Edit
+  - Glob
+  - Grep
+cli:
+  command: harness align-design-system
+  args:
+    - name: path
+      description: Project root path
+      required: false
+    - name: dry-run
+      description: Compute diffs without writing files
+      required: false
+    - name: files
+      description: Optional file/glob scope (standalone mode only)
+      required: false
+    - name: mode
+      description: standalone (default) or pipeline (read findings from handoff.json)
+      required: false
+mcp:
+  tool: run_skill
+  input:
+    skill: align-design-system
+    path: string
+type: rigid
+tier: 2
+phases:
+  - name: gather
+    description: Load drift findings (run detect-design-drift in standalone mode; read pipeline.driftFindings in pipeline mode)
+    required: true
+  - name: classify
+    description: Per-finding pre-flight check decides safe-codemod vs suggestion
+    required: true
+  - name: apply
+    description: Write codemods for safe T001/T002/T003; emit suggestions for everything else
+    required: true
+  - name: report
+    description: Emit FixOutcome[] + aggregate summary; write pipeline.fixesApplied in pipeline mode
+    required: true
+state:
+  persistent: false
+depends_on:
+  - detect-design-drift