@harness-engineering/cli 1.14.0 → 1.16.0

package/dist/agents/skills/codex/harness-verify/SKILL.md

@@ -0,0 +1,159 @@
+# Harness Verify
+
+> Binary pass/fail quick gate. Runs test, lint, typecheck — returns structured result. No judgment calls, no deep analysis. Pass or fail.
+
+## When to Use
+
+- After every task as a final sanity check
+- As the final step in any code-producing skill
+- When you need a fast mechanical answer: "does the code compile, lint, and pass tests?"
+- NOT for deep verification or architectural review (use `harness-verification` for that)
+- NOT for AI-powered code review (use `harness-code-review` for that)
+
+## Relationship to Other Skills
+
+| Skill                     | What It Does                                   | Time  |
+| ------------------------- | ---------------------------------------------- | ----- |
+| **harness-verify** (this) | Mechanical quick gate: typecheck, lint, test   | ~30s  |
+| **harness-verification**  | Deep audit: architecture, patterns, edge cases | ~5min |
+
+`harness-verify` is the fast, deterministic gate. `harness-verification` is the slow, thorough audit. They serve different purposes and should not be confused.
+
+## Process
+
+### Phase 1: DETECT
+
+Auto-detect project commands by inspecting the project root:
+
+1. **package.json** — Look for `scripts.test`, `scripts.lint`, `scripts.typecheck` (or `scripts.tsc`, `scripts.type-check`)
+2. **Makefile** — Look for `test`, `lint`, `typecheck` targets
+3. **Conventions** — Fall back to common commands:
+   - Typecheck: `npx tsc --noEmit`, `mypy .`, `go vet ./...`
+   - Lint: `npx eslint .`, `ruff check .`, `golangci-lint run`
+   - Test: `npm test`, `pytest`, `go test ./...`
+4. **Language detection** — Use file extensions to determine which convention set applies
+
+For each of the three checks (typecheck, lint, test), record either the detected command or `NONE` if no command can be determined.
+
+### Phase 2: EXECUTE
+
+Run all detected commands in this order: **typecheck -> lint -> test**.
+
+Rules:
+
+- Run ALL commands regardless of earlier failures. Do not short-circuit.
+- Capture exit code, stdout, and stderr for each command.
+- Exit code 0 = PASS. Any non-zero exit code = FAIL.
+- If a command was `NONE` (not detected), mark that check as `SKIPPED`.
+- Run each command from the project root directory.
+- Do not modify any files. Do not install dependencies. Do not fix errors.
+
+### Design Constraint Check (conditional)
+
+When `harness.config.json` contains a `design` block:
+
+1. **Run design constraint checks** by invoking `harness-accessibility` in scan+evaluate mode against the project.
+2. Apply the `design.strictness` setting to determine severity:
+   - `strict`: accessibility violations are FAIL; anti-pattern violations are WARN
+   - `standard`: accessibility and anti-pattern violations are WARN; nothing blocks
+   - `permissive`: all design violations are INFO
+3. Capture the result as `Design: [PASS/WARN/FAIL/SKIPPED]`.
+4. If no `design` block exists in config, mark Design as `SKIPPED`.
+
+The design check runs AFTER test/lint/typecheck. It does not short-circuit on earlier failures.
+
+### Phase 3: REPORT
+
+Output a structured result in this exact format:
+
+```
+Verification: [PASS/FAIL]
+- Typecheck: [PASS/FAIL/SKIPPED]
+- Lint:      [PASS/FAIL/SKIPPED]
+- Test:      [PASS/FAIL/SKIPPED]
+```
+
+When design config is present, include the design line:
+
+```
+Verification: [PASS/FAIL]
+- Typecheck: [PASS/FAIL/SKIPPED]
+- Lint:      [PASS/FAIL/SKIPPED]
+- Test:      [PASS/FAIL/SKIPPED]
+- Design:    [PASS/WARN/FAIL/SKIPPED]
+```
+
+Rules:
+
+- Overall `Verification: PASS` only if all non-skipped checks passed.
+- If all checks are SKIPPED, overall result is `PASS` (nothing to fail).
+- On FAIL, include a brief summary of what failed (e.g., "3 type errors", "2 lint errors", "5 tests failed") below the structured block.
+
+### Roadmap Sync (conditional)
+
+When all non-skipped checks pass (overall `Verification: PASS`) and `docs/roadmap.md` exists:
+
+1. Trigger a roadmap sync to update feature statuses based on the verified state.
+2. Use the `manage_roadmap` MCP tool with `sync` action if available, or note to the caller that a roadmap sync is recommended.
+3. Features linked to plans whose tasks are all complete and verified may be marked as `done`.
+
+If `docs/roadmap.md` does not exist, skip this step silently. If verification failed, do not sync — the roadmap should only reflect verified completions.
+
+## Deterministic Checks
+
+This skill is entirely deterministic. There are no LLM judgment calls anywhere in the process.
+
+- Exit code 0 = PASS. Always.
+- Exit code non-zero = FAIL. Always.
+- No "it looks like this might be okay" reasoning. No interpretation of output.
+- The same codebase with the same commands will always produce the same result.
+
+## Harness Integration
+
+- Follows Principle 7 (Deterministic-vs-LLM Split) — this skill is entirely deterministic
+- Invoked as the final step by code-producing skills (harness-execution, harness-tdd)
+- Complements harness-verification (deep audit) — use verify for quick checks, verification for milestones
+- Output format is consumed by harness-integrity for the unified pipeline
+- Invokes `harness-accessibility` for design constraint checking when `design` config exists
+- Design violations respect `design.strictness` from `harness.config.json`
+- **Roadmap sync** — When verification passes and `docs/roadmap.md` exists, triggers `manage_roadmap sync` to mark verified features as `done`. Only fires on overall PASS.
+
+## Success Criteria
+
+- [ ] All detected commands were executed
+- [ ] Report follows the structured format exactly
+- [ ] Overall verdict correctly reflects individual results
+- [ ] Failed checks include error output summary
+
+## Examples
+
+### Example: Node.js Project
+
+```
+Verification: PASS
+- Types: PASS (no errors)
+- Lint: PASS (0 warnings)
+- Tests: PASS (42/42)
+```
+
+### Example: Failing Project
+
+```
+Verification: FAIL
+- Types: FAIL (3 type errors in src/auth/login.ts)
+- Lint: PASS
+- Tests: NOT RUN
+```
+
+## Gates
+
+- **No judgment calls.** The exit code is the only signal.
+- **No skipping.** If a command is detected, it runs. Period.
+- **Fresh execution only.** Do not cache results. Do not reuse previous runs. Execute the commands right now.
+- **No file modifications.** This skill is read-only (plus command execution). It must not change the codebase.
+
+## Escalation
+
+- **Timeout:** Each command has a 120-second timeout. If a command exceeds this, mark it as FAIL with reason "TIMEOUT".
+- **No commands detected:** If no typecheck, lint, or test commands can be detected, all three checks are SKIPPED and the overall result is PASS. Log a note that no verification commands were found.
+- **Environment errors:** If a command fails due to missing tooling (e.g., `tsc` not installed), mark it as FAIL. Do not attempt to install the tooling.

package/dist/agents/skills/codex/harness-verify/skill.yaml

@@ -0,0 +1,41 @@
+name: harness-verify
+version: "1.0.0"
+description: Binary pass/fail quick gate — runs test, lint, typecheck commands and returns structured result
+triggers:
+  - manual
+  - on_task_complete
+platforms:
+  - claude-code
+  - gemini-cli
+tools:
+  - Bash
+  - Read
+  - Glob
+cli:
+  command: harness skill run harness-verify
+  args:
+    - name: path
+      description: Project root path
+      required: false
+mcp:
+  tool: run_skill
+  input:
+    skill: harness-verify
+    path: string
+type: rigid
+tier: 2
+cognitive_mode: meticulous-verifier
+phases:
+  - name: detect
+    description: Auto-detect project commands from package.json, Makefile, or conventions
+    required: true
+  - name: execute
+    description: Run test, lint, typecheck commands
+    required: true
+  - name: report
+    description: Return structured pass/fail result
+    required: true
+state:
+  persistent: false
+  files: []
+depends_on: []

package/dist/agents/skills/codex/harness-visual-regression/SKILL.md

@@ -0,0 +1,257 @@
+# Harness Visual Regression
+
+> Screenshot comparison, visual diff detection, and baseline management. Catches unintended CSS regressions, layout shifts, and rendering inconsistencies before they reach production.
+
+## When to Use
+
+- Adding visual regression coverage for UI components or pages
+- Reviewing visual changes in a pull request before merge
+- Updating baselines after intentional design changes
+- NOT when testing interactive user flows (use harness-e2e instead)
+- NOT when testing component behavior or state (use unit tests or harness-tdd instead)
+- NOT when auditing accessibility compliance (use harness-accessibility instead)
+
+## Process
+
+### Phase 1: DETECT -- Identify UI Components and Rendering Infrastructure
+
+1. **Scan for existing visual test infrastructure.** Search for:
+   - Storybook configuration (`.storybook/`, `*.stories.tsx`, `*.stories.ts`)
+   - Visual testing tools (Chromatic config, `percy.yml`, Playwright screenshot tests)
+   - Existing baseline directories (`screenshots/`, `__image_snapshots__/`, `visual-tests/`)
+
+2. **Catalog testable components.** Identify UI surfaces that benefit from visual testing:
+   - Shared design system components (buttons, forms, modals, navigation)
+   - Page-level layouts (dashboard, settings, landing page)
+   - Responsive breakpoints (mobile, tablet, desktop)
+   - Theme variants (light mode, dark mode)
+   - States (loading, empty, error, populated)
+
+3. **Determine the rendering strategy.** Choose how screenshots are captured:
+   - **Storybook + Chromatic/Percy:** best for component libraries with existing stories
+   - **Playwright screenshots:** best for full-page and integration-level visual tests
+   - **Jest + jest-image-snapshot:** best for lightweight component rendering with jsdom or happy-dom
+   - **Cypress + Percy plugin:** best when Cypress is already the E2E framework
+
+4. **Identify viewport and theme matrix.** Define the combinations to test:
+   - Viewports: 375px (mobile), 768px (tablet), 1280px (desktop), 1920px (wide)
+   - Themes: light, dark (if supported)
+   - Locales: LTR, RTL (if internationalized)
+
+5. **Report findings.** Summarize: components to cover, rendering strategy, viewport matrix, and estimated baseline count.
+
+### Phase 2: BASELINE -- Capture Reference Screenshots
+
+1. **Configure the visual testing tool.** Set up:
+   - Screenshot output directory with `.gitkeep` or add to `.gitignore` as appropriate
+   - Threshold for pixel-level diff tolerance (recommended: 0.1% for component tests, 0.5% for full-page tests)
+   - Anti-aliasing handling to avoid false positives across different rendering engines
+   - Font loading: wait for web fonts to load before capture, or use a system font fallback in test mode
+
+2. **Stabilize rendering for deterministic screenshots.** Address common sources of non-determinism:
+   - Disable CSS animations and transitions in test mode
+   - Mock dates and times to prevent timestamp-based changes
+   - Replace dynamic content (avatars, ads, user-generated content) with stable placeholders
+   - Set a fixed random seed for any randomized UI elements
+
+3. **Capture baseline screenshots.** For each component in the test matrix:
+   - Render the component in each viewport and theme combination
+   - Wait for fonts, images, and lazy-loaded content to fully render
+   - Capture and save the screenshot to the baseline directory
+
+4. **Review baselines manually.** Before committing, visually inspect every baseline screenshot. Confirm:
+   - The component renders correctly at each viewport
+   - No rendering artifacts (clipped text, missing icons, broken layouts)
+   - The screenshot captures the full component without excessive whitespace
+
+5. **Commit baselines.** Add baseline screenshots to version control with a descriptive commit message. These baselines become the source of truth for future comparisons.
+
+### Phase 3: COMPARE -- Run Visual Diffs Against Baselines
+
+1. **Execute visual comparison.** Run the visual test suite, which:
+   - Renders each component in the same viewport/theme matrix
+   - Captures a new screenshot for each combination
+   - Compares the new screenshot against the stored baseline pixel-by-pixel
+   - Reports differences that exceed the configured threshold
+
+2. **Classify each diff.** For every screenshot that exceeds the threshold:
+   - **Intentional change:** the diff corresponds to a deliberate design update in the current PR. Mark for baseline update.
+   - **Regression:** the diff is unintended and represents a visual bug. Flag for investigation.
+   - **Environmental noise:** the diff is caused by rendering differences (sub-pixel anti-aliasing, font hinting). Increase threshold or stabilize rendering.
+
+3. **Investigate regressions.** For each regression:
+   - Identify the CSS or component change that caused the visual shift
+   - Determine if the change is localized (one component) or cascading (layout shift affecting multiple components)
+   - Check if the change is caused by a dependency update (CSS framework, icon library)
+
+4. **Update baselines for intentional changes.** When a visual change is confirmed intentional:
+   - Re-capture the baseline for affected screenshots
+   - Review the updated baseline to confirm it matches the design intent
+   - Commit updated baselines alongside the code change
+
+5. **Generate a diff report.** Produce a summary showing:
+   - Total screenshots compared
+   - Screenshots unchanged (passed)
+   - Screenshots with intentional changes (baselines updated)
+   - Screenshots with regressions (flagged for fix)
+
+### Phase 4: REPORT -- Generate Visual Diff Report and Approval Workflow
+
+1. **Create a visual diff summary for PR review.** Include:
+   - Side-by-side comparison images for each changed screenshot
+   - Diff overlay highlighting the pixels that changed
+   - Percentage of pixels changed per screenshot
+   - Grouped by component and viewport
+
+2. **Integrate with CI pipeline.** Configure the visual test suite to:
+   - Run automatically on every pull request
+   - Block merge when unapproved visual changes are detected
+   - Provide a link to the visual diff report in the PR status check
+
+3. **Define the approval workflow.** Establish:
+   - Who can approve visual changes (design team, frontend lead)
+   - How approvals are recorded (PR comment, Chromatic approval, Percy review)
+   - What constitutes an "auto-approve" (changes below threshold, test-only files)
+
+4. **Run `harness validate`.** Confirm the project passes all harness checks with visual testing infrastructure in place.
+
+5. **Document the visual testing workflow.** Record:
+   - How to run visual tests locally
+   - How to update baselines after intentional changes
+   - How to add visual tests for new components
+   - Where to find the diff report in CI
+
+### Graph Refresh
+
+If a knowledge graph exists at `.harness/graph/`, refresh it after code changes to keep graph queries accurate:
+
+```
+harness scan [path]
+```
+
+## Harness Integration
+
+- **`harness validate`** -- Run in REPORT phase after visual testing infrastructure is complete. Confirms project health.
+- **`harness check-deps`** -- Run after BASELINE phase to verify visual testing dependencies are in devDependencies.
+- **`emit_interaction`** -- Used to present visual diff results and request human approval for baseline updates.
+- **Glob** -- Used in DETECT phase to find Storybook stories, existing screenshots, and component files.
+- **Grep** -- Used to search for CSS animation properties, dynamic content patterns, and non-deterministic rendering.
+
+## Success Criteria
+
+- Every shared design system component has visual baselines for at least mobile and desktop viewports
+- Visual diffs are deterministic: running the same code produces the same screenshots every time
+- No false positives: environmental noise (font rendering, anti-aliasing) does not trigger diff failures
+- Intentional changes are distinguished from regressions in the diff report
+- Baselines are committed to version control and updated alongside code changes
+- CI blocks merge when unapproved visual changes are detected
+- `harness validate` passes with visual testing infrastructure in place
+
+## Examples
+
+### Example: Playwright Visual Regression for a React App
+
+**BASELINE -- Capture component screenshots:**
+
+```typescript
+// visual-tests/components.spec.ts
+import { test, expect } from '@playwright/test';
+
+const viewports = [
+  { name: 'mobile', width: 375, height: 812 },
+  { name: 'desktop', width: 1280, height: 720 },
+];
+
+for (const viewport of viewports) {
+  test.describe(`${viewport.name} viewport`, () => {
+    test.use({ viewport: { width: viewport.width, height: viewport.height } });
+
+    test('dashboard renders correctly', async ({ page }) => {
+      await page.goto('/dashboard');
+      await page.waitForLoadState('networkidle');
+      // Disable animations for deterministic screenshots
+      await page.addStyleTag({
+        content:
+          '*, *::before, *::after { animation: none !important; transition: none !important; }',
+      });
+      await expect(page).toHaveScreenshot(`dashboard-${viewport.name}.png`, {
+        maxDiffPixelRatio: 0.005,
+        fullPage: true,
+      });
+    });
+
+    test('settings page renders correctly', async ({ page }) => {
+      await page.goto('/settings');
+      await page.waitForLoadState('networkidle');
+      await page.addStyleTag({
+        content:
+          '*, *::before, *::after { animation: none !important; transition: none !important; }',
+      });
+      await expect(page).toHaveScreenshot(`settings-${viewport.name}.png`, {
+        maxDiffPixelRatio: 0.005,
+      });
+    });
+  });
+}
+```
+
+### Example: Storybook with Chromatic
+
+**DETECT output:**
+
+```
+Storybook: v7.6 detected (.storybook/main.ts)
+Stories: 47 stories across 23 components
+Chromatic: not configured
+Existing baselines: none
+Components without stories: Modal, Toast, DatePicker (3 gaps)
+```
+
+**BASELINE -- Configure Chromatic and run first build:**
+
+```json
+// package.json (relevant scripts)
+{
+  "scripts": {
+    "chromatic": "chromatic --project-token=${CHROMATIC_PROJECT_TOKEN}",
+    "chromatic:ci": "chromatic --project-token=${CHROMATIC_PROJECT_TOKEN} --exit-zero-on-changes --auto-accept-changes main"
+  }
+}
+```
+
+```typescript
+// .storybook/preview.ts -- stabilize rendering
+import { Preview } from '@storybook/react';
+
+const preview: Preview = {
+  parameters: {
+    chromatic: {
+      pauseAnimationAtEnd: true,
+      viewports: [375, 768, 1280],
+    },
+  },
+  decorators: [
+    (Story) => (
+      <div style={{ fontFamily: 'Arial, sans-serif' }}>
+        <Story />
+      </div>
+    ),
+  ],
+};
+
+export default preview;
+```
+
+## Gates
+
+- **No non-deterministic screenshots.** If the same code produces different screenshots on consecutive runs, the rendering is not stabilized. Fix animations, dynamic content, and font loading before capturing baselines.
+- **No uncommitted baselines.** Baseline screenshots must be in version control. If baselines exist only on a developer's machine, CI cannot compare against them. Commit baselines with the code that creates them.
+- **No threshold above 1%.** A pixel diff threshold above 1% hides real regressions. If environmental noise requires a higher threshold, fix the noise source (fonts, animations) rather than raising the threshold.
+- **No visual tests without review workflow.** Visual tests that run but whose results are never reviewed provide false confidence. Every visual diff must have a defined approval path.
+
+## Escalation
+
+- **When screenshots differ between local and CI environments:** This is usually caused by different font rendering, display scaling, or browser versions. Standardize by running visual tests in Docker with a fixed browser version and system fonts. Do not try to match local and CI rendering -- pick one as the source of truth.
+- **When baseline updates flood a PR with hundreds of changed screenshots:** Group changes by root cause. If a single CSS variable change cascades to 200 screenshots, approve the root cause and batch-update baselines. Consider whether the cascade indicates a design system architecture issue.
+- **When dynamic content (user avatars, timestamps, ads) causes false positives:** Mock or replace dynamic content in the test environment. Use Storybook args or Playwright route interception to inject stable placeholder content.
+- **When the visual test suite takes too long (> 15 minutes):** Prioritize components by change frequency. Run the full visual suite nightly, and only test changed components on each PR using affected-story detection.

package/dist/agents/skills/codex/harness-visual-regression/skill.yaml

@@ -0,0 +1,74 @@
+name: harness-visual-regression
+version: "1.0.0"
+description: Screenshot comparison, visual diff detection, and baseline management
+cognitive_mode: meticulous-verifier
+triggers:
+  - manual
+  - on_pr
+platforms:
+  - claude-code
+  - gemini-cli
+tools:
+  - Bash
+  - Read
+  - Write
+  - Edit
+  - Glob
+  - Grep
+  - emit_interaction
+cli:
+  command: harness skill run harness-visual-regression
+  args:
+    - name: path
+      description: Project root path
+      required: false
+    - name: tool
+      description: "Visual testing tool: chromatic, percy, pixelmatch, or playwright. Auto-detected when omitted."
+      required: false
+    - name: update-baselines
+      description: "Update baseline screenshots instead of comparing. Defaults to false."
+      required: false
+mcp:
+  tool: run_skill
+  input:
+    skill: harness-visual-regression
+    path: string
+type: rigid
+tier: 3
+internal: false
+keywords:
+  - visual regression
+  - screenshot
+  - visual diff
+  - Chromatic
+  - Percy
+  - Pixelmatch
+  - Storybook
+  - baseline
+  - snapshot
+  - visual testing
+  - CSS regression
+stack_signals:
+  - "visual-tests/"
+  - ".chromatic/"
+  - "percy.yml"
+  - ".storybook/"
+  - "tests/visual/"
+  - "screenshots/"
+phases:
+  - name: detect
+    description: Identify UI components, existing visual tests, and rendering infrastructure
+    required: true
+  - name: baseline
+    description: Capture or verify baseline screenshots for target components and viewports
+    required: true
+  - name: compare
+    description: Run visual diff against baselines and classify changes as intentional or regression
+    required: true
+  - name: report
+    description: Generate visual diff report with annotated screenshots and approval workflow
+    required: true
+state:
+  persistent: false
+  files: []
+depends_on: []