@vizzly-testing/cli 0.11.0 → 0.11.2

package/claude-plugin/skills/debug-visual-regression/SKILL.md

@@ -0,0 +1,269 @@
+---
+name: Debug Visual Regression
+description: Analyze visual regression test failures in Vizzly. Use when the user mentions failing visual tests, screenshot differences, visual bugs, diffs, or asks to debug/investigate/analyze visual changes. Works with both local TDD and cloud modes.
+allowed-tools: Read, WebFetch, mcp__plugin_vizzly_vizzly__read_comparison_details, mcp__plugin_vizzly_vizzly__search_comparisons, mcp__plugin_vizzly_vizzly__accept_baseline, mcp__plugin_vizzly_vizzly__approve_comparison, mcp__plugin_vizzly_vizzly__reject_comparison
+---
+
+# Debug Visual Regression
+
+Automatically analyze visual regression failures when the user mentions them. This Skill helps identify the root cause of visual differences and suggests whether to accept or fix changes.
+
+## When to Use This Skill
+
+Activate this Skill when the user:
+- Mentions "failing visual test" or "screenshot failure"
+- Asks "what's wrong with my visual tests?"
+- Says "the homepage screenshot is different" or similar
+- Wants to understand why a visual comparison failed
+- Asks to "debug", "analyze", or "investigate" visual changes
+- Mentions specific screenshot names that are failing
+
+## How This Skill Works
+
+1. **Automatically detect the mode** (local TDD or cloud)
+2. **Fetch comparison details** using the screenshot name or comparison ID
+3. **View the actual images** to perform visual analysis
+4. **Provide detailed insights** on what changed and why
+5. **Suggest next steps** (accept, reject, or fix)
+
+## Instructions
+
+### Step 1: Call the Unified MCP Tool
+
+Use `read_comparison_details` with the identifier:
+- Pass screenshot name (e.g., "homepage_desktop") for local mode
+- Pass comparison ID (e.g., "cmp_abc123") for cloud mode
+- The tool automatically detects which mode to use
+- Returns a response with `mode` field indicating local or cloud
+
+### Step 2: Check the Mode in Response
+
+The response will contain a `mode` field:
+- **Local mode** (`mode: "local"`): Returns filesystem paths (`baselinePath`, `currentPath`, `diffPath`)
+- **Cloud mode** (`mode: "cloud"`): Returns URLs (`baselineUrl`, `currentUrl`, `diffUrl`)
+
+### Step 3: Analyze Comparison Data
+
+Examine the comparison details:
+- Diff percentage and threshold
+- Status (failed/new/passed)
+- Image references (paths or URLs depending on mode)
+- Viewport and browser information
+
+### Step 4: View the Actual Images
+
+**CRITICAL:** You MUST view the baseline and current images to provide accurate analysis.
+
+**If mode is "local":**
+- Response contains filesystem paths (`baselinePath`, `currentPath`, `diffPath`)
+- **Use the Read tool to view ONLY baselinePath and currentPath**
+- **DO NOT read diffPath** - it causes API errors
+
+**If mode is "cloud":**
+- Response contains public URLs (`baselineUrl`, `currentUrl`, `diffUrl`)
+- **Use the WebFetch tool to view ONLY baselineUrl and currentUrl**
+- **DO NOT fetch diffUrl** - it causes API errors
+
+### Step 5: Provide Detailed Visual Insights
+
+Based on what you observe in the images:
+
+**Describe the specific visual differences:**
+- Which UI components, elements, or layouts changed
+- Colors, spacing, typography, positioning changes
+- Missing or added elements
+
+**Categorize the change by diff percentage:**
+- **< 1%:** Anti-aliasing, font rendering, subpixel differences
+- **1-5%:** Layout shifts, padding/margin changes, color variations
+- **> 5%:** Significant layout changes, missing content, major visual updates
+
+**Identify possible causes:**
+- CSS changes (margin, padding, positioning)
+- Content changes (text, images)
+- State issues (hover, focus, loading states)
+- Browser/viewport rendering differences
+
+### Step 6: Suggest Next Steps
+
+**If local mode:**
+- Whether to accept using `accept_baseline` tool
+- Specific code areas to investigate if unintentional
+- How to fix common issues
+
+**If cloud mode:**
+- Whether to approve using `approve_comparison` tool
+- Whether to reject using `reject_comparison` tool with reason
+- Team coordination steps
+
+**If changes are intentional:**
+- Explain why it's safe to accept/approve
+- Confirm this matches expected behavior
+
+**If changes are unintentional:**
+- Specific files to check (CSS, templates, components)
+- Git commands to investigate recent changes
+- How to reproduce and fix
+
+## Example Analysis (Local TDD Mode)
+
+```
+User: "The homepage screenshot is failing"
+
+Step 1: Call tool
+Tool: read_comparison_details({ identifier: "homepage" })
+
+Response:
+{
+  "name": "homepage",
+  "status": "failed",
+  "diffPercentage": 2.3,
+  "threshold": 0.1,
+  "mode": "local",
+  "baselinePath": "/Users/you/project/.vizzly/baselines/homepage.png",
+  "currentPath": "/Users/you/project/.vizzly/screenshots/homepage.png"
+}
+
+Step 2: View images
+Read(baselinePath)
+Read(currentPath)
+
+Step 3: Visual Analysis
+Comparing the two images, I can see the following changes:
+
+Visual Differences:
+- The navigation header has shifted down by approximately 10-15 pixels
+- Logo position moved from y:20px to y:35px
+- Navigation menu items are now overlapping with the hero section
+- The "Sign Up" button background changed from blue (#2563eb) to darker blue (#1e40af)
+
+Root Cause Analysis:
+Based on the 2.3% diff and visual comparison, this appears to be:
+1. A margin or padding change on the header element
+2. The button color change suggests a hover state being captured
+
+Recommendations:
+If intentional:
+- This appears to be a deliberate redesign of the header spacing
+- Accept as new baseline if these changes match your design updates
+
+If unintentional:
+- Check recent CSS changes to `.header` or `nav` margin-top/padding-top
+- Verify button states during screenshot capture to avoid hover state
+- Investigate with: `git diff src/styles/header.css`
+- Fix the header positioning and re-run tests
+```
+
+## Example Analysis (Cloud Mode)
+
+```
+User: "Can you check comparison cmp_xyz789?"
+
+Step 1: Call tool
+Tool: read_comparison_details({
+  identifier: "cmp_xyz789",
+  apiToken: "vzt_..."
+})
+
+Response:
+{
+  "name": "login-page",
+  "status": "failed",
+  "diffPercentage": 0.8,
+  "threshold": 0.1,
+  "mode": "cloud",
+  "baselineUrl": "https://app.vizzly.dev/screenshots/abc123/baseline.png",
+  "currentUrl": "https://app.vizzly.dev/screenshots/abc123/current.png",
+  "comparisonId": "cmp_xyz789"
+}
+
+Step 2: View images
+WebFetch(baselineUrl, "show me the baseline screenshot")
+WebFetch(currentUrl, "show me the current screenshot")
+
+Step 3: Visual Analysis
+The differences are very subtle (0.8% diff):
+
+Visual Differences:
+- Input field border color slightly different (#e5e7eb vs #d1d5db)
+- Submit button has very slight shadow change
+- Font anti-aliasing appears slightly different
+
+Root Cause Analysis:
+This appears to be browser rendering differences or minor CSS updates:
+- The 0.8% diff suggests this is not a major visual change
+- Could be Chrome version differences
+- Could be sub-pixel rendering variations
+
+Recommendations:
+This is likely acceptable to approve:
+- The changes are extremely minor (< 1%)
+- No functional UI changes
+- Probably browser/rendering variations
+
+Approve with: approve_comparison tool
+Or adjust threshold to 1% if these variations are expected
+```
+
+## Cross-Build Debugging (Cloud Only)
+
+When debugging visual regressions in cloud mode, you can track a screenshot across multiple builds to identify when changes were introduced.
+
+### When to Use Search
+
+Use `search_comparisons` when:
+- The user asks "when did this screenshot start failing?"
+- They want to track a visual change across builds
+- They're investigating a regression that appeared recently
+- They want to see the history of a specific screenshot
+
+### How to Search
+
+```javascript
+// Search for all comparisons of a screenshot
+search_comparisons({
+  name: "homepage_desktop",
+  branch: "main",  // optional: filter by branch
+  limit: 10        // optional: limit results
+})
+```
+
+### Example Workflow
+
+```
+User: "When did the homepage screenshot start failing?"
+
+Step 1: Search for the screenshot across builds
+Tool: search_comparisons({ name: "homepage", branch: "main", limit: 10 })
+
+Response shows 10 comparisons from most recent to oldest, each with:
+- Build name, branch, and creation date
+- Diff percentage and status
+- Screenshot URLs
+
+Step 2: Analyze the timeline
+- Build #45 (today): 5.2% diff - FAILED
+- Build #44 (yesterday): 5.1% diff - FAILED
+- Build #43 (2 days ago): 0.3% diff - PASSED
+- Build #42 (3 days ago): 0.2% diff - PASSED
+
+Step 3: Report findings
+"The homepage screenshot started failing between Build #43 and #44
+(approximately 1-2 days ago). The diff jumped from 0.3% to 5.1%,
+suggesting a significant visual change was introduced."
+
+Step 4: Deep dive into the failing build
+Tool: read_comparison_details({ identifier: "cmp_xyz_from_build44" })
+[View images and provide detailed analysis as usual]
+```
+
+## Important Notes
+
+- **Always use `read_comparison_details`** - it automatically detects the mode
+- **Use `search_comparisons` for cloud debugging** - to track changes across builds
+- **Check the `mode` field** to know which viewing tool to use (Read vs WebFetch)
+- **Never view diff images** - only baseline and current
+- **Visual inspection is critical** - don't rely solely on diff percentages
+- **Be specific in analysis** - identify exact elements that changed
+- **Provide actionable advice** - specific files, commands, or tools to use
+- **Consider context** - small diffs might be acceptable, large ones need investigation

package/dist/plugin-loader.js

@@ -131,21 +131,16 @@ async function loadPlugin(pluginPath) {
 
 /**
  * Zod schema for validating plugin structure
+ * Note: Using passthrough() to allow configSchema without validating its structure
+ * to avoid Zod version conflicts when plugins have nested config objects
  */
 const pluginSchema = z.object({
   name: z.string().min(1, 'Plugin name is required'),
   version: z.string().optional(),
   register: z.custom(val => typeof val === 'function', {
     message: 'register must be a function'
-  }),
-  configSchema: z.record(z.any()).optional()
-});
-
-/**
- * Zod schema for validating plugin config values
- * Supports: string, number, boolean, null, arrays, and nested objects
- */
-const configValueSchema = z.lazy(() => z.union([z.string(), z.number(), z.boolean(), z.null(), z.array(configValueSchema), z.record(configValueSchema)]));
+  })
+}).passthrough();
 
 /**
  * Validate plugin has required structure
@@ -157,11 +152,8 @@ function validatePluginStructure(plugin) {
     // Validate basic plugin structure
     pluginSchema.parse(plugin);
 
-    // If configSchema exists, validate it contains valid config values
-    if (plugin.configSchema) {
-      let configSchemaValidator = z.record(configValueSchema);
-      configSchemaValidator.parse(plugin.configSchema);
-    }
+    // Skip deep validation of configSchema to avoid Zod version conflicts
+    // configSchema is optional and primarily for documentation
   } catch (error) {
     if (error instanceof z.ZodError) {
       let messages = error.issues.map(e => `${e.path.join('.')}: ${e.message}`);

package/dist/services/api-service.js

@@ -128,7 +128,37 @@ export class ApiService {
    * @returns {Promise<Object>} Comparison data
    */
   async getComparison(comparisonId) {
-    return this.request(`/api/sdk/comparisons/${comparisonId}`);
+    let response = await this.request(`/api/sdk/comparisons/${comparisonId}`);
+    return response.comparison;
+  }
+
+  /**
+   * Search for comparisons by name across builds
+   * @param {string} name - Screenshot name to search for
+   * @param {Object} filters - Optional filters (branch, limit, offset)
+   * @param {string} [filters.branch] - Filter by branch name
+   * @param {number} [filters.limit=50] - Maximum number of results (default: 50)
+   * @param {number} [filters.offset=0] - Pagination offset (default: 0)
+   * @returns {Promise<Object>} Search results with comparisons and pagination
+   */
+  async searchComparisons(name, filters = {}) {
+    if (!name || typeof name !== 'string') {
+      throw new VizzlyError('name is required and must be a non-empty string');
+    }
+    let {
+      branch,
+      limit = 50,
+      offset = 0
+    } = filters;
+    const queryParams = new URLSearchParams({
+      name,
+      limit: String(limit),
+      offset: String(offset)
+    });
+
+    // Only add branch if provided
+    if (branch) queryParams.append('branch', branch);
+    return this.request(`/api/sdk/comparisons/search?${queryParams}`);
   }
 
   /**

package/dist/services/tdd-service.js

@@ -132,10 +132,35 @@ export class TddService {
           logger.warn(`⚠️  Build ${buildId} has status: ${baselineBuild.status} (expected: completed)`);
         }
       } else if (comparisonId) {
-        // Use specific comparison ID
+        // Use specific comparison ID - download only this comparison's baseline screenshot
         logger.info(`📌 Using comparison: ${comparisonId}`);
         const comparison = await this.api.getComparison(comparisonId);
-        baselineBuild = comparison.baselineBuild;
+
+        // A comparison doesn't have baselineBuild directly - we need to get it
+        // The comparison has baseline_screenshot which contains the build_id
+        if (!comparison.baseline_screenshot) {
+          throw new Error(`Comparison ${comparisonId} has no baseline screenshot. This comparison may be a "new" screenshot with no baseline to compare against.`);
+        }
+
+        // The original_url might be in baseline_screenshot.original_url or comparison.baseline_screenshot_url
+        let baselineUrl = comparison.baseline_screenshot.original_url || comparison.baseline_screenshot_url;
+        if (!baselineUrl) {
+          throw new Error(`Baseline screenshot for comparison ${comparisonId} has no download URL`);
+        }
+
+        // For a specific comparison, we only download that one baseline screenshot
+        // Create a mock build structure with just this one screenshot
+        baselineBuild = {
+          id: comparison.baseline_screenshot.build_id || 'comparison-baseline',
+          name: `Comparison ${comparisonId.substring(0, 8)}`,
+          screenshots: [{
+            id: comparison.baseline_screenshot.id,
+            name: comparison.baseline_name || comparison.current_name,
+            original_url: baselineUrl,
+            metadata: {},
+            properties: {}
+          }]
+        };
       } else {
         // Get the latest passed build for this environment and branch
         const builds = await this.api.getBuilds({
@@ -152,10 +177,12 @@ export class TddService {
         baselineBuild = builds.data[0];
       }
 
-      // For specific buildId, we already have screenshots, otherwise get build details
+      // For specific buildId, we already have screenshots
+      // For comparisonId, we created a mock build with just the one screenshot
+      // Otherwise, get build details with screenshots
       let buildDetails = baselineBuild;
-      if (!buildId) {
-        // Get build details with screenshots for non-buildId cases
+      if (!buildId && !comparisonId) {
+        // Get build details with screenshots for non-buildId/non-comparisonId cases
         const actualBuildId = baselineBuild.id;
         buildDetails = await this.api.getBuild(actualBuildId, 'screenshots');
       }

package/dist/types/services/api-service.d.ts

@@ -28,6 +28,20 @@ export class ApiService {
      * @returns {Promise<Object>} Comparison data
      */
     getComparison(comparisonId: string): Promise<any>;
+    /**
+     * Search for comparisons by name across builds
+     * @param {string} name - Screenshot name to search for
+     * @param {Object} filters - Optional filters (branch, limit, offset)
+     * @param {string} [filters.branch] - Filter by branch name
+     * @param {number} [filters.limit=50] - Maximum number of results (default: 50)
+     * @param {number} [filters.offset=0] - Pagination offset (default: 0)
+     * @returns {Promise<Object>} Search results with comparisons and pagination
+     */
+    searchComparisons(name: string, filters?: {
+        branch?: string;
+        limit?: number;
+        offset?: number;
+    }): Promise<any>;
     /**
      * Get builds for a project
      * @param {Object} filters - Filter options

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@vizzly-testing/cli",
-  "version": "0.11.0",
+  "version": "0.11.2",
   "description": "Visual review platform for UI developers and designers",
   "keywords": [
     "visual-testing",
@@ -48,7 +48,7 @@
     "bin",
     "dist",
     "docs",
-    ".claude-plugin",
+    "claude-plugin",
     "README.md",
     "LICENSE"
   ],

package/.claude-plugin/README.md

@@ -1,114 +0,0 @@
-# Vizzly Claude Code Plugin
-
-> Integrate Vizzly visual regression testing seamlessly into your Claude Code workflow
-
-This plugin brings Vizzly's powerful visual testing capabilities directly into Claude Code, helping you debug visual regressions, manage baselines, and integrate visual testing into your development workflow—all with AI assistance.
-
-## Installation
-
-### From GitHub (Recommended)
-
-```bash
-# Add the marketplace
-/plugin marketplace add vizzly-testing/vizzly-cli
-
-# Install the plugin
-/plugin install vizzly@vizzly-marketplace
-```
-
-### From Local Source
-
-```bash
-# Clone the repository
-git clone https://github.com/vizzly-testing/vizzly-cli.git
-cd vizzly-cli
-
-# Add the marketplace
-/plugin marketplace add ./.claude-plugin
-
-# Install the plugin
-/plugin install vizzly@vizzly-marketplace
-```
-
-## Features
-
-### 🔍 **TDD Status Checking**
-- `/vizzly:tdd-status` - Check current TDD status and comparison results
-- See failed/new/passed screenshot counts
-- Direct links to diff images and dashboard
-
-### 🐛 **Smart Diff Analysis**
-- `/vizzly:debug-diff <screenshot-name>` - Analyze visual regression failures
-- AI-powered analysis with contextual suggestions
-- Guidance on whether to accept or fix changes
-
-### 💡 **Screenshot Suggestions**
-- `/vizzly:suggest-screenshots` - Analyze test files for screenshot opportunities
-- Framework-specific code examples
-- Respect your test structure and patterns
-
-### ⚡ **Quick Setup**
-- `/vizzly:setup` - Initialize Vizzly configuration
-- Environment variable guidance
-- CI/CD integration help
-
-## MCP Server Tools
-
-The plugin provides an MCP server with direct access to Vizzly data:
-
-### Local TDD Tools
-- `detect_context` - Detect if using local TDD or cloud mode
-- `get_tdd_status` - Get current TDD comparison results
-- `read_comparison_details` - Detailed info for specific screenshot
-- `accept_baseline` - Accept a screenshot as new baseline
-- `reject_baseline` - Reject a baseline with reason
-
-### Cloud API Tools
-- `list_recent_builds` - List recent builds with filtering
-- `get_build_status` - Get build status with commit context
-- `get_comparison` - Get comparison details with screenshots
-- `approve_comparison` - Approve a comparison with comment
-- `reject_comparison` - Reject a comparison with reason
-- `create_build_comment` - Add comment to build
-
-## Authentication
-
-The plugin automatically uses your Vizzly authentication with the following priority:
-
-1. **Explicitly provided token** (via tool parameters)
-2. **Environment variable** (`VIZZLY_TOKEN`)
-3. **Project mapping** (configured via `vizzly project:select`)
-4. **User access token** (from `vizzly login`)
-
-### Getting Started
-
-**For local development:**
-```bash
-vizzly login                # Authenticate with your Vizzly account
-vizzly project:select       # Optional: set project-specific token
-```
-
-**For CI/CD:**
-```bash
-export VIZZLY_TOKEN=vzt_your_project_token
-```
-
-The plugin will automatically use the appropriate token based on your context.
-
-## Requirements
-
-- Claude Code
-- Node.js 20+
-- Vizzly CLI (`@vizzly-testing/cli`) installed in your project
-- TDD mode running for local features
-- Authentication configured (see above) for cloud features
-
-## Documentation
-
-- [Vizzly CLI](https://github.com/vizzly-testing/vizzly-cli) - Official CLI documentation
-- [Vizzly Platform](https://vizzly.dev) - Web dashboard and cloud features
-- [Claude Code](https://claude.com/claude-code) - Claude Code documentation
-
-## License
-
-MIT © Vizzly Team